Yes - Java desktop and server applications are basically secure.
When you run a desktop application - Skype, Picassa, whatever - you give that software full access to your computer. You have to trust the software.
In contrast, when you run a Java applet in your web browser, the applet runs in a restricted environment called a sandbox. The sandbox exists so you do not have to trust the Java applet.
Java has had a lot of vulnerabilities; almost all of them are "sandbox escapes". In other words, if you're running an old version of Java, a malicious applet can break out of the sandbox and take control of your computer.
Not many technologies support sandboxes. In fact, there are only three common technologies where people routinely run untrusted software: Java, JavaScript and Flash. All of these have had many sandbox escape vulnerabilities, which demonstrates the difficulty of writing a secure sandbox.
When you run Java on your desktop, or on a server, you trust the Java code you are running, so you are not relying on the sandbox. In that context the main concern is whether untrusted data can interfere with the application. For example, if you're talking to someone on Skype, could they send a malicious message that Skype mishandles and allows them to take control of your computer. (I'm just using Skype as an example here).
There have been very few instances where bugs in the Java runtime would allow a desktop or server application to be hacked. Typically this happens because of bugs in the application code, not Java itself.