At its heart, this is a question about Control. You want to control access to this KeyStore file, so that only your application can access it but nobody else can. You don't want to put too few Controls on this, because that would open up the possibility of unauthorized access. Neither do you want to put too many controls on this, because that would be too cumbersome and too expensive to use.
To allow the application, and not one else, access to its KeyStore, you need to enable the application to assert an identity.
You have the following options:
- Put the passphrase to the KeyStore in a configuration file and have the application read this on startup. This allows you to control the identity of the application instance by manipulating file system permissions (the application user can read, but not write, and nobody else can read). If your KeyStore is based on only the KeyStore file, consider putting the OS permission controls on the KeyStore file and doing away with the passphrase altogether. If your adversary can impersonate your application user, they have root on your system and you have much bigger problems.
- When the application starts up, have someone type the passphrase to KeyStore on the console before the application loads the KeyStore. This obviously falls squarely into the 'cumbersome' category: it prevents automated startup, and it may be attacked by bribing the persons(s) who have to know the passphrase.
- Use a Hardware Security Module (HSM) to back the KeyStore so that subverting file system security on the application server has to be combined with a physical attack on the hosting facility before the keys can be used. Same considerations as above for the security of the credentials used to log into the HSM. This is where I have to disclose that I sell HSMs for a living.
- Use an HSM in combination with HSM-enforced access controls so that multiple operators have to provide a hardware credential (smart card) and type a passphrase on the system console before the application can start. This is the far end of 'cumbersome', but it protects against the bribery issue in 2. above.
Note that none of these are based on adding layers of encryption: it's all about enforcing Controls on access to the KeyStore contents, based on the identity of an application instance or authorization by one or more operators.
In the end, it all depends on how important that private key is, and what the consequences are when that private key is lost or compromised. Which Controls to use should be a business decision, based on your assessment of the risks and consequences.