Encrypting a salted password hash with RSA before storing in a database

Question

Why is it a bad idea to encrypt a salted password hash with RSA (or maybe other public-key algorithm) before storing it?

Answer 1

As you mentioned in a previous question, the goal of encrypting a hashed password is to add pepper to it, to make it harder (even impossible when a good pepper is used) to brute-force passwords, in the event that the database is breached.

Adding a pepper in a password-hashing function and symmetrically encrypting the hash are functionally equivalent. To obtain the recommended level of cryptographic security, the pepper or the key should be at least 128 bit long and chosen from a cryptographic strong uniformly random source.

Adding the random key as a pepper in the hashing function provides benefits over using the symmetric encryption of the hash:

• The key does not need to be derived to ensure it conforms to its requirements by the encryption algorithm. The hashing function will take care of that.
• It is less computationally expensive, while providing exactly the same amount of security. Moreover, most encryption libraries are made to encrypt messages of varying length, while using the fixed length of the hash equal to the block size of the encryption algorithm could allow optimizations (no padding for example). So, using a encryption library could introduce an useless overhead, even though it's better to use a library rather than rolling your own cryptography.
• It avoids to make mistakes in the use of encryption: no padding and no random nonce to handle. No weak choice of encryption algorithm possible.
• It reduces the surface of attack and the maintenance cost: by not using yet another library, one avoids the potential bugs in it and the cost to keep it up-to-date.

Using asymmetric encryption, such as RSA, adds the following drawbacks over just hashing with a pepper:

• It still needs a symmetric cipher, so all the previous points about symmetric encryption applies here.
• The recipient and sender are the same: it's the server. Both the public and private keys are in the same hands. Asymmetric encryption is a functionally useless layer over symmetric encryption here. See nobody's answer for more details on this point.
• It adds yet another algorithm, one that is far more error prone than symmetric encryption, as history has proven. Even cryptographers often make mistakes when using asymmetric encryption.
• Asymmetric encryption is rather inefficient (read "slow"). Especially RSA, which is very slow and needs a very long key.
• No optimization possible: the use of random nonce and correct padding handling is unavoidable. This is hard and a lot of cryptographic library fail here. Do not assume your code is safe even when correctly using a library: most continue to use PKCS#1 v1.5 padding, even though it has been broken for more than 20 years. That's only the most common vulnerability, but others are often found.

On the other hand, in my knowledge, encrypting hashed passwords offers no real benefit over adding pepper; assuming the pepper is handled as a symmetric cryptographic key.

Claimed benefits cited by the OP in his other question are:

• From stackoverflow and from security.SE: One can rotate the encryption keys. However, once the database is breached, that is useless: either the encryption is good, and changing the passwords or rotating the key is useless; or the encryption is weak and the passwords need to be changed.
• From stackoverflow: Encryption is more secure than hashing. That's wrong.
• From stackoverflow "since he knows the salt and the output, he can brute force the pepper" The same applies with encryption. This is not an issue with a pepper handled as a key.

Answer 2

First, RSA is an asymmetric algorithm and there is no point of using an asymmetric algorithm here. Asymmetric cryptography is used when you need to exchange information with somebody you haven't established a shared key with.

RSA is rarely used to encrypt data directly (mostly its used to encrypt a key for a symmetric cipher which is what is actually used to encrypt data), due to performance constraints and the fact that you can only encrypt a limited amount of data with it (although hashes will probably fit within that limit). Also, quantum computers, although still a long way away, would completely break RSA (and some other asymmetric schemes too). So when quantum computers do materialize, you would have to change your whole scheme.

Using an asymmetric algorithm in this case won't add any security to your scheme over a symmetric algorithm. 128 Bit AES is stronger than 2048 bit RSA so it doesn't make sense to use the less efficient and more fragile approach.