When generating SSH authentication keys on a Unix/Linux system with ssh-keygen, you're given the choice of creating a RSA or DSA key pair (using -t type).
What is the difference between RSA and DSA keys? What would lead someone to choose one over the other?
Go with RSA.
DSA is faster for signature generation but slower for validation, slower when encrypting but faster when decrypting and security can be considered equivalent compared to an RSA key of equal key length. That's the punch line, now some justification.
The security of the RSA algorithm is based on the fact that factorization of large integers is known to be "difficult", whereas DSA security is based on the discrete logarithm problem. Today the fastest known algorithm for factoring large integers is the General Number Field Sieve, also the fastest algorithm to solve the discrete logarithm problem in finite fields modulo a large prime p as specified for DSA.
Now, if the security can be deemed as equal, we would of course favour the algorithm that is faster. But again, there is no clear winner.
You may have a look at this study or, if you have OpenSSL installed on your machine, run openssl speed. You will see that DSA performs faster in generating a signature but much slower when verifying a signature of the same key length. Verification is generally what you want to be faster if you deal e.g. with a signed document. The signature is generated once - so it's fine if this takes a bit longer - but the document signature may be verified much more often by end users.
Both do support some form of encryption method, RSA out of the box and DSA using an El Gamal. DSA is generally faster in decryption but slower for encryption, with RSA it's the other way round. Again you want decryption to be faster here because one encrypted document might be decrypted many times.
In commercial terms, RSA is clearly the winner, commercial RSA certificates are much more widely deployed than DSA certificates.
But I saved the killer argument for the end: man ssh-keygen says that a DSA key has to be exactly 1024 bits long to be compliant with NIST's FIPS 186-2. So although in theory longer DSA keys are possible (FIPS 186-3 also explicitly allows them) you are still restricted to 1024 bits. And if you take the considerations of this [article], we are no longer secure with 1024 bits for either RSA or DSA.
So today, you are better off with an RSA 2048 or 4096 bit key.
Right now the question is a bit broader: RSA vs. DSA vs. ECDSA vs. Ed25519. So:
A presentation at BlackHat 2013 suggests that significant advances have been made in solving the problems on complexity of which the strength of DSA and some other algorithms is founded, so they can be mathematically broken very soon. Moreover, the attack may be possible (but harder) to extend to RSA as well.
The presentation suggests using elliptic curve cryptography instead. The ECC algorithms supported by OpenSSH are ECDSA and, since OpenSSH 6.5, Ed25519.
OpenSSH only supports NIST curves for ECDSA and according to this study those curves look really suspicious for NSA backdoors. And if NSA can already crack it, then it won't be as hard to crack for somebody else as a proper curve would be. Ed25519 is the same thing but with a better curve, so it's the safest bet against the underlying algorithm being mathematically broken.
Also, DSA and ECDSA have a nasty property: they require a parameter usually called k to be completely random, secret, and unique. In practice that means that if you connect to your server from a machine with a poor random number generator and e.g. the the same k happens to be used twice, an observer of the traffic can figure out your private key. (source: Wikipedia on DSA and ECDSA, also this).
The bottom line is:
In SSH, on the client side, the choice between RSA and DSA does not matter much, because both offer similar security for the same key size (use 2048 bits and you will be happy).
Historically, version 1 of the SSH protocol supported only RSA keys. When version 2 was defined, RSA was still patented, so support of DSA was added, so that an opensource patent-free implementation could be made. RSA patent expired more than 10 years ago, so there is no worry now.
Theoretically, in some very specific situations, you can have a performance issue with one or the other: if the server is a very small machine (say, an i486), it will prefer clients with RSA keys, because verifying a RSA signature is less computationally expensive than verifying a DSA signature. Conversely, a DSA signature is shorter (typically 64 bytes vs 256) so if you are very short on bandwidth you would prefer DSA. Anyway, you will have a hard time detecting those effects, let alone find them important.
On the server, a DSA key is preferred, because then the key exchange will use a transient Diffie-Hellman key, which opens the road for "Perfect Forward Secrecy" (i.e. if a bad guy steals the server private key, he still cannot decrypt past connections that he would have recorded).
Another important advantage of RSA over both DSA and ECDSA is that you don't ever need a secure random number generator to create signatures.
To generate a signature, (EC)DSA needs a value that has to be random, secret/unpredictable and can never be used again. If one of those properties is violated, it's possible to trivially recover the private key from one or two signatures.
This has happened before (once with a broken patch of the OpenSSL PRNG in Debian, and once with a bug in Android's SecureRandom implementation), and is pretty hard to completely prevent.
With RSA, in those situations only your ephemeral session key would have been compromised, if the actual authentication key pairs have been created using a properly seeded PRNG before.
Theoretically, there is a way to make (EC)DSA signatures depend on the message and private key, which can avoid total failure in case of a broken RNG, but until this gets integrated into your SSH client (there is no OpenSSL release version including the patch right now), I'd definitely go with RSA keys.
Not knowing much about cryptography, as a OpenSSH user I would stick to a simple fact: in http://www.openssh.com/legacy.html, which states that SHA1 is now disabled by default because it is considered weak:
OpenSSH 7.0 and greater similarly disables the ssh-dss (DSA) public key algorithm. It too is weak and we recommend against its use.
The math might not matter. This thread seems pre-Snowden. Here is a Reuters article dated December 20, 2013:
(Reuters) - As a key part of a campaign to embed encryption software that it could crack into widely used computer products, the U.S. National Security Agency arranged a secret $10 million contract with RSA, one of the most influential firms in the computer security industry, Reuters has learned.
Documents leaked by former NSA contractor Edward Snowden show that the NSA created and promulgated a flawed formula for generating random numbers to create a "back door" in encryption products, the New York Times reported in September. Reuters later reported that RSA became the most important distributor of that formula by rolling it into a software tool called Bsafe that is used to enhance security in personal computers and many other products.
Undisclosed until now was that RSA received $10 million in a deal that set the NSA formula as the preferred, or default, method for number generation in the BSafe software, according to two sources familiar with the contract. Although that sum might seem paltry, it represented more than a third of the revenue that the relevant division at RSA had taken in during the entire previous year, securities filings show.
During this Science Friday podcast Ira asks Matt Green, Martin Hellman (inventor of Public Key Cryptography), and Phil Zimmerman (creator of PGP) what they think the NSA has cracked:
(around 17:26)
Ira: What are some of the things that we know that the NSA has broken into?
Matt: So we have heard a number of things that we can probably credit for real. … random number generators … we know that NSA through NIST … has very likely put back doors in some of those standard algorithms that allow them to essentially break those systems entirely.
Ira: You mean the NSA created those back doors?
Matt: That's exactly right. So NIST works with NSA --- and they're required to by law. We thought NSA was helping NIST by developing more secure standards for Americans to use. We now suspect --- and have strong evidence to believe --- that the situation was exactly the opposite; that NIST was being used to put out standards that the NSA could break.
Considering these recent revelations the strength of the algorithms seems largely irrelevant. RSA appears to have been a private company somehow bought by the NSA, and DSA was created by NIST itself, which, according to these experts is largely a front for NSA crypto research.
In other words, it really doesn't matter if you are using the random number generators that come with pretty much any modern computer, which OpenSSH and others do.
Pick the one that is the fastest for what you want. In my case, I reuse the same key for a lot of stuff so DSA's faster generation speed is less desirable. Also, maybe there is some wild chance that RSA was actually an independent entity from NIST and NSA, whereas we know DSA was created by NIST.
I personally just use 1028 bit keys because, as we've seen, it really doesn't matter unless someone is placing some requirement on you who still believes bigger keys will protect you. The whole thing is largely just an annoyance to anyone seeking to break in.
RSA and DSA are two completely different algorithms. RSA keys can go up to 4096 bits, where DSA has to be exactly 1024 bits (although OpenSSL allows for more.) According to Bruce Schneier, "both DSA and RSA with the same length keys are just about identical in difficulty to crack."
The problem with ECDSA is not so much backdoors. Bernstein/Lange specifically mention that curve cryptanalysis does not attack specific curves, but classes of curves (see slide 6).
The problem with ECDSA is that NIST curves are hard to implement correctly (ie. constant time and with all proper verification) compared to Curve25519. OpenSSL has a constant-time P256 implementation, so OpenSSH is safe in that regard.
If you're still worried about NIST curves, OpenSSH recently added support for the Ed25519 scheme
