Why can't we use transposition cipher on tls?

Question

I've been interested in tls and rsa algorithms but I saw an example of RSA here. If we have a certain value for each character, then why can't we use transposition cipher?

I mean if "a" is always a definite character (let's assume it becomes "d") after RSA, can't we just find it on our machine (that 'a' becomes 'd') and then since everyone uses the same public key, just substitute values of our finding?

Even if we use symmetric encryption afterwards, we should be able to log those encrypted text from our initial process (handshake) and then use it to find symmetric key. Then we should be able to mitm right? Please explain why this doesn't work.

My understanding is this: The server distributes the same public key to everyone. So, if we perform a handshake with the server then we'll know exactly what characters were transformed into what. Then we could use that knowledge to extract AES key and hence their data?

Answer 1

Based on your comments, it seems that you have misunderstood RSA; it is not a substitution cipher. If it were, you are correct that you could just reverse the process and get they key back out. While RSA encryption is deterministic (i.e. same inputs = same outputs) the use of such operations in TLS involve randomly generated values that should never repeat, or at least be statistically unlikely to repeat (on the order of 2^-128 or better).

RSA is an asymmetric cipher. The way it works internally is a long answer full of maths and I can't really explain it any better than the Wikipedia article does. The short answer is that RSA has two keys (public and private) and can do two things: encrypt, and sign. The most important features are as follows:

1. If you encrypt something with the public key, only the private key can decrypt it.
2. If you sign something with the private key, the public key can be used to verify that the signature is correct.

In older TLS cipher suite modes, we used something called "static RSA key exchange". Effectively what this means is that the client generates a session key and encrypts it with the RSA public key, which is in the certificate. Because of property 1 above, only someone in possession of the private key can decrypt that session key. The server has the private key so therefore it can decrypt the key. Now both the client and server know the same session key and can use it to encrypt the rest of the traffic with a symmetric cipher such as AES. Since the attacker does not have the private key they cannot decrypt the session key or any traffic.

In newer TLS cipher suite modes, we use a separate algorithm for key exchange, called Diffie-Hellman (DH for short). DH allows for two parties to agree upon a value securely, even if an attacker is watching the exchange. The problem with DH is that if an attacker can man-in-the-middle the exchange they pretend to be the other party and get access to the keys. We call this "anonymous DH" and it isn't secure. In order to fix this, we need to digitally sign the DH exchange. We can utilise property 2 in order to do so. The server generates its half of the DH exchange and signs it with the private key. The client can verify the signature using the public key. The client then encrypts its half of the DH exchange using the RSA key and gives it to the server. This prevents an attacker from interfering with the key exchange and prevents the exchanged key from being discovered.

I recommend reading this question for a better understanding of SSL/TLS in general: How does SSL/TLS work?