Beware that there are a few mistakes in what you think you know. You've probably been reading misinformed or outright wrong simplifications. I'll address a few below, but I recommend reading some good references with an open mind. Good references include TLS: every byte explained and How does SSL/TLS work?.
TLS version: TLS 1.2 CIPHER SUITE : TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (any cipher suite that involves creation of a pre-master secret)
Every cipher suite involves the creation of a pre-master secret apart from a few TLS ≤1.2 cipher suites that don't encrypt data (null cipher suites) and are very rarely used.
In terms of understanding a TLS handshake, the “mainstream” cipher suites are the ones that use a key agreement protocol to establish a one-time session key, and a signature mechanism to verify that the server is who it claims to be (and, with mutual TLS, also to verify that the client is who it claims to be). Those cipher suites have DHE or ECDHE in the name ((elliptic-curve) Diffie-Hellman, ephemeral (= one-time)), as well as a signature method (RSA, DSS, ECDSA, EDDSA).
TLS ≤1.2 has various other mechanisms that I won't go into. I'll just mention RSA encryption (RSA without any kind of DH in the cipher suite name), which is somewhat popular in tutorials because it's a bit simpler to explain, but is deprecated because it has security and performance downsides. Forget about RSA-encryption cipher suites unless you have a specific need to understand them. They don't exist in TLS 1.3 any more. There are also PSK (pre-shared key) cipher suites, even in TLS 1.3; these are only useful in special circumstances where the client and the server have a shared symmetric key, mostly for tiny embedded devices that don't have enough computing power for asymmetric cryptography, and, in TLS 1.3, for session resumption.
Note: I understand in TLS 1.3 , the handshakes are encrypted after Server Hello.
If that is the case and in case of TLS 1.3 a middle man cannot get the public key's shared during the handshake then, only scenario, I am looking at is, when an attacker breaks in to a device and gets the certificates.
Correct. However, this only matters for privacy: when you don't want attackers to know who is connecting to who. For the core security of the connection (preventing attackers from impersonating one side, or from knowing what data is exchanged), the public keys are considered public knowledge: any security analysis assumes that attackers already know the public keys.
Server verification is done using the Server CA's certificate by the client.
(Public key of CA is used to obtain the has from the sign and then this has is compared to be identical and validated).
This is correct with respect to the general flow of information, but not with respect to how the verification works. In the simple case where the server sends one certificate which is signed by a CA that the client trusts, what the client checks is:
- Is the certificate well-formed and does is have acceptable parameters?
- Is the server name in the certificate identical to the name that the client is trying to connect to?
- Is the signature in the certificate a correct signature made by the CA's private key?
Note that the signature verification process is not an equality comparison. Making a signature and verifying a signature are complementary processes. It would not be possible to verify a public-key signature by signing and comparing results, since the verifier does not have the private key, which is required to calculate the signature.
Client verification on the server is done using Client CA's certificate by the server.
This is the only place where the public key of the respective CA's are used.
Correct.
The client random is encrypted using the server's public key in the server's ssl certificate and sent to the server for decryption.
No. The client random is sent in plain text. It's the very first message sent by the client and no key has been established apart from the server's key pair, so this would only work if the server was capable of decryption, but servers that are up to modern standards only have signature keys. The point of the client random and the server random is to avoid replay attacks. There would be no point in keeping them secret.
Question 1.
So, where exactly is the client's public key used?
Is the client's ssl certificate shared,
limited to only verification and has no contribution to key exchange and thus encryption?
The client public key is only used to verify that the client is who it claims to be. Likewise, the server public key is only used to verify that the server is who it claims to be. The session keys (pre-master secret and derived symmetric keys) are derived from the Diffie-Hellman key exchange.
Question 2:
Given that the client certificate and server's certificate are exposed in the handshake,
even though it cannot be used for getting useful data,
can the info in the certs (like CN,public key) be used to do DDoS kind of attacks?
The certificates are assumed to be public, at least for security properties other than privacy, so they aren't an attack concern, DoS or otherwise.
Regarding privacy, anybody can obtain a server's certificate by initiating a TLS connection to the server. In TLS ≤1.2, if a client sends a certificate as soons as it connects, this is also transmitted in clear text. In TLS 1.3, the latter part of the handshake is encrypted, so the client's certificate is private.
Note that when a client contacts a server machine that hosts multiple sites, it sends the name of the server it wants in a TLS protocol message called SNI. In TLS ≤1.2, a passive observer can see this information. In TLS 1.3, the SNI can be encrypted (this protects at least against passive attacks, I'm not sure how good it is against active attacks).
Question 3: Given that the scenario addresses a self signed certificate,
if an attacker manages to get the CA's certificate on either the client/server,
can this be used to perform any kind of attacks?
This is a strange question: if the certificate is self-signed then there is no CA.
Assuming there is a CA, no. In most realistic scenarios, the attacker already knows the CA's certificate anyway. Anybody can just connect to the server to find out which CA it used: that information is in the server's certificate. The client CA can be private in TLS 1.3, like the client certificate, but usually it's an organization CA which the attacker would have plenty of ways of obtaining anyway.
But crux is , are there any attacks that can be done using the public keys alone? Be it the public key of the CA's , Client public key, Server's Public Key
As I explained above, the only such attacks are attacks on privacy.
If that is the case, can this be solved by:
1.sending an encrypted version of the certificate in the handshake such that no middle man can actually access it.
TLS 1.3 indeed encrypts part of the handshake for privacy.
- locking the CA certs inside HSM modules etc.
Putting certificates in an HSM has no security benefit. The main reason to put a certificate in an HSM is if the HSM holds the corresponding private key, because it's convenient to store certificate(s) for the private key close to the private key.
(Note that some software documentation confusingly uses “certificate” to mean a bundle containing both a private key and a certificate for that key. I strongly recommend not to use this terminology because it's hard to reason about security properties when the same word is used for public and private information.)
I somewhat feel, the idea where public key can be used to perform an attack is unwarranted, but I am not that aware of different possible attacks.
The main attacks related to public keys are of a different type: sending a malformed public key or certificate or other message containing a public key, and hoping to trigger a bug in the software that processes this message. The quality of mainstream security-related software has improved to a point where such vulnerabilities are pretty rare. But they're still a possibility, especially on never-upgraded embedded devices (such as many network appliances).
