Does the SSL record protocol encrypt the handshake protocol too?

Question

As I know the first communication protocol which establishes between client and server in SSL is handshaking protocol and as this site says:

The SSL record protocol is used to transfer any data within a session - both messages and other SSL protocols (for example the handshake protocol)...

My question is how does record protocol encrypts data between client and server before even their first communication protocol(before handshaking protocol)

for example, What keys does record protocol uses for encryption and how does it share this key with the client?

Answer 1

From RFC 8446, Section 5 (TLS 1.3, Record Protocol):

The TLS record protocol takes messages to be transmitted, fragments the data into manageable blocks, protects the records, and transmits the result. Received data is verified, decrypted, reassembled, and then delivered to higher-level clients.

The cipher for encrypting fragments at the Record layer can change over time. Initially a null cipher will be in use since no keys have been established. This effectively means that no encryption is applied, the "ciphertext" is the same as the original fragment.

The Handshake Protocol (carried over the Record Layer) initially starts using this null cipher. After the client and server share security parameters (using Client Hello and Server Hello handshake messages), they can start performing a key exchange to establish a shared secret. Following this, they will be able to switch the Record Layer to a more secure cipher based on the shared secret. In TLS 1.2 and earlier, this switch is initiated using the Change Cipher Spec message.

SSL/TLS has a bit of a history, and there are two types of key exchanges based on asymmetric cryptography. One is based on RSA encryption, another is based on Diffie-Hellman (DH). In TLS 1.3, only the DH key exchange mechanism remains.

With the RSA key exchange, the client generates a random premaster secret, encrypts it using the RSA public key from the certificate provided by the server, and sends the result to the server. The server will be able to decrypt the result using the corresponding private RSA key and therefore obtains the same premaster secret.

In case of a Diffie-Hellman key exchange, both parties generate a new key pair (a private key and a corresponding public key) and share the public key with each other. By combining their own private key and the peer's public key in a special way, they will obtain the same shared secret.

In either case, a shared secret is obtained which is transformed into a symmetric key. This key will be used for encrypting fragments at the Record Layer.

For a more extended treatment of TLS, see How does SSL/TLS work?