1

I'm trying to manually decrypt DTLS packets sent by OpenSSL.

The cipher suite used between the server and the client is TLS_RSA_WITH_AES_256_CBC_SHA256 and my understanding is that the IV used by the cipher is prepended to the encrypted application data. I also know the master secret used by the server.

Now to my question, how OpenSSL derives the encryption key from the master secret ?

turnl
  • 11
  • 2
  • 2
    Possible duplicate of [In PSK TLS, how is the key used for encryption derived?](http://security.stackexchange.com/questions/81597/in-psk-tls-how-is-the-key-used-for-encryption-derived) – RoraΖ Feb 03 '16 at 15:45
  • While the duplicate above is for PSK, the links to relevant RFC sections should still apply. – RoraΖ Feb 03 '16 at 15:45
  • 1
    81579 is for TLSv1.0, and is also correct for 1.1, but this Q is for 1.2 where key derivation (specifically PRF) is changed; see rfc 5246 instead. Also, you say 'the master secret' as if there is one, but the master secret is different for every session. If you mean you have the server's *private key*, you need to use that to decrypt the premaster secret, then derive the master secret, then derive the encryption keys and MAC keys (four total). If you only want the decrypt results (rather than learning to do it) just use wireshark. – dave_thompson_085 Feb 03 '16 at 18:27

1 Answers1

2

Meta: Doesn't add much to the dupe but on rereading I notice I missed the "D".

Key establishment and encryption/decryption and authentication processes for TLSv1.2 are specified in RFC 5246 TLSv1.2 plus for ECC key exchanges (not here) RFC 4492 TLS ECC as amended by appendix A.7 of 5246. These processes also apply to DTLSv1.2 RFC 6347 DTLSv1.2, although the record headers, actual transmission and some error handling differs for DTLS compared to TLS.

SSL/TLS/DTLS key establishment has two parts: one which varies based on (only) the key-exchange algorithm used and produces a "premaster secret", and one which varies based on (only) the bulk-cipher and MAC algorithm(s)/mode selected and uses the premaster secret to produce a master secret and then the several working keys, IVs and nonces. The encryption/decryption and authentication processes vary similarly.

For TLS_RSA_WITH_AES_256_CBC_SHA256 the key-exchange algorithm is RSA. The premaster secret is chosen by the client, RSA encrypted under the server's public key and sent to the server which RSA decrypts it and checks it. See in 7.4.2 the item for RSA, 7.4.7.1, and 8.1.1. To do this outside the server you need (a copy of) the server's private key.

The premaster secret is then used with the nonces to derive the master secret as specified in 8.1 using the PRF (PseudoRandom Function) defined in sec 5; this is then used again with the nonces to derive working keys as specified in 6.3 using the same PRF for the sizes in appendix C. For TLS_RSA_WITH_AES_256_CBC_SHA256 the cipher is AES_256_CBC which requires a key of 32 octets in each direction and the MAC is HMAC-SHA256 with a key of 32 octets in each direction.

Each data record is MACed and encrypted by the sender, and decrypted and verified by the receiver. The MAC/verify process is specified by 6.2.3.1 (for all non-AEAD ciphers/suites, and this suite is not AEAD) and the encryption/decryption process for CBC-mode (block) cipher is specified by 6.2.3.2; it does indeed choose a per-record IV and prepend it to the encrypted record data.

For another ciphersuite the outline is the same but in some places different specific sections of RFC 5246, and for ECC-based key exchange (ECDH, ECDHE, or ECDH_anon) RFC 4492, apply.

Community
  • 1
dave_thompson_085
  • 10,064
  • 1
  • 26
  • 29