OpenSSL dgst authentication failure

Question

I've tried to replicate the workflow presented on this blog in OpenSSL: http://farid.hajji.name/blog/2009/07/27/public-key-cryptography-with-openssl/

However, authentication seems to fail despite many variations. What's going wrong? Please see the code below you can copy and paste into OpenSSL. Note that I am using the compiled Windows binary version of OpenSSL.

//================Phase 1 - Setup================

//Generate my private key (myprivatekey.txt)
genpkey -algorithm RSA -out C:\myprivatekey.txt -pass pass:abc123 -pkeyopt rsa_keygen_bits:2048

//Generate friend's private key (friendprivatekey.txt)
genpkey -algorithm RSA -out C:\friendprivatekey.txt -pass pass:123abc -pkeyopt rsa_keygen_bits:2048

----------------

//Extract my public key (mypublickey.txt) from my private key (myprivatekey.txt)
rsa -passin pass:abc123 -in C:\myprivatekey.txt -pubout -out C:\mypublickey.txt

//Extract friend's public key (friendpublickey.txt) from my private key (friendprivatekey.txt)
rsa -passin pass:123abc -in C:\friendprivatekey.txt -pubout -out C:\friendpublickey.txt

----------------

//Generate my password (a random base64 string password saved mypassword.txt)
rand -base64 -out C:\mypassword.txt 128

//Generate friend's password (a random base64 string password saved to friendpassword.txt)
rand -base64 -out C:\friendpassword.txt 128

//Delete the .rnd file that's generated?  Not sure what it is.

----------------

//Encrypt my password using my private key (encrypted password saved to a binary file - myencryptedpassword.txt)
pkeyutl -in C:\mypassword.txt -out C:\myencryptedpassword.txt -inkey C:\myprivatekey.txt -passin pass:abc123

//Encrypt friend's password using friend's private key (encrypted password saved to a binary file - friendencryptedpassword.txt)
pkeyutl -in C:\friendpassword.txt -out C:\friendencryptedpassword.txt -inkey C:\friendprivatekey.txt -passin pass:123abc

----------------

//Convert my encrypted password to base64 from binary (saved as myencryptedpasswordbase64.txt)
base64 -in C:\myencryptedpassword.txt -out C:\myencryptedpasswordbase64.txt

//Convert friend's encrypted password to base64 from binary (saved as friendencryptedpasswordbase64.txt)
base64 -in C:\friendencryptedpassword.txt -out C:\friendencryptedpasswordbase64.txt

----------------

//Create a signed hash of my password so my friend knows it's coming from me (signed hash saved as mysignedhash.txt and is in binary form)
dgst -sha256 -sign C:\myprivatekey.txt -passin pass:abc123 -out C:\mysignedhash.txt C:\myencryptedpasswordbase64.txt

//Create a signed hash of friend's password so I know it's coming from my friend (signed hash saved as friendsignedhash.txt and is in binary form)
dgst -sha256 -sign C:\friendprivatekey.txt -passin pass:123abc -out C:\friendsignedhash.txt C:\friendencryptedpasswordbase64.txt

----------------

//Convert my signed hash from binary to base64
base64 -in C:\mysignedhash.txt -out C:\mysignedhashbase64.txt

//Convert friend's signed hash from binary to base64
base64 -in C:\friendsignedhash.txt -out C:\friendsignedhashbase64.txt

//================Phase 2 - Authentication================

//Now, we reverse the process and authenticate the friend.  Let's prefix all output files with "phase2"

//I provide friend with my public key and my encrypted password 
//Friend provides me with their public key

//Convert friend's encrypted password from base64 to binary.  The output file will be the same as friendsignedhash.txt
base64 -d -in C:\friendsignedhashbase64.txt -out C:\phase2friendsignedhash.txt

//Convert friend's signed hash from base64 to binary.  The output file will be the same as C:\friendsignedhash.txt
base64 -d -in C:\friendencryptedpasswordbase64.txt -out C:\phase2friendencryptedpassword.txt

//Verify if the password originates from my friend (by checking against my friend's public key)
dgst -sha256 -verify C:\friendpublickey.txt -signature C:\phase2friendsignedhash.txt -out C:\friendresult.txt C:\phase2friendencryptedpassword.txt

Any idea of why the verification failure occurs?

Answer 1

The immediate reason of the verification failure is that the signature was generated over friendencryptedpasswordbase64.txt, but you try to verify it over phase2friendencryptedpassword.txt, which has not the same contents. Signatures are computed over sequences of bytes and don't care whether you envision these bytes as the Base64-encoding of some other bytes. If you want the signature to be verifiable, you must use the same sequence of bytes as signed message for both generation and verification.

---

On the whole, what you are trying to do is unclear; in particular, what you call "Encrypt friend's password using friend's private key" does not do that. What you obtain with that command-line is not encryption; it rather is a half-signature (the input file, friendpassword.txt, is taken "as is" as if it was a hash value, embedded in a "PKCS#1 v1.5 type 1 padding", and subject to the RSA core modular exponentiation). In particular, using the value friendencryptedpassword.txt and the public key friendpublickey.txt, both of which being public (since they are sent "on the wire"), it is trivial to rebuild friendpassword.txt, and I bet that's not what you would want to happen.

If what you want is that user A conveys to user B some secret value V (e.g. a "password") so that eavesdropper cannot see V, but B has some guarantee that what was received is indeed a message from A, then what should happen is that:

• A encrypts the value V with B's public key, yielding W.
• A signs W with A's private key, yielding a signature S.
• A sends W and S to B.
• B verifies S over input W, using A's public key. This gives B some guarantee that W is indeed the value that A sent (it was not altered in transit by some malevolent interloper).
• B decrypts W with B's private key, yielding V.

Note that A keeps his private key private, and so does B. A never learns B's private key, and B never learns A's private key. Moreover, B's key pair is used for encryption and A's key pair for signatures. Encryption and signature are distinct activities which use distinct algorithms and distinct types of keys. It is an unfortunate source of confusion that there is an encryption algorithm called RSA, and also a signature algorithm called RSA, and both kinds of RSA can share the same key structure; even more confusingly, a lot of people erroneously refer to signatures as "encrypting with the private key", which is wrong and makes the overall picture especially obscure. You had better keep signatures and encryption separate (and there are good reasons for that).

---

There are a gazillion details which can go wrong at any point in the design and implementation of such protocols, so the safe way is to rely on standard protocols which have been carefully specified and analysed, and on tools which implement these protocols and have been thoroughly tested for correctness. To be brief, use GnuPG.

Answer 2

Latest corrected version:

//A is me
//B is friend
//V is my password
//W is my encrypted password
//S is my encrypted password's signature
//The purpose of using base64 encoding is for regular text-based transmission

----------------Phase I - Creating Keys, Password and Signature----------------

//Generate my private key (myprivatekey.txt)
genpkey -algorithm RSA -out C:\myprivatekey.txt -pass pass:abc123 -pkeyopt rsa_keygen_bits:2048

//Generate friend's private key (friendprivatekey.txt)
genpkey -algorithm RSA -out C:\friendprivatekey.txt -pass pass:123abc -pkeyopt rsa_keygen_bits:2048

----------------

//Extract my public key (mypublickey.txt) from my private key (myprivatekey.txt)
rsa -passin pass:abc123 -in C:\myprivatekey.txt -pubout -out C:\mypublickey.txt

//Extract friend's public key (friendpublickey.txt) from my private key (friendprivatekey.txt)
rsa -passin pass:123abc -in C:\friendprivatekey.txt -pubout -out C:\friendpublickey.txt

----------------

//Generate my password (a random base64 string password saved mypassword.txt)
rand -base64 -out C:\mypassword.txt 128

//Generate friend's password (a random base64 string password saved to friendpassword.txt)
rand -base64 -out C:\friendpassword.txt 128

----------------

//Encrypt my password with my friend's public key for later use (A encrypts the value V with B's public key, yielding W.)
rsautl -encrypt -pubin -inkey C:\friendpublickey.txt -in C:\mypassword.txt -out C:\myencryptedpassword.txt

//Encrypt friend's password with my public key for later use
rsautl -encrypt -pubin -inkey C:\mypublickey.txt -in C:\friendpassword.txt -out C:\friendencryptedpassword.txt

----------------

//Create a signed hash of my password so my friend knows it's coming from me (signed hash saved as mysignedhash.txt and is in binary form) (A signs W with A's private key, yielding a signature S.)
dgst -sha256 -sign C:\myprivatekey.txt -passin pass:abc123 -out C:\mysignedencryptedpassword.txt C:\myencryptedpassword.txt

//Create a signed hash of friend's password so I know it's coming from my friend (signed hash saved as friendsignedhash.txt and is in binary form)
dgst -sha256 -sign C:\friendprivatekey.txt -passin pass:123abc -out C:\friendsignedencryptedpassword.txt C:\friendencryptedpassword.txt

----------------

//Create base64 versions of my password and my signature for easy transmission
//Password
base64 -in C:\myencryptedpassword.txt -out C:\myencryptedpasswordbase64.txt
//Signature
base64 -in C:\mysignedencryptedpassword.txt -out C:\mysignedencryptedpasswordbase64.txt

//Create base64 versions of friend's password and friend's signature for easy transmission
//Password
base64 -in C:\friendencryptedpassword.txt -out C:\friendencryptedpasswordbase64.txt
//Signature
base64 -in C:\friendsignedencryptedpassword.txt -out C:\friendsignedencryptedpasswordbase64.txt

----------------Phase II - Verifying Signature and Decrypting Password----------------

//For testing purposes, let's now say that I sent my password and signature to my friend.  My friend now wants to verify the password is from me.

//Convert my password and signature from base64 to binary
//Password
base64 -d -in C:\myencryptedpasswordbase64.txt -out C:\phase2myencryptedpassword.txt
//Signature
base64 -d -in C:\mysignedencryptedpasswordbase64.txt -out C:\phase2mysignedencryptedpassword.txt

//Verify the signature using my public key (B verifies S over input W, using A's public key. This gives B some guarantee that W is indeed the value that A sent (it was not altered in transit by some malevolent interloper))
dgst -sha256 -verify C:\mypublickey.txt -signature C:\phase2mysignedencryptedpassword.txt -out C:\phase2verificationresult.txt C:\phase2myencryptedpassword.txt
//Yay - this produces "Verified OK"

//Decrypt my password (B decrypts W with B's private key, yielding V.)
rsautl -decrypt -inkey C:\friendprivatekey.txt -in C:\phase2myencryptedpassword.txt -out C:\phase2mydecryptedpassword.txt
//Yay - this produces my password for my friend to see