Payment Gateway rejecting Digitally Signed Payment Message

Question

I am new to digital signatures. I read many problem statements, solutions, and documents but nothing came to my rescue.

Payment Message (Text) needs to be digitally signed with the following specs as per the payment system mandate.

The hash algorithm used is SHA-256 algorithm.
The signature should be generated in PKCS7 format.
The signature should be detached signature.
There should be serial no. existing in the signature.
Signature should be included between the tag -----BEGIN PKCS7----- and -----END PKCS7-----.
Each line should have a maximum of 64 bytes.
\n should be used as End-of-line.

Payment Message being sent to gateway is Payment Message + Signature attached at the end in one single message transmitted via MQ

When the existing system signs the message, it is accepted by Payment System Gateway but when I try to do the same, it always fails in verification. We have been given a PFX file as input from the client and a relevant password which is valid. Using this PFX file, the existing system signs the message and is successful but then it always fails when I generate a digital signature and send it across.

I used OpenSSL commands

openssl dgst -sha256 -out hashfile inputpaymsg.txt 
openssl smime -sign -in hashfile -out output.p7s -signer certificate.pem -inkey private_key.pem -outform PEM -noattr earlier generated certificate.pem, private_key.pem

but I am not able to match the signature for a given input which is seen in the successful existing system message.

One difference I saw is that when I use openssl asn1parse on the digital signature which is accepted by Payment System generated by the existing system I see "md5" instead of "sha256".

                                SEQUENCE
   30:d=5  hl=2 l=   8 prim:    OBJECT            :md5
   40:d=5  hl=2 l=   0 prim:    NULL
   42:d=3  hl=2 l=  11 cons:    SEQUENCE
   44:d=4  hl=2 l=   9 prim:    OBJECT            :pkcs7-data

Tried hashing with md5 but still, it shows SHA256 only and is not matching with the signature as I see in the message sent by the existing system.

Would appreciate it if someone can give me step-by-step openssl commands via which I can use the PFX file and input text messages and generate digital signatures like existing system might be generating.

Another Doubt/Question : From one of the tutorial videos, got to learn that Digital Signature is encryted hash and how it is useful to ensure that original message is not tampered. I also understood that Digital Signature could be decrypted to original hash. If this is right, is it possible for anyone to share correct "unix level sso command" to decrypt the signature to retrieve hash. Also can i generate hash of original message with all possible algorithms and then check and compare hash from which algorithm matches? — Vishal, Jan 23 '23 at 12:24
No, that's not how digital signatures work. That's a simplified explanation to help with understanding, but it doesn't work that way. — schroeder, Jan 23 '23 at 19:17

dave_thompson_085 · Answer 1 · 2023-01-24T19:37:17.990

the signature operation done by PKCS7/CMS/SMIME automatically includes the hash step so you don't want to do that explicitly, only a single sign operation like

 openssl smime -sign -outform pem -in paymentmsg [-md sha256] -signer cert -inkey key [-noattr] -out sig

If the hash really is sha256 and you use OpenSSL 1.0.2 up, which at this point is quite likely, you don't need to specify -md sha256 because it is the default; if the hash really is md5 (which would be very bad) you do need to specify -md md5 for any OpenSSL. The requirements you quote don't say whether using signedAttrs is prohibited, allowed, or required, but since you have a known-good case it should work to follow that. Similarly it isn't clear if the signer cert should be included, omitted, or doesn't matter, but you could look at and follow the known-good case.

EDIT: I forgot originally, but if the separate data contains bare-LF linebreak(s) (i.e. like Unix not CRLF like DOS/Windows or more relevantly NVT), or NUL(s), or possibly some other control character(s), OpenSSL by default applies S/MIME canonicalization -- even when using CMS/PKCS7 formats that don't call for it; this will likely mismatch with the verifier(s). To avoid this add -binary.

I don't know what #4 'serial no. existing in message' means; if it means the SignerInfo uses issuerAndSerial to identify the cert, OpenSSL does that; if it means something else, OpenSSL may not do it.

For RSA only, which you didn't say but questions about an unidentified asymmetric algorithm are usually by people who don't know there are algorithms other than RSA just like questions that don't identify an OS are usually by people who don't know there are OSes other than Windows, you can recover (not decrypt) the signed hash value, which will match a hash of the data; see Can OpenSSL decrypt the encrypted signature in an Amazon Alexa request to a web service? .

$ openssl req -newkey rsa:2048 -keyout se267928.key -nodes -x509 -days 3650 -subj /C=XX/O=test/CN=se267928 -out se267928.crt
Generating a RSA private key
.............+++++
...................................+++++
writing new private key to 'se267928.key'
-----
$ printf payment message >se267928.in
$ openssl smime -sign -outform pem -noattr -in se267928.in -signer se267928.crt -inkey se267928.key -out se267928.sig
$ openssl asn1parse <se267928.sig -i
    0:d=0  hl=4 l=1265 cons: SEQUENCE
    4:d=1  hl=2 l=   9 prim:  OBJECT            :pkcs7-signedData
   15:d=1  hl=4 l=1250 cons:  cont [ 0 ]
   19:d=2  hl=4 l=1246 cons:   SEQUENCE
   23:d=3  hl=2 l=   1 prim:    INTEGER           :01
   26:d=3  hl=2 l=  15 cons:    SET
   28:d=4  hl=2 l=  13 cons:     SEQUENCE
   30:d=5  hl=2 l=   9 prim:      OBJECT            :sha256
   41:d=5  hl=2 l=   0 prim:      NULL
   43:d=3  hl=2 l=  11 cons:    SEQUENCE
   45:d=4  hl=2 l=   9 prim:     OBJECT            :pkcs7-data
   56:d=3  hl=4 l= 835 cons:    cont [ 0 ]
   60:d=4  hl=4 l= 831 cons:     SEQUENCE
   64:d=5  hl=4 l= 551 cons:      SEQUENCE
   68:d=6  hl=2 l=   3 cons:       cont [ 0 ]
   70:d=7  hl=2 l=   1 prim:        INTEGER           :02
   73:d=6  hl=2 l=  20 prim:       INTEGER           :6A43249F600271F0B24C4CA83EC152C23A2C1956
   95:d=6  hl=2 l=  13 cons:       SEQUENCE
   97:d=7  hl=2 l=   9 prim:        OBJECT            :sha256WithRSAEncryption
  108:d=7  hl=2 l=   0 prim:        NULL
  110:d=6  hl=2 l=  47 cons:       SEQUENCE
  112:d=7  hl=2 l=  11 cons:        SET
  114:d=8  hl=2 l=   9 cons:         SEQUENCE
  116:d=9  hl=2 l=   3 prim:          OBJECT            :countryName
  121:d=9  hl=2 l=   2 prim:          PRINTABLESTRING   :XX
  125:d=7  hl=2 l=  13 cons:        SET
  127:d=8  hl=2 l=  11 cons:         SEQUENCE
  129:d=9  hl=2 l=   3 prim:          OBJECT            :organizationName
  134:d=9  hl=2 l=   4 prim:          UTF8STRING        :test
  140:d=7  hl=2 l=  17 cons:        SET
  142:d=8  hl=2 l=  15 cons:         SEQUENCE
  144:d=9  hl=2 l=   3 prim:          OBJECT            :commonName
  149:d=9  hl=2 l=   8 prim:          UTF8STRING        :se267928
  159:d=6  hl=2 l=  30 cons:       SEQUENCE
  161:d=7  hl=2 l=  13 prim:        UTCTIME           :230124005316Z
  176:d=7  hl=2 l=  13 prim:        UTCTIME           :330121005316Z
  191:d=6  hl=2 l=  47 cons:       SEQUENCE
  193:d=7  hl=2 l=  11 cons:        SET
  195:d=8  hl=2 l=   9 cons:         SEQUENCE
  197:d=9  hl=2 l=   3 prim:          OBJECT            :countryName
  202:d=9  hl=2 l=   2 prim:          PRINTABLESTRING   :XX
  206:d=7  hl=2 l=  13 cons:        SET
  208:d=8  hl=2 l=  11 cons:         SEQUENCE
  210:d=9  hl=2 l=   3 prim:          OBJECT            :organizationName
  215:d=9  hl=2 l=   4 prim:          UTF8STRING        :test
  221:d=7  hl=2 l=  17 cons:        SET
  223:d=8  hl=2 l=  15 cons:         SEQUENCE
  225:d=9  hl=2 l=   3 prim:          OBJECT            :commonName
  230:d=9  hl=2 l=   8 prim:          UTF8STRING        :se267928
  240:d=6  hl=4 l= 290 cons:       SEQUENCE
  244:d=7  hl=2 l=  13 cons:        SEQUENCE
  246:d=8  hl=2 l=   9 prim:         OBJECT            :rsaEncryption
  257:d=8  hl=2 l=   0 prim:         NULL
  259:d=7  hl=4 l= 271 prim:        BIT STRING
  534:d=6  hl=2 l=  83 cons:       cont [ 3 ]
  536:d=7  hl=2 l=  81 cons:        SEQUENCE
  538:d=8  hl=2 l=  29 cons:         SEQUENCE
  540:d=9  hl=2 l=   3 prim:          OBJECT            :X509v3 Subject Key Identifier
  545:d=9  hl=2 l=  22 prim:          OCTET STRING      [HEX DUMP]:04146C99D8E22DADE9E69231A373AB7D57E3B0C1F1DE
  569:d=8  hl=2 l=  31 cons:         SEQUENCE
  571:d=9  hl=2 l=   3 prim:          OBJECT            :X509v3 Authority Key Identifier
  576:d=9  hl=2 l=  24 prim:          OCTET STRING      [HEX DUMP]:301680146C99D8E22DADE9E69231A373AB7D57E3B0C1F1DE
  602:d=8  hl=2 l=  15 cons:         SEQUENCE
  604:d=9  hl=2 l=   3 prim:          OBJECT            :X509v3 Basic Constraints
  609:d=9  hl=2 l=   1 prim:          BOOLEAN           :255
  612:d=9  hl=2 l=   5 prim:          OCTET STRING      [HEX DUMP]:30030101FF
  619:d=5  hl=2 l=  13 cons:      SEQUENCE
  621:d=6  hl=2 l=   9 prim:       OBJECT            :sha256WithRSAEncryption
  632:d=6  hl=2 l=   0 prim:       NULL
  634:d=5  hl=4 l= 257 prim:      BIT STRING
  895:d=3  hl=4 l= 370 cons:    SET
  899:d=4  hl=4 l= 366 cons:     SEQUENCE
  903:d=5  hl=2 l=   1 prim:      INTEGER           :01
  906:d=5  hl=2 l=  71 cons:      SEQUENCE
  908:d=6  hl=2 l=  47 cons:       SEQUENCE
  910:d=7  hl=2 l=  11 cons:        SET
  912:d=8  hl=2 l=   9 cons:         SEQUENCE
  914:d=9  hl=2 l=   3 prim:          OBJECT            :countryName
  919:d=9  hl=2 l=   2 prim:          PRINTABLESTRING   :XX
  923:d=7  hl=2 l=  13 cons:        SET
  925:d=8  hl=2 l=  11 cons:         SEQUENCE
  927:d=9  hl=2 l=   3 prim:          OBJECT            :organizationName
  932:d=9  hl=2 l=   4 prim:          UTF8STRING        :test
  938:d=7  hl=2 l=  17 cons:        SET
  940:d=8  hl=2 l=  15 cons:         SEQUENCE
  942:d=9  hl=2 l=   3 prim:          OBJECT            :commonName
  947:d=9  hl=2 l=   8 prim:          UTF8STRING        :se267928
  957:d=6  hl=2 l=  20 prim:       INTEGER           :6A43249F600271F0B24C4CA83EC152C23A2C1956
  979:d=5  hl=2 l=  13 cons:      SEQUENCE
  981:d=6  hl=2 l=   9 prim:       OBJECT            :sha256
  992:d=6  hl=2 l=   0 prim:       NULL
  994:d=5  hl=2 l=  13 cons:      SEQUENCE
  996:d=6  hl=2 l=   9 prim:       OBJECT            :rsaEncryption
 1007:d=6  hl=2 l=   0 prim:       NULL
 1009:d=5  hl=4 l= 256 prim:      OCTET STRING      [HEX DUMP]:024274CC4588E55CAE53C3B433C9BC440A76CF742117B657A540B429329BA984CD32530F811C1B2359A4C45985461E2BDED4C1766F111D9F12F79B51EEFB8F2AF0C90C3BCDCA4B6FC1BBB7CAB2164CC06E20F2625A1FEFF2711F2603A2B92CD221F54419DE2B25A9E2A58000B5CF320C4DCC2CAB122DB60690D8856D3FCC4A0568B3F7C5010C8CFF12F9930EEDFE52FAA9AF9F0F029AE29BF0AC9EB416FA8D164F9E7E65EB3A580F452A87954F1F5761703EFD245E226326EC1ABDEDBB3E99A5F7CEDBF3DAB6DB20C711894FC84F71A598F48CDE08DE8469B62CABA591BFED613A6DA951FD8572F68874FFE8018914293DE5B8342A7F0C98D3F6D9102237AACB
### The SEQUENCE from 908 to 979 is the IssuerAndSerialNumber to identify the cert; notice the serial at 957 matches the serial of the actual cert at 73. 
$ openssl asn1parse -in se267928.sig -strparse 1009 -out se267928.tmp
    0:d=0  hl=2 l=  66 prim: INTEGER           :74CC4588E55CAE53C3B433C9BC440A76CF742117B657A540B429329BA984CD32530F811C1B2359A4C45985461E2BDED4C1766F111D9F12F79B51EEFB8F2AF0C90C3B
Error in encoding
140519004668352:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:../crypto/asn1/asn1_lib.c:101:
### ignore this error; the RSA signature is not ASN.1, but -strparse -out still extracts it
$ openssl rsautl -verify -in se267928.tmp -inkey se267928.crt -certin -asn1parse
    0:d=0  hl=2 l=  49 cons: SEQUENCE
    2:d=1  hl=2 l=  13 cons:  SEQUENCE
    4:d=2  hl=2 l=   9 prim:   OBJECT            :sha256
   15:d=2  hl=2 l=   0 prim:   NULL
   17:d=1  hl=2 l=  32 prim:  OCTET STRING
      0000 - 04 0f fd 59 25 d4 0e 11-c6 7b 72 38 a7 fc 99 57   ...Y%....{r8...W
      0010 - 85 0b 8b 9a 46 e9 72 9f-ab 88 c2 4d 6a 98 af f2   ....F.r....Mj...
$ openssl sha256 <se267928.in
(stdin)= 040ffd5925d40e11c67b7238a7fc9957850b8b9a46e9729fab88c24d6a98aff2

Million Thanks @dave_thompson_085 - never been so close in solving this puzzle. I followed below command and now difference between existing system signature and mine is only 6 lines which are all at the end. `openssl smime -sign -outform pem -in paymentmsg -md md5 -signer certificate.pem -inkey private_key.pem -noattr -out mysig` Also when i compare asn1parse dump of passed signature and my signature then it is also limited lines following below tag ` 1701:d=5 hl=4 l= 256 prim: OCTET STRING ` Looks like some switch usage is missing or some tweek reqd in commands -- Any idea — Vishal, Jan 24 '23 at 03:37
Few questions to sovle this remaining bit. 1) Is openssl version has some dependency? -- basically can the output be different if openssl version changes and rest everything remaining same? 2) Is the assumption right that if INPUTS remain same then above command should always generate same signature every time as i am trying to compare my generated signature with one generated by another system and being accepted by payment gateway.? --- is time stamp of signature generation plays any role or any any other parameter which may answer above difference of 6 lines? — Vishal, Jan 24 '23 at 05:23
@Vishal: If you specify the hash (not default) there is no openssl version difference; if you use noattr there is no signingtime. Some signature schemes, which you should identify by looking at the AlgorithmIdentifiers before the signature (which is indeed the final OCTET STRING), are randomized, but DSA and ECDSA can't have length 256 (which does correspond to nearly 6 lines of base64 in PEM) and RSA-PSS isn't used without explicit specification. But I remembered one more possibility; see edit. ... — dave_thompson_085, Jan 24 '23 at 19:41
... And if they are using MD5 with RSA-PKCS1v1_5 for payments _in 2023_ their security is crap and you should be prepared to lose your money. — dave_thompson_085, Jan 24 '23 at 19:41
Thanks for your help @dave_thompson_085. I did add -binary as well but again problem continues. my generated signature does not match with what other system generates and accepted by payment system. Trying since last 6 weeks + but no idea why it is not matching. Unfortunately no technical specs/sample code is shared by payment system which is making this a little more difficult to crack. Existing system won't share details from their side. i am using -noattr as otherwise difference is huge. — Vishal, Jan 25 '23 at 13:03
@Vishal: then I'm out of ideas. If they are both RSA-PKCS1v1_5 as appears, recover the hashes as I showed, confirm if they mismatch, and try to find a way they do or do not match your data. — dave_thompson_085, Jan 26 '23 at 01:19

Payment Gateway rejecting Digitally Signed Payment Message

1 Answers1