It looks like you asked a second similar question shortly after asking this question. However, the bounty is here, so I'll give it a shot!
In your other question, you mention that you're looking for confidentiality and non-repudiation as part of your file exchange scheme. This means that you're looking to send data from Alice to Bob such that Bob can verify that Alice (and only Alice) could have sent the file and Alice can verify that Bob (and only Bob) can read the contents. You're also, as far as I can tell, looking to send this data in a single package, so you only have to send one communication with all of the information required.
PKCS7 is an older format that's been largely wrapped up in the Cryptographic Message Syntax (CMS) format, which includes PKCS7 data, but also has many more features. This format is primarily used for sending encrypted email (S/MIME), but you could theoretically use it for your purposes. There are other formats for sending an encrypted package, e.g. using XML, that are standard in various contexts.
But this is overkill. Don't re-implement the wheel.
There are existing tools (e.g. these) that can do public-key file encryption and signing out of the box so that you don't have to risk implementing your own cryptosystem. If you can use GNU Privacy Guard (GPG) or another tool like it for this purpose, I would recommend doing that. I would be shocked if these tools couldn't output messages in CMS/PKCS-compatible formats.
If you need something more basic, OpenSSL is capable of encrypting and decrypting signed CMS content on the command line and via the library. There are plenty of examples if you search.
There are also a variety of answers here on Stack Exchange on this same topic, e.g. What's the standard way to encrypt a file with a public key in Java?:
One common, standard encryption format is Cryptographic Message Syntax (CMS, evolved from PKCS#7), and e.g. BouncyCastle supports it. It allows for signatures also, and is the basis, e.g., for S/MIME mail message security. It may be overkill, but is widely supported.
Example code is provided by Bouncy Castle. For CMS, see org.bouncycastle.cms.test, probably the Enveloped Data examples.
Another option is the XML Encryption format, standardized by W3C. For Java examples, see e.g. Exploring XML Encryption, Part 1 or do it with XSS4J: Implementing XML Encryption in Java
For non-repudiation purposes, there are established third-party services that provide timestamping services, along the lines of a notary public validating a paper document, if you need that. These are called Trusted Timestamp Authorities. This way, Bob can verify that Alice signed the document no later than the trusted timestamp, since the document was already signed and encrypted when timestamped by the TTA, and Alice cannot claim to have signed the document later. This could (IANAL) also allow a secure trace for court purposes, if, for instance, Alice and Bob are negotiating a contract and Bob needs to sue Alice later.
It's generally a bad idea, security-wise, for a non-expert to re-implement an cryptography standard. You could, and probably will, make a mistake, usually a protocol error, which would blow your system wide open. As Schneier says, it's trivial to create a crypto system that you yourself cannot break. Using a standard system also allows you to send your data to any recipient, who doesn't necessarily need to know your protocols and interfaces.
Cryptosystems like PGP, GPG, and OpenSSL have been around for two decades and have had the proverbial kitchen sink thrown at them. Security experts still depend on them. So use one of those, if possible.
If it's not possible to use existing libraries and systems (and there aren't very many situations where that would be the case - university homework is one of the few I can think of), you'll need to do at least the following:
- Alice and Bob will each need a public and private keypair. When Alice sends a file to Bob, she will encrypt it using her private key and Bob's public key. This means that only Alice could have sent the file, and only Bob can decrypt the contents.
- You will need a secure way to communicate keys from Alice to Bob, or at least to verify that the key you received is the key Bob sent. Ideally, you'll need to have your keys signed by a certificate authority like Verisign. If you can't, you'll need to use some kind of trusted out-of-band method of verifying your exchanged public keys. Otherwise, all you've got is a key that claims to be Bob's key.
- Please please use a standard mechanism to generate your keys (e.g. OpenSSL).
- Do not re-use a symmetric key. Ever. Similarly, don't re-use an initialization vector (IV) for your symmetric system. Ever.
- If you need trusted timestamping, as mentioned above, using a trusted third-party service is one of the most reliable ways to accomplish this. You would send them your already-encrypted file, and they would return a package including a signed timestamp and hash of your encrypted blob. If you need this and can't use a third-party, you'll need to figure out a way of securely synchronizing timestamps with Bob before sending the document.
- You'll need some way to keep your private keys safe. Using a keystore on the local machine means that your 2048-bit RSA keys are exactly as secure as the sum of your local system password and your keystore password. It's better to store keys in a dedicated crypto module, or perhaps a USB drive kept in a safe when not being used.
- You can use a standard format like CMS to transmit your certificate data, or you can roll your own. I'd recommend going with standard formats whenever possible, since the recipient can then use standard tools for decryption.
- You may want to consider using separate keys for signing the document vs. encrypting the document. In other words, Alice would have two keypairs - one used to encrypt the symmetric key and a second one used to encrypt the hash. Bob would need both corresponding public keys. The advantage is that you can then swap out your encryption keys on a regular basis without necessarily losing your non-repudiation. The disadvantage is that you'd have to securely communicate each new public key to Bob.
Please let me know if you need any further clarification. I cannot stress strongly enough that you should use an existing, tested, and accepted cryptography system if it's available for your purposes.
