This protocol is mainly used to transfer end-to-end encrypted data.
The protocol mainly consists of:
1) X3DH (Extended Triple Diffie-Hellman) key agreement protocol
X3DH establishes a shared secret key between two parties who mutually
authenticate each other based on public keys. X3DH provides forward
secrecy and cryptographic deniability.
the Double Ratchet algorithm, which is used by two parties to exchange
encrypted messages based on a shared secret key. The parties derive
new keys for every Double Ratchet message so that earlier keys cannot
be calculated from later ones. The parties also send Diffie-Hellman
public values attached to their messages. The results of
Diffie-Hellman calculations are mixed into the derived keys so that
later keys cannot be calculated from earlier ones. These properties
give some protection to earlier or later encrypted messages in case of
a compromise of a party's keys.
By using Signal Protocol, you can transfer safely any kind of traffic between some parties and even you can't see the photos your users are sharing using your tool.
Also Signal Protocol is not vulnerable to MiTM attacks as long as the users are mindful with the chat code (known as Safety number in Signal and security code in whatsapp) that is generated after doing the X3DH handshake (you will know about it from a video in the resources down below).
About managing file encryption keys:
you can use a zero knowledge authentican protocol to authenticate your users, by this you can't know what passwords are the users using. So you can (in the client-side) use the password of the account of the user (if you have an account system) to derive from it an encryption key to encrypt the encryption keys of the encrypted photos to be stored safely in the cloud, so you won't worry about how to manage encryption keys in the client-side.
Note: if the user forgot his password the encryption keys cannot be recovered.
Other useful resources:
- This video will help you understand more about end-to-end encryption
- This video will help you understand X3DH key agreement protocol
- This video will help you understand the Double Ratchet algorithm
- This video will help you understand how to share end-to-end encrypted data between more than two parties
- This video will help you understand more about zero knowledge authentication