TL;DR: Skip to "Using result of DH".
Because you started with "I want to create a message exchange program" together with "I don’t know how to use DH", a tedious preface:
Security
People trust secrets to secure information systems and when those systems turn out to be insecure people get hurt, sometimes die. We want secure systems that are actually hard to break, even for a competent adversary.
What if your system is not meant to protect anything from a competent adversary? Then why are you building a system that is a messenger with RSA and AES? Obviously to advertise it as secure. Don’t advertise something as secure when it is not secure, because people will trust secrets to it and then they will get hurt.
You think your system is secure? It is not. The definition of "secure" is "multiple people who have a proven good track record of breaking comparable systems have tried to break this system for a long time and got nowhere". Because attacks only get better, even issues that do not immediately lead to a full break make the system broken or at least not recommended.
You think that standard is too high? You want me to break your system right in front of you and unless I can do that, you’ll call it secure and me jealous? The people who are capable of breaking non-trivial systems (I’m not one of them) don’t work for free. You might get a sketch of a way to break it for free, but without a lot of studying you won’t even understand what they mean. Professional security audits by real cryptographers and security engineers are expensive, you are in a bidding war for their time against financial institutions and the like.
A way to see your system get broken for free is to make it high value. Using the system to coordinate terrorist attacks will get the NSA on it, but they won’t tell you how they broke it. A system that would let an attacker steal a few bitcoins might attract qualified attackers.
The traditional way to get cryptographers work on breaking your system for free is to get an advanced degree in cryptography (or equivalent), publish breaks (or at least publishable results) in journals and conferences (they accept good papers by unknowns), become a known cryptographer, then publish your own design for others to break. Competitions like NIST LWC are a good venue.
What for?
Why do you want to build a secure messenger app? There are plenty of existing apps to choose from. Signal is the gold standard of secure messenger apps, it goes above and beyond to not let the server know anything about you. Even its controversial use of phone numbers is because they don’t want to store your contacts list on the server.
You mentioned RSA and AES.
Because they are household names. Non-experts just trust brands they recognize, regardless of all the information an expert would use to evaluate a product.
RSA
RSA is obsolete. It is not used in new designs. It has a number of problems:
It exposes a footgun in the form of "encrypt to public key" which invites people to build systems without PFS.
It needs many bits to be secure, this makes it somewhat inefficient. To get 256 bits of security, you need impractically many bits.
It is under-specified in the standards, so you won’t find a misuse resistant RSA API that makes all the security decisions for you (e.g. Public exponent? PKCS#1 v1.5? PSS salt length? MGF1 hash function?).
For those foolish enough to implement RSA on their own, there are a great many more issues, chief among them is that many people were shown textbook RSA as a throwaway nifty thing in math class, without all the details that make it secure, and countless people have just implemented RSA insecurely based only on that information. But professionals mess up too.
What has superseded RSA? First, NIST P-256 ECDH, ECDSA and ECIES early in the 21st century and then X25519 and Ed25519 in about 2012.
AES
AES is secure, when implemented well in hardware or when you use a slow (but secure) bitslicing implementation. Fast implementations of AES in software usually leak the key via side channels. Intel chips have had good AES for 10 years, so problem solved. There is a catch though: "if you have to type the letters "A-E-S" into your source code, you’re doing it wrong".
AES is a block cipher, the only way to use it is in a "mode of operation" (there are so many to choose from!), those need fresh IVs (TLS did it wrong!) or nonces which you must not repeat (the TLS site of the GCHQ did it wrong!), and possibly padding (padding oracles are a classic and the go aws s3 encryption sdk did it wrong!), after that the encryption is malleable so you have to add message authentication (iMessage did it wrong!), then you need to combine the encryption and the authentication correctly (TLS did it wrong!), then you need to verify the authentication tag in constant time, etc. etc.
The way these problems have been solved by everyone is by using an AEAD. The most popular one is AES-GCM (because it is fastest among the non-patented), but it requires hardware support and is easy to mess up, so I recommend using ChaCha20-Poly1305 (or XChaCha, which is even better). Those two options are what TLS, SSH and IPSec converged on. WireGuard, the modern replacement for IPSec, offers ChaPoly only, for extra security by eliminating cipher suite negotiation.
What about DH and PFS?
In the old days, people thought it’s a good idea to use the identity key as an encryption key, usually a key-encryption-key in a hybrid encryption scheme.
The way that works is that everyone has a single keypair that they intend to use for decades and not lose the private key and also not get it leaked (so should you backup the key or shouldn’t you?). You can use your own private key to digitally sign things so that others can verify your signatures offline, and you can use the other person’s public key to encrypt things so that they can decrypt it using their private key, offline. This is how PGP works and this is also how the TLS RSA cipher suites work.
The major problem with that way of doing things is that when the key inevitably leaks, the adversary can decrypt all previously recorded intercepted messages and also impersonate you to your friends and sometimes even impersonate other people to you. People impersonating you is an unavoidable risk of losing your key, but decrypting previous messages is catastrophic and new designs that don't provide PFS are non-starters.
With TLS, this means the key for a certificate is valuable even years after the certificate has expired and cannot be used for impersonation.
PGP/GPG is not recommended by infosec professionals, HTTP/2 has disallowed the non-PFS TLS cipher suites in 2015 and TLS 1.3 has finally removed the non-PFS cipher suites in 2018. AFAIK, SSH has never been used without PFS.
The way this is solved, and the way to get PFS, is by first using DH. Each side generates new single-use (called "ephemeral") keypairs which are not linked to any identity, and they exchange public keys in the clear. Then one computes "g^(ab) mod p" and the other computes "g^(ba) mod p" and those two values are the same and nobody who saw their exchange can know this number. This is basically a magic trick, it takes people a long time to believe asymmetric cryptography is possible.
Using result of DH
The result of performing the DH key exchange is not a smooth uniform array of bits of the particular length that you need to use an AEAD so you pass this value through a KDF (HKDF is fine) and extract as many bytes as you need. For example, the first 32 bytes are the client’s sending key and the next 32 bytes are the server’s sending key. The client is the side that initiated the network connection or sent the message. The server uses the client’s key to decrypt the client’s message and its own key to encrypt its reply.
Now, this is an anonymous secure channel. Nobody knows what’s inside, but the two sides don’t know who they are talking to. The keys they used are not linked to anything, and they might be Man-in-The-Middled and they have no way to know. The first thing they should do using this channel is to use asymmetric authentication to verify they are talking to someone who holds the long term identity key they expect (or the long term identity key is itself vouched by someone they trust, like in PKI). This can be done using digital signatures (the usual way) or using DH and HMAC, the way the Noise Protocol Framework does it.
After they have performed authentication, they can exchange regular messages.
Note that the PGP way of doing things does not require a message round trip or for the recipient to be online. Simple DH requires a round trip and requires both sides to be online at the same time. The round trip can be avoided by the recipient publishing a bunch of signed DH public keys on the server, which the sender can use to encrypt messages and store them on the server for the recipient to retrieve when they are back online. This is what Signal does (including whatsapp). The Noise Protocol Framework enumerates a bunch of "patterns" and the features you get in exchange for more round trips.
Classic DH, today called FFDHE, has some of the same problems as RSA, and has been superseded by NIST P-256 ECDH and later by X25519, but conceptually they work the same way.
How to build?
The people who have learned to not roll their own cryptographic primitives sometimes roll their own cryptographic protocols, and then in the joinery between otherwise secure primitives they create game-over bugs.
What should we do instead of rolling our own? Use good out-of-the-box solutions, the higher level the better. If you can just use WireGuard or correctly configured TLS or SSH (disable all the backwards compatibility and leave single secure choice), or age and minisign, you should do that.
If you have to use a library, use libsodium or Google Tink. Both try hard to have an API that is hard to use insecurely.
If you use those and don't use any cryptography outside of what's inside those, you have a much higher chance of not having a major cryptographic issue.
Secure messenger
Suppose we have solved the problem of exchanging messages. Secure messenger apps have more features than that.
If users have accounts they can recover after losing their device, password recovery (or equivalent) is a popular way to get game-over bugs.
Replacing a device, even if you have access to your previous device, or adding a device to be logged in on two devices simultaneously, is a way to lose custody of your private key if done incorrectly.
The question of how the key directory service works is critical, because without ability to verify the public key of the other party, you cannot trust the system. If the server can replace public keys then it can MiTM users.
Group chats are more complicated because you need to rekey when group membership changes.
If you want the server to store the user’s contact list, this is a privacy problem. Storing group membership on the server is a privacy problem. Storing user profiles on the server is a privacy problem.
Various messenger apps solve these problems in various ways.
You should study the Signal Protocol whitepapers to learn the state of the art.
Please do a lot more reading before you implement anything anyone might accidentally use for anything important.
openness to new people and ideas
What if you have a brilliant idea and I won’t find out because I was dismissive of your apparent lack of knowledge about the current state of the art? That’s a hard problem. People will not invest the time to learn about your brilliant new idea, unless you demonstrate first that you know enough about the existing ideas.
