The main risk is that the initial connection can be intercepted by a Man-In-The-Middle, so an attacker can retrieve the password.
The first time a user connects to an SSH server, something similar to the following is displayed:
$ ssh scanme.nmap.org
The authenticity of host 'scanme.nmap.org (45.33.32.156)' can't be established.
ECDSA key fingerprint is SHA256:8iz5L6iZxKJ6YONmad4oMbC+m/+vI9vx5C5f+qTTGDc.
Are you sure you want to continue connecting (yes/no)?
Now, unless every user is going to verify that fingerprint to ensure that it's your server, then there is a risk that their connection has been intercepted (e.g. MITM on their network directly intercepting traffic, or some type of DNS hijack).
This is because SSH is "Trust on First Use". If you've connected to that host before and the key suddenly changes then the following message is displayed instead, warning the user.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
5c:9b:16:56:a6:cd:11:10:3a:cd:1b:a2:91:cd:e5:1c.
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending key in /home/user/.ssh/known_hosts:1
RSA host key for ras.mydomain.com has changed and you have requested strict checking.
Host key verification failed.
The warning will mitigate this attack for users that have already connected from that particular client.
Other, less critical risks of having SSH open on the internet is that there are many bots out there finding and attempting logins against such servers. If you have a strong password on all accounts, then the risk of compromise is very low (by strong I mean one that definitely won't be in a word list the attacker may use or create). Usual rules apply - eg no password reuse, password stored securely, etc. The downside is that log files will quickly build up with all the authentication attempts received.
Regarding SSH itself in any authentication mode, as this is another service there is a chance of vulnerabilities being discovered that could automatically be exploited by such bots. However, the risk of this is no greater to that of a web server or web application of being exploited. Like any server with public services, a good vulnerability management strategy is recommended.
Also see this answer, the part related to SSH warning messages and the risk of continuing, even with public key authentication.