We do not know how exactly Facebook manages user passwords, however, we can imagine this scenario:
A new user logs into his Facebook account using the password: Password1
. It is likely that, immediately, Facebook generates similar passwords such as: Passw0rd1
. pAssword2
, Passmord1
...etc.
Facebook then stores their respective hashes (which are different, of course) so that if a user user types his password (after reset), the hash of the new password is compared to all these stored hashes: Facebook can tell you then if your new password is similar to the previous one or not.
EDIT:
Following @Philipp comment saying Facebook does not hash passwords, I want to share this information that I quoted from the official Facebook documentation (Keeping Passwords Secure):
We hash each password using our internal password hashing algorithm
and the unique salt for that person. Since Facebook stores passwords
securely as hashes, we can't simply compare a password directly to the
database. We need to hash it first and compare the hashes.