The security of your filesystem mostly depends on two main things: the algorithms that you are using for encryption and key derivation, and the implementation of those algorithms.
The algorithms that dm-crypt supports are well known, audited, and have been around for a while, so we consider them cryptographically secure because the only known way to attack them is through a brute-force attack, to perform a brute-force attack on a algorithm that has 256bit key (like AES-256), there are 2^256 possibilities that the attacker needs to guess, which is a really huge number and probably nobody has enough resources to do so.
But the attacker doesn't have to guess the key itself, attacker only needs to guess the passphrase that your key derived from, and passwords/passphrases have less amount of entropy, and they are usually short, so it will be easier for the attacker to guess the passphrase/password because it's less expensive and requires less resources, but that is not something you can't fix, you can configure your KDF(Key Derivation Function) to require more resources to derive the key, so the attacker will have to use more resources to guess the passphrase/password, or you can choose a better passphrase/password.
Note: making the key derivation process more expensive will make it more expensive for you as well, and you will have to wait a few more seconds to derive the key.
Note: you can use memory-hard KDFs like argon2 or scrypt to make it more expensive for the attacker.
And archlinux wiki has a good guide on setting up keyfile on a external media: https://wiki.archlinux.org/title/Dm-crypt/Device_encryption#With_a_keyfile_stored_on_an_external_media