Yes you can use the key file instead of a password: simply specify the file at the time:
cryptsetup luksAddKey /dev/the /your/key/file
When you wish to unlock, cryptsetup open ... --key-file /your/key/file /dev/the (My understanding is that cryptsetup will still use a KDF though, just one that requires no user input.)
If you wish to have a key file and a passphrase, you need to 'unlock' your key file using some tool that demands user input, then make this file available to cryptsetup. (If this is your goal let me know and I'll update this answer later on with some options.)
If you use 10k iterations and 2GB of ram, you'll be waiting a long time for Argon2 to complete. 10 iterations might be more usable (at the time of writing) and still provide good security, as long as your password is 'good', eg: b/pz8prnzx2sv.r2o/y3khykkacqja6ghx9u442o (which, according to KeepassXC, provides "228.40 bits" of input to key derivation functions!)
MASTER KEY As you pointed out in your comment, LUKS stores a hash-digest for the master key in the header. This value MK-digest is PBKDF2-derived from the master-key with MK-salt and MK-iterations as input. The underlying hash function is the algorithm you specify using cryptsetup luksFormat --hash, by default, SHA2/256 on my version of cryptsetup: 2+. The number of iterations is 1000 and I don't believe you can change this, however, this may be incorrect. (See the LUKS1 spec, Figure-4, p10)
KEY-SLOTS Each attempted key-slot derivation becomes the key to decrypt that key-slot's copy of the master key, as described in this answer. It's not clear from the spec whether a key-slot file is also fed into a key-slot's corresponding password-based KDF, however, I believe it is because if you tune an Argon2 slot to take a few seconds, it will take the same time regardless of whether its via --key-file or a typed password.
This answer gives a good overview as well as details about the LUKS header arif '19.
So yes, an attacker can perform an exhaustive search to find an input that produces the MK-digest that is stored in the header. This input will be the master key. You want your adversary to be in the worst-case position of having to search, on average, half the key-space to find a match. (More follows subsequently on exhaustive searches.)
You can provide the master key to cryptsetup at the time of creation with the --master-key-file option. By default, cryptsetup will draw random bytes in accordance with its default setting, or, if you specify a random input file. Read the man page carefully, as you can provide the master key file at time of luksFormat, at open, as well as other circumstances.
You can also open the LUKS volume directly with your master key, however, I personally think the better option is to unlock the key-slot and then somehow destroy the insecure copy of your key-file. (I like modprobe brd and shred.) This leaves the kernel to manage the master key. It also allows you to have two or more separate copies of the master key, in case of hardware failure, but this will double etc. the likelihood that an attacker can find your volume master key.
You also need to think about what file-system the master key file or any key-slot file will be stored on during its use, because this has the potential to hang around in its unencrypted form.
RANT ON KEY SIZES From a security persepective, there's no point producing more than 256 bits of high-quality random input for your aes-xts-plain64 key derivation. The reason why is, that this is an impossible wall to climb when it comes to guessing... but feel free to provide as much as you're comfortable with - there might be other good reasons to produce large input. (It's a lot easier to lose small things!)
Joking aside, in fact you can't even enumerate each possible number in a fraction of this range with present-day technology, (in your lifetime), let alone test them all in an exhaustive search. To put it into perspective, the newer AntMiner s19 can hash 95 T-hash/sec of SHA2/256, so if your security rests on this primitive, then to test say 70 bits of truly random input, it will take this single piece of hardware 2⁷⁰ ÷ (95 × 10¹²), or ~12 million seconds, say, 3k 24-hour periods. Also remember that PBKDF2 has a tunable number of iterations which you'll then need to divide 95 × 10¹² by. (Ps. I'm not suggesting 70 bits is enough, but this is a tractable number for people to understand - remember - each additional bit doubles the previous search space.)
(Incidentally, the same computation shows why a poor choice of password falls swiftly in the face of parallel, offline dictionary attacks. Also of note, the AntMiner s19 operates at roughly 8-times the rate of the s9, ~12 T-hash/sec, for roughly 3-times the power consumption.)
The point is: you want any key to come from a sufficiently large, non-deterministic and equally likely number of possibilities, ie. you want it to be of high entropy.
