The encoding of the value (32 bytes of binary, 256 bytes of ASCII ones and zeros, 64 bytes of hex, 44 bytes of base64, etc.) does not affect the security of the key in any way.
The important distinction between cryptographic keys and passwords is that when humans choose passwords, they generally only get about 20 bits of entropy. That means there are about 1 million possible passwords they could have chosen. A dictionary attack that tries the common things people do (add numbers, l33t speak, etc.) will break most passwords. Also, people reuse passwords.
A cryptographic key is generated by a computer, not by a human, and it must be generated using a CSPRNG. Reading from /dev/urandom on linux is perfectly fine. CryptGenRandom on Windows is fine. A cryptographic key generated this way genuinely has the amount of entropy that is assumed by its size. So if it is 256 bits long, then is has 256 bits of entropy. No dictionary attack or using the birthday of the user's kids or the password the user used on their luggage would work at breaking it.
So although the API might accept any 32 bytes as a cryptographic key, using a human generated password directly will break security and should be flagged in any audit/review.
There are ways to securely use passwords for cryptography (see PAKE), but the correct thing to do almost always is to generate a secure cryptographic key using a CSPRNG, almost always using the CSPRNG provided by the OS kernel.
The real amount of entropy and possibility of reuse also dictate how the value is stored: human generated passwords have to be stored using secure password storage (designed to be slow to thwart brute force dictionary attacks on GPUs, argon2, scrypt, bcrypt, PBKDF2) while high entropy computer generated keys/tokens don't need that and can be stored with a single hash (only if all you need is to verify it when it is provided from outside. If you need that key in the clear, use a secure way to manage keys).
There are ways to encode strong cryptographic keys in a way that humans can memorize them. This is rarely needed, but some people want to carry a secure electronic wallet in their head. For that, see BIP-39 and things like it.
To try to explain how much entropy is in 256 bits, I'll say that it means 100 throws of a truly fair 6 sided dice or alternatively, 24 words chosen from a list of 2048 words in such a way that knowing any 23 of the 24 words would not help the attacker in any way: their chance of guessing the missing 24th word is still exactly 1/2048 (that means for example that words can repeat). It is very difficult for humans to generate that much entropy.