Cargo Cult
It is no secret that a lot of people who are tasked to design security for systems know very little about information security and are ill equipped for the task. As a source for that claim, I will take years of personal experience as pentester, and the absolute bewilderment for the things I have seen people develop over the years. Not because the developers are bad developers - on the contrary - but because information security has different requirements than normal software development. It's like asking a painter to paint a blueprint of a house.
As such, developers who lack the understanding of what is required of a system that handles authentication and authorization often fall back to the most primitive form of learning: Imitation. We see other people do things, and without understanding why these things are done, we imitate them. "After all, it's done that way for a reason." This process has a name: Cargo Cult Programming.
Originally, cargo cults were cults of tribespeople, who saw modern soldiers receive air cargo with supplies. They thought that this air cargo was a gift from a divine entity, and that the activities of the soldiers were a ritual to summon said entity to bring gifts. As a result, after the soldiers had left, the tribespeople began to immitate their activities to the best of their understanding, making makeshift uniforms and marching up and down, in hopes that this would summon some divine favour.
Cargo Cult Programming is a similar process, in which some developer does something in a particular way, due to some circumstance. The other programmers then blindly implement their solution in the same way, without understanding why the solution was implemented in said way, and what problem it aimed to solve.
About Password Lengths
For example, "DES crypt" would only allow for a maximum of 8 bytes for password length due to export restrictions - which was primarily a legal problem. So any system that used DES crypt would require a maximum character length of 8. Someone who saw this implemented may not know the reason why 8 was chosen for this, and may blindly copy this implementation, possibly not even using DES crypt in the background. So they are not limited by a maximum password length of 8, yet still arbitrarily impose it, simply because they have seen someone else do the same thing.
Misinformation
As for why companies require short, complex passwords rather than long, easy-to-remember phrases? That one is unfortunately on us. For a very long time, security experts tried to get people to make "better" passwords. And in a sense, it's true: <3BZg2Ck is a better password than ILoveNYC.
However, security experts are not infallible and we too learn painful lessons, such as AviD's Rule of Usability:
Security at the expense of usability comes at the expense of security.
Short, highly complex passwords are really difficult to remember for people, and people started to use "systems to make good passwords", like tGbHuJm! (I'll leave it up to the reader to figure out why this is a bad password). Experts changed their advice, instead advocating for long passphrases, like It's monday again and the coffee still tastes like wet socks, but "common wisdom" is difficult to change. When people hear that passphrases should be complex, they see the half-truth how a complex password is better than a simple one, and urge their users to make passwords as complex as possible.
Recommendation
In a system that requires passwords, users should be urged to use a password manager to create a safe, new password. A sufficiently long, randomly generated password will always beat any long, easy-to-remember passphrase, and it's more easily usable too if the browser automatically fills it in for the user.
If that is not a possibility, users should be urged to make long passphrases. Phrases are generally easier to remember for people, and length is king when it comes to strength. However, users can stick to phrases that are publicly known and not secure at all, such as In the beginning God created the heaven and the earth. - any automated check will tell you that this is an amazing password, but my wordlist on my cracking machine will beg to differ. As such, you can try to generate a memorable sentence for the user, or just believe that they know what they are doing.
Finally, after a user has entered a passphrase, it is recommended to check that passphrase with a database like Have I Been Pwned? to ensure that password isn't already known to be insecure.
