Wanted to post my own reasoning in the question as I have given this some thought and the reason I am posting this as a question is to check my reasoning. For all I know this answer might get downvoted :) .
I mean, the general argument - as I understand it - is that password complexity requirements require the user to choose a password from a greater 'space', thus decreasing the chances of picking an unsafe password as a lot of unsafe passwords are only lowercase alpha (though clearly not all, such as Password1!
).
When something like haveibeenpawned is checked that is an improved check removing the large majority of the "unsafe password"-space. Instead of considering all lowercase alpha passwords unsafe, it instead uses actual data to determine which passwords are or are not unsafe (the set of compromised passwords is assumed to be extremely similar to unsafe passwords). Combining both complexity rules with a compromised password database check thus primarily eliminates the 'secure' part of the lowercase alpha space, blocking safe passwords which are caught by password complexity rules such as passphrases (e.g. catwindowlightpencil
).
The above figure doesn't encompass all passwords, it instead compares the passwords password complexity rules are blocking with the passwords you wish to block. The unsafe password set is the one you wish to block, the compromised password set is a subset you can block, the lowercase alpha set is the one you block with password complexity rules.
Note: Obviously new 'dumb' passwords can be 'created'. For example, a new pop idol might become popular during the past week or Microsoft might buy Github, but not yet show up in a compromised password database. Similarly someone might use a product name as a password of a product that has not been compromised yet.