I'm not aware of any scientific studies that focus specifically on the security impacts of password history controls.
However, it doesn't seem like we need a study to be fairly confident that reusing old passwords decreases security. An old password may not have been exposed during its use, but if it was exposed then reusing it provides an opportunity for an attacker to abuse it. So in some, likely small, percent of cases the choice not to maintain a blacklist of previous passwords is a detriment to security. In the cases where passwords remained unexposed it has a neutral impact on security.
Now if you expand the scope to include user behaviors that might be influenced by password history controls I could understand hypothesizing how this might negatively affect security. Maybe some user has tried to rotate between two very securely constructed passwords but when foiled by this control just decided to use a simple password and increment the number at the end after every change. You could argue that while rotating between two passwords is bad, using a simpler password that changes predictably is worse.
One problem is that the password history control is very often implemented in conjunction with scheduled password expiration, so password history can be difficult to study in isolation. You'd be left waiting to observe passwords changing organically. A user would have to decide to change their own password for some reason, or the app owners would have to force users to change passwords due to a security event. Then over time you could analyze whether being unable to use 'n' number of previous passwords had an observable influence on user new password choices.
This obstacle may be the main reason that researchers have chosen to focus more on the effects of password expiration controls or cross-site password reuse rather than password history. But I suspect it is also because most of us don't predict that there will be greater negative effects compared to the positive effects.