While I have not deployed it yet, I know many people who have deployed it. Every single one of them have noted a significant reduction in the amount of bandwidth consumed by things like SSH brute-force attacks as a result.
However, that is not to say that there are not downsides. AFAIK, there are no kernel-based port knocking implementations available, which for me would be the real key to adoption. Port knocking daemons rely on reading failed (and filtered/prohibited) log file entries from a firewall system. That's all fine and dandy, but what happens if the filesystem gets full? What happens when the daemon gets killed because of some runaway process eating up the system's RAM and swap? What happens if something else which either of those two things depend on just up and stop working? You most likely end up with a server that you will have to physically access. That could wind up being more costly than is reasonable, especially if you are more than a few tens of miles away from the server and do not have anyone that you can call to get there in a hurry.
One thing that I can say is that it is not "security through obscurity". Port knocking is a form of authentication, and like any authentication system it can be made to be as simple or complex as desired. Something as simple as "knock on port 10,000 + realPortNumber" can be done, which would amount to a trivial break, or the port knocking might itself be used to transmit some form of real authentication (say, 1 block of AES encoded data given a key derived by some other method). It would not be feasible to use port knocking to transmit large amounts of data, though, because it would take significantly longer than just sending a single packet, and if the packet is over TCP than at least it can be known if it was received successfully or encountered some form of error.
One interesting question that this brings up, however, is how to manage the log files---userland implementations mostly require the log files in order to determine whether or not a knock has been successfully sent, and what happens if those logs are leaked? Authentication data becomes known, and that is obviously not a very good thing.
I cannot tell you whether or not to use port knocking in your setup. I am not yet, and I am not 100% certain that I ever will be. It makes more sense to me to use strong authentication systems that are based on strong cryptography (such as a PKI infrastructure) than it does to throw port knocking in the way. Also, adding a single point of failure to access critical infrastructure, to me anyway, seems like a bad idea, and way more difficult to properly support with any sort of guarantee. Again, though, that is based on the notion of the port-knocking software being not integrated with the firewall at the operating system kernel level; if that ever changes, I may also change how I feel about using it.