I'm trying to brainstorm a security scheme for the problem of verifying server software integrity. The domain of this problem is Game Servers built on Valve's Source Dedicated Servers. These servers are used in competitive gaming, and the integrity of these servers can be incredibly important in running tournaments and whatnot.
Game clients are already verified quite well in a number of ways by Valve's Steam system. Because of various content-verifying and anti-cheat systems (VAC), we can be fairly sure that no client-side content is modified.
However, on the server side, there are little to no protection schemes in place to ensure that clients are playing on unmodified versions of the software. The software itself is inherently configurable (through config variables and commands), and it allows extensions on functionality through Server Plugins. In addition, there are no checks on the server code on filesystem or in memory. SRCDS servers are inherently modifiable on many levels.
This poses a problem in competitive gaming on the Source engine. It is possible for any number of ConVar modifications, plugins, or server binary modifications could be used to "cheat"--to give an advantage in gameplay mechanics to one team over another, or one player over another. There is currently no way for a client or third party to verify that the SRCDS server is unmodified.
The basic solution to this issue seems to be open-ness. We can make changes to ConVar values be announced to clients, make lists of plugins running on the servers available to clients, and even do CRC checks on files on disk and in memory and provide clients with resulting values. However, none of this can be differentiated from a server which falsifies the announced/provided information. Anything down this path essentially feels like obscurity rather than security.
How can I design a system to verify server software's integrity using only challenge/responses to that server?
P.S. Feel free to ask questions or ask for more specifications.