First, some background: I've got a C++-based client/server system that uses the OpenSSL C library to encrypt the client/server connections. This is done in (what I believe is) the canonical OpenSSL fashion; that is to say, the server holds a .pem file containing both an RSA private key and the corresponding public key "certificate" (as generated by openssl req -x509 -days 3650 -nodes -newkey rsa:1024 -keyout my_private_key.pem -out my_public_key.pem ; cat my_public_key.pem >> my_private_key.pem), and the client holds a .pem file containing only the public key "certificate". When the client connects, the client uses the OpenSSL protocol to verify that the server really is who it says it is (that is, that the server really has the private key and is not an imposter or "man in the middle") before allowing any application-level data to be encrypted and sent across the connection.
This all works fine, but the problem is that authenticating the identity of the server isn't quite what I want to accomplish; in this scenario, the server and client are both on the same private LAN, and the server is physically connected to some custom hardware, so a fake-server isn't a big concern -- any fake-server or man-in-the-middle attacker would still need to connect to the real server in order to do any damage; the worst a fake-server could do is pretend that the hardware had done things it really hadn't, and since the user will generally be physically in the room with the hardware, the user would immediately see (by observing the hardware) that his commands were not being acted on.
Rather, my primary goal for OpenSSL here is to authenticate the client -- in particular, before I allow the client machine to send any commands to the hardware, I want to verify that the client machine is actually an authorized user, and just not some hacker/saboteur who has somehow gained LAN access and is trying to cause trouble by sending malicious commands to the hardware.
In my (admittedly naive) understanding of how public/private authentication works, this should be straightforward to accomplish, simply by switching the roles of the client and the server; that is, in this scenario, I think the client machine should hold the private key, while the server should hold the public key, and that way it is the server verifying the client's identify before proceeding, rather than the other way around.
However, when I try to set up the above using OpenSSL (by having the client call both SSL_use_certificate() and SSL_use_PrivateKey_file(), while the server calls only SSL_use_certificate()), it doesn't work (the client emits an error like SSL_write() ERROR!4560385472:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl/record/rec_layer_s3.c:1536:SSL alert number 40), and I'm not sure if that's because I'm simply calling something wrong, or if it's an indication that what I'm doing is unsupported by OpenSSL, or even fundamentally unsupportable for some reason.
I'm pretty certain I could obtain exactly the semantics I want by having the server make an outgoing TLS connection back to the client (so that the "client" becomes the "server" of the new connection), but I'd prefer to avoid that approach if possible (since the client could have a firewall installed that would make that reverse-connection difficult).
My questions are:
1) Is there anything fundamentally wrong-headed or futile about what I'm attempting here?
2) Assuming not, is it known that OpenSSL (as it is currently implemented in v1.1.1c) cannot or does not support this use-case?
3) Is there some other approach I should be considering (other than the usual "figure out where the mistake is in your software" one?)
