(This is based on my answer to the same question on SO. It's often better to ask on only one SE site.)
The TLS protocol only allows for certificates to be exchanged, not raw public keys. Even "PGP keys" (if you wanted to replace X.509 with OpenPGP for authentication in TLS, which is much less supported) are in fact certificates (they're the signed combination of a public key and a set of identifiers and attributes).
As Thomas says, you could possibly define your own type of certificate and make them be a plain public key (although I'm not sure they could still be called "certificates" technically).
From a practical point of view, you should be able to achieve what you need using existing SSL/TLS stacks, and tweaking the way they deal with the verification X.509 certificates, with caution. Indeed, most SSL/TLS stacks expose the verification of X.509 certificates through their APIs, but few allow for other types of certificates (e.g. OpenPGP) or would let you customise their code easily. Customisation for non-X.509 certificates could be quite difficult, might results in additional bugs if you're new to SSL/TLS implementation and would also be difficult to deploy in practice, since all the potential clients would also need to support your customisations.
You can perform client authentication using self-signed client X.509 certificates and rely on their public keys (but you will need to verify this public key against something your server already knows, such as a known list). You need to understand the security implications for implementing this first. This is not quite the same problem as self-signed server certificates. (I'd suggest keeping a more traditional approach to verifying the server certificate.)
You can make the client send a self-signed certificate (possibly using certain tricks regarding the CA list advertised by the server) and either perform the verification in the TLS stack or later in the application.
Note that many application containers (e.g. Tomcat/Jetty in Java) expect the SSL/TLS layer to verify the client certificate. Hence, if you skip the authentication there (and prefer to do it later on within the container or as part of the application), many application frameworks will be confused. You need to be quite careful to make sure that authentication is actually performed somewhere before performing any action that requires authentication in your application.
For example, it can be OK to have a trust manager that lets any client certificate through in Tomcat/Jetty, but you can't rely on the javax.servlet.request.X509Certificate
request attribute to have been verified in any way (which most frameworks would otherwise expect). You'd need to implement some verification logic within your application: for example, having a filter before any authenticated feature that compares the public key in this client certificate with your known list of public keys (or however you want to match the public key to an identifier). Alternatively, you can also perform this verification within your custom trust manager (in Java), this will require less work within the applications.
You could do something similar in an Apache Httpd + PHP setup (for example), using SSLVerifyClient optional_no_ca
. Again, your PHP application could not rely on the certificate having been verified, so you would have to implement so verification there too.
Don't do any of this unless you understand at which stage the certificate information you get has been verified.