Seems like a relatively obvious way to prevent (software) keylogging would be to force only the current (in-focus) app to be able to receive keystrokes.
There could be a way to make explicit exceptions for macro apps etc. Querying the exception list would make finding a keylogger trivial.
Is there any reason operating systems don't enforce this policy by default?
Because it wouldn't help.
Most keyloggers are installed at the operating-system level, and the operating system needs to have access to the keystrokes. Alt-Tab program switching, using Ctrl-Alt-Del to terminate malfunctioning programs, and detecting keyboard activity to keep your screensaver from activating all require the OS to see keystrokes.
There's also the minor matter that if you eliminated OS access to the keyboard, every application would need to have a complete set of keyboard drivers built into it.
The keyboard to application interface goes through several phases, some of which the OS has little control, and some that is provides explicit hooks into for additional functionality. The basic design goes like this: hardware events are received by driver chains, which then pass messages to the kernel, that then dispatches it to a global hotkey chain, and finally to the intended application (if not cancelled by any prior step in the chain).
The driver chain allows the kernel to not care about "how" the keystrokes are generated, only that they are. They could be from a keyboard, from a IR device, or any other source that could send a signal designed to be interpreted as a keyboard. A hardware keyboard logger, for example, is a dongle that has a USB or PS/2 input on one end, and a USB port or PS/2 on the other, such that the keyboard passes data through this device and is intercepted. The OS literally cannot detect that such logging is going on.
The other common kind of logging happens in software, which can happen either before the OS has a chance to see the keyboard messages, or after. Drivers can do pretty much whatever they want, and the OS can't strictly detect that a driver is diverting messages for nefarious purposes, because they get to inspect messages before the OS does. This is the nature of the hardware abstraction layer (HAL) that the drivers are a part of. Fortunately, since they are in memory, anti-malware software can detect and disable such behavior.
Finally, you have an intentional "hole" in the OS, usually referred to as "global hot keys", that allows any application to request that the keyboard messages are passed to them before the in-focus application. This allows not just Alt-Tab to work (the window manager intercepts these messages to switch apps), but also most media programs request handlers to support multi-media keys like play, rewind, and fast forward, and other user-land apps for volume control, etc. Without all of these global hot keys, the OS would be very annoying to use, and apps would be far more complex as a result. However, just as this is a great feature to have, it can also be abused by a program.
However, you should note that not "all" programs get a copy of a keyboard message, only drivers, global hot key handlers, and the in-focus application. The problem has nothing to do with the fact that every program gets a copy of a keyboard event, but the fact that the HAL needs to be able to transform messages from hardware to kernel messages, and global hot keys are necessary to provide features to the user without each program needing to be built to provide the same features.
There have been advances to lock down the process though, such as requiring "signed drivers," which reduces the likelihood of malicious drivers getting into the driver chain, and anti-viruses that can detect bad behavior by apps. However, until many of the security vulnerabilities are addressed, such as hardware level keyboard logging and insecure global hot key registration exists, loggers will still have an opportunity to log keystrokes. Even though normal keystrokes appear to go simply from hardware to an app, there's actually several intervening steps necessary, and these steps are required for basic compatibility (drivers) and functionality (global hot keys).
The reason this isn't done by default is because the previous-generation operating system design didn't have a huge focus on sandboxing and the like, so right now it would require big architectural changes to make such changes work. Mark touches upon those to some extent in his answer, but it boils down to that you can't allow applications to blindly run with OS privileges.
It is however far more interesting to note that modern OSes like for example Google's Android, Apple's iOS and even Google's (desktop) ChromeOS all do limit keystrokes only to current applications1. Now, focusing just on ChromeOS as it's the only desktop OS in the list it's also important to note that global shortcuts create no problems in such a case. An application can 'simply' tell the OS that they wish to bind to a specific shortcut which then can be configured in the OS by the user. The relevant specification can be found here for those of you who're curious how this looks.
Similarly, by taking a look at Android, we can find that accessibility software that requires global keyboard access can still be written in such modern environments by exposing such information if and only if the application is explicitly granted such permissions in the accessibility settings panel. This makes setting up such software a bit of a pain, but it does prevent keyloggers from being distributed.
In conclusion, the only reason it's not done is not because it doesn't help or is impossible, but because due to historical priorities it's just taking a bit longer to get there. We will get there in time and in the meantime it can make sense to use a more modern, locked-down OS in secure environments.
1 I know that Mac OS X has recently been expanding their sandboxing effort of applications. I presume that a properly sandboxed application (one not requiring administrator privileges) will also be unable to act as a keylogger, however I have spent very little time reading up on how their sandboxing really works. If anybody knows for sure, do share!
On Windows, there is very little protection between applications running as the same user. If you try to take away SetWindowsHookEx, then malware writers will switch to DLL injection and a whole set of other techniques. You could even just draw a transparent window over the targeted application which would have focus and recieve keystrokes, then pass on those keystrokes by sending Windows messages. Fundamentally Windows was not designed with the possibility of malicious executables in mind.
A system designed for running untrusted applications can sandbox them from one another. But it's still vulnerable to (much rarer) exploits that punch out of the sandbox into the kernel.
There's also another technique that could be used: arbitary code execution in the browser gives an exploit the ability to record every keystroke the browser sees, even if it can't escape a sandbox.
There are very valid reasons you want keystrokes visible to applications outside of the currently running foreground process. Unless every programmer implements cut/copy/paste for example, the OS must monitor for certain keystrokes. Lets say you have a program for taking screenshots and you want it activated by a certain keystroke, that program must be able to monitor the keyboard activity to detect when its keystrokes are activated. It would make more sense to run everything in a sandbox and if monitoring input was a requirement for a background program it is a permission that must be allowed, but no matter what is done, there is always a way around it. If someone wants to know what your typing they will find a way. All we can do is try to make it harder. Obfuscation could be an interesting method. Inject a lot of false input messages to the OS that the running program generates to fool anything outside of it?
Several have observed that other legitimate applications may need to transform the original keyboard input (which comes in the form of keycap X was pressed/ released) into some kind of text. People who have lost the use of one hand, for instance.
There are also keyboard mapping tools that allow people to type in languages for which the OS does not supply a keyboard map for that language (or the user doesn't want to use the OS-defined keyboard map--for instance, a Dvorak-like layout for some language other than English). One such app is Keyman (keyman.com).
And finally, there are people like me (or maybe I'm the only one) who have keyboard mappers that map things like -H to the left cursor arrow, -B to -, and so forth. So we lazy people (or just me) don't have to move our hands from the alphabetic part of the keyboard to move around.
There are in fact some operating systems that allow this, namely Mac OS X. Key presses that aren't 'reserved' system key combinations or 'modifier' keys (alt, ctrl, shift etc) are only sent to the application currently in focus.
Of course, that would make it very annoying for some applications designed for accessibility and programs that use VoIP that need push-to-talk keys. Because of this, there's a part of System Preferences where you can enable this functionality for specific applications.
As for Windows, I would imagine that it's not done because the current method of handling keypresses is quite complex and is designed to work with all of the current drivers and programs.
One of the reasons is that for some software, being able to read keystrokes while being active but not currently selected is vital to the working of the software.
Let’s give some practical example.
Some disabled people find typing hard and are very slow at typing. They often use “word prediction” software that shows a list of the words the match what they have typed so far, allowing a word from the list to be picked with a single keystroke.
Blind people use software that reads the text displayed on the screen (screen readers). This software can also speak out each letter as it is pressed on the keyboard. The software has a few hot keys that read the current line, etc.
In the UK, there is software that allows you to type an address into a letter by typing the postcode, then hitting a hotkey to get the postcode expanded into nearly the complete address. This software is often used with customer database, etc. that don’t support “quick” address entry themselves.
Testers wish to be able to record all keystrokes and mouse moves they make while testing software, so they can then play them back to repeat the test.
In all the above cases the software DEPENDS on being able to see what keystrokes are being sent to other programs.
Smartphones and tablets limit how applications can interact with each other, to the extent that most people still turn to a “PC” or Mac when they wish to get work done that need the use of more than one application.
Safety and freedom to be productive does not always go together; there is a tradeoff to be made. The iPhone sits on one end of the spectrum and Windows/Max on the other end.
The X Window system already does this to some extent. It allows programs to grab the mouse and keyboard and nothing else can intercept the inputs. Not even the window manager. It is rather annoying for full screen games since there is no way to switch away from the game. Try opening xterm and holding down the ctrl key and right clicking (I think) and select grab keyboard. Now what you type in to xterm cannot be seen by any other client connecting to the X server. Edit: it looks like X still has an XQueryKeymap function that can bypass this and be used by a keylogger.
Edit: Windows implements such a feature by making it so that pressing ctrl+alt+del will always bring up a valid Windows account management screen, as long as fast user switching is turned off. So fake password entry screens are not possible.
