Basically any process which can successfully connect to a X11 server has full access to what occurs on that server. The X11 security model assumes that attackers are rejected at connection time. The usual security system is that clients must send a specific "magic cookie" as part of their first message, the cookie being a random blob which is created when the server launches or as part of the login procedure; the cookie is also copied to a location which is accessible only to the "allowed" applications. On Unix-like systems, the cookie is traditionally in the $HOME/.Xauthority file, but may be elsewhere (on my recent Linux system, it seems to be somewhere in a sub-directory of /var/run/gdm/), the point being that the file is readable only to a single Unix user. When forwarding X11 connections, SSH uses the xauth command to transparently transport the current cookie from the client to the server (so that applications launched on the remote host can send the cookie when they connect back).
So there is isolation with X11 -- but it is between users (as controlled by the OS), not between applications executed on behalf of the same user.
In older times, the magic cookies were not very common, hence some applications (notably xterm) included countermeasures such as refusing synthetic events (i.e. events resulting from a XSendEvent() called, as opposed to events coming from the hardware). Such measures were before many of the extensions which have grown on the X11 protocol over years, such as XTest, which make them mostly obsolete and useless.
Qubes is an operating system which builds on Xen to isolate some applications from each other -- each VM having its own exclusive display, with no possible interactions with the X11 servers from other VM. I have trouble reading "Beta 1" and "secure" in the same sentence, though...
For the X11 part, the same kind of isolation has existed for many years as Xnest (with an updated successor called Xephyr): you run the potentially evil application under a specific, isolated user ID, and let it connect only to a Xephyr instance which itself connects to the main X11 server as if it was a simple application. To evade that isolation, the application should hijack Xephyr (e.g. through a buffer overflow). The Qubes solution is only more thorough in that it replaces the Unix basic isolation ("distinct user ID") with a much more massive VM layer.
Wayland does not seem to advertise anything about isolation so it is a safe bet that it does nothing in that respect. Also, Wayland tries to give almost-direct hardware access to applications when possible, so any security model in that situation would be relative to how the undocumented 3D hardware and closed-source driver collaborate to avoid a misplaced DMA transfer to gain extra rights -- this does not look like the best way to build security.
