you always know the IP address of the person your sending to
Oh, do you? Personally I have no clue under which IP addresses I can find my favourite web services. Usually I type in a domain and my computer contacts some third party to ask about the corresponding IP. This is known as Domain Name Service (DNS). You'll see in a second why I mention this.
A classic example of a potential man-in-the-middle (MITM) attacker is the person controlling the router you are connected to. Without PKI in place, they can easily pretend to be someone else.
Let's say you go to a public cafe and want to access your Gmail inbox. You type 'gmail.com' into your browser's address bar, and you are presented with a page that asks for your Gmail account name and password. How do you know that page is in fact the one hosted by Google, and not, say, served from a local machine in the back room of the cafe? One issue here is that your computer will have sent a request to a remote DNS to ask for the IP required, but this request goes through the router, so without additional security measures in place, the MITM can return an arbitrary IP. Second, the router might have left your DNS request and answer unchanged but then served up the fake page regardless of what IP you were actually trying to reach. After all, you relied on them to relay your request and Gmail's answer.
And this is where transport-layer encryption and the public-key infrastructure come into play. When you try to contact a server that runs HTTPS, it will present you with a public key to encrypt further communications, and that key will usually be signed by a trusted CA. Thus, without access to the server's private key, an intermediary can not read or manipulate the exchanged data (hence the name end-to-end encryption for this kind of mechanism). What the CA does is ensure that the public key originally presented cannot be spoofed without you noticing it.
Similarly, DNSSEC (DNS Security Extensions) relies on PKI to detect a MITM who is manipulating DNS requests/responses.
