How does malware become executable code?

Question

I'm a software engineer and I still don't understand how malware works or what an exploit actually is in software engineering terms. How do the contents of a website, say, or a file attachment to an email, become executable code on a PC?

File formats like HTML, JPEG, MP3, PNG, BMP, etc, are all read by another piece of software and so should only be able to do what that reader allows it to do. Something that reads a JPEG file can basically light up pixels using certain RGB values in a display window; there's nothing in the JPEG spec that says "create a .exe file and put it here in the file system and then execute it". So how do they corrupt a JPEG file to achieve that? Why would a legitimate JPEG reader have the capacity to execute arbitrary code on a PC, and how would those bits get written to the PC?

Even javascript, which is a scripting-language, is still read by an interpreter and doesn't have direct access to the host's file system - it's sandboxed and can't directly write to the host's file system.

Modern programming languages don't even do direct memory access with pointers like we used to with C in the 80's and I assume that the "buffer-overrun" vulnerabilities that existed in the old days are a thing of the past with modern OSes and hardware memory management. So even if I wanted to, I don't know how I would write bits into a block of memory I didn't have legal access to, and even if I managed to do that, how would I trick the OS into treating those bits as executable code and execute them?

So how, exactly, does a nefarious website create executable code on a modern PC with a modern OS?

Answer 1

Many of the questions in the original post have already been asked and answered elsewhere on this site. Entire books have been written on some of these topics.

I'm a software engineer and I still don't understand how malware works or what an exploit actually is in software engineering terms.

If the definition of what an exploit is in the context of computer security is not clear enough, it may be helpful to look at examples of exploit code. exploit-db is a good resource for this.

File formats like HTML, JPEG, MP3, PNG, BMP, etc, are all read by another piece of software and so should only be able to do what that reader allows it to do. Something that reads a JPEG file can basically light up pixels using certain RGB values in a display window; there's nothing in the JPEG spec that says "create a .exe file and put it here in the file system and then execute it". So how do they corrupt a JPEG file to achieve that? Why would a legitimate JPEG reader have the capacity to execute arbitrary code on a PC, and how would those bits get written to the PC?

How to inject executable, malicious code into PDF, JPEG, MP3, etc.?

How can normal files hide a virus?

Is it possible for a file that is non-executable and read-only to run malicious code?

How can a virus exist in an image?

Modern programming languages don't even do direct memory access with pointers like we used to with C in the 80's and I assume that the "buffer-overrun" vulnerabilities that existed in the old days are a thing of the past with modern OSes and hardware memory management. So even if I wanted to, I don't know how I would write bits into a block of memory I didn't have legal access to, and even if I managed to do that, how would I trick the OS into treating those bits as executable code and execute them?

Stack Overflows - Defeating Canaries, ASLR, DEP, NX

So how, exactly, does a nefarious website create executable code on a modern PC with a modern OS?

How can a modern Windows/Linux system be compromised by a heap overflow in Chrome?

Browser Exploits: Attacks and Defense

The AnC attack

CWE-94: Improper Control of Generation of Code ('Code Injection')

Answer 2

A very common way to exploit for instance jpeg-viewers and other harmless software tools is to trigger a buffer overflow. And then you could do exerything...