How do hackers find vulnerabilities in closed source operating systems?

Question

I'm not planning on hacking anything, but I'm just wondering how hackers find vulnerabilities in closed source operating systems such as Windows, since they don't know exactly what the code is doing? From some online searches I know that some experienced hackers write and use "scanning tools", but how would they know how to write these in the first place? And how would you find vulnerabilities without a scanning tool program.

Answer 1

Reverse engineering, is the process by which a man-made object is deconstructed to reveal its designs, architecture, or to extract knowledge from the object; similar to scientific research, the only difference being that scientific research is about a natural phenomenon. This is how we begin.

This is not an exploit I coded and deployed, but rather noticed on Windows 7 startup recovery would sometimes after finishing checks open the results in notepad.exe. More importantly, I presumed notepad.exe was running as an elevated administrator, also notepad.exe has the ability to open Windows Explorer. So, now I had elevated administrator privileges over all listed drive partitions. Now I could ensure cmd.exe could be executed from the lock screen, which also runs as a restricted elevated administrator. This is how I became a local admin on a computer which I was only a user and the machine had bios passwords, to protect against this type of attack. Furthermore, this attack could bypass BitLocker if it had been used.

This is one type of way, which I deduced Windows 7 had an attack vector, which I happened to be right about. Although, I suspect your question was focused around zero-day exploits. So, reverse engineering tools are often the way. This becomes black box testing.

With compiled code, I am presuming the programmer coded the program in a particular manner while observing how the input affects the output.

• Input() = 3
• Code() = ... (We don't know, it's compiled)
• Output() = 10

We can deduce now, based on the input being 3 and the output being 10. A likely calculation could be 3*3 + 1 = 10. Although, it could also be 3*4 - 2 = 10... What about if we do another:

• Input() = 5
• Code() = ...
• Output () = 16

So, the pattern 5*3 + 1 = 16 could be a common pattern, we deduce other results and find which match between our tested inputs and outputs.

So a common arithmetic formula, for the code could be sum = $input * 3 + 1.

Now we have narrowed down what Reverse engineering is, let's focus on exploiting a networking service, Safari on iOS 4.0.2 which allowed '.pdf' Local Privilege Escalation 'Jailbreak'. An exploit like this would have likely, been detected by noticing the browser would crash (often a buffer overflow) when loading a PDF under certain conditions. Specifically CVE-2010-1797

Multiple stack-based buffer overflows in the cff_decoder_parse_charstrings function in the CFF Type2 CharStrings interpreter in cff/cffgload.c in FreeType before 2.4.2, as used in Apple iOS before 4.0.2 on the iPhone and iPod touch and before 3.2.2 on the iPad, allow remote attackers to execute arbitrary code or cause a denial of service (memory corruption) via crafted CFF opcodes in embedded fonts in a PDF document, as demonstrated by JailbreakMe. NOTE: some of these details are obtained from third party information.

This exploit allowed shown in JailbreakMe, allow arbitrary code within the web browser to be executed and elevated to root privileges without your permission. The browser would crash followed by the iOS restarting, somewhat often on iOS 4.

OllyDbg is a 32-bit assembler level analysing debugger for Microsoft® Windows®. Emphasis on binary code analysis makes it particularly useful in cases where a source is unavailable.

Beginning Reverse Engineering Tutorial 1 [HD] outlines how OllyDbg works and should introduce you to using this particular tool for reverse engineering.

This is far from a comprehensive guide about reverse engineering. As points regarding Sqlmap, dex2jar, jd-gui, edb-debugger and Valgrind, just to name a few, have not been included. Let alone reverse engineering a closed directory web server or hardware reverse engineering.

Answer 2

Aside from what Safesploit mentioned, there's also the concept of fuzzing. This involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks.