7

Lets assume I am responsible for an application in my company and I decide to hire security experts to perform a security audit. Lets assume further that my company owns the source code of the application and I am allowed to hand it over to the hired experts.

Are there any good reasons to prefer a blackbox penetration test over a security source code review? In my opinion a source code audit identifies critical vulnerabilities much faster and more efficient than a blackbox penetration test. Why should I let a security expert hammer at my application in the dark, when i can provide him with internal informations like system configuration and the source code to help him directing his efforts better.

To avoid confusion - when I talk about a security source code review, I assume that I will allow the security experts to perform tests on the application as well, to validate their findings and try different attacks. The goal of the audit is to identify weaknesses and improve the security of the application afterwards.

Demento
  • 7,339
  • 5
  • 37
  • 46

1 Answers1

4

Source code review is generally more effective than black-box pentesting. @Demento, you articulated the reasons why well. And, apart from these first-principle justifications, empirically source code review finds more vulnerabilities than black-box pentesting. This is especially true if source code review is combined with architectural risk analysis (what Microsoft calls threat modelling).

The big advantage of black-box pentesting is that it is considerably less expensive and requires less expertise. This is one reason why you see so many people use black-box pentesting: it is a lot cheaper, and still catches a significant fraction of vulnerabilities.

Personally, I recommend that if you use source code review, you should also combine it with black-box pentesting. Black-box pentesting will cost only a fraction of the cost of the source code review. And empirical data suggests that black-box pentesting + source code review find more bugs than either alone finds. Therefore, the marginal utility of black-box pentesting is probably worth it, even if you are already doing source code review.

D.W.
  • 98,860
  • 33
  • 271
  • 588
  • +1 but one more question. You say "empirical data suggests that black-box pentesting + source code review find more bugs than either alone". Do you have any sources for that? This would be really helpful. – Demento Oct 09 '11 at 08:18
  • @Demento, there's a been a lot of great stuff written on this topic, but [here's one source](http://www.cs.berkeley.edu/~daw/papers/webapps11.pdf). Hope that helps! – D.W. Oct 10 '11 at 03:19
  • 2
    @Demento: Even outside the pentesting arena, the evidence is consistent that source code review is more effective than testing. See "Code Complete" by McConnell which has good references; almost anything by Watts Humphrey at the SEI; or "Software Inspection" by Gilb and Graham. My experience bears this out too -- I can find 5-10x the bugs per hour when reviewing vs testing. – bstpierre Oct 16 '11 at 23:57