Effectiveness of Interactive Application Security Testing

Question

There are a number of IAST tools available on the market such as Acunetix Web Vulnerability Scanner and HP WebInspect Real-Time.

How effective are these at finding vulnerabilities? Is there any evidence that these can find more or less than a black box application pen test (DAST) or by source code review (SAST)?

I found this paper but as it is appears to be branded by a IAST vendor I wondered if there were any independent comparisons.

I've also read this post with interest: What fraction of vulnerabilities does black-box pentesting find?

Answer 1

Each method (DAST vs SAST) has its strengths and weaknesses.

For example, DAST method is much better positioned at evaluating XSS type vulnerabilities because it is actually able to execute "on the client side" to validate if execution is indeed possible (rather than benign reflection). SAST in this case for the most part is limited to proving that input made it (or has a possible path) from source to sink and can be reflected back in the output, but typically wouldn't be able to confirm that reflection can be executed to deliver XSS payload. In this case DAST has some advantages over SAST.

OTOH, detecting vulnerabilities such as a password input being written clear-text into backend log file would prove just about impossible for DAST scanner, but a manageable job for SAST analysis.

So IAST (or sometimes referred to as hybrid) can take advantage of strengths of both methods and help manage/reduce inherent weaknesses of the other. It can also help reduce false-positive detection rates by having one method confirm the other (e.g. in dynamic testing of SQLI having confirmation that attack indeed made it to DB); help speed up testing by narrowing attack surface (e.g. no need to test a page for SQLI if page doesn't do any SQL), actually discover attack surface that may be hidden (i.e. sub-application or pages that are not linked to and are not easily discoverable in the blind), etc.

Disclaimer: I used to work for and develop one of such scanners.

Answer 2

How effective are IAST tools at finding vulnerabilities?

HP WIRT, Acunetix with Acusensor, Contrast Security, Quotium, and Appscan Enterprise can find more vulnerabilities when they are in the best-fit scenario. Yes, I have used Contrast Security and WIRT to find vulnerabilities faster than without the SAST-side capabilities (note that WIRT gives feedback to both WebInspect in-tool via the SecurityScope hooks as well as makes Fortify merge-capable with the SecurityScope log output), but I would (and have) only utilize their capabilities when targeting JEE apps. For Acusensor, you get even less results and only against PHP apps. Quotium and Appscan Enterprise don't necessarily add more value to .NET, JEE, or any other languages I've run across -- but they certainly have their niches there as well, under the right set and setting.

HP WIRT/SecScope vs. JEE: 9/10
Contrast Security vs. JEE: 9.5/10
Acusensor vs. PHP: 6/10
Quotium or Appscan benefits: 7 or 8 out of 10
HP WIRT vs. ASP.NET: 8.5 out of 10

Is there any evidence that these can find more or less than a black box application pen test (DAST) or by source code review (SAST)?

There is absolutely conclusive evidence of this, especially given the time and resource constraints put on any given DAST assessment. Let me be a vocal giant when it comes to this topic: IAST, whether strict (Contrast, Quotium) or loosely-defined (WIRT/SecScope, et al) are enormous aids when configuring fault-injection test cases after a hybrid brain-tool code understanding (which normally involves -- at the very least -- walking and crawling the app's execution flow).

In the case of WIRT/SecScope, I was able to configure and find vulnerabilities that I definitely would have missed in several scenarios -- against JEE and .NET apps, especially apps that were over 20 MLOC with tons of complexity, and especially when they included Web Services that were SOAP and/or REST interfaces.

Personally, I wouldn't even touch an assessment against JEE, .NET, or PHP built apps without considering IAST as a primary, first-run approach. It saves so much time and provides so much focus.

Comparing SAST to IAST: IAST is faster than SAST and provides focus to the risk management. Even with only one IAST finding that SAST may or may not future-find, this can be of enormous value when looking at control-set gap analyses or similar risk-management problems. The fact that a security bug was found with IAST makes a software weakness uniquely positioned for higher prioritization. Additionally, IAST findings, when combined with SAST findings (especially custom rules-based) are amazing time savers when it comes to further customization of views, understanding of code, design of code, and combining of resources.

Comparing DAST to IAST: In my hands, IAST will simply find more, and understand the vulnerabilities deeper which probably will lead to exploitation or exploit chaining easier/faster as well.

In summary: use IAST whenever possible, as early as possible, and provide IAST artifacts to future SAST (and customized SAST) assessments.

Answer 3

Which is better? a hammer or a screwdriver? It all depends on the problem. Given that you do not know the weaknesses beforehand, the approach recommended by IARPA' STONESOUP is to use multiple tools with multiple approaches.

NIST's Paul Black highlights the advantages and disadvantages of static analysis in his CrossTalk March/April 2009 article. "New tests must be developed when new attacks or failure modes are discovered. Static analyzers have some advantage in this case ... "Most importantly, static analyzers have the potential to find rare occurrences or hidden back doors. Since they consider the code independently of any particular execution, they can enumerate all possible interactions. The number of interactions tends to increase exponentially, defying comprehensive static analysis and test execution alike. Static analysis can focus on the interaction without testing’s need to re-establish initial conditions or artificially constrain the system to produce the desired interaction. Worse, black-box testing cannot realistically be expected to discover, let’s say, a backdoor accessible when the user ID is “JoshuaCaleb” since there are a nearly infinite number of arbitrary strings to test."

"Testing and static analysis complement each other. Testing has the advantage of possibly revealing completely unexpected failures. Embedded systems can be tested, even when it is utterly impractical to analyze any software that may be tucked away in a component."

"Static analysis is no panacea. Complex and subtle vulnerabilities can always defeat the reasoning in a static analyzer. The utter lack of an important requirement, such as auditing or encryption, cannot reasonably be deduced from only the examination of post-production artifacts. Software with no resiliency or self-monitoring is open to errors in installation or operation, but static analysis can be one of the last lines of defense against vulnerabilities." ... Work from the June 2008 Static Analysis Tool Exposition [3], [4] shows that current analyzers vary widely. An analyzer may produce few false alarms for some weaknesses, but many false alarms for other weaknesses. Likewise, the rate of missed weaknesses differs greatly. Analyzers also only cover a subset of documented weaknesses [5]. Thus, the most comprehensive static analysis would result from a carefully used combination of analyzers.

Dr. Black leads NIST's SAMATE - Software Assurance Metrics And Tool Evaluation Project. See SAMATE's Tool Survey and Shay Chen's SecTool page for Price and Feature Comparison of Web Application Scanners. Full disclosure: I am a guest researcher supporting the SAMATE Project.

Answer 4

I'm working for the company who wrote the case study Quotium (http://www.quotium.com) but I will try to be objective in my answer !

The point is not to find vulnerabilities but to be able to have developers fix quickly the ones which pose a real risk to your business and specifically your data (main scope of hackers).

Why ?

When you are developing in an Agile environment with short release cycles. You can't stop the process to do SAST+DAST, analyse reports, do result triage, fix issues in unused functions, analyse code (sometime located in third parties) to pinpoint the source, understand false positive ecc... No time !

The main issue with SAST and DAST is time and expertise. For both, you need impact analysis and code fix analysis. SAST+DAST is not Agile.

SAST and DAST knows only one side of the system, they don't look at vulnerabilities in an overall threat context and data context. IAST does, as it tracks data from the front-end to the back-end.

A real IAST is one tool which have been built from scratch specifically to act from the hacker point of view to the code.

IAST gives you the ability to be better integrated in the development process.

For example, you plug your IAST tool where you have your build servers... on one side you plug your automated testing scripts (selenium for ex.) on the other side your bug tracking tool. You have an automated security testing process which runs by night.

In the morning, your developers find their vulnerable code list with the remediation to apply, a replay video of the attack on the tested application, the path to the vulnerable code on the different components to be able to patch third parties interfaces if the source code is unavailable.

• Developers focus only on proved vulnerabilities
• Testers have a clear view of vulnerabilities risks tailored by OWASP Top 10, SANS/CWE, PCI ecc... for a GO/NO GO

A big gain of time in the process !

Well here I'm not objective but it's how it works :)

To be objective:

There are only 2 tools that have been created to be IAST : Quotium's Seeker and Contrast from Aspect.

All the other vendors, tried to merge separate tools for marketing purpose but they are DAST+SAST not IAST !

We have clients who implemented IAST as a continuous process and continue to have a SAST tool for audit.

SAST helps you to have better secure coding practice. IAST still find vulnerabilities after SAST, but IAST will not highlight unsecure coding practice if the piece of code is not touched by a vulnerable process.

So you may better do IAST+SAST than DAST+SAST !