121

Today I experienced a situation where a person responsible for the security of a company required a pentesting company to withdraw a clause in the contract that says that:

"during the pentest there exist the possibility to delete or modify sensitive data in the production environment unintentionally due to the execution of some tools, exploits, techniques, etc."

The client says that he is not going to accept that clause and that he believes that no company would accept that clause. He thinks that during a pentest information could be accessed but never deleted or modified.

We know that the execution of some tools like web crawlers or spiders can delete data if the web application is very badly programmed, so the possibility always exists if those types of tools are going to be used.

I know that these are the conditions of the client, and should be accepted, but:

Can a skilled and professional pentester always assure that no data will be deleted or modified in production during a pentest?

Can a pentest really be done if the pentest team has the limitation that data cannot be created nor modified?

Should the pentesting company always include the disclaimer clause just in case?

NULLZ
  • 11,446
  • 18
  • 80
  • 111
kinunt
  • 2,769
  • 2
  • 24
  • 30
  • 67
    This reminds me of a wtf story where Google indexed a page that had a link that reset the entire database when it was clicked. Google just blindly went and indexed the page, followed the link, and wiped the database. There is no way to guarantee that a pentester might not do the same thing by accident. – Mark Henderson Jul 17 '13 at 22:41
  • 14
    Who knows what kind of logic bomb you might stumble upon. It's CYA against malicious traps and horrible misconfigurations that were a hidden disaster waiting to take out data the first time somebody triggered the domino string. Basically a "Sorry you don't have working backups, but that issue predates my breathing on your system" clause. – Fiasco Labs Jul 18 '13 at 01:02
  • 3
    @MarkHenderson I'd love to see a site with details of that case... :) – woliveirajr Jul 18 '13 at 02:54
  • 22
    @woliveirajr - http://thedailywtf.com/Articles/The_Spider_of_Doom.aspx and http://thedailywtf.com/Articles/WellIntentioned-Destruction.aspx - my memory wasn't quite spot on, but close enough – Mark Henderson Jul 18 '13 at 03:18
  • 54
    Surely a client insisting on this clause being removed is not being unreasonable, but rather **not understanding why this clause is there**. They think *you* will do something to break their system during testing. Just explain to them that if *their* code has some awful bug in it that deletes the database when you click "save", you obviously can't be held responsible for it. As often happens, this sounds like a human communication failure, not a technical question. – MGOwen Jul 18 '13 at 04:16
  • 2
    If the client has a working (tested) backup/restore procedure (with a mechanism of file integrity checking), then they've got nothing to worry about, right? – Tom O'Connor Jul 18 '13 at 08:17
  • 8
    @tom-oconnor Even more so, better to lose data in a controlled and prepared way with recent backups than at a random time with some hacker slashing his way through. If data loss during pentesting is even an issue, it's propbably the least of the company's worries. And in this light I side with the remark that it's also more a communication problem than a technical problem. – Grimace of Despair Jul 18 '13 at 09:30
  • 6
    Here's another way to look at it. The *supposition* of a penetration test is that if the attacker succeeds then the system is *broken*; if it were *correct* then the attack would not have succeeded. If the attack is successful then the system is *broken*; if the system is broken then *its behavior when given any input is not necessarily that predicted by its specification*, and if its behavior is not predicted by its specification then *who knows what will happen*? – Eric Lippert Jul 18 '13 at 17:36
  • Is it professional that your company hasn't cloned the systems you plan to pen test - if not only for the fact they could affect your system, but also where do you stage your development changes? – Ryan McDonough Jul 18 '13 at 18:22
  • direct your client to this page – Alex Gordon Jul 18 '13 at 22:31
  • @RyanMcDonough That's only really practical with small targets. For a general test on XYZ Co's infrastructure when they're large enough to have numerous racks of servers would not only be prohibitively expensive to clone; but the likelyhood of not having everything configured identically in the copy will go up reducing the validity of results. – Dan Is Fiddling By Firelight Jul 19 '13 at 18:01
  • @DanNeely Even if you clone the system, couldn't what you've cloned reach out across the network to a box on the original system via a hardcoded network address which you've cloned along with the rest? Yes, you could isolate the system, but at that point, you've changed the environment so radically that your tests don't test the thing you're testing. – deworde Jul 22 '13 at 08:38

8 Answers8

148

There's no way that a pentester can 100% assure that data will not be modified or deleted, in the same way as they can't assure that system availability won't be affected (I've knocked systems over with a port scan or a single ' character). as you say a web crawler can delete data from a system if it's been set-up badly.

I'd say that what should be said is something like "every care will be taken to ensure that the testing does not negatively affect the systems under review and no deliberate attempts will be made to modify or delete production data or to negatively affect the availability of in-scope systems. However with all security testing there is a risk that systems will be affected and the customer should ensure that backups of all data and systems are in place prior to the commencement of the review"

Rory McCune
  • 61,541
  • 14
  • 140
  • 221
  • 42
    I knocked a server over by putting a unicode character into a field in the customer's web app. You can never be 100% sure your actions won't affect uptime - unexpected failure vectors are everywhere. – Polynomial Jul 17 '13 at 20:06
  • 13
    I have had members of teams testing accounts provided by the client as 'test' accounts accidentally make international payments :-) It happens. Keep the clause in! – Rory Alsop Jul 17 '13 at 22:04
  • 24
    I don't know a lot about security but I've met a lot of messed up code bases and I approve this message. – Erik Reppen Jul 18 '13 at 01:52
  • This and @MGOwen 's comment, try to explain it the reasoning to the customer. If they still wont accept it, it's probably best to not accept the job. Not that I know much about pentesting, but you might try to do some SQL injection to get access to the system and this mig wreck havoc with the data, if not careful enough. – Holger Jul 18 '13 at 07:42
  • 46
    As a penetration tester, you're essentially and ultimately going to be spending a lot of time poking at bits of the system with the question of "what happens when I input *this* unexpected data *here*?", and unless you're allowed to analyze the code itself at the same time there's no way to *absolutely guarantee* that the answer *won't* turn out to be "the system shits itself, erases its backend database and corrupts the backups before escaping across the border to Tijuana with the company budget". – Shadur Jul 18 '13 at 07:53
93

A pentester who claims that he will never alter production data is either a filthy liar, or thinks himself to be a lot more competent than he really is, or strongly intends to do nothing at all (that's the only surefire way of never breaking anything). In any case, you don't want to work with that guy.

A prospective customer who believes that skilled pentesters will never damage the tested systems, and who refuses to work with pentesters unless they promise exactly that, is a customer who 1. lives in the fairy unicorn dreamland, and 2. is almost guaranteed to ever do business only with filthy liars. He is in for some disillusion at some point, probably in a rather spectacular way.

It is a moral imperative and also basic self-protection for pentesters to include clauses about possible breakage in their contracts. The risk of collateral damage is real, even if the pentester is very good at what he does (because damage does not come from how good the pentester is but from how bad the tested system was designed and implemented). A pentester could be sued for not having warned the customer.

A customer who refuses a contract with such a clause has another name: "trouble". It is usually better to skip such customers altogether.

Tom Leek
  • 170,038
  • 29
  • 342
  • 480
  • 16
    +1 client smells funny, back away. This is 100% standard language included in all pentest engagements. – bobince Jul 17 '13 at 20:03
  • 3
    Sounds to me like he read it as "We can do what we like to your system, and claim 'accident' when we screw up", rather than "We don't want to be sued because of an unforeseeable situation". But yeah, AVOID. – deworde Jul 18 '13 at 07:57
  • @deworde Any clarification to the condition such as "we will take all reasonable steps to avoid breaking the system" can later be sued over their steps not being sufficiently reasonable. Depending in the impact of the damage they do, it could easily exceed what would otherwise be their bill, and require a significant increase in their fee (i.e.: "Why does pen testing cost $1milion?" "Because that is what we expect you could sue us for, there is a 900k discount if you agree not to sue"). – Matt Jul 18 '13 at 14:22
  • @Matt I agree. It's a failure to comprehend, not a failure to convey. If this is the standard language, the contract really *can't* say anything else without massive risk with zero upside. At worst, you'd end up sending the message that you're willing to negotiate your culpability. – deworde Jul 18 '13 at 21:31
36

Caveat: I am not a professional pentester. And I work in backup software.

Can a skilled and professional pentester always assure that no data will be deleted or modified in production during a pentest?

I would say no, there are no absolute guarantees something will not accidentally be deleted when you are trying to break things. However, that said, a pentester should generally try to exploit systems in a way that does not involve deleting data. For example, whilst everyone's favourite SQL statement is:

select * from users where userid='dave'; DROP ALL TABLES;

A more responsible approach would simply be to list all data:

select * from users where userid='2' OR 1=1;

Generally, I imagine most pen testers have developed a series of exploits of the non-destructive variety like the latter.

Javascript injection in its various forms can use simple proof of concept code such as alert("exploited you"); rather than genuine exploit code.

Can a pentest be really done if the pentest team has the limitation that data cannot be created nor modified?

I would say no. How would you show a proof of concept of stored XSS without being able to store your XSS in the database? This is invariably going to create new data.

There are numerous examples where an exploit requires writing data, even if only temporarily.

Should the pentesting company always include the disclaimer clause just in case?

Again, I am stretching my personal knowledge here, but I am going to say yes.

This comes back to the whole point of hiring a pen testing company in the first place, however. The point of a pen test is to identify risks that might need evaluating and fix them before the bad guys find them.

I would also hope any good pentester would ask, and any good business would have thought of, disaster recovery on some scale. Surely you should have a backup of some form on your systems such that you can restore as needed. You need this not just in case the pentest destroys you data, but in case you are really really breached by the bad guys, in which case you might want a non-contaminated backup to roll back to.

  • @medivh probably, yes. You'd hope the company in question would furnish the pentesters with some documentation about their system so the guys know how to execute something that doesn't break it... –  Aug 12 '13 at 10:05
  • @medivh It doesn't even have to be that complicated. Most automated UI testing tools will fail if they can't handle unexpected alerts. – Fax May 04 '20 at 13:19
15

I run penetration tests quite regularly on my sites.

The first time I ran one I ground the site to a halt and flooded their mail server with spam.

I then tweeked a few forms, to get rid of the spam and it allowed be to identify and fix all the other known holes, without grinding the server to a halt.

I was frankly amazed at the deviousness of the exploiters, I had to plug holes I hadn't considered.

But the thing was, prior to running the tests, I thought my sites were secure. I had followed best practices, as far as they went, for website design, but there were still problems.

Now, with the benefit of hindsight, I appreciate that testing may affect my production site.

So I plan ahead.

I make sure that everything is backed up first.

If the site is really, really critical, I will create a complete clone, then test that clone first.

I have learned from experience that if you create the clone, create it on a different server. Otherwise, when the clone grinds to a halt, the server will still affect your production environment.

But I still carry out my tests. How do you think hackers gain access to sites? They presumably run tests like these themselves, in an unauthorised fashion. So until your site is protected it will always be vulnerable. It is much better to get the tests done first, with the precautions (backup etc) in place, than have to pick up the pieces when you least expect it.

So, yes it is acceptable, but it is also predictable, and should be managed accordingly.

Relaxing In Cyprus
  • 271
  • 1
  • 3
  • 2
    you can put the test clone in a throttled VM if hardware is tight. this way when it grinds to a halt it only grinds with 25% of CPU and prod still has 75 to play with – ratchet freak Jul 18 '13 at 10:20
  • 2
    you should never (pen)test on a production server (virualized or not) the vm application can have bugs(ex: leak mem in kernel-space) – borrel Jul 18 '13 at 18:39
  • If you create the clone on a separate server using the backup you took, you are also putting your Disaster Recovery skills to the test. – Relaxing In Cyprus Jul 18 '13 at 21:44
2

The answer to your title is "Yes", and the client needs to know this!

This disclaimer is not a case of CYA, it is instead vital information the client needs in order to prepare for the pentest. I am not a pen tester, but IMO this should't be buried on the last page in the fine print, but should be on page 1 of the contract, front and center and in a big bold font. The client company needs to prepare for things to go wrong -- backups need to be made, recovery plans created and put into place, etc. Letting the client believe that a pen test is risk free is irresponsible. You are by definition attempting to trigger incorrect behavior, with no control over how far that broken behavior extends or does.

The answers to the 3 questions in the body of your post are: (1) no, you can't make any guarantees about others code (2) a limited form of pen testing might be done without INTENTIONALLY modifying or creating data (if you exclude any logging that the app may do) but the results would not be complete, and finally (3) you should ALWAYS include this disclaimer -- as much for the clients benefit as yours.

The tester is basically looking for a flaw to exploit and can in no way guarantee that they won't find a flaw that does damage. Consider a typical software disclaimer for something that is NOT intended to encounter or create flaws, and then consider that you probably didn't write the code you are testing...

Although the story may be fictional, consider The Spider of Doom!, not a pen tester and with no expectations of causing damage. Bad implementation and, boom, massive data loss.

jmoreno
  • 496
  • 2
  • 9
  • It would be unusual, but not irresponsible, to have a contract where you are not to intentionally modify production data in a non-standard way. It is technically impossible to guarantee that an unknown code base would not do damage upon NORMAL use, let alone the kind of unusual use that is involved in pen testing. I see that I should expand upon my answer a bit. – jmoreno Jul 22 '13 at 16:49
1

Probably you can assure that only under certain conditions. Obviously Pentest is trade-off between many factors, some factor are directly antagonist to system availability and one of those is the depth of your analysis. Some people could argue that a pentest without exploitation isn't a pentest.

In the past I had to pentest some SCADA systems which are quite famous for their fragility and I can assure you it is possible to tune most of the tools to be gentle enough. For example disabling NMAP fingerprint and incresing delay between packets helped a lot.

However to answer your question, personally, I will NEVER agree to put a clause to guarantee that to a client of mine. I will give him my word but business-wise you can't be responsable if their application breaks by itself during your pentest.

Francesco Manzoni
  • 21
  • 1
  • management just sees Penetration Test as "expensive" and Vulnerability Assessment as "cheap". If they pay you enough is **ALWAYS** a pentest. – Francesco Manzoni Jul 19 '13 at 10:33
1

"Can a skilled and professional pentester always assure that no data will be deleted or modified in production during a pentest?"

No.

"Can a pentest really be done if the pentest team has the limitation that data cannot be created nor modified?"

Don't think so..

"Should the pentesting company always include the disclaimer clause just in case?"

An internal agreement is going to look much different from a third party agreement. I don't know of any pen test company that doesn't have limitations of liability insurance, among other protections...

TildalWave
  • 10,801
  • 11
  • 46
  • 85
user29367
  • 11
  • 1
-3

Penetration testing should follow a methodology where tests that can cause damage are done only in non-production scenarios, such as a staging or lab environment.

The real issue is not deletion or modification of data because testers can leave those test cases to non-production systems. The real issue is leaving behind new data, such as error logs or superfluous erroneous files that reveals confidential information, or even the existence of the pentest itself, including the penetration tester's client-side data material.

Another issue prevalent in penetration testing is the revelation of other clientele data, work email, or browser bookmarks during demos or screencasts.

atdre
  • 18,945
  • 6
  • 59
  • 108
  • Penetration tests lots of times are done in production and you cannot guarantee that a test that seems to be inoffensive delete something due to the behaviour of a web for example. – kinunt Apr 19 '15 at 18:10
  • I can; maybe you can't – atdre Apr 19 '15 at 23:51
  • 3
    I think that you have not read the rest of the answers neither the valid answer. How can you predict that a button that says "Search" deletes information in the database if a programmer has programmed that functionality and you have not access to the source code? – kinunt Apr 20 '15 at 06:25
  • The difference between some people on this forum who follow the letter of the law, and me, who follows the spirit of the law -- is that I can reinterpret meaning from questions and "question the question". It's ok to debate. It's ok to have different perspectives. This is the world I want to live in. All of my pen tests are full knowledge. I always get the source code first. – atdre Apr 20 '15 at 10:31
  • Programmers are endlessly inventive in coming up with ways to make their programs fragile. For example, one piece of software I work with (a database system) can be crashed by a simple half-open portscan, with attendant risk of data loss. – Mark Apr 21 '15 at 03:27
  • I really don't think that any of the people who commented in response to my post or downvoted me understand the issues here at all. They don't have the experience. Testing must happen. Testing moves things forward. Testing some things in prod and some things in dev or test is the way to go. Understanding what bad can wrong in either is the zen of security testing and pen testing alike. – atdre Apr 21 '15 at 13:19