From reading around on the internet I get the impression that barring physical damage, deleted data can be always be recovered using sophisticated digital forensics.
For this reason the advice is that you should encrypt your data.
So at what point is data irrecoverable even to sophisticated digital forensics?
This depends a lot on what medium is used to store the data and what you consider "irrecoverable".
"Deleting" data mostly does not what most people think it does. Simply put, after a standard deletion, the data isn't gone but only the link between "Data XY lies at 0x000000" and the actual storage location 0x000000 is being deleted. Your data is still at 0x000000. File recovery programs will easily restore this data.
HDDs: When deleting data or formatting/deleting partitions, you are not actively erasing the data but also only making it not actively indexed. To actually erase the data, you need to overwrite every bit.
Actually, to make it really secure, you need to do that multiple times as data on magnetic drives can be recovered even after overwriting it with zeroes.
SSDs: Wear leveling and other features prevent SSDs from actually erasing or overwriting sectors even if specifically advised to. Instead, the SSD controller writes the new data to somewhere else and creates a pointer from the old to the new, actual location. Your old data still lies on the disk.
Most SSDs offer a Secure Erase feature that is designed to tackle those exact problems.
There are research papers dedicated to erasing data on SSDs:
NIST offers an overwriting standard NIST SP-800-88 Rev. 1 that elaborates on 64 pages on the difficulties of erasing data on different mediums.
While the current leading answer dives into the technical details of data recovery, I will (try to) take the broader approach.
How do we define "irrecoverable"?
Depending on the circle you may associate with, "irrecoverable" could mean only some insignificant portion of the data is recoverable.
While I don't practice law, this could prove "sufficient" as a legal defense in some countries. Any critical metadata (timestamps, permissions, etc.) would be missing and thereby limits arguments given by prosecutor against a defendant's alibi.
In other circles—especially malware authors—using XOR operations or base64 encoding makes data "irrecoverable" to the general populous. Good enough for their purposes and easy to implement.
The answer also depends on time scale. Are we done with the data, or do we still want to access the data over time?
I assume that the author is "done" with the data on a given storage medium.
How does data become "unrecoverable"?
It largely depends on:
At a higher level, most operating systems lack a way to truly validate if data is actually "gone" or "irrecoverable". This is important to note because auditing is a cornerstone in information security.
This is because most software (operating systems and such) don't have a view into the raw data storage — how the controller handling storage medium reads and writes data.
For instance, magnetic storage mediums typically require multiple passes (and multiple orientations) to ensure any residual fields are dissipated. Last time I checked, no firmware on a hard drive exposed a "read the direction and strength of the magnetic field at X, Y, Z". They offer a "tell me what bits are in sector/block X".
How does encrypting data make it "irrecoverable"?
It is believed by many that because the sensitive data was encrypted before the storage medium controller got its hands on it, that is only part of the battle.
Claiming "encrypted with AES-256... military-grade encryption" is common in many pieces of marketing literature. Encrypted data can be easily recovered if it uses one or more weak algorithms, implementations, keys, or seeds. An example of this is "export grade crypto".
Depending on the level of sophistication of the "attacker", a false sense of encryption won't matter if attackers look at the storage medium at a high level (mount the device in an OS), or low level (with a scanning electron microscope).
The advice is that you should encrypt your data.
While I may receive some flak for saying this, most consumers don't want or need encryption for data at rest.
Before some flame war starts, I will explain.
Information security deals with risk assessments. If a storage medium is guaranteed to not hold sensitive data, why would it be encrypted? Encryption often incurs a performance penalty.
Most everyday people would find no encryption on data at rest helpful. As much as it pains me to say it, a lack of encryption allows for easy data recovery. In IT support, it is vastly more common to be asked "I lost my files" than "I want to lose my files".
If you do banking on a device with "encryption", you have many other problems besides the data at rest being secure.
Most amateur street thieves are interested in wiping the device and selling it off. Encryption doesn't help in this case, a factory reset makes data "irrecoverable" to amateur street thieves.
Taking your storage medium to a shredding company is more than sufficient if you no longer have any use for it. That is what I assume the question is asking about.
What if I really need to make my data irrecoverable on some level?
What if I really need to make my data irrecoverable over time while still accessing it?
Assuming you do have sensitive files, corporate secrets, or some other legitimate reason, 3DES or even RC4 may prove enough to encrypt your files — given a strong enough key, reasonable implementation, and your potential attackers belonging to the general (and uniformed) population.
You should be asking who or what you are protecting your data from/against.
If you are looking for generally "unrecoverable", pick a random reputable cipher, give it a completely random key that even you don't know (somewhere in the order of 256 bits), and wait for the process to complete.
Assuming every line of code in the pile of software did its job (and every transistor in every chip) this should prove "irrecoverable" to the wide population.
If you have hesitations, encrypt the data on the drive every decade or so. This way you can ensure that you stay "up to date" on any weaknesses in the cryptography you used.
Be sure to flush any caches that may exist, devices use these for performance and likely contain clear-text contents.
If you are interested in securing data over time, that's a whole other area of information security.
On many kinds of magnetic disk drives, overwriting a sector even once with random data is likely to render the data permanently irrecoverable even with advanced forensic techniques. A variety of factors may cause some data to "escape" destruction, however. For example, if a removable disk is written with a drive whose heads produce a stronger magnetic field than the drive used to overwrite it, the width of the stripe that is written by the first drive may be slightly greater than the width that is erased by the second. If e.g. the first stripe was 10% wider than the second, it would be impossible to read the outside portions by conventional means, but if one had a drive whose head that was slightly narrower even than the second, and whose position could be precisely controlled, one could erase the middle 90% of the later track, then try to read the inner and outer edges, then move the head to erase the middle 95%, then try to read the inner and outer edges, etc. If one were to uniformly polarize the entire area covered by the new stripe, then attempting to read the outer portions of the track would yield a signal that was 1/20 of the original speed, but had the newly written data removed.
I don't know whether the amplitude of such a signal would be sufficient to allow reliable recovery, but if one knows where a 256-bit private key is stored and one can identify bit values with 99% reliability per bit, a brute-force search which (oversimplifying) starts with the bit pattern on the disk, and then all patterns that differ by one bit, and then all patterns that differ by two bits, etc. it may be possible to crack the key far more quickly than if one didn't have a partially-recoverable key to use as a starting point.
On the Apple II, if one wants to scrub a floppy, it's possible to use programs that will move the head in half-step increments rather than full-step increments while erasing the disk. If the drive is working properly, data erased by such a program will be well and truly gone. I don't know of any hard drives that offer similar abilities, however, other than--as mentioned elsewhere--by using encryption.
While this is true, and used to me a lot "more true" in the past (is there any such thing as more true?) it is not necessarily a problem.
Of course, if you want to be 100% certain because you are in the top-5 of the CIA's wanted list, there is no other option than to physically destroy the drive with a wrench, and with fire, after formatting it (though, honestly, for 99% of all users, this is total nonsense!). Oh, and shoot everybody who may have seen your face.
But is deleted data recoverable or not? Whether or not that's the case is hard to tell.
First of all, it depends on what "delete" means. For example, deleting a file typically, first and foremost, deleting metadata in the filesystem, this merely marks the actual data blocks as being available for being reused/overwritten eventually. There are exceptions, and there exist "secure delete" tools which actually overwrite the file, sometimes several times. Whether or not these tools actually work is debatable on modern hardware (and with some filesystems). Deleting data from memory will make it unrecoverable (pretty certain!), unless the memory page has been written to swap. In which case you cannot tell if it can ever be deleted at all (short of burning the drive in a furnace). Note that on journalling filesystems, even if the data is deleted, a copy may still be in the journal. A copy-on-write filesystem may, surprise, have a copy that you don't know about, and are unable to overwrite at all.
It also depends a lot on which century you live in. With 1990s tech, it was pretty mainstream to reconstruct data that was overwritten half a dozen times from residual magnetism. On the other hand side, at that time, secure erase was reasonably safe, too, since you had kind of a guarantee that you actually overwrite the data, or get a failure.
With modern drives, things are a bit different. On the one hand side, restoring overwritten magnetic data is a pretty daunting task nowadays. I don't know whether residual charge could be used to restore data in flash memory, but if one is determined to do it, then at least in theory... why not.
On the other hand side, you never know when something is overwritten, or if at all, or if the same sector that you write to is actually the same sector. Often, it is not, and on SSD it is practically guaranteed that this isn't the case. Wear-levelling and reallocation is 100% transparent, and you have no way of telling. You might not be able to overwrite a sector at all, no matter how hard you try. Modern drives (not just "hybrid" drives which are already dying out again) may have several levels of storage, using one or the other for caching. Some SSDs work that way with SLC/MLC combinations, for example. So, even if you do delete something you never know for sure if there's not a copy.
On the other hand side...
Older drives used some sort of bit mixing for technical reasons (more favorable to the hardware, both on magnetic and solid state). Modern drives almost exclusively use AES which is not really significantly better at mixing bits, but it's readily supported in cheap, omni-present hardware, and you can write "AES, secure!" on the box, which is good for sales.
Most, if not all present-day drives are SEDs (self-encrypting drives), although that does not automatically mean that they are secure. It comes down to how/where the decryption key is stored, and how it is accessible. On a self-encrypting drive that has a single built-in decryption key which everyone can access, the fact that it is self-encrypting is irrelevant. A drive where the decryption key is only available to users who have authenticated to the BIOS is a different story.
The same goes for the secure erase / factory reset function that virtually all drives support. Some (Seagate, much to my annoyance, not only is it insecure, it also takes hours) actually overwrite the drive with garbage. Some (Samsung, for example) just erase the encryption key, instantly rendering all data unreadable. Some, I've been told, keep a copy of the decryption key around although you said "secure erase". Not sure if that's actually true (hardly imaginable), but it sure is a theoretic possibility. You have no way of telling for sure.
What does it mean in practice?
For most people, using a properly configured, no-crap self-encrypting drive is just good enough. Before throwing the computer away, secure-erase the disk, and you're good to go. This will prevent a random person pulling your disk out of the trash as well as the average criminal from accessing your stuff.
For most people, just deleting a file (which doesn't happen to be a super special secret file) that isnt needed any longer in Windows Explorer is just good enough.
If you want to be 100% certain that some vital data cannot be accessed if the computer is stolen, you can create an encrypted partition or container on the disk (Veracrypt or the like). But really, few people actually need that.
There's roughly three categories of secrets in the world:
So, long story short, just delete your files normally, and erase the disk when you toss it away.
You should distinguish claims about "theoretical" and "practical" data recovery. By the use of the word "always" I assume you refer to the practical aspect.
Theoretically, if you get a fluctuating value which is interpreted as "1" 51% of the time and as "0" 49% of the time, it still provides some information about the system it's coming from, as opposed to a perfectly random value, so you recover useful data by reading it. If you know for a fact that the bits you are reading belong to an encryption key, you can use this technique to recover it: even guessing 10 correct bits speeds up the brute forcing by a factor of 1000.
Practically, data recovery means you can restore e.g. a lost file from a thumb drive. This means reading millions of bits without a single mistake. A file with 1% of bits flipped is essentially useless. In many cases, even flipping just a dozen of bits makes the recovery impractical.
Theoretically, if you overwrite a file, it may still be recoverable: there's no guarantee that every single bit of it was overwritten. Practically, if you overwrite a file, you kiss it goodbye.
Encryption is not a method of data destruction, encrypted data is explicitly intended to be recovered, but only by authorised persons. By design, data that is encrypted should always be recoverable, although it should be prohibitively difficult for unauthorised users.
To answer your question as written, there are different lengths you can go to delete data, "irrecoverable" depends on who it is trying to recover it and how much money/time they are willing to invest to do so.
Deletion:
Overwriting:
Residue purging
Physical Destruction
To answer you question generally, residue purging and physical destruction will make your data irrecoverable, the specific processes required to do this will depend on the storage medium.
In terms of storage medium (overall integrity):
Magnetic Storage Medium:
For magnetic storage mediums like HDD and Floppy Disks, the state of the magnetic film or platers determines the ability to extract any data from the storage medium. If an HDD were to be jack hammered, the HDD casing and platers would be destroyed into many pieces. Another scenario where the HDD or a Floppy Disk got induced to extreme magnetic forces, the magnetic fields on the disk would be rearranged and the data would be lost.
In the case of a jack hammered hard drive, even if someone tried to put the pieces of hard disk platters together like pieces of a puzzle, there would still be some data lost. Presuming a specially designed magnetic reader, that moves over the plater, were to be created to read over as many blocks on the disk platers. It still will not be able to recover majority of the data. Magnetic objects that break or have their magnetic fields rearranged, and the edges of the break will also experience the most magnetic rearrangement. As for deleted data on the disk, it depends on what can be read and how sophisticated forensic technology is at that point to read fragmented parts of an HDD.
A strong magnetic field affect could possibly make it even harder or near impossible to get back the deleted data. Unlike the effects of breaking or banging an HDD, the effects of strong magnetic fields can rearrange all the magnetic fields of all blocks on the HDD platters. If there was deleted data on the disk, it can be considered at this point that the data is irrecoverable.
Flash storage medium:
A surge of electricity induced on the NAND memory chip of a flash drive or SSD can fry or damage it. Depending on how the memory chip has been fried or damage, can determine how much data can be recovered. Now consider the situation where you use an electric chainsaw and cut straight thru the middle of the memory chip, perpendicular to the edges of the chip that have the circuit board connector pins. Not only the memory chip is cut in half, but it would require a lot of electrical engineering to be able to connect the two halves together or use them separately on a circuit board or some forensic equipment.
As @dmuensterer stated, deleted files is not automatically overwritten or deleted from an SSD. This is for wear leveling and SSD life enhancement purposes. They also said how special tools are required to carry out such tasks for really deleting deleted files immediately.
With the above two paragraphs into consideration, it can be assumed that the current state of the memory chip and how the deleted files data were treated, determines if the deleted data is recoverable or irrecoverable.
Optical Disks:
Some formats of optical disks like CD(±RW), DVD(±RW), and Blu-Ray (RE), when formatted for computer data, does support file deletion. The deletion on optical disks acts similar to deletion on a hard drive, where the file metadata and pointer is removed. When adding a new file or folder, the old data is over written. The only difference is the trace of the earlier data is a lot less. This is because physical structure is involved here. More writes over the same area will further change the topology of the surface. Thus, the deleted data's recoverability on optical disks depends on, how much information can be obtained from the topology of the surface of the optical disk.
Using a blowtorch is the easiest way to make the data irrecoverable. This is done by putting a blowtorch on the data surface of the optical disk and heating it long enough so that the data composite layer melts or changes shape at a microscopic level. At this point the data is lost and is irrecoverable.
In terms of Read and Write Cycles:
HDD:
Data on an HDD is not really reflected by read and write cycles in theory. As long as the arm does not hit platter the HDD can go thru multiple read and write cycles, there are no problems. Platters are made from metal, and metal objects never loses the property of becoming magnetized and demagnetized. We are referring to the platter that what holds the data, not the arm which only reads or writes it. In theory if a dead arm is replaced the HDD data is still accessible again.
SSD:
As others have said above, an SSD will stop working to a point where it may no longer be able to be written or even read. Because the data is stored in an electric state (not magnetic or physical), there will be no way to access the data again in that situation. A dead SSD can be considered that any deleted data is irrecoverable.
Optical Disk:
All rewritable optical disks have a write limit. There is no read limit like an SSD, although there is something called an optical disk age limit. Rewritable DVDs typically have around less write cycles than a rewritable CD (1). The pits or bumps on a rewritable DVD can be changed around 1,000 over the same spot(2). This is because optical disks are usually made from dyes. After data cannot be written over, the only way to destroy deleted data is to blowtorch the optical disk in the manner said previously. There is also a point where the optical disk can no longer be used (even read), and it is because of these dyes. The dyes can decay and deform over years (usually decades), after which the optical disk is no longer readable. In this situation the deleted data becomes irrecoverable. Referenced is a study carried out by NIST (3). Reference from somewhere else (4).
In terms of encryption:
Encrypted disk data containing deleted data, can be considered another avenue for making deleted data irrecoverable. However, there are a few catches. There are conditions that must be met which are:
Deleted encryption key with no copies is important here:
A deleted encryption key with no copies of it, means the only way to recover the data is by either using a flaw in the encryption algorithm or by brute force. Which leads to the final section.
A strong encryption algorithm with no feasible brute force:
An encryption algorithm that has a flaw, take RC4 for example, can gets its key found.(5). RC5 will take 105 years to break (6 and 7). Also an encryption algorithm that is too easy like Caesar Cipher will take no time to decrypt.
A strong encryption algorithm like AES-256, will take till eternity to brute force and thus deleted data can be considered irrecoverable for now.
