Techniques for ensuring verifiability of event log files

Question

Bit of newbie at the whole forensics stuff - but I'm trying to find out what I should have in place before an attack. While there is no end of material on the internet about forensics from seizure onwards, I'm trying to find out more about how I can make a secure record of events (specifically webserver logs) of adequate quality to be considered as evidence.

There are vague references to non-volatile media, hashing and signatures in the stuff I've read; these certainly provide a means for demonstrating a consistent snapshot - but do not intrinsically provide a mechanism for proving the data has not changed between initial capture and the snapshot, e.g. I could take today's log files and do a search/replace to overwrite the date with something else before committing the snapshot.

How does an electronic signature prove the data has not been tampered with between the initial capture and the signing - does it just support the signers assertion?

Must the integrity verification method be implemented in real-time? E.g. it's not very practical to write data directly to DVD, at best a track at a time is as near to real-time as you can get - but at a huge performance penalty.

Any pointers on content suitable for a non-lawyer? (pref with a EU/UK bias).

Answer 1

This is an excellent and important question. There are several important techniques to know about:

• Remote logging. Rather than store the log entries on the webserver, the webserver should be configured to send each log entry over the network to a log server. The log server should be a custom machine, configured for a single use (log recording only), and hardened. You should carefully minimize who has access to the log server, firewall it off from the outside world, and make sure it is running a minimum of services. This will help prevent tampering with logs after they are generated.

• Hash chaining. Another important technique is to use cryptographic methods to protect the integrity of log records, once they are stored. A log entry should be a pair X_n = (M_n, T_n), where M_n is the nth log message received, and where T_n = Hash(X_n-1) is the cryptographic hash of the last log entry. Use SHA256 or some other collision-resistant cryptographic hash function for this purpose. What this does is ensure that an attacker can't tamper with entries in the middle of the log, without being detected. The attacker can replace all the entries in the "chain", or throw away a suffix and replace them with something new, but this constraints what the attacker can do. Importantly, the crypto ensures that if you write any log entry to write-once media, then the attacker can not change any earlier log entry without being detected. That's huge.

  For more, read about secure timestamping using hash chaining.

• Write-once media. If you have write-once storage media, you can write the logs to the write-once media, and that will prevent tampering after they are written. Unfortunately, the choices here aren't great. As far as I know, they're basically:

  • CD-ROMs and DVD-ROMS are write-once, and allow appending if you use TAO mode, but also slightly clunky.

  • A line printer, printing onto paper, is a surprisingly effective write-once medium, as long as you keep the paper supply fed.

  • PROMs are write-once, if you use a Manchester encoding. Unfortunately, PROMs have mostly been replaced by EEPROMs these days, so true PROMs are hard to find.

  • Wikipedia also has a list of write-once media.

  • SanDisk has a SD card with 1GB of write-once storage, which is specifically designed for forensics and log storage purposes.

  Write-once media complement hash chaining extremely well. What you can do is store all log entries on some ordinary storage medium (e.g., a hard drive), and then once an hour store the latest log entry (or just its hash) to write-once storage. This ensures that an attacker who compromises or otherwise tampers with the log server can only modify records going back an hour, as well as (of course) all future records, but not records that were logged more than an hour before the compromise. That's huge.

• Replication. You can store multiple copies of the log entries on multiple servers, to provide redundancy and protect against tampering. An attacker who breaks into one server and tampers with it won't be able to tamper with the copy on the other servers.

  For this to effective, you need the servers to be independent, so that it is unlikely that an attacker can compromise them all. For instance, you might physically locate the servers in different locations or different machine rooms, set up so that no one person has physical access to all locations. You might have separate individuals administering them, so that no one person has log-in access to all of the servers. You might have one log server running on your system, and a replica hosted remotely (e.g., running in the cloud).

  Replication complements hash chaining very nicely. For instance, rather than replicating every log entry, you can replicate just 1 out of every m log entries. Also, if you don't want to expose private log data to some of the replicas (e.g., ones running in the cloud), you don't have to store the entire log entry: you can store just the hash T_n. This hash value reveals nothing about the log entries themselves, since cryptographic hash functions are one-way.

I'm sure there is lots more that could be said about this, but I hope this helps introduce you to several technological methods that are available to protect logs.

Answer 2

Usual disclaimer: I'm not a lawyer. Please seek legal advice for admissibility in a specific court case.

What I have been told by lawyers though is that logs are admissible in courts if they have been captured as part of the ordinary course of business. So you do not have to do anything specific for this.

You do have to satisfy the evidentiary requirements though that the other side may throw up. So having a chain of custody and evidence that the logs have not been tampered with maybe useful and increases the weight of the logs in arguing your point e.g. proving a specific user or machine was used in an incident. Chain of custody in layman's terms means being able to demonstrate the "seizure, custody, control, transfer, analysis, and disposition of evidence, physical or electronic". For forensics and especially in criminal investigations the authorities need to be able to show chain of custody for the evidence to be admissible in a court

For this exporting the logs from the source system to a central Security Event and Information Management (SIEM) system that the administrators do not have access to is a good measure. Archiving this to write once read many (WORM) media is also good. Some SIEM's will allow you to generate a cryptographic hash of the log with the timestamp and store this on a separate media or system which will allow you to prove by re-calculating the hash that the log has not been tampered with.

Obviously if you can hash and archive in real time or at least daily that will provide you the best argument in a court. But having it even if done monthly would provide you a better argument than not I would have thought.

Answer 3

Snare allows you to transmit an MD5 hash appended to every message via standard Syslog. You can receive this using pretty much any SIEM. For the most part SIEM's in the top tier all support hashing of retained logs. This is in main part for searching and indexing, but are possible to use as a component of the chain-of-custody.

In combination with a strong separation of duties and a WORM archival of the retained and checksummed logs you've got a good chunk of it. Without WORM archival (not every runs Solaris after all) you still have your backup tapes that can be used for the same purpose.

The tapes themselves, if encrypted, and managed by another individual can assist in building that strong chain-of-custody as well. At that point, archival documentation is essential, most companies have requirements for removal and passing their backup tapes with a strong chain of custody for legal reasons.

With those pieces in place you can have a strong set of evidence. Once you get the lawyers involved, well, that's a different story.