It there a reliable method of “monitoring” system calls under Linux?
There is strace for example to monitor system calls and signals. Is there a way for a process to dodge out of strace? If yes, is there another reliable, secure method of "monitoring" system calls (and, maybe receiving signals), that a process cannot escape from (assuming a proper Linux implementation)?
On Linux, you can reliably monitor a selection of system calls or file accesses with the audit subsystem. Make sure the auditd daemon is running, then configure what you want to log with auditctl. Each logged operation is recorded in /var/log/audit/audit.log (on typical configurations).
You'll find simple examples of auditctl usage on this site, on Server Fault, and on Unix Stack Exchange.
strace or associated programs using ptrace are reliable ways of monitoring system calls, but I would be wary of using them on a malicious program. I couldn't tell you how off the top of my head, but it should be possible for a monitored program to make the right ptrace calls to evade monitoring.
Note that a malicious program could spawn a process that is not audited and can execute code that won't be logged. For example, it could use mmap to write to a file without the file contents ever appearing as the arguments of system calls, make this file executable and spawn a process executing it. The spawned process can typically break the process tree with something like ssh localhost. If you audit all the processes executed by a given user (as opposed to only a single process and its descendants), you'll be able to log everything.
If yes, Is there another reliable, secure method of "monitoring" system calls (and, maybe receiving signals), that process can not break (assuming proper Linux implementation) ?
To re-iterate in a slightly different way what D.W. has already said, ptrace is a system call that strace, gdb and the like make to monitor a processes' actions. There are two problems with this approach:
ptrace, provide you the output of another process or some other such malfeasance. Processes are not always written to submit willingly to debuggers. You might like to read this challenge set (win32 focused - see the very first entry and keep reading for making it difficult) from an appsec company (I have no links with them). Whilst focused on the IsDebuggerPresent() mechanism, similar solutions exist for ptrace. If you want to see this in the wild and have skype installed on a Linux box, try debugging it.
Repeating those techniques here, there are two clear anti-ptrace mechanisms:
if (ptrace(PTRACE_TRACEME, 0, 0, 0) < 0) {
printf("being ptraced\n");
exit(1);
}
This method relies on the fact a process cannot be traced twice. If you can't ptrace yourself, you're being ptraced.
struct timespec spec;
signal(SIGALRM, SIG_IGN);
alarm(1);
spec.tv_sec = 2;
spec.tv_nsec = 0;
if (nanosleep(&spec, NULL) < 0) {
/* EINTR */
printf("being ptraced\n");
exit(1);
}
To explain this one, have a look at nanosleep() and read the original article. In simple terms, nanosleep() is a non-restartable system call and will return early when a signal is handled by the process. This particular process will, when not being debugged, just not handle that particular signal and so will not be woken up. However, a ptraced process will handle it, causing nanosleep to return early. Another example where this happens is the select() system call.
Ultimately, you can mitigate the effects of 1 by ensuring the integrity of your system prior to starting and applying sufficient security measures and an appropriately configured kernel.
What can you do reliably about 2? Not a lot without modifying the original binary code, since any technique for debugging is going to introduce observable inconsistencies or implementation issues somewhere.
tl;dr ptrace will help you if the target process wasn't written with debuggers in mind.
The Linux Audit Framework supports syscall monitoring - I believe it is what you are looking for.
Yes. strace is a reasonable way to monitor system calls and their arguments, as long as the process being monitored is not malicious. If the process being monitored is malicious and was written to evade strace, I expect that it can do so. strace was not written as a security tool, and I can hypothesize several ways that the process might defeat it. See, e.g., Robert Watson, Exploiting Concurrency Vulnerabilities in System Call Wrappers or Tal Garfinkel, Traps and Pitfalls: Practical Problems in System Call Interposition Based Security Tools.
If you are worried about malicious code, you'll want to use a sandbox that was designed for security, rather than a tool like strace that wasn't designed for security. The general approach to building such a sandbox is to use system call interposition both to contain the monitored process, and to monitor its actions. One portable method is to use ptrace, though this can introduce a non-trivial performance overhead as it forces a context switch on every system call. On Solaris, you can use /proc; /proc lets you specify the subset of system calls that you are interested in wrapping, which lets you achieve better performance at the cost of compatibility.
Take a look at Plash, Systrace, and Subterfugue, to see some worked systems that use these sorts of methods. Also look at Chrome's sandbox, which uses a variety of mechanisms (including seccomp on Linux).
What about monitoring the system calls on the side of the kernel using an external gdb instance.
This could be done by setting up a virtual machine that is configured to run the code of interest. Then QEMU and KVM (by my knowledge) have to be configured to open a port for gdb debugging of the kernel. (See guides blow.)
If this VM is started gdb could be attached to its kernel during boot.
The next step is to set gdb properties and breakpoints to fire on any execve (and consorts) that sets the code of interest as new program. Then let the gdb run until it hits this breakpoint. At this point during the execution of the program the pid of the process running the code of interest could be extracted and breakpoints could be set in gdb that are hit (in the kernel code) on any system call of this process (including fork and execve calls that might lead to additional processes to observe).
In theory this should be a good solution that is hard to doge.
One problem is that everything in the guest system becomes horribly slow, and you might get a huge amount of unwanted calls as bycatch (that you have to filter using gdb...). Additional gdb might have to be extended using python to get the conditional breakpoints working with the required conditions (especially for an automatic child process detection).
Guides how to connect gdb to the guest: Whamcloud Wiki, ReadHat Helpdesk, Stackoverflow
(I did not try these guides. I used gdb some years ago to debug some details of the kernel for a students project. There I used a simple condition on a breakpoint to detect fork calls of a specific process.)
On top of these there are some other techniques to debug a kernel.
PS: Be aware that there are ways to escape a virtual machine (an old example).
