Linux command logging?

Question

Recently someone broke into one of our linux servers as a non root user.

An excerpt from .bash_history:

[...]
perl
perl
set +o history
set +o history
set +o history
passwd
sdfsdf
passwd
exit

I assume that set +o history toggles command logging to .bash_history

Is there a way to make it so history logging can't be disabled?

Is there a way to find out what happened if history is disabled?

score 14 · Accepted Answer · edited Jun 18 '11 at 15:55

14

Bash history won't help you against a semi-competent attacker in the usual case.

There's an alternative: The auditing subsystem. Install auditd and configure it, so it logs for example program executions and other things you configure. In basically logs the system calls that are made according to filters you have created.

Now, if you really want to depend on bash history, you can make it more secure: Recent bash versions can log to syslog instead of a local user file. Then, syslog can be redirected to a remote syslog server if that's imporant to you.

edited Jun 18 '11 at 15:55

nealmcb

20,693
6
71
117

answered Jun 17 '11 at 13:59

john

10,998
1
36
43

I'm a bit fan of the auditd subsystem. For anything beyond default setup it's a pain to configure, but gives you an abundance of data. – Scott Pack Jun 17 '11 at 18:43
This can be "solved" by performing chattr +a (append only) on the users .bash_history files. This however can be avoided by turning of the history feature. – Dog eat cat world Jun 30 '11 at 15:07
I disagree, I don't think you can secure the bash history at all. Any semi-competent attacker could simply use `ssh root@yourbox /bin/bash` to execute bash without a login, leaving no `wtmp` entry and leaving `.bash_history` unmodified (thus nullifying the remote syslog approach). Be assured, an actual competent attacker will leave no stone unturned in covering their tracks. – Daniel Hanrahan Oct 08 '12 at 23:52
@QuasarDonkey: At least (if not altered, or when syslog-forwarding is in place), one can see the connection in /var/log/secure. – Ursula Dec 21 '13 at 01:03

score 9 · Answer 2 · edited Mar 17 '17 at 13:14

An attacker who actually wanted to hide his tracks would just edit the history file by hand.

You can make it harder for a non-root user to hide commands that she's already typed into bash. But this only requires the attacker to be moderately competent.

If you make the HISTFILE variable read-only, then bash will save the last $HISTFILESIZE commands into that file when it exits. You can make HISTFILESIZE (and HISTSIZE) read-only as well and set them to a large value. This is very easily defeated though, for example by using the history built-in to sanitize the history before exiting. And since bash writes history entries when it exits and not after each command, just exit bash with kill -9 $$ and no history will be saved. Zsh can be told to save the history after each command, but not in any way that the attacker can't trivially undo.
Under Linux, you can make the history file append-only with chattr +a ~user/.bash_history. Only root can remove the append-only attribute from a file. As seen above, this only means that if the attacker makes a mistake, she won't be able to erase her traces.

Even if you managed to have all commands from a particular shell logged, this wouldn't be useful for security. The attacker could just emit commands from another application, in a way that wouldn't raise any red flag and be plausibly deniable. (zsh, fish: ok, so he prefers that shell. emacs, vi: so he edited a text file. ssh: presumably he just wanted to connect to another machine. ~/bin/foo: it's just an innocent script, but who knows what it was two hours ago?)

Linux (like most unices) has a simple command auditing system called process accounting that's usually enabled by default: the executable name, time and executing user for each executed command are stored in /var/log/account (the exact location may vary between distributions). You can view that log file with lastcomm. This may help you in your current forensic endeavor, but it can be defeated by a moderately competent attacker (e.g. by hard-linking every executable to /var/tmp/$name/ls) — all you get to know reliably is that there was activity from that user at a certain time.

There are more thorough auditing packages, in particular the audit subsystem that john suggested. Search for auditctl to find a few examples of how to set it up. This is typically not turned on by default as it comes with a performance hit (I don't know how bad this is in practice).

Another simple tool that's (at least partially) default-on is file times. If a file's ctime is from before the intrusion, then the attacker didn't modify it. If you have file access times enabled (this used to be the case by default, but is now often disabled for performance; check the *atime mount options), .

All of that assumes the attacker didn't gain root, otherwise all bets are off. Consider that the attacker may even deliberately have left traces that make it look like she didn't escalate her access by cleverly wiping only the more compromising traces. If you're at all worried, reinstall that system from clean backups (as well as other systems that the attacker may have reached following the initial break-in).

score 8 · Answer 3 · answered Jun 17 '11 at 11:35

8

it won't really help if you allow executing non-shell commands from the shell. For example - you can omit bash by running perl (or some other scripting language) and call commands from there - it will evade bash logging.

answered Jun 17 '11 at 11:35

martin

356
2
5

score 1 · Answer 4 · answered Jun 17 '11 at 18:39

1

Set all the users shells to the bash restricted shell. In /etc/passwd set the shell to /bin/rbash (it should be a simlink to /bin/bash), or /bin/bash -r

The Bash restricted shell prevents users from changing shell options like command history.

answered Jun 17 '11 at 18:39

this.josh

8,843
2
29
51

score 1 · Answer 5 · answered Jun 18 '11 at 08:27

Also, to supplement the ideas listed above, I would install and configure logwatch to run daily (or more frequently) via cron and email you the results for auditing purposes.

Installation and configuration:

Debian-based Linux type the following command-

apt-get install logwatch

Redhat-based Linux type the following command-

yum install logwatch

To customize logwatch go to /usr/share/doc/logwatch-*/ directory and read the file HOWTO-Customize-LogWatch

Edit logwatch.conf file:

vi /etc/logwatch/conf/logwatch.conf

Or

vi /usr/share/logwatch/default.conf/logwatch.conf

Read man page of logwatch for more information.

Linux command logging?

5 Answers5

Linked