7

An Ubuntu server of my company has been hacked to carry out a DoS attack. I found the shellshock bug had not been fixed by my colleagues, and I think it's the problem. Then, I found an ELF file that sends thousands messages, and the script is auto-generated by something. Even if I try to remove it, it creates newly by itself by using a new name (in /boot, /etc/init.d). Besides, I see the netstat command doesn't show me all real open ports. Maybe has the command been replaced? How is it possible to re-install it?

Gudradain
  • 6,941
  • 2
  • 26
  • 44
roghan
  • 97
  • 1
  • 5
  • 54
    Wipe the machine and re-install it from scratch. – Stephen Touset Nov 28 '14 at 11:37
  • 1
    For now I cannot re-install the system since it's an important customer's server. We need to restore it, and soon we will schedule the configuration of a new server. – roghan Nov 28 '14 at 12:02
  • 23
    It's important to take the server offline as quickly as possible. You cannot trust any data passed through the server, and the attack may already have any data stored or accessible by the server. Further, it could be involved in an attack against another person (DDoS for example), and you wouldn't want to be participating in an attack. – Chris Murray Nov 28 '14 at 12:05
  • Yes, the server is down now. I can log in to it only locally in order to solve the problem. As soon as I fix the issue, I will restore its Internet connection. – roghan Nov 28 '14 at 12:23
  • 1
    You could try my little [ddosBotKiller](http://f-hauri.ch/vrac/ddosIrcBotPerlKiller_pl.txt) I recently wrote for this, and let my know if it work. – F. Hauri - Give Up GitHub Nov 28 '14 at 14:19
  • It can be a useful forensics exercise for you. I'd suggest that you use this opportunity to learn as much as possible, the gained experience can be quite useful! If you can take your time with it, that is. – Milen Nov 28 '14 at 15:17
  • 1
    Throw out the computer. See http://spritesmods.com/?art=hddhack&page=6 – ike Nov 28 '14 at 16:05
  • 12
    "For now I cannot re-install the system since it's an important customer's server" - you know how weird that sounds?. important customers' stuff should be on servers which can be rebuilt in seconds... see puppet/chef or any similar tool. whatever you do now, please wipe that machine later. – Karoly Horvath Nov 28 '14 at 18:09
  • 2
    Burnerate the server. – Zoey Mertes Nov 28 '14 at 23:52
  • 1
    Apparent duplicate of http://security.stackexchange.com/q/24195/971 or http://security.stackexchange.com/q/67124/971. – D.W. Nov 29 '14 at 01:08
  • Hello Karoly, unless you have a good baseline strategy, Puppet can take hours to rebuild a machine, not seconds. Also if they don't have a DR or Failover strategy it is unlikely they will have the "extra" machines required by Chef or for Puppet as it sounds like a shoestring budget. Puppet is generally used for any "quarantined" instance that needs to be regressed to older or custom packages for stability. Sparing matters, what if it was a hardware failure ? – mckenzm Jul 04 '15 at 22:36

2 Answers2

59

You should "nuke it from orbit": wipe and reinstall the OS and applications from clean source media, and then carefully restore the data from backup.

Graham Hill
  • 15,474
  • 37
  • 63
  • 6
    +1 This is the only really reliable way to deal with a compromised system. I would clarify that for anything possible, it would be better to reinstall software from scratch instead of using the back up. Only actual user-generated data should really need to come from the back up. – jpmc26 Nov 28 '14 at 18:53
8

It's always a question of effort vs result. If an attacker has gained complete control over the system, there are thousands of things that can be replaced or trojanized. Hunting them down one by one is a very time consuming task.

If the attackers aren't very sophisticated, you can start by searching for files that have been modified recently. One starting point is to look for files on your system that have last been modified in /usr/bin and /sbin (where most commands are).

ls -ltr /usr/bin

Check what's going on in /etc by looking at files that have been modified in the last 10 days:

find /etc -mtime -10

These are really basic first steps, and they don't cover looking at currently running (and potentially hidden) processes, that may hide any recent changes to the system. That's why this can turn into a cat and mouse game that can last for weeks if you miss just a single point of entry.

Unfortunately, reinstalling is the best option - least time consuming and almost guaranteed result!

Milen
  • 1,203
  • 8
  • 12
  • 5
    that's provided `ls` and `find` have not been modified themselves. – njzk2 Nov 28 '14 at 15:30
  • Good point, that's why I mentioned in my original answer "if the attackers aren't very sophisticated"... :-) – Milen Nov 28 '14 at 15:32
  • 1
    otherwise you'll need to do the analysis from the outside, by mounting the disk on a clean machine (noexec obviously) and check it with clean tools. – njzk2 Nov 28 '14 at 15:36
  • 1
    What about a hard disk firmware hack? http://spritesmods.com/?art=hddhack&page=6 – ike Nov 28 '14 at 16:07
  • 4
    Assuming netstat and ps were compromised I'd say the attackers are a bit sophisticated (it's not your usual script kiddie IRC bot) and there is a good chance that find and ls are compromised as well. –  Nov 28 '14 at 21:37
  • "if the attackers aren't very sophisticated" How can one know that? – Siyuan Ren Nov 29 '14 at 02:59