Cryptomining Malware - Ubuntu - Stratum tcp

Question

My LAMP server has been hit with some kind of crypto mining malware. Crontab is clear and unused, Clam can't seem to detect anything.

It appears to automatically create these files in my /tmp folder at random times of the day.

• phpIij8W8.c
• phpIij8W8_fkk2qr2rqjikiewz (Always has a random name after I delete)

Inside of this phpIij8W8_fkk2qr2rqjikiewz:

threads = 1
mine = stratum+tcp://44XNuUyCFUgjG23yPfCHp(cut for content...etc):x@monerohash.com:3333/xmr

They all begin with "php" so I'm assuming some kind of PHP script is on the server that is automatically creating these files in the tmp folder and running them.

I've attempted to use iNotify to determine what/how these have been created and all I've been able to determine is this:

File phpvlnebP_fkk2qr2rqjikiewz has been just created by MOVED_TO

Can anyone give me some tips to narrow down the script that is creating these random files? Or if there's some way to get iNotify to give me more information when the file is created/accessed. Or heck, if there's a better tool our there to diagnose this problem.

Ubuntu 16.04.3

Answer 1

I had same problem, and I've narrowed down, hackers were able to intrude in some old and non-upgraded, wordpress.

Probably the best and fastest way to see who is consuming how much time on your Ubuntu 16.04.3 server is to install htop

sudo apt install htop

then type sudo htop

It will show you under which username is eating how much CPU

And if you identify some process that's eating much cpu, you can check it by lsof -p <pid>

lsof stands for list open files, to see it's full set of commands, type man lsof

However, it all depends upon how is your PHP being executed, and what hackers actually have done to your system.

Another good way to see, what is Apache exactly doing is to enable mod_status

Usually, on newer ubuntu's it's:

sudo a2enmod status

And after that add this to your 000-default.conf website:

<Location /server-status>
    SetHandler server-status
    Require ip 127.0.0.1
    Require ip ::1
    Require ip X.X.X.X
</Location>

Replace X with your actual IP...

Or you can access it with lync for example from your server's console like

lynx 127.0.0.1/server-status

And the output should look like:

On my other post https://security.stackexchange.com/questions/172396/some-bot-keeps-posting-this-to-my-server/172571 you can see how to improve security of your server and prevent such attacks.

Answer 2

To be clear, your best bet here is to nuke it from orbit. You presumably have a copy of your application somewhere, and also a copy of your database. If you really want to try to figure it out, then you should probably start from the perspective that you have one or two intrusion vectors:

1. Your web application
2. Your system

While I'm guessing wildly, it seems that it is a safe bet that the PHP file you keep deleting isn't the actual malware, but rather a by-product of the malware. I know you know this, but to be clear: you haven't done anything to remove the malware, just the symptoms of it. Discovering and removing the malware itself will vary wildly depending on whether or not it is in your web application or in your system. Who is the owner of the file you keep deleting? If it is the user your webserver runs on, then the infection probably lies with your web application. If so, here's some details that might help:

Web Application Intrusion

1. You know when the payload is being created in your temporary directory. See if there are any suspicious looking requests in your HTTP logs that correlate with that time period. If the intrusion vector is your web application, then there are likely calls coming in through your HTTP server from a C&C server that is triggering the payload execution.
2. Immediately update your software application to the latest version, presuming that you are deploying third-party software (wordpress, etc...)
3. Does your web user have write access to the folder where the web application lives? If so, you're SOL. If they got in through your web application, this means that they have found a remote-code execution vulnerability. With write access to your application directory, that means that they could have installed backdoors anywhere they wanted to inside your application. Unless you have old backups and can determine for sure that nothing malicious was added to your code base (which is a big pain to do), your best bet is to NUKE IT FROM ORBIT
4. Conversely, it is possible that you have an open vulnerability which isn't patched, and your system itself is unmodified (more likely if your web server doesn't have write access to the web application). In this case the payload keeps returning because the C&C server keeps re-exploiting the security vulnerability. In that sense the problem will go away once you get your security problems patched. Make sure and upgrade all parts of your web application. If this is a home-grown application then you may very well be SOL: unless you can find and close your own vulnerabilities, nuking it from orbit may not stop them from getting back in. In this case you will hopefully get some help from your server logs. If you can find evidence of the initial intrusion in your logs, that will help you find and patch the vulnerability.

System Intrusion

This one is probably less likely, but it is possible your system itself was compromised (insecure service, crackable SSH passwords, etc...). Short of lots of digging through logs, it may be hard to determine for sure if it is your system itself that was the weak point. It will probably be much easier to determine that your web application is the problem (from my own experience, it probably is your web application that is the culprit anyway). Either way, if you can't determine for sure that it was your web application, your system is the only other culprit.

The answer

Either way, the trouble is that it is going to be very hard to determine for sure where the problem got started, and harder still to tell whether or not you got them out. Even if you manage to get the cryptominer to stop putting itself in place, what guarantees do you have that there isn't a more silent and more malicious process hiding elsewhere? Background process that scrapes your database and sends it off to a remote C&C server once a week? That's a great way to have the personal information of anyone who logs into your system get leaked. Not to mention the threat to your own passwords and account credentials. This is why, regardless of where the intrusion started, there is only one answer:

1. NUKE IT FROM ORBIT
2. Spin up a new server following all best practices (minimum services, proper firewall, everything fully patched)
3. Use all new passwords for all services.
4. Find a copy of your application that is KNOWN TO BE CLEAN. If using third party software, make sure and fully upgrade/patch.
5. Restore from the latest copy of your database. Make sure there aren't any extra "users" with administrator access, and change all administrator passwords
6. Make sure the new database has no triggers/procedures that aren't from you.
7. If this application is a home-grown system then it is time to invest heavily in security training and re-writes, unless you know for sure that your system was not at fault.

Answer 3

One of my servers somehow managed to catch it at one datacentre, but 2 of my other servers running the same software, each hosted with a different provider, did not get the infection - a nasty one at that, causing the CPU (quad core) to max out.

After searching for an answer I came across this board and although I like the NUKE IT OUT OF ORBIT approach, I may have found a solution after trying many approaches for hours.

Here's the synopses:

I figured if I wanted to stop the existing processes from firing up, I need to obviously delete the executables - not exactly. Since we're dealing with an infection, the processes kept coming back after a few milliseconds. After I deleted the executables, they came back - I managed to catch an SSH connection going out to the C&C server, will post it here later once I log it on my experimental machine. Next step I did to prevent it from launching, was to find all executables and instead of deleting them, I set their permission to nobody:nobody and 444 (I wanted to still review the contents you can set it to lower). What this did is occupy the name of the file used by the rogue program so that the SSH connection could not replace the contents of the file, and also prevented it from executing rendering it useless. This seems to work for stopping the processes from automatically starting and allowing Remote Code Execution and effectively quarantining the executables .

Please note this is only a patch solution and not a remedy to fix the problem. I will be reviewing the code commit by commit (in git) and eyeballing log files for a while until I figure out how the payload got in there in the first place.

CentOS7

Steps:

install htop:
#: sudo yum install htop

#: htop

Sort by CPU% and note the name of the process consuming 99% or 100%

find the executable:

#: sudo find / -type f -iname "[FILENAME]"

replace [FILENAME] with the name you grabbed from htop

#: sudo chown nobody:nobody /path/to/[FILENAME]
#: sudo chmod 0444 /path/to/[FILENAME]

Shut down the currently running executables

#: sudo htop

find the process, select and hit the "K" key to kill the process. At this point, it should not come back. If a new process popped back up, repeat the steps above.

Maybe someone will have time to write a sh script to do it automatically.

[UPDATE:] ALso watch out for CRON jobs for user apache - runs every 15 mins to download the payload.