PHP malware/shell keeps resurrecting

Question

So I've been fighting this problem for months now and decided that it's beyond my limited (if at all) server skills, and that I need help from the pros.

I have a VPS (with root access) which hosts several different PHP websites, some of which are WordPress-based. Some of the sites got infected with the a malware as a result of the MailPoet vulnerability. I cleaned the infected sites, completely removed MailPoet, backdoor accounts, and related stuff, but the malware keeps resurrecting once in a while. Below is what I can describe about it:

• There are two malware signatures (sorry if I'm using the wrong term), both are injected at the very top of PHP pages. Once looks like this <?php $ozufdqjmhx = '7825h!>!%x5c%x7825tdz)%x5c%x7825bbT-%x5c%x782vg}... with the variable $ozufdqjmhx changes from time to time, the other begins with <?php if(!isset($GLOBALS[\'\a\e\0... etc etc
• The malware comes back at random intervals. Sometimes it comes back a day after cleaning, sometimes a week, or several weeks.
• Only previously infected files/directories/websites get infected again. New directories, or old unaffected ones, are always clean. New files in old infected directories though, get infected.
• maldet (using ClamAV I believe) can't detect any malware. PHP Shell Detector can, but it cannot fix due to being a detector only.

Can you guys help, or give a direction I should be heading to? A million thanks in advance!

(Also I'm sorry if this question doesn't fit the site's regulations. When I'm a daily user of StackOverflow, this is my first time on this Security subsite).

EDIT: I really appreciate any recommendation from you guys, but wiping the sever and start from scratch is not an option. If it was, why would I ask this question to begin with, right? :)

EDIT 2: Following @Mints97's answer, I've checked all the open ports -- looks normal:

21/tcp    open  ftp
22/tcp    open  ssh
25/tcp    open  smtp
53/tcp    open  domain
80/tcp    open  http
110/tcp   open  pop3
143/tcp   open  imap
443/tcp   open  https
465/tcp   open  smtps
587/tcp   open  submission
993/tcp   open  imaps
995/tcp   open  pop3s
3000/tcp  open  ppp
3306/tcp  open  mysql
5432/tcp  open  postgresql
8000/tcp  open  http-alt
8080/tcp  open  http-proxy
8082/tcp  open  blackice-alerts
10000/tcp open  snet-sensor-mgmt
20000/tcp open  dnp

EDIT 3: This is for @QuestionOverflow: When searching for the 4 domains you mentioned in your other answer, I came across a script to eliminate the malware here. In the code we can see if (preg_match('/^<\?php \$[a-z]{10} = \'/', $fh_str)) {, which targets EXACTLY the first signature. I would say now it's the same malware, or at least from the same guy via the same vulnerability. Pretty interesting.

EDIT 4: The second malware has already been discussed here if it may help, and yes, apparently both fetch some payload randomly from 4 domains: "33db9538.com", "9507c4e8.com", "e5b57288.com", "54dfa1cb.com". I've added all 4 into my hosts files, pointing to 127.0.0.1. Let's see what's next.

EDIT 5: Several suggest that this question has already been answered here at How do you explain the necessity of “nuke it from orbit” to management and users?. Honestly I fail to see how the other question answers mine. I'm asking how to eliminate a malware, not to explain to my boss why I should reinstall a server.

Answer 1

I would enable auditd to monitor changes to the files you expect to be backdoored. You will be able to determine which account and process that is responsible for doing these changes.

After installing auditd (not installed pr default on all systems), you can start monitoring changes in files. To do this, simply run the command:

auditctl -w /var/www -p wa

This command will log all file changes to the auditd log file. Usually you will find it in /var/log/audit/, but it is system dependent.

If you are concerned that the attacker might notice this (and maybe remove the rule), you can lock it by running the command:

auditctl -e 2

No further changes can be made to the audit rules until the system is rebooted. Before doing this, be sure that the audit rule above do not generate an insane amount of logs.

Lastly, the audit logs are a bit cryptic. Once you have found that the system is backdoored again, you can search for the audit trail, using the command:

ausearch -f /var/www/backdoored_file.php

I hope this will provide you with the clues on what is going on. Good luck!

Additional:

To make the audit rules survive reboot, you have to define them in /etc/audit/audit.rules

Answer 2

I originally posted this as a comment, but I think this could do with a little explanation.

From my experience with website takeover scenarios, when a shell is uploaded to a website, the hacker either manages to exploit a vulnerability in the server, gain root access, backdoor your SSH and compromise all other sites on the server, or he simply doesn't manage this and makes do with a simple shell. I think that, in your case, it is the second scenario, given that you say that only one website on the VPS was infected.

Now for how does the shell "ressurect". If your server wasn't "rooted", that leaves only four other options I can think of:

1. a "bindport" backdoor
2. a "backconnect" backdoor (rare)
3. a custom backdoor or a "downloader" in one of your PHP files
4. a compromised MySQL account with file access and remote login priveliges.

Let's talk about the first two and how to deal with them first. "bindport" and "backconnect" are two small programs, usually Perl scripts, that are traditionally shipped with web shells. They are usually created in (and executed from) the /tmp folder, which is writable to everything. To find them, you can monitor all of your processes for weird scripts or programs and have a good look in the /tmp folder. Also, it is advisable to set up a firewall.

"Bindport" opens a new port for incoming connections and provides Unix shell access to anyone who knocks in (it's usually password-protected). To find it, look for weird open ports (many hackers just have it open port 31337 or something like that).

"Backconnect" does exactly what it is called - it opens a connection to a remote server, also granting shell access. It is used more rarely than "bindport", mainly because most hackers are too lazy to bother with using it. They normally resort to this method only if "bindport" fails for some reason (like firewall settings).

Now, about the custom backdoors and "downloaders". These are seldom used, because the attacker needs to know PHP at least a bit to use it (and most site-jackers today are nothing more than skriptkiddies or badly-made bots). They're mostly stand-alone files like simplistic shells, or a couple of extra lines of code injected into one of your scripts. They either execute commands given to them (PHP code, shell commands) or do a simple file write with data pased to them (which can be another backdoor). You can try looking for files that have PHP code like eval, preg_replace with the e modifier, exec, system, fopen/fwrite and other file functions, etc. However, the best and most certain way to deal with this stuff is simply restoring the entire website from a backup. If you decide to do this, make sure you wipe all other of the site's files from the server beforehand, just in case you missed a stand-alone shell or backdoor.

And the last highly improbable, but still possible case. If you've been running WordPress with the MySQL root user (or just a user with file write rights), or simply with a user that had enough rights to view the password hash of another user with file write rights, and that user with file write rights has the privelige to connect to the database from anywhere, well... you get it. I'd recommend changing all your MySQL passwords.

Now to why only a certain set of directories gets infected. There are two possible cases: either the hacker, having failed to "root" the server, make do with the limited amount of directories available to them (probably all of your website's directories), or they have access to other directories but simply don't use them.

If it is the latter case, I'd wager that either you're dealing with an idiot, or someone extremely lazy. Or with a bot. Yes, this may disappoint you, but I doubt that your site is really important enough for an experienced hacker: it might be used to grab a few clicks from your users, to host malware or simply churn out a metric ton of spam a couple of times. You're either dealing with a scriptkiddie, or an inexperienced fool trying to scrape a couple of cents from click-trading and spam, or with a bot. However, I think it is more likely a living person: that'd explain the irregularity of the shell's "ressurections".

Answer 3

I've had a very similar thing happen to a site I manage. After much frustration of deleting the malicious code and then it appearing about 2 weeks later, I discovered this:

I took note of the date stamp of when all the files got modified, then I looked up the access log for that minute. I saw a certain page was requested that seemed suspicious, since it was a 404.php of a wordpress theme that wasn't active. I checked that page and saw one line which was basically eval($_POST['php']) encoded in base64.

So I didn't delete the code, instead I changed it save a log of anything that gets sent to post of that page. Sure enough 2 weeks later, my site remained safe, but a log file recorded some interesting code sent to it.

Answer 4

It sounds like the attackers have installed a rootkit on your server. A rootkit can provide a backdoor even if everything looks clean.

The best approach now in my opinion would be to wipe the server and reinstall from scratch. Patch the websites to eliminate the vulnerability. If you need to restore from backup (you have backups, right? :) ) make sure it's a clean one.

Set up a script to look for the traces of infection (for instance the ID 10001 user) in case there are some other vulnerabilities around in addition to the MailPoet problem.

Answer 5

This might seem like a simple question, but have you checked for anything suspicious in you /tmp directory? While different variants of malware are all over the place, many commonly take advantage of system access bugs—such as the bash shellshock bug—to plant executable code right in the /tmp directory.

If you are unsure what should/shouldn’t be in /tmp/ there is an easy—but extreme—thing you can do to clear out the bad stuff. Just run this online in the command line:

rm -rf /tmp && mkdir /tmp && chown root:root /tmp && chmod 1777 /tmp

Or run each command individually like this:

sudo rm -rf /tmp 
sudo mkdir /tmp
sudo chown root:root /tmp
sudo chmod 1777 /tmp

Then reboot the server to see if that clears things up. If it does, congrats! But you are not out of the woods yet since it whatever caused the original system infection might still penetrate your system, it’s only a matter of time before they reinfect you again. Meaning, hopefully this will clean up the mess caused by a weakness in your system you hopefully have plugged up already. But you need to to be sure to find out what that weak-point might have been and be sure to harden it.

I would also recommend checking the crontab entries for the root user via a simple crontab -l like this:

sudo crontab -l

And perhaps even running the bash script taken from this answer on Stack Overflow which will give you a nice, overview report of all crontabs installed on the system:

#!/bin/bash

# System-wide crontab file and cron job directory. Change these for your system.
CRONTAB='/etc/crontab'
CRONDIR='/etc/cron.d'

# Single tab character. Annoyingly necessary.
tab=$(echo -en "\t")

# Given a stream of crontab lines, exclude non-cron job lines, replace
# whitespace characters with a single space, and remove any spaces from the
# beginning of each line.
function clean_cron_lines() {
    while read line ; do
        echo "${line}" |
            egrep --invert-match '^($|\s*#|\s*[[:alnum:]_]+=)' |
            sed --regexp-extended "s/\s+/ /g" |
            sed --regexp-extended "s/^ //"
    done;
}

# Given a stream of cleaned crontab lines, echo any that don't include the
# run-parts command, and for those that do, show each job file in the run-parts
# directory as if it were scheduled explicitly.
function lookup_run_parts() {
    while read line ; do
        match=$(echo "${line}" | egrep -o 'run-parts (-{1,2}\S+ )*\S+')

        if [[ -z "${match}" ]] ; then
            echo "${line}"
        else
            cron_fields=$(echo "${line}" | cut -f1-6 -d' ')
            cron_job_dir=$(echo  "${match}" | awk '{print $NF}')

            if [[ -d "${cron_job_dir}" ]] ; then
                for cron_job_file in "${cron_job_dir}"/* ; do  # */ <not a comment>
                    [[ -f "${cron_job_file}" ]] && echo "${cron_fields} ${cron_job_file}"
                done
            fi
        fi
    done;
}

# Temporary file for crontab lines.
temp=$(mktemp) || exit 1

# Add all of the jobs from the system-wide crontab file.
cat "${CRONTAB}" | clean_cron_lines | lookup_run_parts >"${temp}" 

# Add all of the jobs from the system-wide cron directory.
cat "${CRONDIR}"/* | clean_cron_lines >>"${temp}"  # */ <not a comment>

# Add each user's crontab (if it exists). Insert the user's name between the
# five time fields and the command.
while read user ; do
    crontab -l -u "${user}" 2>/dev/null |
        clean_cron_lines |
        sed --regexp-extended "s/^((\S+ +){5})(.+)$/\1${user} \3/" >>"${temp}"
done < <(cut --fields=1 --delimiter=: /etc/passwd)

# Output the collected crontab lines. Replace the single spaces between the
# fields with tab characters, sort the lines by hour and minute, insert the
# header line, and format the results as a table.
cat "${temp}" |
    sed --regexp-extended "s/^(\S+) +(\S+) +(\S+) +(\S+) +(\S+) +(\S+) +(.*)$/\1\t\2\t\3\t\4\t\5\t\6\t\7/" |
    sort --numeric-sort --field-separator="${tab}" --key=2,1 |
    sed "1i\mi\th\td\tm\tw\tuser\tcommand" |
    column -s"${tab}" -t

rm --force "${temp}"

As complex as malware infections might seem, once you know where the common choke-points are, you can focus your efforts to purge your system of that junk once and for all.

Answer 6

If you don't know how you got infected in the first place, then my guess would be the resurrection is coming from you getting hit by the same vulnerability scanner that hit you the first time round - if this is the case, following @scrollup's advice to compare file timestamps to web traffic should find you the vulnerable script, as well as giving you some good IPs to block from your site.

In my experience, at least when targeting Apache, page-modifying scripts will target a very specific subset of the folders on your machine: those that they can find in your /etc/httpd/conf/httpd.conf folder. I had some other web folders defined in included files, but the script that searched for the folders to inject was clearly only interested in grepping the folders from the one file.

So, see if there's anything significantly different in your Apache config between those sites that were impacted, and those that were not. It may give you a good approach to protect most of your sites from it in future, while leaving one "canary" or "honeypot" site that you can monitor for recurrences.

Answer 7

There isn't enough information in your post to determine thie cause of this. You'll have to do some investigating. The first thing to check is the creation/modification time of the files and then compare that to your access and ftp logs around the same time in order to determine how the files were modified. This will hopefully lead you to some /plugin/.shell.php file that is the left over backdoor from the initial infection. If your logs don't appear to contain anything useful it is possible that a backdoor account has been created in your WP install and what you are seeing is the result of "normal operations" by a malicious user.

Answer 8

Why not set a watch on the creation of the new user? And once that happens, you send an offsite mail to inform you of the creation.