3

This question had already been brought up here, but was poorly answered with generic answers, like reinstall everything and no actual solution and cause.

So here's my story: There is a new virus out there which creates a process called "WATCHBOG" which eats all CPU from the server. After further investigation, I found out that this virus is mining cryptocurrency.

Full story:

Server specs: OS: Ubuntu server 16.04.5 LTS Software: Apache2 with virtualhosts, MYSQL, PHP5.6, PHP7

When I found out that this web-server is infected, web pages still worked but awfully slow. The first thing I see is the watchbog process which eats all CPU, so I tried to kill it, but it reappears instantly, and everything is so slow that it is almost impossible to operate the server. Next what I thought- "ok I need to terminate this process" So I thought I will put "* * * * * killall watchbog" in crontab , so I opened crontab, and found out that its compromised as well. I removed this new entry from crontab, i deleted the watchbog file, and after a minute or two everything magically reappeared. Crontab got compromised again to run some kind of remote script and the watchbog process was up and eating cpu again.

I tried to find anything useful on the web, and found these articles:

Here's is the most helpful one: https://sudhakarbellamkonda.blogspot...50061219193777

After following this article the watchbog virus still reappeared so I came up with this solution: Open a screen session as root and then run this loop: ( while true ; do killall watchbog ; done ) and leave it running in background by detaching screen session with CTRL+A+D. I posted as well this solution in that blog. and here is one other post, but nothing really is helpful there
https://unix.stackexchange.com/questions/487437/a-strange-process-called-watchbog-is-hogging-my-entire-cpu-and-i-cant-get-rid

So I tried fighting this virus many ways, changed passwords, reinstalled SSH, etc, with no luck. Meanwhile I created a new Ubuntu server with the latest 18.04.1 LTS. We installed all latest webserver stuff, enabled UFW, opened web and ftp ports, then migrated WWW data and SQL, changed IP back to original servers IP, and..... there it is AGAIN!

The virus came back on the new machine

I think probably this virus infects the system by using some vulnerability in web-server software, So we found out that WGET and CURL is responsible for distributing the virus around the system, now we are trying to understand how it got there.

If you have any tips, please help me to find the vulnerability.

P.S. This is my first question, judge me softly please :)

IconDaemon
  • 109
  • 4
r. tihovs
  • 63
  • 1
  • 5
  • 3
    @r.tihovs If you don't want this closed as a duplicate I would recommend focusing much more on "how do I prevent this from infecting my fresh install" and less on "how do I remove it". The only real answer to "how to remove it" is to reinstall everything, otherwise as many have noted, you could very easily leave backdoors lurking unnoticed. – AndrolGenhald Jan 11 '19 at 14:09
  • 1
    I think you must review the security of your website too. If you have an outdated Wordpress and plugins, you can be infected again. – ThoriumBR Jan 11 '19 at 14:22
  • To be precise, the watchbog cryptominer is not a virus, it's a payload. It could've been delivered into the system in any number of ways, including you running it yourself. If you simply reinstall and setup everything the way you had before the infection, you'll likely be setting up a machine with the same vulnerability, and if the attacker delivered their payload with a bot, it may try to hit your machine again when it detected that your machine failed to report to the bot's command and control. You need to remove the vulnerability before reintroducing the new machine back to the network. – Lie Ryan Jan 12 '19 at 05:02

3 Answers3

6

Did a lot of digging on this. It's a pretty new script that runs some older code. Shout out to twitter, google translate, notpad, and general boredom.

Watchbog is a miner script that downloads several utilities to compromise a system in order to perform cryptanalysis. As part of the package it has several features.

  1. It determines the architecture of the OS being used
  2. It determines the kernel version for exploit
  3. It downloads the appropriate tools to a location within /tmp for exploit.

Once the script is up, it looks for several commands from Pastebin such as to update, run scripts, or other. It's my assumption that the attacker is still using pastebin and there do not appear to be any updates for this yet, but the script is smart enough to look for that.

After checking for updates, the script backsup cron and completely re-writes it with SEVERAL references to the script to ensure that it is getting updated and run inspite of any user attempts. It's pretty exhausting reading through this part of the code, but it is definitely doable.

Once the CRON tab is running the way it should, the account can look at it's credentials. If needed, it leverages cve-2017-16995. More information can be found here:

https://www.exploit-db.com/exploits/45010

This allows a standard user to attain root and begin executing scripts as needed. It runs a few basic tests to make sure all the components are working (Curl, wget, python, OS calls) and then launches the full application to begin the mining.

Looking online, it appears that the file can be easily used to impact a site which allows XSS attacks. In addition to locking down the allowed sites for script building, there are a couple of other targetted areas that you may want to review for patching.

  1. They use the mail spooler for the root account. There is an email that has a shell script embedded in it. The one that launches Watchbog.

  2. The above comment mentioned SOLR. The source I found from China mentioned that there was a potential XSS opportunity after the use of CKeditor, a popular rich text editor for websites. Definitely something to keep in mind.

Take aways:

PATCH everything. The vulnerability used impacts the following kernels:

'4.4.0-31-generic',
'4.4.0-62-generic',
'4.4.0-81-generic',
'4.4.0-116-generic',
'4.8.0-58-generic',
'4.10.0.42-generic',
'4.13.0-21-generic',
'4.9.0-3-amd64',
'4.9.0-deepin13-amd64',
'4.8.0-52-generic',
'4.8.6-300.fc25.x86_64',
'4.11.8-300.fc26.x86_64',
'4.13.9-300.fc27.x86_64',
'4.5.2-aufs-r',
'4.4.0-89-generic',
'4.8.0-58-generic',
'4.13.0-16-generic',
'4.9.35-desktop-1.mga6',
'4.4.28-2-MANJARO',
'4.12.7-11.current',
'4.4.0-89-generic',
'4.8.0-45-generic',
'4.10.0-28-generic',
'4.10.0-19-generic',
'4.8.0-39-generic'

Don't let root get email.

Keep yourself protected from XSS.

Connor Peoples
  • 1,431
  • 6
  • 12
2

I'm probably one of those answer you judged as a "poor generic answer", but I'll said it again.

For now, little is known about said virus, and it could have much more that the process runing right now, and could be a critical breach to sensitive data throught, for instance, a backdoor. Therefore, the only SAFE thing to do is redo a fresh install after handpicking what file you want to save and, as AndrolGenhald said, concentrate on how to prevent a new infection on the fresh system

Thryn
  • 301
  • 1
  • 7
0

Finally we found the vulnerability!

The cause was SOLR search engine module ( http://lucene.apache.org/solr/), it was outdated and had root permissions, after removing it, the virus is not reappearing any more. So now the plan is to start from scratch and hope its over...

r. tihovs
  • 63
  • 1
  • 5
  • Can you share how you determined this dependency to be the root cause? – Connor Peoples Jan 12 '19 at 03:52
  • We found the root of this problem in this file, and by checking what had root permisions. File location: https://mywebsite.com/solr/server/solr/drdrdr/conf/configoverlay.json. heres the code: https://pastebin.com/raw/Y3Fz8ULL – r. tihovs Jan 14 '19 at 15:30
  • For the record, I am infected by the wathbog malware, but I'm running nginx, not apache. So this is not applicable in my case, and I guess for a number of other people that fall in the same case. – Giorgos Sfikas Nov 21 '19 at 14:11