0

I run an airflow instance which python process scheduler, which is used to trigger different python ML process which has no relation to crypto mining. This is the process which is running this dockerized process https://github.com/puckel/docker-airflow I still have no idea if it's a false positive from google's side or if someone actually got access to my instance and crypto-mined.

I received this E-mail from Google

Dear Developer,
We've detected that your Google Cloud Project xxxxx (id: xxxxx) is engaging in cryptocurrency mining, resulting in the suspension of all project resources displaying this behavior.

So I contacted support and got it reinstated and deleted and recreated the instance and it kept happening again and again. One funny thing is, after 2 hours of creating the instance (4 CPU, 10 GB RAM), the CPU usage goes up to 100% and I have no idea how it's happening. This is just a development server so my redis server wasn't password protected. Could it have caused the vulnerability?

To see the exact code that was running on the instance, you can look at another issue I posted: https://github.com/puckel/docker-airflow/issues/507

I got in touch with the trust and safety team and they gave the following information.

The flag that was raised and some discussion details from the GCP trust and safety team:

"The resources associated with your project are being suspended for cryptocurrency mining in violation of our Terms of Service.

abuse_start_timestamp: "2020-02-17 16:17"
abuse_stop_timestamp: "2020-02-17 16:25"
source_ips: "34.87.94.235"
destination_ips: "107.173.160.165"
urls: "107.173.160.165"
total_core_hours: 101.0
vm_resource_id_zone_name: "2080774995510078591:asia-southeast1-b"
vm_hostname_zone_name: "airflow:asia-southeast1-b"
remote_port_list: "5555"

"Is your project really meant to terminate to HIFormance or ColoCrossing in California, USA? That is the destination IP and URL (107.173.160.165) of the project per the log given. This is the result when I did multiple IP lookups."

" Port 5555 is what you would call a well known port. Sadly, this is associated to threats and trojans. More and more, it looks like the instance may have been hijacked. However, rather than play the guessing game, I would like to wait for the information from our Trust & Safety team."

I'm very new to security, so apologies if my questions are too stupid but

  • How can someone gain access to my instance to crypto mine on it?

  • Is CPU usage going 100% a sign of crypto mining?

  • Can an unprotected redis server be responsible for it?

  • I use SSH to log in to the VM, that couldn't be the reason, would it? if it was my other instances would also be affected.

  • Would changing the IAM be helpful?

  • Could malicious python/js libraries be executing in the background crypto mining?

I quite lost on what to do since my instance has been reinstated but I never got an explanation for how it was happening. Any help would be appreciated.

isht3
  • 101
  • 1
  • Unfortunately, since stackoverflow isn't a forum, this just isn't the kind of place where you can get a helpful answer to broad questions like this. In an attempt to be helpful anyway: the number of possibilities for how this happened are large, bordering on infinite. There could be an application vulnerability, a key could have leaked, someone may have broken into an unprotected redis server and from their into other parts of your infrastructure, etc... Only someone with full access to your full systems will really have any hope of answering your questions. – Conor Mancone Mar 13 '20 at 12:27
  • If it helps though what you are really trying to do is "Cyber Incident Analysis" or just "Incident Analysis" (to pick two of many possible names for this). In essence you have had a breach and now are trying to figure out how it happened, when it happened, and how to stop it from happening again. If you can't solve it yourself your only option may be to hire someone to help. Certainly, until you figure out the root cause, putting your resources back on will obviously just result in it getting compromised again. – Conor Mancone Mar 13 '20 at 12:29
  • And yes, 100% CPU usage is absolutely a sign of cryptocurrency mining (although it could be a sign of other things as well). Worth a read: https://security.stackexchange.com/questions/138606/help-my-home-pc-has-been-infected-by-a-virus-what-do-i-do-now – Conor Mancone Mar 13 '20 at 12:30
  • That docker image is [configured](https://github.com/puckel/docker-airflow#ui-links) to use port 5555 for the Flower user interface. Whether or not the docker image has anything malicious in it, or if the software running on it was compromised is anyone's guess. – user Mar 13 '20 at 14:04

0 Answers0