5

I got a mongodb server which from the logfiles got a connection from a remote ip address, though its obviously not allowed (or to be more precise: the mongod is not bound to any public interface) due to the mongodb configuration file as shown below.

The ip address has tried to connect without ssl and access has been denied - but, there are obviously logs missing. So, I assume, they cleaned parts of the mongodb.log, though not all, so for example theres one entry left where the remote ip address is shown (could be one of them they used?).

So, the question is: How can one get with a public ip address to the mongodb server? They seem to got ssh access because of various facts below but even then they would need to use the local interface to connect to mongodb, right?

What I have found out so far:

  • /var/log/auth.log has been removed inside the mongodb docker container (or it didn't exist yet, because I did not directly ssh to the docker containers), but it still exists on the deamon parent machine though it starts at Apr 24 06:25:29 so I assume they removed it.
  • they left a READ_ME (empty) folder & PLEASE_READ_ME folder inside the mongodb root directory, with a file "./PLEASE_READ_ME/collection-0-*****.wt" (masked) starting with the following message: Don't panic. Your DB is in safety and backed up (check logs). To restore send 0.1 BTC and email with your server ip or domain name. Each 48 hours we erase all the data...
  • they did not create additional db users (or at least, additional users do not exist atm)

  • this is the list of the service on the parent machine (netstat)

    tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1281/sshd
    tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1296/exim4
    tcp 0 0 0.0.0.0:54885 0.0.0.0:* LISTEN 707/rpc.statd
    tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 697/rpcbind
    tcp6 0 0 :::22 :::* LISTEN 1281/sshd
    tcp6 0 0 ::1:25 :::* LISTEN 1296/exim4
    tcp6 0 0 :::35619 :::* LISTEN 707/rpc.statd
    tcp6 0 0 :::27017 :::* LISTEN 1172/docker-proxy tcp6 0 0 :::111 :::* LISTEN 697/rpcbind
    udp 0 0 127.0.0.1:883 0.0.0.0:* 707/rpc.statd
    udp 0 0 0.0.0.0:39217 0.0.0.0:* 707/rpc.statd
    udp 0 0 0.0.0.0:872 0.0.0.0:* 697/rpcbind
    udp 0 0 0.0.0.0:111 0.0.0.0:* 697/rpcbind
    udp6 0 0 :::49550 :::* 707/rpc.statd
    udp6 0 0 :::872 :::* 697/rpcbind
    udp6 0 0 :::111 :::* 697/rpcbind

Architecture:

  • Docker deamon running on a virtual machine with debian 8.7
  • Each docker container also runs debian 8.7
  • One docker container for mongodb, one for application (connected via docker network)

Version MongoDB shell version: 3.2.11

Log (database name, principal name & ip's masked):

2017-04-23T12:03:03.511Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:44575 #16 (5 connections now open)
2017-04-23T12:03:03.540Z I ACCESS   [conn16] Successfully authenticated as principal *** on testdatabase
2017-04-23T12:03:15.461Z I NETWORK  [conn16] end connection ***.***.***.***:44575 (4 connections now open)
2017-04-24T05:07:57.711Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:27790 #17 (5 connections now open)
2017-04-24T05:07:57.724Z I NETWORK  [conn17] AssertionException handling request, closing client connection: 17189 The server is configured to only allow SSL connections
2017-04-24T05:45:35.997Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:54235 #18 (5 connections now open)
2017-04-24T05:45:36.027Z I ACCESS   [conn18] Successfully authenticated as principal *** on testdatabase
2017-04-24T05:45:36.053Z I NETWORK  [conn18] end connection ***.***.***.***:54235 (4 connections now open)
2017-04-24T05:45:36.063Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:54236 #19 (5 connections now open)
2017-04-24T05:45:36.086Z I ACCESS   [conn19] Successfully authenticated as principal *** on testdatabase
2017-04-24T05:45:39.843Z I NETWORK  [conn19] end connection ***.***.***.***:54236 (4 connections now open)
2017-04-24T05:45:39.853Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:54237 #20 (5 connections now open)
2017-04-24T05:45:39.874Z I ACCESS   [conn20] Successfully authenticated as principal *** on testdatabase
2017-04-24T05:45:40.071Z I NETWORK  [conn20] end connection ***.***.***.***:54237 (4 connections now open)
2017-04-24T05:45:40.080Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:54238 #21 (5 connections now open)
2017-04-24T05:45:40.101Z I ACCESS   [conn21] Successfully authenticated as principal *** on testdatabase
2017-04-24T05:45:40.108Z I NETWORK  [conn21] end connection ***.***.***.***:54238 (4 connections now open)                                                                                                                                                                
2017-04-24T05:45:40.117Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:54239 #22 (5 connections now open)
2017-04-24T05:45:40.138Z I ACCESS   [conn22] Successfully authenticated as principal *** on testdatabase
2017-04-24T05:45:44.784Z I NETWORK  [conn22] end connection ***.***.***.***:54239 (4 connections now open)
2017-04-24T05:45:44.797Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:54245 #23 (5 connections now open)
2017-04-24T05:45:44.820Z I ACCESS   [conn23] Successfully authenticated as principal *** on testdatabase
2017-04-24T05:45:44.824Z I NETWORK  [conn23] end connection ***.***.***.***:54245 (4 connections now open)
2017-04-24T05:45:44.833Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:54246 #24 (5 connections now open)
2017-04-24T05:45:44.854Z I ACCESS   [conn24] Successfully authenticated as principal *** on testdatabase
2017-04-24T05:45:44.858Z I NETWORK  [conn24] end connection ***.***.***.***:54246 (4 connections now open)
2017-04-24T05:45:44.868Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:54247 #25 (5 connections now open)
2017-04-24T05:45:44.888Z I ACCESS   [conn25] Successfully authenticated as principal *** on testdatabase
2017-04-24T05:45:56.683Z I NETWORK  [conn25] end connection ***.***.***.***:54247 (4 connections now open)
2017-04-24T06:00:01.936Z I NETWORK  [conn3] end connection ***.***.***.***:39610 (3 connections now open)
2017-04-24T06:00:01.937Z I NETWORK  [conn2] end connection ***.***.***.***:39609 (3 connections now open)
2017-04-24T06:00:01.937Z I NETWORK  [conn4] end connection ***.***.***.***:39636 (3 connections now open)
2017-04-24T06:00:01.937Z I NETWORK  [conn1] end connection ***.***.***.***:39548 (3 connections now open)
2017-04-24T06:00:05.915Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:49695 #26 (1 connection now open)
2017-04-24T06:00:05.951Z I ACCESS   [conn26] Successfully authenticated as principal *** on testdatabase
2017-04-24T06:00:09.089Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:49758 #27 (2 connections now open)
2017-04-24T06:00:09.103Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:49760 #28 (3 connections now open)
2017-04-24T06:00:09.125Z I ACCESS   [conn28] Successfully authenticated as principal *** on testdatabase
2017-04-24T06:00:20.916Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:49790 #29 (4 connections now open)
2017-04-24T06:05:32.404Z I NETWORK  [initandlisten] connection accepted from ***.***.***.***:54712 #30 (5 connections now open)
2017-04-24T06:05:32.414Z I ACCESS   [conn30] Unauthorized: not authorized on admin to execute command { getLog: "startupWarnings" }

Config File (database name masked):

storage:
    dbPath: "/***"
    directoryPerDB: true
    journal:
        enabled: true
systemLog:
    destination: file
    path:   "/***/mongodb.log"                                                                                                     
    logAppend: true
    timeStampFormat: iso8601-utc
processManagement:
    fork: true
net:
    bindIp: [127.0.0.1,172.0.0.5]
    port: 27017
    wireObjectCheck : false
    unixDomainSocket: 
        enabled : true
    http:
        enabled: false
    ssl:
        mode: requireSSL
        PEMKeyFile: /certs/server-***.***.***.***.pem
        CAFile: /certs/ca.pem
        #CRLFile: <string>
        allowConnectionsWithoutCertificates: false
        allowInvalidCertificates: false
        allowInvalidHostnames: true
        #disabledProtocols: false
        #FIPSMode: <boolean>
security:
    authorization: enabled

Head of parent docker deamons /var/log/auth.log (ip's masked)

Apr 24 06:25:29 *** sshd[5332]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:25:32 *** sshd[4660]: Received disconnect from ***.***.***.***: 11: disconnected by user
Apr 24 06:25:32 *** sshd[4660]: pam_unix(sshd:session): session closed for user root
Apr 24 06:25:41 *** CRON[5111]: pam_unix(cron:session): session closed for user root
Apr 24 06:25:44 *** sshd[5342]: Invalid user admin from ***.***.***.***
Apr 24 06:25:44 *** sshd[5342]: input_userauth_request: invalid user admin [preauth]
Apr 24 06:25:44 *** sshd[5342]: Disconnecting: Too many authentication failures for invalid user admin from ***.***.***.*** port 41822 ssh2 [preauth]
Apr 24 06:26:19 *** sshd[5393]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:28:26 *** sshd[5423]: Invalid user pat from ***.***.***.***
Apr 24 06:28:26 *** sshd[5423]: input_userauth_request: invalid user pat [preauth]
Apr 24 06:28:26 *** sshd[5423]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:28:28 *** sshd[5421]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:30:46 *** sshd[5450]: Accepted publickey for root from ***.***.***.*** port 47739 ssh2: RSA 74:72:c0:b4:6e:6d:11:3a:2e:c5:4e:be:34:60:e1:99
Apr 24 06:30:46 *** sshd[5450]: pam_unix(sshd:session): session opened for user root by (uid=0)
Apr 24 06:30:49 *** sshd[5499]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:33:16 *** sshd[5842]: Invalid user postgres from ***.***.***.***
Apr 24 06:33:16 *** sshd[5842]: input_userauth_request: invalid user postgres [preauth]
Apr 24 06:33:16 *** sshd[5842]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:36:16 *** sshd[5844]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:38:02 *** sshd[5846]: Invalid user nginx from ***.***.***.***
Apr 24 06:38:02 *** sshd[5846]: input_userauth_request: invalid user nginx [preauth]
Apr 24 06:38:02 *** sshd[5846]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:38:28 *** sshd[3979]: Timeout, client not responding.
Apr 24 06:38:28 *** sshd[3979]: pam_unix(sshd:session): session closed for user root
Apr 24 06:38:32 *** sshd[5450]: Timeout, client not responding.
Apr 24 06:38:32 *** sshd[5450]: pam_unix(sshd:session): session closed for user root
Apr 24 06:39:14 *** sshd[4027]: Timeout, client not responding.
Apr 24 06:39:14 *** sshd[4027]: pam_unix(sshd:session): session closed for user root
Apr 24 06:41:42 *** sshd[5848]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:42:07 *** sshd[5850]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:42:08 *** sshd[5852]: Invalid user  from ***.***.***.***
Apr 24 06:42:08 *** sshd[5852]: input_userauth_request: invalid user  [preauth]
Apr 24 06:42:08 *** sshd[5852]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:42:55 *** sshd[5854]: Invalid user spark from ***.***.***.***
Apr 24 06:42:55 *** sshd[5854]: input_userauth_request: invalid user spark [preauth]
Apr 24 06:42:55 *** sshd[5854]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:47:06 *** sshd[5859]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:47:48 *** sshd[5861]: Invalid user odoo from ***.***.***.***
Apr 24 06:47:48 *** sshd[5861]: input_userauth_request: invalid user odoo [preauth]
Apr 24 06:47:48 *** sshd[5857]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:47:48 *** sshd[5861]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:52:32 *** sshd[5872]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:52:34 *** sshd[5870]: Invalid user steam from ***.***.***.***
Apr 24 06:52:34 *** sshd[5870]: input_userauth_request: invalid user steam [preauth]
Apr 24 06:52:34 *** sshd[5870]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:54:00 *** sshd[5874]: Disconnecting: Too many authentication failures for root from ***.***.***.*** port 46803 ssh2 [preauth]
Apr 24 06:57:22 *** sshd[5877]: Invalid user teste from ***.***.***.***
Apr 24 06:57:22 *** sshd[5877]: input_userauth_request: invalid user teste [preauth]
Apr 24 06:57:22 *** sshd[5877]: Connection closed by ***.***.***.*** [preauth]
Apr 24 06:57:58 *** sshd[5879]: Connection closed by ***.***.***.*** [preauth]
Apr 24 07:02:13 *** sshd[5881]: Invalid user hadoop from ***.***.***.***
Apr 24 07:02:13 *** sshd[5881]: input_userauth_request: invalid user hadoop [preauth]
Apr 24 07:02:13 *** sshd[5881]: Connection closed by ***.***.***.*** [preauth]
Apr 24 07:03:16 *** sshd[5885]: Invalid user admin from ***.***.***.***
Apr 24 07:03:16 *** sshd[5885]: input_userauth_request: invalid user admin [preauth]
Apr 24 07:03:16 *** sshd[5883]: Disconnecting: Too many authentication failures for root from ***.***.***.*** port 4792 ssh2 [preauth]
Apr 24 07:03:17 *** sshd[5885]: Disconnecting: Too many authentication failures for invalid user admin from ***.***.***.*** port 4802 ssh2 [preauth]
Techradar
  • 177
  • 5
  • 1
    What is your intention? Make a security forensic? Or do you want to fix the problem? For forensic purpose, you should create image snapshot and move it to virtual environment for inspection. OTH, you should backup and redeploy everything in a fresh OS with security patches. – mootmoot Apr 25 '17 at 15:10
  • Important for me is fixing the issue in first instance, I mean fixing the security exploit or preventing the attack method they used to get into the system before a redeploy of the system to a fresh os (which I will do definitely, but without solving the issue first I think it doesnt take too long to see such an attack again)... – Techradar Apr 25 '17 at 17:56
  • 2
    It is time consuming to tackle each running services. Please refer to the step to be takne after server compromised : https://security.stackexchange.com/questions/39231/how-do-i-deal-with-a-compromised-server – mootmoot Apr 26 '17 at 07:38
  • The config says that mongodb is listening on two interfaces: both 127.0.0.1 and 172.0.0.5. That second one might be a problem? – Vince Bowdren May 09 '17 at 17:23
  • well the second one is an ip from the virtual private network of docker, so I guess its not, because, how would one with a publich ip address get to the mongo instance via the private network? that doesn't make sense to me.. – Techradar May 09 '17 at 17:34
  • 3
    I'm guessing this is a bit late for an answer now, but looking at your netstat there looks like docker-proxy was bound to the mongodb port, which makes me think it's likely that when the mongo container was started the port was forwarded from the hosts interface to the container's interface, which would explain how your attackers made the connection. – Rory McCune Jun 07 '17 at 15:30
  • I'm curious, did you paid or they erased the data finally? did you disconnected the server to perform a forensic action? what happened? – OscarAkaElvis Jun 21 '17 at 19:21
  • @OscarAkaElvis No I did not. Fortunately it was just an environment with demo data... I shutdown the instance so dunno whether they would have erased another data. Setup fresh instance... forensic still waiting for some time ;-) – Techradar Jun 21 '17 at 19:43
  • There's not really enough info here to figure out that actual point of ingress; however, I do see some things that could be tightened up: specifically, it appears you are not using fail2ban to prevent user / pass brute forcing (but you've masked IPs, so I can't really tell). So, without having direct access to do actual forensics, I'll just point to the normal precautions: install only services you use, minimize attack surface, open only ports you use. Put ssh on non-standard port, use key-authentication only, service users should have no shell, etc... – DrDamnit Aug 04 '17 at 17:03

2 Answers2

2

As lack of information next time plizz include kernel version, docker version and if the client application connects with root user.

Possible exploits that they used:

  • compromised database client the application so check that too for vulnerabilities

  • as docker containers share kernel with the host and other containers check kernel version for possible vulnerabilities

  • exploited ssh. As I don't see any information is there fail2ban or key-based ssh auth or just password or VPN (recommended to change ssh port, public key auth and fail2ban or only allow connection to ssh via private VPN)

The thing you need to understand is that there isn't maybe just one vulnerability that turned into exploit, it can be a chain of vulnerabilities.

Hrvoje Milković
  • 507
  • 3
  • 6
-2

Check how many interfaces are active on the MDB host. it is possible that you are protecting one network and one is vulnerable. a simple ifconfig may give you some idea.

Hugo R
  • 177
  • 2