68

When checking the auth log of a server with the command:

grep sshd.\*Failed /var/log/auth.log | less

I see thousands of lines like this:

    Jan 12 11:27:10 ubuntu-leno1 sshd[8423]: Failed password for invalid user admins from 172.25.1.1 port 44216 ssh2
    Jan 12 11:27:13 ubuntu-leno1 sshd[8425]: Failed password for invalid user phoenix from 172.25.1.1 port 20532 ssh2
    Jan 12 11:27:17 ubuntu-leno1 sshd[8428]: Failed password for invalid user piglet from 172.25.1.1 port 24492 ssh2
    Jan 12 11:27:22 ubuntu-leno1 sshd[8430]: Failed password for invalid user rainbow from 172.25.1.1 port 46591 ssh2
    Jan 12 11:27:25 ubuntu-leno1 sshd[8432]: Failed password for invalid user runner from 172.25.1.1 port 57129 ssh2
    Jan 12 11:27:34 ubuntu-leno1 sshd[8434]: Failed password for invalid user sam from 172.25.1.1 port 11960 ssh2
    Jan 12 11:27:37 ubuntu-leno1 sshd[8437]: Failed password for invalid user abc123 from 172.25.1.1 port 5921 ssh2
    Jan 12 11:27:40 ubuntu-leno1 sshd[8439]: Failed password for invalid user passwd from 172.25.1.1 port 21208 ssh2
    Jan 12 11:27:43 ubuntu-leno1 sshd[8441]: Failed password for invalid user newpass from 172.25.1.1 port 65416 ssh2
    Jan 12 11:27:46 ubuntu-leno1 sshd[8445]: Failed password for invalid user newpass from 172.25.1.1 port 26332 ssh2
    Jan 12 11:27:49 ubuntu-leno1 sshd[8447]: Failed password for invalid user notused from 172.25.1.1 port 51126 ssh2
    Jan 12 11:27:52 ubuntu-leno1 sshd[8449]: Failed password for invalid user Hockey from 172.25.1.1 port 14949 ssh2
    Jan 12 11:27:56 ubuntu-leno1 sshd[8451]: Failed password for invalid user internet from 172.25.1.1 port 35105 ssh2
    Jan 12 11:27:59 ubuntu-leno1 sshd[8453]: Failed password for invalid user asshole from 172.25.1.1 port 7916 ssh2
    Jan 12 11:28:02 ubuntu-leno1 sshd[8456]: Failed password for invalid user Maddock from 172.25.1.1 port 26431 ssh2
    Jan 12 11:28:05 ubuntu-leno1 sshd[8458]: Failed password for invalid user Maddock from 172.25.1.1 port 53406 ssh2
    Jan 12 11:28:09 ubuntu-leno1 sshd[8460]: Failed password for invalid user computer from 172.25.1.1 port 23350 ssh2
    Jan 12 11:28:15 ubuntu-leno1 sshd[8462]: Failed password for invalid user Mickey from 172.25.1.1 port 37232 ssh2
    Jan 12 11:28:19 ubuntu-leno1 sshd[8465]: Failed password for invalid user qwerty from 172.25.1.1 port 16474 ssh2
    Jan 12 11:28:22 ubuntu-leno1 sshd[8467]: Failed password for invalid user fiction from 172.25.1.1 port 29600 ssh2
    Jan 12 11:28:26 ubuntu-leno1 sshd[8469]: Failed password for invalid user orange from 172.25.1.1 port 44845 ssh2
    Jan 12 11:28:30 ubuntu-leno1 sshd[8471]: Failed password for invalid user tigger from 172.25.1.1 port 12038 ssh2
    Jan 12 11:28:33 ubuntu-leno1 sshd[8474]: Failed password for invalid user wheeling from 172.25.1.1 port 49099 ssh2
    Jan 12 11:28:36 ubuntu-leno1 sshd[8476]: Failed password for invalid user mustang from 172.25.1.1 port 29364 ssh2
    Jan 12 11:28:39 ubuntu-leno1 sshd[8478]: Failed password for invalid user admin from 172.25.1.1 port 23734 ssh2
    Jan 12 11:28:42 ubuntu-leno1 sshd[8480]: Failed password for invalid user jennifer from 172.25.1.1 port 15409 ssh2
    Jan 12 11:28:46 ubuntu-leno1 sshd[8483]: Failed password for invalid user admin from 172.25.1.1 port 40680 ssh2
    Jan 12 11:28:48 ubuntu-leno1 sshd[8485]: Failed password for invalid user money from 172.25.1.1 port 27060 ssh2
    Jan 12 11:28:52 ubuntu-leno1 sshd[8487]: Failed password for invalid user Justin from 172.25.1.1 port 17696 ssh2
    Jan 12 11:28:55 ubuntu-leno1 sshd[8489]: Failed password for invalid user admin from 172.25.1.1 port 50546 ssh2
    Jan 12 11:28:58 ubuntu-leno1 sshd[8491]: Failed password for root from 172.25.1.1 port 43559 ssh2
    Jan 12 11:29:01 ubuntu-leno1 sshd[8494]: Failed password for invalid user admin from 172.25.1.1 port 11206 ssh2
    Jan 12 11:29:04 ubuntu-leno1 sshd[8496]: Failed password for invalid user chris from 172.25.1.1 port 63459 ssh2
    Jan 12 11:29:08 ubuntu-leno1 sshd[8498]: Failed password for invalid user david from 172.25.1.1 port 52512 ssh2
    Jan 12 11:29:11 ubuntu-leno1 sshd[8500]: Failed password for invalid user foobar from 172.25.1.1 port 35772 ssh2
    Jan 12 11:29:14 ubuntu-leno1 sshd[8502]: Failed password for invalid user buster from 172.25.1.1 port 18745 ssh2
    Jan 12 11:29:17 ubuntu-leno1 sshd[8505]: Failed password for invalid user harley from 172.25.1.1 port 38893 ssh2
    Jan 12 11:29:20 ubuntu-leno1 sshd[8507]: Failed password for invalid user jordan from 172.25.1.1 port 64367 ssh2
    Jan 12 11:29:24 ubuntu-leno1 sshd[8509]: Failed password for invalid user stupid from 172.25.1.1 port 27740 ssh2
    Jan 12 11:29:27 ubuntu-leno1 sshd[8511]: Failed password for invalid user apple from 172.25.1.1 port 22873 ssh2
    Jan 12 11:29:30 ubuntu-leno1 sshd[8514]: Failed password for invalid user fred from 172.25.1.1 port 54420 ssh2
    Jan 12 11:29:33 ubuntu-leno1 sshd[8516]: Failed password for invalid user admin from 172.25.1.1 port 58507 ssh2
    Jan 12 11:29:42 ubuntu-leno1 sshd[8518]: Failed password for invalid user summer from 172.25.1.1 port 48271 ssh2
    Jan 12 11:29:45 ubuntu-leno1 sshd[8520]: Failed password for invalid user sunshine from 172.25.1.1 port 5645 ssh2
    Jan 12 11:29:53 ubuntu-leno1 sshd[8523]: Failed password for invalid user andrew from 172.25.1.1 port 44522 ssh2

It seems that I'm experiencing an ssh brute force attack. Is this a common occurrence, or am I being specifically targeted? What should I do now? Should I consider the attack successful and take measures?

-----Edit-------

The fact that the attack comes from an internal IP address is explained by this server having a ssh redirection from outside. It happened really quickly after opening the port, are every public IP scanned in the wild in search of an existing server behind ?

syldor
  • 771
  • 1
  • 5
  • 8
  • 32
    172.25.1.1 is I believe in the Private IP range (e.a. not rout able over the internet) so most likely this is someone near you, or a machine near you. – LvB Jan 15 '16 at 10:17
  • Comments are not for extended discussion; this conversation has been [moved to chat](http://chat.stackexchange.com/rooms/34508/discussion-on-question-by-syldor-am-i-experiencing-a-brute-force-attack). – Rory Alsop Jan 18 '16 at 23:09

8 Answers8

63

Yes it looks like you are experiencing a brute force attack. The attacker is in on a class B private address, so it is likely to be someone with access to your organization's network that is conducting the attack. From the usernames it looks like they are running though a dictionary of common usernames.

Have a look at 'How to stop/prevent SSH bruteforce' (Serverfault) and 'Preventing brute force SSH attacks' (Rimu Hosting) on how to take measures to mitigate some of the risk relating to SSH bruteforce attacks.

TheJulyPlot
  • 7,729
  • 6
  • 30
  • 44
  • 50
    Reminds me of [this](https://xkcd.com/742/) :) – Matt Lyons-Wood Jan 15 '16 at 10:50
  • 5
    meanwhile, to the OP, update your passwords. Disable PermitRootLogin if you don't need it. Better yet, disable password-based logins (ie, only RSA key logins). – Otheus Jan 15 '16 at 14:38
  • When independent we used to map the location of attacks at company days @matt. 98% of all attacks came from Paris, the location of our second office... – Ben Jan 15 '16 at 18:35
  • 1
    How likely is it that the network has been compromised (e.g., virus on a user's machine) rather than an actual person within the organization? – jpmc26 Jan 15 '16 at 19:39
  • 1
    @jpmc26 It is *possible* another machine has been compromised and the attacker may be attempting to pivot through the network. A virus *could* have been at play in any possible compromise. This is only conjecture though and remains one of many potential explanations. – TheJulyPlot Jan 16 '16 at 07:40
  • 2
    Many information to go through ! It will be a funny monday far for my usual programming. – syldor Jan 16 '16 at 13:54
  • @syldor make sure to show them the xkcd comic and see if anyone face grimaces. – PyRulez Jan 16 '16 at 17:38
  • The attacker is not in a class B private address. The port is redirected by the firewall and as such the logs have the internal firewall address. Not the ideal configuration. – Rui F Ribeiro Feb 23 '18 at 04:38
41

Yes, this looks exactly like a brute-force attack and after googling admins phoenix piglet rainbow it looks like this is the wordlist the attacker is using: https://github.com/hydrogen18/kojoney/blob/master/fake_users
Check out line 116 onwards. The wordlist is being used in the exact same order.
This appears to be a generic wordlist as it also present on other sites. e.g. http://src.gnu-darwin.org/ports/net/kojoney/work/kojoney/fake_users

@TheJulyPlot has provided some good information on what to do to mitigate this attack.

30

Quick note added about fail2ban, as a lot of people have been mentioning it: The front-end is a corporate firewall, the back-end only sees the redirection/proxy/internal address that comes from the firewall.

So no, 172.25.1.1 is not an internal machine compromised (and both the comment in the answer, and other answer here stating it is an internal machine are wrong when commenting that). It is one of the internal IP addresses of the firewall.

Fail2ban in the back-end would only block all possibility to use SSH for stretches at a time as it only sees failed attempts from 172.25.1.1. So please, read on for my answer.

There are no doubts, as other posts mention, it is painfully obvious you are under a brute force attack.

However, that does not mean at all you are compromised in any way from the logs you have shown us. Alas, these days ssh brute force attacks are all too common. Most of the time they are really automated, and you are not necessarily being targeted.

As an anecdotal warning tale, some years ago the first day I set up new servers in an ISP provider, the ssh one open to the Internet got 200k+ ssh scanning probes on a single night.

As for the "internal IP address," either you are using SNAT or a 22/TCP proxy redirect at best, and the source Internet IPs do not show (which is not the best practice), or at worst your router/cable modem is compromised.

If you do really are having a SNAT/proxy SSH configuration, I do recommend you to think it over. You want logs of the real IP addresses, and not of your network.

As for measures, I do recommend some:

  1. Do not allow passwords in SSH; only logins using RSA certificates;
  2. Do not open ssh for the outside; restrict it to your internal network;
  3. For accessing from the outside, access via a VPN; do not expose SSH to the Internet at large;
  4. rate-limiting SYNs from the outside.

If you absolutely insist in still having the SSH Internet open to the outside, be aware that changing the default SSH port gives only a fake sense of security, and blocking temporarily IP addresses only slows the attack down as often we are talking about coordinated farms of zombie machines.

You may want to have a look at fail2ban if you do change your configuration to receive directly public IP addresses connecting to you; however take into account that if indeed any external IP arrives with the IP address of your gateway, you are effectively locking out all external SSH access using it.

There are also other caveats; please do note that nowadays zombies/malware take fail2ban into account, and will return after the default timeout period or take turns with different IP addresses. (I have seen it happen.) Even if you use mandatory RSA certificate authentication, the password attacks will still be logged and will burn I/O, disk space, and CPU cycles.

As for the VPN, you do not need dedicated hardware; it is trivial to configure a VPN in a Linux or FreeBSD server.

If you do not feel comfortable setting one up from scratch, I do recommend a VM with pfSense. for FreeBSD; strongSwan for setting up a VPN in a Linux server.

If your front-end server is Linux, another alternative is port knocking. However due to its inherent working, I only recommend it for domestic settings. As correctly pointed out by @GroundZero, "port knocking is security by obscurity and an attacker can actively monitor your network traffic to discover the knocking sequence."

How To Use Port Knocking to Hide your SSH Daemon from Attackers on Ubuntu

As an additional measure, you can also rate-limit with firewall/iptables rules SYNs in port 22. In this way, you won't rate limit your legitimate connections, and either iptables in linux, or most commercial firewalls allow that configuration. I have seen nasty tricks as bots attacking some daemons as fast as possible before security rules kick-in. However I do believe actually ssh has built-in defenses against that.

To answer you about the rampant scanning, yes indeed. You have a lot of bad actors, zombie networks and malware constantly scanning the IP address space to find servers with compromised versions of sshd, vulnerable/old sshd versions, servers with default/bad passwords/known backdoors, and simply servers with openssh to gain a foothold in a non-privileged user either via brute force or dual pronged attacks via phishing.

After three successful attacks via phishing (in separate events) in my bastion host in my current work, I decided to do a dual ssh configuration in that all users are asked for a RSA certificate in sshd_config, and only the internal network allows password authentication, adding to the end of the configuration file as such:

# sshd configuration allowing only RSA certificates
Match Address 10.0.0.0/8,172.16.0.0/12,192.168.0.0/24
PasswordAuthentication yes

As another anecdotal scary tale, before I changed to mandatory RSA certificates, one of the phishing compromises was done in the exact week there were two kernel updates for vulnerabilities that allowed privilege escalation to root, and had I not updated on the spot, I would have been root compromised. (and if my memory does not fail me, it was around the 4th of July... hackers love to save those nasty attacks for holiday times)

As for graphs of actual live attacks real-time, have a look at:

For finishing the answer:

No, you probably are not compromised in any way.

Yes, you have to take measures to increase your security level. I would advise a VPN tunnel/client from the outside to your corporate VPN server. It is what I am doing actually.

I have also a last and important piece of advice: to make sure a regular account isn't compromised, check your /var/log/auth.log authentication logs for a successful authentication. There are ways of logging into openssh with any account without it being registered in /var/log/wtmp and consequently not appearing in the last command.

It goes without saying that if a regular account is compromised in an old machine without updates, all bets are off. And in the unfortunate case of privilege escalation to root, even the logs can be compromised.

Rui F Ribeiro
  • 1,736
  • 9
  • 15
  • 1
    Great answer, I do have an ssh redirection and your anecdote shows that this kind of things happen extremely quickly after opening to the outside. – syldor Jan 16 '16 at 13:42
  • I am sorry for the English glitches as it is not my mother tongue. I have forbidden all the Unix operators to use passwords, and the server used by professors to log from the outside has an SSH configuration as such that while it allows login+password from the internal network, it forces RSA certificates from the Internet. You can only reach my internal network at home via VPN+dynamic DNS, and only after being authenticated by the VPN, you have SSH+HTTP+DNS+voIP – Rui F Ribeiro Jan 16 '16 at 15:06
  • (and I have configured my VPN to be used natively by OS/X and iPhone. Very convenient) – Rui F Ribeiro Jan 16 '16 at 16:21
  • syldor, are you being a cable modem, a linux server, or behind a corporate firewall (as 172.x.x.x are not that common in domestic settings) – Rui F Ribeiro Jan 17 '16 at 10:31
  • At work today, and I added a few bits very *pertinent* to your answer. – Rui F Ribeiro Jan 18 '16 at 08:53
  • 1
    By authentification logs, do you mean /var/log/auth.log ? That's where i found the logs, i didnt know about wtmp :) – syldor Jan 18 '16 at 09:02
  • 1
    And It's a Linux server behind a corporate firewall yes. – syldor Jan 18 '16 at 09:03
  • wtmp is a binary file where the `last` command drinks - it is rotated by default every month; and yes, auth.log. Only if you find a successful authentication there, it means you were compromised. – Rui F Ribeiro Jan 18 '16 at 09:04
  • (I also updated the answer with firewall rules even before you confirming it is indeed a corporate firewall). Have a nice week. – Rui F Ribeiro Jan 18 '16 at 09:14
  • Coming back...about the firewall being the ssh conduit...I always oppose vehemently a firewall having any kind of service active/open. It is a vector of attack. A firewall should only forward packets and not even answer to PINGs. Thanks for selecting the answer btw. – Rui F Ribeiro Jan 18 '16 at 09:33
  • 2
    Note that port knocking is also a form of obfuscation and does not truly add security. However, it will stop most automated bots and will (in most cases) require an attacker to actively monitor your network traffic to discover the knocking sequence – BlueCacti Jan 18 '16 at 17:10
  • 1
    I do agree with you @GroundZero. It can be useful in domestic settings, however I do use VPNs both at work and at home. – Rui F Ribeiro Jan 18 '16 at 17:35
  • 1
    @GroundZero, I expanded the post with your suggestion for the sake of completeness. The OP has already stated he has a corporate firewall. – Rui F Ribeiro Jan 18 '16 at 17:43
  • 1
    I use RSA, and I usually pull the log from my servers and ban the brute forcers by iptables. – ave Jan 19 '16 at 07:12
  • In this particular case, it would involve some convoluted setup, unless we are talking about the actual syslog server for the corporate firewall. And developing a plug-in for fail2ban. At home I have a VPN with strongswan, at work going through a dedicated VPN server to the IT team ; cannot be bothered with ssh brute-force logs eating resources, and leaving another port (SSH) open to vulnerability attacks. You need ssh, you are out, you are coming via VPN. I have a central syslog server here, no log pulling. I have VPN+ssh clients both in my notebook and my iPhone, and they work pretty well. – Rui F Ribeiro Jan 19 '16 at 08:16
28

Yes, you're being bruteforced. But I don't think you should worry about any bruteforce you detect coming from the internet. You should, however, be worried about brute-force attacks coming from your own network.

Being bruteforced is very common, and as long you don't use passwords for SSH (or use good passwords), the attack won't be successful at all (especially with one guess every 3-4 seconds).

Though, the IP (172.25.1.1) is on the same network than you. This is the real problem, and you should check whether this machine is compromised as soon as possible.

Mark Buffalo
  • 22,508
  • 8
  • 74
  • 91
Benoit Esnard
  • 13,979
  • 7
  • 65
  • 65
  • _"I don't think you should worry about any bruteforce you detect coming from the internet"_ I do. Install basic security software like _fail2ban_ to prevent these attacks (a) stumbling on a correct password, (b) wasting your bandwidth, and (c) filling up your logs. – Lightness Races in Orbit Jan 17 '16 at 16:50
  • 1
    What Benoit meant is that the OP shouldn't be worried about this particular attack - as it is very common when carried out from the Internet - but instead about the fact that the attacker has escalated privileges and is now attacking from the internal network, having therefore access to the OP's organization's resources and servers. – dr_ Jan 17 '16 at 18:47
  • I am facing brute force from different external IPs but in addition to that; the thing is that i am facing `brute-force attacks coming from your own network` on my Web Application URLS as well and its consuming all my API Quota. I am getting IP address of my own server on req.headers['user-agent']. Please help me in this regards. Thanking You! – Dynamic Remo Dec 02 '18 at 17:47
8

You are surviving the attack, from appearances. SSH is doing what it's supposed to do. However, see below for some steps you should take ASAP to ensure continued survival.

Also, and unfortunately, a system inside the network has been compromised. Too common these days. You should ascertain the nature of this apparently interior attack, but do not assume that this system is compromised simply on the basis of this logfile. Anyone who runs a web-facing SSHD has seen this before, over and over, for years. Fix up the SSHD installation on this host, then investigate the system at 172.25.1.1.

Follow SSHD best practices such as these ...

Disable password authentication in favor of key-based authentication:

# grep PasswordAuth sshd_config | grep no
PasswordAuthentication no

As mentioned, disable root logins:

PermitRootLogin no

Add allowed users to sshd_config:

AllowUsers myusername the_other_sysop_guy

If you do the last one, the log message will change from 'invalid user' to "illegal user".

Depending on your business situation, you can also think about firewalling the SSHD port (allow SSH only from authorized IP addresses) or changing it to using a different port (which is really security by obscurity, but most attacks of this nature are actually scripted). Alas, the fact that the attacker appears to be inside your LAN might cause these to not offer as much help as they would otherwise.

Incidentally, that fact that the device is at 1.1 makes me wonder if they've gotten into your router? ;-)

Kevin_Kinsey
  • 181
  • 5
  • 1
    In addition to `root`, I also tend to forbid SSH logins to users with broad sudo privileges, since they are just as bad if compromized. On modern systems `root` itself often doesn't even have a password to begin with. – Dmitry Grigoryev Jan 16 '16 at 19:56
  • While generally that seems a good idea, what do you do for the sysop who needs "broad" privileges? And what "modern systems" do you refer to ... no root password seems a tad counterintuitive ... unless there's no root account? :) – Kevin_Kinsey Jan 21 '16 at 17:43
4

The standard look from the logs seem to be a common brute force attack as the names enlisted in the logs are from a standard dictionary being used. Even the common word-lists consider these as the primary target usernames. Also, the injection is from a Private Class B IP. Don't worry though, as long as the passwords that you have enforced are strong enough and you have a load balance to manage this wont be a problem for you. Look for measures to protect this in SSH Brute Force Protection measures in SSH.

D3X
  • 171
  • 6
3

Since all SSH logins to your server are redirected via the same local IP address, I would advise to use fail2ban with care, if you decide to use it. Installing fail2ban on the same server as sshd will result in the IP 172.25.1.1 being blocked on the spot. After that, nobody will be able to login via SSH to your server.

If you can install fail2ban on the server which redirects SSH traffic, do it. I use fail2ban on my public SSH server, and it does a great job limiting the attacker to 3 attempts in 10 minutes. Most "cracking" scripts the script kiddies are using will simply timeout after these 3 attempts, and won't bother me again for days.

Dmitry Grigoryev
  • 10,122
  • 1
  • 26
  • 56
  • 1
    `fail2ban` has a look at logs for failed authentications do do it´s magic, and the frontend server wont have the logs. – Rui F Ribeiro Jan 17 '16 at 10:37
  • (so placing it on a frontend server that redirects the service wont work as fail2ban wont have absolutely no ideia wether it is dealing with legitimate or failed attempts in the backend) – Rui F Ribeiro Jan 17 '16 at 21:46
3

You can also slow down the password guessing attacker fairly easily.

In the file /etc/pam.d/sshd, you can add a line like this:

auth     optional   pam_faildelay.so   delay=7000000

On every failed sshd login attempt, the PAM module will wait 7 seconds. You may want to increase or decrease the delay, because if you fat-finger your own password, you wait 7 seconds to get another prompt. I would hazard a guess that most SSH password guessing programs are so unsophisticated that they don't have a timeout surrounding waiting for a new prompt, but it's possible that a really long delay would just cause better guessing programs to timeout at some point.

Bruce Ediger
  • 4,552
  • 2
  • 25
  • 26