Prevent a bot accessing login page with multiple IPs and massive list of username/ passwords

Question

For the second time my website seems to be the target of a large automated attack. It seems complex enough and very well executed. I have the following systems in place:

• Captcha on 3rd failed login from IP
• Account lock for 30 min after 5 failed login attempts (using same email)
• Minimum password requirements (8 chrs, letter, number, capitals)
• Failed login attempts return a non-specific error (i.e. your email or password is incorrect)
• Rate limited requests (from same IP)

Over the last half an hour or so, my website has been 20,000 failed login requests. Each request is using a different email (from spot checking) and each one has failed with a 401 and no information. Each request is coming from a different public IP address (all seem to be coming out of Phoenix, Arizona from my manual spot check)

All of the requests are coming via an mobile app I built which loads the login webpage via a webview. Below is a sample of the full details from one request.

I can't think of a way to mitigate this attack. It seems like someone is fishing for email/ password matches. 99% of the emails are not in my system anyway, so it seems to just be a bot with a list of emails and passwords trying to gain access.

My questions are. Should I be worried about this? My biggest concern is the DDOS element with regards to system load. Why would someone even bother doing this? Are there any additional things I could be doing to mitigate the risk?

Sample payload:

{
    "path": "/auth/login/email",
    "method": "POST",
    "query": "POST /auth/login/email",
    "startts": 1598474644337,
    "endts": 1598474644342,
    "responsetime": 5,
    "node": {
        "name": "ip-XXX-XX-XX-XX",
        "version": "",
        "hostname": "ip-XXX-XX-XX-XX",
        "ip": "172.31.15.58"
    },
    "http": {
        "request": {
            "url": "/email",
            "headers": {
                "host": "api.domain.com",
                "x-forwarded-for": "XXX.XXX.XXX.XXX",
                "x-forwarded-proto": "https",
                "x-forwarded-port": "443",
                "x-amzn-trace-id": "Root=1-5f46c994-168fa61913c6b3a2153fe9dd",
                "accept-encoding": "gzip,deflate",
                "content-type": "application/x-www-form-urlencoded",
                "accept": "application/json, text/plain, */*",
                "appsecret": "12312312312313123123",
                "origin": "file://",
                "user-agent": "Mozilla/5.0 (Linux; Android 5.1.1; SM-G973N Build/LYZ28N; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/XX.X.XXXX.XXX Mobile Safari/537.36",
                "accept-language": "en-US,en;q=0.9",
                "x-requested-with": "myapp.bundle.app",
                "x-forwarded-host": "api.domain.com",
                "x-forwarded-server": "ip-XXX-XX-XX-XX.us-east-1.compute.internal",
                "connection": "Keep-Alive",
                "content-length": "45"
            },
            "clength": 45,
            "route_path": "/auth/login/email",
            "params": {},
            "query": {},
            "body": {
                "email": "{\"email\":\"user@domain.co.uk\",\"password\":\"realplaintextpassword\"}",
                "password": "{\"email\":\"user@domain.co.uk\",\"password\":\"realplaintextpassword\"}"
            }
        },
        "response": {
            "code": 401,
            "class": "client_error",
            "phrase": "Unauthorized",
            "headers": {
                "x-dns-prefetch-control": "off",
                "x-frame-options": "SAMEORIGIN",
                "strict-transport-security": "max-age=15552000; includeSubDomains",
                "x-download-options": "noopen",
                "x-content-type-options": "nosniff",
                "x-xss-protection": "1; mode=block",
                "vary": "X-HTTP-Method-Override, Origin",
                "access-control-allow-origin": "file://",
                "uuid": "1231y239hndn9u13u123",
                "server": "Apache",
                "x-ratelimit-limit": 10,
                "x-ratelimit-remaining": 9
            },
            "clength": 5
        }
    },
    "ip": "::ffff:127.0.0.1",
    "real_ip": "107.178.110.130",
    "port": 8081,
    "@timestamp": "2020-08-26T20:44:04.337Z",
    "api": {
        "path": "/auth/login/email",
        "query": "POST /auth/login/email"
    }
}

Answer 1

Attack is automated. You can inspect packets coming from attack vs packets coming from your customers. It can be as simple as the HTTP user agent string or can be some TCP header difference (e.g. some strange flag). Then filter out on the firewall level.

Answer 2

My biggest concern is the DDOS element with regards to system load

Then your defences are not appopriate - unless you are blocking the packets before they get to your webserver they are consuming resources (although even if you drop the packets inside your network they will use your bandwidth, but that is likely to be less of a problem).

I've just implemented fail2ban on my sites (actually I got someone else to do the hard work) and its working a treat.

But do bear in mind that mobile devices are much more likely to be using shared IP addresses - ipv6 POPs or "accelerators".

Answer 3

Attacks generally have an end time- attackers don't spend unlimited time on any one target. You could temporarily blackhole route Arizona logins from that ip range coming in via the app to a "we're sorry" page.

You could also leave them able to log in, but put captcha on first attempt vs letting them fail at all.

For unique ip analysis, it can help to throw the IP list in an excel spreadsheet and remove duplicates, see what vanished.

Answer 4

It seems to be a Credential Stuffing Attack. If the attacker is not using a large set of IP addresses and you can find out the source country and/or service provider, you can block the IP set with a traditional firewall even in your machines, or use a WAF provided by Cloud and CDN providers. They can even block DDoS attacks. Some of them have a free tier.

If the attack impacts your business and the security of your users, maybe you should consider solutions from Identity providers that have automatic and transparent countermeasures.

Answer 5

As you described, those attempts are pretty much indistinguishable from real logins of your clients. Even if you can narrow them down to IPs from specific location or some peculiarity in payload, blackholing them outright, as suggested in other answers is not a good option if you have any real clients from that location or whose software can realistically generate similar payload. Unless you think that dealing with alienating those client is simpler.

So before going for drastic measures do two other things to reduce impact: First: review your login handler. Profile and improve sub-optimal code and database access. If you primary login/password DB is still slow after that, try introducing more lightweight caching through simpler key/value stores, preferably in-memory.

Maybe you won't need to do anything else if implementing those measures already makes load from the bot negligible. Extra speed is good for your service in any case.

If bot is still taking too much resources after that, then do the second thing: reduce your resource consumption even further and slow bot to a crawl through of a lightweight reverse proxy in front of your service - those are often extremely optimized to handle very high load while taking much less resources than "real" service, thus lowering your chance to be DDoSed, but not fit to perform complex business logic.

You don't need anything complex though - set up a small rules that can be checked with something fast and simple like pattern matching on request without accessing any external storage and route everything that "path goes to login handler" + "have IP from specific range" + "...any other peculiarity you noted...". Network-based check like IP range can even be performed at firewall level, sending potentially problematic traffic to separate host assigned to this proxy, while letting the rest of traffic to avoid checks altogether.

Let anything that doesn't match you problematic traffic pattern pass directly to real service and stall potentially problematic requests for some extra time like 5 seconds before letting them go to real logic. Some proxies may even allow you to dynamically adjust delay depending on load. You don't have infinite resources, but neither does the attacker. As long as your mini-handlers on proxy are doing nothing but sleeping, it will be hard to completely flood it - attackers most often do limit amount of sessions open from same IP and wait for answer - they don't have infinite resources too, after all. With attack scripts generally written in dynamic languages that waste attacking device's resource faster than your optimized proxy compiled to binary, you can comfortably hold a big enough load. And if attacker uses a real browser, even headless, or automates your real application client, then you hold even bigger advantage. But even if proxy is completely overloaded, if you placed it on separate host and used firewall to route traffic, as mentioned above, it still won't disturb other clients.

With this setup your potential legitimate clients from networks with sources of bad traffic will only experience some delay on login, which is hardly noticeable by human unless he knows he's deliberately held up, as opposed from being completely denied from your service (i.e. self-DoS).

Answer 6

You are suffering a L7 attack based on your user email, so from my point of view you have the first option that is rate limiting how many times a URI can be called (by the client) per second, for example (if your backend supports that). The other option is to check on black lists, for example IPVoid, and check the reputation of the IPs that generate fails and build a small database, with this database later on you can block the IP addresses certain time for example.

Also you can look to JA3 signatures and try to build something in order to detect bogus clients before they hit your service.

Hope it helps

Answer 7

Another possible denial of service vector is the automatic account lockout.

If they attempt five logins for a legitimate user, that user will be locked out from using your service for a few minutes, and might even have valid sessions terminated. Presumably, the attacker has a list of accounts they want to have service denied to, and they have stuffed that with a set of random email addresses to make it appear like a non-targeted attack.

Mitigation step one is to find whether the attacks all come from the same IP block, and if they have a sensible abuse handling process. If it's a "bulletproof hosting" operation, you're out of luck, but large providers do care about that kind of thing and don't want it on their network. This might be able to get the attack itself shut down.

Second, analysis: I'd filter the list of account names attempted so far down to the valid accounts, and further down to the list of accounts that were actually locked by the automatic lockout, to see if there are some common properties to some of these. It's still possible that the attack is random, but if it isn't, you definitely want to know.