F5 SIRT ESRP Case Study: Regional Bank

 

 

 

 

 

A regional bank was experiencing a credential stuffing attack and opened a case with F5 Support to invoke the ESRP process.   Credential stuffing attacks are a subset of brute force attacks where the attacker has a list of usernames and passwords that they are trying against a target.    We see similar attacks with credit card details, but with those its usually targeted against some site that makes it easy to try credit card numbers to test their validity, like donation portals.    

 

When the case was opened it was assigned to a F5 SIRT Specialist II who was trained on Application Security Manager  (The predecessor of Advanced WAF) and a Zoom meeting was assembled with the customer contacts, the F5 SIRT Specialist and an F5 SIRT Security Engineer.    After a bit of initial triage the Security Engineer sent out a notification to the F5 Sales team for the customer to let them know about the attack and potential need for trial access to F5 products to stem the attack.    One of these products was Shape Security.

 

In the case of this bank's attack, the attacker had a mix of usernames with passwords and emails with passwords, so it was initially easy to slow them down by blocking access that involved a username with an @ in it.      After that it became a bit harder, ASM as it was called at the time (now Advanced WAF) was employed and bot protections were set up.    We were also able to leverage ASM's protections against credential stuffing attacks.

 

This effort was able to slow down the attackers enough that they stopped several times and came back with attempts to bypass the protections.    When we saw bypasses we were able to escalate those issues to the ASM Engineering Services team and during at least some of this incident's Zoom calls we brought the ASM ENEs on the call to help gather data and test configuration fixes live during the attack.

 

During this F5 Sales was working to put together a proof of concept for deploying Shape Security (now F5 Distributed Cloud Bot Defense)  in-front of the bank's web banking portal.    Ultimately this test proved to shut down the attackers, who tried very hard to bypass the Shape iRule but ultimately their attempts allowed us to harden the iRule against bypass attempts.    Since the shape solution was fully deployed and hardened, the customer has not had an incident that required activating ESRP and the customer continues to use F5 Distributed Cloud Bot Defense to protect their web banking application to this day.

 

You may be wondering what information was being collected during this, the bank had a robust SIEM system that was able to provide both statistical data about login dispositions and sessions as well as individual HTTP requests, this combined with violation reports from ASM allowed us to examine the techniques the attacker was using.  

 

 

The fast triage of the issue and assignment of an engineer that has experience in dealing with these attacks helped the customer start responding to the attack right away.   While not all of the attack could be blocked, it was brought under enough control that they could effectively deal with compromised web banking accounts while allowing customers to continue to use the web banking application like nothing was happening.    There were times when certain types of attacks were anticipated that did not happen because the infrastructure allowed for protections to be added without disrupting existing mobile applications.

 

During the time this attack was happening we had seen a series of attacks against the API endpoints for mobile banking applications and protections ASM could provide for those were limited because, well, you cant get a human signature from an app.    The structure of this regional bank's application allowed us to push updates to the authentication flow from the ASM and so it slowed down automated access enough to make it not worth the effort for the attacker.     Sometimes that's exactly what shuts down an attack, if an attacker can move on to somewhere else easier than continuing their attack on you, they may very much do that, allowing you time to regroup and shore up defenses.

 

 

One of the most common tools used to analyze attacks has been Wireshark. F5 has a special plugin for Wireshark that allows metadata about connections to be examined and filtered on.  This plugin is included in Wireshark as of version 2.6, so you don't have to go hunting for it.  Also handy is the graphs and tmctl parts of F5 iHealth.    Each QKView contains statistical information over time that allows us to dig into changes over time.    This means that if you grab a QKView every week or so and store it away you have the ability to look back on the state of F5 BIG-IPs, a valuable tool when trying to get to the bottom of when something started happening or went wrong.    I recommend to customers that they should consider periodically generating and storing QKView and UCS files, as well as generating them before and after every change window.

 

With the rise of DNS NXDOMAIN attacks in the last few months, another tool that we have been using a bit of is Network Miner.  This tool allows you to open up a packet capture and a view a number of things, like a list of hosts with some OS fingerprinting done on that list, but more important to DNS attacks is the DNS tab, which makes it really easy and fast to have a look at what kind of requests are being made, so NXDOMAIN attacks can be quickly identified.

 

How NXDOMAIN attacks work is that a distributed attacker makes a large number of requests for nonexistent subdomains on a domain they are targeting, because these requests are for randomly generated subdomains each request bypasses the cache on DNS servers and goes straight to the authoritative servers for that domain.  Because the attacker generates a high volume of these requests  and they all go to the authoritative servers these attacks often result in a very large increase in DNS requests and can be especially hard for DNS domains using DNSSEC to respond to, since each response will need to be signed. 

 

You can mitigate NXDOMAIN attacks using DNS Express to serve DNS replies from BIG-IP memory and filter them by using Advanced Firewall Manager's DoS protections.

 

 

If you are an F5 customer, you can visit support.f5.com to open a case for security incident.  We also have a number of regional tool free numbers in case you want to raise a case by telephone.    When raising a case for a security emergency please provide as much information as possible, especially what products are involved, so we can triage the case to an engineer trained on those products.