WorldTech IT - Who Ya Gonna Call? Scary Hack Story
At WorldTech IT, our specialty for Always-On emergency support means that when things go bump in the night on F5 devices, we're the ones who wake up and investigate. We've seen our fair share of headless entities, killer bugs, gremlins, possessions, zombies, and daemons, but our scariest hack story started like any other day. I was making coffee in our headquarters, a haunted house built on a BIG-IP 1600s burial site, when my phone lit up in the darkness. Ten letters burn on the screen, more horrifying than anything spelled on a Ouija Board: SEV1 BREACH.
This particular client hired us as break-glass support, and we knew that any call from inside the house would be deadly serious. We had spent months trying to convince them to update their numerous BIG-IPs, but like warning someone to stay away from a haunted house or a legacy loadbalancer, our warnings fell on deaf ears. Luckily for our client, the source of the hack was more Halloween trick than coordinated attack—your neighborhood script kiddies armed with spray paint and firecrackers instead of trained professionals.
Despite that, it was a perfect storm: A recent CVE alerted hackers to a TMUI vulnerability, allowing terminal access to each box. The client might have survived the night had they not enabled port 443 on self IP. The hackers, finding the proverbial lights turned on and door left open, loaded up an `rm -rf *` payload and watched as every box was wiped entirely. Our clients had narrowly avoided a ransomware or spyware attack but quickly noticed there was somethin' strange underneath their hood, somethin' weird, and it didn't look good. Who ya gonna call?
Our team loaded up their proton packs, silver bullets, wooden stakes, arcane flavors of Linux, and jumped right into action. If the hackers had run a reboot, the F5 boxes would have been dead with no possibility of reanimation. To add to the horror, the client's most recent backup was 7 months old. The only way to jolt this monster back to life would be to rebuild its old parts one by one. Some we could rescue from the ancient backup, and others had to be deciphered from the past year of change requests. In another killer twist, every encrypted key had vanished into thin air with the data wipe. Our team worked around the encrypted data, resurrecting portion by portion to get as close to the living thing as possible.
The only way to avoid an angry mob of end-users was to rebuild without any abnormalities or mistakes lest our monster break free incomplete. Our engineering team is used to late nights in the lab, but as the shifts stretched into double-digit hours, things took a turn into the bizarre. The wild eyes, lightning-fast typing, and incoherent rambling—that was all normal, but no mere human could have consumed that much Dr. Pepper. Our mad scientists prevailed and ensured no loss of service while the BIG-IPs were repaired.
Our client was thrilled that the worst of the scares were over. Our team had done what none thought possible, completely rebuilt their F5s with incomplete backups and even made some improvements along the way. We saved the client millions of dollars by avoiding downtime or service interruption, and continue to emphasize the importance of regular updates and monitoring for F5.
We have plenty of spooky hack stories to choose from at WorldTech IT, but by relying on F5's killer technology and expert community of threat slayers, we're always able to keep the dark forces at bay. Happy Halloween to our DevCentral community—may your candy bag be full and your support inbox empty!
