A Day in the Life of a Security Engineer from Tel Aviv
October 2022 is the Cybersecurity Awareness Month, so we decided to focus on the human aspect of the F5 SIRT team and share some of our day to day work. When I started writing this, I thought it would be trivial to capture what I do on an average day and write about it. But it turned out to be challenging task simply because we do so much. We interact with many groups and there is always a new top priority. So bouncing back and forth between tasks is the only way to execute when you are deeply involved with security in the organization. There is really no average day as the next security emergency is right around the corner.
First, a little background info on me: I started working in F5 at 2006 as a New Products Introduction (NPI) engineer representing the customer throughout the product life cycle. The job included attending design meetings on new features and their implementation in real world with Product Development (PD) and Product Management (PM). The deliverables were technical presentations for both online and in-person at internal F5 conferences. The feedback that I got from the various departments were consolidated into improvements list to PD and PM, acting as a feedback loop for new features. The product that I represented as subject matter expert was BIG-IP Application Security Manager (ASM) that evolved to BIG-IP Advanced WAF, which is my specialization and my favorite technical topic until today.
Then at end of 2016 I moved to the F5 SIRT team. The shift was beneficial as it started a new chapter in becoming a full time security engineer. Let me describe to you what that looks like.
Morning: coffee & emergency catchup first
It is a 12-minute drive from my apartment to the the office based in Tel Aviv. Living close to the office is great and you can see the sea from the 30th floor. Yes, I know I should/can bike, and I will do more biking now that the summer heat is going away.
First cortado is for reading emails that pile up overnight. We are a "follow the sun" coverage group so here's a quick time orientation: when I arrive to the office at 9AM, it's lunch time for the Singapore guys at 13:00 (5 hours ahead) and the US guys are getting ready to sleep at 11PM Seattle time (10 behind). This means that I usually have a long list of emails and messages to read. I catch up on all the emergency cases that are ongoing and reach my time zone for monitoring or follow up actions.
F5 SIRT is a unique group of top engineers with many years of experience with F5 security products and security in general. We are responsible for three main pillars and the first one is assisting F5 customers when they are under attack. Since we are an emergency team, we are ready to act from the minute we come to work and we like the excitement of solving emergency cases.
The F5 security team is ready to help customers when they need us the most, when the customers are under attack. This is what I call the money time as this is why people buy security products, to mitigate attacks using the F5 products. This moment has arrived.
The first line of defense is the F5 SIRT specialist group which handles the request from the customer and marks it as emergency. If they need assistance from a security engineer, then they will ping us. Working in collaboration with the SIRT specialist group always feels good. It's great to have someone to trust, especially working with EMEA F5 SIRT specialists who always set a high standard.
When I’m needed to help a customer under attack, verbal communication is always more effective and faster than written communication. This call ensures the technical issues, the risks, and the benefits involved in mitigation are the right ones so that the customer can choose the best path forward.
Common action includes understanding the customer environment, the attack indication they have seen, and the severity of the incident. Once we collect the information from the customer, we create a plan that lists all the possibilities to mitigate the attack. Sometimes we simply give good advice on the possible mitigation and how to proceed but sometimes we need to have a full war room where we do deep traffic analysis and provide the specific mitigation to kill the attack.
We have seen many attacks and each attack is different but essentially, we classify them to those main categories:
Graph : Distribution of the attack over time.
Working in SIRT requires understanding of the different environments, the attack landscape and above all a deep understanding of F5 security products. These are your best friends in killing the attack. Finding the best mitigation strategies with our products which leads to a successful prevention is what we do best. It is a very good fealing to lead the way to an incident win.
At the end of each incident, we create a report with recommendation to the customers as well as internal analysis because if it's not documented it doesn’t exist. We have a high success rate in mitigating attacks mostly because F5 product suite is one of the best in the industry for mitigating network and web application attacks. We usually get a lot of warm words from customers.
And with that, it's now lunch time. Time flies when you are busy.
Noon: lunch and CVE’s
Deciding what to eat should be evaluated carefully: too heavy and you fall asleep, too light and you will be hungry in 2-3 hours. If time permits, I'll walk 10 minutes each direction to the local food market with the local F5 employees and have fun conversations over lunch.
When I get back it is time to take a black coffee and review the additional work that needs to be done for the day and decide which of the items I can delay. Most of the time we define our own deadlines, so we plan ahead. This means we have no one to blame if we are late. So don’t be late.
This is also a good time to read some of the security industry news. If there is something notable, I will paste the link in the group team’s chat. If it's my turn to write the This Week in Security (TWIS), then this is where I will mark topics to write about. Writing TWIS can be time consuming, but it provides the ability to express yourself and keep up to date with the security industry around the globe.
Now, it is CVE work time, which is our second pilar of responsibility: vulnerability management for F5 products. F5 SIRT owns the vulnerability management and publishes public CVEs as part of the F5's commitment to security best practices with F5 products. We have public policy that we follow: K4602: Overview of the F5 security vulnerability response policy.
CVEs can originate from internal or external sources such as a security researcher who approached the F5 SIRT team directly. We evaluate CVEs to make sure we understand the vulnerability from both the exploitation aspect and the relevant fix introduced by Product Engineering (PE). After Interacting with PE, and once the software fix is in place, we start writing the security advisory which is the actual article that will be published.
All CVEs are under embargo until publication day and just before we publish we provide briefing to internal audience to inform them of what to expect and which type of questions they might encounter. We work as a group to cover all the regions and keep everyone on the same page.
Publication day is always a big event for us. This is where all the hard work comes into the light. We are constantly monitoring customers inquiries about fresh CVEs and are ready to solve any challenges customers may face. We always invest a lot of time and effort, so we created a well-defined playbook and a common language so that we can publish well-documented CVEs. Vulnerabilities and their CVEs will never run out, this is the nature of software and hardware.
Time for ristretto and the Zero Day (0day) aka the OMG scenario.
Every now and then, a new high-profile 0day is being published. This is the start of a race to mitigation and our play books are ready for those situations. We start by collecting all the possible information available and evaluate the situation.
If F5 products are affected by the 0day, a software fix will be issued ASAP and customer notification will be released by us describing the actions that need to be taken.
If we are not affected, then we want to find a mitigation to help our customer protect themselves.
In both cases we will write a security article in AskF5, as well as internal communication and briefing on our findings and remediations. Those will include all possible mitigations such as WAF signatures, iRules, AFM IPS signature, LTM configuration and more.
The Log4j 0day was a good example of how a solid process works like magic and we published mitigation list articles and email notifications very fast. In such cases we work with the Security Research Team from the local Tel Aviv office, a very talented group of people that assists and collaborates with us all the time with full dedication for high profile CVEs. This is where the power of F5 as a company shows its face.
Once we have our mitigations plan for the 0day, F5 SIRT will send a notification email to our customers and publish information on the Ask F5 site and on social media (of course). This typically increases customers inquiries about the level of exposure they have from this new 0day, so publishing articles and knowledge is critical to fast mitigations for our customers.
And it is afternoon already.
Afternoon: tea, knowledge share and projects
Technology is constantly improving and new features, products and services are being released to confront upcoming attacks. Therefore learning and practicing new releases is mandatory. The more we learn, know and get our hands on, the better we can mitigate security challenges when dealing with customers under attack and vulnerability management.
This is also our third pillar: security advisor, which is about learning and building security mindset by sharing knowledge and experience. We write knowledge base articles on Ask F5, we mentor whenever we have good advice, and we answer security inquiries from both internal and external sources.
This knowledge and experience translates to projects that we chose to do every quarter. My favorite project that I was leading (and is still very alive and relevant today) is the Attack Matrix that is used as a battle cards for customers and F5 personnel. The basic concept is to have attacks and their corresponding mitigations with F5 products. This is a very effective tool for customers and demonstrates the power of the F5 security capabilities. I mostly liked doing the WAF section (remember my favorite F5 product is BIG-IP Advance WAF) which IMHO is the best WAF technology in the industry.
Late afternoon: meet the team
You probably already figured out that my time zone is EMEA. Together with AaronJB, we cover the three pillars of the F5 SIRT team for the EMEA region. We discuss new ideas often and sometimes it feels like we can talk about security for weeks. So thank you, Aaron, for helping me and for being around.
No matter how good you are as an individual, you must have a team to really succeed! As the day comes to an end and North America wakes up (8AM Seattle time is 6PM in Tel Aviv), we have a sync calls for the core team and other teams. It always feels good to talk to the F5 SIRT core personnel from APCJ and NA whom I work with every day. With our fearless leader who established this security A-Team, it is such a pleasure working in this group.
Day report
This was a day in the life of an F5 SIRT team memeber and it is totally subject to immediate changes, an emergency can arrive at any time of the day. There are days where everything becomes a war room, when there is a worldwide high-profile security incident is invoked. And there are days where I can have a cup of coffee and write an article like this. Security became a necessity, every aspect of software and computer system is affected directly by threats. So security mitigation is here to stay and is key to keeping it all going.
There is much more to these organized and erratic workdays and I can talk and talk but the day has ended so until next time...
Keep it up.
