Happy 20th Birthday, BIG-IP TMOS!
I wasn’t in the waiting room with the F5 family, ears and eyes perked for the release announcement of BIG-IP version 9.0. I was a customer back in 2004, working on a government contract at Scott AFB, Illinois. I shared ownership of the F5 infrastructure, pairs of BIG-IPs running version 4.5 on Dell PowerEdge 2250 servers with one other guy. But maybe a month or two before the official first release of TMOS, my F5 account manager dropped off some shiny new hardware. And it was legit purpose-built and snazzy, not some garage-style hacked Frankenstein of COTS parts like the earlier stuff. And you wonder why we chose Dell servers! Anyway, I was a hard-core network engineer at this time, with very little exposure to anything above layer four, and even there, my understanding was limited to ports and ACLs and maybe a little high-level clarity around transport protocols. But application protocols? Nah. No idea. So with this new hardware and an entirely new full-proxy architecture (what’s a proxy, again?) I was overwhelmed. And honestly, I was frustrated with it for the first few days because I didn’t know what I didn’t know and so I struggled to figure out what to do with it, even to replicate my half-proxy configuration in the “new way”. But I’m a curious person. Given enough time and caffeine, I can usually get to the bottom of a problem, at least well enough to arrive at a workable solution. And so I did. My typical approach to anything is to make it work, make it work better, make it work reliably better, then finally make it work reliably and more performantly better. And the beauty here with this new TMOS system is that I was armed with a treasure trove of new toys. The short list I dug into during my beta trial, which lasted for a couple of weeks: The concept of a profile. When you support a few applications, this is no big deal. When you support hundreds, being able to macro configuration snippets within your application and across applications was revolutionary. Not just for the final solution, but also for setting up and executing your test plans. iRules. Yes, technically they existed in 4.x, but they were very limited in scope. With TMOS, F5 introduced the Tcl-based and F5 extended live-traffic scripting environment that unleashed tremendous power and flexibility for network and application teams. I dabbled with this, and thought I understood exactly how useful this was. More on this a little later. A host operating system. I was a router, switch, and firewall guy. Nothing I worked on had this capability. I mean, a linux system built in to my networking device? YES!!! Two things I never knew I always needed during my trial: 1) tcpdump ON BOX. Seriously--mind blown; and 2) perl scripting against config and snmp. Yeah, I know, I laugh about perl now. But 20 years ago, it was the cats pajamas. A fortunate job change Shortly after my trial was over, I interviewed for an accepted a job offer from a major rental car company that was looking to hire an engineer to redesign their application load balancing infrastructure and select the next gear purchase for the effort. We evaluated Cisco, Nortel/Alteon, Radware, and F5 on my recommendation. With our team’s resident architect we drafted the rubric with which we’d evaluate all the products, and whereas there were some layer two performance issues in some packet sizes that were arguably less than real-world, the BIG-IP blew away the competitors across the board. Particularly, though, in configurability and instrumentation. Tcpdump on box was such a game-changer for us. Did we have issues with TMOS version 9? For sure. My first year with TMOS was also TMOS's first year. Bugs are going to happen with any release, but a brand new thing is guaranteed. But F5 support was awesome, and we worked through all the issues in due time. Anyway, I want to share three wins in my first year with TMOS. Win #1 Our first production rollout was in the internet space, on BIG-IP version 9.0.5. That’s right, a .0 release. TMOS was a brand new baby, and we had great confidence throughout our testing. During our maintenance, once we flipped over the BIG-IPs, our rental transaction monitors all turned red and the scripted rental process had increased by 50%! Not good. “What is this F5 stuff? Send it back!!” But it was new, and we knew we had a gem here. We took packet captures on box, of course, then rolled back and took more packet captures, this time through taps because our old stuff didn’t have tcpdump on box. This is where Jason started to really learn about the implications of both a full proxy architecture and the TCP protocol. It turns our our application servers had a highly-tuned TCP stack on them specific to the characteristics of the rental application. We didn’t know this, of course. But since we implemented a proxy that terminates clients at the BIG-IP and starts a new session to the servers, all those customizations for WAN traffic were lost. Once we built a TCP profile specifically for the rental application servers and tested it under WAN emulation, we not only reached parity with the prior performance but beat it by 10%. Huzzah! Go BIG-IP custom protocol stack configuration! Win #2 For the next internal project, I had to rearchitect the terminal server farm. We had over 700 servers in two datacenters supporting over 60,000 thin clients around the world for rental terminals. Any failures meant paper tickets and unhappy staff and customers. One thing that was problematic with the existing solution is that sometimes clients would detach and upon reconnect would connect directly to the server, which skewed the load balancers view of the world and frequently overloaded some servers to the point all sessions on that server would hang until metrics (but usually angry staff) would notify. Remember my iRules comment earlier on differentiators? Well, iRules architect David Hansen happened to be a community hero and was very helpful to me in the DevCentral forums and really opened my eyes to the art of possible with iRules. He was able to take the RDP session token that was being returned by the client, read it, translate it from its Microsoft encoding format, and then forward the session on to the correct server in the backend so that all sessions continued to be accounted for in our load balancing tier. This was formative for me as a technologist and as a member of the DevCentral community. Win #3 2004-2005 was the era before security patching was as visible a responsibility as it is today, but even then we had a process and concerns when there were obstacles. We had an internal application that had a plugin for the web tier that managed all the sessions to the app tier, and this plugin was no longer supported. We were almost a year behind on system and application patches because we had no replacement for this. Enter, again, iRules. I was able to rebuild the logic of the plugin in an iRule that IIRC wasn’t more than 30 lines. So the benefits ended up not only being a solution to that problem, but the ability to remove that web tier altogether, saving on equipment, power, and complexity costs. And that was just the beginning... TMOS was mature upon arrival, but it got better every year. iControl added REST-based API access; clustered multi-processing introduced tremendous performance gains; TMOS got virtualized, and all the home-lab technologists shouted with joy; a plugin architecture allowed for product modules like ASM and APM; solutions that began as iRules like AFM and SSLO became products. It’s crazy how much innovation has taken place on this platform! The introduction of TMOS didn’t just introduce me to applications and programmability. It did that and I’m grateful, but it did so much more. It unlocked in me that fanboy level that fans of sports teams, video game platforms, Taylor Swift, etc, experience. It helped me build an online community at DevCentral, long before I was an employee. Happy 20th Birthday, TMOS! We celebrate and salute you!Security Automation with F5 BIG-IP and Event Driven Ansible
Updated (September 19th 2023) INTRODUCTION TO EVENT DRIVEN SECURITY: Event Driven Security is one of the projects I have been working on for the last year or so. The idea of creating automated security that can react similarly to how I would react in situations is fascinating to me, and then comes the BIG Question.... "Can I code it?" Originally our solution we had utilized ELK (Elastic Logstash Kibana) where Elasticsearch was my logging and monitoring tool, Kibana was the frontend GUI for helping me visualize and set up my watchers for my webhook triggers, Logstash would be an intermediary to receive my webhooks to help me execute Ansible related code. While using Logstash, if the Ansible code was simple it had no issues, however when things got more complex (i.e., taking payloads from Elastic and feeding them through Logstash to my playbooks), I would sometimes get intermittent results. Some of this could be my lack of knowledge of the software but for me it needed to be simple! As I want to become more complex with my Event Driven Security, I needed a product that would follow those needs. And luckily in October 2022 that product was announced "Event Driven Ansible" it made it so I didn’t need Logstash anymore i could call Ansible related code directly, it even took in webhooks (JSON based) to trigger the code, so I was already half way there! CODE FOR EVENT DRIVEN SECURITY: So now I have setup the preface let’s get down to the good stuff! I have setup a GitHub repository for the code i have been testing with https://github.com/f5devcentral/f5-bd-ansible-eda-demo which is free for all to use and please feel free to take/fork/expand!!! There are some cool things worth noting in the code specifically the transformation of the watch code into something usable in playbooks. This code will take all the times the watcher finds a match in its filter and then then copies the Source IP from that code and puts it into a CSV list, then it sends the list as a variable within the webhook along with the message to execute the code. Here is the code I am mentioning above about transforming and sending the payloads in an elastic watcher. See the Full code in the GitHub repo. (Github Repo --> elastic --> watch_blocked_ips.json) "actions": { "logstash_exec": { "transform": { "script": { "source": """ def hits = ctx.payload.hits.hits; def transform = ''; for (hit in hits) { transform += hit._source.src_ip; transform += ', ' } return transform; """, "lang": "painless" } }, "webhook": { "scheme": "http", "host": "10.1.1.12", "port": 5000, "method": "post", "path": "/endpoint", "params": {}, "headers": {}, "body": """{ "message": "Ansible Please Block Some IPs", "payload": "{{ctx.payload._value}}" }""" } } } } In the Ansible Rulebook the big thing to note is that from the Pre-GA code (which was all CLI ansible-rulebook based) to the GA version (EDA GUI) rulebooks now are setup to call Ansible Automation Platform (AAP) templates. In the code below you can see that its looking for an existing template "Block IPs" in the organization "Default" to be able to run correctly. (Github Repo --> rulebooks --> webhook-block-ips.yaml) --- - name: Listen for events on a webhook hosts: all ## Define our source for events sources: - ansible.eda.webhook: host: 0.0.0.0 port: 5000 ## Define the conditions we are looking for rules: - name: Block IPs condition: event.payload.message == "Ansible Please Block Some IPs" action: run_job_template: name: "Block IPs" organization: "Default" This shows my template setup in Ansible Automation Platform 2.4.x, there is one CRITICAL piece of information i wanted to share about using EDA GA and AAP 2.4 code is that within the template you MUST tick the checkbox on the "Prompt on launch" in the "variables section". This will allow the payload from EDA (given to it from Elastic) to pass on to the playbook. In the Playbook you can see how we extract the payload from the event using the ansible_eda variable, this allows us to pull in the event we were sent from Elastic to Event Driven Ansible and then sent to the Ansible Automation Platform template to narrow down the specific fields we needed (Message and Payload) from there we create an array from that payload so we can pass it along to our F5 code to start adding Blocked IPs to the WAF Policy. (Github Repo --> playbooks --> block-ips.yaml) --- - name: ASM Policy Update with Blocked IPs hosts: lb connection: local gather_facts: false vars: Blocked_IPs_Events: "{{ ansible_eda.event.payload }}" F5_VIP_Name: VS_WEB F5_VIP_Port: "80" F5_Admin_Port: "443" ASM_Policy_Name: "WAF-POLICY" ASM_Policy_Directory: "/tmp/f5/" ASM_Policy_File: "WAF-POLICY.xml" tasks: - name: Setup provider ansible.builtin.set_fact: provider: server: "{{ ansible_host }}" user: "{{ ansible_user }}" password: "{{ ansible_password }}" server_port: "{{ F5_Admin_Port }}" validate_certs: "no" - name: Blocked IP Events From EDA debug: msg: "{{ Blocked_IPs_Events.payload }}" - name: Create Array from BlockedIPs ansible.builtin.set_fact: Blocked_IPs: "{{ Blocked_IPs_Events.payload.split(', ') }}" when: Blocked_IPs_Events is defined - name: Remove Last Object from Array which is empty array object ansible.builtin.set_fact: Blocked_IPs: "{{ Blocked_IPs[:-1] }}" when: Blocked_IPs_Events is defined ... All of this combined, creates a well-oiled setup that looks like the following diagram below, with the code and the flows setup we can now create proactive event based security! Here is the flow of the code that is in the GitHub repo when executed. The F5 BIG-IP is pushing all the monitoring logs to Elastic. Elastic is taking all that data and storing it while utilizing a watcher with its filters and criteria, The Watcher finds something that matches its criteria and sends the webhook with payload to Event Driven Ansible. Event Driven Ansible's Rulebook triggers and calls a template within Ansible Automation Platform and sends along the payload given to it from Elastic. Ansible Automation Platforms Template executes a playbook to secure the F5 BIG-IP using the payload given to it from EDA (originally from Elastic). In the End we go Full Circle, starting from the F5 BIG-IP and ending at the F5 BIG-IP! Full Demonstration Video: Check out our full demonstration video we recently posted (Sept 13th 2023) is available on-demand via https://www.f5.com/company/events/webinars/f5-and-red-hat-3-part-demo-series This page does require a registration and you can check out our 3 part series. The one related to this lab is the "Event-Driven Automation and Security with F5 and Red Hat Ansible" Proactive Securiy with F5 & Event Driven Ansible Video Demo LINKS TO CODE: https://github.com/f5devcentral/f5-bd-ansible-eda-demo2022 DevCentral MVP Announcement
Congratulations to the 2022 DevCentral MVPs! Without users who take time from their busy days to share their experience and knowledge for others, DevCentral would be more of a corporate news site and not an actual user community. To that end, the DevCentral MVP Award is given annually to the outstanding group of individuals – the experts in the technical F5 user community who go out of their way to engage with the user community. The award is our way of recognizing their significant contributions, because while all of our users collectively make DevCentral one of the top community sites around and a valuable resource for everyone, MVPs regularly go above and beyond in assisting fellow F5 users. We understand that 2021 was difficult for everyone, and we are extra-grateful to this year's MVPs for going out of their ways to help others. MVPs get badges in their DevCentral profiles so everyone can see that they are recognized experts. This year’s MVPs will receive a glass award, certificate, exclusive thank-you gifts, and invitations to exclusive webinars and behind-the-scenes looks at things like roadmaps, new product sneak-previews, and innovative concepts in development. The 2022 DevCentral MVPs are: Aditya K Vlogs AlexBCT Amine_Kadimi Austin_Geraci Boneyard Daniel_Wolf Dario_Garrido David.burgoyne Donamato 01 Enes_Afsin_Al FrancisD iaine jaikumar_f5 Jim_Schwartzme1 JoshBecigneul JTLampe Kai Wilke Kees van den Bos Kevin_Davies Lionel Deval (Lidev) LouisK Mayur_Sutare Neeeewbie Niels_van_Sluis Nikoolayy1 P K Patrik_Jonsson Philip Jönsson Rob_Carr Rodolfo_Nützmann Rodrigo_Albuquerque Samstep SanjayP ScottE Sebastian Maniak Stefan_Klotz StephanManthey Tyler.HattonAgility sessions announced
Good news, everyone! This year's virtual Agility will have over 100 sessions for you to choose from, aligned to 3 pillars. There will be Breakouts (pre-recorded 25 minutes, unlimited audience) Discussion Forums (live content up to 45 minutes, interactive for up to 75 attendees) Quick Hits (pre-recorded 10 minutes, unlimited audience) So, what kind of content are we talking about? If you'd like to learn more about how to Simplify Delivery of Legacy Apps, you might be interested in Making Sense of Zero Trust: what’s required today and what we’ll need for the future (Discussion Forum) Are you ready for a service mesh? (breakout) BIG-IP APM + Microsoft Azure Active Directory for stronger cybersecurity defense (Quick Hits) If you'd like to learn more about how to Secure Digital Experiences, you might be interested in The State of Application Strategy 2022: A Sneak Peak (Discussion Forum) Security Stack Change at the Speed of Business (Breakout) Deploy App Protect based WAF Solution to AWS in minutes (Quick Hits) If you'd like to learn more about how to Enable Modern App Delivery at Scale, you might be interested in Proactively Understanding Your Application's Vulnerabilities (Discussion Forum Is That Project Ready for you? Open Source Maturity Models (Breakout) How to balance privacy and security handling DNS over HTTPS (Quick Hits) The DevCentral team will be hosting livestreams, and the DevCentral lounge where we can hang out, connect, and you can interact directly with session presenters and other technical SMEs. Please go to https://agility2022.f5agility.com/sessions.html to see the comprehensive list, and check back with us for more information as we get closer to the conference.May the Fourth and Security Trivia on DevCentral with a Chance to Win Prizes
DevCentral is hosting two trivia games to celebrate National Password Day AND May The Fourth (Star Wars Day). Come show off your Star Wars security, password security, and F5 security knowledge in this live event hosted by the DevCentral team and some special guests. There will be three rounds of questions during this hour, with prizes for the winners of each round as well as one lucky random player per game. Get details on how to join by clicking on Option 1: 8-9pm Pacific on Thursday, May 3 (7am in the UAE, 8:30am in Bangalore, 11am in Singapore, noon in Tokyo, 1pm in Melbourne and Sydney on May 4 th ) or Option 2: 8-9am Pacific on Thursday, May 4 (4pm BST, 5pm CET, 6pm EEST, and 7pm GST on May 4 th )Hanging with DevCentral at RSA Conference 2023
AubreyKingF5 and buulam spent last week at RSA Conference 2023 in San Francisco connecting with security professionals and colleagues. And, of course, they filmed a lot of those talks to share with the community. Buu connected with Tanya Janca at B-Sides San Francisco, Jason Haddix (BuddoBot), and Joel Violette (IBM) who shares his thoughts on ChatGPT (AI was a hot topic this year across the board). Aubrey shared his RSA Keynote takeaways, interviewed Alyssa Miller before her panel on CISO challenges, and connected with a ton of other organizations across the expo room floor. Also making an appearance: Scheff, Kyle_Roberts, and Cameron_Delano who all shared insights from their conversations at the F5 booth. Check out the full playlist of RSA 2023 content here!That's A Wrap From BlackHat 2022 - API Security, it's time to shine
We hit the road and we hit it HARD! Between the editing and interviews, it was hard to find a lot of time for sleep. This was a truly immersive experience, heading out there. Coming back to producing another show immediately, I've had some time to reflect on our trip to BlakHat USA 2020. There were some resounding themes, for sure. Firstly, it is an API world. If you look at the summary article of BlackHat 2022 announcements I talked about on DevCentral Connects yesterday, you see a trend: I am THRILLED to see the rest of the security industry finally taking an interest on the segment that we've led in terms of industry deployment numbers for 15 years. In 2007, TMOS 9.2.4 delivered an XML firewall to our customers - able to ingest a WSDL for schema baselines - that learned expected API usage over time like any other http delivered application. SOAP followed shortly, thereafter and, of course, REST shortly after that. This was just an evolution of WAF here at F5. Watching NGINX+ delivery of API gateways add on NGINX App Protect features with the F5 WAF engine showed me that we continue to value this level of API defense at every scale, as well. When I saw Gartner and other vendors starting to call the segment 'WAAP,' it bothered me a bit, as we at f5 have always understood WAF to encompass APIs. I really hope that we can capitalize on our pedigree in this space to deliver our customers the most reliable and scalable API Security solutions there are. Regarding the f5 booth, it's been a minute since I've been at a trade show for f5. The thing that stuck out the most for me was the lack of the term 'load balancing.' What I heard more of was 'API Security,' and 'Multi-Cloud Networking'. What struck me the most was that customers seemed to understand that multi-cloud was really our thing now, and I think that having partners like RedHat has helped us to be viewed as a software company, rather than a big iron vendor. This speaks volumes to our ongoing transformation and our ability to get that message out there. Kudos to our sales teams for really making that apparent. Be sure to follow us all on the socials for content, in addition to YouTube: DevCentral handles: https://www.linkedin.com/showcase/f5-devcentral/ | https://twitter.com/devcentral Peter Silva: https://www.linkedin.com/in/psilvas/ | https://twitter.com/psilvas Buu Lam: https://www.linkedin.com/in/buulam/ | https://twitter.com/buulam Aubrey King: https://www.linkedin.com/in/aubreyking | https://twitter.com/aubreykingf5 Content summary: BlackHat USA 2022 Playlist: https://www.youtube.com/playlist?list=PLyqga7AXMtPPL3Xw0qrWO3sZnsbL4amP5SPECIAL DevCentral Connects - on Basic iControl Security
Basic iControl Security - DevCentral Connects Join JRahm and buulam on May 12, 2022 at 8AM Pacific as they look at some common questions for people who are getting to know the BIG-IP iControl API. How To: * find out if you're using iControl * verify what queries are being run * Review the iControl logs and * restrict access to iControl Join our DevCentral Connects group to see show notes and get links to relevant articles.Attempting the new path to BIG-IP Certified Administrator
A couple weeks ago I had kjsalchow on for an episode of DevCentral Connects, which you can watch at your pleasure here: I had reached out to Ken and HeidiSchreifels after one our MVPs made a comment on this new path toward certification. I missed the memo (Heidi's article here), but this was big news and I knew we needed to have a conversation with the community. During our chat, Ken mentioned that all five beta tests required to earn your BIG-IP Certified Administrator would be available on-site at AppWorld (as would the recertification test available to ANYONE who has previously held the cert) and that he needed more people to start at #5 and work backwards. And so I did. I had time Tuesday morning before "The Hub" opening party kicked things off, so I knocked out all five beta tests. Here are my thoughts about the experience. I went in completely blind. I did not study, (but the blueprint is here for you) and I did not do the prep work to get my device ready for testing. I did pre-register for tests #3 - #5. When I arrived at the room, the cert team did a great job helping me get the tools set up on my laptop. The test environment downloads a secure browsing session, and there are some known issues with company laptops that lock things down, so you might be best suited to test from a personal laptop. To my knowledge in discussions with them, tablets are not supported. The certiverse delivery was great. Strong improvement from what I recall for the previous versions. Seeing the questions and the diagrams and being able to reference back and forth was far easier to assess the challenges. I always try to use the flagging system for review and that worked great. As this was a beta, I took extra time to provide feedback. For the betas, I had a range of questions from I think 39 to 58 across the five tests and an hour to complete each. For production tests, I believe that will be 30/30. None of them concerned me on time. I really liked the breakdown in the new format. This allows you to progress through the material when studying without having to keep it all upstairs for one test. BIG-IP Administration Install, Initial Configuration, and Upgrade BIG-IP Administration Data Plane Concepts BIG-IP Administration Data Plane Configuration BIG-IP Administration Control Plane Administration BIG-IP Administration Support and Troubleshooting I felt pretty good about the analysis questions, that stuff is pretty cemented in my brain. I work mostly with the BIG-IP APIs now, so I'm less solid on specific tmsh commands or tmui click paths. I put myself in the cone of shame on a few questions because I filmed lightboard lessons for them but I wasn't confident in the right answer. All that said, I have no idea if I passed them, but I think I hit minimally viable candidate on four of them? As they were betas, there were some questions that probably need to be removed, and some questions might need to be refined a little. This is where the always fascinating psychometrics come into play. But for the most part, I though they were a good summary of the knowledge one should have for basic administration. I got the first three tests completed quickly enough to take the other two. Registering for them on-site and jumping into them was painless. The cert team is the bomb-diggity. They're so helpful, friendly, encouraging, and super eager to make everyone successful. It's always a pleasure to cross paths with them! The downside of betas is they are not scored immediately, so I have to wait. Jason does not like waiting... How about you, community? Anyone else take the betas for the refreshed BIG-IP Certified Administrator (or the recertifying exam) and want to share your experience?
