Avoiding Common iRules Security Pitfalls

[Jordan_Zebor]

Introduction

iRules are an event driven language available on BIG-IP, which allow the operator to extend capabilities that are unavailable in the CLI or GUI. Based on the Tool Command Language (TCL), F5 has customized an implementation of the TCL language in BIG-IP to include additional events and commands. The additional capabilities are broad, from supporting uses cases such as extending our Web Application Firewall, organizing network flows based on URL categorization and/or application classification, rewriting HTTP requests and responses and customized logging attributes of network flows.  Along with adding capabilities to the standard TCL language, F5 has also removed some of the dangerous functionality inherited by the language. We did this to create a bounded operating environment which avoids use of the riskier functions such as invocation of a subprocess, opening arbitrary system files, and installation of TCL packages.

While we strive to create a balance between customer security and usability, just like any programming language there are common security pitfalls even the most seasoned iRule developers can fall into. This article is written with the intention to inform new and experienced iRule developers of common security pitfalls I've observed over the years. While some of these recommendations can be generically applied, this article is scoped to the TCL based iRules implementation and does not specifically cover Node.js iRulesLX use cases.

Double Substitution

One of the main drivers for this article is to inform iRule developers on the issue of double substitution. Double substitution AKA double evaluation is a form of code injection. Code injection attacks are a threat that can allow attackers to gain control of application flow and/or cause other unexpected impacts. In other words, code injection is bad. These can be prevented by following the best practice for TCL scripting, which is to enclose all expressions, ensuring that they are not substituted or evaluated unexpectedly. For more specific guidance on identifying risky use and how to fix it, we've published K15650046: Tcl code injection security exposure. In addition to user education, we've implemented static code analysis on BIG-IP that can detect cases where double substitution may be possible. In addition to local checks, iHealth will trigger heuristic D015332 when the log message emitted by code analysis is observed.

Suggestions

• Review all existing cases for areas of potential double substitution using human code review
• Use iHealth to determine if there are instances of K57410758 in system log files
• If you are managing BIG-IP with BIG-IQ, check for instances of D015332 found on managed devices using this procedure
• Use tooling to check for security problems. Note: the tooling suggestions below should be evaluated for appropriate use in your environment. As with any security testing tools the potential to cause harm exists so only test systems where you have explicit approval and proceed with caution
• Use the Visual Studio F5 Networks iRule Extension for general development
• Perform static analysis. Two useful tools are tclscanner and the recently released irulescan
• Use unit testing frameworks such as TesTCL for general code coverage
  
• Test live iRules with BURP Suite, using the BURP iRule Detector Extension

Unconstrained use of input

Lack of filtering or sanitizing input is a close second to double substitution, especially when user input is passed to specific commands in an iRule. Two particular commands that should be reviewed carefully are node and sideband.

• The node command "causes the specified server node (IP address and port number) to be used directly, thus bypassing any load-balancing".
• The sideband command "can create a connection (TCP or UDP) to any outside resource you choose, send a custom formatted request, await a response if applicable, act on that response, etc."

If an iRule takes input from an untrusted source and passes it directly to the aforementioned commands, an attacker may be able to connect to arbitrary systems defined in a request. This may be intended functionality, but when it's not intended becomes Server Side Request Forgery which results in unexpected network connections to systems reachable by the BIG-IP. This includes directly passing an attacker controlled value to the commands, but an overlooked area is passing in responses from DNS lookups. If a DNS lookup is recursed to an attacker controlled server, the attacker controls the address in the response and therefore controls the destination of the connection by the iRule.

Suggestions

• Segment and/or filter BIG-IP originating egress traffic
• Only pass values to the node and sideband commands from trusted sources
• Verify the address passed into the node command is not controlled by untrusted entities
• Use server pools or a data group for acceptable destinations
• If using DNS to resolve node names check that only trusted DNS infrastructure is used. You probably will want to restrict/disable query recursion to avoid DNS recursion to arbitrary sources

Logging

Logging is highly dependent on the business, so my suggestion is to gather a very clear understanding of the necessary business requirements along with any constraints that you need to consider. I'm reminded of the nursery rhyme "Goldilocks and the Three Bears" where finding the "just right" amount of logging is oftentimes discovered after iterative development to identify what works best for the application. As with many things, the desirable optimizations will often come with tradeoffs and risks that need to be balanced appropriately. These decisions don't need to be iRules specific so I will suggest some general tradeoffs I've encountered.

Logging too much or logging "everything"

• System resources are used when logging, so oversubscribed systems may have trouble servicing clients with extensive logging. This is especially true with "on-box" logging, where messages are all written to disk
• Information overload for defenders. Too much data to sift through makes human review difficult
• Attackers can sometimes use the noise to blend in, making attacks less likely to be noticed
• Secrets and/or Personally Identifiable Information may be logged, putting users and organizations at risk

Logging too little

• Lack of visibility into application use cases
• Hard to debug applications
• Missing important events and/or not meeting business logging requirements

Logging "just right"

• Proper balance of resource use and business requirements
• Ability to enable/disable debugging statements and iterate quickly on changes in development
• Avoids accidental logging of sensitive data

Suggestions

• Be precise with logging requirements that match the business needs. Define the fields which need to be restricted from logging and implement accordingly. Test any assumptions made in this step are correct
• Use High Speed Logging (HSL) to reduce the overall system burden
• Use SIEM/SOAR systems for alerting, event triggers, data filtering and general analytic capabilities. Systems which utilize Machine Learning help identify anomalies and outliers out of seas of data
• For debugging and testing, setup an identical Virtual Server to match traffic only from a specific source  using the "Source Address" configuration on a Virtual Server. This setup enables testing from a specific host to occur without disrupting production traffic
• In the cases where the iRule sends logs to a remote system, beware of logging infrastructure that is still vulnerable to Log4Shell

Known Vulnerabilities

Last but not least - there are known iRule vulnerabilities which range in severity and impact. Reviewing security advisories so you can make an appropriate risk assessment, along with implementing mitigations and/or fixes are important steps in preventing exploitation.

Suggestions

• Stay informed of the latest issues impacting F5 products and services by following our Quarterly Security Notifications
• Subscribe to email notifications regarding F5 products, specifically we will announce vulnerabilities to the "F5 Security Announcements" email list.
• Regularly use iHealth to assess current status of a BIG-IP system
• Regularly perform software updates

 

Conclusion

iRules are a powerful tool that enables business value but come with some risk. I hope that by following the suggestions here, you are more adept at recognizing risky patterns and avoiding potential pitfalls encountered during the iRules software development lifecycle. Thanks for reading, hope this helps.

[JRahm]

This is excellent, @Jordan_Zebor, thanks for sharing!