iRules Event Order
iRules is an event driven language. This has been discussed before, but to recap briefly, here’s what “Event Driven” means in iRules terms: iRules code is written inside event blocks. Those event blocks translate to times within the life-cycle of a connection to your LTM. By putting your code inside a given event block you are ensuring that it executes at a particular point in that transaction. I.E. HTTP_REQUEST code is executed once a request has been received, not before, so you’re safe to look for HTTP header info. SERVER_CONNECTED however isn’t fired until all the client side processing has been done and a load balancing decision is made, but before the actual data is sent to the chosen system. In this way we ensure granularity in the control of when and how your code is executed to allow you the absolute most control possible. It also allows for things to be highly efficient as we only have to execute the needed code each time, not the entire script. So now that we know what “Event Driven” means and how to use events to execute code precisely where we want to within the context of a connection, that leaves two burning questions. What events can I choose from? That is answered handily by the “Events” page in the iRules wiki. The other is: When does each event fire? I.E – what’s the overall order of events for a given connection? This has been something that’s a little harder to nail down as it takes a little bit of disclaimer work since not all events fire for every connection because some are based on given profiles, not to mention iRules is constantly changing. Keeping that in mind, though, I’ve put together a list of events in the order that they execute, checked it twice…then sent it off for PD approval (hey, I’m not an idiot). The below list has gotten the stamp of approval and I’m quite happy to finally get a documented order online as there have been many requests for this. Events are processed in a linear fashion so all CLIENT_ACCEPTED events in all iRules finish processing before the first HTTP_REQUEST event can fire. The linear order of this list is representative of that. Each event will not fire until all events above it have completed. Also, all events within one line, separated by slashes (/) execute at effectively the same time as far as the context of a connection is concerned. Please note that some of the events below are only executed under certain circumstances. For instance the AUTH_RESULT event only occurs if an AUTH command (such as AUTH::authenticate) completes to return results. iRules Event Order: RULE_INIT This is a global iRule event. It occurs once when the iRule is loaded into memory and does not continue to be processed for each subsequent request. Client Request Events - These events occur between the time the client connects to the LTM and the time the LTM makes a load balancing decision. At this point it has finished client data processing and is ready to connect to the chosen server but has not yet sent the data. These events are executed in a client-side context, meaning they behave as if you've used the clientside command with each of them. This means that local_addr type commands behave with the client in mind. This is where you have the opportunity to change the information a user is sending in before it makes it to the server for processing. URI re-writing, authentication and encryption offloading or inspection, server impersonation (acting like you’re a server to dig deeper into the transaction) – it all happens here. CLIENT_ACCEPTED CLIENTSSL_HANDSHAKE CLIENTSSL_CLIENTCERT HTTP_REQUEST / CACHE_REQUEST / RTSP_REQUEST / SIP_REQUEST / HTTP_CLASS_FAILED / HTTP_CLASS_SELECTED STREAM_MATCHED CACHE_UPDATE CLIENT_DATA / RTSP_REQUEST_DATA / HTTP_REQUEST_DATA – Only occur when collected data arrives AUTH_RESULT / AUTH_WANTCREDENTIAL – Only occur when authentication commands return LB_SELECTED / LB_FAILED / PERSIST_DOWN Server Request Events - These events take place after client request processing has finished, a destination has been chosen by the LTM, and the connection to the server is initiated. They happen before the client request is actually sent to the server. This is the end of the request process, which is followed by the first Server Response event. These events are executed in a server-side context. Much like the opposite of the above client-side events, these behave as if you’ve used the serverside command with each of them. SERVER_CONNECTED SERVER_SSL_HANDSHAKE HTTP_REQUEST_SEND / SIP_REQUEST_SEND Server Response Events - These events are triggered after the request has been sent to the server, the server has processed the request and the LTM receives a response from the server. These events are where you can access content generated by the server before it gets sent back to the client. Scrubbing out credit card information, server response manipulation, etc. – this is the place for it. CACHE_RESPONSE HTTP_RESPONSE / RTSP_RESPONSE / SIP_RESPONSE STREAM_MATCHED HTTP_RESPONSE_CONTINUE HTTP_RESPONSE_DATA / SERVER_DATA / RTSP_RESPONSE_DATA – Only occur when collected data arrives Disconnect Events - These are the events that represent the termination of a connection after all processing is completed. SERVER_CLOSED CLIENT_CLOSED This list is representative of iRules as they sit today. Keep in mind that the iRules language is continually evolving and changing, which may impact this list as well. I’ll do my best to keep it updated, because I think it’s a useful resource. Get the Flash Player to see this player. 20090331-iRulesEventOrder.mp3Accessing TCP Options from iRules
I’ve written several articles on the TCP profile and enjoy digging into TCP. It’s a beast, and I am constantly re-learning the inner workings. Still etched in my visual memory map, however, is the TCP header format, shown in Figure 1 below. Since 9.0 was released, TCP payload data (that which comes after the header) has been consumable in iRules via the TCP::payload and the port information has been available in the contextual commands TCP::local_port/TCP::remote_port and of course TCP::client_port/TCP::server_port. Options, however, have been inaccessible. But beginning with version 10.2.0-HF2, it is now possible to retrieve data from the options fields. Preparing the BIG-IP Prior to version 11.0, it was necessary to set a bigpipe database key with the option (or options) of interest: bigpipe db Rules.Tcpoption.settings [option, first|last], [option, first|last] In version 11.0 and forward, the DB keys are no more and you need to create a tcp profile with the these options defined, like so: ltm profile tcp tcp_opt { app-service none tcp-options "{option first|last} {option first|last}" } The option is an integer between 2 and 255, and the first/last setting indicates whether the system will retain the first or last instance of the specified option. Once that key is set, you’ll need to do a bigstart restart for it to take (warning: service impacting). Note also that the LTM only collects option data starting with the ACK of a connection. The initial SYN is ignored even if you select the first keyword. This is done to prevent a SYN flood attack (in keeping with SYN-cookies). A New iRules Command: TCP::option The TCP::option command has the following syntax: TCP::option get <option> v11 Additions/Changes: TCP::option set <option number> <value> <next|all> TCP::option noset <option number> Pretty simple, no? So now that you can access them, what fun can be had? Real World Scenario: Akamai In Akamai’s IPA and SXL product lines, they support client IP visibility by embedding a version number (one byte) and an IPv4 address (four bytes) as part of their overlay path feature in tcp option number 28. To access this data, we first set the database key: tmsh create ltm profile tcp tcp_opt tcp-options "{28 first}" Now, the iRule utilizing the TCP::option command: when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt28 cH8 ver addr if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { scan $addr "%2x%2x%2x%2x" ip1 ip2 ip3 ip4 set optaddr "$ip1.$ip2.$ip3.$ip4" } } } when HTTP_REQUEST { if { [info exists optaddr] } { HTTP::header insert "X-Forwarded-For" $optaddr } } The Akamai version should be one, so we log if not. Otherwise, we take the address (stored in the variable addr in hex) and scan it to get the decimal equivalents to build the address for inserting in the X-Forwarded-For header. Cool, right? Also cool—along with the new TCP::option command , an extension was made to the IP::addr command to parse binary fields into a dotted decimal IP address. This extension is also available beginning in 10.2.0-HF2, but extended in 11.0. Here’s the syntax: IP::addr parse [-ipv4 | -ipv6 [swap]] <binary field> [<offset>] So for example, if you had an IPv6 address in option 28 with a 1 byte offset, you would parse that like: log local0. "IP::addr parse IPv6 output: [IP::addr parse -ipv6 [TCP::option get 28] 1]" ## Log Result ## May 27 21:51:34 ltm13 info tmm[27207]: Rule /Common/tcpopt_test <CLIENT_ACCEPTED>: IP::addr parse IPv6 output: 2601:1930:bd51:a3e0:20cd:a50b:1cc1:ad13 But in the context of our TCP option, we have 5-bytes of data with the first byte not mattering in the context of an address, so we get at the address with this: set optaddr [IP::addr parse -ipv4 [TCP::option get 28] 1] This cleans up the rule a bit: when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt c ver if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { set optaddr [IP::addr parse -ipv4 $opt28 1] } } } when HTTP_REQUEST { if { [info exists optaddr] } { HTTP::header insert "X-Forwarded-For" $optaddr } } No need to store the address in the first binary scan and no need for the scan command at all so I eliminated those. Setting a forwarding header is not the only thing we can do with this data. It could also be shipped off to a logging server, or used as a snat address (assuming the server had either a default route to the BIG-IP, or specific routes for the customer destinations, which is doubtful). Logging is trivial, shown below with the log command. The HSL commands could be used in lieu of log if sending off-box to a log server. when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt c ver if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { set optaddr [IP::addr parse -ipv4 $opt28 1] log local0. "Client IP extracted from Akamai TCP option is $optaddr" } } } If setting the provided IP as a snat address, you’ll want to make sure it’s a valid IP address before doing so. You can use the TCL catch command and IP::addr to perform this check as seen in the iRule below: when CLIENT_ACCEPTED { set addrs [list \ "192.168.1.1" \ "256.168.1.1" \ "192.256.1.1" \ "192.168.256.1" \ "192.168.1.256" \ ] foreach x $addrs { if { [catch {IP::addr $x mask 255.255.255.255}] } { log local0. "IP $x is invalid" } else { log local0. "IP $x is valid" } } } The output of this iRule: <CLIENT_ACCEPTED>: IP 192.168.1.1 is valid <CLIENT_ACCEPTED>: IP 256.168.1.1 is invalid <CLIENT_ACCEPTED>: IP 192.256.1.1 is invalid <CLIENT_ACCEPTED>: IP 192.168.256.1 is invalid <CLIENT_ACCEPTED>: IP 192.168.1.256 is invalid Adding this logic into a functional rule with snat: when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt c ver if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { set optaddr [IP::addr parse -ipv4 $opt28 1] if { [catch {IP::addr $x mask 255.255.255.255}] } { log local0. "$optaddr is not a valid address" snat automap } else { log local0. "Akamai inserted Client IP is $optaddr. Setting as snat address." snat $optaddr } } } Alternative TCP Option Use Cases The Akamai solution shows an application implementation taking advantage of normally unused space in TCP headers. There are, however, defined uses for several option “kind” numbers. The list is available here: http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xml. Some options that might be useful in troubleshooting efforts: Opkind 2 – Max Segment Size Opkind 3 – Window Scaling Opkind 5 – Selective Acknowledgements Opkind 8 – Timestamps Of course, with tcpdump you get all this plus the context of other header information and data, but hey, another tool in the toolbox, right? Addendum I've been working with F5 SE Leonardo Simon on on additional examples I wanted to share here that uses option 28 or 253 to extract an IPv6 address if the version is 34 and otherwise extracts an IPv4 address if the version is 1 or 2. Option 28 when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] binary scan $opt28 c ver #log local0. "version: $ver" if { $ver == 34 } { set optaddr [IP::addr parse -ipv6 $opt28 1] log local0. "opt28 ipv6 address: $optaddr" } elseif { $ver == 1 || $ver == 2 } { set optaddr [IP::addr parse -ipv4 $opt28 1] log local0. "opt28 ipv4 address: $optaddr" } } Option 253 when CLIENT_ACCEPTED { set opt253 [TCP::option get 253] binary scan $opt253 c ver #log local0. "version: $ver" if { $ver == 34 } { set optaddr [IP::addr parse -ipv6 $opt253 1] log local0. "opt253 ipv6 address: $optaddr" } elseif { $ver == 1 || $ver == 2 } { set optaddr [IP::addr parse -ipv4 $opt253 1] log local0. "opt253 ipv4 address: $optaddr" } }Preventing Brute Force Password Guessing Attacks with APM - Part 1
F5er and DevCentral community member ystephie is back with another great solution (check out her first solution here: BIG-IP APM Customized Logon Page), this time tackling brute force attacks utilizing customizations with the BIG-IP Access Policy Manager. This solution requires BIG-IP10.2.2 Hotfix 1 or later. Introduction Exposing applications or services to the Internet opens inherent security risks. BIG-IP Access Policy Manager (APM) provides edge authentication and access control services for applications, BIG-IP Edge Gateway provides secure SSL VPN services, and BIG-IP Application Security Manager (ASM) provides protection against a variety of attacks. In this series of APM deployment examples, we will cover a couple of techniques for protecting against brute force password-guessing attacks. We’ll start with examples where a CAPTCHA challenge is used to block automated password guessing attacks, followed by an example providing temporary account lockout after a configured number of authentication failures. CAPTCHA stands for Completely Automated Public Turing Test to Tell Computers and Humans Apart (quite a mouthful), but basically consists of a challenge that a human can pass but a computer program cannot. It is used to protect against bots, and in the examples here can help protect against an automated password guessing attack. We can take advantage of Google’s reCAPTCHA web service and APM’s flexible advanced customization to provide basic defense against automated password guessing attacks. In addition, we will play around with the general look of your logon page. With reCAPTCHA available as a web service, we’ll be incorporating the CAPTCHA challenge within the APM logon page via advanced customization. The JavaScript added to the logon page will request a challenge (image with distorted text) from the reCAPTCHA web service and display it within the page. We’ll then create a custom APM Access Policy where we validate the user’s CAPTCHA challenge answer against the same reCAPTCHA web service (using the APM HTTP Auth agent). The links below describe the Google reCAPTCHA service in greater detail: http://code.google.com/apis/recaptcha/intro.html http://code.google.com/apis/recaptcha/docs/display.html http://code.google.com/apis/recaptcha/docs/verify.html Initial Setup – Create a Google Account for the reCAPTCHA Project Sign up for Google’s reCAPTCHA project through http://www.google.com/recaptcha/whyrecaptcha. Fill in a domain name and jot down the private and public keys for we’ll be using them later. Device Wizard For the purpose of this example, we’ll be using the Network Access Setup Wizard for Remote Access option under Templates and Wizards -> Device Wizards shown in Figure 1. Select HTTP Authentication with the following setup. This is required to verify the CAPTCHA challenge answer from the user against the reCAPTCHA web service. Follow the steps in the wizard (AAA Server, Lease Pool, Network Access, and etc.) to get to the summary page shown below in Figure 2. Before clicking finished, enter the Visual Policy Editor (VPE) to make a few changes. Click on Logon Page and modify field 3 and 4 under Logon Page Agent with the following configuration and save. Note: The logon page agent will only parse and store POST parameters it knows about (that are defined here). We’ll be hiding these two new fields on the logon page via advanced customization later. You should add an Ad Auth after the success leg of HTTP Auth. In my examples, I will be using ADAuth but feel free to use any sort of additional authenticate. That way, you only need to check their credentials once we know for sure that this is a human user (passes the CAPTCHA challenge). Update the access policy by clicking Apply Access Policy and finish the Device Wizard. Advanced Customization Follow steps 1-3 under the section “Customize the Logon Page” in the BIG-IP APM-Customized Logon Page article. We’ll be replacing the auto-generated logon form with HTML that includes the username and password fields, along with some JavaScript that calls the reCAPTCHA service and includes the challenge within the page. Edit logon_en.inc file: remove this block of PHP code: <? //------------------------------------------------------------ foreach( $fields_settings as $field_settings ) { if( $field_settings["type"] != "none" ) { if( $GLOBALS["label_position"] == "above" ){ ?> <tr> <td colspan=2 class="credentials_table_unified_cell" ><label for="<? print( $field_settings["type"] ); ?>"><? print( $field_settings["caption"] ); ?></label><input type=<? print( $field_settings["type"] ); ?> name=<? print( $field_settings["name"] ); ?> class="credentials_input_<? print( $field_settings["type"] ); ?>" <? print( ( $field_settings["rw"] == 0 ? "disabled" : "" ) ); ?> value="<? print( $field_settings["value"] ); ?>" autocomplete="off"></td> </tr> <? }else{ ?> <tr> <td class="credentials_table_label_cell" ><? print( $field_settings["caption"] ); ?></td> <td class="credentials_table_field_cell"><input type="<? print( $field_settings["type"] ); ?>" name="<? print( $field_settings["name"] ); ?>" class="credentials_input_<? print( $field_settings["type"] ); ?>" <? print( ( $field_settings["rw"] == 0 ? "disabled" : "" ) ); ?> value="<? print( $field_settings["value"] ); ?>" autocomplete="off"></td> </tr> <? } } } //------------------------------------------------------------ ?> In its place, paste this second block of code. Make sure to replace the red text with your own information. <tr> <td colspan=2 class="credentials_table_unified_cell" ><label for="text">Username</label><input type=text name=username s="credentials_input_text" value="" autocomplete="off" autocapitalize="off"></td> </tr> <tr> <td colspan=2 class="credentials_table_unified_cell" ><label for="password">Password</label><input type=password =password class="credentials_input_password" value="" autocomplete="off" autocapitalize="off"></td> </tr> d colspan=2 class="credentials_table_unified_cell"> cript type="text/javascript" src="https://www.google.com/recaptcha/api/challenge?k=replace_with_your_public_key"> script> oscript> <iframe src="https://www.google.com/recaptcha/api/noscript?k=replace_with_your_public_key" height="300" width="500" frameborder="0"></iframe><br> <textarea name="recaptcha_challenge_field" rows="3" cols="40"> </textarea> <input type="hidden" name="recaptcha_response_field" value="manual_challenge"> noscript> td> > Apply the customizations to the policy with the following commands: b customization group <your policy name>_act_logon_page_ag action update b profile access <your policy name> generation action increment Extra Touches Currently the page should like like Figure 3. But we can easily customize this page. First, let us start by adding a helpful message before the CAPTCHA. Open the logon_en.inc file again and add some HTML like below. Place it between the <td...> tag and <script…> tag we added earlier. <td colspan=2 class="credentials_table_unified_cell"> <label for="text">Security Check<p>Enter <b>both words</b> below, <b>separated by a space</b>.</p></label> <script type="text/javascript" Edit this message as fits your organization. Don’t forget to update the access policy! The page now looks like Figure 4: The color scheme of the CAPTCHA may not work for every page so Google provides a few more templates shown in Figure 5. If you feel that you would like to do more customization, see the documentation found on this page -http://code.google.com/apis/recaptcha/docs/customization.html. <script type="text/javascript"> var RecaptchaOptions = { theme : 'theme_name' }; </script> To display a standard theme, add the following script into logon_en.inc anywhere before the <form> element where we inserted our code. Replace ‘theme_name’ with one of the above theme names. In Figure 6, I’m using the ‘white’ theme. Remember to update! More Extra Touches – Changing the Look of Your Page To customize the look of your page, click on your access profile- Access Policy -> Access Profiles -> <your access policy> -> Customization (third tab from the left on the top) -> general UI -> Find Customization. Feel free to make whatever changes you like, in Figure 7, I changed the color of the Header background color, and Form background color to #63919E and #94BBC2 respectively. Advanced Customization (Logon Page) Checklist When you copy and paste to a template file, it has the following formatting: logon_<language>.inc Set permissions using the following command chmod a+r logon_<language>.inc After editing, update with the following two commands b customization group <your policy name>_act_logon_page_ag action update b profile access <your policy name> generation action increment Final Notes Exposing applications or services to the Internet opens inherent security risks. APM can help by providing advanced authentication, authorization, and endpoint security checks. With a bit of customization you can integrate with web services such as the Google reCAPTCHA project to provide additional security layers. In our next example, we’ll build on this work to display the CAPTCHA only after the user has failed full authentication, to reduce the inconvenience of typing CATPCHA challenges. We’ll be demonstrating how to do an AD Query and use the bad password count attribute to determine when to show the CAPTCHA challenge. About the Author Stephanie is a summer intern at F5, heading back to school soon to continue her EECS degree at UC Berkeley, and has been having a blast creating interesting solutions for BIG-IP. Stephanie’s passion for engineering, and smile, is contagious.These Are Not The Scrapes You're Looking For - Session Anomalies
In my first article in this series, I discussed web scraping -- what it is, why people do it, and why it could be harmful. My second article outlined the details of bot detection and how the ASM blocks against these pesky little creatures. This last article in the series of web scraping will focus on the final part of the ASM defense against web scraping: session opening anomalies and session transaction anomalies. These two detection modes are new in v11.3, so if you're using v11.2 or earlier, then you should upgrade and take advantage of these great new features! ASM Configuration In case you missed it in the bot detection article, here's a quick screenshot that shows the location and settings of the Session Opening and Session Transactions Anomaly in the ASM. You'll find all the fun when you navigate to Security > Application Security > Anomaly Detection > Web Scraping. There are three different settings in the ASM for Session Anomaly: Off, Alarm, and Alarm and Block. (Note: these settings are configured independently...they don't have to be set at the same value) Obviously, if Session Anomaly is set to "Off" then the ASM does not check for anomalies at all. The "Alarm" setting will detect anomalies and record attack data, but it will allow the client to continue accessing the website. The "Alarm and Block" setting will detect anomalies, record the attack data, and block the suspicious requests. Session Opening Anomaly The first detection and prevention mode we'll discuss is Session Opening Anomaly. But before we get too deep into this, let's review what a session is. From a simple perspective, a session begins when a client visits a website, and it ends when the client leaves the site (or the client exceeds the session timeout value). Most clients will visit a website, surf around some links on the site, find the information they need, and then leave. When clients don't follow a typical browsing pattern, it makes you wonder what they are up to and if they are one of the bad guys trying to scrape your site. That's where Session Opening Anomaly defense comes in! Session Opening Anomaly defense checks for lots of abnormal activities like clients that don't accept cookies or process JavaScript, clients that don't scrape by surfing internal links in the application, and clients that create a one-time session for each resource they consume. These one-time sessions lead scrapers to open a large number of new sessions in order to complete their job quickly. What's Considered A New Session? Since we are discussing session anomalies, I figured we should spend a few sentences on describing how the ASM differentiates between a new or ongoing session for each client request. Each new client is assigned a "TS cookie" and this cookie is used by the ASM to identify future requests from the client with a known, ongoing session. If the ASM receives a client request and the request does not contain a TS cookie, then the ASM knows the request is for a new session. This will prove very important when calculating the values needed to determine whether or not a client is scraping your site. Detection There are two different methods used by the ASM to detect these anomalies. The first method compares a calculated value to a predetermined ceiling value for newly opened sessions. The second method considers the rate of increase of newly opened sessions. We'll dig into all that in just a minute. But first, let's look at the criteria used for detecting these anomalies. As you can see from the screenshot above, there are three detection criteria the ASM uses...they are: Sessions opened per second increased by: This specifies that the ASM considers client traffic to be an attack if the number of sessions opened per second increases by a given percentage. The default setting is 500 percent. Sessions opened per second reached: This specifies that the ASM considers client traffic to be an attack if the number of sessions opened per second is greater than or equal to this number. The default value is 400 sessions opened per second. Minimum sessions opened per second threshold for detection: This specifies that the ASM considers traffic to be an attack if the number of sessions opened per second is greater than or equal to the number specified. In addition, at least one of the "Sessions opened per second increased by" or "Sessions opened per second reached" numbers must also be reached. If the number of sessions opened per second is lower than the specified number, the ASM does not consider this traffic to be an attack even if one of the "Sessions opened per second increased by" or "Sessions opened per second" reached numbers was reached. The default value for this setting is 200 sessions opened per second. In addition, the ASM maintains two variables for each client IP address: a one-minute running average of new session opening rate, and a one-hour running average of new session opening rate. Both of these variables are recalculated every second. Now that we have all the basic building blocks. let's look at how the ASM determines if a client is scraping your site. First Method: Predefined Ceiling Value This method uses the user-defined "minimum sessions opened per second threshold for detection" value and compares it to the one-minute running average. If the one-minute average is less than this number, then nothing else happens because the minimum threshold has not been met. But, if the one-minute average is higher than this number, the ASM goes on to compare the one-minute average to the user-defined "sessions opened per second reached" value. If the one-minute average is less than this value, nothing happens. But, if the one-minute average is higher than this value, the ASM will declare the client a web scraper. The following flowchart provides a pictorial representation of this process. Second Method: Rate of Increase The second detection method uses several variables to compare the rate of increase of newly opened sessions against user-defined variables. Like the first method, this method first checks to make sure the minimum sessions opened per second threshold is met before doing anything else. If the minimum threshold has been met, the ASM will perform a few more calculations to determine if the client is a web scraper or not. The "sessions opened per second increased by" value (percentage) is multiplied by the one-hour running average and this value is compared to the one-minute running average. If the one-minute average is greater, then the ASM declares the client a web scraper. If the one-minute average is lower, then nothing happens. The following matrix shows a few examples of this detection method. Keep in mind that the one-minute and one-hour averages are recalculated every second, so these values will be very dynamic. Prevention The ASM provides several policies to prevent session opening anomalies. It begins with the first method that you enable in this list. If the system finds this method not effective enough to stop the attack, it uses the next method that you enable in this list. The following screenshots show the different options available for prevention. The "Drop IP Addresses with bad reputation" is tied to Rate Limiting, so it will not appear as an option unless you enable Rate Limiting. Note that IP Address Intelligence must be licensed and enabled. This feature is licensed separately from the other ASM web scraping options. Here's a quick breakdown of what each of these prevention policies do for you: Client Side Integrity Defense: The system determines whether the client is a legal browser or an illegal script by sending a JavaScript challenge to each new session request from the detected IP address, and waiting for a response. The JavaScript challenge will typically involve some sort of computational challenge. Legal browsers will respond with a TS cookie while illegal scripts will not. The default for this feature is disabled. Rate Limiting: The goal of Rate Limiting is to keep the volume of new sessions at a "non-attack" level. The system will drop sessions from suspicious IP addresses after the system determines that the client is an illegal script. The default for this feature is also disabled. Drop IP Addresses with bad reputation: The system drops requests from IP addresses that have a bad reputation according to the system’s IP Address Intelligence database (shown above). The ASM will drop all request from any "bad" IP addresses even if they respond with a TS cookie. IP addresses that do not have a bad reputation also undergo rate limiting. The default for this option is disabled. Keep in mind that this option is available only after Rate Limiting is enabled. In addition, this option is only enforced if at least one of the IP Address Intelligence Categories is set to Alarm mode. Prevention Duration Now that we have detected session opening anomalies and mitigated them using our prevention options, we must figure out how long to apply the prevention measures. This is where the Prevention Duration comes in. This setting specifies the length of time that the system will prevent an attack. The system prevents attacks by rejecting requests from the attacking IP address. There are two settings for Prevention Duration: Unlimited: This specifies that after the system detects and stops an attack, it performs attack prevention until it detects the end of the attack. This is the default setting. Maximum <number of> seconds: This specifies that after the system detects and stops an attack, it performs attack prevention for the amount of time indicated unless the system detects the end of the attack earlier. So, to finish up our Session Opening Anomaly part of this article, I wanted to share a quick scenario. I was recently reading several articles from some of the web scrapers around the block, and I found one guy's solution to work around web scraping defense. Here's what he said: "Since the service conducted rate-limiting based on IP address, my solution was to put the code that hit their service into some client-side JavaScript, and then send the results back to my server from each of the clients. This way, the requests would appear to come from thousands of different places, since each client would presumably have their own unique IP address, and none of them would individually be going over the rate limit." This guy is really smart! And, this would work great against a web scraping defense that only offered a Rate Limiting feature. Here's the pop quiz question: If a user were to deploy this same tactic against the ASM, what would you do to catch this guy? I'm thinking you would need to set your minimum threshold at an appropriate level (this will ensure the ASM kicks into gear when all these sessions are opened) and then the "sessions opened per second" or the "sessions opened per second increased by" should take care of the rest for you. As always, it's important to learn what each setting does and then test it on your own environment for a period of time to ensure you have everything tuned correctly. And, don't forget to revisit your settings from time to time...you will probably need to change them as your network environment changes. Session Transactions Anomaly The second detection and prevention mode is Session Transactions Anomaly. This mode specifies how the ASM reacts when it detects a large number of transactions per session as well as a large increase of session transactions. Keep in mind that web scrapers are designed to extract content from your website as quickly and efficiently as possible. So, web scrapers normally perform many more transactions than a typical application client. Even if a web scraper found a way around all the other defenses we've discussed, the Session Transaction Anomaly defense should be able to catch it based on the sheer number of transactions it performs during a given session. The ASM detects this activity by counting the number of transactions per session and comparing that number to a total average of transactions from all sessions. The following screenshot shows the detection and prevention criteria for Session Transactions Anomaly. Detection How does the ASM detect all this bad behavior? Well, since it's trying to find clients that surf your site much more than other clients, it tracks the number of transactions per client session (note: the ASM will drop a session from the table if no transactions are performed for 15 minutes). It also tracks the average number of transactions for all current sessions (note: the ASM calculates the average transaction value every minute). It can use these two figures to compare a specific client session to a reasonable baseline and figure out if the client is performing too many transactions. The ASM can automatically figure out the number of transactions per client, but it needs some user-defined thresholds to conduct the appropriate comparisons. These thresholds are as follows: Session transactions increased by: This specifies that the system considers traffic to be an attack if the number of transactions per session increased by the percentage listed. The default setting is 500 percent. Session transactions reached: This specifies that the system considers traffic to be an attack if the number of transactions per session is equal to or greater than this number. The default value is 400 transactions. Minimum session transactions threshold for detection: This specifies that the system considers traffic to be an attack if the number of transactions per session is equal to or greater than this number, and at least one of the "Sessions transactions increased by" or "Session transactions reached" numbers was reached. If the number of transactions per session is lower than this number, the system does not consider this traffic to be an attack even if one of the "Session transactions increased by" or "Session transaction reached" numbers was reached. The default value is 200 transactions. The following table shows an example of how the ASM calculates transaction values (averages and individual sessions). We would expect that a given client session would perform about the same number of transactions as the overall average number of transactions per session. But, if one of the sessions is performing a significantly higher number of transactions than the average, then we start to get suspicious. You can see that session 1 and session 3 have transaction values higher than the average, but that only tells part of the story. We need to consider a few more things before we decide if this client is a web scraper or not. By the way, if the ASM knows that a given session is malicious, it does not use that session's transaction numbers when it calculates the average. Now, let's roll in the threshold values that we discussed above. If the ASM is going to declare a client as a web scraper using the session transaction anomaly defense, the session transactions must first reach the minimum threshold. Using our default minimum threshold value of 200, the only session that exceeded the minimum threshold is session 3 (250 > 200). All other sessions look good so far...keep in mind that these numbers will change as the client performs additional transactions during the session, so more sessions may be considered as their transaction numbers increase. Since we have our eye on session 3 at this point, it's time to look at our two methods of detecting an attack. The first detection method is a simple comparison of the total session transaction value to our user-defined "session transactions reached" threshold. If the total session transactions is larger than the threshold, the ASM will declare the client a web scraper. Our example would look like this: Is session 3 transaction value > threshold value (250 > 400)? No, so the ASM does not declare this client as a web scraper. The second detection method uses the "transactions increased by" value along with the average transaction value for all sessions. The ASM multiplies the average transaction value with the "transactions increased by" percentage to calculate the value needed for comparison. Our example would look like this: 90 * 500% = 450 transactions Is session 3 transaction value > result (250 > 450)? No, so the ASM does not declare this client as a web scraper. By the way, only one of these detection methods needs to be met for the ASM to declare the client as a web scraper. You should be able to see how the user-defined thresholds are used in these calculations and comparisons. So, it's important to raise or lower these values as you need for your environment. Prevention Duration In order to save you a bunch of time reading about prevention duration, I'll just say that the Session Transactions Anomaly prevention duration works the same as the Session Opening Anomaly prevention duration (Unlimited vs Maximum <number of> seconds). See, that was easy! Conclusion Thanks for spending some time reading about session anomalies and web scraping defense. The ASM does a great job of detecting and preventing web scrapers from taking your valuable information. One more thing...for an informative anomaly discussion on the DevCentral Security Forum, check out this conversation. If you have any questions about web scraping or ASM configurations, let me know...you can fill out the comment section below or you can contact the DevCentral team at https://devcentral.f5.com/s/community/contact-us.iRules IP Comparison Considerations with IP::addr Command
Anyone utilizing IP network comparisons in iRules is probably familiar with this syntax: if { [IP::addr [IP::client_addr]/24 equals 10.10.20.0] } { # Do this } In fact, there are several methods for doing a comparison. Here are three functional equivalents that include the most common form shown above: [IP::addr [IP::remote_addr]/24 equals 10.10.20.0] [IP::addr [IP::remote_addr]/255.255.255.0 equals 10.10.20.0] [IP::addr "[IP::remote_addr] mask 255.255.255.0" equals 10.10.20.0] All three work, returning true if there is match and false if not. These formats, however, are not as ideal as it was never intended to work this way. What occurs when performing the comparison this way is the system has to convert the internal IP address to string form, apply the network mask, then re-convert the result back into an IP network object for comparison to the last argument. While possible, it isn’t as efficient and technically is an oversight in the syntax validation checking. It’s not slated to be “fixed” at this point, but it could be in the future, so you should consider updating any iRules to one of these formats: [IP::addr [IP::remote_addr] equals 10.10.20.0/24] [IP::addr [IP::remote_addr] equals 10.10.20.0/255.255.255.0] [IP::addr [IP::remote_addr] equals "10.10.20.0 mask 255.255.255.0"] In these formats, the input address does not need to be masked and can be directly compared with the pre-parsed network specification. Finally, there is one more format that works: [IP::addr [IP::addr [IP::remote_addr] mask 255.255.255.0] equals 10.10.20.0] This also doesn’t require any additional conversion, but the sheer volume of commands in that statement is an immediate indicator (to me, anyway) that it’s probably not very efficient. Performance Before running each format through a simple test (ab –n 10000 –c 25 http://<vip>), I set up a control so I could isolate the cycles for each format: when HTTP_REQUEST timing on { ### CONTROL ### if { 1 } { } } Since all the formats will return true if the match is found, then subtracting out the average cycles from this iRule should give me a pretty accurate accounting of cycles required for each specific format. So I can replace the “1” from the conditional with each format, as shown in this iRule: when HTTP_REQUEST timing on { ### FUNCTIONAL EQUIVALENTS "RECOMMENDED" ### # Format #1 Cycles: 6839 - 1136 = 5703 # if { [IP::addr [IP::remote_addr] equals 10.10.20.0/24] } { } # Format #2 Cycles: 6903 - 1136 = 5767 # if { [IP::addr [IP::remote_addr] equals 10.10.20.0/255.255.255.0] } { } # Format #3 Cycles: 7290 - 1136 = 6154 # if { [IP::addr [IP::remote_addr] equals "10.10.20.0 mask 255.255.255.0"] } { } ### FUNCTIONAL EQUIVALENTS "NOT RECOMMENDED" ### # Format #4 Cycles: 8500 - 1136 = 7364 # if { [IP::addr [IP::remote_addr]/24 equals 10.10.20.0] } { } # Format #5 Cycles: 8543 - 1136 = 7407 # if { [IP::addr [IP::remote_addr]/255.255.255.0 equals 10.10.20.0] } { } # Format #6 Cycles: 8827 - 1136 = 7691 # if { [IP::addr "[IP::remote_addr] mask 255.255.255.0" equals 10.10.20.0] } { } ### ALTERNATE FORMAT ### # Format #7 Cycles: 9124 - 1136 = 7988 # if { [IP::addr [IP::addr [IP::remote_addr] mask 255.255.255.0] equals 10.10.20.0] } { } ### CONTROL ### # Cycles: 1136 # if { 1 } { } } You can see from the average cycles data I added to each of the formats comments above that the recommended formats are all faster than the formats that are not recommended. What’s interesting is the subtle differences in the “/” formats within each group of functional equivalents and then the significant outliers that the “<net> mask <mask>” formats are within their group. Also of note is that the last format, while acceptable, is really inefficient and should probably be avoided. The table below breaks down the increase in cycles for each of the formats compared to the best performer:[IP::addr [IP::remote_addr] equals 10.10.20.0/24]. Whereas the number of cycles required to execute this operation are all quite small, the difference beyond the first two similar formats is quite significant. Conclusion Many ways to skin a cat, some good, some not so good. Just to be clear, using the mask on the initial argument of the comparison should be avoided, and if you currently have iRules utilizing this format, it would be best to update them sooner rather than later. I’ll be doing the same on all the DevCentral wikis, forums, articles, and blogs. If you find some references where we haven’t made this distinction, please comment below. Related Articles IP::addr and IPv6 Why do we need to use [IP::addr [IP::client_addr]]?BIG-IP APM - Customized Logon Page
Note: This solution is only applicable to version 10.x, version 11 doesn't use advCustHelp. The default logon page for the Access Policy Manager module is pretty basic, particularly so if only the minimal username and password is configured. However, APM is wildly flexible. In this tech tip, I’ll cover customizing the logon page by adding a dropdown box in addition to the standard username and password fields. Introduction Background Information The goal here is to provide access to multiple web applications behind APM through the use of an admin-defined dropdown menu and different LTM pools for each web application. We will be generating the list dynamically through the use of data groups so there will be no need to manually edit the iRule code each time an admin decides to add another option. Solution Overview Combining advanced customization, data groups, and iRules, we can dynamically generate html code for each key value pair in the data group. We simply add a session variable in the logon page through advanced customization and insert our html code, generated with iRules, through the session variable. The data group serves as a user friendly way of adding more applications as a layer of indirection. Create the Access Policy Before creating the custom logon page, an access policy needs to be defined. This can be done by utilizing the Device Wizards option under the Templates and Wizards main tab. Select Web Application Access Management for Local Traffic Virtual Servers as shown below in Figure 1. Follow the steps in the wizard (AAA server, virtual IP, pool info, etc) to get to the summary page shown below in Figure 2. Before clicking finished, enter the Visual Policy Editor to make a few changes. Immediately after the start box in the VPE, add an iRule Event as show in Figure 3. This will trigger the iRule event ACCESS_POLICY_AGENT_EVENT in the iRule featured later in this article. Make sure to assign an ID to the event before saving it (it's significant only in differentiating the use of multiple iRule events). Next, click on the Logon Page event in the VPE and add a text field immediately after the password field. Set the Post Variable Name and the Session Variable Name to ‘appname’. (The name is not significant but will need to match a statement in the HTML that will be replaced later in the article. This ensures that the logon page agent will know to expect appname as one of the POST parameters.) In the Logon Page Input Field #3 box, enter ‘Application’ and click save. See Figure 4 below for details. Note: The logon page agent will only parse and store POST parameters it knows about. Also, the 'Application' entry in Logon Page Input Field #3 is entered for the sake of completeness, but is technically unnecessary as the custom HTML will override this. Finally, update the access policy by clicking Apply Access Policy and finish the Device Wizard. Customize the Logon Page Now that the policy is created, login via ssh to the command line interface to complete several steps. 1. Change Directory into the specific policy’s logon directory: cd /config/customization/advanced/logon/<policy_name>_act_logon_page_ag (where <policy_name> is your policy name. devCenEx_act_logon_page_ag in this case.) 2. Make a copy of the tmp_logon_en.inc file (the name logon_en.inc is significant.) cp tmp_logon_en.inc logon_en.inc 3. Add group and world read permissions to the logon_en.inc file chmod a+r logon_en.inc 4. Edit and save the logon_en.inc file, replacing this PHP code with the HTML code below it. Notice that the label and select tags reference Applications and appname (respectively) from our Logon Page in Figure 4. PHP (remove) <? //------------------------------------------------------------ foreach( $fields_settings as $field_settings ) { if( $field_settings["type"] != "none" ) { if( $GLOBALS["label_position"] == "above" ){ ?> <tr> <td colspan=2 class="credentials_table_unified_cell" ><label for="<? print( $field_settings["type"] ); ?>"><? print( $field_settings["caption"] ); ?></label><input type=<? print( $field_settings["type"] ); ?> name=<? print( $field_settings["name"] ); ?> class="credentials_input_<? print( $field_settings["type"] ); ?>" <? print( ( $field_settings["rw"] == 0 ? "disabled" : "" ) ); ?> value="<? print( $field_settings["value"] ); ?>" autocomplete="off"></td> </tr> <? }else{ ?> <tr> <td class="credentials_table_label_cell" ><? print( $field_settings["caption"] ); ?></td> <td class="credentials_table_field_cell"><input type="<? print( $field_settings["type"] ); ?>" name="<? print( $field_settings["name"] ); ?>" class="credentials_input_<? print( $field_settings["type"] ); ?>" <? print( ( $field_settings["rw"] == 0 ? "disabled" : "" ) ); ?> value="<? print( $field_settings["value"] ); ?>" autocomplete="off"></td> </tr> <? } } } //------------------------------------------------------------ ?> Custom HTML (add) <tr> <td colspan=2 class="credentials_table_unified_cell" ><label for="text">Username</label> <input type=text name=username class="credentials_input_text" value="" autocomplete="off" autocapitalize="off"></td> </tr> <tr> <td colspan=2 class="credentials_table_unified_cell" ><label for="password">Password</label> <input type=password name=password class="credentials_input_password" value="" autocomplete="off" autocapitalize="off"></td> </tr> <tr> <td colspan=2 class="credentials_table_unified_cell" ><label for="text">Applications</label> <select name="appname"> %{session.custom.logon_opt} </select> </td> </tr> Note: The HTML above can be further customized if desired. The important section of code is: <select name>=”appname”> %{session.custom.logon_opt} </select> The POST variable was created in the Logon page and the session variable will be utilized in the iRule. UPDATE:As an alternative to replacing the php code with html as shown above, the javascript below can be added to the javascript section of the same file: function dropdown(){ var allTDs = document.getElementsByTagName('td'); for(i=0; i < allTDs.length; i++) { if(allTDs[i].innerHTML.indexOf('appname') > 0 && allTDs[i].innerHTML.indexOf('auth_form') == -1) { var replacetext = '<label for="text">Application</label><select name="appname" value="" autocomplete="off">%{session.custom.logon_opt}</select>'; allTDs[i].innerHTML = replacetext; } } } To ensure this javascript is run, insert the dropdown() function shown above in line 1 of the OnLoad() function defined in the same file. 5. Apply the customizations to the policy advCustHelp <your policy name> (The following two commands are included with instructions on updated the access policy when this command is run.) b customization group <your policy_name>_act_logon_page_ag action update b profile access <your policy name> generation action increment 6. Create a string class for the applications. The string should be the name of the application to be displayed in the Logon Page and the value should be the pool that hosts the applicable service. The resulting configuration of the class is shown entered below via bigpipe then via tmsh (See Figure 5 immediatedly after for the GUI entry for the class.) b class ExampleDataGroup '{ { "OWA" { "intranet-pool" } "Share Point" { "sharepoint-pool" } } }' tmsh create /ltm data-group ExampleDataGroup type string records add { "OWA" { data "intranet-pool" } "Share Point" { data "sharepoint-pool" } } 7. Create the iRule (via iRule editor, GUI, or tmsh) when ACCESS_POLICY_AGENT_EVENT { if {[ACCESS::policy agent_id] eq "GenHtml"} { set htmlstr "" # Pull data from the data group set keys [class names ExampleDataGroup] # log local0. "DATA GROUP:: $keys" foreach key $keys { # log local0. "KEY:: $key" # Add a new option in the drop down box for each key append htmlstr "<option value=\"$key\">$key</option>" } # log local0. "HTML STRING:: $htmlstr" # Using the session variable we inserted through advanced customization, # we can insert the html code we generated above ACCESS::session data set "session.custom.logon_opt" $htmlstr } } when ACCESS_ACL_ALLOWED { set appname [ACCESS::session data get "session.logon.last.appname"] # log local0. "appname:: $appname" set value [class search -value ExampleDataGroup equals $appname] # log local0. "value:: $value" if {[string length $value] > 0} { pool $value # log local0. "POOL:: $value" } } Note that the class name referenced in the iRule should match the actual class name (instead of ExampleDataGroup as shown above) 8. Finally, apply the iRule to the APM virtual server and the process is complete! tmsh modify /ltm virtual custom_dropdown_vs rules { custom_dropdown_iRule } Figure 6 below shows the resulting logon page from the customizations performed. This is just the surface of what can be done to customize the logon page. Many thanks to F5er ystephie for writing up the documentation steps for this solution. Related Articles DevCentral Wiki: BIG-IP Access Policy Manager (APM) Wiki Home Auto-launch Remote Desktop Sessions with APM > DevCentral > F5 ... NTLM/ Outlook Anywhere/ Big-IP APM - DevCentral - F5 DevCentral ... Web Application Login Integration with APM > DevCentral > F5 ... F5 Tutorial: BIG-IP APM with SecureAuth Set APM Cookies to HttpOnly - DevCentral - F5 DevCentral ... DevCentral Wiki: APM nested virtuals with APM - DevCentral - F5 DevCentral > Community ... Pete Silva - apm BigIp_VE_10.1.0 — APM module and Hypervisor Support - DevCentral ...IP::addr and IPv6
Did you know that all address internal to tmm are kept in IPv6 format? If you’ve written external monitors, I’m guessing you knew this. In the external monitors, for IPv4 networks the IPv6 “header” is removed with the line: IP=`echo $1 | sed 's/::ffff://'` IPv4 address are stored in what’s called “IPv4-mapped” format. An IPv4-mapped address has its first 80 bits set to zero and the next 16 set to one, followed by the 32 bits of the IPv4 address. The prefix looks like this: 0000:0000:0000:0000:0000:ffff: (abbreviated as ::ffff:, which looks strickingly simliar—ok, identical—to the pattern stripped above) Notation of the IPv4 section of the IPv4-formatted address vary in implementations between ::ffff:192.168.1.1 and ::ffff:c0a8:c8c8, but only the latter notation (in hex) is supported. If you need the decimal version, you can extract it like so: % puts $x ::ffff:c0a8:c8c8 % if { [string range $x 0 6] == "::ffff:" } { scan [string range $x 7 end] "%2x%2x:%2x%2x" ip1 ip2 ip3 ip4 set ipv4addr "$ip1.$ip2.$ip3.$ip4" } 192.168.200.200 Address Comparisons The text format is not what controls whether the IP::addr command (nor the class command) does an IPv4 or IPv6 comparison. Whether or not the IP address is IPv4-mapped is what controls the comparison. The text format merely controls how the text is then translated into the internal IPv6 format (ie: whether it becomes a IPv4-mapped address or not). Normally, this is not an issue, however, if you are trying to compare an IPv6 address against an IPv4 address, then you really need to understand this mapping business. Also, it is not recommended to use 0.0.0.0/0.0.0.0 for testing whether something is IPv4 versus IPv6 as that is not really valid a IP address—using the 0.0.0.0 mask (technically the same as /0) is a loophole and ultimately, what you are doing is loading the equivalent form of a IPv4-mapped mask. Rather, you should just use the following to test whether it is an IPv4-mapped address: if { [IP::addr $IP1 equals ::ffff:0000:0000/96] } { log local0. “Yep, that’s an IPv4 address” } These notes are covered in the IP::addr wiki entry. Any updates to the command and/or supporting notes will exist there, so keep the links handy. Related Articles F5 Friday: 'IPv4 and IPv6 Can Coexist' or 'How to eat your cake ... Service Provider Series: Managing the ipv6 Migration IPv6 and the End of the World No More IPv4. You do have your IPv6 plan running now, right ... Question about IPv6 - BIGIP - DevCentral - F5 DevCentral ... Insert IPv6 address into header - DevCentral - F5 DevCentral ... Business Case for IPv6 - DevCentral - F5 DevCentral > Community ... We're sorry. The IPv4 address you are trying to reach has been ... Don MacVittie - F5 BIG-IP IPv6 Gateway Module
My Buddy Byron: Security iRules
So, I was having lunch with my buddy Byron, who is a fantabulous Sales Engineer here at F5. I know, the world sales sends shivers down my spine to, but trust me, he’s a sharp engineer. We do what most co-workers do at lunch, complain about the weather, the Seahawks, and of course… work. I forget how we got on the topic, but somehow we ended up in a rousing discussion around iRules and their usefulness. Both of us had gotten multiple requests around “Hey, what can iRules do to make me safer”? Luckily for us, Byron being the driven guy he is, took the time to create a fact sheet completely around security iRules. (Disclaimer: This has been slightly modified from original content) Fact Sheet: Security iRules The iRule The Solution Transparent Web App Bot Protection Block illegitimate requests from automated bots that bombard a contact form Distributed Apache Killer Deny application requests that cause a Web server Denial of Service (DoS) DNS Blackhole with iRules Prevent employees from accessing known bad websites at the DNS level Thwart Dictionary Attacks Restrict excessive login attempts using the Exponential Backoff algorithm SSL Renegotiation DOS attack Drop connections that renegotiate SSL sessions more than 5 times a minute So, what you have here are 5 iRules that in 5 minutes can help improve your security posture. Transparent Web Application Bot Protection: This iRule is rather cool alternative to CAPTCHA’s that can work well in situations. It also includes an admin interface iRule. That’s pretty shiny Distributed Apache Killer: We security folks are so lucky. It seems like every other month or so, another “Killer” comes out. The loic, hoic, D-A-K, all as deadly as a Drink Delivering Dalek DNS Blackhole: DNS is all the rage on the interwebs. Dan Kaminski has consistently preached on the importantly of DNS (much tech love/thank to Dan for his work), and rightfully so. The DNS blackhole iRule allows an admin to intercept requests to DNS and respond back with a DNS blocking page. Great application: Known malware sites, HR policy blocks, and of course world of warcraft. It doesn’t stop users from surfing via IP though (keep in mind) Thwart Dictionary Attacks: Brute Force.. it’s a brutal use of force to attempt to login. Attacker simply tries to login with every potential password that a user might choose. Given the time and cycles, they will succeed. So, what do we do? Limit a user to 3 attempts before locking them out for 5 minutes? That could work, but lets look at a more elegant weapon, for a more civilized age The exponential backoff algorithm determines the amount of time a user is locked out, based upon the number of times they failed login attempts during a window. It uses some nifty math to make it all happen. Ssl renegotiation dos attack: Ah the classics. SSL renegotiation was quite in fashion, until it was proven that you can DoS a server with ease from a single system, due to the processing expense of ssl renegotiation. While the F5 was a tougher nut to crack than a standard server, we wanted get out a mitigation quickly, just in case. I want to thank Byron for his great work on putting the document together! Have a great day all. Me and Byron at LunchTen Steps to iRules Optimization
Ever wondered whether to use switch or if/else? How about the effects of regular expressions? Or even what the overhead of variables are? We've come up with a list of the top 10 optimization techniques you can follow to write a healthier happier iRule. #1 - Use a Profile First This one seems obvious enough but we continually get questions on the forums about iRules that are already built into the product directly. iRules does add extra processing overhead and will be slower than executing the same features in native code. Granted, they won't be that slow, but any little bit you can save will help you in the long run. So, if you aren't doing custom conditional testing, let the profiles do the work. The following common features implemented in iRules can be handled by native profiles: HTTP header insert and erase HTTP fallback HTTP compress uri <exclude|include> HTTP redirect rewrite HTTP insert X-Forwarded-For HTTP ramcache uri <exclude|include|pinned> Stream profile for content replacement Class profile for URI matching. Even if you do need to do some conditional testing, several of the profiles such as the stream profile will allow you to enable/disable and control the expression directly from the iRule. #2 - A Little Planning Goes A Long Way iRules are fun and it is really easy to find yourself jumping right in and coding without giving proper thought to what you are trying to accomplish. Just like any software project, it will pay off in the long run if you come up with some sort of feature specification and try to determine the best way to implement it. Common questions you can ask yourself include: What are the protocols involved? What commands should I use to achieve the desired result? How can I achieve the desired result in the least amount of steps? What needs to be logged? Where and how will I test it? By taking a few minutes up front, you will save yourself a bunch of rewrites later on. #3 - Understand Control Statements Common control statements include if/elseif/else, switch, and the iRule matchclass/findclass commands with datagroups. The problem is that most of these do the same thing but, depending on their usage, their performance can vary quite a bit. Here are some general rules of thumb when considering which control statement to use: Always think: "switch", "data group", and then "if/elseif" in that order. If you think in this order, then in most cases you will be better off. Use switch for < 100 comparisons. Switches are fastest for fewer than 100 comparisons but if you are making changes to the list frequently, then a data group might be more manageable. Use data groups for > 100 comparisons. Not only will your iRule be easier to read by externalizing the comparisons, but it will be easier to maintain and update. Order your if/elseif's with most frequent hits at the top. If/elseif's must perform the full comparison for each if and elseif. If you can move the most frequently hit match to the top, you'll have much less processing overhead. If you don't know which hits are the highest, think about creating a Statistics profile to dynamically store your hit counts and then modify you iRule accordingly. Combine switch/data group/if's when appropriate. No one said you have to pick only one of the above. When it makes since, embed a switch in an if/elseif or put a matchclass inside a switch. User your operators wisely. "equals" is better than "contains", "string match/switch-glob" is better than regular expressions. The "equals" operator checks to see if string "a" is contained in string "b". If you are trying to do a true comparison, the overhead here is unnecessary - use the "equals" command instead. As for regular expressions, see #4. Use break, return, and EVENT::disable commands to avoid unnecessary iRule processing. Since iRules cannot be broken into procedures they tend to be lengthy in certain situations. When applicable use the "break", "return", or "EVENT::disable" commands to halt further iRule processing. Now, these are not set in stone but just general rules of thumb so use your best judgement. There will always be exceptions to the rule and it is best if you know the cost/benefits to make the right decision for you. #4 - Regex is EVIL! Regex's are cool and all, but are CPU hogs and should be considered pure evil. In most cases there are better alternatives. Bad when HTTP_REQUEST { if { [regex {^/myPortal} [HTTP::uri] } { regsub {/myPortal} [HTTP::uri] "/UserPortal" newUri HTTP::uri $newUri } Good when HTTP_REQUEST { if { [HTTP::uri] starts_with "/myPortal" } { set newUri [string map {myPortal UserPortal [HTTP::uri]] HTTP::uri $newUri } But sometimes they are a necessary evil as illustrated in the CreditCardScrubber iRule. In the cases where it would take dozens or more commands to recreate the functionality in a regular expression, then a regex might be your best option. when HTTP_RESPONSE_DATA { # Find ALL the possible credit card numbers in one pass set card_indices [regexp -all -inline -indices \ {(?:3[4|7]\d{2})(?:[ ,-]?(?:\d{5}(?:\d{1})?)){2}|(?:4\d{3})(?:[ ,-]?(?:\d{4})){3}|(?:5[1-5]\d{2})(?:[ ,-]?(?:\d{4})){3}|(?:6011)(?:[ ,-]?(?:\d{4})){3}} \ [HTTP::payload]] } #5 - Don't Use Variables The use of variables does incur a small amount of overhead that in some cases can be eliminated. Consider the following example where the value of the Host header and the URI are stored in variables and later are compared and logged. These take up additional memory for each connection. when HTTP_REQUEST { set host [HTTP::host] set uri [HTTP::uri] if { $host equals "bob.com" } { log "Host = $host; URI = $uri" pool http_pool1 } } Here is the exact same functionality as above but without the overhead of the variables. when HTTP_REQUEST { if { [HTTP::host] equals "bob.com" } { log "Host = [HTTP::host]; URI = [HTTP::uri]" pool http_pool1 } } You may ask yourself: "Self, who cares about a spare 30 bytes or so?" Well, for a single connection, it's likely no big deal but you need to think in terms of 100's of 1000's of connections per second. That's where even the smallest amount of overhead you can save will add up. Removing those variables could gain you a couple thousand connections per second on your application. And never fear, there is no more overhead in using the builtin commands like [HTTP::host] over a varable reference as both are just direct pointers to the data in memory. #6 - Use Variables Now that I've told you not to use variables, here's the case where using variables makes sense. When you need to do repetitive costly evaluations on a value and can avoid it by storing that evaluation in a variable, then by all means do so. Bad - Too many memory allocations when HTTP_REQUEST { if { [string tolower [HTTP::uri]] starts_with "/img" } { pool imagePool } elseif { ([string tolower [HTTP::uri]] ends_with ".gif") || ([string tolower [HTTP::uri]] ends_with ".jpg") } { pool imagePool } } Bad - The length of a variable name can actually consume more overhead due to the way TCL stores variables in lookup tables. when HTTP_REQUEST { set theUriThatIAmMatchingInThisiRule [string tolower [HTTP::uri]] if { $theUriThatIAmMatchingInThisiRule starts_with "/img" } { pool imagePool } elseif { ($theUriThatIAmMatchingInThisiRule ends_with ".gif") || ($theUriThatIAmMatchingInThisiRule ends_with ".jpg") } { pool imagePool } } Good - Keep your variables names to a handful of characters. when HTTP_REQUEST { set uri [string tolower [HTTP::uri]] if { $uri starts_with "/img" } { pool imagePool } elseif { ($uri ends_with ".gif") || ($uri ends_with ".jpg") } { pool imagePool } } #7 - Understand Polymorphism TCL is polymorphic in that variables can "morph" back and forth from data type to data type depending on how it is used. This makes loosly-typed languages like TCL easier to work with since you don't need to declare variables as a certain type such as integer or string but it does come at a cost. You can minimize those costs by using the correct operators on your data types. Use the correct operator for the correct type. Use eq,ne when comparing strings. Use ==,!= when comparing numbers Use [IP::addr] when comparing IP addresses. If you aren't careful, things may not be what they seem as illustrated below set x 5 if { $x == 5 } { } # evaluates to true if { $x eq 5 } { } # evaluates to true if { $x == 05 } { } # evaluates to true if { $x eq 05 } { } # evaluates to false #8 - Timing The timing iRule command is a great tool to help you eek the best performance out of your iRule. CPU cycles are stored on a per event level and you can turn "timing on" for any and all of your events. These CPU metrics are stored and can be retrieved by the bigpipe CLI, the administrative GUI, or the iControl API. The iRule Editor can take these numbers and convert them into the following statistics CPU Cycles/Request Run Time (ms)/Request Percent CPU Usage/Request Max Concurrent Requests Just make sure you turn those timing values off when you are done optimizing your iRule as that will have overhead as well! #9 - Use The Community DevCentral is a great resource whether you are just starting your first iRule or a seasoned veteran looking for advice on something you may have overlooked. The forums and wikis can be a life saver when trying to squeeze every bit of performance out of your iRule. #10 - Learn about the newest features Read the release documentation for new BIG-IP releases as we are constantly adding new functionalty to make your iRules faster. For instance, look for the upcoming "class" command to make accessing your data groups lightning fast. Conclusion Take these 10 tips with you when you travel down the road of iRule development and you will likley have a faster iRule to show for it. As always, if you have any tips that you've found to be helpful when optimizing your iRules, please pass them along and we'll add them to this list. Get the Flash Player to see this player. 20081030-iRulesOpt_TenSteps.mp3More Web Scraping - Bot Detection
In my last article, I discussed the issue of web scraping and why it could be a problem for many individuals and/or companies. In this article, we will dive into some of the technical details regarding bots and how the BIG-IP Application Security Manager (ASM) can detect them and block them from scraping your website. What Is A Bot? A bot is a software application that runs automated tasks and typically performs these tasks much faster than a human possibly could. In the context of web scraping, bots are used to extract data from websites, parse the data, and assemble it into a structured format where it can be presented in a useful form. Bots can perform many other actions as well, like submitting forms, setting up schedules, and connecting to databases. They can also do fun things like add friends to social networking sites like Twitter, Facebook, Google+, and others. A quick Internet search will show that many different bot tools are readily available for download free of charge. We won't go into the specifics of each vendor's bot application, but it's important to understand that they are out there and are very easy to use. Bot Detection So, now that we know what a bot is and what it does, how can we distinguish between malicious bot activity and harmless human activity? Well, the ASM is configured to check for some very specific activities that help it determine if the client source is a bot or a human. By the way, it's important to note that the ASM can accurately detect a human user only if clients have JavaScript enabled and support cookies. There are three different settings in the ASM for bot detection: Off, Alarm, and Alarm and Block. Obviously, if bot detection is set to "Off" then the ASM does not check for bot activity at all. The "Alarm" setting will detect bot activity and record attack data, but it will allow the client to continue accessing the website. The "Alarm and Block" setting will detect bot activity, record the attack data, and block the suspicious requests. These settings are shown in the screenshot below. Once you apply the setting for bot detection, you can then tune the ASM to begin checking for bots that are accessing your website. The bot detection utilizes four different techniques to detect and defend against bot activity. These include Rapid Surfing, Grace Interval, Unsafe Interval, and Safe Interval. Rapid Surfing detects bot activity by counting the client's page consumption speed. A page change is counted from the page load event to its unload event. The ASM configuration allows you to set a maximum number of page changes for a given time period (measured in milliseconds). If a page changes more than the maximum allowable times for the given time interval, the ASM will declare the client as a bot and perform the action that was set for bot detection (Off, Alarm, Alarm and Block). The default setting for Rapid Surfing is 5 page changes per second (or 1000 milliseconds). The Grace Interval setting specifies the maximum number of page requests the system reviews while it tries to detect whether the client is a human or a bot. As soon as the system makes the determination of human or bot it ends the Grace Interval and stops checking for bots. The default setting for the Grace Interval is 100 requests. Once the system determines that the client is valid, the system does not check the subsequent requests as specified in the Safe Interval setting. This setting allows for normal client activity to continue since the ASM has determined the client is safe (during the Grace Interval). Once the number of requests sent by the client reaches the value specified in the Safe Interval setting, the system reactivates the Grace Interval and begins the process again. The default setting for the Safe Interval is 2000 requests. This Safe Interval is nice because it lowers the processing overhead needed to constantly check every client request. If the system does not detect a valid client during the Grace Interval, the system issues and continues to issue the "Web Scraping Detected" violation until it reaches the number of requests specified in the Unsafe Interval setting. The Unsafe Interval setting specifies the number of requests that the ASM considers unsafe. Much like in the Safe Interval, after the client sends the number of requests specified in the Unsafe Interval setting, the system reactivates the Grace Interval and begins the process again. The default setting for the Unsafe Interval is 100 requests. The following figure shows the settings for Bot Detection and the values associated with each setting. Interval Timing The following picture shows a timeline of client requests and the intervals associated with each request. In the example, the first 100 client requests will fall into the Grace Interval, and during this interval the ASM will be determining whether or not the client is a bot. Let's say a bot is detected at client request 100. Then, the ASM will immediately invoke the Unsafe Interval and the next 100 requests will be issued a "Web Scraping Detected" violation. When the Unsafe Interval is complete, the ASM reverts back to the Grace Interval. If, during the Grace Interval, the system determines that the client is a human, it does not check the subsequent requests at all (during the Safe Interval). Once the Safe Interval is complete, the system moves back into the Grace Interval and the process continues. Notice that the ASM is able to detect a bot before the Grace Interval is complete (as shown in the latter part of the diagram below). As soon as the system detects a bot, it immediately moves into the Unsafe Interval...even if the Grace Interval has not reached its set threshold. Setting Thresholds As you can see from the timeline above, it's important to establish the correct thresholds for each interval setting. The longer you make the Grace Interval, the longer you give the ASM a chance to detect a bot, but keep in mind that the processing overhead can become expensive. Likewise, the Unsafe Interval setting is a great feature, but if you set it too high, your requesting clients will have to sit through a long period of violation notices before they can access your site again. Finally, the Safe Interval setting allows your users free and open access to your site. If you set this too low, you will force the system to cycle through the Grace Interval unnecessarily, but if you set it too high, a bot might have a chance to sneak past the ASM defense and scrape your site. Remember, the ASM does not check client requests at all during the Safe Interval. Also, remember the ASM does not perform web scraping detection on traffic from search engines that the system recognizes as being legitimate. If your web application has its own search engine, it's recommended that you add it to the system. Go to Security > Options > Application Security > Advanced Configuration > Search Engines and add it to the list (the ASM comes preconfigured with Ask, Bing, Google, and Yahoo already loaded). A Quick Test I loaded up a sample virtual web server (in this case it was a fictitious online auction site) and then configured the ASM to Alarm and Block bot activity on the site. Then, I fired up the iMacro plugin from Firefox to scrape the site. Using the iMacro plugin, I sent many client requests in a short amount of time. After several requests to the site, I received the response page shown below. You can see that the response page settings in the ASM are shown in the Firefox browser window when web scraping is detected. These settings can be adjusted in the ASM by navigating to Security > Application Security > Blocking > Response Pages. Well, thanks for coming back and reading about ASM bot detection. Be sure to swing by again for the final web scraping article where I will discuss session anomalies. I can't possibly think of anything that could be more fun than that!
