Controlling a Pool Members Ratio and Priority Group with iControl
A Little Background A question came in through the iControl forums about controlling a pool members ratio and priority programmatically. The issue really involves how the API’s use multi-dimensional arrays but I thought it would be a good opportunity to talk about ratio and priority groups for those that don’t understand how they work. In the first part of this article, I’ll talk a little about what pool members are and how their ratio and priorities apply to how traffic is assigned to them in a load balancing setup. The details in this article were based on BIG-IP version 11.1, but the concepts can apply to other previous versions as well. Load Balancing In it’s very basic form, a load balancing setup involves a virtual ip address (referred to as a VIP) that virtualized a set of backend servers. The idea is that if your application gets very popular, you don’t want to have to rely on a single server to handle the traffic. A VIP contains an object called a “pool” which is essentially a collection of servers that it can distribute traffic to. The method of distributing traffic is referred to as a “Load Balancing Method”. You may have heard the term “Round Robin” before. In this method, connections are passed one at a time from server to server. In most cases though, this is not the best method due to characteristics of the application you are serving. Here are a list of the available load balancing methods in BIG-IP version 11.1. Load Balancing Methods in BIG-IP version 11.1 Round Robin: Specifies that the system passes each new connection request to the next server in line, eventually distributing connections evenly across the array of machines being load balanced. This method works well in most configurations, especially if the equipment that you are load balancing is roughly equal in processing speed and memory. Ratio (member): Specifies that the number of connections that each machine receives over time is proportionate to a ratio weight you define for each machine within the pool. Least Connections (member): Specifies that the system passes a new connection to the node that has the least number of current connections in the pool. This method works best in environments where the servers or other equipment you are load balancing have similar capabilities. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the current number of connections per node or the fastest node response time. Observed (member): Specifies that the system ranks nodes based on the number of connections. Nodes that have a better balance of fewest connections receive a greater proportion of the connections. This method differs from Least Connections (member), in that the Least Connections method measures connections only at the moment of load balancing, while the Observed method tracks the number of Layer 4 connections to each node over time and creates a ratio for load balancing. This dynamic load balancing method works well in any environment, but may be particularly useful in environments where node performance varies significantly. Predictive (member): Uses the ranking method used by the Observed (member) methods, except that the system analyzes the trend of the ranking over time, determining whether a node's performance is improving or declining. The nodes in the pool with better performance rankings that are currently improving, rather than declining, receive a higher proportion of the connections. This dynamic load balancing method works well in any environment. Ratio (node): Specifies that the number of connections that each machine receives over time is proportionate to a ratio weight you define for each machine across all pools of which the server is a member. Least Connections (node): Specifies that the system passes a new connection to the node that has the least number of current connections out of all pools of which a node is a member. This method works best in environments where the servers or other equipment you are load balancing have similar capabilities. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the number of current connections per node, or the fastest node response time. Fastest (node): Specifies that the system passes a new connection based on the fastest response of all pools of which a server is a member. This method might be particularly useful in environments where nodes are distributed across different logical networks. Observed (node): Specifies that the system ranks nodes based on the number of connections. Nodes that have a better balance of fewest connections receive a greater proportion of the connections. This method differs from Least Connections (node), in that the Least Connections method measures connections only at the moment of load balancing, while the Observed method tracks the number of Layer 4 connections to each node over time and creates a ratio for load balancing. This dynamic load balancing method works well in any environment, but may be particularly useful in environments where node performance varies significantly. Predictive (node): Uses the ranking method used by the Observed (member) methods, except that the system analyzes the trend of the ranking over time, determining whether a node's performance is improving or declining. The nodes in the pool with better performance rankings that are currently improving, rather than declining, receive a higher proportion of the connections. This dynamic load balancing method works well in any environment. Dynamic Ratio (node) : This method is similar to Ratio (node) mode, except that weights are based on continuous monitoring of the servers and are therefore continually changing. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the number of current connections per node or the fastest node response time. Fastest (application): Passes a new connection based on the fastest response of all currently active nodes in a pool. This method might be particularly useful in environments where nodes are distributed across different logical networks. Least Sessions: Specifies that the system passes a new connection to the node that has the least number of current sessions. This method works best in environments where the servers or other equipment you are load balancing have similar capabilities. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the number of current sessions. Dynamic Ratio (member): This method is similar to Ratio (node) mode, except that weights are based on continuous monitoring of the servers and are therefore continually changing. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the number of current connections per node or the fastest node response time. L3 Address: This method functions in the same way as the Least Connections methods. We are deprecating it, so you should not use it. Weighted Least Connections (member): Specifies that the system uses the value you specify in Connection Limit to establish a proportional algorithm for each pool member. The system bases the load balancing decision on that proportion and the number of current connections to that pool member. For example,member_a has 20 connections and its connection limit is 100, so it is at 20% of capacity. Similarly, member_b has 20 connections and its connection limit is 200, so it is at 10% of capacity. In this case, the system select selects member_b. This algorithm requires all pool members to have a non-zero connection limit specified. Weighted Least Connections (node): Specifies that the system uses the value you specify in the node's Connection Limitand the number of current connections to a node to establish a proportional algorithm. This algorithm requires all nodes used by pool members to have a non-zero connection limit specified. Ratios The ratio is used by the ratio-related load balancing methods to load balance connections. The ratio specifies the ratio weight to assign to the pool member. Valid values range from 1 through 100. The default is 1, which means that each pool member has an equal ratio proportion. So, if you have server1 a with a ratio value of “10” and server2 with a ratio value of “1”, server1 will get served 10 connections for every one that server2 receives. This can be useful when you have different classes of servers with different performance capabilities. Priority Group The priority group is a number that groups pool members together. The default is 0, meaning that the member has no priority. To specify a priority, you must activate priority group usage when you create a new pool or when adding or removing pool members. When activated, the system load balances traffic according to the priority group number assigned to the pool member. The higher the number, the higher the priority, so a member with a priority of 3 has higher priority than a member with a priority of 1. The easiest way to think of priority groups is as if you are creating mini-pools of servers within a single pool. You put members A, B, and C in to priority group 5 and members D, E, and F in priority group 1. Members A, B, and C will be served traffic according to their ratios (assuming you have ratio loadbalancing configured). If all those servers have reached their thresholds, then traffic will be distributed to servers D, E, and F in priority group 1. he default setting for priority group activation is Disabled. Once you enable this setting, you can specify pool member priority when you create a new pool or on a pool member's properties screen. The system treats same-priority pool members as a group. To enable priority group activation in the admin GUI, select Less than from the list, and in the Available Member(s) box, type a number from 0 to 65535 that represents the minimum number of members that must be available in one priority group before the system directs traffic to members in a lower priority group. When a sufficient number of members become available in the higher priority group, the system again directs traffic to the higher priority group. Implementing in Code The two methods to retrieve the priority and ratio values are very similar. They both take two parameters: a list of pools to query, and a 2-D array of members (a list for each pool member passed in). long [] [] get_member_priority( in String [] pool_names, in Common__AddressPort [] [] members ); long [] [] get_member_ratio( in String [] pool_names, in Common__AddressPort [] [] members ); The following PowerShell function (utilizing the iControl PowerShell Library), takes as input a pool and a single member. It then make a call to query the ratio and priority for the specific member and writes it to the console. function Get-PoolMemberDetails() { param( $Pool = $null, $Member = $null ); $AddrPort = Parse-AddressPort $Member; $RatioAofA = (Get-F5.iControl).LocalLBPool.get_member_ratio( @($Pool), @( @($AddrPort) ) ); $PriorityAofA = (Get-F5.iControl).LocalLBPool.get_member_priority( @($Pool), @( @($AddrPort) ) ); $ratio = $RatioAofA[0][0]; $priority = $PriorityAofA[0][0]; "Pool '$Pool' member '$Member' ratio '$ratio' priority '$priority'"; } Setting the values with the set_member_priority and set_member_ratio methods take the same first two parameters as their associated get_* methods, but add a third parameter for the priorities and ratios for the pool members. set_member_priority( in String [] pool_names, in Common::AddressPort [] [] members, in long [] [] priorities ); set_member_ratio( in String [] pool_names, in Common::AddressPort [] [] members, in long [] [] ratios ); The following Powershell function takes as input the Pool and Member with optional values for the Ratio and Priority. If either of those are set, the function will call the appropriate iControl methods to set their values. function Set-PoolMemberDetails() { param( $Pool = $null, $Member = $null, $Ratio = $null, $Priority = $null ); $AddrPort = Parse-AddressPort $Member; if ( $null -ne $Ratio ) { (Get-F5.iControl).LocalLBPool.set_member_ratio( @($Pool), @( @($AddrPort) ), @($Ratio) ); } if ( $null -ne $Priority ) { (Get-F5.iControl).LocalLBPool.set_member_priority( @($Pool), @( @($AddrPort) ), @($Priority) ); } } In case you were wondering how to create the Common::AddressPort structure for the $AddrPort variables in the above examples, here’s a helper function I wrote to allocate the object and fill in it’s properties. function Parse-AddressPort() { param($Value); $tokens = $Value.Split(":"); $r = New-Object iControl.CommonAddressPort; $r.address = $tokens[0]; $r.port = $tokens[1]; $r; } Download The Source The full source for this example can be found in the iControl CodeShare under PowerShell PoolMember Ratio and Priority.
Intro to Load Balancing for Developers – The Algorithms
If you’re new to this series, you can find the complete list of articles in the series on my personal page here If you are writing applications to sit behind a Load Balancer, it behooves you to at least have a clue what the algorithm your load balancer uses is about. We’re taking this week’s installment to just chat about the most common algorithms and give a plain- programmer description of how they work. While historically the algorithm chosen is both beyond the developers’ control, you’re the one that has to deal with performance problems, so you should know what is happening in the application’s ecosystem, not just in the application. Anything that can slow your application down or introduce errors is something worth having reviewed. For algorithms supported by the BIG-IP, the text here is paraphrased/modified versions of the help text associated with the Pool Member tab of the BIG-IP UI. If they wrote a good description and all I needed to do was programmer-ize it, then I used it. For algorithms not supported by the BIG-IP I wrote from scratch. Note that there are many, many more algorithms out there, but as you read through here you’ll see why these (or minor variants of them) are the ones you’ll see the most. Plain Programmer Description: Is not intended to say anything about the way any particular dev team at F5 or any other company writes these algorithms, they’re just an attempt to put the process into terms that are easier for someone with a programming background to understand. Hopefully a successful attempt. Interestingly enough, I’ve pared down what BIG-IP supports to a subset. That means that F5 employees and aficionados will be going “But you didn’t mention…!” and non-F5 employees will likely say “But there’s the Chi-Squared Algorithm…!” (no, chi-squared is theoretical distribution method I know of because it was presented as a proof for testing the randomness of a 20 sided die, ages ago in Dragon Magazine). The point being that I tried to stick to a group that builds on each other in some connected fashion. So send me hate mail… I’m good. Unless you can say more than 2-5% of the world’s load balancers are running the algorithm, I won’t consider that I missed something important. The point is to give developers and software architects a familiarity with core algorithms, not to build the worlds most complete lexicon of algorithms. Random: This load balancing method randomly distributes load across the servers available, picking one via random number generation and sending the current connection to it. While it is available on many load balancing products, its usefulness is questionable except where uptime is concerned – and then only if you detect down machines. Plain Programmer Description: The system builds an array of Servers being load balanced, and uses the random number generator to determine who gets the next connection… Far from an elegant solution, and most often found in large software packages that have thrown load balancing in as a feature. Round Robin: Round Robin passes each new connection request to the next server in line, eventually distributing connections evenly across the array of machines being load balanced. Round Robin works well in most configurations, but could be better if the equipment that you are load balancing is not roughly equal in processing speed, connection speed, and/or memory. Plain Programmer Description: The system builds a standard circular queue and walks through it, sending one request to each machine before getting to the start of the queue and doing it again. While I’ve never seen the code (or actual load balancer code for any of these for that matter), we’ve all written this queue with the modulus function before. In school if nowhere else. Weighted Round Robin (called Ratio on the BIG-IP): With this method, the number of connections that each machine receives over time is proportionate to a ratio weight you define for each machine. This is an improvement over Round Robin because you can say “Machine 3 can handle 2x the load of machines 1 and 2”, and the load balancer will send two requests to machine #3 for each request to the others. Plain Programmer Description: The simplest way to explain for this one is that the system makes multiple entries in the Round Robin circular queue for servers with larger ratios. So if you set ratios at 3:2:1:1 for your four servers, that’s what the queue would look like – 3 entries for the first server, two for the second, one each for the third and fourth. In this version, the weights are set when the load balancing is configured for your application and never change, so the system will just keep looping through that circular queue. Different vendors use different weighting systems – whole numbers, decimals that must total 1.0 (100%), etc. but this is an implementation detail, they all end up in a circular queue style layout with more entries for larger ratings. Dynamic Round Robin (Called Dynamic Ratio on the BIG-IP): is similar to Weighted Round Robin, however, weights are based on continuous monitoring of the servers and are therefore continually changing. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the current number of connections per node or the fastest node response time. This Application Delivery Controller method is rarely available in a simple load balancer. Plain Programmer Description: If you think of Weighted Round Robin where the circular queue is rebuilt with new (dynamic) weights whenever it has been fully traversed, you’ll be dead-on. Fastest: The Fastest method passes a new connection based on the fastest response time of all servers. This method may be particularly useful in environments where servers are distributed across different logical networks. On the BIG-IP, only servers that are active will be selected. Plain Programmer Description: The load balancer looks at the response time of each attached server and chooses the one with the best response time. This is pretty straight-forward, but can lead to congestion because response time right now won’t necessarily be response time in 1 second or two seconds. Since connections are generally going through the load balancer, this algorithm is a lot easier to implement than you might think, as long as the numbers are kept up to date whenever a response comes through. These next three I use the BIG-IP name for. They are variants of a generalized algorithm sometimes called Long Term Resource Monitoring. Least Connections: With this method, the system passes a new connection to the server that has the least number of current connections. Least Connections methods work best in environments where the servers or other equipment you are load balancing have similar capabilities. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the current number of connections per node or the fastest node response time. This Application Delivery Controller method is rarely available in a simple load balancer. Plain Programmer Description: This algorithm just keeps track of the number of connections attached to each server, and selects the one with the smallest number to receive the connection. Like fastest, this can cause congestion when the connections are all of different durations – like if one is loading a plain HTML page and another is running a JSP with a ton of database lookups. Connection counting just doesn’t account for that scenario very well. Observed: The Observed method uses a combination of the logic used in the Least Connections and Fastest algorithms to load balance connections to servers being load-balanced. With this method, servers are ranked based on a combination of the number of current connections and the response time. Servers that have a better balance of fewest connections and fastest response time receive a greater proportion of the connections. This Application Delivery Controller method is rarely available in a simple load balancer. Plain Programmer Description: This algorithm tries to merge Fastest and Least Connections, which does make it more appealing than either one of the above than alone. In this case, an array is built with the information indicated (how weighting is done will vary, and I don’t know even for F5, let alone our competitors), and the element with the highest value is chosen to receive the connection. This somewhat counters the weaknesses of both of the original algorithms, but does not account for when a server is about to be overloaded – like when three requests to that query-heavy JSP have just been submitted, but not yet hit the heavy work. Predictive: The Predictive method uses the ranking method used by the Observed method, however, with the Predictive method, the system analyzes the trend of the ranking over time, determining whether a servers performance is currently improving or declining. The servers in the specified pool with better performance rankings that are currently improving, rather than declining, receive a higher proportion of the connections. The Predictive methods work well in any environment. This Application Delivery Controller method is rarely available in a simple load balancer. Plain Programmer Description: This method attempts to fix the one problem with Observed by watching what is happening with the server. If its response time has started going down, it is less likely to receive the packet. Again, no idea what the weightings are, but an array is built and the most desirable is chosen. You can see with some of these algorithms that persistent connections would cause problems. Like Round Robin, if the connections persist to a server for as long as the user session is working, some servers will build a backlog of persistent connections that slow their response time. The Long Term Resource Monitoring algorithms are the best choice if you have a significant number of persistent connections. Fastest works okay in this scenario also if you don’t have access to any of the dynamic solutions. That’s it for this week, next week we’ll start talking specifically about Application Delivery Controllers and what they offer – which is a whole lot – that can help your application in a variety of ways. Until then! Don.Accessing TCP Options from iRules
I’ve written several articles on the TCP profile and enjoy digging into TCP. It’s a beast, and I am constantly re-learning the inner workings. Still etched in my visual memory map, however, is the TCP header format, shown in Figure 1 below. Since 9.0 was released, TCP payload data (that which comes after the header) has been consumable in iRules via the TCP::payload and the port information has been available in the contextual commands TCP::local_port/TCP::remote_port and of course TCP::client_port/TCP::server_port. Options, however, have been inaccessible. But beginning with version 10.2.0-HF2, it is now possible to retrieve data from the options fields. Preparing the BIG-IP Prior to version 11.0, it was necessary to set a bigpipe database key with the option (or options) of interest: bigpipe db Rules.Tcpoption.settings [option, first|last], [option, first|last] In version 11.0 and forward, the DB keys are no more and you need to create a tcp profile with the these options defined, like so: ltm profile tcp tcp_opt { app-service none tcp-options "{option first|last} {option first|last}" } The option is an integer between 2 and 255, and the first/last setting indicates whether the system will retain the first or last instance of the specified option. Once that key is set, you’ll need to do a bigstart restart for it to take (warning: service impacting). Note also that the LTM only collects option data starting with the ACK of a connection. The initial SYN is ignored even if you select the first keyword. This is done to prevent a SYN flood attack (in keeping with SYN-cookies). A New iRules Command: TCP::option The TCP::option command has the following syntax: TCP::option get <option> v11 Additions/Changes: TCP::option set <option number> <value> <next|all> TCP::option noset <option number> Pretty simple, no? So now that you can access them, what fun can be had? Real World Scenario: Akamai In Akamai’s IPA and SXL product lines, they support client IP visibility by embedding a version number (one byte) and an IPv4 address (four bytes) as part of their overlay path feature in tcp option number 28. To access this data, we first set the database key: tmsh create ltm profile tcp tcp_opt tcp-options "{28 first}" Now, the iRule utilizing the TCP::option command: when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt28 cH8 ver addr if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { scan $addr "%2x%2x%2x%2x" ip1 ip2 ip3 ip4 set optaddr "$ip1.$ip2.$ip3.$ip4" } } } when HTTP_REQUEST { if { [info exists optaddr] } { HTTP::header insert "X-Forwarded-For" $optaddr } } The Akamai version should be one, so we log if not. Otherwise, we take the address (stored in the variable addr in hex) and scan it to get the decimal equivalents to build the address for inserting in the X-Forwarded-For header. Cool, right? Also cool—along with the new TCP::option command , an extension was made to the IP::addr command to parse binary fields into a dotted decimal IP address. This extension is also available beginning in 10.2.0-HF2, but extended in 11.0. Here’s the syntax: IP::addr parse [-ipv4 | -ipv6 [swap]] <binary field> [<offset>] So for example, if you had an IPv6 address in option 28 with a 1 byte offset, you would parse that like: log local0. "IP::addr parse IPv6 output: [IP::addr parse -ipv6 [TCP::option get 28] 1]" ## Log Result ## May 27 21:51:34 ltm13 info tmm[27207]: Rule /Common/tcpopt_test <CLIENT_ACCEPTED>: IP::addr parse IPv6 output: 2601:1930:bd51:a3e0:20cd:a50b:1cc1:ad13 But in the context of our TCP option, we have 5-bytes of data with the first byte not mattering in the context of an address, so we get at the address with this: set optaddr [IP::addr parse -ipv4 [TCP::option get 28] 1] This cleans up the rule a bit: when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt c ver if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { set optaddr [IP::addr parse -ipv4 $opt28 1] } } } when HTTP_REQUEST { if { [info exists optaddr] } { HTTP::header insert "X-Forwarded-For" $optaddr } } No need to store the address in the first binary scan and no need for the scan command at all so I eliminated those. Setting a forwarding header is not the only thing we can do with this data. It could also be shipped off to a logging server, or used as a snat address (assuming the server had either a default route to the BIG-IP, or specific routes for the customer destinations, which is doubtful). Logging is trivial, shown below with the log command. The HSL commands could be used in lieu of log if sending off-box to a log server. when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt c ver if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { set optaddr [IP::addr parse -ipv4 $opt28 1] log local0. "Client IP extracted from Akamai TCP option is $optaddr" } } } If setting the provided IP as a snat address, you’ll want to make sure it’s a valid IP address before doing so. You can use the TCL catch command and IP::addr to perform this check as seen in the iRule below: when CLIENT_ACCEPTED { set addrs [list \ "192.168.1.1" \ "256.168.1.1" \ "192.256.1.1" \ "192.168.256.1" \ "192.168.1.256" \ ] foreach x $addrs { if { [catch {IP::addr $x mask 255.255.255.255}] } { log local0. "IP $x is invalid" } else { log local0. "IP $x is valid" } } } The output of this iRule: <CLIENT_ACCEPTED>: IP 192.168.1.1 is valid <CLIENT_ACCEPTED>: IP 256.168.1.1 is invalid <CLIENT_ACCEPTED>: IP 192.256.1.1 is invalid <CLIENT_ACCEPTED>: IP 192.168.256.1 is invalid <CLIENT_ACCEPTED>: IP 192.168.1.256 is invalid Adding this logic into a functional rule with snat: when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt c ver if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { set optaddr [IP::addr parse -ipv4 $opt28 1] if { [catch {IP::addr $x mask 255.255.255.255}] } { log local0. "$optaddr is not a valid address" snat automap } else { log local0. "Akamai inserted Client IP is $optaddr. Setting as snat address." snat $optaddr } } } Alternative TCP Option Use Cases The Akamai solution shows an application implementation taking advantage of normally unused space in TCP headers. There are, however, defined uses for several option “kind” numbers. The list is available here: http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xml. Some options that might be useful in troubleshooting efforts: Opkind 2 – Max Segment Size Opkind 3 – Window Scaling Opkind 5 – Selective Acknowledgements Opkind 8 – Timestamps Of course, with tcpdump you get all this plus the context of other header information and data, but hey, another tool in the toolbox, right? Addendum I've been working with F5 SE Leonardo Simon on on additional examples I wanted to share here that uses option 28 or 253 to extract an IPv6 address if the version is 34 and otherwise extracts an IPv4 address if the version is 1 or 2. Option 28 when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] binary scan $opt28 c ver #log local0. "version: $ver" if { $ver == 34 } { set optaddr [IP::addr parse -ipv6 $opt28 1] log local0. "opt28 ipv6 address: $optaddr" } elseif { $ver == 1 || $ver == 2 } { set optaddr [IP::addr parse -ipv4 $opt28 1] log local0. "opt28 ipv4 address: $optaddr" } } Option 253 when CLIENT_ACCEPTED { set opt253 [TCP::option get 253] binary scan $opt253 c ver #log local0. "version: $ver" if { $ver == 34 } { set optaddr [IP::addr parse -ipv6 $opt253 1] log local0. "opt253 ipv6 address: $optaddr" } elseif { $ver == 1 || $ver == 2 } { set optaddr [IP::addr parse -ipv4 $opt253 1] log local0. "opt253 ipv4 address: $optaddr" } }One Time Passwords via an SMS Gateway with BIG-IP Access Policy Manager
One time passwords, or OTP, are used (as the name indicates) for a single session or transaction. The plus side is a more secure deployment, the downside is two-fold—first, most solutions involve a token system, which is costly in management, dollars, and complexity, and second, people are lousy at remembering things, so a delivery system for that OTP is necessary. The exercise in this tech tip is to employ BIG-IP APM to generate the OTP and pass it to the user via an SMS Gateway, eliminating the need for a token creating server/security appliance while reducing cost and complexity. Getting Started This guide was developed by F5er Per Boe utilizing the newly released BIG-IP version 10.2.1. The “-secure” option for the mcget command is new in this version and is required in one of the steps for this solution. Also, this solution uses the Clickatell SMS Gateway to deliver the OTPs. Their API is documented at http://www.clickatell.com/downloads/http/Clickatell_HTTP.pdf. Other gateway providers with a web-based API could easily be substituted. Also, there are steps at the tail end of this guide to utilize the BIG-IP’s built-in mail capabilities to email the OTP during testing in lieu of SMS. The process in delivering the OTP is shown in Figure 1. First a request is made to the BIG-IP APM. The policy is configured to authenticate the user’s phone number in Active Directory, and if successful, generate a OTP and pass along to the SMS via the HTTP API. The user will then use the OTP to enter into the form updated by APM before allowing the user through to the server resources. BIG-IP APM Configuration Before configuring the policy, an access profile needs to be created, as do a couple authentication servers. First, let’s look at the authentication servers Authentication Servers To create servers used by BIG-IP APM, navigate to Access Policy->AAA Servers and then click create. This profile is simple, supply your domain server, domain name, and admin username and password as shown in Figure 2. The other authentication server is for the SMS Gateway, and since it is an HTTP API we’re using, we need the HTTP type server as shown in Figure 3. Note that the hidden form values highlighted in red will come from your Clickatell account information. Also note that the form method is GET, the form action references the Clickatell API interface, and that the match type is set to look for a specific string. The Clickatell SMS Gateway expects the following format: https://api.clickatell.com/http/sendmsg?api_id=xxxx&user=xxxx&password=xxxx&to=xxxx&text=xxxx Finally, successful logon detection value highlighted in red at the bottom of Figure 3 should be modified to response code returned from SMS Gateway. Now that the authentication servers are configured, let’s take a look at the access profile and create the policy. Access Profile & Policy Before we can create the policy, we need an access profile, shown below in Figure 4 with all default settings. Now that that is done, we click on Edit under the Access Policy column highlighted in red in Figure 5. The default policy is bare bones, or as some call it, empty. We’ll work our way through the objects, taking screen captures as we go and making notes as necessary. To add an object, just click the “+” sign after the Start flag. The first object we’ll add is a Logon Page as shown in Figure 6. No modifications are necessary here, so you can just click save. Next, we’ll configure the Active Directory authentication, so we’ll add an AD Auth object. Only setting here in Figure 7 is selecting the server we created earlier. Following the AD Auth object, we need to add an AD Query object on the AD Auth successful branch as shown in Figures 8 and 9. The server is selected in the properties tab, and then we create an expression in the branch rules tab. To create the expression, click change, and then select the Advanced tab. The expression used in this AD Query branch rule: expr { [mcget {session.ad.last.attr.mobile}] != "" } Next we add an iRule Event object to the AD Query OK branch that will generate the one time password and provide logging. Figure 10 Shows the iRule Event object configuration. The iRule referenced by this event is below. The logging is there for troubleshooting purposes, and should probably be disabled in production. 1: when ACCESS_POLICY_AGENT_EVENT { 2: expr srand([clock clicks]) 3: set otp [string range [format "%08d" [expr int(rand() * 1e9)]] 1 6 ] 4: set mail [ACCESS::session data get "session.ad.last.attr.mail"] 5: set mobile [ACCESS::session data get "session.ad.last.attr.mobile"] 6: set logstring mail,$mail,otp,$otp,mobile,$mobile 7: ACCESS::session data set session.user.otp.pw $otp 8: ACCESS::session data set session.user.otp.mobile $mobile 9: ACCESS::session data set session.user.otp.username [ACCESS::session data get "session.logon.last.username"] 10: log local0.alert "Event [ACCESS::policy agent_id] Log $logstring" 11: } 12: 13: when ACCESS_POLICY_COMPLETED { 14: log local0.alert "Result: [ACCESS::policy result]" 15: } On the fallback path of the iRule Event object, add a Variable Assign object as show in Figure 10b. Note that the first assignment should be set to secure as indicated in image with the [S]. The expressions in Figure 10b are: session.logon.last.password = expr { [mcget {session.user.otp.pw}]} session.logon.last.username = expr { [mcget {session.user.otp.mobile}]} On the fallback path of the AD Query object, add a Message Box object as shown in Figure 11 to alert the user if no mobile number is configured in Active Directory. On the fallback path of the Event OTP object, we need to add the HTTP Auth object. This is where the SMS Gateway we configured in the authentication server is referenced. It is shown in Figure 12. On the fallback path of the HTTP Auth object, we need to add a Message Box as shown in Figure 13 to communicate the error to the client. On the Successful branch of the HTTP Auth object, we need to add a Variable Assign object to store the username. A simple expression and a unique name for this variable object is all that is changed. This is shown in Figure 14. On the fallback branch of the Username Variable Assign object, we’ll configure the OTP Logon page, which requires a Logon Page object (shown in Figure 15). I haven’t mentioned it yet, but the name field of all these objects isn’t a required change, but adding information specific to the object helps with readability. On this form, only one entry field is required, the one time password, so the second password field (enabled by default) is set to none and the initial username field is changed to password. The Input field below is changed to reflect the type of logon to better queue the user. Finally, we’ll finish off with an Empty Action object where we’ll insert an expression to verify the OTP. The name is configured in properties and the expression in the branch rules, as shown in Figures 16 and 17. Again, you’ll want to click advanced on the branch rules to enter the expression. The expression used in the branch rules above is: expr { [mcget {session.user.otp.pw}] == [mcget -secure {session.logon.last.otp}] } Note again that the –secure option is only available in version 10.2.1 forward. Now that we’re done adding objects to the policy, one final step is to click on the Deny following the OK branch of the OTP Verify Empty Action object and change it from Deny to Allow. Figure 18 shows how it should look in the visual policy editor window. Now that the policy is completed, we can attach the access profile to the virtual server and test it out, as can be seen in Figures 19 and 20 below. Email Option If during testing you’d rather send emails than utilize the SMS Gateway, then configure your BIG-IP for mail support (Solution 3664), keep the Logging object, lose the HTTP Auth object, and configure the system with this script to listen for the messages sent to /var/log/ltm from the configured Logging object: #!/bin/bash while true do tail -n0 -f /var/log/ltm | while read line do var2=`echo $line | grep otp | awk -F'[,]' '{ print $2 }'` var3=`echo $line | grep otp | awk -F'[,]' '{ print $3 }'` var4=`echo $line | grep otp | awk -F'[,]' '{ print $4 }'` if [ "$var3" = "otp" -a -n "$var4" ]; then echo Sending pin $var4 to $var2 echo One Time Password is $var4 | mail -s $var4 $var2 fi done done The log messages look like this: Jan 26 13:37:24 local/bigip1 notice apd[4118]: 01490113:5: b94f603a: session.user.otp.log is mail,user1@home.local,otp,609819,mobile,12345678 The output from the script as configured looks like this: [root@bigip1:Active] config # ./otp_mail.sh Sending pin 239272 to user1@home.local Conclusion The BIG-IP APM is an incredibly powerful tool to add to the LTM toolbox. Whether using the mail system or an SMS gateway, you can take a bite out of your infrastructure complexity by using this solution to eliminate the need for a token management service. Many thanks again to F5er Per Boe for this excellent solution!v11: iRules Data Group Updates
Several months ago I wrote up the v10 formatting for internal and external datagroups: iRules Data Group Formatting Rules. In v11, however, there is a change to the format of the internal data group and the data group reference to external class files (the formatting in the external class file itself is unchanged). The formatting rules in v11 for data groups more closely resembles the tmsh commands necessary to build the class at the CLI (these command attributes are masked if you are using the GUI). I’ll follow the same format as the original write-up in showing the various data group types. The format is the same among internal data group types. If there is no value associated with the key, there is a curly bracket pair trailing the key on the same line. If there is an associated value with a key, the curly bracket opens the value, followed by a newline with the keyword data and the value, then another newline with the closing curly bracket. After the records are listed, the type is specified. For external data groups, the file name and the type are specified. If the filename is in /var/class, the path is omitted from the filename reference. Address Data Groups Internal Data Group ltm data-group internal addr_testclass { records { 192.168.1.1/32 { } 192.168.1.2/32 { data "host 2" } 192.168.2.0/24 { } 192.168.3.0/24 { data "network 2" } } type ip } External Data Group ltm data-group external addr_testclass_ext { external-file-name addr_testclass.class type ip } Integer Data Groups Internal Data Group ltm data-group internal int_testclass { records { 1 { data "test 1" } 2 { data "test 2" } } type integer } External Data Group ltm data-group external int_testclass_ext { external-file-name int_testclass type integer } String Data Groups Internal Data Group ltm data-group internal str_testclass { records { str1 { data "value 1" } str2 { data "value 2" } } type string } External Data Group ltm data-group external str_testclass_ext { external-file-name str_testclass.class type string } External Datagroup File Management Beginning in v11, external datagroups are imported into a local filestore rather than simply existing someplace on the file system (/config/filestore). However, this filestore is not meant to edited manually. Please follow the steps below for creating or modifying external datagroups. Thanks to hoolio for the steps in this external datagroup section. Create a New External Datagroup from the CLI Non-Interactively 1. Create or copy over to LTM a temporary file containing the external data group contents. If copying, make sure the line terminators are \n only not \r\n. # cat /var/tmp/string_name_value_external_dg.txt "name1" := "value1", "name2" := "value2", "name3" := "value3", 2. Create the new external data group file tmsh create /sys file data-group string_name_value_external_dg_file separator ":=" source-path file:/var/tmp/string_name_value_external_dg.txt type string 3. Create the external data group referencing the file tmsh create /ltm data-group external string_name_value_external_dg external-file-name string_name_value_external_dg_file Modify the External Datagroup File for Existing Datagroup 1. Create a new temporary file containing the updated external data group contents # cat /var/tmp/string_name_value_external_v2_dg.txt "name1" := "valueA", "name2" := "valueB", "name3" := "valueC", 2. Import the new data group file tmsh create /sys file data-group string_name_value_external_v2_dg_file separator ":=" source-path file:/var/tmp/string_name_value_external_v2_dg.txt type string 3. Modify the data group definition to reference the new external data group file tmsh modify /ltm data-group external string_name_value_external_dg external-file-name string_name_value_external_v2_dg_file 4. Delete the old data group file if it’s unneeded tmsh delete sys file data-group string_name_value_external_dg_file Handling Line Terminator Discrepencies Whether imporing external datagroups in the GUI or from the CLI, the system does not accept files with \r\n line terminators, it only accepts \n. If you copy files over from windows, most likely you have the wrong terminator in your file format. To check, you can use the od command. Datagroup Contents created in vi on LTM [root@golgotha:Active] data_group_d # od -c /var/tmp/string_name_value_external_dg.txt 0000000 " n a m e 1 " : = " v a l u 0000020 e 1 " , \n " n a m e 2 " : = 0000040 " v a l u e 2 " , \n " n a m e 3 0000060 " : = " v a l u e 3 " , \n 0000077 Datagroup Contents created in Notepad on Windows [root@golgotha:Active] data_group_d # od -c /var/tmp/notepad_dg.txt 0000000 " n a m e 1 " : = " v a l u 0000020 e 1 " , \r \n " n a m e 2 " : = 0000040 " v a l u e 2 " , \r \n " n a m 0000060 e 3 " : = " v a l u e 3 " , 0000100 If your line terminators are incorrect, you can use the tr command to remove the \r's. [root@golgotha:Active] tmp # cat /var/tmp/notepad_dg.txt | tr -d '\r' > /var/tmp/notepad_dg_update.txt [root@golgotha:Active] tmp # od -c /var/tmp/notepad_dg_update.txt 0000000 " n a m e 1 " : = " v a l u 0000020 e 1 " , \n " n a m e 2 " : = 0000040 " v a l u e 2 " , \n " n a m e 3 0000060 " : = " v a l u e 3 " , 0000076
Intermediate iRules: Nested Conditionals
Conditionals are a pretty standard tool in every programmer's toolbox. They are the functions that allow us to decided when we want certain actions to happen, based on, well, conditions that can be determined within our code. This concept is as old as compilers. Chances are, if you're writing code, you're going to be using a slew of these things, even in an Event based language like iRules. iRules is no different than any other programming/scripting language when it comes to conditionals; we have them. Sure how they're implemented and what they look like change from language to language, but most of the same basic tools are there: if, else, switch, elseif, etc. Just about any example that you might run across on DevCentral is going to contain some example of these being put to use. Learning which conditional to use in each situation is an integral part to learning how to code effectively. Once you have that under control, however, there's still plenty more to learn. Now that you're comfortable using a single conditional, what about starting to combine them? There are many times when it makes more sense to use a pair or more of conditionals in place of a single conditional along with logical operators. For example: if { [HTTP::host] eq "bob.com" and [HTTP::uri] starts_with "/uri1" } { pool pool1 } elseif { [HTTP::host] eq "bob.com" and [HTTP::uri] starts_with "/uri2" } { pool pool2 } elseif { [HTTP::host] eq "bob.com" and [HTTP::uri] starts_with "/uri3" } { pool pool3 } Can be re-written to use a pair of conditionals instead, making it far more efficient. To do this, you take the common case shared among the example strings and only perform that comparison once, and only perform the other comparisons if that result returns as desired. This is more easily described as nested conditionals, and it looks like this: if { [HTTP::host] eq "bob.com" } { if {[HTTP::uri] starts_with "/uri1" } { pool pool1 } elseif {[HTTP::uri] starts_with "/uri2" } { pool pool2 } elseif {[HTTP::uri] starts_with "/uri3" } { pool pool3 } } These two examples are logically equivalent, but the latter example is far more efficient. This is because in all the cases where the host is not equal to "bob.com", no other inspection needs to be done, whereas in the first example, you must perform the host check three times, as well as the uri check every single time, regardless of the fact that you could have stopped the process earlier. While basic, this concept is important in general when coding. It becomes exponentially more important, as do almost all optimizations, when talking about programming in iRules. A script being executed on a server firing perhaps once per minute benefits from small optimizations. An iRule being executed somewhere in the order of 100,000 times per second benefits that much more. A slightly more interesting example, perhaps, is performing the same logical nesting while using different operators. In this example we'll look at a series of if/elseif statements that are already using nesting, and take a look at how we might use the switch command to even further optimize things. I've seen multiple examples of people shying away from switch when nesting their logic because it looks odd to them or they're not quite sure how it should be structured. Hopefully this will help clear things up. First, the example using if statements: when HTTP_REQUEST { if { [HTTP::host] eq "secure.domain.com" } { HTTP::header insert "Client-IP:[IP::client_addr]" pool sslServers } elseif { [HTTP::host] eq "www.domain.com" } { HTTP::header insert "Client-IP:[IP::client_addr]" pool httpServers } elseif { [HTTP::host] ends_with "domain.com" and [HTTP::uri] starts_with "/secure"} { HTTP::header insert "Client-IP:[IP::client_addr]" pool sslServers } elseif {[HTTP::host] ends_with "domain.com" and [HTTP::uri] starts_with "/login"} { HTTP::header insert "Client-IP:[IP::client_addr]" pool httpServers } elseif { [HTTP::host] eq "intranet.myhost.com" } { HTTP::header insert "Client-IP:[IP::client_addr]" pool internal } } As you can see, this is completely functional and would do the job just fine. There are definitely some improvements that can be made, though. Let's try using a switch statement instead of several if comparisons for improved performance. To do that, we're going to have to use an if nested inside a switch comparison. While this might be new to some or look a bit odd if you're not used to it, it's completely valid and often times the most efficient you’re going to get. This is what the above code would look like cleaned up and put into a switch: when HTTP_REQUEST { HTTP::header insert "Client-IP:[IP::client_addr]" switch -glob [HTTP::host] { "secure.domain.com" { pool sslServers } "www.domain.com" { pool httpServers } "*.domain.com" { if { [HTTP::uri] starts_with "/secure" } { pool sslServers } else { pool httpServers } } "intranet.myhost.com" { pool internal } } } As you can see this is not only easier to read and maintain, but it will also prove to be more efficient. We've moved to the more efficient switch structure, we've gotten rid of the repeat host comparisons that were happening above with the /secure vs /login uris, and while I was at it I got rid of all those examples of inserting a header, since that was happening in every case anyway. Hopefully the benefit this technique can offer is clear, and these examples did the topic some justice. With any luck, you'll nest those conditionals with confidence now.Implementing HTTP Strict Transport Security in iRules
Last month I ran across a blog entry by Extreme Geekboy discussing a patch (now in the most recent nightly forthcoming 4.0 builds) for Firefox he submitted that implements the user agent components of HTTP Strict Transport Security. Strict Transport Security, or HSTS (or STS if that extra character is taxing to type) is an internet draft that allows site owners to specify https as the only acceptable means of accessing the site. This is accomplished by the site inserting a header that the browser will evaluate and for x number of seconds (specified in the header) will rewrite all requests, either from the user or returned in a link from the site to https. This first part is good, but is only half of the implementation. If you are under a man-in-the-middle attack, it matters not if your data is encrypted because the attacker has the keys and is quite happy to decrypt your session unbeknownst to you. This is where the second half of the draft comes in. It disallows the use of untrusted certificates (self-signed, untrusted-CA signed, etc). Any link to an untrusted destination should result in an error in the browser. The goals of the draft are to thwart passive and active network attackers as well as imperfect web developers. It does not address phishing or malware. For details on the threat vectors, read section 2.3 of the draft. Implementation of this draft is actually quite trivial. To get there, I’ll walk you configuring your own certificate authority for use in testing, a BIG-IP (Don’t have one? Get the VE trial!), and a server. All this testing for me is completely contained on my laptop, utilizing Joe’s excellent article on laptop load balancing configuration with LTM VE and VMware, though full-disclosure: I deployed Apache instead of IIS. Working with Certificates I’ve worked with certificates on windows and linux, but for this go I’ll create the certificate authority on my linux virtual machine and prepare the certificates. Many have mad cli skills with the openssl command switches, but I do not. So I’m a big fan of the CA.pl script for working with certificates, which hides a lot of the magic. Make a directory a copy a couple tools into it for testing (my Ubuntu system file locations, ymmv) jrahm@jrahm-dev:~$ mkdir catest jrahm@jrahm-dev:~$ cd catest jrahm@jrahm-dev:~/catest$ cp /usr/lib/ssl/misc/CA.pl . jrahm@jrahm-dev:~/catest$ cp /usr/lib/ssl/openssl.cnf . Create the certificate authority. Questions are pretty self explanatory, make sure the common name is the name you want the CA to be referenced as. jrahm@jrahm-dev:~/catest$ ./CA.pl –newca Create the certificate and sign in. Similar questions to the CA process. Common name should be the name of your site. In my case, this is test.testco.com jrahm@jrahm-dev:~/catest$ ./CA.pl -newreq jrahm@jrahm-dev:~/catest$ ./CA.pl –sign Export the root certificate to Windows compatible format (had to use the openssl command for this one) jrahm@jrahm-dev:~/catest$ openssl x509 -in cacert.pem -outform DER -out ca.der Copy the files to the desktop (using pscp) C:\Users\jrahm>pscp jrahm@10.10.20.200:/home/jrahm/catest/*.pem . C:\Users\jrahm>pscp jrahm@10.10.20.200:/home/jrahm/catest/demoCA/ca.der . Install the root certificate in Windows Install the test.testco.com key and certificate to BIG-IP Create the SSL Profile for CA-signed certificate Create a self-signed certificate in BIG-IP for host test.testco.com Create an additional clientssl profile for the self-signed certificate Preparing the BIG-IP Configuration To test this properly we need four virtual servers, a single pool, and a couple iRules. The first two virtuals are for the “good” site and support the standard ports for http and https. The second two virtuals are for the “bad” site and this site will represent our man-in-the-middle attacker. The iRules support a response rewrite on the good site http virtual (as recommended in the draft), and the insert of the HSTS header on the https virtual only (as required by the draft). Not specified in the draft is the appropriate length for the max-age. I’m adding logic to expire the max-age a day in advance of the certificate expiration date, but you can set a static length of time. I read on one blog that a user was setting it for 50 years. It’s not necessary in my example, but I’m setting the includeSubDomains as well, so that will instruct browsers to securely request and link from test.testco.com and any subdomains of this site (ie, my.test.testco.com). 1: ### iRule for HSTS HTTP Virtuals ### 2: # 3: when HTTP_REQUEST { 4: HTTP::respond 301 Location "https://[HTTP::host][HTTP::uri]" 5: } 6: 7: ### iRule for HSTS HTTPS Virtuals ### 8: # 9: when RULE_INIT { 10: set static::expires [clock scan 20110926] 11: } 12: when HTTP_RESPONSE { 13: HTTP::header insert Strict-Transport-Security "max-age=[expr {$static::expires - [clock seconds]}]; includeSubDomains" 14: } HSTS & MITM Virtuals ### "Good" Virtuals ### # virtual testco_http-vip { snat automap pool testco-pool destination 10.10.20.111:http ip protocol tcp rules hsts_redirect profiles { http {} tcp {} } } virtual testco_https-vip { snat automap pool testco-pool destination 10.10.20.111:https ip protocol tcp rules hsts_insert profiles { http {} tcp {} testco_clientssl { clientside } } } ### "Bad" Virtuals ### # virtual testco2_http-vip { snat automap pool testco-pool destination 10.10.20.112:http ip protocol tcp profiles { http {} tcp {} } } virtual testco2_https-vip { snat automap pool testco-pool destination 10.10.20.112:https ip protocol tcp profiles { http {} tcp {} testco-bad_clientssl { clientside } } } The Results I got the expected results on both Firefox 4.0 and Chrome. Once I switched the virtual from the known good site to the bad site, both browsers presented error pages that I could not click through. HSTS TestResults Great! Where is it Supported? Support already exists in the latest releases of Google Chrome, and if you use the NoScript add-on for current Firefox releases you have support as well. As mentioned above in the introductory section, Firefox 4.0 will support it as well when it releases. Conclusion HTTP Strict Transport Security is a promising development in thwarting some attack vectors between client and server, and is a simple yet effective deployment in iRules. One additional thing worth mentioning is the ability on the user agent (browser or browser add-on) to “seed” known HSTS servers. This would provide additional protection against the initial http connection users might make before redirecting to the https site where the STS header is delivered. Section 12.2 of the draft discusses the bootstrap vulnerability without the seeds in place prior to the first connection to the specified site. Technorati Tags: F5 DevCentral,HTTP Strict Transport Security,HSTS,STS,MITM Attack,Jason Rahm Strict+Transport+Security" rel="tag">HTTP Strict Transport Security,HSTS,STS,MITM Attack,Jason Rahm20 Lines or Less #1
Yesterday I got an idea for what I think will be a cool new series that I wanted to bring to the community via my blog. I call it "20 lines or less". My thought is to pose a simple question: "What can you do via an iRule in 20 lines or less?". Each week I'll find some cool examples of iRules doing fun things in less than 21 lines of code, not counting white spaces or comments, round them up, and post them here. Not only will this give the community lots of cool examples of what iRules can do with relative ease, but I'm hoping it will continue to show just how flexible and light-weight this technology is - not to mention just plain cool. I invite you to follow along, learn what you can and please, if you have suggestions, contributions, or feedback of any kind, don't hesitate to comment, email, IM, whatever. You know how to get a hold of me...please do. ;) I'd love to have a member contributed version of this once a month or quarter or ... whatever if you guys start feeding me your cool, short iRules. Ok, so without further adieu, here we go. The inaugural edition of 20 Lines or Less. For this first edition I wanted to highlight some of the things that have already been contributed by the awesome community here at DevCentral. So I pulled up the Code Share and started reading. I was quite happy to see that I couldn't even get halfway through the list of awesome iRule contributions before I found 5 entries that were neat, and under 20 lines (These are actually almost all under 10 lines of code - wow!) Kudos to the contributors. I'll grab another bunch next week to keep highlighting what we've got already! Cipher Strength Pool Selection Ever want to check the type of encryption your users are using before allowing them into your most secure content? Here's your solution. when HTTP_REQUEST { log local0. "[IP::remote_addr]: SSL cipher strength is [SSL::cipher bits]" if { [SSL::cipher bits] < 128 }{ pool weak_encryption_pool } else { pool strong_encryption_pool } } Clone Pool Based On URI Need to clone some of your traffic to a second pool, based on the incoming URI? Here you go... when HTTP_REQUEST { if { [HTTP::uri] starts_with "/clone_me" } { pool real_pool clone pool clone_pool } else { pool real_pool } } Cache No POST Have you been looking for a way to avoid sending those POST responses to your RAMCache module? You're welcome. when HTTP_REQUEST { if { [HTTP::method] equals "POST" } { CACHE::disable } else { CACHE::enable } } Access Control Based on IP Here's a great example of blocking unwelcome IP addresses from accessing your network and only allowing those Client-IPs that you have deemed trusted. when CLIENT_ACCEPTED { if { [matchclass [IP::client_addr] equals $::trustedAddresses] }{ #Uncomment the line below to turn on logging. #log local0. "Valid client IP: [IP::client_addr] - forwarding traffic" forward } else { #Uncomment the line below to turn on logging. #log local0. "Invalid client IP: [IP::client_addr] - discarding" discard } } Content Type Tracking If you're looking to keep track of the different types of content you're serving, this iRule can help in a big way. # First, create statistics profile named "ContentType" with following entries: # HTML # Images # Scripts # Documents # Stylesheets # Other # Now associate this Statistics Profile to the virtual server. Then apply the following iRule. # To view the results, go to Statistics -> Profiles - Statistics when HTTP_RESPONSE { switch -glob [HTTP::header "Content-type"] { image/* { STATS::incr "ContentType" "Images" } text/html { STATS::incr "ContentType" "HTML" } text/css { STATS::incr "ContentType" "Stylesheets" } *javascript { STATS::incr "ContentType" "Scripts" } text/vbscript { STATS::incr "ContentType" "Scripts" } application/pdf { STATS::incr "ContentType" "Documents" } application/msword { STATS::incr "ContentType" "Documents" } application/*powerpoint { STATS::incr "ContentType" "Documents" } application/*excel { STATS::incr "ContentType" "Documents" } default { STATS::incr "ContentType" "Other" } } } There you have it, the first edition of "20 Lines or Less"! I hope you enjoyed it...I sure did. If you've got feedback or examples to be featured in future editions, let me know. #ColinWhat is iControl?
What is iControl? So, you've arrived at DevCentral and are trying to figure out the various extensibility features of our products. Well, hopefully this article will set you on the right path for whatever your task shall be and help you in deciding which technology you'll want to use to get there. The Two: There may have been One Ring to Rule Them All in J.R.R. Tolkein's Middle Earth, but Mr. Tolkein better watch out for F5. We've got not one, but two rings of power: iRules and iControl. iRules iRules is F5's embedded scripting language used to inspect, modify, and enforce policy on content flowing through the network. Need to inspect all out-going webpages to ensure no sensitive informationis leaked? iRules is your answer. Need to dynamically assign priority to certain classes of client connections? iRules is for you again. Need to add a featureto BIG-IP? It's iRules again. We'll discuss more on iRules in an upcoming article. In the mean time, check out the iRules CodeShare to get an idea what others are doing with them. iRules are very powerful but the solve only one class of management issues. Let's say you are about to deploy a new web application in your production webfarm. How do you automate that process? It's iControl this time around. iControl So, what exactly is iControl? Well, Marketing describes iControl as: iControl is the first open API that enables applications to work in concert with the underlying network based on true software integration. Utilizing SOAP/XML to ensure open communications between dissimilar systems, iControl helps F5 customers, leading independent software vendors ( ISVs ), and Solution Providers realize new levels of automation and configuration management efficiency. Whether monitoring network-level traffic statistics, automating network configuration and management, or facilitating next generation service-oriented architectures, iControl gives organizations the power and flexibility to ensure that applications and the network work together for increased reliability, security, and performance. Further, iControl has proven itself as a valuable technology that can help reduce the cost of managing complex environments. Got all that? To put it in simpler terms, iControl is a Remote Management Application Programming Interface (API). Think of it as an extension to the management GUI or Command Line Interface (CLI) allowing custom scripts/applications/processes/etc. to be created that would normally be achieved by operators working within the Admin interfaces. iControl was built using Web Services technologies (SOAP and WSDL) which makes integration into developer platforms (VS.Net, WebLogic, Eclipse, JDeveloper, ...) a sinch. Heck, iControl even works with dynamic scripting lanaguages like Perl and Microsoft Windows PowerShell. Unlike the awkwardness of SNMP, iControl has a very robust and rich set of interfaces and methods allowing management of everything from system daemons, user accounts, load balancing (local and global), security, iRules, and everything else you could do with the GUI. Under the covers, iControl is implemented as a set of SOAP Based Web Services described by a set of Web Service Description Language (WSDL) files. Implemented over HTTP+SSL, iControl applications communicate to the BIG-IP devices in a similar way that a browser would connect to the administration GUI. The application opens a secure HTTP connection to the BIG-IP's iControl Portal using the same authentication credentials setup for the Admin GUI accounts. The portal then acts as a broker for all iControl method calls and forwards them to the appropriate subsystem on the device. Here are a few examples of how iControl is being used by customers and partners. Automatic Content Deployment to webfarms (1000s of servers daily) Enabling server provisioning in dynamic grid networks Secure remote access for vending machines (via the FirePass iControl Client API) Server identification masking (hiding specifics of backend servers) Response content filtering (scrubbing Credit Card and Social Security numbers) Extracting and archiving statistics for reporting and auditing Custom monitoring and alerting applications Notification alerting agents (rss feeds, email, pager) Listening application for network configuration change events The iRules Editor Many Many more... Are you hesitant to get started because you aren't a developer? Never fear, we have an article series of getting started with iControl, and we are always available on the forums to help out with coding/debugging issues that come up. Coupled with support for almost all mainstream development environments, you are bound to have someone in your organization with relevant experience. So what are you waiting for? Grab your favorite editor and get on the road to automation bliss!iRule Security 101 - #07 - FTP Proxy
We get questions all the time about custom application protocols and how one would go about writing an iRule to "understand" what's going on with that protocol. In this article, I will look at the FTP protocol and show you how one could write the logic to understand that application flow and selectively turn on and off support for various commands within the protocol. Other articles in the series: iRule Security 101 – #1 – HTTP Version iRule Security 101 – #02 – HTTP Methods and Cross Site Tracing iRule Security 101 – #03 – HTML Comments iRule Security 101 – #04 – Masking Application Platform iRule Security 101 – #05 – Avoiding Path Traversal iRule Security 101 – #06 – HTTP Referer iRule Security 101 – #07 – FTP Proxy iRule Security 101 – #08 – Limiting POST Data iRule Security 101 – #09 – Command Execution FTP FTP, for those who don't know, stands for File Transfer Protocol. FTP is designed to allow for the remote uploading and downloading of documents. I'm not going to dig deep into the protocol in this document, but for those who want to explore further, it is defined in RFC959. The basics of FTP are as follows. Requests are made with single line requests formatted as: COMMAND COMMAND_ARGS CRLF Some FTP commands include USER, PASS, & ACCT for authentication, CWD for changing directories, LIST for requesting the contents of a directory, and QUIT for terminating a session. Responses to commands are made in two ways. Over the main "control" connection, the server will process the request and then return a response in this format CODE DESCRIPTION CRLF Where code is the status code defined for the given request command. These have some similarity to HTTP response codes (200 -> OK, 500 -> Error), but don't count on them being exactly the same for each situation. For commands that do not requests content from the server (USER, PASS, CWD, etc), the control connection is all that is uses. But, there are other commands that specifically request data from the server. RETR (downloading a file), STOR (uploading a file), and LIST (for requesting a current directory listing) are examples of these types of commands. For these commands, the status is still returned in the control channel, but the data is passed back in a separate "data" channel that is configured by the client with either the PORT or PASV commands. Writing the Proxy We'll start of the iRule with a set of global variables that are used across all connections. In this iRule will will only inspect on the following FTP commands: USER, PASV, RETR, STOR, RNFR, FNTO, PORT, RMD, MKD, LIST, PWD, CWD, and DELE. This iRule can easily be expanded to include other commands in the FTP command set. In the RULE_INIT event we will set some global variables to determine how we want the proxy to handle the specific commands. A value of 1 for the "block" options will make the iRule deny those commands from reaching the backend FTP server. Setting a value of 0 for the block flag, will allow the command to pass through. when RULE_INIT { set DEBUG 1 #------------------------------------------------------------------------ # FTP Commands #------------------------------------------------------------------------ set sec_block_anonymous_ftp 1 set sec_block_passive_ftp 0 set sec_block_retr_cmd 0 set sec_block_stor_cmd 0 set sec_block_rename_cmd 0 set sec_block_port_cmd 0 set sec_block_rmd_cmd 0 set sec_block_mkd_cmd 0 set sec_block_list_cmd 0 set sec_block_pwd_cmd 0 set sec_block_cwd_cmd 0 set sec_block_dele_cmd 1 } Since we will not be relying on a BIG-IP profile to handle the application parsing, we'll be using the low level TCP events to capture the requests and responses. When a client establishes a connection, the CLIENT_ACCPETED event will occur, from within this event we'll have to trigger a collection of the TCP data so that we can inspect it in the CLIENT_DATA event. when CLIENT_ACCEPTED { if { $::DEBUG } { log local0. "client accepted" } TCP::collect TCP::release } In the CLIENT_DATA event, we will look at the request with the TCP::payload command. We will then feed that value into a switch statement with options for each of the commands. For commands that are found that we want to disallow, we will issue an FTP error response code with description string, empty out the payload, and return from the iRule - thus breaking the connection. For all other cases, we allow the TCP engine to continue on with it's processing and then enter into data collect mode again. when CLIENT_DATA { if { $::DEBUG } { log local0. "----------------------------------------------------------" } if { $::DEBUG } { log local0. "payload [TCP::payload]" } set client_data [string trim [TCP::payload]] #--------------------------------------------------- # Block or alert specific commands #--------------------------------------------------- switch -glob $client_data { "USER anonymous*" - "USER ftp*" { if { $::DEBUG } { log local0. "LOG: Anonymous login detected" } if { $::sec_block_anonymous_ftp } { TCP::respond "530 Guest user not allowed\r\n"; reject } } "PASV*" { if { $::DEBUG } { log local0. "LOG: passive request detected" } if { $::sec_block_passive_ftp } { TCP::respond "502 Passive commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } "RETR*" { if { $::DEBUG } { log local0. "LOG: RETR request detected" } if { $::sec_block_retr_cmd } { TCP::respond "550 RETR commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } "STOR*" { if { $::DEBUG } { log local0. "LOG: STOR request detected" } if { $::sec_block_stor_cmd } { TCP::respond "550 STOR commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } "RNFR*" - "RNTO*" { if { $::DEBUG } { log local0. "LOG: RENAME request detected" } if { $::sec_block_rename_cmd } { TCP::respond "550 RENAME commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } "PORT*" { if { $::DEBUG } { log local0. "LOG: PORT request detected" } if { $::sec_block_port_cmd } { TCP::respond "550 PORT commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } "RMD*" { if { $::DEBUG } { log local0. "LOG: RMD request detected" } if { $::sec_block_rmd_cmd } { TCP::respond "550 RMD commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } "MKD*" { if { $::DEBUG } { log local0. "LOG: MKD request detected" } if { $::sec_block_mkd_cmd } { TCP::respond "550 MKD commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } "LIST*" { if { $::DEBUG } { log local0. "LOG: LIST request detected" } if { $::sec_block_list_cmd } { TCP::respond "550 LIST commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } "PWD*" { if { $::DEBUG } { log local0. "LOG: PWD request detected" } if { $::sec_block_pwd_cmd } { TCP::respond "550 PWD commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } "CWD*" { if { $::DEBUG } { log local0. "LOG: CWD request detected" } if { $::sec_block_cwd_cmd } { TCP::respond "550 CWD commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } "DELE*" { if { $::DEBUG } { log local0. "LOG: DELE request detected" } if { $::sec_block_dele_cmd } { TCP::respond "550 DELE commands not allowed\r\n" TCP::payload replace 0 [string length $client_data] "" return } } } TCP::release TCP::collect } Once a connection has been made to the backend server, the SERVER_CONNECTED event will be raised. In this event we will release the context and issue a collect to occur for the server data. The server data will then be returned, and optionally logged, in the SERVER_DATA event. when SERVER_CONNECTED { if { $::DEBUG } { log "server connected" } TCP::release TCP::collect } when SERVER_DATA { if { $::DEBUG } { log local0. "payload <[TCP::payload]>" } TCP::release TCP::collect } And finally when the client closes it's connection,. the CLIENT_CLOSED event will be fired and we will log the fact that the session is over. when CLIENT_CLOSED { if { $::DEBUG } { log local0. "client closed" } } Conclusion This article shows how one can use iRules to inspect, and optionally secure, an application based on command sets within that application. Not all application protocols behave like FTP (TELNET for instance sends one character at a time and it's up to the proxy to consecutively request more data until the request is complete). But this should give you the tools you need to start inspection on your TCP based application. Get the Flash Player to see this player.
