Controlling a Pool Members Ratio and Priority Group with iControl
A Little Background A question came in through the iControl forums about controlling a pool members ratio and priority programmatically. The issue really involves how the API’s use multi-dimensional arrays but I thought it would be a good opportunity to talk about ratio and priority groups for those that don’t understand how they work. In the first part of this article, I’ll talk a little about what pool members are and how their ratio and priorities apply to how traffic is assigned to them in a load balancing setup. The details in this article were based on BIG-IP version 11.1, but the concepts can apply to other previous versions as well. Load Balancing In it’s very basic form, a load balancing setup involves a virtual ip address (referred to as a VIP) that virtualized a set of backend servers. The idea is that if your application gets very popular, you don’t want to have to rely on a single server to handle the traffic. A VIP contains an object called a “pool” which is essentially a collection of servers that it can distribute traffic to. The method of distributing traffic is referred to as a “Load Balancing Method”. You may have heard the term “Round Robin” before. In this method, connections are passed one at a time from server to server. In most cases though, this is not the best method due to characteristics of the application you are serving. Here are a list of the available load balancing methods in BIG-IP version 11.1. Load Balancing Methods in BIG-IP version 11.1 Round Robin: Specifies that the system passes each new connection request to the next server in line, eventually distributing connections evenly across the array of machines being load balanced. This method works well in most configurations, especially if the equipment that you are load balancing is roughly equal in processing speed and memory. Ratio (member): Specifies that the number of connections that each machine receives over time is proportionate to a ratio weight you define for each machine within the pool. Least Connections (member): Specifies that the system passes a new connection to the node that has the least number of current connections in the pool. This method works best in environments where the servers or other equipment you are load balancing have similar capabilities. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the current number of connections per node or the fastest node response time. Observed (member): Specifies that the system ranks nodes based on the number of connections. Nodes that have a better balance of fewest connections receive a greater proportion of the connections. This method differs from Least Connections (member), in that the Least Connections method measures connections only at the moment of load balancing, while the Observed method tracks the number of Layer 4 connections to each node over time and creates a ratio for load balancing. This dynamic load balancing method works well in any environment, but may be particularly useful in environments where node performance varies significantly. Predictive (member): Uses the ranking method used by the Observed (member) methods, except that the system analyzes the trend of the ranking over time, determining whether a node's performance is improving or declining. The nodes in the pool with better performance rankings that are currently improving, rather than declining, receive a higher proportion of the connections. This dynamic load balancing method works well in any environment. Ratio (node): Specifies that the number of connections that each machine receives over time is proportionate to a ratio weight you define for each machine across all pools of which the server is a member. Least Connections (node): Specifies that the system passes a new connection to the node that has the least number of current connections out of all pools of which a node is a member. This method works best in environments where the servers or other equipment you are load balancing have similar capabilities. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the number of current connections per node, or the fastest node response time. Fastest (node): Specifies that the system passes a new connection based on the fastest response of all pools of which a server is a member. This method might be particularly useful in environments where nodes are distributed across different logical networks. Observed (node): Specifies that the system ranks nodes based on the number of connections. Nodes that have a better balance of fewest connections receive a greater proportion of the connections. This method differs from Least Connections (node), in that the Least Connections method measures connections only at the moment of load balancing, while the Observed method tracks the number of Layer 4 connections to each node over time and creates a ratio for load balancing. This dynamic load balancing method works well in any environment, but may be particularly useful in environments where node performance varies significantly. Predictive (node): Uses the ranking method used by the Observed (member) methods, except that the system analyzes the trend of the ranking over time, determining whether a node's performance is improving or declining. The nodes in the pool with better performance rankings that are currently improving, rather than declining, receive a higher proportion of the connections. This dynamic load balancing method works well in any environment. Dynamic Ratio (node) : This method is similar to Ratio (node) mode, except that weights are based on continuous monitoring of the servers and are therefore continually changing. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the number of current connections per node or the fastest node response time. Fastest (application): Passes a new connection based on the fastest response of all currently active nodes in a pool. This method might be particularly useful in environments where nodes are distributed across different logical networks. Least Sessions: Specifies that the system passes a new connection to the node that has the least number of current sessions. This method works best in environments where the servers or other equipment you are load balancing have similar capabilities. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the number of current sessions. Dynamic Ratio (member): This method is similar to Ratio (node) mode, except that weights are based on continuous monitoring of the servers and are therefore continually changing. This is a dynamic load balancing method, distributing connections based on various aspects of real-time server performance analysis, such as the number of current connections per node or the fastest node response time. L3 Address: This method functions in the same way as the Least Connections methods. We are deprecating it, so you should not use it. Weighted Least Connections (member): Specifies that the system uses the value you specify in Connection Limit to establish a proportional algorithm for each pool member. The system bases the load balancing decision on that proportion and the number of current connections to that pool member. For example,member_a has 20 connections and its connection limit is 100, so it is at 20% of capacity. Similarly, member_b has 20 connections and its connection limit is 200, so it is at 10% of capacity. In this case, the system select selects member_b. This algorithm requires all pool members to have a non-zero connection limit specified. Weighted Least Connections (node): Specifies that the system uses the value you specify in the node's Connection Limitand the number of current connections to a node to establish a proportional algorithm. This algorithm requires all nodes used by pool members to have a non-zero connection limit specified. Ratios The ratio is used by the ratio-related load balancing methods to load balance connections. The ratio specifies the ratio weight to assign to the pool member. Valid values range from 1 through 100. The default is 1, which means that each pool member has an equal ratio proportion. So, if you have server1 a with a ratio value of “10” and server2 with a ratio value of “1”, server1 will get served 10 connections for every one that server2 receives. This can be useful when you have different classes of servers with different performance capabilities. Priority Group The priority group is a number that groups pool members together. The default is 0, meaning that the member has no priority. To specify a priority, you must activate priority group usage when you create a new pool or when adding or removing pool members. When activated, the system load balances traffic according to the priority group number assigned to the pool member. The higher the number, the higher the priority, so a member with a priority of 3 has higher priority than a member with a priority of 1. The easiest way to think of priority groups is as if you are creating mini-pools of servers within a single pool. You put members A, B, and C in to priority group 5 and members D, E, and F in priority group 1. Members A, B, and C will be served traffic according to their ratios (assuming you have ratio loadbalancing configured). If all those servers have reached their thresholds, then traffic will be distributed to servers D, E, and F in priority group 1. he default setting for priority group activation is Disabled. Once you enable this setting, you can specify pool member priority when you create a new pool or on a pool member's properties screen. The system treats same-priority pool members as a group. To enable priority group activation in the admin GUI, select Less than from the list, and in the Available Member(s) box, type a number from 0 to 65535 that represents the minimum number of members that must be available in one priority group before the system directs traffic to members in a lower priority group. When a sufficient number of members become available in the higher priority group, the system again directs traffic to the higher priority group. Implementing in Code The two methods to retrieve the priority and ratio values are very similar. They both take two parameters: a list of pools to query, and a 2-D array of members (a list for each pool member passed in). long [] [] get_member_priority( in String [] pool_names, in Common__AddressPort [] [] members ); long [] [] get_member_ratio( in String [] pool_names, in Common__AddressPort [] [] members ); The following PowerShell function (utilizing the iControl PowerShell Library), takes as input a pool and a single member. It then make a call to query the ratio and priority for the specific member and writes it to the console. function Get-PoolMemberDetails() { param( $Pool = $null, $Member = $null ); $AddrPort = Parse-AddressPort $Member; $RatioAofA = (Get-F5.iControl).LocalLBPool.get_member_ratio( @($Pool), @( @($AddrPort) ) ); $PriorityAofA = (Get-F5.iControl).LocalLBPool.get_member_priority( @($Pool), @( @($AddrPort) ) ); $ratio = $RatioAofA[0][0]; $priority = $PriorityAofA[0][0]; "Pool '$Pool' member '$Member' ratio '$ratio' priority '$priority'"; } Setting the values with the set_member_priority and set_member_ratio methods take the same first two parameters as their associated get_* methods, but add a third parameter for the priorities and ratios for the pool members. set_member_priority( in String [] pool_names, in Common::AddressPort [] [] members, in long [] [] priorities ); set_member_ratio( in String [] pool_names, in Common::AddressPort [] [] members, in long [] [] ratios ); The following Powershell function takes as input the Pool and Member with optional values for the Ratio and Priority. If either of those are set, the function will call the appropriate iControl methods to set their values. function Set-PoolMemberDetails() { param( $Pool = $null, $Member = $null, $Ratio = $null, $Priority = $null ); $AddrPort = Parse-AddressPort $Member; if ( $null -ne $Ratio ) { (Get-F5.iControl).LocalLBPool.set_member_ratio( @($Pool), @( @($AddrPort) ), @($Ratio) ); } if ( $null -ne $Priority ) { (Get-F5.iControl).LocalLBPool.set_member_priority( @($Pool), @( @($AddrPort) ), @($Priority) ); } } In case you were wondering how to create the Common::AddressPort structure for the $AddrPort variables in the above examples, here’s a helper function I wrote to allocate the object and fill in it’s properties. function Parse-AddressPort() { param($Value); $tokens = $Value.Split(":"); $r = New-Object iControl.CommonAddressPort; $r.address = $tokens[0]; $r.port = $tokens[1]; $r; } Download The Source The full source for this example can be found in the iControl CodeShare under PowerShell PoolMember Ratio and Priority.The BIG-IP Application Security Manager Part 1: What is the ASM?
tl;dr - BIG-IP Application Security Manager (ASM) is a layer 7 web application firewall (WAF) available on F5's BIG-IP platforms. Introduction This article series was written a while back, but we are re-introducing it as a part of our Security Month on DevCentral. I hope you enjoy all the features of this very powerful module on the BIG-IP! This is the first of a 10-part series on the BIG-IP ASM. This module is a very powerful and effective tool for defending your applications and your peace of mind, but what is it really? And, how do you configure it correctly and efficiently? How can you take advantage of all the features it has to offer? Well, the purpose of this article series is to answer these fundamental questions. So, join me as we dive into this really cool technology called the BIG-IP ASM! The Basics The BIG-IP ASM is a Layer 7 ICSA-certified Web Application Firewall (WAF) that provides application security in traditional, virtual, and private cloud environments. It is built on TMOS...the universal product platform shared by all F5 BIG-IP products. It can run on any of the F5 Application Delivery Platforms...BIG-IP Virtual Edition, BIG-IP 2000 -> 11050, and all the VIPRION blades. It protects your applications from a myriad of network attacks including the OWASP Top 10 most critical web application security risks It is able to adapt to constantly-changing applications in very dynamic network environments It can run standalone or integrated with other modules like BIG-IP LTM, BIG-IP DNS, BIG-IP APM, etc Why A Layer 7 Firewall? Traditional network firewalls (Layer 3-4) do a great job preventing outsiders from accessing internal networks. But, these firewalls offer little to no support in the protection of application layer traffic. As David Holmes points out in his article series on F5 firewalls, threat vectors today are being introduced at all layers of the network. For example, the Slowloris and HTTP Flood attacks are Layer 7 attacks...a traditional network firewall would never stop these attacks. But, nonetheless, your application would still go down if/when it gets hit by one of these. So, it's important to defend your network with more than just a traditional Layer 3-4 firewall. That's where the ASM comes in... Some Key Features The ASM comes pre-loaded with over 2,200 attack signatures. These signatures form the foundation for the intelligence used to allow or block network traffic. If these 2,200+ signatures don't quite do the job for you, never fear...you can also build your own user-defined signatures. And, as we all know, network threats are always changing so the ASM is configured to download updated attack signatures on a regular basis. Also, the ASM offers several different policy building features. Policy building can be difficult and time consuming, especially for sites that have a large number of pages. For example, DevCentral has over 55,000 pages...who wants to hand-write the policy for that?!? No one has that kind of time. Instead, you can let the system automatically build your policy based on what it learns from your application traffic, you can manually build a policy based on what you know about your traffic, or you can use external security scanning tools (WhiteHat Sentinel, QualysGuard, IBM AppScan, Cenzic Hailstorm, etc) to build your policy. In addition, the ASM comes configured with pre-built policies for several popular applications (SharePoint, Exchange, Oracle Portal, Oracle Application, Lotus Domino, etc). Did you know? The BIG-IP ASM was the first WAF to integrate with a scanner. WhiteHat approached all the WAFs and asked about the concept of building a security policy around known vulnerabilities in the apps. All the other WAFs said "no"...F5 said "of course!" and thus began the first WAF-scanner integration. The ASM also utilizes Geolocation and IP address intelligence to allow for more sophisticated and targeted defense measures. You can allow/block users from specific locations around the world, and you can block IP addresses that have built a bad reputation on other sites around the Internet. If they were doing bad things on some other site, why let them access yours? The ASM is also built for Payment Card Industry Data Security Standard (PCI DSS) compliance. In fact, you can generate a real-time PCI compliance report at the click of a button! The ASM also comes loaded with the DataGuard feature that automatically blocks sensitive data (Credit Card numbers, SSN, etc) from being displayed in a browser. In addition to the PCI reports, you can generate on-demand charts and graphs that show just about every detail of traffic statistics that you need. The following screenshot is a representative sample of some real traffic that I pulled off a site that uses the ASM. Pretty powerful stuff! I could go on for days here...and I know you probably want me to, but I'll wrap it up for this first article. I hope you can see the value of the ASM both as a technical solution in the defense of your network and also a critical asset in the long-term strategic vision of your company. So, if you already have an ASM and want to know more about it or if you don't have one yet and want to see what you're missing, come on back for the next article where I will talk about the cool features of policy building. What is the BIG-IP ASM? Policy Building The Importance of File Types, Parameters, and URLs Attack Signatures XML Security IP Address Intelligence and Whitelisting Geolocation Data Guard Username and Session Awareness Tracking Event LoggingSSL Profiles Part 8: Client Authentication
This is the eighth article in a series of Tech Tips that highlight SSL Profiles on the BIG-IP LTM. SSL Overview and Handshake SSL Certificates Certificate Chain Implementation Cipher Suites SSL Options SSL Renegotiation Server Name Indication Client Authentication Server Authentication All the "Little" Options This article will discuss the concept of Client Authentication, how it works, and how the BIG-IP system allows you to configure it for your environment. Client Authentication In a TLS handshake, the client and the server exchange several messages that ultimately result in an encrypted channel for secure communication. During this handshake, the client authenticates the server's identity by verifying the server certificate (for more on the TLS handshake, see SSL Overview and Handshake - Article 1in this series). Although the client always authenticates the server's identity, the server is not required to authenticate the client's identity. However, there are some situations that call for the server to authenticate the client. Client authentication is a feature that lets you authenticate users that are accessing a server. In client authentication, a certificate is passed from the client to the server and is verified by the server. Client authentication allow you to rest assured that the person represented by the certificate is the person you expect. Many companies want to ensure that only authorized users can gain access to the services and content they provide. As more personal and access-controlled information moves online, client authentication becomes more of a reality and a necessity. How Does Client Authentication Work? Before we jump into client authentication, let's make sure we understand server authentication. During the TLS handshake, the client authenticates the identity of the server by verifying the server's certificate and using the server's public key to encrypt data that will be used to compute the shared symmetric key. The server can only generate the symmetric key used in the TLS session if it can decrypt that data with its private key. The following diagram shows an abbreviated version of the TLS handshake that highlights some of these concepts. Ultimately, the client and server need to use a symmetric key to encrypt all communication during their TLS session. In order to calculate that key, the server shares its certificate with the client (the certificate includes the server's public key), and the client sends a random string of data to the server (encrypted with the server's public key). Now that the client and server each have the random string of data, they can each calculate (independently) the symmetric key that will be used to encrypt all remaining communication for the duration of that specific TLS session. In fact, the client and server both send a "Finished' message at the end of the handshake...and that message is encrypted with the symmetric key that they have both calculated on their own. So, if all that stuff works and they can both read each other's "Finished" message, then the server has been authenticated by the client and they proceed along with smiles on their collective faces (encrypted smiles, of course). You'll notice in the diagram above that the server sent its certificate to the client, but the client never sent its certificate to the server. When client authentication is used, the server still sends its certificate to the client, but it also sends a "Certificate Request" message to the client. This lets the client know that it needs to get its certificate ready because the next message from the client to the server (during the handshake) will need to include the client certificate. The following diagram shows the added steps needed during the TLS handshake for client authentication. So, you can see that when client authentication is enabled, the public and private keys are still used to encrypt and decrypt critical information that leads to the shared symmetric key. In addition to the public and private keys being used for authentication, the client and server both send certificates and each verifies the certificate of the other. This certificate verification is also part of the authentication process for both the client and the server. The certificate verification process includes four important checks. If any of these checks do not return a valid response, the certificate verification fails (which makes the TLS handshake fail) and the session will terminate. These checks are as follows: Check digital signature Check certificate chain Check expiration date and validity period Check certificate revocation status Here's how the client and server accomplish each of the checks for client authentication: Digital Signature: The client sends a "Certificate Verify" message that contains a digitally signed copy of the previous handshake message. This message is signed using the client certificate's private key. The server can validate the message digest of the digital signature by using the client's public key (which is found in the client certificate). Once the digital signature is validated, the server knows that public key belonging to the client matches the private key used to create the signature. Certificate Chain: The server maintains a list of trusted CAs, and this list determines which certificates the server will accept. The server will use the public key from the CA certificate (which it has in its list of trusted CAs) to validate the CA's digital signature on the certificate being presented. If the message digest has changed or if the public key doesn't correspond to the CA's private key used to sign the certificate, the verification fails and the handshake terminates. Expiration Date and Validity Period: The server compares the current date to the validity period listed in the certificate. If the expiration date has not passed and the current date is within the period, everything is good. If it's not, then the verification fails and the handshake terminates. Certificate Revocation Status: The server compares the client certificate to the list of revoked certificates on the system. If the client certificate is on the list, the verification fails and the handshake terminates. As you can see, a bunch of stuff has to happen in just the right way for the Client-Authenticated TLS handshake to finalize correctly. But, all this is in place for your own protection. After all, you want to make sure that no one else can steal your identity and impersonate you on a critically important website! BIG-IP Configuration Now that we've established the foundation for client authentication in a TLS handshake, let's figure out how the BIG-IP is set up to handle this feature. The following screenshot shows the user interface for configuring Client Authentication. To get here, navigate to Local Traffic > Profiles > SSL > Client. The Client Certificate drop down menu has three settings: Ignore (default), Require, and Request. The "Ignore" setting specifies that the system will ignore any certificate presented and will not authenticate the client before establishing the SSL session. This effectively turns off client authentication. The "Require" setting enforces client authentication. When this setting is enabled, the BIG-IP will request a client certificate and attempt to verify it. An SSL session is established only if a valid client certificate from a trusted CA is presented. Finally, the "Request" setting enables optional client authentication. When this setting is enabled, the BIG-IP will request a client certificate and attempt to verify it. However, an SSL session will be established regardless of whether or not a valid client certificate from a trusted CA is presented. The Request option is often used in conjunction with iRules in order to provide selective access depending on the certificate that is presented. For example: let's say you would like to allow clients who present a certificate from a trusted CA to gain access to the application while clients who do not provide the required certificate be redirected to a page detailing the access requirements. If you are not using iRules to enforce a different outcome based on the certificate details, there is no significant benefit to using the "Request" setting versus the default "Ignore" setting. In both cases, an SSL session will be established regardless of the certificate presented. Frequency specifies the frequency of client authentication for an SSL session. This menu offers two options: Once (default) and Always. The "Once" setting specifies that the system will authenticate the client only once for an SSL session. The "Always"setting specifies that the system will authenticate the client once when the SSL session is established as well as each time that session is reused. The Retain Certificate box is checked by default. When checked, the client certificate is retained for the SSL session. Certificate Chain Traversal Depth specifies the maximum number of certificates that can be traversed in a client certificate chain. The default for this setting is 9. Remember that "Certificate Chain" part of the verification checks? This setting is where you configure the depth that you allow the server to dig for a trusted CA. For more on certificate chains, see article 2 of this SSL series. Trusted Certificate Authorities setting is used to specify the BIG-IP's Trusted Certificate Authorities store. These are the CAs that the BIG-IP trusts when it verifies a client certificate that is presented during client authentication. The default value for the Trusted Certificate Authorities setting is None, indicating that no CAs are trusted. Don't forget...if the BIG-IP Client Certificate menu is set to Require but the Trusted Certificate Authorities is set to None, clients will not be able to establish SSL sessions with the virtual server. The drop down list in this setting includes the name of all the SSL certificates installed in the BIG-IP's /config/ssl/ssl.crt directory. A newly-installed BIG-IP system will include the following certificates: default certificate and ca-bundle certificate. The default certificate is a self-signed server certificate used when testing SSL profiles. This certificate is not appropriate for use as a Trusted Certificate Authorities certificate bundle. The ca-bundle certificate is a bundle of CA certificates from most of the well-known PKIs around the world. This certificate may be appropriate for use as a Trusted Certificate Authorities certificate bundle. However, if this bundle is specified as the Trusted Certificate Authorities certificate store, any valid client certificate that is signed by one of the popular Root CAs included in the default ca-bundle.crt will be authenticated. This provides some level of identification, but it provides very little access control since almost any valid client certificate could be authenticated. If you want to trust only certificates signed by a specific CA or set of CAs, you should create and install a bundle containing the certificates of the CAs whose certificates you trust. The bundle must also include the entire chain of CA certificates necessary to establish a chain of trust. Once you create this new certificate bundle, you can select it in the Trusted Certificate Authorities drop down menu. The Advertised Certificate Authorities setting is used to specify the CAs that the BIG-IP advertises as trusted when soliciting a client certificate for client authentication. The default value for the Advertised Certificate Authorities setting is None, indicating that no CAs are advertised. When set to None, no list of trusted CAs is sent to a client with the certificate request. If the Client Certificate menu is set to Require or Request, you can configure the Advertised Certificate Authorities setting to send clients a list of CAs that the server is likely to trust. Like the Trusted Certificate Authorities list, the Advertised Certificate Authorities drop down list includes the name of all the SSL certificates installed in the BIG-IP /config/ssl/ssl.crt directory. A newly-installed BIG-IP system includes the following certificates: default certificate and ca-bundle certificate. The default certificate is a self-signed server certificate used for testing SSL profiles. This certificate is not appropriate for use as an Advertised Certificate Authorities certificate bundle. The ca-bundle certificate is a bundle of CA certificates from most of the well-known PKIs around the world. This certificate may be appropriate for use as an Advertised Certificate Authorities certificate bundle. If you want to advertise only a specific CA or set of CAs, you should create and install a bundle containing the certificates of the CA to advertise. Once you create this new certificate bundle, you can select it in the Advertised Certificate Authorities setting drop down menu. You are allowed to configure the Advertised Certificate Authorities setting to send a different list of CAs than that specified for the Trusted Certificate Authorities. This allows greater control over the configuration information shared with unknown clients. You might not want to reveal the entire list of trusted CAs to a client that does not automatically present a valid client certificate from a trusted CA. Finally, you should avoid specifying a bundle that contains a large number of certificates when you configure the Advertised Certificate Authorities setting. This will cut down on the number of certificates exchanged during a client SSL handshake. The maximum size allowed by the BIG-IP for native SSL handshake messages is 14,304 bytes. Most handshakes don't result in large message lengths, but if the SSL handshake is negotiating a native cipher and the total length of all messages in the handshake exceeds the 14,304 byte threshold, the handshake will fail. The Certificate Revocation List (CRL) setting allows you to specify a CRL that the BIG-IP will use to check revocation status of a certificate prior to authenticating a client. If you want to use a CRL, you must upload it to the /config/ssl/ssl.crl directory on the BIG-IP. The name of the CRL file may then be entered in the CRL setting dialog box. Note that this box will offer no drop down menu options until you upload a CRL file to the BIG-IP. Since CRLs can quickly become outdated, you should use either OCSP or CRLDP profiles for more robust and current verification functionality. Conclusion Well, that wraps up our discussion on Client Authentication. I hope the information helped, and I hope you can use this to configure your BIG-IP to meet the needs of your specific network environment. Be sure to come back for our next article in the SSL series. As always, if you have any other questions, feel free to post a question here or Contact Us directly. See you next time!SSL Profiles Part 7: Server Name Indication
This is the seventh article in a series of Tech Tips that highlight SSL Profiles on the BIG-IP LTM. The other articles are: SSL Overview and Handshake SSL Certificates Certificate Chain Implementation Cipher Suites SSL Options SSL Renegotiation Server Name Indication Client Authentication Server Authentication All the "Little" Options This article will discuss the concept of Server Name Indication (SNI) and how the BIG-IP system allows you to configure it for your environment. What is Server Name Indication? SNI (listed in RFC 4366) is an extension to the TLS protocol that allows the client to include the requested hostname in the first message of the SSL handshake (Client Hello). This allows the server to determine the correct named host for the request and setup the connection accordingly from the start. Prior to the introduction of SNI, the client could not establish secure connections to multiple virtual servers hosted on a single IP address. This happened because the destination server name can only be decoded from the HTTP request header after the SSL connection has been established. As you can see from the following diagram (taken from Jason Rahm's first articlein this series), the standard TLS handshake involves several messages between the client and the server. The server sends the certificate (step 3) to the client long before the handshake is complete. If a web server is used to host multiple DNS hostnames on a single IP address, the certificate passing between the server and the client could get problematic. Using the standard TLS protocol, the server might send the wrong certificate to the client because it does not yet know which certificate the client is looking for. Then, if the client receives a certificate with the wrong name, it will either abort the connection (assuming a Man-in-the-Middle attack) or at least display a warning page to the user stating that there is a problem with the certificate. This scenario is shown in the following figure. A client wants to visit https://www.securesite1.com, but the virtual web server doesn't know to pass the certificate for securesite1.com until it finishes the TLS handshake and reads the HTTP request header from the client. This is where SNI comes in really handy. With the introduction of SNI, the client can indicate the name of the server to which he is attempting to connect as part of the "Client Hello" message in the handshake process. The server then uses this information to select the appropriate certificate to return to the client when it sends back the "Server Hello" packet during the handshake. This allows a server to present multiple certificates on the same IP address and port number and thus allows multiple secure (HTTPS) sites to be served off the same IP address without requiring all the sites to use the same certificate. Having this extension available (per RFC 4366) is a great feature, but it does no good if a client is using a browser that doesn't send the correct SNI message to the server. So, to make SNI practical, the vast majority of your users must use web browsers that support it. Browsers that do not support SNI will be presented with the server's default certificate and are likely to receive a certificate warning. According to Wikipedia, the following browsers support SNI: Internet Explorer 7 or later, on Windows Vista or higher. Does not work on Windows XP, even Internet Explorer 8 (the support of this feature is not browser version dependent, it depends on SChannel system component which introduced the support of TLS SNI extension, starting from Windows Vista, not XP). Mozilla Firefox 2.0 or later Opera 8.0 (2005) or later (the TLS 1.1 protocol must be enabled) Opera Mobile at least version 10.1 beta on Android Google Chrome (Vista or higher. XP on Chrome 6 or newer. OS X 10.5.7 or higher on Chrome 5.0.342.1 or newer) Safari 3.0 or later (Mac OS X 10.5.6 or higher and Windows Vista or higher) Konqueror/KDE 4.7 or later MobileSafari in Apple iOS 4.0 or later Android default browser on Honeycomb (v3.x) or newer Windows Phone 7 MicroB on Maemo Odyssey on MorphOS Now that you know the background of SNI, let's dig into how the BIG-IP is set up for SNI configuration. BIG-IP Configuration Beginning in v11.1.0, the BIG-IP allows you to assign multiple SSL profiles to a virtual server for supporting the use of the TLS SNI feature. The TLS SNI feature is not available in previous BIG-IP versions, so you'll want to upgrade if you are not on v11.1.0 or higher! To support this feature, a virtual server must be assigned a default SSL profile for fallback as well as one SSL profile per HTTPS site. The fallback SSL profile is used when the server name does not match the client request or when the browser does not support the SNI extensions. Using the example from the figures above, suppose you need to host the three domains securesite1.com, securesite2.com, and securesite3.com on the same HTTPS virtual server. Each domain has its own server certificate to use, such as securesite1.crt, securesite2.crt, and securesite3.crt, and each has different security requirements. To ensure that the BIG-IP presents the correct certificate to the browser, you enable SNI, which sends the name of a domain as part of the TLS negotiation. To enable SNI, you configure the Server Name and other settings on an SSL profile, and then assign the profile to a virtual server. For SSL profiles (Client and Server), you type the name for the HTTPS site in the Server Name box. SNI configuration is found by navigating to Local Traffic > Profiles > SSL > Client | Server. The following screenshot shows the three settings used for SNI in the BIG-IP. Server Name specifies the fully qualified DNS hostname of the server that is used in SNI communications. Using the server name, the Local Traffic Manager can choose from multiple SSL profiles prior to the SSL Handshake. If no value is specified, the system uses the Common Name value from the default certificate. The default for this setting leaves the name blank. Default SSL Profile for SNI indicates that the system uses this profile as the default SSL profile when there is no match to the server name, or when the client provides no SNI extension support. Note that when assigning multiple SSL profiles to a single virtual server, you can enable this setting on one Client SSL profile only and on one Server SSL profile only. The default for this setting is unchecked. Require Peer SNI support requires that all network peers provide SNI support as well. If you enable both "Default SSL Profile for SNI" and "Require Peer SNI Support," the system terminates the connection when the client provides no SNI extension. The default for this setting is unchecked. Conclusion I hope this helps with setting up this great feature in your environment. SNI is a powerful tool, and it could go a long way in saving you precious IP addresses for your secure sites! As we noted before, this feature is only supported in BIG-IP version 11.1.0 and later. If you are using a version prior to 11.1.0, you can read this articleon DevCentral that shows how you can use an iRule to take advantage of SNI on previous versions. Finally, you can read more on our DevCentral Wiki about using iRules for SNI. Well, thanks for reading about SNI. Be sure to come back for the next article in the SSL series where I will talk about SSL Forward Proxy. See you then!SSL Profiles Part 6: SSL Renegotiation
This is the sixth article in a series of Tech Tips that highlight SSL Profiles on the BIG-IP LTM. SSL Overview and Handshake SSL Certificates Certificate Chain Implementation Cipher Suites SSL Options SSL Renegotiation Server Name Indication Client Authentication Server Authentication All the "Little" Options This article will discuss SSL renegotiation, some OpenSSL features, some of the attack vectors against SSL renegotiation, and then how the BIG-IP is configured to handle this feature. What Is SSL Renegotiation? Before we get into the details of SSL renegotiation, let me start by outlining a quick scenario. Let's say a user visits an online shopping site that is SSL encrypted. Initially, the user shops around anonymously, but after some time on the site, the user decides to log in and make a purchase. The user will need to authenticate to the site before making the purchase, so the SSL connection will need to be adjusted to allow the authentication. Ideally, all the information the user collected as an anonymous visitor will be maintained and protected after the status change to an authenticated customer. This basic (yet frequent) scenario outlines the need for SSL renegotiation. The figure below shows a representation of this scenario and shows that a new SSL session must be established in order to authenticate the user. This new session completes the full handshake, but it does it using the security of the already-established SSL connection. SSL renegotiation messages (including types of ciphers and encryption keys) are encrypted and then sent over the existing SSL connection. This feature is nice because it can create a secure SSL session based on the fact that a secure connection is already established. SSL renegotiation is useful in several situations where an ordinary SSL session is already established. Here are a few examples: Client authentication is required A different set of encryption and decryption keys are needed A different set of encryption and hashing algorithms are used The client or the server can request renegotiation at any time. For the client to request renegotiation, the client sends a "Client Hello" message in the already-established encrypted channel and the server responds with a "Server Hello" and then the negotiation follows the normal handshake process. The server can initiate the renegotiation by sending the client a Hello Request message. When the client receives the request, the client sends the "Client Hello" message and the handshake process takes place. One quick note of clarification: the client and server can also complete a session resumption vice a session renegotiation. A session resumption basically resumes a previous session (using a previous session ID), and it saves the client and server the computing work of generating new encryption keys. Renegotiation is a process by which the full handshake process takes place over an already existing SSL connection. A Fly In The Ointment... Life would be much easier if the bad guys weren't out there doing their thing, but I guess it would also be less interesting. Nonetheless, some pretty smart guys were working with SSL renegotiation back in the old days (circa 2009) and found out that they could exploit a vulnerability in its design. The Common Vulnerabilities and Exposures (CVE) database outlines the details behind this SSL renegotiation vulnerability in CVE-2009-3555. You can read the details for yourself, but here's what the CVE basically says: TLS and SSLv3 do not properly associate renegotiation handshakes with an existing connection, and this allows attackers to insert data into existing sessions by sending an unauthenticated request that is processed retroactively by a server in a post-renegotiation context related to a "plaintext injection" attack. Essentially, this means that bad guys can hijack the SSL session and start using your "secure" communications on your behalf. Houston, we have a problem. Fortunately, the good folks at the Internet Engineering Task Force (IETF) found out about this problem, and they responded with a little magic of their own. It's outlined in Request for Comments (RFC) 5746. Here's what they had to say: "Secure Socket Layer (SSL) and Transport Layer Security (TLS) renegotiation are vulnerable to an attack in which the attacker forms a TLS connection with the target server, injects content of his choice, and then splices in a new TLS connection from a client. The server treats the client's initial TLS handshake as a renegotiation and thus believes that the initial data transmitted by the attacker is from the same entity as the subsequent client data. This specification defines a TLS extension to cryptographically tie renegotiations to the TLS connections they are being performed over, thus preventing this attack." Well, take that bad guys!! A Few Notes About OpenSSL Now that we have a basic understanding of RFC 5746, let's spend some time on OpenSSL and how it relates to F5's BIG-IP. After all, how can you have an article on SSL and not talk about OpenSSL? This stuff is free (Apache style) and it's awesome! The BIG-IP supports the popular OpenSSL toolkit, and it even has a built-in OpenSSL utility. It's important to understand how OpenSSL handles the behavior of clients and servers regarding the feature of SSL renegotiation. OpenSSL 0.9.8m and later always attempts to use secure renegotiation as described in RFC 5746. On the BIG-IP side of the house, versions prior to 10.2.3 do not support this RFC. However, beginning in version 10.2.3 and 11.x, the BIG-IP SSL profiles do support it. So, pause for a few minutes from reading this article and upgrade your BIG-IP! Patching When describing SSL implementations, systems that support secure renegotiation (per RFC 5746) are referred to as "patched" and systems that do not support secure renegotiation are referred to as "unpatched." If a client and server are both patched, connections and renegotiation are always permitted by OpenSSL implementations. If a server is patched but the client is unpatched, the initial connection will succeed but client renegotiation will be denied by the server with a no_renegotiation warning alert if TLS v1.0 is used or a fatal handshake_failure alert if SSL v3.0 is used. If the patched server attempts to renegotiate with the unpatched client, a fatal handshake_failure alert is sent. This is because the server code may be unaware of the unpatched nature of the client. Here's what happens if a client is patched but the server is unpatched. If the option "SSL_OP_LEGACY_SERVER_CONNECT" or "SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION" is set then initial connections as well as all renegotiations between patched clients and unpatched servers will succeed. However, if neither option is set then initial connections to unpatched servers will fail. (note: the option "SSL_OP_LEGACY_SERVER_CONNECT" is currently set by default, but as more servers become patched, this option will not be set by default in future versions of OpenSSL). Renegotiation Attacks Speaking of highly broken...let's spend a little time on SSL renegotiation attacks before we dive into the BIG-IP configurations. Here are two fairly well known attacks that are targeted at the SSL renegotiation feature. And, guess what? I'm starting with the..."Man in the Middle" (sung to Michael Jackson's hit song) Man In The Middle Since we discussed RFC 5746 earlier, why not go over the details of what it fixed? As we've discussed, SSL allows both servers and clients to initiate a complete renegotiation of the encryption parameters used for SSL connections. However, the design of the initial handshake and the renegotiation process had a gap that could allow an attacker to send a “Client Hello” and insert content into a client’s SSL session and then intercept secure communications between the server and the client. The following illustration shows how a bad guy could exploit the Man in the Middle vulnerability. In this example, the client sends a "Client Hello" to start an SSL session with his bank's server. The bad guy temporarily blocks the "Client Hello" and then creates an SSL connection of his own with the server. After the SSL session is created between the bad guy and the bank server, the bad guy asks for a money transfer. The bank server needs to authenticate the user, so it requests a renegotiation by sending a "Hello Request" to the bad guy...which is exactly what he wanted. This is where the bad guy sends the client's initial "Client Hello" to the server. The client thinks he is beginning a new SSL session, and the server thinks an SSL renegotiation has just taken place. But, nonetheless, all parties are happy with everything...especially the bad guy! See, now you know why you should be using a "patched" system! Denial of Service Last year, the IETF posted a paper that outlines a Denial of Service (DoS) attack against a server using the SSL renegotiation feature. The DoS attack is based on the fact that an SSL handshake requires up to 10 times more processing power on the server than on the client. Therein lies the problem...and the ability to exploit it! When SSL renegotiation is enabled on the server, a user is allowed to send a renegotiation request any time he wants...and this request initiates a new handshake. Since it takes much fewer resources for a client to perform a handshake than a server, the client can request multiple handshakes per second and cause a DoS on the server side SSL interface. Therefore, if a malicious user requests multiple renegotiation requests it will exhaust the server’s resources and not allow any other user to establish a connection. This attack is different than a Distributed Denial of Service (DDoS) as it does not require a large amounts of network volume to exhaust the network connection. Instead, it exhausts the server resources from a single host requiring only a single TCP/IP socket. A single server can perform between 150-300 handshakes per second while a single client can request up to 1,000 handshakes per second. By the way, Openssl doesn't consider this DoS exploit an actual vulnerability (which is why they haven't done anything about it in their latest releases). Their response is that this exploit is a way of DoS'ing a server by requesting lots of expensive crypto operations. Any app that allows an operation like SSL renegotiation to an untrusted host is subject to resource exhaustion. They would say the correct answer is to restrict hosts if this is an issue. Note also that if the OS firewall is not enabled, any OS is subject to a DoS through resource exhaustion, even if it's just TCP port exhaustion. For more information on a great SSL renegotiation DoS countermeasure, see this DevCentral article. David Holmes and Jason Rahm teamed up to talk about the DoS vulnerability, and then they wrote an iRule that elegantly defeats the attack...really cool and useful stuff!! In addition, here's a quick video clip that talks about the same issue...F5 knocks it out of the park with this iRule stuff! F5 Security Vignette: SSL Renegotiation BIG-IP Configuration Now that we've talked through this SSL renegotiation thing, let's finish up with the features of the BIG-IP and how it handles SSL renegotiation. The following screenshot shows the different options for SSL renegotiation on the BIG-IP. You can find this screen by navigating to Local Traffic > Profiles > SSL > Client | Server. Notice that you can either enable or disable SSL renegotiation altogether. By default, it is enabled. Renegotiation controls (on a per-connection basis) how the BIG-IP responds to mid-stream SSL reconnection requests. When enabled, the system processes mid-stream SSL renegotiation requests. When disabled, the system terminates the connection, or ignores the request, depending on system configuration. The Renegotiate Period setting specifies the number of seconds from the initial connect time that the system renegotiates an SSL session. The options are "Specify" (in seconds) and "Indefinite" with "Indefinite" being the default. The "Indefinite" setting means that you never want the system to force an SSL renegotiation. The system will renegotiate the session after the specified period of time...that's why the "Indefinite" setting never forces a renegotiation. For example, setting the renegotiate period to 3600 seconds triggers session renegotiation at least once an hour. The Renegotiate Size setting forces the BIG-IP to renegotiate an SSL session after a specified number of megabytes of application data have been transmitted over the secure channel. The values for this setting are "Indefinite" and "Specify" (in megabytes) with "Indefinite" being the default. This setting follows the same premise as the Renegotiate Period but it uses data size to determine when it will force a renegotiation vice a period of time. The Renegotiate Max Record Delay setting forces the BIG-IP to terminate an SSL session after receiving the specified maximum number of SSL records. The number of records are counted while the BIG-IP system waits for the client to initiate a renegotiation. If the BIG-IP system receives more than the maximum number of SSL records, it closes the connection. The values for this setting are "Indefinite" and "Specify" (number of records). The default setting is "Specify" 10 records. Secure Renegotiation is a fairly new feature in the BIG-IP (not available prior to v10.2.3). This setting handles the RFC 5746 problem, and it allows the user to specify the method of secure renegotiation for SSL connections. The values for this setting are "Request," "Require," and "Require Strict" with "Require" being the default value for the Client-side SSL profile. The default value for the Server-side SSL profile depends on the BIG-IP version: in v10.2.3 and v10.2.4, the default value for the Server-side SSL profile is "Require" and the default value in v11.x is "Require Strict." The "Request" value specifies that the system simply requests secure renegotiation of SSL connections. The "Require" value specifies that the system requires secure renegotiation of SSL connections. In this mode, the system permits initial SSL handshakes from clients, but terminates renegotiations from unpatched clients. The "Require Strict" value specifies that the system requires strict secure renegotiation of SSL connections. In this mode, the system does not accept new SSL connections from unpatched clients. Within the context of the Server-side SSL profile, there is no behavioral difference between "Require" and "Require Strict." In either mode, initial SSL connections from the BIG-IP to unpatched servers will fail. Conclusion Well, I think that about does it for SSL Renegotiation. Come back for more great SSL discussion in our next article that will feature Server Name Indication (SNI). As always, if you have any questions (about SSL, iRules, or any other F5 technology), feel free to comment here or contact the DevCentral team directly. See you soon!Accessing TCP Options from iRules
I’ve written several articles on the TCP profile and enjoy digging into TCP. It’s a beast, and I am constantly re-learning the inner workings. Still etched in my visual memory map, however, is the TCP header format, shown in Figure 1 below. Since 9.0 was released, TCP payload data (that which comes after the header) has been consumable in iRules via the TCP::payload and the port information has been available in the contextual commands TCP::local_port/TCP::remote_port and of course TCP::client_port/TCP::server_port. Options, however, have been inaccessible. But beginning with version 10.2.0-HF2, it is now possible to retrieve data from the options fields. Preparing the BIG-IP Prior to version 11.0, it was necessary to set a bigpipe database key with the option (or options) of interest: bigpipe db Rules.Tcpoption.settings [option, first|last], [option, first|last] In version 11.0 and forward, the DB keys are no more and you need to create a tcp profile with the these options defined, like so: ltm profile tcp tcp_opt { app-service none tcp-options "{option first|last} {option first|last}" } The option is an integer between 2 and 255, and the first/last setting indicates whether the system will retain the first or last instance of the specified option. Once that key is set, you’ll need to do a bigstart restart for it to take (warning: service impacting). Note also that the LTM only collects option data starting with the ACK of a connection. The initial SYN is ignored even if you select the first keyword. This is done to prevent a SYN flood attack (in keeping with SYN-cookies). A New iRules Command: TCP::option The TCP::option command has the following syntax: TCP::option get <option> v11 Additions/Changes: TCP::option set <option number> <value> <next|all> TCP::option noset <option number> Pretty simple, no? So now that you can access them, what fun can be had? Real World Scenario: Akamai In Akamai’s IPA and SXL product lines, they support client IP visibility by embedding a version number (one byte) and an IPv4 address (four bytes) as part of their overlay path feature in tcp option number 28. To access this data, we first set the database key: tmsh create ltm profile tcp tcp_opt tcp-options "{28 first}" Now, the iRule utilizing the TCP::option command: when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt28 cH8 ver addr if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { scan $addr "%2x%2x%2x%2x" ip1 ip2 ip3 ip4 set optaddr "$ip1.$ip2.$ip3.$ip4" } } } when HTTP_REQUEST { if { [info exists optaddr] } { HTTP::header insert "X-Forwarded-For" $optaddr } } The Akamai version should be one, so we log if not. Otherwise, we take the address (stored in the variable addr in hex) and scan it to get the decimal equivalents to build the address for inserting in the X-Forwarded-For header. Cool, right? Also cool—along with the new TCP::option command , an extension was made to the IP::addr command to parse binary fields into a dotted decimal IP address. This extension is also available beginning in 10.2.0-HF2, but extended in 11.0. Here’s the syntax: IP::addr parse [-ipv4 | -ipv6 [swap]] <binary field> [<offset>] So for example, if you had an IPv6 address in option 28 with a 1 byte offset, you would parse that like: log local0. "IP::addr parse IPv6 output: [IP::addr parse -ipv6 [TCP::option get 28] 1]" ## Log Result ## May 27 21:51:34 ltm13 info tmm[27207]: Rule /Common/tcpopt_test <CLIENT_ACCEPTED>: IP::addr parse IPv6 output: 2601:1930:bd51:a3e0:20cd:a50b:1cc1:ad13 But in the context of our TCP option, we have 5-bytes of data with the first byte not mattering in the context of an address, so we get at the address with this: set optaddr [IP::addr parse -ipv4 [TCP::option get 28] 1] This cleans up the rule a bit: when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt c ver if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { set optaddr [IP::addr parse -ipv4 $opt28 1] } } } when HTTP_REQUEST { if { [info exists optaddr] } { HTTP::header insert "X-Forwarded-For" $optaddr } } No need to store the address in the first binary scan and no need for the scan command at all so I eliminated those. Setting a forwarding header is not the only thing we can do with this data. It could also be shipped off to a logging server, or used as a snat address (assuming the server had either a default route to the BIG-IP, or specific routes for the customer destinations, which is doubtful). Logging is trivial, shown below with the log command. The HSL commands could be used in lieu of log if sending off-box to a log server. when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt c ver if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { set optaddr [IP::addr parse -ipv4 $opt28 1] log local0. "Client IP extracted from Akamai TCP option is $optaddr" } } } If setting the provided IP as a snat address, you’ll want to make sure it’s a valid IP address before doing so. You can use the TCL catch command and IP::addr to perform this check as seen in the iRule below: when CLIENT_ACCEPTED { set addrs [list \ "192.168.1.1" \ "256.168.1.1" \ "192.256.1.1" \ "192.168.256.1" \ "192.168.1.256" \ ] foreach x $addrs { if { [catch {IP::addr $x mask 255.255.255.255}] } { log local0. "IP $x is invalid" } else { log local0. "IP $x is valid" } } } The output of this iRule: <CLIENT_ACCEPTED>: IP 192.168.1.1 is valid <CLIENT_ACCEPTED>: IP 256.168.1.1 is invalid <CLIENT_ACCEPTED>: IP 192.256.1.1 is invalid <CLIENT_ACCEPTED>: IP 192.168.256.1 is invalid <CLIENT_ACCEPTED>: IP 192.168.1.256 is invalid Adding this logic into a functional rule with snat: when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] if { [string length $opt28] == 5 } { binary scan $opt c ver if { $ver != 1 } { log local0. "Unsupported Akamai version: $ver" } else { set optaddr [IP::addr parse -ipv4 $opt28 1] if { [catch {IP::addr $x mask 255.255.255.255}] } { log local0. "$optaddr is not a valid address" snat automap } else { log local0. "Akamai inserted Client IP is $optaddr. Setting as snat address." snat $optaddr } } } Alternative TCP Option Use Cases The Akamai solution shows an application implementation taking advantage of normally unused space in TCP headers. There are, however, defined uses for several option “kind” numbers. The list is available here: http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xml. Some options that might be useful in troubleshooting efforts: Opkind 2 – Max Segment Size Opkind 3 – Window Scaling Opkind 5 – Selective Acknowledgements Opkind 8 – Timestamps Of course, with tcpdump you get all this plus the context of other header information and data, but hey, another tool in the toolbox, right? Addendum I've been working with F5 SE Leonardo Simon on on additional examples I wanted to share here that uses option 28 or 253 to extract an IPv6 address if the version is 34 and otherwise extracts an IPv4 address if the version is 1 or 2. Option 28 when CLIENT_ACCEPTED { set opt28 [TCP::option get 28] binary scan $opt28 c ver #log local0. "version: $ver" if { $ver == 34 } { set optaddr [IP::addr parse -ipv6 $opt28 1] log local0. "opt28 ipv6 address: $optaddr" } elseif { $ver == 1 || $ver == 2 } { set optaddr [IP::addr parse -ipv4 $opt28 1] log local0. "opt28 ipv4 address: $optaddr" } } Option 253 when CLIENT_ACCEPTED { set opt253 [TCP::option get 253] binary scan $opt253 c ver #log local0. "version: $ver" if { $ver == 34 } { set optaddr [IP::addr parse -ipv6 $opt253 1] log local0. "opt253 ipv6 address: $optaddr" } elseif { $ver == 1 || $ver == 2 } { set optaddr [IP::addr parse -ipv4 $opt253 1] log local0. "opt253 ipv4 address: $optaddr" } }
Two-Factor Authentication With Google Authenticator And APM
Introduction Two-factor authentication (TFA) has been around for many years and the concept far pre-dates computers. The application of a keyed padlock and a combination lock to secure a single point would technically qualify as two-factor authentication: “something you have,” a key, and “something you know,” a combination. Until the past few years, two-factor authentication in its electronic form has been reserved for high security environments: government, banks, large companies, etc. The most common method for implementing a second authentication factor has been to issue every employee a disconnected time-based one-time password hard token. The term “disconnected” refers to the absence of a connection between the token and a central authentication server. A “hard token” implies that the device is purpose-built for authentication and serves no other purpose. A soft or “software” token on the other hand has other uses beyond providing an authentication mechanism. In the context of this article we will refer to mobile devices as a soft tokens. This fits our definition as the device an be used to make phone calls, check email, surf the Internet, all in addition to providing a time-based one-time password. A time-based one-time password (TOTP) is a single use code for authenticating a user. It can be used by itself or to supplement another authentication method. It fits the definition of “something you have” as it cannot be easily duplicated and reused elsewhere. This differs from a username and password combination, which is “something you know,” but could be easily duplicated by someone else. The TOTP uses a shared secret and the current time to calculate a code, which is displayed for the user and regenerated at regular intervals. Because the token and the authentication server are disconnected from each other, the clocks of each must be perfectly in sync. This is accomplished by using Network Time Protocol (NTP) to synchronize the clocks of each device with the correct time of central time servers. Using Google Authenticator as a soft token application makes sense from many angles. It is low cost due to the proliferation of smart phones and is available from the “app store” free of charge on all major platforms. It uses an open standard (defined by RFC 4226), which means that it is well-tested, understood, secure. Calculation as you will later see is well-documented and relatively easy to implement in your language of choice (iRules in our case). This process is explained in the next section. This Tech Tip is a follow-up to Two-Factor Authentication With Google Authenticator And LDAP. The first article in this series highlighted two-factor authentication with Google Authenticator and LDAP on an LTM. In this follow-up, we will be covering implementation of this solution with Access Policy Manager (APM). APM allows for far more granular control of network resources via access policies. Access policies are rule sets, which are intuitively displayed in the UI as flow charts. After creation, an access policy is applied to a virtual server to provide security, authentication services, client inspection, policy enforcement, etc. This article highlights not only a two-factor authentication solution, but also the usage of iRules within APM policies. By combining the extensibility of iRules with the APM’s access policies, we are able to create virtually any functionality we might need. Note: A 10-user fully-featured APM license is included with every LTM license. You do not need to purchase an additional module to use this feature if you have less than 10 users. Calculating The Google Authenticator TOTP The Google Authenticator TOTP is calculated by generating an HMAC-SHA1 token, which uses a 10-byte base32-encoded shared secret as a key and Unix time (epoch) divided into a 30 second interval as inputs. The resulting 80-byte token is converted to a 40-character hexadecimal string, the least significant (last) hex digit is then used to calculate a 0-15 offset. The offset is then used to read the next 8 hex digits from the offset. The resulting 8 hex digits are then AND’d with 0x7FFFFFFF (2,147,483,647), then the modulo of the resultant integer and 1,000,000 is calculated, which produces the correct code for that 30 seconds period. Base32 encoding and decoding were covered in my previous Tech Tip titled Base32 Encoding And Decoding With iRules . The Tech Tip details the process for decoding a user’s base32-encoded key to binary as well as converting a binary key to base32. The HMAC-SHA256 token calculation iRule was originally submitted by Nat to the Codeshare on DevCentral. The iRule was slightly modified to support the SHA-1 algorithm, but is otherwise taken directly from the pseudocode outlined in RFC 2104. These two pieces of code contribute the bulk of the processing of the Google Authenticator code. The rest is done with simple bitwise and arithmetic functions. Triggering iRules From An APM Access Policy Our previously published Google Authenticator iRule combined the functionality of Google Authenticator token verification with LDAP authentication. It was written for a standalone LTM system without the leverage of APM’s Visual Policy Editor. The issue with combining these two authentication factors in a single iRule is that their functionality is not mutually exclusive or easily separable. We can greatly reduce the complexity of our iRule by isolating functionality for Google Authenticator token verification and moving the directory server authentication to the APM access policy. APM iRules differ from those that we typically develop for LTM. iRules assigned to LTM virtual server are triggered by events that occur during connection or payload handling. Many of these events still apply to an LTM virtual server with an APM policy, but do not have perspective into the access policy. This is where we enter the realm of APM iRules. APM iRules are applied to a virtual server exactly like any other iRule, but are triggered by custom iRule event agent IDs within the access policy. When the access policy reaches an iRule event, it will trigger the ACCESS_POLICY_AGENT_EVENT iRule event. Within the iRule we can execute the ACCESS::policy agent_id command to return the iRule event ID that triggered the event. We can then match on this ID string prior to executing any additional code. Within the iRule we can get and set APM session variables with the ACCESS::session command, which will serve as our conduit for transferring variables to and from our access policy. A visual walkthrough of this paragraph is shown below. iRule Trigger Process Create an iRule Event in the Visual Policy Editor Specify a Name for the object and an ID for the Custom iRule Event Agent Create an iRule with the ID referenced and assign it to the virtual server 1: when ACCESS_POLICY_AGENT_EVENT { 2: if { [ACCESS::policy agent_id] eq "ga_code_verify" } { 3: # get APM session variables 4: set username [ACCESS::session data get session.logon.last.username] 5: 6: ### Google Authenticator token verification (code omitted for brevity) ### 7: 8: # set APM session variables 9: ACCESS::session data set session.custom.ga_result $ga_result 10: } 11: } Add branch rules to the iRule Event which read the custom session variable and handle the result Google Authenticator Two-Factor Authentication Process Two-Factor Authentication Access Policy Overview Rather than walking through the entire process of configuring the access policy from scratch, we’ll look at the policy (available for download at the bottom of this Tech Tip) and discuss the flow. The policy has been simplified by creating macros for the redundant portions of the authentication process: Google Authenticator token verification and the two-factor authentication processes for LDAP and Active Directory. The “Google Auth verification” macro consists of an iRule event and 5 branch rules. The number of branch rules could be reduced to just two: success and failure. This would however limit our diagnostic capabilities should we hit a snag during our deployment, so we added logging for all of the potential failure scenarios. Remember that these logs are sent to APM reporting (Web UI: Access Policy > Reports) not /var/log/ltm. APM reporting is designed to provide per-session logging in the user interface without requiring grepping of the log files. The LDAP and Active Directory macros contain the directory server authentication and query mechanisms. Directory server queries are used to retrieve user information from the directory server. In this case we can store our Google Authenticator key (shared secret) in a schema attribute to remove a dependency from our BIG-IP. We do however offer the ability to store the key in a data group as well. The main portion of the access policy is far simpler and easier to read by using macros. When the user first enters our virtual server we look at the Landing URI they are requesting. A first time request will be sent to the “normal” logon page. The user will then input their credentials along with the one-time password provided by the Google Authenticator token. If the user’s credentials and one-time password match, they are allowed access. If they fail the authentication process, we increment a counter via a table in our iRule and redirect them back to an “error” logon page. The “error” logon page notifies them that their credentials are invalid. The notification makes no reference as to which of the two factors they failed. If the user exceeds the allowed number of failures for a specified period of time, their session will be terminated and they will be unable to login for a short period of time. An authenticated user would be allowed access to secured resources for the duration of their session. Deploying Google Authenticator Token Verification This solution requires three components (one optional) for deployment: Sample access policy Google Authenticator token verification iRule Google Authenticator token generation iRule (optional) The process for deploying this solution has been divided into four sections: Configuring a AAA server Login to the Web UI of your APM From the side panel select Access Policy > AAA Servers > Active Directory, then the + next to the text to create a new AD server Within the AD creation form you’ll need to provide a Name, Domain Controller, Domain Name, Admin Username, and Admin Password When you have completed the form click Finished Copy the iRule to BIG-IP and configure options Download a copy of the Google Authenticator Token Verification iRule for APM from the DevCentral CodeShare (hint: this is much easier if you “edit” the wiki page to display the source without the line numbers and formatting) Navigate to Local Traffic > iRules > iRule List and click the + symbol Name the iRule '”google_auth_verify_apm,” then copy and paste the iRule from the CodeShare into the Definition field At the top of the iRule there are a few options that need to be defined: lockout_attempts - number of attempts a user is allowed to make prior to being locked out temporarily (default: 3 attempts) lockout_period - duration of lockout period (default: 30 seconds) ga_code_form_field - name of HTML form field used in the APM logon page, this field is define in the "Logon Page" access policy object (default: ga_code_attempt) ga_key_storage - key storage method for users' Google Authenticator shared keys, valid options include: datagroup, ldap, or ad (default: datagroup) ga_key_ldap_attr - name of LDAP schema attribute containing users' key ga_key_ad_attr - name of Active Directory schema attribute containing users' key ga_key_dg - data group containing user := key mappings Click Finished when you’ve configured the iRule options to your liking Import sample access policy From the Web UI, select Access Policy > Access Profiles > Access Profiles List In the upper right corner, click Import Download the sample policy for Two-Factor Authentication With Google Authenticator And APM and extract the .conf from ZIP archive Fill in the New Profile Name with a name of your choosing, then select Choose File, navigate to the extracted sample policy and Open Click Import to complete the import policy The sample policy’s AAA servers will likely not work in your environment, from the Access Policy List, click Edit next to the imported policy When the Visual Policy Editor opens, expand the macro (LDAP or Active Directory auth) that describe your environment Click the AD Auth object, select the AD server from the drop-down that was defined earlier in the AAA Servers step, then click Save Repeat this process for the AD Query object Assign sample policy and iRule to a virtual server From the Web UI, select Local Traffic > Virtual Servers > Virtual Server List, then the create button (+) In the New Virtual Server form, fill in the Name, Destination address, Service Port (should be HTTPS/443), next select an HTTP profile and anSSL Profile (Client). Next you’ll add a SNAT Profile if needed, an Access Profile, and finally the token verification iRule Depending on your deployment you may want to add a pool or other network connectivity resources Finally click Finished At this point you should have a function virtual server that is serving your access policy. You’ll now need to add some tokens for your users. This process is another section on its own and is listed below. Generating Software Tokens For Users In addition to the Google Authenticator Token Verification iRule for APM we also wrote a Google Authenticator Soft Token Generator iRule that will generate soft tokens for your users. The iRule can be added directly to an HTTP virtual server without a a pool and accessed directly to create tokens. There are a few available fields in the generator: account, pre-defined secret, and a QR code option. The “account” field defines how to label the soft token within the user’s mobile device and can be useful if the user has multiple soft token on the same device (I have 3 and need to label them to keep them straight). A 10-byte string can be used as a pre-defined secret for conversion to a base32-encoded key. We will advise you against using a pre-defined key because a key known to the user is something they know (as opposed to something they have) and could be potentially regenerate out-of-band thereby nullifying the benefits of two-factor authentication. Lastly, there is an option to generate a QR code by sending an HTTPS request to Google and returning the QR code as an image. While this is convenient, this could be seen as insecure since it may wind up in Google’s logs somewhere. You’ll have to decide if that is a risk you’re willing to take for the convenience it provides. Once the token has been generated, it will need to be added to a data group on the BIG-IP: Navigate to Local Traffic > iRules > Data Group Lists Select Create from the upper right-hand corner if the data group does not yet exist. If it exists, just select it from the list. Name the data group “google_auth_keys” (data group name can be changed in the beginning section of the iRule) The type of data group will be String Type the “username” into the String field and paste the “Google Authenticator key” into the Value field Click Add and the username/key pair should appear in the list as such: user := ONSWG4TFOQYTEMZU Click Finished when all your username/key pairs have been added. Your user can scan the QR code or type it into their device manually. After they scan the QR code, the account name should appear along with the TOTP for the account. The image below is how the soft token appears in the Google Authenticator iPhone application: Once again, do not let the user leave with a copy of the plain text key. Knowing their key value will negate the value of having the token in the first place. Once the key has been added to the BIG-IP, the user’s device, and they’ve tested their access, destroy any reference to the key outside the BIG-IPs data group.If you’re worried about having the keys in plain text on the BIG-IP, they can be encrypted with AES or stored off-box in LDAP and only queried via secure connection. This is beyond the scope of this article, but doable with iRules. Code Google Authenticator Token Verification iRule for APM – Documentation and code for the iRule used in this Tech Tip Google Authenticator Soft Token Generator iRule – iRule for generating soft tokens for users Sample Access Policy: Two-Factor Authentication With Google Authenticator And APM – APM access policy Reference Materials RFC 4226 - HOTP: An HMAC-Based One-Time Password Algorithm RFC 2104 - HMAC: Keyed-Hashing for Message Authentication RFC 4648 - The Base16, Base32, and Base64 Data Encodings SOL3122: Configuring the BIG-IP system to use an NTP server using the Configuration utility – Information on configuring time servers Configuration Guide for BIG-IP Access Policy Manager – The “big book” on APM configurations Configuring Authentication Using AAA Servers – Official F5 documentation for configuring AAA servers for APM Troubleshooting AAA Configurations – Extra help if you hit a snag configuring your AAA server
Rewriting Redirects
While best practices for virtualized web applications may indicate that relative self-referencing links and redirects (those which don't include the protocol or the hostname) are preferable to absolute ones (those which do), many applications load balanced by our gear still send absolute self-references. This drives a fairly common requirement when proxying or virtualizing HTTP applications: To manipulate any redirects the servers may set such that they fully support the intended proxy or virtualization scheme. In some cases the requirement is as simple as changing "http://" to "https://" in every redirect the server sends because it is unaware of SSL offloading. Other applications or environments may require modifications to the host, URI, or other headers. LTM provides a couple of different ways to manage server-set redirects appropriately. HTTP profile option: "Rewrite Redirects" The LTM http profile contains the "Rewrite Redirects" option which supports rewriting server-set redirects to the https protocol with a hostname matching the one requested by the client. The possible settings for the option are "All", "Matching", Node", and "None". Rewrite Redirects settings for http profile Setting Effect Resulting Redirect Use Case All Rewrites all HTTP 301, 302, 303, 305, or 307 redirects https://<requested_hostname>/<requested_uri> Use "All" if all redirects are self-referencing and the applicaiton is intended to be secure throughout. You should also use "All" if your application is intended to be secure throughout, even if redirected to another hostname. Matching Rewrites redirects when the request and the redirect are identical except for a trailing slash. See K14775 . https://<requested_hostname>/<requested_uri>/ Use "Matching" to rewrite only courtesy redirects intended to append a missing trailing slash to a directory request. Node Rewrites all redirects containing pool member IP addresses instead of FQDN https://<vs_address>/<requested_uri> If your servers send redirects that include the server's own IP address instead of a hostname. None No redirects are rewritten N/A Default Setting Note that all options will rewrite the specified redirects to HTTPS, so there must be an HTTPS virtual enabled on the same address as the HTTP virtual server. iRule Options While these options cover a broad range of applications, they may not be granular enough to meet your needs. For example, you might only want to re-write the hostname, not the protocol, to support HTTP-only proxying scenarios. You might need it to temporarily work around product issues such as those noted in SOL8535/CR89873 . In these cases, you can use an iRule that uses the HTTP::is_redirect command to identify server-set redirects and selectively rewrite any part of the Location header using the HTTP::header command with the "replace" option. Here's an iRule that rewrites just one specific hostname to another, preserving the protocol scheme and URI as set by the server: when HTTP_RESPONSE { if { [HTTP::is_redirect] }{ HTTP::header replace Location [string map {"A.internal.com" "X.external.com"} [HTTP::header Location]] } } Here's one that rewrites both relative and absolute redirects to absolute HTTPS redirects, inserting the requested hostname when re-writing the relative redirect to absolute: when HTTP_REQUEST { # save hostname for use in response set fqdn_name [HTTP::host] } when HTTP_RESPONSE { if { [HTTP::is_redirect] }{ if { [HTTP::header Location] starts_with &quot;/&quot; }{ HTTP::header replace Location &quot;https://$fqdn_name[HTTP::header Location]&quot; } else { HTTP::header replace Location &quot;[string map {&quot;http://&quot; &quot;https://&quot;} [HTTP::header Location]]&quot; } } } The string map example could quite easily be adjusted or extended to meet just about any redirect rewriting need you might encounter. (The string map command will accept multiple replacement pairs which can come in handy if multiple hostnames or directory strings need to be re-written -- in many cases you can perform the intended replacements with a single string map command.) Taking it a step further As I mentioned earlier, redirects are only one place server self-references may be found. If absolute self-referencing links are embedded in the HTTP payload, you may need to build and apply a stream profile to perform the appropriate replacements. An iRule could also be used for more complex payload replacements if necessary. For the ultimate in redirect rewriting and all other things HTTP proxy, I direct your attention to the legendary ProxyPass iRule contributed to the DevCentral codeshare by Kirk Bauer (thanks, Kirk, for a very comprehensive & instructive example!)SSL Profiles Part 1: Handshakes
This is the first in a series of tech tips on the F5 BIG-IP LTM SSL profiles. SSL Overview and Handshake SSL Certificates Certificate Chain Implementation Cipher Suites SSL Options SSL Renegotiation Server Name Indication Client Authentication Server Authentication All the "Little" Options SSL, or the Secure Socket Layer, was developed by Netscape back in the ‘90s to secure the transport of web content. While adopted globally, the standards body defined the Transport Layer Security, or TLS 1.0, a few years later. Commonly interchanged in discussions, the final version of SSL (v3) and the initial version of TLS (v1) do not interoperate, though TLS includes the capabality to downgrade to SSLv3 if necessary. Before we dive into the options within the SSL profiles, we’ll start in this installment with a look at the SSL certificate exchanges and take a look at what makes a client versus a server ssl profile. Server-Only Authentication This is the basic TLS handshake, where the only certificate required is on the serverside of the connection. The exchange is shown in Figure 1 below. In the first step. the client sends a ClientHello message containing the cipher suites (I did a tech tip a while back on manipulating profiles that has a good breakdown of the fields in a ClientHello message), a random number, the TLS version it supports (highest), and compression methods. The ServerHello message is then sent by the server with the version, a random number, the ciphersuite, and a compression method from the clients list. The server then sends its Certificate and follows that with the ServerDone message. The client responds with key material (depending on cipher selected) and then begins computing the master secret, as does the server upon receipt of the clients key material. The client then sends the ChangeCipherSpec message informing the server that future messages will be authenticated and encrypted (encryption is optional and dependent on parameters in server certificate, but most implementations include encryption). The client finally sends its Finished message, which the server will decrypt and verify. The server then sends its ChangeCipherSpec and Finished messages, with the client performing the same decryption and verification. The application messages then start flowing and when complete (or there are SSL record errors) the session will be torn down. Question—Is it possible to host multiple domains on a single IP and still protect with SSL/TLS without certificate errors? This question is asked quite often in the forums. The answer is in the details we’ve already discussed, but perhaps this isn’t immediately obvious. Because the application messages—such as an HTTP GET—are not received until AFTER this handshake, there is no ability on the server side to switch profiles, so the default domain will work just fine, but the others will receive certificate errors. There is hope, however. In the TLSv1 standard, there is an option called Server Name Indication, or SNI, which will allow you to extract the server name and switch profiles accordingly. This could work today—if you control the client base. The problem is most browsers don’t default to TLS, but rather SSLv3. Also, TLS SNI is not natively supported in the profiles, so you’ll need to get your hands dirty with some serious iRule-fu (more on this later in the series). Anyway, I digress. Client-Authenticated Handshake The next handshake is the client-authenticated handshake, shown below in Figure 2. This handshake adds a few steps (in bold above), inserting the CertificateRequest by the server in between the Certificate and ServerHelloDone messages, and the client starting off with sending its Certificate and after sending its key material in the ClientKeyExchange message, sends the CertificateVerify message which contains a signature of the previous handshake messages using the client certificate’s private key. The server, upon receipt, verifies using the client’s public key and begins the heavy compute for the master key before both finish out the handshake in like fashion to the basic handshake. Resumed Handshake Figure 3 shows the abbreviated handshake that allows the performance gains of not requiring the recomputing of keys. The steps passed over from the full handshake are greyed out. It’s after step 5 that ordinarily both server and client would be working hard to compute the master key, and as this step is eliminated, bulk encryption of the application messages is far less costly than the full handshake. The resumed session works by the client submitting the existing session ID from previous connections in the ClientHello and the server responding with same session ID (if the ID is different, a full handshake is initiated) in the ServerHello and off they go. Profile Context As with many things, context is everything. There are two SSL profiles on the LTM, clientssl and serverssl. The clientssl profile is for acting as the SSL server, and the serverssl profile is for acting as the SSL client. Not sure why it worked out that way, but there you go. So if you will be offloading SSL for your applications, you’ll need a clientssl profile. If you will be offloading SSL to make decisions/optimizations/inspects, but still require secured transport to your servers, you’ll need a clientssl and a serverssl profile. Figure 4 details the context. Conclusion Hopefully this background on the handshake process and the profile context is beneficial to the profile options we’ll cover in this series. For a deeper dive into the handshakes (and more on TLS), you can check out RFC 2246 and this presentation on SSL and TLS cryptography. Next up, we’ll take a look at Certificates.LTM External Monitors: The Basics
LTM's external monitors are incredibly flexible, fairly easy to implement, and especially useful for monitoring applications for which there is no built-in monitor template. They give you the ability to effectively monitor the health of just about any application by writing custom scripts to interact with your servers in the same way users would. In this article, I will attempt to explain the basic LTM external monitoring paradigm, then dissect and explain one of the sample monitors from the Advanced Design & Config codeshare. (Thanks to poster pgroven for inspiring me to finally write this up.) An "External Monitor" is a script that is "external" to the configuration file which contains specific logic designed to interact with your servers to verify the health of load balanced services. LTM runs a unique instance of the custom-crafted script against each pool member to which it is applied, passing command line arguments and environment variables as specified in the monitor definition calling the script. The script logic formulates and submits a request (or requests) to the target pool member, evaluates the response(s), and manages the pool member's availability based on the results of the response evaluation. The Tools The sample monitor scripts The external script itself should be a shell script (if at all possible) to minimize overhead. If absolutely necessary, a perl script may be used instead, but keep in mind that the overhead of invoking the intepreter and required modules for multiple instances may negatively impact performance overall. However, LTM was not intended to be a development platform or a dedicated monitoring device, and thus has a limited set of development tools and modules included in the software build, so you may not find the perl modules you need. You can add them, but it is not recommended or supported to do so, and those customizations will likely not survive an upgrade. (You can also use an external monitor to invoke a compiled program, but that discussion is beyond the scope of this article.) cURL cURL is a very flexible command line tool you can use in shell and perl scripts for complex interactions with HTTP and FTP servers. netcat netcat is another useful command line tool that facilitates interaction with TCP and UDP services. The LTM external monitor template The LTM external monitor template allows you to specify the name of the script to run, the interval & timeout, command line arguments and variables the script requires, and alternate destination for the monitor traffic. The Tips ("good to know" stuff and best practices recommendations) There are a few special considerations you need to make when writing the script and configuring the LTM monitor definition that calls it. Do you really need an external monitor? Never use an external monitor when a built-in one will work as well. Forking a shell and running even the simplest shell script takes a significant amount of system resources, so external monitors should be avoided whenever possible. If possible, have the server administrator script execution of the required transaction on the server itself (or locate/author an alternative script on the server) that reliably reflects its availability. Then, instead of an external monitor, you can define a built-in monitor that requests that dynamic script from the server, and let the server run the script locally and report results. For example, the simple request/response HTTP transaction in the sample script below would be much better implemented using the built in basic HTTP monitor. Optimization Use the lowest overhead tools, make the simplest possible request, & minimize the amount of response parsing required to determine the pool member's status. The logic of the script can contain just about any logic you want to determine if that server is healthy. You can use commandline tools like netcat and cURL to replicate server transactions, from a basic request and response parsing for an expected string, to more complicated exchanges where cookies or persistence tokens are used, login is required or some other dynamic transaction must take place in order to establish that usability of a server by its intended users. Redundant pairs Both units in a redundant pair will independently run the configured monitor, even when running as Standby. Monitor status is not shared between redundant pairs. Variables Variables may be passed to indicate service or hostname, the URI you need to request, or just about any piece of information that would be needed to construct a valid query and receive a valid response from the server. Variables can contain static values, basic regex expressions, or even expressions that contain other variables. As long as your script receives the expected variables from the monitor definition and the logic handles them appropriately, the possibilities are fairly limitless. Authentication If your script must pass authentication tokens to the pool members to sufficiently transact with them, make sure the authentication method will allow multiple concurrent logins. Each pool-member-specific instance on each member of a redundant pair may attempt to log in simultaneously. If only a single login is allowed per credential, authentication collisions will most likely result in rolling multiple concurrent false downs as only one monitor request can succeed at a time. Script against one pool member The script should be written to determine the health of one specific pool member. An LTM monitor script is really a template for monitoring a single pool member. Whether you apply an LTM monitor to an individual pool member or to the entire pool in the GUI, a separate copy of the monitor runs for each pool member, passing only that specific IP & port to be tested and maintaining only that single tested pool member's availability. (Discrete monitoring of a single pool member by an external monitor isespecially important if other monitors will also be applied to the pool members.) Minimize the work Keep the amount of work your monitor script must perform as small as possible. Both the script that runs on LTM and the request against the server itself should represent the minimum interaction required to adequately determine the server's health. If you consider how often the monitor will make that request against each pool members, you can get an idea of the scale of the work that you're asking both big IP and your servers to do. The Ins & Outs A script intended for use as an external monitor must conform to some specific input and output requirements. Command line arguments IP and port of the pool member are passed automatically as the first 2 command line arguments for all external monitors. The IP address is always passed in the IPv6 format (TMOS' internal address format). IPv4 addresses are passed using IPv6's special "IPv4 mapped address" transition notation: The IPv4 address prefixed with "::ffff:". In that notation, the IP address for pool member 10.0.0.1:80 would be "::ffff:10.0.0.1". The proper address type is critical to proper operation of your monitor script. More on that later. Additional command line arguments may be defined in the monitor configuration. When defined, they are passed to the script by the monitoring daemon as the 3rd, 4th, and subsequent arguments. Variables Variables in the form of Name/Value pairs may also be defined in the monitor configuration. When defined, they are created as environment variables in the shell forked for each instance of the script. Script Output IF ANY VALUE AT ALL GOES TO STANDARD OUTPUT, THE POOL MEMBER WILL BE MARKED UP. If the pool member is determined to be healthy enough to receive load balance traffic by successfully satisfying the script logic, the script should output any value but null to standard output, and the monitoring daemon will mark the pool member up. If the pool member does not respond as expected, the script will output nothing to stdout, and the lack of output will cause the monitoring daemon to mark the pool member down at the expiration of the timeout. All other outputs from the script are ignored by the monitoring daemon. The Timing The interval is the amount of time that will elapse between the start of each monitor attempt. In order to avoid creating a Denial of Service situation by sending your servers excessive monitor traffic, you should increase the interval as much as possible. The interval MUST be longer that the longest possible healthy response should take, since each successive instance of the script run against a pool member will kill off any already-running previous instances, assuming they are hung and will never complete. F5 recommends a timeout value 3 times greater than the interval value plus 1 second, but you can use a different ratio if necessary. Setting the timeout shorter than the interval is not recommended. If you consider the monitor will make that request every <interval> against each pool member, you can get an idea of the scale of the work that you're asking both LTM and your servers to do, so some careful testing is in order with the goal of minimizing the timeout value and maximizing the interval. (If you notice that your healthy pool members are being marked down and then back up again on the next interval, your timeout may be to short, and some further experimentation may be in order.) There are also ways that you can control and tighten up the tolerance for timing in some monitors. In another article, we will take a closer look at a different external monitor that marks pool members down a little bit more aggressively than waiting for the monitor timeout. The Gory Details Here's a sample monitor from the codeshare: HTTPMonitor_cURL_BasicGET Let's go through the script a section at a time and take a closer look at what's going on. First of all, notice that the script documentation tells us it is expecting 2 variable definitions: # This example expects the following Name/Value pairs: # URI = the URI to request from the server # RECV = the expected response (not case sensitive) For this example, we are going to request the URI "/testpath/testfile.html" over HTTP for each server, and expect a string that says "Server is UP!!!". (As noted earlier, the simple request / response HTTP transaction demonstrated here would be much better implemented using the built in basic HTTP monitor with static request/receive strings, but is still helpful in demonstrating the basic requirements for external monitor implementation.) Now that we know what variables we need to define, the monitor configuration will look like this: monitor ExternalHTTP { defaults from external RECV "Server is UP!!!" run "" URI "/testpath/testfile.html" } Once the monitor is defined, it can be applied to the pool members. (The monitor can be applied to individual pool members or the entire pool. Either way, a unique instance of the script is run for each pool member at each interval to monitor each pool member independently.) When the monitoring daemon (bigd) runs the script according to the monitor definition, it forks a new shell and and creates the required environment variables, then invokes the script with the 2 default command line arguments (the target pool member's IP address and port). At the start of the script, the command line arguments are processed. First it checks if the IPv6 address passed is in the IPv4 mapped format, and if so, converts it to a standard IPv4 address instead, and assigns both arguments to named environment variables: # remove IPv6/IPv4 compatibility prefix (LTM passes addresses in IPv6 format) IP=`echo ${1} | sed 's/::ffff://'` PORT=${2} Once the IP and PORT variables are defined, they are used to set up a process management scheme intended to prevent multiple copies of the monitor from running against the same pool member at the same time. It works like this: Each instance of the script first looks for a unique file named "monitorname.IP_port.pid" in /var/run containing the process ID of the last instance of the script run against that pool member. If it exists, it means the last instance of the script has not completed. Since multiple copies of the same script funning against the same pool member may interfere with proper monitor operations, the script kills that process, then re-writes the PID file containing the process ID of the current instance for reference by the next instance. PIDFILE="/var/run/`basename ${0}`.${IP}_${PORT}.pid" # kill of the last instance of this monitor if hung and log current pid if [ -f $PIDFILE ] then kill -9 `cat $PIDFILE` > /dev/null 2>&1 fi echo "$$" > $PIDFILE Now the heavy lifting begins. In this example, we're simply sending a URI, and examining the response to see if it contains the RECV string: # send request & check for expected response curl -fNs http://${IP}:${PORT}${URI} | grep -i "${RECV}" 2>&1 > /dev/null (Remember this is a simplified example. In a real world example, the logic inserted here would replicate whatever transactions you identified earlier as the minimum required interaction to determine the pool member's health. cURL has a wide range of options you can use to mimic almost any browser operation, including sending and receiving cookies, to replicate multi-step transactions or validate complex responses.) If the expected response contained the value of the RECV variable, the "grep" command will return 0, causing the script to send the string "UP" to stdout, and the pool member will be marked up immediately. If the expected response did NOT contain the value of the RECV variable, the "grep" command will return a non-zero value, and the script will output nothing to stdout, and the pool member will be marked down when the timeout expires. # mark node UP if expected response was received if [ $? -eq 0 ] then echo "UP" fi And finally the script will delete the PID file written earlier (since it has finished cleanly and won't need to be killed of by the next instance) and then exit: rm -f $PIDFILE exit It doesn't work... what now? Troubleshooting external monitors can be challenging. In my next article, I'll cover the basic process you can follow to track down and resolve any issues that may interfere with proper monitor operation. (LTM External Monitors: Troubleshooting)
