Any questions? Post'em
Hi everyone, If you have any questions or comments about the performance report or it's supporting documents, please feel free to post them here. I'm one of the engineers who helped to create the performance report, and I'll be actively monitoring this forum to answer questions. Mike Lowell
Throughput per virtual server
Hi! Not quite sure if my question fits this forum, but I'm trying anyway :) I'm trying to measure performance through our LTMs. Is there any way to get throughput numbers per virtual server? The default graph in the dashboard gives a good indication of the total traffic, but it would be nice to be able to drill down to a specific vip. Is it perhaps possible to query with snmp? /Niklas
WILS: SSL TPS versus HTTP TPS over SSL
The difference between these two performance metrics is significant so be sure you know which one you’re measuring, and which one you wanted to be measuring. It may be the case that you’ve decided that SSL is, in fact, a good idea for securing data in transit. Excellent. Now you’re trying to figure out how to implement support and you’re testing solutions or perhaps trying to peruse reports someone else generated from testing. Excellent. I’m a huge testing fan and it really is one of the best ways to size a solution specifically for your environment. Some of the terminology used to describe specific performance metrics in application delivery, however, can be misleading. The difference between SSL TPS (Transactions per second) and HTTP TPS over SSL, for example, are significant and therefore should not be used interchangeably when comparing performance and capacity of any solution – that goes for software, hardware, or some yet-to-be-defined combination thereof. The reasons why interpreting claims of SSL TPS are so difficult is due to the ambiguity that comes from SSL itself. SSL “transactions” are, by general industry agreement (unenforceable, of course) a single transaction that is “wrapped” in an SSL session. Generally speaking one SSL transaction is considered: 1. Session establishment (authentication, key exchange) 2. Exchange of data over SSL, often a 1KB file over HTTP 3. Session closure Seems logical, but technically speaking a single SSL transaction could be interpreted as any single transaction conducted over an SSL encrypted session because the very act of transmitting data over the SSL session necessarily requires SSL-related operations. SSL session establishment requires a handshake and an exchange of keys, and the transfer of data within such a session requires the invocation of encryption and decryption operations (often referred to as bulk encryption). Therefore it is technically accurate for SSL capacity/performance metrics to use the term “SSL TPS” and be referring to two completely different things. This means it is important that whomever is interested in such data must do a little research to determine exactly what is meant by SSL TPS when presented with such data. Based on the definition the actual results mean different things. When used to refer to HTTP TPS over SSL the constraint is actually on the bulk encryption rate (related more to response time, latency, and throughput measurements), while SSL TPS measures the number of SSL sessions that can be created per second and is more related to capacity than response time metrics. It can be difficult to determine which method was utilized, but if you see the term “SSL ID re-use” anywhere, you can be relatively certain the test results refer to HTTP TPS over SSL rather than SSL TPS. When SSL session IDs are reused, the handshaking and key exchange steps are skipped, which reduces the number of computationally expensive RSA operations that must be performed and artificially increases the results. As always, if you aren’t sure what a performance metric really means, ask. If you don’t get a straight answer, ask again, or take advantage of all that great social networking you’re doing and find someone you trust to help you determine what was really tested. Basing architectural decisions on misleading or misunderstood data can cause grief and be expensive later when you have to purchase additional licenses or solutions to bring your capacity up to what was originally expected. WILS: Write It Like Seth. Seth Godin always gets his point across with brevity and wit. WILS is an ATTEMPT TO BE concise about application delivery TOPICS AND just get straight to the point. NO DILLY DALLYING AROUND. The Anatomy of an SSL Handshake When Did Specialized Hardware Become a Dirty Word? WILS: Virtual Server versus Virtual IP Address Following Google’s Lead on Security? Don’t Forget to Encrypt Cookies WILS: What Does It Mean to Align IT with the Business WILS: Three Ways To Better Utilize Resources In Any Data Center WILS: Why Does Load Balancing Improve Application Performance? WILS: Application Acceleration versus Optimization All WILS Topics on DevCentral What is server offload and why do I need it?Implementing SSL Orchestrator - Validation & Troubleshooting
Introduction This article is part of a series on implementing BIG-IP SSL Orchestrator. It includes high availability and central management with BIG-IQ. Implementing SSL/TLS Decryption is not a trivial task. There are many factors to keep in mind and account for, from the network topology and insertion point, to SSL/TLS keyrings, certificates, ciphersuites and on and on. This article focuses on verification that the solution is working. This article is divided into the following high level sections: How to check if content is being decrypted How to check if content is being blocked How to check if content is being bypassed Logging and Troubleshooting Please forgive me for using SSL and TLS interchangeably in this article. Software versions used in this article: BIG-IP Version: 14.1.2 SSL Orchestrator Version: 5.5 BIG-IQ Version: 7.0.1 How to check if content is being decrypted Go to a computer that connects to the internet through SSL Orchestrator.Go to an encrypted website like www.cisco.com.The page should load without error.Click the padlock icon in the Address Bar. Notice that the connection is secure and the certificate is valid.Click the Certificate to view more details.The certificate was issued to www.cisco.com but has been issued by subrsa.f5labs.com.The connection to this web site was decrypted and re-encrypted by SSL Orchestrator. Switch to the Palo Alto UI and go to Monitor > Traffic.It should look like this. Click the link on the left in the red box. This drills into more details. Notice the IP address of 72.163.10.10 and port 80 (not port 443).A quick lookup of that IP shows it belongs to Cisco. How to check if content is being blocked Now go to the website www.eicar.org.This site hosts a harmless test malware file so you can see if your security measures are effective.Click Download Anti Malware Testfile on the right. Click the Download link on the left. Scroll down on the next page.You should see a section with 8 different downloads. There are 4 different file types.The first group uses HTTP.The second group uses HTTPS.Click any of the HTTPS links to verify it is detected and blocked. You should get a block page like the following. Notice the URL indicates an attempt to download one of the compressed .zip files. How to check if content is being bypassed Next try a Banking website and make sure it is bypassed.On the client open a browser and go to www.chase.com.Click the Padlock icon in the address bar and it should look like this. Click the highlighted section for more details about this certificate. This certificate was issued by Entrust CA.The SSL Orchestrator is not decrypting this connection.We can check the Palo Alto logs to make sure it doesn’t have a record of this transaction. The image below shows ping traffic generated by SSL Orchestrator to test the health of the L2 Service.Notice there is no web-browsing or port 80 traffic. Logging and Troubleshooting To enable logging: -Connect to the WebUI -Go to SSL Orchestrator and click on Logs -Enable the required logging levels Note: By default, logs are stored locally on the BIG-IP. Below is a reasonably-ordered list of troubleshooting steps. ·If the SSL Orchestrator deployment process fails, review the ensuing error message. It would be impossible to list here all of the possible error messages and their meanings, but often enough the messages will reveal the issue. ·Re-review the lab steps for any missing or misconfigured settings. ·Enable debug logging in the SSL Orchestrator configuration. Tail the APM log from a BIG-IP command line or from the logs page in the management UI. Debug logging will very often reveal important issues. Specifically, it will indicate traffic classification matches, mismatches or deployment issues. tail –f /var/log/apm tail -f /var/log/restnoded/restnoded.log tail -f /var/log/restjavad.0.log ·If the SSL Orchestrator deployment process succeeds, but traffic isn’t flowing through the environment made evident by lack of access to remote sites from the client: o Never enable debug logging on the Per-Request Policy logging option in a production environment. If needed, be sure to turn it back off after capture logs. The PRP logging is extremely verbose and WILL affect system performance. o /var/log/apm is used to log data plane traffic flow o /var/log/restnoded/restnoded.log is used to log control plane (when SSLO configurations are deployed) o /var/log/restjavad is similarly used to log control plane restjava functions o It’s also sometimes useful to tail /var/log/ltm, which is where SSL and generic data plane issues will show up. oEnsure that the client is properly configured to either default route to the ingress VLAN and self-IP of the BIG-IP for transparent proxy access or has the correct browser proxy settings defined for explicit proxy access. oEnsure that traffic is flowing to the BIG-IP from the client with a tcpdump capture at the ingress interface. oReview the LTM configuration created by the SSL Orchestrator. Specifically, look at the pools and respective monitors for any failures. oIsolate service chain services. If at least one service chain has been created, and debug logging indicates that traffic is matching this chain, remove all but one service from that chain and test. Add one service back at a time until traffic flow stops. If a single added service breaks traffic flow, this service will typically be the culprit. oIf a broken service is identified, insert probes to verify inbound and outbound traffic flow. Inline services will have a source (S) VLAN and destination (D) VLAN, and ICAP and receive only services will each have a single source VLAN. Insert a tcpdump capture at each VLAN in order to determine if traffic is getting to the device, and if traffic is leaving the device through its outbound interface. o How to insert probes. The services VLANs are wrapped in application services so must be addressed accordingly in the tcpdump. Note that each inline service will create two separate VLANs – one for inbound and one for outbound, so it becomes easy to surgically insert captures at specific points in the flow (ie. to the service, coming from the service). So a tcpdump capture of a service named “FEYE” might look like this: tcpdump -lnni ssloS_FEYE.app/ssloS_FEYE Where “ssloS_FEYE.app” is the application service container, which contains the “ssloS_FEYE” VLAN. oBy default the ‘all traffic’ rule doesn’t attach a service chain, so it can be as simple as removing service chains from all of the security rules. If traffic doesn’t flow with no service chains attached anywhere, then the problem is in SSLO proper, likely a routing or decryption issue. If the traffic only fails with a service chain applied, this is when it’s useful to isolate the services. If a broken service is identified, insert tcpdump probes as described above. oIf traffic is flowing through all of the security devices, insert a tcpdump probe at the egress point to verify traffic is leaving the BIG-IP to the gateway router. oIf traffic is flowing to the gateway router, perform a more extensive packet analysis to determine if SSL if failing between the BIG-IP egress point and the remote server. tcpdump –i 0.0:nnn –nn –Xs0 –vv –w <file.pcap> <any additional filters> Then either export this capture to WireShark or send to ssldump: ssldump –nr <file.pcap> -H –S crypto > text-file.txt oIf the WireShark or ssldump analysis verifies an SSL issue: oPlug the site’s address into the SSLLabs.com server test site at: www.ssllabs.com/ssltest/ This report will indicate any specific SSL requirements that this site has. oVerify that the SSL Orchestrator server SSL profiles (two of them) have the correct cipher string to match the requirements of this site. To do that, perform the following command at the BIG-IP command line: tmm --clientciphers ‘CIPHER STRING AS DISPLAYED IN SERVER SSL PROFILES’ Further SSL/TLS issues are beyond the depth of this guide. Seek assistance. • If all else fails, seek assistance. Summary In this article you learned how to verify that SSL Orchestrator is decrypting SSL and passing it to an inline security device. You can verify that SSL is decrypted simply by viewing web site certificates. You can also use logging or reporting capabilities of an inline security device to verify that SSL is decrypted. You learned how to test whether the inline security device was blocking malicious content. You also learned how to verify that a policy to bypass SSL decryption is working. Then you learned how to enable logging and perform troubleshooting steps. Next Steps Click Next to proceed to the next article in the series.View current bandwidth for particular interface
Is it possible (with some built in command or such) to through ssh see how much bandwidth (in mbit/s or so) a F5 BigIp unit utilizes over a particular interface? ifconfig can be used to see number of packets and tcpdump can be used to capture the actual traffic but I want for debugging purposes see how many mbit/s there are for a particular interface.
Ideas on how to measure performance hit of SSL on a VIP?
Running LTM 9.3.1, users complain of response time problems for websites through it, as compared to hitting the website bypassing the F5. Lots of places to look, I know, but we use a tool called HTTPWatch and it points out that on a web page load that might have 50 objects, some of them (always gifs and jpgs), the "time chart" or breakdown of where the time was spent to download that item shows an unusually long time in the TCP Connect portion of the Get. HTTPWatch help says "Connect is the time required to create a TCP connection to the web server (or proxy). If a secure HTTPS connection is being used this time includes the SSL handshake process. Keep-Alive connections are often used to avoid the overhead of repeatedly connecting to the web server." SSL Overhead is one part of this slice of time, and when my testers go around the F5 they're not hitting the server using SSL (and time to load a page is cut in half or better). I set up a test VIP to try and prevent SSL Offload, however some of the redirects on the page still go to HTTPS for a subset of GIFs (I haven't gotten Redirect Rewrite to rewrite all the Redirects correctly). The thing I notice, which drives this question, is that it's pretty consistent when there's a "[1 to 3 second] delay" in grabbing an object on the webpage, it's always when that object is at HTTPS://.... and never when the object is just HTTP://... So I'm wondering if you have any thoughts on how to measure just how much overhead/delay SSL processing adds. It's not consistently on the same objects for each repeat of the same page loading, but there's always at least one or two objects SSL protected on the page that throw out one of these relatively long delays. Changing our website to not use SSL is not an option, but if our LTM is the bottleneck in it's ability to handle the SSL TPS (which according to the onboard Performance graph is <=20), I'd like to know that. If you look at the screenshot attached to this post, the picture really is worth a thousand words.
High CPU usage?
Hi, one of our ASM biggest customer is worried about the performance of his new LTM + ASM machine. They own a pair of 6900 running 9.4.8.HF4 with LTM + ASM (full of licensed capabilities). And what they can see in performance graphics is almost (always) a baseline of 50% cpu usage for CPU00, CPU01 and CPU02 (CPU03 is 100% by TMM, that's normal). If we have a look at CPU usage with top we can see the following: 3 bd processes, 2 running in CPU 1 and 2 in CPU 2, using 12.2% CPU. TMM running in CPU 3 is using 25% of CPU (normal), mysqld is just using 01% of CPU... As I can see it, there is no justification for that 50% that is always shown in performance graphics. The machine has a baseline of 3K HTTP connections with peaks of 5K. New connections has a baseline of 200 each second with peaks of 500. SSL transactions are 200 per second, with peaks of 300. CPU usage graphics show 50% of constant usage with peaks of 70% (matching new connections and HTTP transactions peaks). ASM CPU utilization graphics show 12% of constant CPU usage, with peaks of 20% (matching the other peaks). As I can see it, maybe just system CPU usage (showed by top as 16-23% for the 3 CPU) is a little high, but the customer is not sure. Any appreciation?About 100% tmm cpu (50% in dual CPU) usage for BIG-IP 6400 LTM
Hello, We use two F5 BIG-IP 6400 LTM for Active-Standby mode. I've found that our tmm use 50% CPU resource in 6400 (in fact it's 100% in cpu01). First question is, is this normal behavior ? Both active & standby LTM use 100% in single cpu. Second question is how many req/sec does 6400 LTM can handle ? Thanks,
iRule test & analysis tool
I'm looking for an iRule analyzer that would allow testing simple traffic scenarios independent of configuring a BIG-IP device. The user would specify values for system variables such as [HTTP:uri] and [HTTP:cookie] and the analyzer would output rewrite url and name of pool routed to , or redirect url , etc. Also would be nice if the tool would also analyzed the iRule for optimum code and performance load testing
