ADC03: Incomplete Observability – A Critical Application Delivery Challenge
Observability is the backbone of modern application delivery, enabling the detection of performance issues, analyzing system usage, and monitoring overall health. However, Incomplete Observability, characterized by insufficient logging, inadequate monitoring tools, and inconsistent data collection, introduces significant business risks. These risks range from limited visibility into performance bottlenecks and prolonged service disruptions to flawed scaling decisions and inefficient operations. To address these challenges effectively, it is crucial to understand the core issues at hand and implement robust strategies and tools, such as F5 BIG-IP and OpenTelemetry, that enhance observability across the infrastructure. Let's explore the impacts of Incomplete Observability and practical solutions, incorporating lessons from a real-world use case. AI Reference Architecture In an AI-powered application ecosystem, observability plays a pivotal role in coordinating and monitoring interactions between end users, frontend applications, and inference services. The following AI Reference Architecture diagram illustrates a typical flow: Diagram Overview: End Users initiate requests through frontend applications. These applications connect with backend Large Language Models (LLMs) based on user-specific needs. Inference Services operate at the core, processing data to deliver accurate and efficient results. Monitoring critical pipelines ensures reliability and scalability while maintaining secure data flows. By aligning observability with such an architecture, AI systems can handle complex pipelines effectively, optimizing performance, security, and governance. Consequences of Incomplete Observability Impact on Performance The absence of complete observability limits an organization’s ability to proactively detect and resolve performance bottlenecks. Without detailed insights into key metrics like latency, response times, and resource utilization, it becomes nearly impossible to identify root causes or improve application responsiveness. For example, undetected spikes in CPU or memory usage can lead to degraded user experiences and even system crashes. Impact on Availability Incomplete observability hampers availability—an essential component of application delivery. Downtime and overlooked critical failures are costly, with 32% of organizations reporting an average outage cost exceeding $500,000 per hour (New Relic). For distributed systems, limited visibility can cause cascading failures, with a minor issue in one system component triggering widespread service interruptions before being detected. Impact on Scalability Dynamic and scalable infrastructure is essential for supporting modern applications with variable workloads. Incomplete observability creates significant obstacles in tracking traffic trends and resource utilization accurately, leading to resource under-provisioning or over-provisioning that wastes budgetary resources or results in outages. Impact on Operational Efficiency Operational inefficiencies arise when IT teams are forced to sift through fragmented, inconsistent data sets to identify issues. Logs spread across incompatible formats or disconnected tools lead to delays in troubleshooting and limited optimization opportunities. This reduces teams' ability to respond to incidents promptly and improve overall system performance. Best Practices for Overcoming Observability Gaps F5's BIG-IP and OpenTelemetry address these challenges by delivering end-to-end observability capabilities requiring real-time insights into application health, performance bottlenecks, and operational metrics. These tools facilitate timely root cause analysis and enable proactive management of distributed systems. Enhanced Observability Framework: Use Case Overview The following diagram illustrates a practical implementation of comprehensive observability using tools like F5 BIG-IP and OpenTelemetry: Use Case Breakdown Consolidate Traffic via F5 BIG-IP LT BIG-IP LTM acts as a centralized point for SSL termination, iRules, and high-speed logging, capturing critical metrics like latency, VIP health, and trace IDs. Traffic is centrally analyzed to provide real-time visibility into application flow dynamics. Capture and Export Logs & Metrics Key metrics, logs, traces, and request IDs are captured and exported for downstream analysis. Logs are standardized across systems, ensuring that valuable data isn't lost in noise. Standardize Observability with OpenTelemetry OpenTelemetry normalizes diverse observability patterns into a unified data model. This enables cross-system compatibility and real-time trend comparisons in distributed environments. Implement Dynamic Alerts & Automated Responses Configure dynamic alerting systems to notify teams when anomalies are detected and integrate automated responses for tasks such as scaling resources or rerouting traffic. Create Unified Dashboards & Analytics Observability platforms like ELK, Prometheus, and Datadog aggregate logs and metrics into a central dashboard, delivering actionable intelligence to IT teams. Establish Feedback Loops for Continuous Improvement Feedback loops using historical performance data enable ongoing improvements in application delivery processes. Insights refine operational decisions and better align infrastructure with real-time demand. Key Benefits Enhanced visibility into application flows, including API interactions, access patterns, and system utilization. Rapid issue detection and mitigation using real-time analytics and automated responses. Resource optimization ensures cost-effective scaling aligned with workload demands. Improved governance and security through dynamic control of inter-application communications. Conclusion Incomplete observability disrupts critical aspects of performance, availability, scalability, and operational efficiency. By leveraging solutions like F5 BIG-IP and OpenTelemetry, alongside enhanced observability frameworks, organizations can address visibility gaps effectively. Dynamic alerting systems, unified dashboards, and standardization tools enable real-time insights, fostering a culture of data-driven decisions and continuous service improvement. Observability is no longer just a supporting feature. It has become the strategic foundation for reliable, high-performing, and secure digital ecosystems. Start improving your observability practices today to achieve long-term success in application delivery. Reference Articles Enhancing BIG-IP with F5 Distributed Cloud: Automated Service Discovery for Scalable Application Delivery and Security Adopting SRE practices with F5: Observability and beyond with ELK Stack Monitor Application Availability with F5 BIG-IP LTM Why Application Observability and Insights Matter Gain insights into the performance of your F5 BIG-IP LTM and DNS solutions ADSP Platform overview The Application Delivery Top 10 AI reference architecture
Devops Proverb: Process Practice Makes Perfect
#devops Tools for automating – and optimizing – processes are a must-have for enabling continuous delivery of application deployments Some idioms are cross-cultural and cross-temporal. They transcend cultures and time, remaining relevant no matter where or when they are spoken. These idioms are often referred to as proverbs, which carries with it a sense of enduring wisdom. One such idiom, “practice makes perfect”, can be found in just about every culture in some form. In Chinese, for example, the idiom is apparently properly read as “familiarity through doing creates high proficiency”, i.e. practice makes perfect. This is a central tenet of devops, particularly where optimization of operational processes is concerned. The more often you execute a process, the more likely you are to get better at it and discover what activities (steps) within that process may need tweaking or changes or improvements. Ergo, optimization. This tenet grows out of the agile methodology adopted by devops: application release cycles should be nearly continuous, with both developers and operations iterating over the same process – develop, test, deploy – with a high level of frequency. Eventually (one hopes) we achieve process perfection – or at least what we might call process perfection: repeatable, consistent deployment success. It is implied that in order to achieve this many processes will be automated, once we have discovered and defined them in such a way as to enable them to be automated. But how does one automate a process such as an application release cycle? Business Process Management (BPM) works well for automating business workflows; such systems include adapters and plug-ins that allow communication between systems as well as people. But these systems are not designed for operations; there are no web servers or databases or Load balancer adapters for even the most widely adopted BPM systems. One such solution can be found in Electric Cloud with its recently announced ElectricDeploy. Process Automation for Operations ElectricDeploy is built upon a more well known product from Electric Cloud (well, more well-known in developer circles, at least) known as ElectricCommander, a build-test-deploy application deployment system. Its interface presents applications in terms of tiers – but extends beyond the traditional three-tiers associated with development to include infrastructure services such as – you guessed it – load balancers (yes, including BIG-IP) and virtual infrastructure. The view enables operators to create the tiers appropriate to applications and then orchestrate deployment processes through fairly predictable phases – test, QA, pre-production and production. What’s hawesome about the tools is the ability to control the process – to rollback, to restore, and even debug. The debugging capabilities enable operators to stop at specified tasks in order to examine output from systems, check log files, etc..to ensure the process is executing properly. While it’s not able to perform “step into” debugging (stepping into the configuration of the load balancer, for example, and manually executing line by line changes) it can perform what developers know as “step over” debugging, which means you can step through a process at the highest layer and pause at break points, but you can’t yet dive into the actual task. Still, the ability to pause an executing process and examine output, as well as rollback or restore specific process versions (yes, it versions the processes as well, just as you’d expect) would certainly be a boon to operations in the quest to adopt tools and methodologies from development that can aid them in improving time and consistency of deployments. The tool also enables operations to determine what is failure during a deployment. For example, you may want to stop and rollback the deployment when a server fails to launch if your deployment only comprises 2 or 3 servers, but when it comprises 1000s it may be acceptable that a few fail to launch. Success and failure of individual tasks as well as the overall process are defined by the organization and allow for flexibility. This is more than just automation, it’s managed automation; it’s agile in action; it’s focusing on the processes, not the plumbing. MANUAL still RULES Electric Cloud recently (June 2012) conducted a survey on the “state of application deployments today” and found some not unexpected but still frustrating results including that 75% of application deployments are still performed manually or with little to no automation. While automation may not be the goal of devops, but it is a tool enabling operations to achieve its goals and thus it should be more broadly considered as standard operating procedure to automate as much of the deployment process as possible. This is particularly true when operations fully adopts not only the premise of devops but the conclusion resulting from its agile roots. Tighter, faster, more frequent release cycles necessarily puts an additional burden on operations to execute the same processes over and over again. Trying to manually accomplish this may be setting operations up for failure and leave operations focused more on simply going through the motions and getting the application into production successfully than on streamlining and optimizing the processes they are executing. Electric Cloud’s ElectricDeploy is one of the ways in which process optimization can be achieved, and justifies its purchase by operations by promising to enable better control over application deployment processes across development and infrastructure. Devops is a Verb 1024 Words: The Devops Butterfly Effect Devops is Not All About Automation Application Security is a Stack Capacity in the Cloud: Concurrency versus Connections Ecosystems are Always in Flux The Pythagorean Theorem of Operational RiskRequiring an SSL Certificate for Parts of an Application
When building many enterprise web-based applications, security must be taken seriously. iRules provide powerful capabilities for influencing security decisions when processing for your web services and applications. This is a rule for requiring a client certificate for certain parts of an application. The example below requires a certificate when the URL path is for a certain directory. Alternatively, a rule could be written to check the host name of file extension if that is more appropriate for your requirements. Special Notes: this rule requires version 9.0.2 to operate correctly. Log statements are commented out. For testing, they can be uncommented. When the client connects, we set up variables to record two things - whether a certificate has been received and whether a certficate needs to be received. These variables start out with a value of zero, which means "false". when CLIENT_ACCEPTED { set needcert 0 set gotcert 0 } When a client does an SSL handshake, this rule event is triggered. This is the time to validate that a certificate has been received. If a certificate has not been received, but we were expecting one ($needcert == 1), then the connection is rejected. If the certificate has been received, we note that for future reference (set gotcert 1) and we release the current request (HTTP::release) if we were waiting for a certificate before releasing the request. when CLIENTSSL_HANDSHAKE { # log LOCAL0.warn "cert count=[SSL::cert count] result=[SSL::verify_result]" if { [SSL::cert count] == 0 or [SSL::verify_result] != 0 } { # log LOCAL0.warn "Bad cert!" if { $needcert == 1 } { reject } } else { # log LOCAL0.warn "Good cert! ($needcert)" set gotcert 1 if { $needcert == 1 } { HTTP::release } } } Here we process an HTTP request. If the request is for a directory that has been designated for extra security, then several things happen. We freeze the HTTP request until the client certificate is received, we tell SSL to require a certificate, we tell SSL to renegotiate now, and then we set a flag that indicates we need a certificate. when HTTP_REQUEST { if { $gotcert == 0 and [HTTP::uri] starts_with "/needcert" } { # log LOCAL0.warn "Requiring certificate..." HTTP::collect SSL::cert mode require SSL::renegotiate set needcert 1 } else { # log LOCAL0.warn "No cert needed." } } Questions about this iRule? Post them in the Technical Forum.Offload Authentication with iRules
As the applications being driven by webservers become more and more complex, Applications Developers are always looking for ways to increase efficiency or do away with unneeded processing time. One of the ways that I believe that Applications can do that is by making use of an intelligent network infrastructure. Assuming the network that is delivering the applications is an intelligent, application aware one, there are many things that Developers can do to help offload some of the work their code would normally have the web server processing to the network level. One such thing that can be offloaded in many cases, is authentication. By leaving the heavy lifting of ensuring only authorized users are acessing the application(s) in question to the network, the web server is free to use its processing power to deliver the application it's hosting faster and more reliably. This is especially true in highly taxed environments. The example below shows one way in which this can be done. This example uses an HTTP cookie to store authentication information for each user, which is a common practice for many applications. It is getting the data to be stored, I.E. whether or not a user is properly authenticated, by enlisting the authentication server already in place on this hypothetical network. In this specific example, that authentication system is a Radius system, but the iRule works equally well with LDAP, tacacs, etc. If the authentication attempt is successful, a cookie will be sent to client with the appropriate data to be stored. The next time that client tries to access the application, the AUTH cookie is present and valid, so the client will be passed immediately without being re-checked for authentication. If it is not succesful, well, then you can decide what experience that user should have by altering the code in the "AUTH_FAILURE" section, or leave the standard 401 error message that stands there now. In this example, the cookie name, password, domain should be properly modified for your environment. This code comes from the DevCentral iRules CodeShare where you can find many useful iRules examples, not to mention post your own to share with the community. Just another great example of how with iRules, you can... 😉 when CLIENT_ACCEPTED { set authinsck 0 set forceauth 1 set ckname BIGXAUTH set ckpass 1xxx5678 set ckvalue [IP::client_addr] set ckdomain .y.z set asid [AUTH::start pam default_radius] } when HTTP_REQUEST { if {[HTTP::cookie exists $ckname]} { HTTP::cookie decrypt $ckname $ckpass 128 if {[HTTP::cookie value $ckname] eq $ckvalue} { set forceauth 0 } HTTP::cookie remove $ckname } if {$forceauth eq 1} { AUTH::username_credential $asid [HTTP::username] AUTH::password_credential $asid [HTTP::password] AUTH::authenticate $asid HTTP::collect } } when HTTP_RESPONSE { if {$authinsck eq 1} { HTTP::cookie insert name $ckname value $ckvalue path / domain $ckdomain HTTP::cookie secure $ckname enable HTTP::cookie encrypt $ckname $ckpass 128 } } when AUTH_SUCCESS { if {$asid eq [AUTH::last_event_session_id]} { set authinsck 1 HTTP::release } } when AUTH_FAILURE { if {$asid eq [AUTH::last_event_session_id]} { HTTP::respond 401 "WWW-Authenticate" "Basic realm=\"\"" } } when AUTH_WANTCREDENTIAL { if {$asid eq [AUTH::last_event_session_id]} { HTTP::respond 401 "WWW-Authenticate" "Basic realm=\"\"" } } when AUTH_ERROR { if {$asid eq [AUTH::last_event_session_id]} { HTTP::respond 401 } }
LTM: HTTP Monitoring with POST Request
I recently discovered that you can use LTM's built in HTTP monitor template to send a POST request to a server to verify its health. It's surprisingly straightforward in comparison with the external monitor approach I've used in the past. This article will cover the details you'll need to properly implement an HTTP POST monitor using the built in monitor template. The built in HTTP monitor template The HTTP monitor template provides a way to send a single request to the server, monitor the response for an expected string, and mark the pool member down if the expected response isn't seen within the configured timeout. The options of interest for this exercise are the Send string and the Receive string. The Send string is where the request is configured. It should include the method (in this case, POST), the URI to request from the server, the HTTP version, some standard HTTP headers (including Basic Auth, if applicable), and in this case, the POST body. The server should respond in a predictable way to this request each time it is issued. The Receive string should include a phrase or non-dictionary word that will appear only in that expected response. The Receive string can use basic regular expressions to accommodate per-server or other minor variations in the expected response. Capturing the the transaction The first thing you'll need to do, as with any monitor, is to identify and capture the transaction you'd like to replicate in order to validate the health of your application servers. You can run tcpdump to watch live production traffic and capture the target request and response, but it's safer to run an isolated test request at the command line to more exactly replicate the action of the LTM monitor. The approach I recommend uses both techniques. First capture a trace of production traffic so you have a known-good working example with all the expected headers. Then log into the LTM command line and use telnet or cURL to send the same request to the server along with the observed headers, and monitor the response, adjusting the request until the expected response is received. Here's the cURL syntax you'll need to use for command line request submission: curl -N -H <headers> -u <user:password> -d <post body> http://<pool_mmeber_IP>:<pool_mmeber_port><URI> You'll need to examine the actual data that's being passed, including whitespace characters. A packet trace can be used to look directly at the conversation on the wire. Capture on the server-facing interface. To examine only production traffic, filter for the IP/port of the pool member, excluding the non-floating self IP on the server-facing VLAN: tcpdump -nni <server_vlan> -Xs 0 host <pool_member_IP> and port <pool_member_port> and not host <internal_non-floating_selfIP> (Filtering out the non-floating self IP prevents monitor traffic from appearing in the trace.) To examine your command line request, filter instead for the non-floating self IP on the server-facing VLAN and the IP/port of the pool member: tcpdump -nni <server_vlan> -Xs 0 host <internal_non-floating_selfIP> and host <pool_member_IP> and port <pool_member_port> Here's a sample transaction trace for a SOAP/xml request: Once you have identified the transaction and validated the required headers, you are ready to build your custom monitor. Creating the custom HTTP monitor The information you've gathered so far should be everything you need to configure the custom HTTP monitor. Start by creating a monitor of type "http" in the LTM GUI (Local Traffic -> Monitors -> Create). Configure Interval and Timeout as appropriate for your application. Set Username and Password if Basic Auth credentials are required by the server. Then you'll need to dig into your transaction trace data to construct the Send string containing all the request details. For the Send string, the HTTP method must be specified along with the URI and HTTP version v1.1 on the same line, followed by an encoded LF/CR (\r\n), then the Host header (required for HTTP 1.1). For the trace above, we'll start with: POST /XAIApp/xaiserver HTTP/1.1\r\nHost: 191.168.1.1:4200 Add the other headers, all separated by a single encoded CR/LF. (Line breaks are only for display. No literal carriage returns are inserted into the Send string.) Include the Basic Auth header, if applicable, even though it is specified elsewhere in the monitor template. With headers, the Send string now looks like this: POST /XAIApp/xaiserver HTTP/1.1\r\nHost: 191.168.1.1:4200\r\n Authorization: Basic c3lzdXNlcjpzeXN1c2VyMDA= \r\n Pragma: no-cache\r\nAccept: */*\r\nContent-Length: 673\r\nContent-Type: application/x-www-form-urlencoded Finally add the POST body, preceded by a double CR/LF to indicate the end of the headers. If the POST body contains double quotes or braces, you will need to escape them with a preceding backslash so they are interpreted correctly by LTM. The completed Send string now looks like this: POST /XAIApp/xaiserver HTTP/1.1\r\nHost: 191.168.1.1:4200\r\n Authorization: Basic c3lzdXNlcjpzeXN1c2VyMDA= \r\n Pragma: no-cache\r\nAccept: */*\r\nContent-Length: 673\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n <SOAP-ENV:Envelope xmlns:SOAP-ENV=\"urn:schemas-xmlsoap-org:envelope\"> <SOAP-ENV:Body><SSvcInstallationOptionsMaintenance dateTimeTagFormat=\"CdxDateTime\" transactionType =\"READ\" ><SSvcInstallationOptionsMaintenanceService> <SSvcInstallationOptionsMaintenanceHeader InstallationOptionID=\"1\"/><SSvcInstallationOptionsMaintenanceDetails InstallationOptionID=\"1\"> <InstallMsg><InstallMsgHeader InstallationOptionID=\"1\"/></InstallMsg><InstallAlg><InstallAlgHeader InstallationOptionID=\"1\"/></InstallAlg> <Module></Module></SSvcInstallationOptionsMaintenanceDetails></SSvcInstallationOptionsMaintenanceService></SSvcInstallationOptionsMaintenance> </SOAP-ENV:Body></SOAP-ENV:Envelope> To complete the monitor definition, enter the phrase or string expected in a healthy server response into the Receive string field. For the trace above, we'll use: Standard automatic payment creation The resulting monitor configuration will look like this: monitor http-post-soapenv { defaults from http interval 10 timeout 31 password "password" recv "Standard automatic payment creation" send "POST /XAIApp/xaiserver HTTP/1.1\r\nHost: 191.168.1.1:4200\r\nAuthorization: Basic c3lzdXNlcjpzeXN1c2VyMDA= \r\n Pragma: no-cache\r\nAccept: */*\r\n Content-Length: 673\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n <SOAP-ENV:Envelope xmlns:SOAP-ENV=\"urn:schemas-xmlsoap-org:envelope\"> <SOAP-ENV:Body><SSvcInstallationOptionsMaintenance dateTimeTagFormat=\"CdxDateTime\" transactionType =\"READ\" ><SSvcInstallationOptionsMaintenanceService> <SSvcInstallationOptionsMaintenanceHeader InstallationOptionID=\"1\"/><SSvcInstallationOptionsMaintenanceDetails InstallationOptionID=\"1\"> <InstallMsg><InstallMsgHeader InstallationOptionID=\"1\"/></InstallMsg><InstallAlg><InstallAlgHeader InstallationOptionID=\"1\"/></InstallAlg> <Module></Module></SSvcInstallationOptionsMaintenanceDetails></SSvcInstallationOptionsMaintenanceService></SSvcInstallationOptionsMaintenance> </SOAP-ENV:Body></SOAP-ENV:Envelope>" username "user" } Related information These 2 previous articles are specific to external monitors, but contain some more useful details about command line testing and capturing traffic: LTM External Monitors: The Basics LTM External Monitors: TroubleshootingPart 3: Monitoring the health of BIG-IP APM network access PPP connections with a periodic iCall handler
In this part, you monitor the health of PPP connections on the BIG-IP APM system by monitoring the frequency of a particular log message in the /var/log/apm file. In this log file, when the system is under high CPU load, you may observe different messages indicating users' VPN connections are disconnecting. For example: You observe multiple of the following messages where the PPP tunnel is started and closed immediately: Mar 13 13:57:57 hostname.example notice tmm3[16095]: 01490505:5: /Common/accessPolicy_policy:Common:ee6e0ce7: PPP tunnel 0x5604775f7000 (ID: 2c4507ea) started. Mar 13 13:57:57 hostname.example notice tmm3[16095]: 01490505:5: /Common/accessPolicy_policy:Common:ee6e0ce7: PPP tunnel 0x560478482000 (ID: acf63e47) closed. You observe an unusually high rate of log messages indicating users' APM sessions terminated due to various reasons in a short span of time. Mar 13 14:01:10 hostname.com notice tmm3[16095]: 01490567:5: /Common/AccessPolicy_policy:Common:a7cf25d9: Session deleted due to user inactivity. You observe an unusually high rate of log messages indicating different user apm session IDs are attempting to reconnect to network access, VPN. Mar 13 12:18:13 hostname notice tmm1[16095]: 01490549:5: /Common/AP_policy:Common:2c321803: Assigned PPP Dynamic IPv4: 10.10.1.20 ID: 033d0635 Tunnel Type: VPN_TUNNELTYPE_TLS NA Resource: /Common/AP_policy_na_res Client IP: 172.2.2.2 – Reconnect In this article, you monitor the Reconnect message in the last example because it indicates that the VPN connection terminated and the client system still wants to reconnect. When this happens for several users and repeatedly, it usually indicates that the issue is not related to the client side. The next question is, what is an unusually high rate of reconnect messages, taking into account the fact that some reconnects may be due to reasons on the client side and not due to high load on the system. You can create a periodic iCall handler to run a script once every minute. Each time the script runs, it uses grep to find the total number of Reconnect log messages that happened over the last 3 minutes for example. When the average number of entries per minute exceeds a configured threshold, the system can take appropriate action. The following describes the procedures: Creating an iCall script to monitor the rate of reconnect messages Creating a periodic iCall handler to run the iCall script once a minute Testing the implementation using logger 1. Creating an iCall script to monitor the rate of reconnect messages The example script uses the following values: Periodic handler interval 1 minute: This runs the script once a minute to grep the number of reconnect messages logged in the last 3.0 minutes. Period 3.0 minutes: The script greps for the Reconnect messages logged in the last three minutes. Critical threshold value 4: When the average number of Reconnect messages per minute exceeds four or a total of 12 in the last three minutes, the system logs a critical message in the /var/log/apm file. Alert threshold value 7: When the average number of Reconnect messages per minute exceeds seven or a total of 21 in the last three minutes, the system logs an alert message and requires immediate action. Emergency threshold 10: When the BIG-IP system is logging 10 Reconnect messages or more every minute for the last three minutes or a total of 30 messages, the system logs an emergency message, indicating the system is unusable. You should reconfigure these values at the top of the script according to the behavior and set up of your environment. By performing the count over an interval of three minutes or longer, it reduces the possibility of high number of reconnects due to one-off spikes. Procedure Perform the following procedure to create the script to monitor CPU statistics and log an alert message in the /var/log/ltm file when traffic exceeds a CPU threshold value. To create an iCall script, perform the following procedure: Log in to tmsh. Enter the following command to create the script in the vi editor: create sys icall script vpn_reconnect_script 3. Enter the following script into the definition stanza in the editor: This script counts the average number of entries/min the Reconnect message is logged in /var/log/apm over a period of 3 minutes. definition { #log file to grep errormsg set apm_log "/var/log/apm" #This is the string to grep for. set errormsg "Reconnect" #num of entries = grep $errormsg $apm_log | grep $hourmin | wc -l #When (num of entries < crit_threshold), no action #When (crit_threshold <= num of entries < alert_threshold), log crtitical message. #When (alert_threshold <= num of entries < emerg_threshold), log alert message. #When (emerg_threhold <= num of entries log emerg message set crit_threshold 4 set alert_threshold 7 set emerg_threshold 10 #Number of minutes to take average of. E.g. Every 1.0, 2.0, 3.0, 4.0... minutes set period 3.0 #Set this to 1 to log to /var/tmp/scriptd.out. Set to 0 to disable. set DEBUG 0 set total 0 puts "\n[clock format [clock seconds] -format "%b %e %H:%M:%S"] Running script..." for {set i 1} {$i <= $period} {incr i} { set hourmin [clock format [clock scan "-$i minute"] -format "%b %e %H:%M:"] set errorcode [catch {exec grep $errormsg $apm_log | grep $hourmin | wc -l} num_entries] if {$errorcode} { set num_entries 0 } if {$DEBUG} {puts "DEBUG: $hourmin \"$errormsg\" logged $num_entries times."} set total [expr {$total + $num_entries}] } set average [expr $total / $period] set average [format "%.1f" $average] if {$average < $crit_threshold} { if {$DEBUG} {puts "DEBUG: $hourmin \"$errormsg\" logged $average times on average. Below all threshold. No action."} exit } if {$average < $alert_threshold} { if {$DEBUG} {puts "DEBUG: $hourmin \"$errormsg\" logged $average times on average. Reached critical threshold $crit_threshold. Log Critical msg."} exec logger -p local1.crit "01490266: \"$errormsg\" logged $average times on average in last $period mins. >= critical threshold $crit_threshold." exit } if {$average < $emerg_threshold} { if {$DEBUG} {puts "DEBUG: $hourmin \"$errormsg\" logged $average times on average. Reached alert threshold $alert_threshold. Log Alert msg."} exec logger -p local1.alert "01490266: \"$errormsg\" logged $average times on average in last $period mins. >= alert threshold $alert_threshold." exit } if {$DEBUG} {puts "DEBUG: $hourmin \"$errormsg\" logged $average times on average in last $period mins. Log Emerg msg"} exec logger -p local1.emerg "01490266: \"$errormsg\" logged $average times on average in last $period mins. >= emerg threshold $emerg_threshold." exit } 4. Configure the variables in the script as needed and exit the editor by entering the following command: :wq! y 5. Run the following command to list the contents of the script: list sys icall script vpn_reconnect_script 2. Creating a periodic iCall handler to run the script In this example, you create the iCall handler to run the script once a minute. You can increase this interval to once every two minutes or longer. However, you should consider this value together with the period value of the script in the previous procedure to ensure that you're notified on any potential issues early. Procedure Perform the following procedure to create the periodic handler that runs the script once a minute. To create an iCall periodic handler, perform the following procedure: Log in to tmsh. Enter the following command to create a periodic handler: create sys icall handler periodic vpn_reconnect_handler interval 60 script vpn_reconnect_script 3. Run the following command to list the handler: list sys icall handler periodic vpn_reconnect_handler 4. You can start and stop the handler by using the following command syntax: <start|stop> sys icall handler periodic vpn_reconnect_handler 3. Testing the implementation using logger You can use the logger command to log test messages to the /var/log/apm file to test your implementation. To do so, run the following command the required number of times to exceed the threshold you set: Note: The following message must contain the keyword that you are searching for in the script. In this case, the keyword is Reconnect. logger -p local1.notice "01490549:5 Assigned PPP Dynamic IPv4: 10.10.1.20 ID: 033d0635 Tunnel Type: VPN_TUNNELTYPE_TLS NA Resource: /Common/AP_policy_na_res Client IP: 172.2.2.2 - Reconnect" Follow the /var/tmp/scriptd.out and /var/log/apm file entries to verify your implementation is working correctly. Conclusion This article lists three use cases for using iCall to monitor the health of your BIG-IP APM system. The examples mainly log the appropriate messages in the log files. You can extend the examples to monitor more parameters and also perform different kinds of actions such as calling another script (Bash, Perl, and so on) to send an email notification or perform remedial action.Web Application Login Integration with APM
As we hurtle forward through the information age we continue to find ourselves increasingly dependant on the applications upon which we rely. Whether it's your favorite iPhone app or the tools that allow you to do your job, the applications that you interact with and the information contained within are a sacred thing to the average IT warrior. As such, we hold the security of said applications and data in high regard. This leads to entire security teams dreaming up more complex ways to protect your information along more hoops for you to jump through to gain access. Their intentions are pure and the added barriers to entry by way of complex login processes, password security requirements, access enforcement, etc. are justified, but that makes them no less cumbersome. For the remote users the saga is even worse. While the local application user gripes about the latest policy enforced for their own good by the security team the remote user deals with the same restrictions and then some, often including the added step of having to "remote in" via VPN of some sort before even being allowed access to the resources they require. Fortunately though, it doesn't have to be this way. One of the many features F5's Access Policy Manager (APM) offers is seamless integration into your existing application or portal. By configuring APM to accept credentials passed via a simple POST from any application, it can truly be a transparent gateway into your authentication infrastructure. With this configuration, whether the user is internal or external, simply logging into the portal/app in question will pass their credentials to APM which can them seamlessly assign them to the appropriate resources, granting or limiting their access as necessary. The setup is simple. Let's walk through it, assuming we have a basic Virtual IP set up on the APM, and our application has a simple login form, something like: Note that the action on the above form is https://10.10.12.8/fake. This is a "fake" URI on the APM device that's going to be doing the heavy lifting here (The Virtual Server's IP on the APM is 10.10.12.8). The form will look like you would expect, like this: Now that we have a form to play with we'll need to capture the credentials being passed when a user submits the form to log in. This is done via a simple, flexible iRule. We'll input the "/fake" as the URI we're searching for, then do some straight-forward logic to identify the username and password in the POST from the form. Once we have that data singled out, we'll store it in session variables for later use in APM's auth process. The whole thing looks like this: Once the iRule is created and assigned to the APM virtual (10.10.12.8, from above) we should have the data collected and stored as needed. Next we need to instruct APM to access those session variables we set earlier. To do so we'll use the Visual Policy Editor built into APM to make constructing login flows simple. First add an empty event to check for the credentials. We'll call that "have creds". Next, be sure you have some form of authentication set up to actually authenticate the credentials passed in via the form once the "have creds" event succeeds, otherwise anyone will be able to log in, and that's not what we're looking for. For that we've set up a standard AD Auth event in APM which will fire and perform the actual authentication step once we've successfully collected the data from the POST. The setup in the VPE should look something like this: Now that we have the basic structure in place, let's make it do something. Create a new Branch Rule (also named have creds for this example, appearing on the line between have creds and AD Auth) that makes use of an advanced check. We'll tell it to look for the session.logon.last.username variable that we set in the iRule to ensure that the rule was successful. If this check is successful it means that the iRule found a username in the POST from the form and we can safely pass the username and password variables on to the AD Auth event for processing. So, assuming the username and password entered in the original HTML form are correct, once you hit "Logon" to submit the form, your information will be passed back through APM for authentication via AD, and then assigned whatever resources are deemed appropriate for that user based on the "Resource Assign" action. There you have it, login integration to assign resource access. This could of course be done with any login form, not just the simple HTML example above, to save users one more hoop to jump through before getting access to the applications they depend on.iRule Maintenance Windows
A fun, but not well known, feature on BIG-IP is the Statistics Profile. This tech tip is the second in a series on how the Statistics Profile and iRules, when working together, can save time, productivity, and your sleep! The Scenario: Imagine yourself, Mr./Mrs. IT Manager, working for a company that's business relies on updated content on it's back-end web servers. Your company has a web team that has decided that it's test cycle will always end at 3:00AM (your time) on Saturday morning. So, here you are, arriving at home after a night of wild karaoke and one too many drinks with cheap umbrellas and you get a call from the dev team that they need the site taken off line so they can push out the new content. In the past, you gave the dev lead the keys to the castle (meaning the admin credentials for your BIG-IP) and he/she somehow thought "rm -rf /" was how you took a virtual server offline. After an emergency restore of the software you vowed to never give that department direct access to your servers - ever... So, now you are stuck with being on call whenever they deem updates are needed. If only, you could provide your dev team access to control taking their application offline while they are doing updates. Oh, and they should also be able to view the current status as well as being able to bring it back online. Never fear, help is on the way! There are a couple of handy features within iRules that make this problem a thing of the past. The Setup: To get things going, you'll need a Virtual Server setup fronting your web application. Since your application is up and running, that step is already taken care of. Next, you'll need to create a Statistics Profile, assign it to your Virtual Server and write an iRule. So, first we'll create the statistics profile... Create the Statistics Profile Login to the BIG-IP Administrative GUI. Select the Profiles option under Local Traffic and Virtual Servers Select the Statistics from the Other menu. Click the Create button Enter "maintenance_window" for the Profile name Add the following fields: day, start_time, end_time Click Update to create the profile Create the iRule Now that the virtual server is setup with the statistics profile, you'll need to create an iRule to to add the controlling functions as well as the maintenance window enforcement. Select Rules from the Local Traffic/Virtual Servers menu. Click the Create button. Enter "maintenance_window" for the iRule name and enter the iRule from below into the Definition text box. Click Finished to save the iRule Apply the statistics profile to your virtual server The statistics profile doesn't do much until you apply it to your virtual server properties. Here's how: Select Virtual Servers from the Local Traffic menu. Click on your Virtual Server to enter it's properties Make sure the Advanced Configuration option is selected, and scroll down to the Statistics Profile option Select the previously created profile maintenance_window Click the Update Button Select the Resources Menu Click the Manage Rules button Make sure the maintenance_window iRule is in the Enabled list box and click the Finished button. The iRule Now for the fun, here's the iRule that will give you peace, tranquility (and a good nights sleep). This iRule will look for commands in the URI to control the maintenance window settings (see the "/usage" section for details). when HTTP_REQUEST { set TITLE "iRule Maintenance Window Control" set PROFILE_NAME "maintenance_window" # Look for embedded commands to control the maintenance window. switch -glob [string tolower [HTTP::uri]] { "/enable*" { # Extract the maintenance window day, start time, and end time from the URI. # If not there, return an error message. set params [split [HTTP::uri] "/"] if { [llength $params] < 5 } { HTTP::respond 200 content " <head><center><title>$TITLE</title> <body><h1>Usage: http://[HTTP::host]/enable/day_number/start_hour/end_hour</h1> </body></html>" } else { # Update the maintenance window values in the statistics profile. STATS::set $PROFILE_NAME "day" [lindex $params 2] STATS::set $PROFILE_NAME "start_time" [lindex $params 3] STATS::set $PROFILE_NAME "end_time" [lindex $params 4] HTTP::respond 200 content " <head><center><title>$TITLE</title> <body><h1>Maintenance Window enabled</h1> </body></html>" } } "/disable" { # By zero'ing out the values in the statistics profile, the maintenance window is not enforced. STATS::set $PROFILE_NAME "day" 0 STATS::set $PROFILE_NAME "start_time" 0 STATS::set $PROFILE_NAME "end_time" 0 HTTP::respond 200 content " <head><center><title>$TITLE</title> <body><h1>Maintenance Window disabled</h1> </body></html>" } "/check" { # Return the current time and the current settings for the maintenance window set day [STATS::get $PROFILE_NAME "day"] set start_time [STATS::get $PROFILE_NAME "start_time"] set end_time [STATS::get $PROFILE_NAME "end_time"] set l [split [clock format [clock seconds] -format {%u %H %M}] " "] set cur_day [lindex $l 0] set cur_time [expr [expr [lindex $l 1]*100] + [lindex $l 2]] HTTP::respond 200 content " <head><center><title>$TITLE</title> <body><h1>Current Date and time: $cur_day, $cur_time</h1> <h1>Maintenance Window: $day, $start_time - $end_time</h1> </body></html>" } "/usage" { # Usage instructions for the commands HTTP::respond 200 content " <head><center><title>$TITLE</title> <body><h1>Usage: http://[HTTP::host]/\[command\]</h1> <table><tr><td><ul> <li>/enable/day_of_week_number(1:mon-7:sun)/start_time(hhmm)/end_time(hhmm) - enable maintenance window.</li> <li>/disable - disable maintenance window.</li> <li>/check - check if in maintenance window.</li> <li>/usage - display this usage message</li> </ul></td></tr></table> </body></html>" } default { # no secret command entered, so check the maintenance window # Get the values from the statistics profile. set day [STATS::get $PROFILE_NAME "day"] set start_time [STATS::get $PROFILE_NAME "start_time"] set end_time [STATS::get $PROFILE_NAME "end_time"] if { ($day ne 0) and ($start_time ne 0) and ($end_time ne 0) } { # Use the TCL "clock" command to get the current time settings. set l [split [clock format [clock seconds] -format {%u %H %M}] " "] set cur_day [lindex $l 0] # Math right here is a bit overkill. I've changed it to string concatenation. #set cur_time [expr [expr [lindex $l 1]*100] + [lindex $l 2]] set cur_time "[lindex $l 1][lindex $l 2]" if { ($cur_day eq $day) && ($cur_time >= $start_time) && ($cur_time <= $end_time) } { # The current time is in the maintenance window, so either issue a HTTP::redirect here to a # prettied up page, or you can do as I've done here, and return a HTML response direction # to the client. HTTP::respond 200 content " <head><center><title>$TITLE</title> <body><h1>This site is down for maintenance, please come back at $end_time ...</h1> </body></html>" } else { # Not in maintenance window, so allow connection to continue to application. } } } } } Conclusion: So, now you should be all set. While your dev team now has control over their site's availability and the 3:00am calls are a thing of the past...RADIUS Load Balancing with iRules
What is RADIUS? “Remote Authentication Dial In User Service” or RADIUS is a very mature and widely implemented protocol for exchanging ”Triple A” or “Authentication, Authorization and Accounting” information. RADIUS is a relatively simple, transactional protocol. Clients, such as remote access server, FirePass, BIG-IP, etc. originate RADIUS requests (for example, to authenticate a user based on a user/password combination) and then wait for a response from the RADIUS server. Information is exchanged between a RADIUS client and server in the form of attributes. User-name, user-password, IP Address, port, and session state are all examples of attributes. Attributes can be in the format of text, string, IP address, integer or timestamp. Some attributes are variable in length, some are fixed. Why is protocol-specific support valuable? In typical UDP Load Balancing (not protocol-specific), there is one common challenge: if a client always sends requests with the same source port, packets will never be balanced across multiple servers. This behavior is the default for a UDP profile. To allow load balancing to work in this situation, using "Datagram LB" is the common recommendation or the use of an immediate session timeout. By using Datagram LB, every packet will be balanced. However, if a new request comes in before the reply for the previous request comes back from the server, BIG-IP LTM may change source port of that new request before forwards it to the server. This may result in an application not acting properly. In this later case, “Immediate timeout” must then be used. An additional virtual server may be needed for outbound traffic in order to route traffic back to the client. In short, to enable load balancing for RADIUS transaction-based traffic coming from the same source IP/source port, Datagram LB or immediate timeout should be employed. This configuration works in most cases. However, if the transaction requires more than 2 packets (1 request, 1 response), then, further BIG-IP LTM work is needed. An example where this is important occurs in RADIUS challenge/response handshakes, which require 4 packets: * Client ---- access-request ---> Server * Client <-- access-challenge --- Server * Client --- access-request ----> Server * Client <--- access-accept ----- Server For this traffic to succeed, all packets associated with the same transaction must be returned to the same server. In this case, custom layer 7 persistence is needed. iRules can provide the needed persistency. With iRules that understand the RADIUS protocol, BIG-IP LTM can direct traffic based on any attribute sent by client or persist sessions based on any attribute sent by client or server. Session management can then be moved to the BIG-IP, reducing server-side complexity. BIG-IP can provide almost unlimited intelligence in an iRule that can even re-calculate md5, modify usernames, detect realms, etc. BIG-IP LTM can also provide security at the application level of the RADIUS protocol, rejecting malformed traffic, denial of service attacks, or similar threats using customized iRules. Solution Datagram LB UDP profile or immediate timeout may be used if requests from client always use the same source IP/port. If immediate timeout is used, there should be an additional VIP for outbound traffic originated from server to client and also an appropriate SNAT (same IP as VIP). Identifier or some attributes can be used for Universal Inspection Engine (UIE) persistence. If immediate timeout/2-side-VIP technique are used, these should be used in conjunction with session command with "any" option. iRules 1) Here is a sample iRule which does nothing except decode and log some attribute information. This is a good example of the depth of fluency you can achieve via an iRule dealing with RADIUS. when RULE_INIT { array set ::attr_code2name { 1 User-Name 2 User-Password 3 CHAP-Password 4 NAS-IP-Address 5 NAS-Port 6 Service-Type 7 Framed-Protocol 8 Framed-IP-Address 9 Framed-IP-Netmask 10 Framed-Routing 11 Filter-Id 12 Framed-MTU 13 Framed-Compression 14 Login-IP-Host 15 Login-Service 16 Login-TCP-Port 17 (unassigned) 18 Reply-Message 19 Callback-Number 20 Callback-Id 21 (unassigned) 22 Framed-Route 23 Framed-IPX-Network 24 State 25 Class 26 Vendor-Specific 27 Session-Timeout 28 Idle-Timeout 29 Termination-Action 30 Called-Station-Id 31 Calling-Station-Id 32 NAS-Identifier 33 Proxy-State 34 Login-LAT-Service 35 Login-LAT-Node 36 Login-LAT-Group 37 Framed-AppleTalk-Link 38 Framed-AppleTalk-Network 39 Framed-AppleTalk-Zone 60 CHAP-Challenge 61 NAS-Port-Type 62 Port-Limit 63 Login-LAT-Port } } when CLIENT_ACCEPTED { binary scan [UDP::payload] cH2SH32cc code ident len auth \ attr_code1 attr_len1 log local0. "code = $code" log local0. "ident = $ident" log local0. "len = $len" log local0. "auth = $auth" set index 22 while { $index < $len } { set hsize [expr ( $attr_len1 - 2 ) * 2] switch $attr_code1 { 11 - 1 { binary scan [UDP::payload] @${index}a[expr $attr_len1 - 2]cc \ attr_value attr_code2 attr_len2 log local0. " $::attr_code2name($attr_code1) = $attr_value" } 9 - 8 - 4 { binary scan [UDP::payload] @${index}a4cc rawip \ attr_code2 attr_len2 log local0. " $::attr_code2name($attr_code1) =\ [IP::addr $rawip mask 255.255.255.255]" } 13 - 12 - 10 - 7 - 6 - 5 { binary scan [UDP::payload] @${index}Icc attr_value \ attr_code2 attr_len2 log local0. " $::attr_code2name($attr_code1) = $attr_value" } default { binary scan [UDP::payload] @${index}H${hsize}cc \ attr_value attr_code2 attr_len2 log local0. " $::attr_code2name($attr_code1) = $attr_value" } } set index [ expr $index + $attr_len1 ] set attr_len1 $attr_len2 set attr_code1 $attr_code2 } } when SERVER_DATA { binary scan [UDP::payload] cH2SH32cc code ident len auth \ attr_code1 attr_len1 log local0. "code = $code" log local0. "ident = $ident" log local0. "len = $len" log local0. "auth = $auth" set index 22 while { $index < $len } { set hsize [expr ( $attr_len1 - 2 ) * 2] switch $attr_code1 { 11 - 1 { binary scan [UDP::payload] @${index}a[expr $attr_len1 - 2]cc \ attr_value attr_code2 attr_len2 log local0. " $::attr_code2name($attr_code1) = $attr_value" } 9 - 8 - 4 { binary scan [UDP::payload] @${index}a4cc rawip \ attr_code2 attr_len2 log local0. " $::attr_code2name($attr_code1) =\ [IP::addr $rawip mask 255.255.255.255]" } 13 - 12 - 10 - 7 - 6 - 5 { binary scan [UDP::payload] @${index}Icc attr_value \ attr_code2 attr_len2 log local0. " $::attr_code2name($attr_code1) = $attr_value" } default { binary scan [UDP::payload] @${index}H${hsize}cc \ attr_value attr_code2 attr_len2 log local0. " $::attr_code2name($attr_code1) = $attr_value" } } set index [ expr $index + $attr_len1 ] set attr_len1 $attr_len2 set attr_code1 $attr_code2 } } This iRule could be applied to many areas of interest where a particular value needs to be extracted. For example, the iRule could detect the value of specific attributes or realm and direct traffic based on that information. 2) This second iRule allows UDP Datagram LB to work with 2 factor authentication. Persistence in this iRule is based on "State" attribute (value = 24). Another great example of the kinds of things you can do with an iRule, and how deep you can truly dig into a protocol. when CLIENT_ACCEPTED { binary scan [UDP::payload] ccSH32cc code ident len auth \ attr_code1 attr_len1 set index 22 while { $index < $len } { set hsize [expr ( $attr_len1 - 2 ) * 2] binary scan [UDP::payload] @${index}H${hsize}cc attr_value \ attr_code2 attr_len2 # If it is State(24) attribute... if { $attr_code1 == 24 } { persist uie $attr_value 30 return } set index [ expr $index + $attr_len1 ] set attr_len1 $attr_len2 set attr_code1 $attr_code2 } } when SERVER_DATA { binary scan [UDP::payload] ccSH32cc code ident len auth \ attr_code1 attr_len1 # If it is Access-Challenge(11)... if { $code == 11 } { set index 22 while { $index < $len } { set hsize [expr ( $attr_len1 - 2 ) * 2] binary scan [UDP::payload] @${index}H${hsize}cc attr_value \ attr_code2 attr_len2 if { $attr_code1 == 24 } { persist add uie $attr_value 30 return } set index [ expr $index + $attr_len1 ] set attr_len1 $attr_len2 set attr_code1 $attr_code2 } } } Conclusion With iRules, BIG-IP can understand RADIUS packets and make intelligent decisions based on RADIUS protocol information. Additionally, it is also possible to manipulate RADIUS packets to meet nearly any application need. Contributed by: Nat Thirasuttakorn Get the Flash Player to see this player.Part 2: Monitoring the CPU usage of the BIG-IP system using a periodic iCall handler
In this part series, you monitor the CPU usage of the BIG-IP system with a periodic iCall handler. The specific CPU statistics you want to monitor can be retrieved from either Unix or tmsh commands. For example, if you want to monitor the CPU usage of the tmm process, you can monitor the values from the output of the tmsh show sys proc-info tmm.0 command. An iCall script can iterate and retrieve a list of values from the output of a tmsh command. To display the fields available from a tmsh command that you can iterate from an iCall Tcl script, run the tmsh command with the field-fmt option. For example: tmsh show sys proc-info tmm.0 field-fmt You can then use a periodic iCall handler which runs an iCall script periodically every interval to check the value of the output of the tmsh command. When the value exceeds a configured threshold, you can have the script perform an action; for example, an alert message can be logged to the /var/log/ltm file. The following describes the procedures: Creating an iCall script to monitor the required CPU usage values Creating a periodic iCall handler to run the iCall script once a minute 1. Creating an iCall script to monitor the required CPU usage values There are different Unix and tmsh commands available to display CPU usage. To monitor CPU usage, this example uses the following: tmsh show sys performance system detail | grep CPU: This displays the system CPU Utilization (%). The script monitors CPU usage from the Average column for each CPU. tmsh show sys proc-info apmd: Monitors the CPU usage System Utilization (%) Last5-mins value of the apmd process. tmsh show sys proc-info tmm.0: Monitors the CPU usage System Utilization (%) Last5-mins value of the tmm process. This is the sum of the CPU usage of all threads of the tmm.0 process divided by the number of CPUs over five minutes. You can display the number of TMM processes and threads started, by running different commands. For example: pstree -a -A -l -p | grep tmm | grep -v grep grep Start /var/log/tmm.start You can also create your own script to monitor the CPU output from other commands, such as tmsh show sys cpu or tmsh show sys tmm-info. However, a discussion on CPU usage on the BIG-IP system is beyond the scope of this article. For more information, refer to K14358: Overview of Clustered Multiprocessing (11.3.0 and later) and K16739: Understanding 'top' output on the BIG-IP system. You need to set some of the variables in the script, specifically the threshold values: cpu_perf_threshold, tmm.0_threshold, apmd_threshold respectively. In this example, all the CPU threshold values are set at 80%. Note that depending on the set up in your specific environment, you have to adjust the threshold accordingly. The threshold values also depend on the action you plan to run in the script. For example, in this case, the script logs an alert message in the /var/log/ltm file. If you plan to log an emerg message, the threshold values should be higher, for example, 95%. Procedure Perform the following procedure to create the script to monitor CPU statistics and log an alert message in the /var/log/ltm file when traffic exceeds a CPU threshold value. To create an iCall script, perform the following procedure: Log in to tmsh. Enter the following command to create the script in the vi editor: create sys icall script cpu_script 3. Enter the following script into the definition stanza of the editor. The 3 threshold values are currently set at 80%. You can change it according to the requirements in your environment. definition { set DEBUG 0 set VERBOSE 0 #CPU threshold in % from output of tmsh show sys performance system detail set cpu_perf_threshold 80 #The name of the process from output of tmsh show sys proc-info to check. The name must match exactly. #If you would like to add another process, append the process name to the 'process' variable and add another line for threshold. #E.g. To add tmm.4, "set process apmd tmm.0 tmm.4" and add another line "set tmm.4_threshold 75" set process "apmd tmm.0" #CPU threshold in % for output of tmsh show sys proc-info set tmm.0_threshold 80 set apmd_threshold 80 puts "\n[clock format [clock seconds] -format "%b %d %H:%M:%S"] Running CPU monitoring script..." #Getting average CPU output of tmsh show sys performance set errorcode [catch {exec tmsh show sys performance system detail | grep CPU | grep -v Average | awk {{ print $1, $(NF-4), $(NF-3), $(NF-1) }}} result] if {[lindex $result 0] == "Blade"} { set blade 1 } else { set blade 0 } set result [split $result "\n"] foreach i $result { set cpu_num "[lindex $i 1] [lindex $i 2]" if {$blade} {set cpu_num "Blade $cpu_num"} set cpu_rate [lindex $i 3] if {$DEBUG} {puts "tmsh show sys performance->${cpu_num}: ${cpu_rate}%."} if {$cpu_rate > $cpu_perf_threshold} { if {$DEBUG} {puts "tmsh show sys performance->${cpu_num}: ${cpu_rate}%. Exceeded threshold ${cpu_perf_threshold}%."} exec logger -p local0.alert "\"tmsh show sys performance\"->${cpu_num}: ${cpu_rate}%. Exceeded threshold ${cpu_perf_threshold}%." } } #Getting output of tmsh show sys proc-info foreach obj [tmsh::get_status sys proc-info $process] { if {$VERBOSE} {puts $obj} set proc_name [tmsh::get_field_value $obj proc-name] set cpu [tmsh::get_field_value $obj system-usage-5mins] set pid [tmsh::get_field_value $obj pid] set proc_threshold ${proc_name}_threshold set proc_threshold [set [set proc_threshold]] if {$DEBUG} {puts "tmsh show sys proc-info-> Average CPU Utilization of $proc_name pid $pid is ${cpu}%"} if { $cpu > ${proc_threshold} } { if {$DEBUG} {puts "$proc_name process pid $pid at $cpu% cpu. Exceeded ${proc_threshold}% threshold."} exec logger -p local0.alert "\"tmsh show sys proc-info\" $proc_name process pid $pid at $cpu% cpu. Exceeded ${proc_threshold}% threshold." } } } 4. Configure the variables in the script as needed and exit the editor by entering the following command: :wq! y 5. Run the following command to list the contents of the script: list sys icall script cpu_script 2. Creating a periodic iCall handler to run the iCall script once a minute Procedure Perform the following procedure to create the periodic handler that runs the script once a minute. To create an iCall periodic handler, perform the following procedure: Log in to tmsh Enter the following command to create a periodic handler: create sys icall handler periodic cpu_handler interval 60 script cpu_script 3. Run the following command to list the handler: list sys icall handler periodic cpu_handler 4. You can start and stop the handler by using the following command syntax: <start|stop> sys icall handler periodic cpu_handler Follow the /var/tmp/scriptd.out and /var/log/ltm file entries to verify your implementation is working correctly.
