iCall

162 Topics

What is iCall?
tl;dr - iCall is BIG-IP’s event-based granular automation system that enables comprehensive control over configuration and other system settings and objects. The main programmability points of entrance for BIG-IP are the data plane, the control plane, and the management plane. My bare bones description of the three: Data Plane - Client/server traffic on the wire and flowing through devices Control Plane - Tactical control of local system resources Management Plane - Strategic control of distributed system resources You might think iControl (our SOAP and REST API interface) fits the description of both the control and management planes, and whereas you’d be technically correct, iControl is better utilized as an external service in management or orchestration tools. The beauty of iCall is that it’s not an API at all—it’s lightweight, it’s built-in via tmsh, and it integrates seamlessly with the data plane where necessary (via iStats.) It is what we like to call control plane scripting. Do you remember relations and set theory from your early pre-algebra days? I thought so! Let me break it down in a helpful way: P = {(data plane, iRules), (control plane, iCall), (management plane, iControl)} iCall allows you to react dynamically to an event at a system level in real time. It can be as simple as generating a qkview in the event of a failover or executing a tcpdump on a server with too many failed logins. One use case I’ve considered from an operations perspective is in the event of a core dump to have iCall generate a qkview, take checksums of the qkview and the dump file, upload the qkview and generate a support case via the iHealth API, upload the core dumps to support via ftp with the case ID generated from iHealth, then notify the ops team with all the appropriate details. If I had a solution like that back in my customer days, it would have saved me 45 minutes easy each time this happened! iCall Components Three are three components to iCall: events, handlers, and scripts. Events An event is really what drives the primary reason to use iCall over iControl. A local system event (whether it’s a failover, excessive interface or application errors, too many failed logins) would ordinarily just be logged or from a system perspective, ignored altogether. But with iCall, events can be configured to force an action. At a high level, an event is "the message," some named object that has context (key value pairs), scope (pool, virtual, etc), origin (daemon, iRules), and a timestamp. Events occur when specific, configurable, pre-defined conditions are met. Example (placed in /config/user_alert.conf) alert local-http-10-2-80-1-80-DOWN "Pool /Common/my_pool member /Common/10.2.80.1:80 monitor status down" { exec command="tmsh generate sys icall event tcpdump context { { name ip value 10.2.80.1 } { name port value 80 } { name vlan value internal } { name count value 20 } }" } Handlers Within the iCall system, there are three types of handlers: triggered, periodic, and perpetual. Triggered A triggered handler is used to listen for and react to an event. Example (goes with the event example from above:) sys icall handler triggered tcpdump { script tcpdump subscriptions { tcpdump { event-name tcpdump } } } Periodic A periodic handler is used to react to an interval timer. Example: sys icall handler periodic poolcheck { first-occurrence 2017-07-14:11:00:00 interval 60 script poolcheck } Perpetual A perpetual handler is used under the control of a deamon. Example: handler perpetual core_restart_watch sys icall handler perpetual core_restart_watch { script core_restart_watch } Scripts And finally, we have the script! This is simply a tmsh script moved under the /sys icall area of the configuration that will “do stuff" in response to the handlers. Example (continuing the tcpdump event and triggered handler from above:) modify script tcpdump { app-service none definition { set date [clock format [clock seconds] -format "%Y%m%d%H%M%S"] foreach var { ip port count vlan } { set $var $EVENT::context($var) } exec tcpdump -ni $vlan -s0 -w /var/tmp/${ip}_${port}-${date}.pcap -c $count host $ip and port $port } description none events none } Resources iCall Codeshare Lightboard Lessons on iCall Threshold violation article highlighting periodic handler
JRahm
Jul 14, 2017 Place Technical Articles
8.3KViews
2likes
10Comments
iCall CRL update with Route Domains and Auto-Sync
Problem this snippet solves: iCall script to update CRL file within F5 BIG-IP when the HTTP request must run from a specific Route Domain and also uses logger to write logs to the default LTM location. The original was to also update an iFile of the CRL file for use within an iRule however I have removed that due to it being a very special case (I may add another snippet later to detail that one). Important point here is we update the CRL file located within a folder (or partition) that was linked to a Sync-Only Device Group with auto-sync enabled e.g. CRL files are created and saved to /Common/ crl / This way the iCall script does not need to trigger any sort sync and the rest of the configuration can be left as manual sync. Code : sys icall handler periodic /Common/someCrl-CrlUpdate { arguments { { name rd value 2 } { name url value https://172.31.0.1/somepath/to/crlUpdateFile.crl } { name host value somecrl.CADomein.com } { name folder value tempCrlDirectory } { name sslCrl value /Common/crl/someCrlFile.crl } } interval 600 script /Common/iCallCrlUpdate } sys icall script /Common/iCallCrlUpdate { app-service none definition { set logTag "iCallCrlUpdate" set logLevel "notice" # Getting handler provided arguments foreach arg { rd url host folder sslCrl ifileCrl } { set $arg $EVENT::context($arg) } # Create a directory to save files to disk set crlDir /var/tmp/$folder exec mkdir -p $crlDir exec /bin/logger -i -t $logTag -p local0.$logLevel "Running, CRL URL=$url, Host=$host, SSL CRL=$sslCrl, iFile CRL=$ifileCrl, Directory=$crlDir, rd=$rd" # Download CRL file from provided route domain (rd) and url arguments and save to temporary directory set status [exec /usr/bin/rdexec $rd /usr/bin/curl-apd -s -o $crlDir/LatestCRL.crl -w %{http_code} -H Host:$host $url] if {$status == 200} { # Update F5 SSL CRL file tmsh::modify sys file ssl-crl $sslCrl source-path file:$crlDir/LatestCRL.crl exec /bin/logger -t $logTag -p local0.$logLevel "F5 CRL files update complete." } else { exec /bin/logger -i -t $logTag -p local0.error "Command /usr/bin/rdexec $rd /usr/bin/curl-apd -s -o $crlDir/LatestCRL.crl -w '%{http_code}' -H 'Host: onsitecrl.trustwise.com' $url, failed with status=$status" } } description none events none } Tested this on version: 12.1
Andy_McGrath
Sep 24, 2018 Place CodeShare
810Views
2likes
0Comments
iCall for working as if LLCF (4EA Interface)
hello? My name is Muntae. Using iCall, the following LLCF operation is required, so please review whether it is possible to operate according to the work method I wrote. **When all interfaces are disabled, it is an environment where only one interface can be manually enabled. ** [Implementation Goal] - When interface 1.1 is down, 1.2, 2.1, 2.2 down at the same time - When interface 1.2 down, 1.1, 2.1, 2.2 down at the same time - When interface 2.1 down, 1.1, 1.2, 2.2 down at the same time - When interface 2.2 is down, 1.1, 1.2, 2.1 down at the same time - When interface 1.1 up, 1.2, 2.1, 2.2 up at the same time - When interface 1.2 up, 1.1, 2.1, 2.2 up at the same time - When interface 2.1 up, 1.1, 1.2, 2.2 up at the same time - When interface 2.2 up, 1.1, 1.2, 2.1 up at the same time [Working procedure] 1. Create /config/llcf.conf file [Attach the file for details] sys icall script /Common/interface_1_1_down { app-service none definition { exec tmsh modify net interface 1.2 disabled exec tmsh modify sys icall handler triggered interface_1_2_down script null exec tmsh modify net interface 2.1 disabled exec tmsh modify sys icall handler triggered interface_2_1_down script null exec tmsh modify net interface 2.2 disabled exec tmsh modify sys icall handler triggered interface_2_2_down script null } description none events none } sys icall script /Common/interface_1_1_up { app-service none definition { exec tmsh modify net interface 1.2 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_1_2_down script interface_1_2_down exec tmsh modify net interface 2.1 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_2_1_down script interface_2_1_down exec tmsh modify net interface 2.2 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_2_2_down script interface_2_2_down } description none events none } sys icall script /Common/interface_1_2_down { app-service none definition { exec tmsh modify net interface 1.1 disabled exec tmsh modify sys icall handler triggered interface_1_1_down script null exec tmsh modify net interface 2.1 disabled exec tmsh modify sys icall handler triggered interface_2_1_down script null exec tmsh modify net interface 2.2 disabled exec tmsh modify sys icall handler triggered interface_2_2_down script null } description none events none } sys icall script /Common/interface_1_2_up { app-service none definition { exec tmsh modify net interface 1.1 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_1_1_down script interface_1_1_down exec tmsh modify net interface 2.1 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_2_1_down script interface_2_1_down exec tmsh modify net interface 2.2 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_2_2_down script interface_2_2_down } description none events none } sys icall script /Common/interface_2_1_down { app-service none definition { exec tmsh modify net interface 2.2 disabled exec tmsh modify sys icall handler triggered interface_2_2_down script null exec tmsh modify net interface 1.1 disabled exec tmsh modify sys icall handler triggered interface_1_1_down script null exec tmsh modify net interface 1.2 disabled exec tmsh modify sys icall handler triggered interface_1_2_down script null } description none events none } sys icall script /Common/interface_2_1_up { app-service none definition { exec tmsh modify net interface 2.2 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_2_2_down script interface_2_2_down exec tmsh modify net interface 1.1 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_1_1_down script interface_1_1_down exec tmsh modify net interface 1.2 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_1_2_down script interface_1_2_down } description none events none } sys icall script /Common/interface_2_2_down { app-service none definition { exec tmsh modify net interface 2.1 disabled exec tmsh modify sys icall handler triggered interface_2_2_down script null exec tmsh modify net interface 1.1 disabled exec tmsh modify sys icall handler triggered interface_1_1_down script null exec tmsh modify net interface 1.2 disabled exec tmsh modify sys icall handler triggered interface_1_2_down script null } description none events none } sys icall script /Common/interface_2_2_up { app-service none definition { exec tmsh modify net interface 2.1 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_2_1_down script interface_2_1_down exec tmsh modify net interface 1.1 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_1_1_down script interface_1_1_down exec tmsh modify net interface 1.2 enabled exec sleep 5 exec tmsh modify sys icall handler triggered interface_1_2_down script interface_1_2_down } description none events none } sys icall script /common/null { app-service none definition { } description none events none } 2. Create the /config/user_alert.conf file [Attach the file for details] alert interface_1_1_down "Link: 1.1 is DOWN" { exec command="tmsh generate sys icall event interface_1_2_down" exec command="tmsh generate sys icall event interface_2_1_down" exec command="tmsh generate sys icall event interface_2_2_down" } alert interface_1_2_down "Link: 1.2 is DOWN" { exec command="tmsh generate sys icall event interface_1_1_down" exec command="tmsh generate sys icall event interface_2_1_down" exec command="tmsh generate sys icall event interface_2_2_down" } alert interface_2_1_down "Link: 2.1 is DOWN" { exec command="tmsh generate sys icall event interface_1_1_down" exec command="tmsh generate sys icall event interface_1_2_down" exec command="tmsh generate sys icall event interface_2_2_down" } alert interface_2_2_down "Link: 2.2 is DOWN" { exec command="tmsh generate sys icall event interface_1_1_down" exec command="tmsh generate sys icall event interface_1_2_down" exec command="tmsh generate sys icall event interface_2_1_down" } alert interface_1_1_up "Link: 1.1 is UP" { exec command="tmsh generate sys icall event interface_1_2_up" exec command="tmsh generate sys icall event interface_2_1_up" exec command="tmsh generate sys icall event interface_2_2_up" } alert interface_1_2_up "Link: 1.2 is UP" { exec command="tmsh generate sys icall event interface_1_1_up" exec command="tmsh generate sys icall event interface_2_1_up" exec command="tmsh generate sys icall event interface_2_2_up" } alert interface_2_1_up "Link: 2.1 is UP" { exec command="tmsh generate sys icall event interface_1_1_up" exec command="tmsh generate sys icall event interface_1_2_up" exec command="tmsh generate sys icall event interface_2_2_up" } alert interface_2_2_up "Link: 2.2 is UP" { exec command="tmsh generate sys icall event interface_1_1_up" exec command="tmsh generate sys icall event interface_1_2_up" exec command="tmsh generate sys icall event interface_2_1_up" } 3. Merge script configuration: #tmsh load sys config merge file /config/llcf.conf 4. Apply settings: - #tmsh save sys config 5. Check to apply script settings: #list sys icall handler triggered interface 1.1 down #list sys icall handler triggered interface 1.2 down #list sys icall handler triggered interface 2.1 down #list sys icall handler triggered interface 2.2 down #list sys icall handler triggered interface 1.1 up #list sys icall handler triggered interface 1.2 up #list sys icall handler triggered interface 2.1 up #list sys icall handler triggered interface 2.2 up
Solved
muntae_kim
Sep 19, 2022 Place Technical Forum
1.5KViews
1like
5Comments
Part 3: Monitoring the health of BIG-IP APM network access PPP connections with a periodic iCall handler
In this part, you monitor the health of PPP connections on the BIG-IP APM system by monitoring the frequency of a particular log message in the/var/log/apmfile. In this log file, when the system is under high CPU load, you may observe different messages indicating users' VPN connections are disconnecting. For example: You observe multiple of the following messages where the PPP tunnel is started and closed immediately: Mar 13 13:57:57 hostname.example notice tmm3[16095]: 01490505:5: /Common/accessPolicy_policy:Common:ee6e0ce7: PPP tunnel 0x5604775f7000 (ID: 2c4507ea) started. Mar 13 13:57:57 hostname.example notice tmm3[16095]: 01490505:5: /Common/accessPolicy_policy:Common:ee6e0ce7: PPP tunnel 0x560478482000 (ID: acf63e47) closed. You observe an unusually high rate of log messages indicating users' APM sessions terminated due to various reasons in a short span of time. Mar 13 14:01:10 hostname.com notice tmm3[16095]: 01490567:5: /Common/AccessPolicy_policy:Common:a7cf25d9: Session deleted due to user inactivity. You observe an unusually high rate of log messages indicating different user apm session IDs are attempting to reconnect to network access, VPN. Mar 13 12:18:13 hostname notice tmm1[16095]: 01490549:5: /Common/AP_policy:Common:2c321803: Assigned PPP Dynamic IPv4: 10.10.1.20 ID: 033d0635 Tunnel Type: VPN_TUNNELTYPE_TLS NA Resource: /Common/AP_policy_na_res Client IP: 172.2.2.2 – Reconnect In this article, you monitor theReconnectmessage in the last example because it indicates that the VPN connection terminated and the client system still wants to reconnect. When this happens for several users and repeatedly, it usually indicates that the issue is not related to the client side. The next question is, what is an unusually high rate of reconnect messages, taking into account the fact that some reconnects may be due to reasons on the client side and not due to high load on the system. You can create a periodic iCall handler to run a script once every minute. Each time the script runs, it uses grep to find the total number of Reconnect log messages that happened over the last 3 minutes for example. When the average number of entries per minute exceeds a configured threshold, the system can take appropriate action. The following describes the procedures: Creating an iCall script to monitor the rate of reconnect messages Creating a periodic iCall handler to run the iCall script once a minute Testing the implementation using logger 1. Creating an iCall script to monitor the rate of reconnect messages The example script uses the following values: Periodic handler interval 1 minute: This runs the script once a minute to grep the number of reconnect messages logged in the last 3.0 minutes. Period 3.0 minutes: The script greps for the Reconnect messages logged in the last three minutes. Critical threshold value 4: When the average number of Reconnect messages per minute exceeds four or a total of 12 in the last three minutes, the system logs a critical message in the/var/log/apmfile. Alert threshold value 7: When the average number of Reconnect messages per minute exceeds seven or a total of 21 in the last three minutes, the system logs an alert message and requires immediate action. Emergency threshold 10: When the BIG-IP system is logging 10 Reconnect messages or more every minute for the last three minutes or a total of 30 messages, the system logs an emergency message, indicating the system is unusable. You should reconfigure these values at the top of the script according to the behavior and set up of your environment. By performing the count over an interval of three minutes or longer, it reduces the possibility of high number of reconnects due to one-off spikes. Procedure Perform the following procedure to create the script to monitor CPU statistics and log an alert message in the/var/log/ltmfile when traffic exceeds a CPU threshold value. To create an iCall script, perform the following procedure: Log in to tmsh. Enter the following command to create the script in the vi editor: create sys icall script vpn_reconnect_script 3. Enter the following script into the definition stanza in the editor: This script counts the average number of entries/min the Reconnect message is logged in /var/log/apm over a period of 3 minutes. definition { #log file to grep errormsg set apm_log "/var/log/apm" #This is the string to grep for. set errormsg "Reconnect" #num of entries = grep $errormsg $apm_log | grep $hourmin | wc -l #When (num of entries < crit_threshold), no action #When (crit_threshold <= num of entries < alert_threshold), log crtitical message. #When (alert_threshold <= num of entries < emerg_threshold), log alert message. #When (emerg_threhold <= num of entries log emerg message set crit_threshold 4 set alert_threshold 7 set emerg_threshold 10 #Number of minutes to take average of. E.g. Every 1.0, 2.0, 3.0, 4.0... minutes set period 3.0 #Set this to 1 to log to /var/tmp/scriptd.out. Set to 0 to disable. set DEBUG 0 set total 0 puts "\n[clock format [clock seconds] -format "%b %e %H:%M:%S"] Running script..." for {set i 1} {$i <= $period} {incr i} { set hourmin [clock format [clock scan "-$i minute"] -format "%b %e %H:%M:"] set errorcode [catch {exec grep $errormsg $apm_log | grep $hourmin | wc -l} num_entries] if{$errorcode} { set num_entries 0 } if {$DEBUG} {puts "DEBUG: $hourmin \"$errormsg\" logged $num_entries times."} set total [expr {$total + $num_entries}] } set average [expr $total / $period] set average [format "%.1f" $average] if{$average < $crit_threshold} { if {$DEBUG} {puts "DEBUG: $hourmin \"$errormsg\" logged $average times on average. Below all threshold. No action."} exit } if{$average < $alert_threshold} { if {$DEBUG} {puts "DEBUG: $hourmin \"$errormsg\" logged $average times on average. Reached critical threshold $crit_threshold. Log Critical msg."} exec logger -p local1.crit "01490266: \"$errormsg\" logged $average times on average in last $period mins. >= critical threshold $crit_threshold." exit } if{$average < $emerg_threshold} { if {$DEBUG} {puts "DEBUG: $hourmin \"$errormsg\" logged $average times on average. Reached alert threshold $alert_threshold. Log Alert msg."} exec logger -p local1.alert "01490266: \"$errormsg\" logged $average times on average in last $period mins. >= alert threshold $alert_threshold." exit } if {$DEBUG} {puts "DEBUG: $hourmin \"$errormsg\" logged $average times on average in last $period mins. Log Emerg msg"} exec logger -p local1.emerg "01490266: \"$errormsg\" logged $average times on average in last $period mins. >= emerg threshold $emerg_threshold." exit } 4. Configure the variables in the script as needed and exit the editor by entering the following command: :wq! y 5. Run the following command to list the contents of the script: list sys icall script vpn_reconnect_script 2. Creating a periodic iCall handler to run the script In this example, you create the iCall handler to run the script once a minute. You can increase this interval to once every two minutes or longer. However, you should consider this value together with theperiodvalue of the script in the previous procedure to ensure that you're notified on any potential issues early. Procedure Perform the following procedure to create the periodic handler that runs the script once a minute. To create an iCall periodic handler, perform the following procedure: Log in to tmsh. Enter the following command to create a periodic handler: create sys icall handler periodic vpn_reconnect_handler interval 60 script vpn_reconnect_script 3. Run the following command to list the handler: list sys icall handler periodic vpn_reconnect_handler 4. You can start and stop the handler by using the following command syntax: <start|stop> sys icall handler periodic vpn_reconnect_handler 3. Testing the implementation using logger You can use theloggercommand to log test messages to the/var/log/apmfile to test your implementation. To do so, run the following command the required number of times to exceed the threshold you set: Note: The following message must contain the keyword that you are searching for in the script. In this case, the keyword is Reconnect. logger -p local1.notice "01490549:5 Assigned PPP Dynamic IPv4: 10.10.1.20 ID: 033d0635 Tunnel Type: VPN_TUNNELTYPE_TLS NA Resource: /Common/AP_policy_na_res Client IP: 172.2.2.2 - Reconnect" Follow the/var/tmp/scriptd.outand/var/log/apm file entries to verify your implementation is working correctly. Conclusion This article lists three use cases for using iCall to monitor the health of your BIG-IP APM system. The examples mainly log the appropriate messages in the log files. You can extend the examples to monitor more parameters and also perform different kinds of actions such as calling another script (Bash, Perl, and so on) to send an email notification or perform remedial action.
Kin
May 12, 2020 Place Technical Articles
1.3KViews
1like
5Comments
Using iCall to monitor BIG-IP APM network access VPN
Introduction During peak periods, when a large number of users are connected to network access VPN, it is important to monitor your BIG-IP APM system's resource (CPU, memory, and license) usage and performance to ensure that the system is not overloaded and there is no impact on user experience. If you are a BIG-IP administrator, iCall is a tool perfectly suited to do this for you. iCall is a Tcl-based scripting framework that gives you programmability in the control plane, allowing you to script and run Tcl and TMOS Shell (tmsh) commands on your BIG-IP system based on events. For a quick introduction to iCall, refer to iCall - All New Event-Based Automation System. Overview This article is made up of three parts that describe how to use and configure iCall in the following use cases to monitor some important BIG-IP APM system statistics: Part 1: Monitoring access sessions and CCU license usage of the system using a triggered iCall handler Part 2: Monitoring the CPU usage of the system using a periodic iCall handler. Part 3: Monitoring the health of BIG-IP APM network access VPN PPP connections with a periodic iCall handler. In all three cases, the design consists of identifying a specific parameter to monitor. When the value of the parameter exceeds a configured threshold, an iCall script can perform a set of actions such as the following: Log a message to the /var/log/apm file at the appropriate severity: emerg: System is unusable alert: Action must be taken immediately crit: Critical conditions You may then have another monitoring system to pick up these messages and respond to them. Perform a remedial action to ease the load on the BIG-IP system. Run a script (Bash, Perl, Python, or Tcl) to send an email notification to the BIG-IP administrator. Run the tcpdump or qkview commands when you are troubleshooting an issue. When managing or troubleshooting iCall scripts and handler, you should take into consideration the following: You use the Tcl language in the editor in tmsh to edit the contents of scripts and handlers. For example: create sys icall script <name of script> edit sys icall script <name of script> The puts command outputs entries to the /var/tmp/scriptd.out file. For example: puts "\n[clock format [clock seconds] -format "%b %d %H:%M:%S"] Running script..." You can view the statistics for a particular handler using the following command syntax: show sys icall handler <periodic | perpetual | triggered> <name of handler> Series 1: Monitoring access sessions and CCU license usage with a triggered iCall handler You can view the number of currently active sessions and current connectivity sessions usage on your BIG-IP APM system by entering the tmsh show apm license command. You may observe an output similar to the following: -------------------------------------------- Global Access License Details: -------------------------------------------- total access sessions: 10.0M current active sessions: 0 current established sessions: 0 access sessions threshold percent: 75 total connectivity sessions: 2.5K current connectivity sessions: 0 connectivity sessions threshold percent: 75 In the first part of the series, you use iCall to monitor the number of current access sessions and CCU license usage by performing the following procedures: Modifying database DB variables to log a notification when thresholds are exceed. Configuring user_alert.conf to generate an iCall event when the system logs the notification. Creating a script to respond when the license usage reaches its threshold. Creating an iCall triggered handler to handle the event and run an iCall script Testing the implementation using logger 1. Modifying database variables to log a notification when thresholds are exceeded. The tmsh show apm license command displays the access sessions threshold percent and access sessions threshold percent values that you can configure with database variables. The default values are 75. For more information, refer to K62345825: Configuring the BIG-IP APM system to log a notification when APM sessions exceed a configured threshold. When the threshold values are exceeded, you will observe logs similar to the following in /var/log/apm: notice tmm1[<pid>]: 01490564:5: (null):Common:00000000: Global access license usage is 1900 (76%) of 2500 total. Exceeded 75% threshold of total license. notice tmm2[<pid>]: 01490565:5: 00000000: Global concurrent connectivity license usage is 393 (78%) of 500 total. Exceeded 75% threshold of total license. Procedure: Run the following commands to set the threshold to 95% for example: tmsh modify /sys db log.alertapmaccessthreshold value 95 tmsh modify /sys db log.alertapmconnectivitythreshold value 95 Whether to set the alert threshold at 90% or 95%, depends on your specific environment, specifically how fast the usage increases over a period of time. 2. Configuring user_alert.conf to generate an iCall event when the system logs the notification You can configure the /config/user_alert.conf file to run a command or script based on a syslog message. In this step, edit the user_alert.conf file with your favorite editor, so that the file contains the following stanza. alert <name> "<string in syslog to match to trigger event>" { <command to run> } For more information on configuring the /config/user_alert.conf file, refer to K14397: Running a command or custom script based on a syslog message. In particular, it is important to read the bullet points in the Description section of the article first; for example, the system may not process the user_alert.conf file after system upgrades. In addition, BIG-IP APM messages are not processed by the alertd SNMP process by default. So you will also have to perform the steps described in K51341580: Configuring the BIG-IP system to send BIG-IP APM syslog messages to the alertd process as well. Procedure: Perform the following procedure: Edit the /config/user_alert.conf file to match each error code and generate an iCall event named apm_threshold_event. Per K14397 Note: You can create two separate alerts based on both error codes or alternatively use the text description part of the log message common to both log entries to capture both in a single alert. For example "Exceeded 75% threshold of total license" # cat /config/user_alert.conf alert apm_session_threshold "01490564:" { exec command="tmsh generate sys icall event apm_threshold_event" } alert apm_ccu_threshold "01490565:" { exec command="tmsh generate sys icall event apm_threshold_event" } 2. Run the following tmsh command: edit sys syslog all-properties 3. Replace the include none line with the following: Per K51341580 include " filter f_alertd_apm { match (\": 0149[0-9a-fA-F]{4}:\"); }; log { source(s_syslog_pipe); filter(f_alertd_apm); destination(d_alertd); }; " 3. Creating a script to respond when the license usage reaches its threshold. When the apm session or CCU license usage exceeds your configured threshold, you can use a script to perform a list of tasks. For example, if you had followed the earlier steps to configure the threshold values to be 95%, you can write a script to perform the following actions: Log a syslog alert message to the /var/log/apm file. If you have another monitoring system, it can pick this up and respond as well. Optional: Run a tmsh command to modify the Access profile settings. For example, when the threshold exceeds 95%, you may want to limit users to one apm session each, decrease the apm access profile timeout or both. Changes made only affect new users. Users with existing apm sessions are not impacted. If you are making changes to the system in the script, it is advisable to run an additional tmsh command to stop the handler. When you have responded to the alert, you can manually start the handler again. Note: When automating changes to the system, it is advisable to err on the side of safety by making minimal changes each time and only when required. In this case, after the system reaches the license limit, users cannot login and you may need to take immediate action. Procedure: Perform the following procedure to create the iCall script: 1. Log in to tmsh. 2. Run the following command: create sys icall script threshold_alert_script 3. Enter the following in the editor: Note: The tmsh commands to modify the access policy settings have been deliberately commented out. Uncomment them when required. sys icall script threshold_alert_script { app-service none definition { exec logger -p local1.alert "01490266: apm license usage exceeded 95% of threshold set." #tmsh::modify apm profile access exampleNA max-concurrent-sessions 1 #tmsh::modify apm profile access exampleNA generation-action increment #tmsh::stop sys icall handler triggered threshold_alert_handler } description none events none } 4. Creating an iCall triggered handler to handle the event and run an iCall script In this step, you create a triggered iCall handler to handle the event triggered by the tmsh generate sys icall event command from the earlier step to run the script. Procedure: Perform the following: 1. Log in to tmsh. 2. Enter the following command to create the triggered handler. create sys icall handler triggered threshold_alert_handler script threshold_alert_script subscriptions add { apm_threshold_event { event-name apm_threshold_event } } Note: The event-name field must match the name of the event in the generate sys icall command in /config/user_alert.conf you configured in step 2. 3. Enter the following command to verify the configuration of the handler you created. (tmos)# list sys icall handler triggered threshold_alert_handler sys icall handler triggered threshold_alert_handler { script threshold_alert_script subscriptions { apm_threshold_event { event-name apm_threshold_event } } } 5. Testing the implementation using logger You can use theloggercommand to log test messages to the/var/log/apmfile to test your implementation. To do so, run the following command: Note: The message below must contain the keyword that you are searching for in the script. In this example, the keyword is01490564or01490565. logger -p local1.notice "01490564:5: (null):Common:00000000: Global access license usage is 1900 (76%) of 2500 total. Exceeded 75% threshold of total license." logger -p local1.notice "01490565:5: 00000000: Global concurrent connectivity license usage is 393 (78%) of 500 total. Exceeded 75% threshold of total license." Follow the /var/log/apm file to verify your implementation is working correctly.
Kin
May 12, 2020 Place Technical Articles
1.8KViews
1like
0Comments
Part 2: Monitoring the CPU usage of the BIG-IP system using a periodic iCall handler
In this part series, you monitor the CPU usage of the BIG-IP system with a periodic iCall handler. The specific CPU statistics you want to monitor can be retrieved from either Unix or tmsh commands. For example, if you want to monitor the CPU usage of the tmm process, you can monitor the values from the output of the tmsh show sys proc-info tmm.0 command. An iCall script can iterate and retrieve a list of values from the output of a tmsh command. To display the fields available from a tmsh command that you can iterate from an iCall Tcl script, run the tmsh command with thefield-fmtoption. For example: tmsh show sys proc-info tmm.0 field-fmt You can then use a periodic iCall handler which runs an iCall script periodically every interval to check the value of the output of the tmsh command. When the value exceeds a configured threshold, you can have the script perform an action; for example, an alert message can be logged to the/var/log/ltmfile. The following describes the procedures: Creating an iCall script to monitor the required CPU usage values Creating a periodic iCall handler to run the iCall script once a minute 1. Creating an iCall script to monitor the required CPU usage values There are different Unix and tmsh commands available to display CPU usage. To monitor CPU usage, this example uses the following: tmsh show sys performance system detail | grep CPU: This displays the systemCPU Utilization (%). The script monitors CPU usage from theAveragecolumn for each CPU. tmsh show sys proc-info apmd: Monitors the CPU usage System Utilization (%) Last5-minsvalue of the apmd process. tmsh show sys proc-info tmm.0: Monitors the CPU usage System Utilization (%) Last5-minsvalue of the tmm process. This is the sum of the CPU usage of all threads of thetmm.0process divided by the number of CPUs over five minutes. You can display the number of TMM processes and threads started, by running different commands. For example: pstree -a -A -l -p | grep tmm | grep -v grep grep Start /var/log/tmm.start You can also create your own script to monitor the CPU output from other commands, such astmsh show sys cpuortmsh show sys tmm-info. However, a discussion on CPU usage on the BIG-IP system is beyond the scope of this article. For more information, refer toK14358: Overview of Clustered Multiprocessing (11.3.0 and later)andK16739: Understanding 'top' output on the BIG-IP system. You need to set some of the variables in the script, specifically the threshold values:cpu_perf_threshold, tmm.0_threshold, apmd_thresholdrespectively. In this example, all the CPU threshold values are set at 80%. Note that depending on the set up in your specific environment, you have to adjust the threshold accordingly. The threshold values also depend on the action you plan to run in the script. For example, in this case, the script logs an alert message in the/var/log/ltmfile. If you plan to log an emerg message, the threshold values should be higher, for example, 95%. Procedure Perform the following procedure to create the script to monitor CPU statistics and log an alert message in the/var/log/ltmfile when traffic exceeds a CPU threshold value. To create an iCall script, perform the following procedure: Log in to tmsh. Enter the following command to create the script in the vi editor: create sys icall script cpu_script 3. Enter the following script into the definition stanza of the editor. The 3 threshold values are currently set at 80%. You can change it according to the requirements in your environment. definition { set DEBUG 0 set VERBOSE 0 #CPU threshold in % from output of tmsh show sys performance system detail set cpu_perf_threshold 80 #The name of the process from output of tmsh show sys proc-info to check. The name must match exactly. #If you would like to add another process, append the process name to the 'process' variable and add another line for threshold. #E.g. To add tmm.4, "set process apmd tmm.0 tmm.4" and add another line "set tmm.4_threshold 75" set process "apmd tmm.0" #CPU threshold in % for output of tmsh show sys proc-info set tmm.0_threshold 80 set apmd_threshold 80 puts "\n[clock format [clock seconds] -format "%b %d %H:%M:%S"] Running CPU monitoring script..." #Getting average CPU output of tmsh show sys performance set errorcode [catch {exec tmsh show sys performance system detail | grep CPU | grep -v Average | awk {{ print $1, $(NF-4), $(NF-3), $(NF-1) }}} result] if {[lindex $result 0] == "Blade"} { set blade 1 } else { set blade 0 } set result [split $result "\n"] foreach i $result { set cpu_num "[lindex $i 1] [lindex $i 2]" if {$blade} {set cpu_num "Blade $cpu_num"} set cpu_rate [lindex $i 3] if {$DEBUG} {puts "tmsh show sys performance->${cpu_num}: ${cpu_rate}%."} if {$cpu_rate > $cpu_perf_threshold} { if {$DEBUG} {puts "tmsh show sys performance->${cpu_num}: ${cpu_rate}%. Exceeded threshold ${cpu_perf_threshold}%."} exec logger -p local0.alert "\"tmsh show sys performance\"->${cpu_num}: ${cpu_rate}%. Exceeded threshold ${cpu_perf_threshold}%." } } #Getting output of tmsh show sys proc-info foreach obj [tmsh::get_status sys proc-info $process] { if {$VERBOSE} {puts $obj} set proc_name [tmsh::get_field_value $obj proc-name] set cpu [tmsh::get_field_value $obj system-usage-5mins] set pid [tmsh::get_field_value $obj pid] set proc_threshold ${proc_name}_threshold set proc_threshold [set [set proc_threshold]] if {$DEBUG} {puts "tmsh show sys proc-info-> Average CPU Utilization of $proc_name pid $pid is ${cpu}%"} if { $cpu > ${proc_threshold} } { if {$DEBUG} {puts "$proc_name process pid $pid at $cpu% cpu. Exceeded ${proc_threshold}% threshold."} exec logger -p local0.alert "\"tmsh show sys proc-info\" $proc_name process pid $pid at $cpu% cpu. Exceeded ${proc_threshold}% threshold." } } } 4. Configure the variables in the script as needed and exit the editor by entering the following command: :wq! y 5. Run the following command to list the contents of the script: list sys icall script cpu_script 2. Creating a periodic iCall handler to run the iCall script once a minute Procedure Perform the following procedure to create the periodic handler that runs the script once a minute. To create an iCall periodic handler, perform the following procedure: Log in to tmsh Enter the following command to create a periodic handler: create sys icall handler periodic cpu_handler interval 60 script cpu_script 3. Run the following command to list the handler: list sys icall handler periodic cpu_handler 4. You can start and stop the handler by using the following command syntax: <start|stop> sys icall handler periodic cpu_handler Follow the /var/tmp/scriptd.out and /var/log/ltm file entries to verify your implementation is working correctly.
Kin
May 12, 2020 Place Technical Articles
2.7KViews
1like
0Comments
Sample Linux script to update CRL file from Certificate Authority
Problem this snippet solves: CRL files are signed lists of revoked serial numbers issued by a specific Certificate Authority (Verisign, Godaddy, GlobalSign, etc). There are several advanced methods of dealing with revoked certificates, the best of which is OCSP stapling. Other methods are OCSP responders or CRL Distribution Points. However, for small or internal projects, some administrators rely on simply using straight-up CRL files. After a time, the administrator will realize he or she needs to automate this process with a script. The administrator can automate this process on the BIG-IP itself with an iCall script. Here's a link to a great example of the iCall solution. However, some administrators many need to use a straight-up Linux device to pull and copy the CRL files around to many different devices, only one of which is the BIG-IP. How to use this snippet: This is a sample of a Linux script that pulls down a CRL file from GoDaddy, verifies it and then copies it to BIG-IP. Ensure that the Linux device can copy files directly to the BIG-IP via ssh-key authentication. Modify the 'f5' variable of the script to point to the BIG-IP. If not using GoDaddy, find the URL of the CRL file for the appropriate CA. If the 'ssl-crl' object hasn't been created on the BIG-IP yet, then it must be done manually the first time. Download the CRL, copy it to the BIG-IP's /var/tmp area. Then login to the BIG-IP and issue the following command: tmsh modify sys file ssl-crl gdcrl source-path file:/var/tmp/CRL After that, the script should work. Code : #!/bin/bash # # script to download a CRL from GoDaddy CA. # See this page for GoDaddy CRL information: # https://certs.godaddy.com/repository # Verify CRL # Convert CRL # Copy to BIG-IP # exit on error to prevent copying corrupt CRL to BIG-IP set -e f5=yourbigip.com f5port=22 crlurl=https://certs.godaddy.com/repository/mastergodaddy2issuing.crl gdcrturl=https://certs.godaddy.com/repository/gdig2.crt gdcrt=gdig2.crt echo "Automated CRL update Script" echo "Downloading from $crlurl" echo "Copying to ${f5}:${f5port}" echo "Last line should be SUCCESS" echo "---- GO ----" if [ ! -f $gdcrt ]; then echo "Fetching GoDaddy Certificate" curl $gdcrturl > $gdcrt fi cf=gdroot.crl last=last.crl if [ -f $cf ]; then if [ ! -s $cf ]; then echo "Found existing zero-length CRL file, deleting" rm -f $cf else echo "Found existing $cf - moving to backup" mv $cf $last fi fi echo "Downloading CRL file $cf" curl $crlurl > $cf echo "Testing $cf for readability" test -f $cf test -r $cf echo "Testing to see if $cf is zero length" test -s $cf if [ -f $last ]; then echo "Testing if $cf is newer than backup file $last" if [ ! $cf -nt $last ]; then echo "File not changed. SUCCESS" fi fi echo "Verifying CRL against certificate, converting to PEM format" openssl crl -inform DER -CAfile $gdcrt -in $cf -outform PEM -out ${cf}.pem echo "Testing PEM for zero length" test -s ${cf}.pem echo "Copying CRL file to bigip" scp -P ${f5port} ${cf}.pem root@${f5}:/var/tmp echo "Importing CRL into system" echo " this may fail if object was never created - replace 'modify' with 'create'" ssh root@${f5} -p ${f5port} "tmsh modify sys file ssl-crl gdcrl source-path file:/var/tmp/${cf}.pem" echo "SUCCESS"
David_Holmes_12
Oct 12, 2015 Place CodeShare
1.2KViews
1like
0Comments
Run tcpdump on event
Problem this snippet solves: This example shows how to trigger an iCall script using a log message as the trigger. In particular, it shows how to watch for messages about pool members failing their monitors, and then running tcpdump to capture some packets sent to that pool member in order to help troubleshoot later. How to use this snippet: Implementation Details This iCall script requires v11.4 or higher. Code : 1. create the following iCall script: modify script tcpdump { app-service none definition { set date [clock format [clock seconds] -format "%Y%m%d%H%M%S"] foreach var { ip port count vlan } { set $var $EVENT::context($var) } exec tcpdump -ni $vlan -s0 -w /var/tmp/${ip}_${port}-${date}.pcap -c $count host $ip and port $port } description none events none } 2. create an event handler for it that looks like the following: sys icall handler triggered tcpdump { script tcpdump subscriptions { tcpdump { event-name tcpdump } } } 3. modify /config/user_alert.conf to contain entries that looks like the following: alert local-http-10-2-80-1-80-DOWN "Pool /Common/my_pool member /Common/10.2.80.1:80 monitor status down" { exec command="tmsh generate sys icall event tcpdump context { { name ip value 10.2.80.1 } { name port value 80 } { name vlan value internal } { name count value 20 } }" } You will need one of these stanzas for each pool member that you want to monitor in this method - or to modify the example to act on different events and different objects.
Brent_Blood_102
Mar 09, 2015 Place CodeShare
997Views
1like
0Comments
Big-IQ snapshot management
Hello community, I have an interesting situation that I want to run by you guys and hopefully somebody/someone can give me a hand. We have in our environment 20 Big-IQ devices, handling nearly 500 devices (between LTMs, ASMs, DNSs and combination of them). Internally we have configured Big-IQs to trigger snapshots for all of them, the only difference we have is that we only keep snapshot files storaged in the Big-IQ is for ASMs. The rest of the modules keep their snapshots locally. Now, we want (need) to create a python script (basically because we make it run from a 3rd party solution that ssh into the Big-IQs and run that python script) to delete snapshots older than 30 days, for example. For that purpose, I have searched for several documentation regarding API for snapshot management on Big IQ , basically there are 2 API calls relevant for me: 1.- Snapshot checkup: GET /cm/<module>/tasks/snapshot-config/<id> 2.- Snapshot deletion: DELETE mgmt/cm/<module>/tasks/snapshot-config/<id> Now the values between <>, which are: 'module' and 'id', in our case module = asm and 'id' is for the snapshot we want to checkup and delete if meets the criteria. The thing here is that, we do not have any 'id' number, because the snapshots are created by several business units outside our team, where they log into the Big-IQ's GUI and deploy the snapshot manually. I have tried to send a GET request (via POSTMAN) using these variations (hoping some sort of response): GET /cm/<module>/tasks/snapshot-config/ GET /cm/<module>/tasks/snapshot-config/* But logically is expecting the 'id' value so it shoots back an error. NOTE: the 'id' is provided when you try the API call for snapshot creation: POST mgmt/cm/<module>/tasks/snapshot-config. That particular POST, only request the name of the snapshot as parameter and then provides the precious 'id' in return. So you can see my dilemma and where I am missing data. QUESTIONS: 1.- Is there any way to fetch the 'id' using any resource? 2.- where exactly is located the repository for these snapshots in the Big-IQ? I mean the Linux's bash prompt for each Big-IQ, I have searched and searched but I haven't found them... This would be a great peace of information. 3.- Is this the right approach? I found some information about 'restcurl' but I am not completely sure about this resource. Thanks.-
Leonardo_Silva1
Jun 22, 2021 Place Technical Forum
783Views
1like
3Comments
Knowledge sharing: Ways to trigger and schedule scripts on the F5 BIG-IP devices.
I think that it is interesting to share how on F5 different scripts can be run at different times and states. 1. You can use the cron job like on any linux device to run a script. As I have used this to restart the tomcat and httpd each night with "bigstart restart <name>" or "tmsh restart /sys service <name>" (https://support.f5.com/csp/article/K89999342), because of a bug till I upgade the devices (https://support.f5.com/csp/article/K25554628 ). https://support.f5.com/csp/article/K03108954 2.Newer versions of F5 also have anacron tool that can add some randomness to the timframe when a script is run and many F5 default scripts use this and not the crontab: https://support.f5.com/csp/article/K33730915 3.You can even trigger scripts on the F5 device if the state changes from active to standby or from standby to active by adding the scripts under /config/failover/<state>. For example if you have a bug for a critical process that causes a failover ( you can use the command show /sys ha-status all-properties to check for this https://support.f5.com/csp/article/K20060182 ) but the device does not reboot or fix the process you can run a script to when the device becomes standby to restart the process. https://support.f5.com/csp/article/K6008 4. You afcource can run scripts at the F5 start time (startup/bootup): https://support.f5.com/csp/article/K11948 5.The final thing thing I can think of is to run a script at the backround that monitors the log and for example when there is a specific message in /var/log/ltm to trigger a tcpdump (in some cases better than creating a rotating tcpdum to catch an issue as per https://support.f5.com/csp/article/K65251607 ). The script can be a bash script with "tail -f" command that is run on the backround or better use the F5 intergrated "icall" feature. Bash: https://www.thegeekstuff.com/2010/12/5-ways-to-execute-linux-command/ Icall: https://devcentral.f5.com/s/articles/what-is-icall-27404 https://devcentral.f5.com/s/articles/run-tcpdump-on-event 5. You can use utility "logger -p" to generate manually log messages in the F5 device's log for testing of your scripts as this is used also for SNMP custom alarm traps tests (for more about SNMP https://support.f5.com/csp/article/K3727 ) https://support.f5.com/csp/article/K86480148 6. You can also trigger scripts from an BIG-IQ device bt you still can't schedule them when to run: https://clouddocs.f5.com/training/community/big-iq-cloud-edition/html/class5/module1/lab6.html 7.Of course the final option is to use ansible or python SDK that uses the F5 rest-api to execute commands on the F5 devices. https://f5-sdk.readthedocs.io/en/latest/ 8. You can even use TCP expect and bash for automations using SSH connection but this is really old way to do things: https://devcentral.f5.com/s/articles/f5-automation-tcl-amp-bash-921 https://f5-sdk.readthedocs.io/en/latest/userguide/ltm_pools_members_code_example.html 9.F5 is well integrated with Ansible and it is better than REST-API Python SDK or TCL for me as even the declarative AS3 interface is supported: https://clouddocs.f5.com/products/orchestration/ansible/devel/ https://clouddocs.f5.com/products/orchestration/ansible/devel/ https://www.f5.com/partners/technology-alliances/ansible Imperative: https://support.f5.com/csp/article/K42420223 https://clouddocs.f5.com/products/orchestration/ansible/devel/usage/playbook_tutorial.html Declaritive: https://www.f5.com/company/blog/f5-as3-and-red-hat-ansible-automation https://clouddocs.f5.com/training/fas-ansible-workshop-101/3.0-as3-intro.html 10. For some automations without rest-api better use the F5 native cli scripts than bash with tmsh commands: https://clouddocs.f5.com/cli/tmsh-reference/v14/modules/cli/cli_script.html https://clouddocs.f5.com/api/tmsh/script__run.html
Nikoolayy1
Jun 09, 2021 Place Technical Forum
2.6KViews
1like
2Comments