Part 2: Monitoring the CPU usage of the BIG-IP system using a periodic iCall handler

In this part series, you monitor the CPU usage of the BIG-IP system with a periodic iCall handler. The specific CPU statistics you want to monitor can be retrieved from either Unix or tmsh commands. For example, if you want to monitor the CPU usage of the tmm process, you can monitor the values from the output of the tmsh show sys proc-info tmm.0 command.

An iCall script can iterate and retrieve a list of values from the output of a tmsh command. To display the fields available from a tmsh command that you can iterate from an iCall Tcl script, run the tmsh command with the field-fmt option. For example:

tmsh show sys proc-info tmm.0 field-fmt

You can then use a periodic iCall handler which runs an iCall script periodically every interval to check the value of the output of the tmsh command. When the value exceeds a configured threshold, you can have the script perform an action; for example, an alert message can be logged to the /var/log/ltm file. The following describes the procedures:

1. Creating an iCall script to monitor the required CPU usage values
2. Creating a periodic iCall handler to run the iCall script once a minute

1. Creating an iCall script to monitor the required CPU usage values

There are different Unix and tmsh commands available to display CPU usage. To monitor CPU usage, this example uses the following:

• tmsh show sys performance system detail | grep CPU: This displays the system CPU Utilization (%). The script monitors CPU usage from the Average column for each CPU.
• tmsh show sys proc-info apmd: Monitors the CPU usage System Utilization (%) Last5-mins value of the apmd process.
• tmsh show sys proc-info tmm.0: Monitors the CPU usage System Utilization (%) Last5-mins value of the tmm process. This is the sum of the CPU usage of all threads of the tmm.0 process divided by the number of CPUs over five minutes. You can display the number of TMM processes and threads started, by running different commands. For example:
• pstree -a -A -l -p | grep tmm | grep -v grep
• grep Start /var/log/tmm.start

You can also create your own script to monitor the CPU output from other commands, such as tmsh show sys cpu or tmsh show sys tmm-info. However, a discussion on CPU usage on the BIG-IP system is beyond the scope of this article. For more information, refer to K14358: Overview of Clustered Multiprocessing (11.3.0 and later) and K16739: Understanding 'top' output on the BIG-IP system.

You need to set some of the variables in the script, specifically the threshold values: cpu_perf_threshold, tmm.0_threshold, apmd_threshold respectively. In this example, all the CPU threshold values are set at 80%. Note that depending on the set up in your specific environment, you have to adjust the threshold accordingly. The threshold values also depend on the action you plan to run in the script. For example, in this case, the script logs an alert message in the /var/log/ltm file. If you plan to log an emerg message, the threshold values should be higher, for example, 95%.

Procedure

Perform the following procedure to create the script to monitor CPU statistics and log an alert message in the /var/log/ltm file when traffic exceeds a CPU threshold value.

To create an iCall script, perform the following procedure:

1. Log in to tmsh.
2. Enter the following command to create the script in the vi editor:

    create sys icall script cpu_script

3. Enter the following script into the definition stanza of the editor.

The 3 threshold values are currently set at 80%. You can change it according to the requirements in your environment.

definition {
      set DEBUG 0
      set VERBOSE 0

      #CPU threshold in % from output of tmsh show sys performance system detail
      set cpu_perf_threshold 80

      #The name of the process from output of tmsh show sys proc-info to check. The name must match exactly.
      #If you would like to add another process, append the process name to the 'process' variable and add another line for threshold.
      #E.g. To add tmm.4, "set process apmd tmm.0 tmm.4" and add another line "set tmm.4_threshold 75"
      set process "apmd tmm.0"
      #CPU threshold in % for output of tmsh show sys proc-info
      set tmm.0_threshold 80
      set apmd_threshold 80

      puts "\n[clock format [clock seconds] -format "%b %d %H:%M:%S"] Running CPU monitoring script..."


      #Getting average CPU output of tmsh show sys performance
      set errorcode [catch {exec tmsh show sys performance system detail | grep CPU | grep -v Average | awk {{ print $1, $(NF-4), $(NF-3), $(NF-1) }}} result]
      if {[lindex $result 0] == "Blade"} { set blade 1 } else { set blade 0 }
      set result [split $result "\n"]
      foreach i $result {
         set cpu_num "[lindex $i 1] [lindex $i 2]"
         if {$blade} {set cpu_num "Blade $cpu_num"}
         set cpu_rate [lindex $i 3]
         if {$DEBUG} {puts "tmsh show sys performance->${cpu_num}: ${cpu_rate}%."}
         if {$cpu_rate > $cpu_perf_threshold} {
             if {$DEBUG} {puts "tmsh show sys performance->${cpu_num}: ${cpu_rate}%. Exceeded threshold ${cpu_perf_threshold}%."}
             exec logger -p local0.alert "\"tmsh show sys performance\"->${cpu_num}: ${cpu_rate}%. Exceeded threshold ${cpu_perf_threshold}%."
         }
      }

     #Getting output of tmsh show sys proc-info
      foreach obj [tmsh::get_status sys proc-info $process] {
          if {$VERBOSE} {puts $obj}
          set proc_name [tmsh::get_field_value $obj proc-name]
          set cpu [tmsh::get_field_value $obj system-usage-5mins]
          set pid [tmsh::get_field_value $obj pid]
          set proc_threshold ${proc_name}_threshold
          set proc_threshold [set [set proc_threshold]]

          if {$DEBUG} {puts "tmsh show sys proc-info-> Average CPU Utilization of $proc_name pid $pid is ${cpu}%"}
          if { $cpu > ${proc_threshold} } {
              if {$DEBUG} {puts "$proc_name process pid $pid at $cpu% cpu. Exceeded ${proc_threshold}% threshold."}
              exec logger -p local0.alert "\"tmsh show sys proc-info\" $proc_name process pid $pid at $cpu% cpu. Exceeded ${proc_threshold}% threshold."
          }
      }
}

4. Configure the variables in the script as needed and exit the editor by entering the following command:

    :wq!
    y

5. Run the following command to list the contents of the script:

    list sys icall script cpu_script

2. Creating a periodic iCall handler to run the iCall script once a minute

Procedure

Perform the following procedure to create the periodic handler that runs the script once a minute.

To create an iCall periodic handler, perform the following procedure:

1. Log in to tmsh
2. Enter the following command to create a periodic handler:

    create sys icall handler periodic cpu_handler interval 60 script cpu_script

3. Run the following command to list the handler:

    list sys icall handler periodic cpu_handler

4. You can start and stop the handler by using the following command syntax:

    <start|stop> sys icall handler periodic cpu_handler

Follow the /var/tmp/scriptd.out and /var/log/ltm file entries to verify your implementation is working correctly.