Using iCall to monitor BIG-IP APM network access VPN

[Kin]

Introduction

During peak periods, when a large number of users are connected to network access VPN, it is important to monitor your BIG-IP APM system's resource (CPU, memory, and license) usage and performance to ensure that the system is not overloaded and there is no impact on user experience. If you are a BIG-IP administrator, iCall is a tool perfectly suited to do this for you. iCall is a Tcl-based scripting framework that gives you programmability in the control plane, allowing you to script and run Tcl and TMOS Shell (tmsh) commands on your BIG-IP system based on events. For a quick introduction to iCall, refer to iCall - All New Event-Based Automation System.

Overview

This article is made up of three parts that describe how to use and configure iCall in the following use cases to monitor some important BIG-IP APM system statistics:

• Part 1: Monitoring access sessions and CCU license usage of the system using a triggered iCall handler
• Part 2: Monitoring the CPU usage of the system using a periodic iCall handler.
• Part 3: Monitoring the health of BIG-IP APM network access VPN PPP connections with a periodic iCall handler.

In all three cases, the design consists of identifying a specific parameter to monitor. When the value of the parameter exceeds a configured threshold, an iCall script can perform a set of actions such as the following:

• Log a message to the /var/log/apm file at the appropriate severity:
•  emerg: System is unusable
•  alert: Action must be taken immediately
•  crit: Critical conditions

You may then have another monitoring system to pick up these messages and respond to them.

• Perform a remedial action to ease the load on the BIG-IP system.
• Run a script (Bash, Perl, Python, or Tcl) to send an email notification to the BIG-IP administrator.
• Run the tcpdump or qkview commands when you are troubleshooting an issue.

When managing or troubleshooting iCall scripts and handler, you should take into consideration the following:

• You use the Tcl language in the editor in tmsh to edit the contents of scripts and handlers. For example:
• create sys icall script <name of script>
• edit sys icall script <name of script>
• The puts command outputs entries to the /var/tmp/scriptd.out file. For example:
• puts "\n[clock format [clock seconds] -format "%b %d %H:%M:%S"] Running script..."
• You can view the statistics for a particular handler using the following command syntax:
• show sys icall handler <periodic | perpetual | triggered> <name of handler>

Series 1: Monitoring access sessions and CCU license usage with a triggered iCall handler

You can view the number of currently active sessions and current connectivity sessions usage on your BIG-IP APM system by entering the tmsh show apm license command. You may observe an output similar to the following:

--------------------------------------------
Global Access License Details:             
--------------------------------------------
 total access sessions: 10.0M              
 current active sessions: 0                
 current established sessions: 0           
 access sessions threshold percent: 75     
 total connectivity sessions: 2.5K         
 current connectivity sessions: 0          
 connectivity sessions threshold percent: 75

In the first part of the series, you use iCall to monitor the number of current access sessions and CCU license usage by performing the following procedures:

1. Modifying database DB variables to log a notification when thresholds are exceed.
2. Configuring user_alert.conf to generate an iCall event when the system logs the notification.
3. Creating a script to respond when the license usage reaches its threshold.
4. Creating an iCall triggered handler to handle the event and run an iCall script
5. Testing the implementation using logger

1. Modifying database variables to log a notification when thresholds are exceeded.

The tmsh show apm license command displays the access sessions threshold percent and access sessions threshold percent values that you can configure with database variables. The default values are 75. For more information, refer to K62345825: Configuring the BIG-IP APM system to log a notification when APM sessions exceed a config....

When the threshold values are exceeded, you will observe logs similar to the following in /var/log/apm:

notice tmm1[<pid>]: 01490564:5: (null):Common:00000000: Global access license usage is 1900 (76%) of 2500 total. Exceeded 75% threshold of total license.
notice tmm2[<pid>]: 01490565:5: 00000000: Global concurrent connectivity license usage is 393 (78%) of 500 total. Exceeded 75% threshold of total license.

Procedure:

Run the following commands to set the threshold to 95% for example:

tmsh modify /sys db log.alertapmaccessthreshold value 95
tmsh modify /sys db log.alertapmconnectivitythreshold value 95

Whether to set the alert threshold at 90% or 95%, depends on your specific environment, specifically how fast the usage increases over a period of time.

2. Configuring user_alert.conf to generate an iCall event when the system logs the notification

You can configure the /config/user_alert.conf file to run a command or script based on a syslog message. In this step, edit the user_alert.conf file with your favorite editor, so that the file contains the following stanza.

alert <name> "<string in syslog to match to trigger event>" {

  <command to run>

}

For more information on configuring the /config/user_alert.conf file, refer to K14397: Running a command or custom script based on a syslog message. In particular, it is important to read the bullet points in the Description section of the article first; for example, the system may not process the user_alert.conf file after system upgrades. In addition, BIG-IP APM messages are not processed by the alertd SNMP process by default. So you will also have to perform the steps described in K51341580: Configuring the BIG-IP system to send BIG-IP APM syslog messages to the alertd process as well.

Procedure:

Perform the following procedure:

1. Edit the /config/user_alert.conf file to match each error code and generate an iCall event named apm_threshold_event. Per K14397

Note: You can create two separate alerts based on both error codes or alternatively use the text description part of the log message common to both log entries to capture both in a single alert. For example "Exceeded 75% threshold of total license"

# cat /config/user_alert.conf
alert apm_session_threshold "01490564:" {
  exec command="tmsh generate sys icall event apm_threshold_event"
}
alert apm_ccu_threshold "01490565:" {
  exec command="tmsh generate sys icall event apm_threshold_event"
}

2. Run the following tmsh command:

edit sys syslog all-properties

3. Replace the include none line with the following: Per K51341580

   include "
   filter f_alertd_apm {
       match (\": 0149[0-9a-fA-F]{4}:\");
   };
   log {
       source(s_syslog_pipe);
       filter(f_alertd_apm);
       destination(d_alertd);
   };
   "

3. Creating a script to respond when the license usage reaches its threshold.

When the apm session or CCU license usage exceeds your configured threshold, you can use a script to perform a list of tasks. For example, if you had followed the earlier steps to configure the threshold values to be 95%, you can write a script to perform the following actions:

• Log a syslog alert message to the /var/log/apm file. If you have another monitoring system, it can pick this up and respond as well.
• Optional: Run a tmsh command to modify the Access profile settings. For example, when the threshold exceeds 95%, you may want to limit users to one apm session each, decrease the apm access profile timeout or both. Changes made only affect new users. Users with existing apm sessions are not impacted.
• If you are making changes to the system in the script, it is advisable to run an additional tmsh command to stop the handler. When you have responded to the alert, you can manually start the handler again.

Note: When automating changes to the system, it is advisable to err on the side of safety by making minimal changes each time and only when required. In this case, after the system reaches the license limit, users cannot login and you may need to take immediate action.

Procedure:

Perform the following procedure to create the iCall script:

1. Log in to tmsh.

2. Run the following command:

  create sys icall script threshold_alert_script

3. Enter the following in the editor:

Note: The tmsh commands to modify the access policy settings have been deliberately commented out. Uncomment them when required.

sys icall script threshold_alert_script {
   app-service none
   definition {
       exec logger -p local1.alert "01490266: apm license usage exceeded 95% of threshold set."
       #tmsh::modify apm profile access exampleNA max-concurrent-sessions 1
       #tmsh::modify apm profile access exampleNA generation-action increment
       #tmsh::stop sys icall handler triggered threshold_alert_handler
   }
   description none
   events none
}

4. Creating an iCall triggered handler to handle the event and run an iCall script

In this step, you create a triggered iCall handler to handle the event triggered by the tmsh generate sys icall event command from the earlier step to run the script.

Procedure:

Perform the following:

1. Log in to tmsh.

2. Enter the following command to create the triggered handler.

create sys icall handler triggered threshold_alert_handler script threshold_alert_script subscriptions add { apm_threshold_event { event-name apm_threshold_event } }

Note: The event-name field must match the name of the event in the generate sys icall command in /config/user_alert.conf you configured in step 2.

3. Enter the following command to verify the configuration of the handler you created.

(tmos)# list sys icall handler triggered threshold_alert_handler
sys icall handler triggered threshold_alert_handler {
   script threshold_alert_script
   subscriptions {
       apm_threshold_event {
           event-name apm_threshold_event
       }
   }
}

5. Testing the implementation using logger

You can use the logger command to log test messages to the /var/log/apm file to test your implementation. To do so, run the following command:

Note: The message below must contain the keyword that you are searching for in the script. In this example, the keyword is 01490564 or 01490565.

logger -p local1.notice "01490564:5: (null):Common:00000000: Global access license usage is 1900 (76%) of 2500 total. Exceeded 75% threshold of total license."

logger -p local1.notice "01490565:5: 00000000: Global concurrent connectivity license usage is 393 (78%) of 500 total. Exceeded 75% threshold of total license."

Follow the /var/log/apm file to verify your implementation is working correctly.