All BIG-IP systems are pre-configured with a set of trap definitions that are helpful for managing the hardware and software components of the device.  In most cases the default traps are sufficient for monitoring and managing the system.  Some deployments, however, require custom SNMP traps on other logged events.  In this article, I will show you the process of building a custom SNMP trap for BIG-IP. 

BIG-IP uses a standard syslog-ng / alertd framework for generating alerts, including SNMP traps.  In BIG-IP version 9.x, SNMP traps are triggered when the alertd process receives input from the syslog-ng utility that matches an alert code or match string. The alertd process then performs the action specified in the /etc/alertd/alert.conf or /config/user_alert.conffile, such as sending an SNMP trap.

Before you attempt to define a custom SNMP trap, you should already have SNMP configured on BIG-IP with at least one trapsink destination.  (If not, please refer to the BIG-IP® Network and System Management Guide: 15 - Configuring SNMP).  

With SNMP configured, the steps for creating a custom trap are:

1. Identify a trappable log event
2. Create the custom match expression
3. Choose a trap OID
4. Modify alertd configuration
5. Test the alert

(Note: This article does not address configuration of the trapsink device for trap processing, only the configuration on BIG-IP required to generate and send the trap.)

First let's take a look at the existing trap definitions.

SNMP trap definitions

You can determine which alerts will trigger an SNMP trap by viewing /etc/alertd/alert.conf and /var/tmpfs/run/bigip_error_maps.dat.

The /etc/alertd/alert.conf file contains the alert definitions, which instruct the system on what actions to take when the alert is triggered. 

View the /etc/alertd/alert.conf file with the following command (The user_alert.conf file may also contain alerts, and should be examined in the same manner.): 

less /etc/alertd/alert.conf

 An alert definition looks like this:

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS {
        snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.10"
}

Alert definitions configured to trigger an SNMP trap will contain the snmptrap command, as in the example above:

snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.10";

The first line always contains the alert name:

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS

The first line of an alert definition may also optionally contain the match string, as in this example, which uses a regular expression to catch all local authentication failure log messages:

alert BIGIP_AUTH_FAIL "FAILED LOGIN (.*) FROM (.*) FOR (.*), Authentication failure" {
        snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.27"
}

If, as in our original example, the match string is not defined in the alert definition itself, you can find it in the /var/tmpfs/run/bigip_error_maps.dat by searching for the alert name with this command:

grep BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS /var/tmpfs/run/bigip_error_maps.dat

to find this match string, which matches on all pool member status changes:

0 LOG_NOTICE    01070638 BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS "Pool member %s:%u monitor status %s."

(Regular expressions may be used in the alert.conf/user_alert.conf file, while C-style variable substitutions such as %s (string) and %u (unsigned integer) are used in bigip_error_maps.dat.

When the alertd process starts, the runtime configuation file, /var/tmpfs/run/alert.conf, is created by appending the /config/user_alert.conf file to the /etc/alertd/alert.conf file, and the /var/run/bigip_error_maps.dat file is dynamically generated using entries from all of the map files in the /etc/alertd/ directory that end with the _maps.h file extension.  The alertd process uses the /var/run/bigip_error_maps.dat file to map input it receives from the syslog-ng utility to an alert name. When alertd receives input from syslog-ng utility that matches an alert code, the alert code is mapped to the alert name. The alertd process then performs the alert actions specified for that alert name in the /etc/alertd/alertd.conf file, such as issuing an LCD warning, or sending an SNMP trap.

So now that we some understanding of the underpinnings, let's move on to defining a custom trap.

Step 1: Identify trappable log event

Any event that is logged by syslog-ng may be used to configure a custom SNMP trap.  For example, a normally functioning system commonly logs messages related to node or pool member status changes.  We will use one of those log messages as an example to demonstrate the creation of a custom SNMP traps.

For example, when a pool member changes status, a line is logged to the LTM Local Traffic log (/var/log/ltm) that indicates the the pool member's identity and current status: 

Jul 22 07:38:28 Primary mcpd[1844]: 01070638:5: Pool member 10.0.0.154:80 monitor status forced down.

From the example above, we know there is already a default trap that is generated for all pool member status changes.  For demonstration purposes, let's assume that the host at 10.0.0.154:80 is a critical system, and we would like to define a custom trap with its own OID.

Step 2: Create the custom match expression

To create a pool member specific trap, you will need to make the match expression specific to that pool member:  Using the existing trap as a template, we can insert the specific pool member IP and port, and since the alert match string for custom traps is always defined in the user_alert.conf file, we must replace the %s status variable with a corresponding regular expression to catch all status changes for this pool member, creating the following match expression:

"Pool member 10.0.0.154:80 monitor status (.*?)."

Step 3: Choose an OID

An SNMP OID (Object ID) is simply a unique numeric index for the event you wish to trap.

In version 4.x, F5 Networks recommended that you use traps in the range 1.3.6.1.4.1.3375.1.1.110.200 - 1.3.6.1.4.1.3375.1.1.110.999, in order to prevent conflict with official traps added by F5 in the future. As of 9.x, however, we recommend updating this to a range of1.3.6.1.4.1.3375.2.4.0.300 - 1.3.6.1.4.1.3375.2.4.0.999 to avoid conflicts. For example, if you're running v9.x or later, when you add your first custom SNMP trap use OID 1.3.6.1.4.1.3375.2.4.0.300. When you add another SNMP trap, use 1.3.6.1.4.1.3375.2.4.0.301.

However, you can use any object ID that meets all of the following criteria:

• is in standard OID format, such as .1.3.6.1.4.1.3375.2.4.0.500
• is in a numeric range that can be processed by your trap receiving tool
• does not already exist in the /usr/share/snmp/mibs/F5-BIGIP-COMMON-MIB.txt MIB file
• is not already used in another custom trap

For our example, we will use 1.3.6.1.4.1.3375.1.1.110.200.

Step 4: Modify the alertd configuration

Note: Custom, user-defined SNMP traps are defined in the /config/user_alert.conf file. Standard, pre-configured SNMP traps are defined in the /etc/alertd/alert.conf file.  Do not add or remove traps from the alert.conf file. When the alertd process starts, it creates a dynamic configuration file, /var/tmpfs/run/alert.conf, by appending the /config/user_alert.conf file to the /etc/alertd/alert.conf file.

To add your custom trap, edit the /config/user_alert.conf file and add a new SNMP trap using the parts we constructed above and a unique alert name, following the expected format: 

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS_SERVERX "Pool member 10.0.0.154:80 monitor status (.*?)." {
        snmptrap OID=".1.3.6.1.4.1.3375.1.1.110.200"
}

Step 5: Test the trap

 To generate test traps, you can use the logger command to push messages through syslog-ng that match the new trap definition.

First Extract the syslog message string contained in double quotes.

Pool member 10.0.0.154:80 monitor status (.*?).

 Then use the extracted syslog message to define your logger command as follows:

logger -p local0.warning "Pool member 10.0.0.154:80 monitor status down."

This command will output a syslog-ng message to the local0.warning facility (the default location is the /var/log/ltm file), and an SNMP trap for this message will be generated.

 (Note: Since we already had some traps defined for pool member status, they will continue to match as well, so you should see at least 2 traps generated from this message).