F5 BIG-IP and ENTRUST nShield HSM SSL key/cert auto synchronization between HA peers with iCall
Code version: The code was tested on 15.1.8. Main Article: For more information about RFS and Client agent I suggest seeing the vendors article. https://nshielddocs.entrust.com/security-world-docs/v13.3/connect-ug-nix/intro.html Useful F5 links for F5 and nShield Integration for GTM and LTM: https://my.f5.com/manage/s/article/K000135349 https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-system-and-nshield-hsm-implementation/setting-up-t... The nShield architecture includes a component called the Remote File System (RFS) that stores and manages the encrypted key files. The RFS can be installed on the BIG-IP system or on another server on your network. Basically the HSM agent/client is installed on the F5 devices hos Linux host system and the F5 devices are also the RFS servers. The RFS commands are bellow as when installed on the BIG-IP the HSM agent and RFS they are available for use: https://nshielddocs.entrust.com/security-world-docs/utilities/rfs-sync.html The issue I solved with iCall script is that when when you create a new HSM key for BIG-IP HA, you must run command ‘rfs-sync --update’ on all standby BIG-IP devices (the devices where the cert/key were not created or changed) to update the local Thales encrypted file object cache. Without this action, SSL traffic using this key will fail when BIG-IP fails over to one of the unsynced standby devices. When you create the the key and cert on the active F5 device "rfs-sync -commit" and "rfs-sync -update" run automatically on it but not the "rfs-sync -update" on the standby devices and the icall script basically is triggered on the standby devices when you run the normal config sync. The iCall script matched an event called "HA_EVENT" that is configured in the custom alarms section and triggers the full command with the path "/opt/nfast/bin/rfs-sync --update" to check if there was an update in the rfs. I suggest reading the links below that explain the iCall (one is from JRahm ) and the HA logs and the last one is mine that is from the time before I learned proper article formatting 😅 and it also shows how to run scripts not only with iCall but also during HA events and so on. Run tcpdump on event | DevCentral What is iCall? | DevCentral https://my.f5.com/manage/s/article/K34291400 https://my.f5.com/manage/s/article/K3727 https://my.f5.com/manage/s/article/K11127 Knowledge sharing: Ways to trigger and schedule scripts on the F5 BIG-IP devices. | DevCentral tmsh list sys icall sys icall handler triggered ha-handler { script ha-script subscriptions { ha-subscription { event-name HA_EVENT } } } sys icall script ha-script { app-service none definition { exec /bin/bash -c "logger -p local0.notice 'yes'" exec /bin/bash -c "/opt/nfast/bin/rfs-sync --update" } description none events none } cat /config/user_alert.conf alert HA_EVENT "(.*)Sync of device group(.*)" { snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.500" } You can use tmsh::log "yes" to log to /var/log/ltm as shown in: iCall script to validate Virtual Server and node with same IP addresses Testing: This can be tested even without HSM as I don't have one at my home using "logger -p" to inject the logs. I have added "yes" in the logs as a test 😎 logger -p local0.notice "010714a0:5: Sync of device group /Common/Failover" cat /var/log/ltm ............ Jul 1 08:34:07 bigip1.com notice root[23762]: yes Jul 1 08:34:53 bigip1.com notice root[23940]: 010714a0:5: Sync of device group /Common/Failover Using Linux bash script: In some versions you can trigger from icall script a bash or sh script with advanced logic inside, for example /bin/sh -c "/var/tmp/ha_script" but I saw issues on 17.1.x triggering from from icall a bash script that in previous versions I solved by adding "HOME=/root <linux bash command>" . In the "What is iCall" it is shown that you can do "if else" or "for" loops inside the iCall script but I find it easy to use bash for advanced logic. Good thing like everything with F5 usually there is more than one way to do things and this case in the user_alert.conf you can actually trigger a bash script from the log messages! What is iCall? | DevCentral iCall script triggers error need ${HOME} to run | DevCentral Running a command or custom script based on a syslog message cat /var/tmp/ha_script #!/bin/bash logger -p local0.notice "yes" /opt/nfast/bin/rfs-sync --update cat /config/user_alert.conf alert HA_EVENT "(.*)Sync of device group(.*)" { exec command="/var/tmp/ha_script" } Extra Notes: Using /opt/nfast/bin/rfs-sync --update or rfs-sync --update depends in some cases on the versions in the iCall script. In the release notes I saw a new bug https://cdn.f5.com/product/bugtracker/ID1429897.html that is solved in the latest 17.1.x versions where if the RFS is on the BIG-IP after a key/cert are created 'rfs-sync -c' needs to be run on the F5 Device that created them as well. The 'rfs-sync -c' can also be automated the way I have shown and my iCall script will work as well for BIG-IP that use external RFS and after the key/cert are created and committed from an F5 device (usually the active one) then an HA config sync needs to be started that will trigger 'rfs-sync -u' on the other F5 devices. Another nice way if you are using something like Ansible for example is to make to trigger the RFS update command on all F5 devices in a cluster as F5 supports bash commands even through API not only CLI. Example: curl -sku admin:XXX https://XXXX/mgmt/tm/util/bash -H "Content-Type: application/json" -X POST -d '{"command":"run", "utilCmdArgs":"-c \"/opt/nfast/bin/rfs-sync --update\""}' https://clouddocs.f5.com/products/orchestration/ansible/devel/f5_bigip/modules_2_0/bigip_command_mod...
Icall script argument
Hello! How I can translate to icall script argument from APM via iRule? Example. I want generate user certificate SSL via APM. I wrote bash script, but it should be called with two argument - UserName and UserDomain. Thank you! sys icall script gcc_script { app-service none definition { exec /home/root/scripts/certificates.sh $UserDN $DomainDN exec istats remove "GCC generate for UserDN" } description none events none }
Regular expression format in user_alert.conf
I'm trying to use iCall and an event from user_alert.conf to fail over a BIG-IP VE cluster if an arbitrary BGP neighbor goes down. I have the handler and script working just fine if the event only looks in my logs for a static phrase, but when I have it look for a regex instead, it no longer works. However, if I test in a tool like regex101 with my expression and a log entry, it matches just fine. Here's my user_alert.conf (sanitized of course) alert bgp_neighbor_down "neighbor 100.200.[0-9]{1,3}.[0-9]{1,3} Down" { exec command="tmsh generate sys icall event neighbordown context { { name protocol value bgp } }" } And one of the logs I'm trying to match on: 2024/06/20 15:04:32 informational: BGP : %BGP-5-ADJCHANGE: neighbor 100.200.30.4 Down BGP Notification CEASE If I then run imish and shut down a neighbor that should match that regex, the device I'm on stays active. Any thoughts on what else I can try?
Run a sync-config from iCall
Hi, I am wrinting a iCall which modify the configuration with tmsh::modify. I have a cluster of Big-IP so after the command tmsh::modify change the configuration, I have to run a config-sync. My question is how to run a config-sync from a iCall? I can not find any command unless maybe with exec. Thank you. Regards,iCall Script that only runs on Active member
Problem this snippet solves: I had a request to run an iCall script only on the active member in a pair. How to use this snippet: This won't work if you're using active/active via traffic-groups. Code : # Only execute if local BIG-IP is active in failover if {[exec cat /var/prompt/ps1] == "Active"} { tmsh::log "I LIKE SOUP!" } Tested this on version: 12.1
Not working my iCall script...
Problem this snippet solves: Hello~ I created iCall and generate event for working iCall's script. But.... It is not working. list sys icall script change_iRule2 sys icall script change_iRule2 { app-service none definition { tmsh::modify ltm virtual v_198.100__443 rules { vip_snatpool } tmsh::modify ltm virtual v_198.101__443 rules { vip_snatpool } } description none events none } list sys icall handler triggered change_iRule2{ script change_iRule2 subscriptions { change_iRule2 { event-name change_iRule2 } } } I gernerated event generate sys icall event change_iRule2 And I checked bellow status...... Sys::iCall::Event Triggered Handler: change_iRule2 Events matching filters 11 Events causing handler to run 11 Creation time 01/24/17 16:38:35 Current status active Time since last status change 01/24/17 16:38:35 But, Configuration(virtual server's iRule) is not change.... Plz help me. T..T Code : sys icall script change_iRule2 { app-service none definition { tmsh::modify ltm virtual v_198.100__443 rules { vip_snatpool } tmsh::modify ltm virtual v_198.101__443 rules { vip_snatpool } } description none events none } create sys icall handler triggered change_iRule2 script change_iRule2 subscriptions add { change_iRule2 { event-name change_iRule2 } }[icall] kill oldest sessions when reaching xx% of the APM license limit
Problem this snippet solves: When dealing with APM authentication, especially when the F5 device act as a SAML 2.0 IDP, the active sessions can increase considerably and easily reach the max access session limit of the license or the device. The following icall script allows an administrator to guarantee that new users can still authenticate through APM IDP under heavy load. The script will kill oldest active access sessions based on the access session consumption. This is a draft that need to be fine tuned. Warning: when using APM Guest on a vCMP host, we are not able to guarantee that the appliance limit is not reached as we just have knowledge of the active sessions within the guest context only. How to use this snippet: TMSH command to create an icall script create sys icall script apm_purge_sessions Then copy/paste the content of the icall script and save it. By default, the command create a script named "apm_purge_sessions". You can easily change the name of the script by modifying "apm_purge_sessions" in the command line. TMSH command to create the icall handler The following command trigger the script every 60 seconds. It can be changed to increase the frequency of the execution of the script. create sys icall handler periodic f5-apm-purge-session interval 60 script apm_purge_sessions Interesting tcl commands used in the script Retrieve the max_access_session variable in the license of the device: [string trim [lindex [split [exec /usr/bin/tmsh show /sys license detail | grep access] " "] 1] "\[\]"] retrieve the ordered list (oldest first) of active APM sessionIDs catch {set output [exec /usr/bin/sessiondump --allkeys | grep starttime | sort -k3 | cut -c1-8]} Use cases kill oldest sessions when reaching xx% of the APM license limit Evolution trigger the icall script based on a specific event (snmptrap, log, ...) sort APM sessions by Access Profile and kill sessions based on the criticity of each AP. Code : # retrieve the ordered list of active APM sessionIDs catch {set output [exec /usr/bin/sessiondump --allkeys | grep starttime | sort -k3 | cut -c1-8]} if {$output != ""} { # move the output to a list of sessionID set output [split $output "\n"] set count [llength $output] # determine the max_access_session allowed for the running platform set max_access [string trim [lindex [split [exec /usr/bin/tmsh show /sys license detail | grep access] " "] 1] "\[\]"] # determine acceptable threshold before triggering set access_threshold [expr round($max_access*0.85)] set diff [expr $count-$access_threshold] # kill oldest APM sessions until reaching 85% of active sessions in the APM device for {set i 0} {$i < $diff} {incr i} { catch { [exec /usr/bin/sessiondump --delete [lindex $output $i]] } } } Tested this on version: 11.6What is iCall?
tl;dr - iCall is BIG-IP’s event-based granular automation system that enables comprehensive control over configuration and other system settings and objects. The main programmability points of entrance for BIG-IP are the data plane, the control plane, and the management plane. My bare bones description of the three: Data Plane - Client/server traffic on the wire and flowing through devices Control Plane - Tactical control of local system resources Management Plane - Strategic control of distributed system resources You might think iControl (our SOAP and REST API interface) fits the description of both the control and management planes, and whereas you’d be technically correct, iControl is better utilized as an external service in management or orchestration tools. The beauty of iCall is that it’s not an API at all—it’s lightweight, it’s built-in via tmsh, and it integrates seamlessly with the data plane where necessary (via iStats.) It is what we like to call control plane scripting. Do you remember relations and set theory from your early pre-algebra days? I thought so! Let me break it down in a helpful way: P = {(data plane, iRules), (control plane, iCall), (management plane, iControl)} iCall allows you to react dynamically to an event at a system level in real time. It can be as simple as generating a qkview in the event of a failover or executing a tcpdump on a server with too many failed logins. One use case I’ve considered from an operations perspective is in the event of a core dump to have iCall generate a qkview, take checksums of the qkview and the dump file, upload the qkview and generate a support case via the iHealth API, upload the core dumps to support via ftp with the case ID generated from iHealth, then notify the ops team with all the appropriate details. If I had a solution like that back in my customer days, it would have saved me 45 minutes easy each time this happened! iCall Components Three are three components to iCall: events, handlers, and scripts. Events An event is really what drives the primary reason to use iCall over iControl. A local system event (whether it’s a failover, excessive interface or application errors, too many failed logins) would ordinarily just be logged or from a system perspective, ignored altogether. But with iCall, events can be configured to force an action. At a high level, an event is "the message," some named object that has context (key value pairs), scope (pool, virtual, etc), origin (daemon, iRules), and a timestamp. Events occur when specific, configurable, pre-defined conditions are met. Example (placed in /config/user_alert.conf) alert local-http-10-2-80-1-80-DOWN "Pool /Common/my_pool member /Common/10.2.80.1:80 monitor status down" { exec command="tmsh generate sys icall event tcpdump context { { name ip value 10.2.80.1 } { name port value 80 } { name vlan value internal } { name count value 20 } }" } Handlers Within the iCall system, there are three types of handlers: triggered, periodic, and perpetual. Triggered A triggered handler is used to listen for and react to an event. Example (goes with the event example from above:) sys icall handler triggered tcpdump { script tcpdump subscriptions { tcpdump { event-name tcpdump } } } Periodic A periodic handler is used to react to an interval timer. Example: sys icall handler periodic poolcheck { first-occurrence 2017-07-14:11:00:00 interval 60 script poolcheck } Perpetual A perpetual handler is used under the control of a deamon. Example: handler perpetual core_restart_watch sys icall handler perpetual core_restart_watch { script core_restart_watch } Scripts And finally, we have the script! This is simply a tmsh script moved under the /sys icall area of the configuration that will “do stuff" in response to the handlers. Example (continuing the tcpdump event and triggered handler from above:) modify script tcpdump { app-service none definition { set date [clock format [clock seconds] -format "%Y%m%d%H%M%S"] foreach var { ip port count vlan } { set $var $EVENT::context($var) } exec tcpdump -ni $vlan -s0 -w /var/tmp/${ip}_${port}-${date}.pcap -c $count host $ip and port $port } description none events none } Resources iCall Codeshare Lightboard Lessons on iCall Threshold violation article highlighting periodic handler
