F5 BIG-IP and ENTRUST nShield HSM SSL key/cert auto synchronization between HA peers with iCall
Code version: The code was tested on 15.1.8. Main Article: For more information about RFS and Client agent I suggest seeing the vendors article. https://nshielddocs.entrust.com/security-world-docs/v13.3/connect-ug-nix/intro.html Useful F5 links for F5 and nShield Integration for GTM and LTM: https://my.f5.com/manage/s/article/K000135349 https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-system-and-nshield-hsm-implementation/setting-up-t... The nShield architecture includes a component called the Remote File System (RFS) that stores and manages the encrypted key files. The RFS can be installed on the BIG-IP system or on another server on your network. Basically the HSM agent/client is installed on the F5 devices hos Linux host system and the F5 devices are also the RFS servers. The RFS commands are bellow as when installed on the BIG-IP the HSM agent and RFS they are available for use: https://nshielddocs.entrust.com/security-world-docs/utilities/rfs-sync.html The issue I solved with iCall script is that when when you create a new HSM key for BIG-IP HA, you must run command ‘rfs-sync --update’ on all standby BIG-IP devices (the devices where the cert/key were not created or changed) to update the local Thales encrypted file object cache. Without this action, SSL traffic using this key will fail when BIG-IP fails over to one of the unsynced standby devices. When you create the the key and cert on the active F5 device "rfs-sync -commit" and "rfs-sync -update" run automatically on it but not the "rfs-sync -update" on the standby devices and the icall script basically is triggered on the standby devices when you run the normal config sync. The iCall script matched an event called "HA_EVENT" that is configured in the custom alarms section and triggers the full command with the path "/opt/nfast/bin/rfs-sync --update" to check if there was an update in the rfs. I suggest reading the links below that explain the iCall (one is from JRahm ) and the HA logs and the last one is mine that is from the time before I learned proper article formatting 😅 and it also shows how to run scripts not only with iCall but also during HA events and so on. Run tcpdump on event | DevCentral What is iCall? | DevCentral https://my.f5.com/manage/s/article/K34291400 https://my.f5.com/manage/s/article/K3727 https://my.f5.com/manage/s/article/K11127 Knowledge sharing: Ways to trigger and schedule scripts on the F5 BIG-IP devices. | DevCentral tmsh list sys icall sys icall handler triggered ha-handler { script ha-script subscriptions { ha-subscription { event-name HA_EVENT } } } sys icall script ha-script { app-service none definition { exec /bin/bash -c "logger -p local0.notice 'yes'" exec /bin/bash -c "/opt/nfast/bin/rfs-sync --update" } description none events none } cat /config/user_alert.conf alert HA_EVENT "(.*)Sync of device group(.*)" { snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.500" } You can use tmsh::log "yes" to log to /var/log/ltm as shown in: iCall script to validate Virtual Server and node with same IP addresses Testing: This can be tested even without HSM as I don't have one at my home using "logger -p" to inject the logs. I have added "yes" in the logs as a test 😎 logger -p local0.notice "010714a0:5: Sync of device group /Common/Failover" cat /var/log/ltm ............ Jul 1 08:34:07 bigip1.com notice root[23762]: yes Jul 1 08:34:53 bigip1.com notice root[23940]: 010714a0:5: Sync of device group /Common/Failover Using Linux bash script: In some versions you can trigger from icall script a bash or sh script with advanced logic inside, for example /bin/sh -c "/var/tmp/ha_script" but I saw issues on 17.1.x triggering from from icall a bash script that in previous versions I solved by adding "HOME=/root <linux bash command>" . In the "What is iCall" it is shown that you can do "if else" or "for" loops inside the iCall script but I find it easy to use bash for advanced logic. Good thing like everything with F5 usually there is more than one way to do things and this case in the user_alert.conf you can actually trigger a bash script from the log messages! What is iCall? | DevCentral iCall script triggers error need ${HOME} to run | DevCentral Running a command or custom script based on a syslog message cat /var/tmp/ha_script #!/bin/bash logger -p local0.notice "yes" /opt/nfast/bin/rfs-sync --update cat /config/user_alert.conf alert HA_EVENT "(.*)Sync of device group(.*)" { exec command="/var/tmp/ha_script" } Extra Notes: Using /opt/nfast/bin/rfs-sync --update or rfs-sync --update depends in some cases on the versions in the iCall script. In the release notes I saw a new bug https://cdn.f5.com/product/bugtracker/ID1429897.html that is solved in the latest 17.1.x versions where if the RFS is on the BIG-IP after a key/cert are created 'rfs-sync -c' needs to be run on the F5 Device that created them as well. The 'rfs-sync -c' can also be automated the way I have shown and my iCall script will work as well for BIG-IP that use external RFS and after the key/cert are created and committed from an F5 device (usually the active one) then an HA config sync needs to be started that will trigger 'rfs-sync -u' on the other F5 devices. Another nice way if you are using something like Ansible for example is to make to trigger the RFS update command on all F5 devices in a cluster as F5 supports bash commands even through API not only CLI. Example: curl -sku admin:XXX https://XXXX/mgmt/tm/util/bash -H "Content-Type: application/json" -X POST -d '{"command":"run", "utilCmdArgs":"-c \"/opt/nfast/bin/rfs-sync --update\""}' https://clouddocs.f5.com/products/orchestration/ansible/devel/f5_bigip/modules_2_0/bigip_command_mod...iCall Script that only runs on Active member
Problem this snippet solves: I had a request to run an iCall script only on the active member in a pair. How to use this snippet: This won't work if you're using active/active via traffic-groups. Code : # Only execute if local BIG-IP is active in failover if {[exec cat /var/prompt/ps1] == "Active"} { tmsh::log "I LIKE SOUP!" } Tested this on version: 12.1
Not working my iCall script...
Problem this snippet solves: Hello~ I created iCall and generate event for working iCall's script. But.... It is not working. list sys icall script change_iRule2 sys icall script change_iRule2 { app-service none definition { tmsh::modify ltm virtual v_198.100__443 rules { vip_snatpool } tmsh::modify ltm virtual v_198.101__443 rules { vip_snatpool } } description none events none } list sys icall handler triggered change_iRule2{ script change_iRule2 subscriptions { change_iRule2 { event-name change_iRule2 } } } I gernerated event generate sys icall event change_iRule2 And I checked bellow status...... Sys::iCall::Event Triggered Handler: change_iRule2 Events matching filters 11 Events causing handler to run 11 Creation time 01/24/17 16:38:35 Current status active Time since last status change 01/24/17 16:38:35 But, Configuration(virtual server's iRule) is not change.... Plz help me. T..T Code : sys icall script change_iRule2 { app-service none definition { tmsh::modify ltm virtual v_198.100__443 rules { vip_snatpool } tmsh::modify ltm virtual v_198.101__443 rules { vip_snatpool } } description none events none } create sys icall handler triggered change_iRule2 script change_iRule2 subscriptions add { change_iRule2 { event-name change_iRule2 } }[icall] kill oldest sessions when reaching xx% of the APM license limit
Problem this snippet solves: When dealing with APM authentication, especially when the F5 device act as a SAML 2.0 IDP, the active sessions can increase considerably and easily reach the max access session limit of the license or the device. The following icall script allows an administrator to guarantee that new users can still authenticate through APM IDP under heavy load. The script will kill oldest active access sessions based on the access session consumption. This is a draft that need to be fine tuned. Warning: when using APM Guest on a vCMP host, we are not able to guarantee that the appliance limit is not reached as we just have knowledge of the active sessions within the guest context only. How to use this snippet: TMSH command to create an icall script create sys icall script apm_purge_sessions Then copy/paste the content of the icall script and save it. By default, the command create a script named "apm_purge_sessions". You can easily change the name of the script by modifying "apm_purge_sessions" in the command line. TMSH command to create the icall handler The following command trigger the script every 60 seconds. It can be changed to increase the frequency of the execution of the script. create sys icall handler periodic f5-apm-purge-session interval 60 script apm_purge_sessions Interesting tcl commands used in the script Retrieve the max_access_session variable in the license of the device: [string trim [lindex [split [exec /usr/bin/tmsh show /sys license detail | grep access] " "] 1] "\[\]"] retrieve the ordered list (oldest first) of active APM sessionIDs catch {set output [exec /usr/bin/sessiondump --allkeys | grep starttime | sort -k3 | cut -c1-8]} Use cases kill oldest sessions when reaching xx% of the APM license limit Evolution trigger the icall script based on a specific event (snmptrap, log, ...) sort APM sessions by Access Profile and kill sessions based on the criticity of each AP. Code : # retrieve the ordered list of active APM sessionIDs catch {set output [exec /usr/bin/sessiondump --allkeys | grep starttime | sort -k3 | cut -c1-8]} if {$output != ""} { # move the output to a list of sessionID set output [split $output "\n"] set count [llength $output] # determine the max_access_session allowed for the running platform set max_access [string trim [lindex [split [exec /usr/bin/tmsh show /sys license detail | grep access] " "] 1] "\[\]"] # determine acceptable threshold before triggering set access_threshold [expr round($max_access*0.85)] set diff [expr $count-$access_threshold] # kill oldest APM sessions until reaching 85% of active sessions in the APM device for {set i 0} {$i < $diff} {incr i} { catch { [exec /usr/bin/sessiondump --delete [lindex $output $i]] } } } Tested this on version: 11.6
Generate qkview or ucs on failover
Problem this snippet solves: This iCall script collects a qkview and a UCS after a failover event. It is hardcoded for traffic-group-1. How to use this snippet: Implementation Details This iCall script requires v11.4 or higher. The script can be loaded via: load sys config merge file /var/tmp/handle-failover.conf Code : sys icall script detect-failover { app-service none definition { # prime the iApp to assume it's standby so that it doesn't generate # a qkview+ucs on the initial run (which wouldn't be a state # transition) set old_status standby while { 1 } { # this will block until another event is present. As we only # subscribe to HA events, that means that whenever an event is # raised it represents a new message from SOD EVENT::get_next set new_status $EVENT::context(/Common/traffic-group-1) # detect if we just went standby if { $new_status eq "standby" && $old_status ne "standby" } { puts "failover detected - i am standby now!" set date [clock format [clock seconds] -format "%Y%m%d%H%M%S"] set settings [tmsh::get_config sys global-settings] set host [tmsh::get_field_value [lindex $settings 0] hostname] puts "generating qkview /var/tmp/$host-$date.qkview" exec /usr/bin/qkview -f /var/tmp/$host-$date.qkview 2> /dev/null & puts "generating UCS /var/local/ucs/$host-$date.ucs" tmsh::save sys ucs /var/local/ucs/$host-$date.ucs } # save the state set old_status $new_status } } description none events none } sys icall handler perpetual handle-failover { script detect-failover subscriptions { failover { event-name FAILOVER_STATE } } }
iCall CRL update with Route Domains and Auto-Sync
Problem this snippet solves: iCall script to update CRL file within F5 BIG-IP when the HTTP request must run from a specific Route Domain and also uses logger to write logs to the default LTM location. The original was to also update an iFile of the CRL file for use within an iRule however I have removed that due to it being a very special case (I may add another snippet later to detail that one). Important point here is we update the CRL file located within a folder (or partition) that was linked to a Sync-Only Device Group with auto-sync enabled e.g. CRL files are created and saved to /Common/ crl / This way the iCall script does not need to trigger any sort sync and the rest of the configuration can be left as manual sync. Code : sys icall handler periodic /Common/someCrl-CrlUpdate { arguments { { name rd value 2 } { name url value https://172.31.0.1/somepath/to/crlUpdateFile.crl } { name host value somecrl.CADomein.com } { name folder value tempCrlDirectory } { name sslCrl value /Common/crl/someCrlFile.crl } } interval 600 script /Common/iCallCrlUpdate } sys icall script /Common/iCallCrlUpdate { app-service none definition { set logTag "iCallCrlUpdate" set logLevel "notice" # Getting handler provided arguments foreach arg { rd url host folder sslCrl ifileCrl } { set $arg $EVENT::context($arg) } # Create a directory to save files to disk set crlDir /var/tmp/$folder exec mkdir -p $crlDir exec /bin/logger -i -t $logTag -p local0.$logLevel "Running, CRL URL=$url, Host=$host, SSL CRL=$sslCrl, iFile CRL=$ifileCrl, Directory=$crlDir, rd=$rd" # Download CRL file from provided route domain (rd) and url arguments and save to temporary directory set status [exec /usr/bin/rdexec $rd /usr/bin/curl-apd -s -o $crlDir/LatestCRL.crl -w %{http_code} -H Host:$host $url] if {$status == 200} { # Update F5 SSL CRL file tmsh::modify sys file ssl-crl $sslCrl source-path file:$crlDir/LatestCRL.crl exec /bin/logger -t $logTag -p local0.$logLevel "F5 CRL files update complete." } else { exec /bin/logger -i -t $logTag -p local0.error "Command /usr/bin/rdexec $rd /usr/bin/curl-apd -s -o $crlDir/LatestCRL.crl -w '%{http_code}' -H 'Host: onsitecrl.trustwise.com' $url, failed with status=$status" } } description none events none } Tested this on version: 12.1Disable Interface if Pool Member Availability Drops Below Threshold
Problem this snippet solves: This iCall script can be used to disable an interface (int 1.3 in this case) if the member availability of a pool drops below a certain threshold (70% in this example.) How to use this snippet: Implementation Details This iCall script requires v11.4 or higher. The script is called by a periodic handler, but could be converted to a triggered handler with some custom work in /config/user_alert.conf on the pool members. Code : ## Script ## sys icall script poolCheck.v1.0.0 { app-service none definition { set pn "/Common/pool4" set total 0 set usable 0 foreach obj [tmsh::get_status /ltm pool $pn detail] { #puts $obj foreach member [tmsh::get_field_value $obj members] { #puts $member incr total if { [tmsh::get_field_value $member status.availability-state] == "available" && \ [tmsh::get_field_value $member status.enabled-state] == "enabled" } { incr usable } } } if { [expr $usable.0 / $total] < 0.7 } { tmsh::log "Not enough pool members in pool $pn, interface 1.3 disabled" tmsh::modify /net interface 1.3 disabled } else { tmsh::log "Enough pool members in pool $pn, interface 1.3 enabled" tmsh::modify /net interface 1.3 enabled } } description none events none } ## Handler ## sys icall handler periodic poolCheck.v1.0.0 { first-occurrence 2014-09-16:11:00:00 interval 60 script poolCheck.v1.0.0 }
Health Score iCall - Dynamic Modification of a Pool Member's Ratio
Problem this snippet solves: Code Use Case: This iApp can be used to control an iCall script that will dynamically modify a pool member's ratio based on the result of a header received from each individual pool member. The iApp exposes a list of typical questions that will in turn modify the iCall scripts behavior without having to dive into the iCall code itself. The most common example and what the original iCall script was written and tested against is SharePoint web servers will return a header called X-SharePointHealthScore with a value ranging from 0-10(0 being best) based on internal monitored metrics. The iApp managed iCall will then look at this value on an interval and apply a ratio to the pool member ranging from 10-1 as the inverse to the score above. This allows for a more in-tune approach to load balancing. Note: This will work against a pool managed by another iApp (tested with SharePoint 2010/2013/2016 iApp) as well as a manually configured pool (or a pool managed by an iApp with strict updates turned off). Link to Original iCall: https://devcentral.f5.com/codeshare/prioritize-sharepoint-nodes-on-reported-health How to use this snippet: Unzip the file, then upload the iApp template to Big-IP. Then create an application service using the template. Code : 67795 Tested this on version: 12.0Logging pool member name (and not just address)
Problem this snippet solves: I wanted to see a pool member's text name, not just the IP address, in logs. We accomplished this by creating an iCall script that routinely created dynamic data-groups from existing pools, then using an iRule to log the pool member's name based on lookup. How to use this snippet: We had this script run periodically as new pool members were not created/added on a regular basis: create sys icall handler periodic create_poolmember_datagroups interval 86400 script create_poolmember_datagroups The iRule would look in "dynpoolmbrdg-[pool_name]" by IP address and log the corresponding member text name. Code : icall script create_poolmember_datagroups { app-service none definition { # Define variable types set poollist [list] set memberlist [list] set datagrouplist [list] # get a list of current data-groups set dglistraw [tmsh::get_config /ltm data-group internal] foreach datagroup $dglistraw { lappend datagrouplist [tmsh::get_name $datagroup] } # process each pool member in the configuration set poollistraw [tmsh::get_config /ltm pool] foreach pool $poollistraw { # retrieve the pool name and current members set poolname [tmsh::get_name $pool] set memberlist [tmsh::get_field_value $pool members] # Create the pool's data group if it doesn't exist set dyndgname dynpoolmbrdg-$poolname if {[lsearch $datagrouplist $dyndgname] >= 0} { } else { tmsh::create ltm data-group internal $dyndgname type string } # Overwrite the data-group with a list of the current members foreach member $memberlist { set membername [tmsh::get_name $member] set memberaddr [tmsh::get_field_value $member "address"] append payload "$memberaddr { data $membername } " set records "{ $payload }" tmsh::modify ltm data-group internal $dyndgname records replace-all-with $records } } } description none events none } Tested this on version: No Version FoundiCall script to "save sys config" only when changes are detected
Problem this snippet solves: This script will check to see if configuration changes have occurred every X seconds, and if so, will issue the "save sys config" command via TMSH. This is helpful for auto-saving functionality, especially when you are using iControl REST in a dynamic environment and don't want to issue multiple "save sys config" commands via the API for each call. How to use this snippet: Create the script and the handler and you're up and running! The interval in the handler is the number of seconds between checks. Code : sys icall script /Common/F5.SaveConfig.OnlyIfModified { app-service none definition { set dbfile_timestamp [exec stat /config/BigDB.dat --format %Z] set conffile_timestamp [exec stat /config/bigip.conf --format %Z] if { $dbfile_timestamp > $conffile_timestamp } { tmsh::save sys config partitions all } } description none events none } sys icall handler periodic /Common/F5.AutoSaveConfigScheduledTask { interval 120 script /Common/F5.SaveConfig.OnlyIfModified } Tested this on version: 12.1
