iCall CRL update with Route Domains and Auto-Sync
Problem this snippet solves: iCall script to update CRL file within F5 BIG-IP when the HTTP request must run from a specific Route Domain and also uses logger to write logs to the default LTM location. The original was to also update an iFile of the CRL file for use within an iRule however I have removed that due to it being a very special case (I may add another snippet later to detail that one). Important point here is we update the CRL file located within a folder (or partition) that was linked to a Sync-Only Device Group with auto-sync enabled e.g. CRL files are created and saved to /Common/ crl / This way the iCall script does not need to trigger any sort sync and the rest of the configuration can be left as manual sync. Code : sys icall handler periodic /Common/someCrl-CrlUpdate { arguments { { name rd value 2 } { name url value https://172.31.0.1/somepath/to/crlUpdateFile.crl } { name host value somecrl.CADomein.com } { name folder value tempCrlDirectory } { name sslCrl value /Common/crl/someCrlFile.crl } } interval 600 script /Common/iCallCrlUpdate } sys icall script /Common/iCallCrlUpdate { app-service none definition { set logTag "iCallCrlUpdate" set logLevel "notice" # Getting handler provided arguments foreach arg { rd url host folder sslCrl ifileCrl } { set $arg $EVENT::context($arg) } # Create a directory to save files to disk set crlDir /var/tmp/$folder exec mkdir -p $crlDir exec /bin/logger -i -t $logTag -p local0.$logLevel "Running, CRL URL=$url, Host=$host, SSL CRL=$sslCrl, iFile CRL=$ifileCrl, Directory=$crlDir, rd=$rd" # Download CRL file from provided route domain (rd) and url arguments and save to temporary directory set status [exec /usr/bin/rdexec $rd /usr/bin/curl-apd -s -o $crlDir/LatestCRL.crl -w %{http_code} -H Host:$host $url] if {$status == 200} { # Update F5 SSL CRL file tmsh::modify sys file ssl-crl $sslCrl source-path file:$crlDir/LatestCRL.crl exec /bin/logger -t $logTag -p local0.$logLevel "F5 CRL files update complete." } else { exec /bin/logger -i -t $logTag -p local0.error "Command /usr/bin/rdexec $rd /usr/bin/curl-apd -s -o $crlDir/LatestCRL.crl -w '%{http_code}' -H 'Host: onsitecrl.trustwise.com' $url, failed with status=$status" } } description none events none } Tested this on version: 12.1F5 BIG-IP and ENTRUST nShield HSM SSL key/cert auto synchronization between HA peers with iCall
Code version: The code was tested on 15.1.8. Main Article: For more information about RFS and Client agent I suggest seeing the vendors article. https://nshielddocs.entrust.com/security-world-docs/v13.3/connect-ug-nix/intro.html Useful F5 links for F5 and nShield Integration for GTM and LTM: https://my.f5.com/manage/s/article/K000135349 https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-system-and-nshield-hsm-implementation/setting-up-t... The nShield architecture includes a component called the Remote File System (RFS) that stores and manages the encrypted key files. The RFS can be installed on the BIG-IP system or on another server on your network. Basically the HSM agent/client is installed on the F5 devices hos Linux host system and the F5 devices are also the RFS servers. The RFS commands are bellow as when installed on the BIG-IP the HSM agent and RFS they are available for use: https://nshielddocs.entrust.com/security-world-docs/utilities/rfs-sync.html The issue I solved with iCall script is that when when you create a new HSM key for BIG-IP HA, you must run command ‘rfs-sync --update’ on all standby BIG-IP devices (the devices where the cert/key were not created or changed) to update the local Thales encrypted file object cache. Without this action, SSL traffic using this key will fail when BIG-IP fails over to one of the unsynced standby devices. When you create the the key and cert on the active F5 device "rfs-sync -commit" and "rfs-sync -update" run automatically on it but not the "rfs-sync -update" on the standby devices and the icall script basically is triggered on the standby devices when you run the normal config sync. The iCall script matched an event called "HA_EVENT" that is configured in the custom alarms section and triggers the full command with the path "/opt/nfast/bin/rfs-sync --update" to check if there was an update in the rfs. I suggest reading the links below that explain the iCall (one is from JRahm ) and the HA logs and the last one is mine that is from the time before I learned proper article formatting 😅 and it also shows how to run scripts not only with iCall but also during HA events and so on. Run tcpdump on event | DevCentral What is iCall? | DevCentral https://my.f5.com/manage/s/article/K34291400 https://my.f5.com/manage/s/article/K3727 https://my.f5.com/manage/s/article/K11127 Knowledge sharing: Ways to trigger and schedule scripts on the F5 BIG-IP devices. | DevCentral tmsh list sys icall sys icall handler triggered ha-handler { script ha-script subscriptions { ha-subscription { event-name HA_EVENT } } } sys icall script ha-script { app-service none definition { exec /bin/bash -c "logger -p local0.notice 'yes'" exec /bin/bash -c "/opt/nfast/bin/rfs-sync --update" } description none events none } cat /config/user_alert.conf alert HA_EVENT "(.*)Sync of device group(.*)" { snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.500" } You can use tmsh::log "yes" to log to /var/log/ltm as shown in: iCall script to validate Virtual Server and node with same IP addresses Testing: This can be tested even without HSM as I don't have one at my home using "logger -p" to inject the logs. I have added "yes" in the logs as a test 😎 logger -p local0.notice "010714a0:5: Sync of device group /Common/Failover" cat /var/log/ltm ............ Jul 1 08:34:07 bigip1.com notice root[23762]: yes Jul 1 08:34:53 bigip1.com notice root[23940]: 010714a0:5: Sync of device group /Common/Failover Using Linux bash script: In some versions you can trigger from icall script a bash or sh script with advanced logic inside, for example /bin/sh -c "/var/tmp/ha_script" but I saw issues on 17.1.x triggering from from icall a bash script that in previous versions I solved by adding "HOME=/root <linux bash command>" . In the "What is iCall" it is shown that you can do "if else" or "for" loops inside the iCall script but I find it easy to use bash for advanced logic. Good thing like everything with F5 usually there is more than one way to do things and this case in the user_alert.conf you can actually trigger a bash script from the log messages! What is iCall? | DevCentral iCall script triggers error need ${HOME} to run | DevCentral Running a command or custom script based on a syslog message cat /var/tmp/ha_script #!/bin/bash logger -p local0.notice "yes" /opt/nfast/bin/rfs-sync --update cat /config/user_alert.conf alert HA_EVENT "(.*)Sync of device group(.*)" { exec command="/var/tmp/ha_script" } Extra Notes: Using /opt/nfast/bin/rfs-sync --update or rfs-sync --update depends in some cases on the versions in the iCall script. In the release notes I saw a new bug https://cdn.f5.com/product/bugtracker/ID1429897.html that is solved in the latest 17.1.x versions where if the RFS is on the BIG-IP after a key/cert are created 'rfs-sync -c' needs to be run on the F5 Device that created them as well. The 'rfs-sync -c' can also be automated the way I have shown and my iCall script will work as well for BIG-IP that use external RFS and after the key/cert are created and committed from an F5 device (usually the active one) then an HA config sync needs to be started that will trigger 'rfs-sync -u' on the other F5 devices. Another nice way if you are using something like Ansible for example is to make to trigger the RFS update command on all F5 devices in a cluster as F5 supports bash commands even through API not only CLI. Example: curl -sku admin:XXX https://XXXX/mgmt/tm/util/bash -H "Content-Type: application/json" -X POST -d '{"command":"run", "utilCmdArgs":"-c \"/opt/nfast/bin/rfs-sync --update\""}' https://clouddocs.f5.com/products/orchestration/ansible/devel/f5_bigip/modules_2_0/bigip_command_mod...
Sample Linux script to update CRL file from Certificate Authority
Problem this snippet solves: CRL files are signed lists of revoked serial numbers issued by a specific Certificate Authority (Verisign, Godaddy, GlobalSign, etc). There are several advanced methods of dealing with revoked certificates, the best of which is OCSP stapling. Other methods are OCSP responders or CRL Distribution Points. However, for small or internal projects, some administrators rely on simply using straight-up CRL files. After a time, the administrator will realize he or she needs to automate this process with a script. The administrator can automate this process on the BIG-IP itself with an iCall script. Here's a link to a great example of the iCall solution. However, some administrators many need to use a straight-up Linux device to pull and copy the CRL files around to many different devices, only one of which is the BIG-IP. How to use this snippet: This is a sample of a Linux script that pulls down a CRL file from GoDaddy, verifies it and then copies it to BIG-IP. Ensure that the Linux device can copy files directly to the BIG-IP via ssh-key authentication. Modify the 'f5' variable of the script to point to the BIG-IP. If not using GoDaddy, find the URL of the CRL file for the appropriate CA. If the 'ssl-crl' object hasn't been created on the BIG-IP yet, then it must be done manually the first time. Download the CRL, copy it to the BIG-IP's /var/tmp area. Then login to the BIG-IP and issue the following command: tmsh modify sys file ssl-crl gdcrl source-path file:/var/tmp/CRL After that, the script should work. Code : #!/bin/bash # # script to download a CRL from GoDaddy CA. # See this page for GoDaddy CRL information: # https://certs.godaddy.com/repository # Verify CRL # Convert CRL # Copy to BIG-IP # exit on error to prevent copying corrupt CRL to BIG-IP set -e f5=yourbigip.com f5port=22 crlurl=https://certs.godaddy.com/repository/mastergodaddy2issuing.crl gdcrturl=https://certs.godaddy.com/repository/gdig2.crt gdcrt=gdig2.crt echo "Automated CRL update Script" echo "Downloading from $crlurl" echo "Copying to ${f5}:${f5port}" echo "Last line should be SUCCESS" echo "---- GO ----" if [ ! -f $gdcrt ]; then echo "Fetching GoDaddy Certificate" curl $gdcrturl > $gdcrt fi cf=gdroot.crl last=last.crl if [ -f $cf ]; then if [ ! -s $cf ]; then echo "Found existing zero-length CRL file, deleting" rm -f $cf else echo "Found existing $cf - moving to backup" mv $cf $last fi fi echo "Downloading CRL file $cf" curl $crlurl > $cf echo "Testing $cf for readability" test -f $cf test -r $cf echo "Testing to see if $cf is zero length" test -s $cf if [ -f $last ]; then echo "Testing if $cf is newer than backup file $last" if [ ! $cf -nt $last ]; then echo "File not changed. SUCCESS" fi fi echo "Verifying CRL against certificate, converting to PEM format" openssl crl -inform DER -CAfile $gdcrt -in $cf -outform PEM -out ${cf}.pem echo "Testing PEM for zero length" test -s ${cf}.pem echo "Copying CRL file to bigip" scp -P ${f5port} ${cf}.pem root@${f5}:/var/tmp echo "Importing CRL into system" echo " this may fail if object was never created - replace 'modify' with 'create'" ssh root@${f5} -p ${f5port} "tmsh modify sys file ssl-crl gdcrl source-path file:/var/tmp/${cf}.pem" echo "SUCCESS"
Run tcpdump on event
Problem this snippet solves: This example shows how to trigger an iCall script using a log message as the trigger. In particular, it shows how to watch for messages about pool members failing their monitors, and then running tcpdump to capture some packets sent to that pool member in order to help troubleshoot later. How to use this snippet: Implementation Details This iCall script requires v11.4 or higher. Code : 1. create the following iCall script: modify script tcpdump { app-service none definition { set date [clock format [clock seconds] -format "%Y%m%d%H%M%S"] foreach var { ip port count vlan } { set $var $EVENT::context($var) } exec tcpdump -ni $vlan -s0 -w /var/tmp/${ip}_${port}-${date}.pcap -c $count host $ip and port $port } description none events none } 2. create an event handler for it that looks like the following: sys icall handler triggered tcpdump { script tcpdump subscriptions { tcpdump { event-name tcpdump } } } 3. modify /config/user_alert.conf to contain entries that looks like the following: alert local-http-10-2-80-1-80-DOWN "Pool /Common/my_pool member /Common/10.2.80.1:80 monitor status down" { exec command="tmsh generate sys icall event tcpdump context { { name ip value 10.2.80.1 } { name port value 80 } { name vlan value internal } { name count value 20 } }" } You will need one of these stanzas for each pool member that you want to monitor in this method - or to modify the example to act on different events and different objects.
