Automate Data Group updates on many Big-IP devices using Big-IQ or Ansible or Terraform
Problem this snippet solves: In many cases generated bad ip address lists by a SIEM (ELK, Splunk, IBM QRADAR) need to be uploaded to F5 for to be blocked but the BIG-IQ can't be used to send data group changes to the F5 devices. 1.A workaround to use the BIG-IQ script option to make all the F5 devices to check a file on a source server and to update the information in the external data group. I hope F5 to add the option to BIG-IQ to schedule when the scrpts to be run otherwise a cron job on the BIG-IQ may trigger the script feature that will execute the data group to refresh its data (sounds like the Matrix). https://clouddocs.f5.com/training/community/big-iq-cloud-edition/html/class5/module1/lab6.html Example command to run in the BIG-IQ script feature: tmsh modify sys file data-group ban_ip type ip source-pathhttps://x.x.x.x/files/bad_ip.txt https://support.f5.com/csp/article/K17523 2.You can also set the command with cronjob on the BIG-IP devices if you don't have BIG-IQ as you just need Linux server to host the data group files. 3.Also without BIG-IQ Ansible playbook can be used to manage many groups on the F5 devices as I have added the ansible playbook code below. Now with the windows subsystem you can run Ansible on Windows! 4.If you have AFM then you can use custom feed lists to upload the external data without the need for Ansible or Big-IQ. The ASM supports IP intelligence but no custom feeds can be used: https://techdocs.f5.com/kb/en-us/products/big-ip-afm/manuals/product/big-ip-afm-getting-started-14-1-0/04.html How to use this snippet: I made my code reading: https://docs.ansible.com/ansible/latest/collections/f5networks/f5_modules/bigip_data_group_module.html https://support.f5.com/csp/article/K42420223 If you want to have an automatic timeout then you need to use the irule table command (but you can't edit that with REST-API, so see the article below as a workaround) that writes in the RAM memory that supports automatic timeout and life time for each entry then there is a nice article for that as I added comment about possible bug resolution, so read the comments! https://devcentral.f5.com/s/articles/populating-tables-with-csv-data-via-sideband-connections Another way is on the server where you save the data group info is to add a bash script that with cronjob deletes from time to time old entries. For example (I tested this). Just write each data group line/text entry with for example IP address and next to it the date it was added. cutoff=$(date -d 'now - 30 days' '+%Y-%m-%d') awk -v cutoff="$cutoff" '$2 >= cutoff { print }' <in.txt >out.txt && mv out.txt in.txt Ansible is a great automation tool that makes changes only when the configuration is modified, so even if you run the same playbook 2 times (a playbook is the main config file and it contains many tasks), the second time there will be nothing (the same is true for terraform). Ansible supports "for" loops but calls them "loop" (before time " with_items " was used) and "if else" conditions but it calls them "when" just to confuse us and the conditions and loops are placed at the end of the task not at the start 😀 A loop is good if you want to apply the same config to multiple devices with some variables just being changed and "when" is nice for example to apply different tasks to different versions of the F5 TMOS or F5 devices with different provisioned modules. https://stackoverflow.com/questions/38571524/remove-line-in-text-file-with-bash-if-the-date-is-older-than-30-days Code : --- - name: Create or modify data group hosts: all connection: local vars: provider: password: xxxxx server: x.x.x.x user: xxxxx validate_certs: no server_port: 443 tasks: - name: Create a data group of IP addresses from a file bigip_data_group: name: block_group records_src: /var/www/files/bad.txt type: address provider: "{{ provider }}" notify: - Save the running configuration to disk handlers: - name: Save the running configuration to disk bigip_config: save: yes provider: "{{ provider }}" The "notify" triggers the handler task after the main task is done as there is no point in saving the config before that and the handler runs only on change, Tested this on version: 15.1 Also now F5 has Terraform Provider and together with Visual Studio you can edit your code on Windows and deploy it from the Visual Studio itself! Visual Studio wil even open for you the teminal, where you can select the folder where the terraform code will be saved after you have added the code run terraform init, terraform plan, terraform apply. VS even has a plugin for writting F5 irules.Terraform's files are called "tf" and the terraform providers are like the ansible inventory file (ansible may also have a provider object in the playbook not the inventory file) and are used to make the connection and then to create the resources (like ansible tasks). Usefull links for Visual Studio and Terraform: https://registry.terraform.io/providers/F5Networks/bigip/1.16.0/docs/resources/bigip_ltm_datagroup https://www.youtube.com/watch?v=Z5xG8HLwIh4 For more advanced terafform stuff like for loops and if or count conditions: https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9 Code : You may need to add also this resource below as to save the config and with "depends_on" it wil run after the date group is created. This is like the handler in Ansible that is started after the task is done and also terraform sometimes creates resources at the same time not like Ansible task after task, resource "bigip_command" "save-config" { commands = ["save sys config"] depends_on = [ bigip_ltm_datagroup.terraform-external1 ] } Tested this on version: 16.1 Ansible and Terraform now can be used for AS3 deployments like the BIG-IQ's "applications" as they will push the F5 declarative templates to the F5 device and nowadays even the F5 AWAF/ASM and SSLO (ssl orchestrator) support declarative configurations. For more info: https://www.f5.com/company/blog/f5-as3-and-red-hat-ansible-automation https://clouddocs.f5.com/products/orchestration/ansible/devel/f5_bigip/playbook_tutorial.html https://clouddocs.f5.com/products/orchestration/terraform/latest/userguide/as3-integration.html https://support.f5.com/csp/article/K23449665 https://clouddocs.f5.com/training/fas-ansible-workshop-101/3.3-as3-asm.html https://www.youtube.com/watch?v=Ecua-WRGyJc&t=105s
Code to create unreachable ELA license files from BIG-IQ
Problem this snippet solves: *NOTE* if you are upgrading your BIG-IP,please refer to F5 solution:https://support.f5.com/csp/article/K13540950 BIG-IQ traditionally expects to be able to reach any BIG-IP devices it is going to license. This code helps create a license file from the ELA SKU offerings which can be applied on an Unreachable BIG-IP. I've added some troubleshoting steps at the end of the article, Dossier errors seen on the BIG-IP, just in case! How to use this snippet: SSH into the BIG-IP device and run the following command to gain the MAC address of the management interface tmsh show sys mac-address | grep -i interface [root@bigip1:Active:Standalone] config # tmsh show sys mac-address | grep -i interface ll:50:56:xx:xx:36net interfacemgmtmac-address xxxxxxxxxxxxxxxxxnet interface1.3mac-address xxxxxxxxxxxxxxxxxnet interface1.1mac-address xxxxxxxxxxxxxxxxx net interface1.2mac-address In the example above the MAC address we need is “ll:50:56:xx:xx:36” Now SSH into the BIG-IQ Move into the /shared directory (cd /shared) Copy over the Create-license.PY python script and run it by typing python Create-license.py The script runs and will prompt you for the following information [root@Preece-bigiq-cm1:Active:Standalone] shared # python Create-license.py Enter BIG-IQ user ID: admin Enter BIG-IQ Password: Enter Management IP address of BIG-IQ: 44.131.176.101 Enter Management IP address of BIG-IP to be licensed: 44.131.176.22 Enter Management MAC address of BIG-IP to be licensed: ll:50:56:xx:kk:36 Enter the name of the License Pool from which to take BIG-IP license: Load-18 Enter the license name to be assigned to the BIG-IP: F5-BIG-MSP-BT-1GIPIF-LIC-DEV Enter hypervisor used, valid options are: aws, azure, gce, hyperv, kvm, vmware,xen: vmware Optional: Enter chargeback tag if required: Department-A Optional: Enter tenant name if required: Customer-B Once the details have been filled in the script authenticates to the BIG-IQ and generates the license (30 seconds) If everything went well, you will be presented with a success message. The license file is saved as IP-address_bigip.license in the same directory as you run the script Using SCP copy the new license file from the BIG-IQ to your desktop. Copy the license file into the /config directory of the BIG-IP device. Rename the file, copy ip-address.bigip.license bigip.license Reload the license by typing reloadlic Observe the BIG-IP device restart its services and show as active. You can review in the GUI (System—License) and provision modules as needed. Code : import getpass # used to hide the users password input import json import os import requests from time import sleep """ This script uses the BIG-IQ API to license an unreachable (dark site) BIG-IP. The BIG-IQ licensing API needs certain details provided in order to license an appliance, these details can either be provided in a file call lic-data.json or if that file does not exist you will be prompted to enter them. The minimum contents of lic-data.json should be: { "licensePoolName": " -- Enter License Pool Name here. License Pool name can be found in BIG-IQ GUI -- ", "command": "assign", "address": " -- Enter MGMT IP Address of BIG-IP here -- ", "assignmentType": "UNREACHABLE", "macAddress": " -- Enter MAC address of MGMT IP for the BIG-IP here -- ", "hypervisor": " -- Enter hypervisor value here options are; aws, azure, gce, hyperv, kvm, vmware, xen: --", "unitOfMeasure": "yearly", "skuKeyword1": "-- Enter License Name here. License Name (or Offering name) can be found in the BIG-IQ GUI -- " } Additional Optional key:value pairs can be added to the JSON file to afix useful tags to the license. The json file with optional key:value pairs looks like: { "licensePoolName": " -- Enter License Pool Name here. License Pool name can be found in BIG-IQ GUI -- ", "command": "assign", "address": " -- Enter MGMT IP Address of BIG-IP here -- ", "assignmentType": "UNREACHABLE", "macAddress": " -- Enter MAC address of MGMT IP for the BIG-IP here -- ", "hypervisor": " -- Enter hypervisor value here options are; aws, azure, gce, hyperv, kvm, vmware, xen: --", "unitOfMeasure": "yearly", "skuKeyword1": "-- Enter License Name here. License Name (or Offering name) can be found in the BIG-IQ GUI -- ", "chargebackTag": "OPTIONAL: Remove this line if you are not going to use it", "tenant": "OPTIONAL: Remove this line if you are not going to use it" } A completed minimal lic-data.json file will look like this: { "licensePoolName": "byol-pool-utility", "command": "assign", "address": "10.1.1.10", "assignmentType": "UNREACHABLE", "macAddress": "06:ce:c2:43:b3:05", "hypervisor": "kvm", "unitOfMeasure": "yearly", "skuKeyword1": "F5-BIG-MSP-BT-P3-3GF-LIC-DEV" } lic-data.json must reside in the directory from which you execute this python script. """ def bigiqAuth(_bigiqAuthUrl, _bigiqCredentials): """ This function authenticates with BIG-IQ and collects the authentication token provided. Theo token will be used for subsequent calls to BIG-IQ """ _errFlag=0 try: _bigiqAuthInfo=_bigiq_session.post(_bigiqAuthUrl, data=json.dumps(_bigiqCredentials), verify=False) print(_bigiqAuthUrl) _bigiqAuthInfo.raise_for_status() print("Response code: %s" %_bigiqAuthInfo.status_code) except requests.exceptions.HTTPError as err: print(err) _errFlag=1 #end try if _errFlag==0: _bigiqResponse=_bigiqAuthInfo.json() _bigiqToken=_bigiqResponse['token'] for _token in _bigiqToken: if (_token == 'token'): _bigiqAuthToken=(_bigiqToken[_token]) # End if # Next _authHeaders={ "X-F5-Auth-Token": "{_authToken}".format(_authToken=_bigiqAuthToken) } else: _authHeaders=0 #end if print("** Completed Authentication ***") return(_authHeaders); #End Def def extractLicense(_rawLicenseJSON): """ This function pulls the generated license from BIG-IQ """ for _license in _rawLicenseJSON: if (_license=='licenseText'): _extractedLicense=_rawLicenseJSON[_license] #end if if (_license=='status'): if (_rawLicenseJSON[_license]=="FINISHED"): print("***** License has been assigned *****") else: _extractedLicense="FAILED" #end if #end if #next return(_extractedLicense); #End def def licenseData(): """ This function read the lic-data.json file. If it does not exist you will be prompted to enter the necessary values. """ if os.path.exists('lic-data.json'): with open('./lic-data.json') as licfile: _licdata = json.load(licfile) else: _bigipAddress=raw_input("Enter Management IP address of BIG-IP to be licensed: ") _bigipMACaddress=raw_input("Enter Management MAC address of BIG-IP to be licensed: ") _licensePoolName=raw_input("Enter the name of the License Pool from which to take BIG-IP license: ") _licenseSKU=raw_input("Enter the license name to be assigned to the BIG-IP: ") _hypervisorType=raw_input("Enter hypervisor used, valid options are: aws, azure, gce, hyperv, kvm, vmware, xen: ") _chargebackTag=raw_input("Optional: Enter chargeback tag if required: ") _tenantTag=raw_input("Optional: Enter tenant name if required: ") _licdata={ "licensePoolName": "{_licensePool}".format(_licensePool=_licensePoolName), "command": "assign", "address": "{_bigipIP}".format(_bigipIP=_bigipAddress), "assignmentType": "UNREACHABLE", "macAddress": "{_bigipMAC}".format(_bigipMAC=_bigipMACaddress), "hypervisor": "{_hypervisor}".format(_hypervisor=_hypervisorType), "unitOfMeasure": "yearly", "skuKeyword1": "{_license}".format(_license=_licenseSKU), "chargebackTag": "{_chargeback}".format(_chargeback=_chargebackTag), "tenant": "{_tenant}".format(_tenant=_tenantTag) } # End if return(_licdata); def urlConstruction(_bigiqUrl, _bigiqIP): """ This function rewrites the selflink URL returned by BIG-IQ to reflect BIG-IQ management IP address rather than localhost """ count=0 _urlDeConstruct=_bigiqUrl.split("/") _urlReConstruct="" for _urlElement in _urlDeConstruct: #print("%d %s" %(count,_urlElement)) if (_urlElement=="https:"): _urlReConstruct=_urlReConstruct+_urlElement+"//" elif (_urlElement=="localhost"): _urlReConstruct=_urlReConstruct+_bigiqIP else: if (_urlElement!=""): _urlReConstruct=_urlReConstruct+"/"+_urlElement #end if #end if count+=1 #Next return(_urlReConstruct); #End Def _userID=raw_input("Enter BIG-IQ user ID: ") _password=getpass.getpass(prompt="Enter BIG-IQ Password: ") _bigiqAddress=raw_input("Enter Management IP address of BIG-IQ: ") _credPostBody={ "username": "{_uname}".format(_uname=_userID), "password": "{_pword}".format(_pword=_password), "loginProvideriName": "RadiusServer" } _deviceToBeLicensed=licenseData() _bigipAddress=_deviceToBeLicensed['address'] print("BIG-IP Address is: %s" %_bigipAddress) _bigiq_session=requests.session() _bigiq_auth_url="https://{_bigiqIP}/mgmt/shared/authn/login".format(_bigiqIP=_bigiqAddress) # Authenticates with BIG-IQ _bigiqAuthHeader=bigiqAuth(_bigiq_auth_url, _credPostBody) # if _bigiqAuthHeader==0: print("Unable to authenticate with BIG-IQ. Check BIG-IQ reachability and credentials") else: _bigiq_url1="https://{_bigiqIP}/mgmt/cm/device/tasks/licensing/pool/member-management".format(_bigiqIP=_bigiqAddress) # # --- This section requests the license from BIG-IQ. Posting the criteria as laid out in the _deviceToBeLicensed JSON blob # _errFlag=0 try: _bigiqLicenseDevice=_bigiq_session.post(_bigiq_url1, headers=_bigiqAuthHeader, data=json.dumps(_deviceToBeLicensed), verify=False) _bigiqLicenseDevice.raise_for_status() print("Response code: %s" %_bigiqLicenseDevice.status_code) except requests.exceptions.HTTPError as err: print("Issue received, check rquest and or check connectivity %s" %err) _errFlag=1 #end try if _errFlag==0: #print(_bigiqLicenseDevice.status_code) _bigiqResponse=_bigiqLicenseDevice.json() print(_bigiqResponse) print(_bigiqResponse['selfLink']) _bigiqLicenseStatus_url=_bigiqResponse['selfLink'] _bigiqLicenseStatus_url=urlConstruction(_bigiqLicenseStatus_url, _bigiqAddress) print(_bigiqLicenseStatus_url) print("--- Standby for 30 seconds whilst BIG-IQ generates license ---") sleep(30) _errFlag1=0 try: _licenseStatus=_bigiq_session.get(_bigiqLicenseStatus_url, headers=_bigiqAuthHeader, verify=False) _licenseStatus.raise_for_status() print("Response code: %s" %_licenseStatus.status_code) except requests.exceptions.HTTPError as err: print("Issue received, check rquest and or check connectivity %s" %err) _errFlag=1 #end try if _errFlag==0: print(_licenseStatus.content) _licenseStatusDetail=_licenseStatus.json() _licenseOutput=extractLicense(_licenseStatusDetail) if (_licenseOutput=="FAILED"): print("***** License Assignment Failed. Most likely a valid license already exists for device, revoke it before applying a new license *****") else: _licenseFname=(_bigipAddress+"_bigip.license") _licensefile=open(_licenseFname, "w") _licensefile.write("%s" %_licenseOutput) _licensefile.close() print(_licenseOutput) print("***** SUCCESS, the license is stored here %s *****" %_licenseFname) #end if #end if #end if #end if Tested this on version: 13.x, 14.x, 15.x and 16.x Troubleshooting When you apply the license to the BIG-IP you may see an error similar to: License is not operational (expired or digital signature does not match contents) This could simply be that you copy and paste the license file badly, please use MD5SUM on the BIG-IQ to the output license file and compare to the same file on the BIG-IP Example: md5sum 10.2.3.4_bigip.license You can also review the /var/log/ltm file for "Dossier error" messages Dossier error: 1 (MAC address is mismatched) Dossier error: 12 (Hypervisor is mismatched) If this does not help, please open a support case and attach a recent qkview file.
Big-IQ bulk trust, discovery and import of Big-IP using REST API
Problem this snippet solves: Attached is a link to github which provides the user with an comprehensive example of how to discover and import many BIGIP device via BIGIQ CM REST API. Script bulkDiscovery.pl is a standalone script installed directly in the BIGIQ shell. Suggested recommendations: 1. Create a /shared/scripts/. directory 2. scp file to BIGIQ, 3. Usage below. This automation will invoke a device trust task to negotiate certificate, discover device to population in resolver groups (maintained per module) and import configuration of BIGIP's as defined in bulk_discovery.csv file. This happens sequentially and is very useful when administrator's goal is to discover and import many BIGIP devices in a programmatic manner. ** tested with perl distribution present on bigiq v5.8.8 How to use this snippet: Usage: ./bulkDiscovery -c bulk_discovery.csv Program: bulkDiscovery.pl Version: v2.00.00 ##### Discover multiple BIG-IP devices. -r Root credentials for every BIG-IP (such as root:default) - overrides root creds in CSV -a Admin credentials for every BIG-IP (such as admin:admin) - overrides any creds in CSV -v Verbose screen output -s Discover ASM -l Discover LTM -p Discover APM -c Path to CSV file with all BIG-IP devices - REQUIRED -u Update framework if needed -h Help -k Keep the CSV file after this finishes (not recommended if it contains creds) -q BIG-IQ admin credentials in form admin:password - REQUIRED if not using default -g access group name if needed -f Discover AFM csv format: ip, user, pw, cluster-name, framework-action, root-user, root-pw ip: ip address of the BigIP to discover. user, pw: username & password of the BigIP. Will be overridden if -a is specified on the command line. configuration csv example format: 1.2.3.4 1.2.3.4, admin, pw 1.2.3.4, admin, pw, ha-name 1.2.3.4,,, ha-name 1.2.3.4, admin, pw,, skip 1.2.3.4, admin, pw,, update, root, root-pw Code : https://github.com/carldubois/bigiq-cm-restapi-bulk Tested this on version: 12.0
as3 Python module
Problem this snippet solves: ThisisapythonmoduletosimplifyusingtheF5NetworksAS3utility. Downloads, installs and uninstalls the AS3 package https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/installation.html How to use this snippet: Installation Installusingpip: pipinstallas3 Example #!/usr/bin/envpython importas3 t=as3.as3(host='1.1.1.1',username='admin',password='admin') #CheckwhetherAS3isinstalled: print(str(t.isInstalled())) #DownloadthelatestAS3versionfromGithub print(str(t.retrieveVersion())) #Installaspecificversiononadifferenthost-ifyouleaveoutfilenameitwilldownloadthelatest t.installAS3(host='2.2.2.2',username='admin',password='admin',filename='f5-appsvcs-3.16.0-6.noarch.rpm') #Uninstallit t.uninstallAS3(host='2.2.2.2',username='admin',password='admin') Methods as3([debug,host,username,password,port,usetoken])-initialiseanAS3object isInstalled([version,host,username,password,usetoken,port])-CheckswhetherAS3isinstalled.Returnsversiondict,TrueorFalse retrieveVersion([release])-DownloadsaspecificreleaseorthelatestreleaseoftheRPMpackage installAS3([version,filename,host,username,password,usetoken,port])-InstallsAS3asapackage.ReturnsTrueorFalse uninstallAS3([version,filename,host,username,password,usetoken,port])-UninstallscurrentAS3package.ReturnsTrueorFalse github(url,[method,data,useragent,stream])-thisisahelpertoretrievefromgithubF5repository.ReturnTrueorFalse versionToId(version)-ReturnsaGithubobjectIDrelatedtoaversionnumber.egversionis'v3.16.0'andIDis22093972 Tested this on version: 13.0Create or edit "Chargeback Tag" in ELA license
Problem this snippet solves: You are using BIG-IQ Centralized Manager or BIG-IQ License Manager to manage your ELA licenses. The Chargeback Tag is visible on the customer usage reports, and can be useful if you bill across multiple departments within the account. When you first provision a license to a BIG-IP, you have the option to add a free text entry into a field call "Chargeback Tag". However, if you missed this or want to change this tag at a later time, it is currently not possible to add or edit the entry via the BIG-IQ GUI. How to use this snippet: You will need to copy the bash code below and then create a simple CSV file including details of the BIG-IP’s IP Address or MachineID and corresponding Chargeback Tag text you want to create or edit into the /shared directory of BIG-IQ. You will then run the script, which will confirm "Updating" followed by each IP address or MachineID and corresponding Chargeback Tag text. The Chargeback Tag can be seen in the BIG-IQ GUI: Devices >> License Management >> Licenses >> select ELA name >> Select Offering Name of the license type. The Chargeback Tag can also be seen in the reports created and shared each quarter by the Customer Success team. In the /shared directory of BIG-IQ, create a CSV file named update.csv and enter the value pair on a line for each BIG-IP device tag to chage using the format: <BIG-IP IP address1>,<new tag text> <BIG-IP IP address2>,<new tag text> Or <BIG-IP MachineID 1>,<new tag text> <BIG-IP MachineID 2>,<new tag text> For example: cd /shared vi update.csv Enter values: 10.11.10.43,new tag value 10.23.12.100, new tag value Or xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,new tag value yyyyyyy-yyyy-yyy-yyy-yyyyyyyyyyyyy,new tag value Save the file. Please note that no inverted commas are needed, and spaces are accepted within the Chargeback Tag value Create a file named update.sh in the same directory using your favourite editor, copy and paste the provided code below and make this file executable with: chmod u+x update.sh Run the script by entering ./update.sh If everything runs correctly, the output showing: Updating <ip address> to 'new tag' Or Updating <MachineId> to 'new tag' will be shown for each entry within the update.csv file Code : > /var/tmp/offerings > /var/tmp/lic-audit curl -s localhost:8100/cm/device/licensing/pool/utility/licenses/ | jq .items[].regKey -r | while read -r regkey ; do curl -s localhost:8100/cm/device/licensing/pool/utility/licenses/$regkey/offerings | jq .items[].id -r | while read -r off ; do curl -s localhost:8100/cm/device/licensing/pool/utility/licenses/$regkey/offerings/$off/members/ | jq .items[].id -r | while read -r id ; do curl -s localhost:8100/cm/device/licensing/pool/utility/licenses/$regkey/offerings/$off/members/$id | jq '.|{add:.deviceAddress,selfLink:.selfLink,deviceMachineId:.deviceMachineId}' -c >> /var/tmp/offerings done done done curl -s localhost:8100/cm/device/licensing/audit/ | jq '.items[]|{add:.address,id:.id,stat:.status,deviceMachineId:.machineId}' -c | grep GRAN >> /var/tmp/lic-audit cat update.csv | while read -r line ; do IP=$(echo $line | cut -d ',' -f1) VAL=$(echo $line | cut -d ',' -f2) echo "Updating $IP to '$VAL'" auditid=$(grep $IP /var/tmp/lic-audit | head -n1 | jq .id -r ) curl -s localhost:8100/cm/device/licensing/audit/$auditid | jq 'del(.generation,.lastUpdateMicros,.chargebackTag)' | jq --arg chargebackTag "$VAL" '. + {chargebackTag: $chargebackTag}' > /var/tmp/update-lic-audit curl -s localhost:8100/cm/device/licensing/audit/$auditid -XPUT -d @/var/tmp/update-lic-audit -o /dev/null offeringid=$(grep $IP /var/tmp/offerings | jq .selfLink -r | egrep -o '/cm/device/.*') curl -s "localhost:8100$offeringid" | jq 'del(.generation,.lastUpdateMicros,.chargebackTag)' | jq --arg chargebackTag "$VAL" '. + {chargebackTag: $chargebackTag}' > /var/tmp/update-offering curl -s "localhost:8100$offeringid" -XPUT -d @/var/tmp/update-offering -o /dev/null done Tested this on version: BIG-IQ 8.3.0BIG-IQ BIG-IP REST API device discovery - HTTPLIB
Problem this snippet solves: In, bigiq-rest-api-device discovery - requests, we provide an example of how to leverage the rest api for BIG-IQ to trust, discover and import modules using python and the requests lib. The example takes advantage of python class inheritance and the requests module python requests. Because the python requests module is supported with python version 2.7.9 and above, the automation cannot run directly within the BIG-IQ shell. Why you ask, because the distribution of Linux used for the BIG-IQ application of rest framework ships with python 2.4.3. For simplification the script attached will use httplib client protocol, python httplib, and will provide a good comprehensive example of trust, discover (LTM, AFM, ASM) and import (LTM, AFM, ASM) running directly on BIG-IQ. How to use this snippet: Log into BIG-IQ as root. Create a directory under /shared/. and call it scripts. Or whatever you'd like. scp discovery.py to root@<bigiq.ip>:/shared/scripts/. ./discovery.py <bigip.ip> all # trust negotiation, discover (LTM, AFM, ASM), import (LTM, AFM, ASM) ./discovery.py <bigip.ip> afm # trust negotiation, discover (LTM, AFM), import (LTM, AFM) ./discovery.py <bigip.ip> asm # trust negotiation, discover (LTM, ASM), import (LTM, ASM) Code : https://github.com/carldubois/bigiq-cm-restapi/blob/master/discover.py Tested this on version: 12.0
CLI tool for working with BIG-IQ Regkey license pools
Problem this snippet solves: A BIG-IQ license manager can import base regkeys into a regkey pool and activate them. It cannot import addon keys from a csv along with the base regkey or use a csv to install addon keys to previously Activated offerings. Doing a large number of these one at a time in the GUI can be time-consuming. This script makes it possible to feed the keys as arguments when running the script. It only installs one license at a time but makes it possible to loop over a file(s) to automate the process. This script can be used to install base regkeys with or without the addonkeys. It can also install addon keys to an existing offering. It cannot be run locally on a BIG-IQ because some modules are not available for import. How to use this snippet: ./reg_pool_tool.py -h usage: reg_pool_tool.py [-h] [-d] [-v] [-a ADDRESS] [-u USERNAME] [-p PASSWORD] [-l] [-o POOL_UUID] [-r REG_KEY] [-A ADD_ON_KEY_LIST] [-i INSTALL_POOL_UUID] [-m MODIFY_POOL_UUID] This is a tool for workingwith regkey pool on BIG-IQ optional arguments: -h, --help show this help message and exit -d, --debug enable debug -v, --verbose enable verbose for options that have it -a ADDRESS, --address ADDRESS IP address of the target host -u USERNAME, --username USERNAME username for auth to host -p PASSWORD, --password PASSWORD password for auth to host -l, --list-pools list the UUIDs for existing regkey pools, requires no args -o POOL_UUID, --offerings POOL_UUID take UUID of pool as arg and list the offerings for a pool use -v to also show the active modules -r REG_KEY, --regkey REG_KEY takes and stores the regkey for use in other options -A ADD_ON_KEY_LIST, --add-on-keys ADD_ON_KEY_LIST takes string of comma sep addon keys for use by other options -i INSTALL_POOL_UUID, --install-offering INSTALL_POOL_UUID takes pool UUID as arg and installs new offering,requires -r, -A can be used to install addon keys atthe same time -m MODIFY_POOL_UUID, --modify-offering-addons MODIFY_POOL_UUID takes pool UUID as arg and installs addon to offering,requires -A [addon_key_list] and -r reg_key Examples: List regkey pools: ./reg_pool_tool.py -a 192.0.44 -l List the offerings in a regkey pool: ./reg_pool_tool.py -a 192.0.2.44 -o 07c77c7f-3170-4b17-93de-31e4f39e4709 Installing a new regkey with addon: ./reg_pool_tool.py -a 192.0.2.44 -i 5678d528-daac-4430-a87e-b67b87acfe20 -r B6083-41023-70161-19033-9429393 -A -A F659828-1850547 Modifying an existing offering adding a new addon key: ./reg_pool_tool.py -a 192.0.2.44 -m 5678d528-daac-4430-a87e-b67b87acfe20 -r K1260-99690-76028-13047-3993585 -A J984388-8738340 Code : #!/usr/bin/env python """ reg_pool_tool.py Python Version: 2.7.15 """ from __future__ import print_function import argparse from base64 import b64encode import json #from pprint import pprint import sys import time import urllib3 import requests ### Arguments parsing section ### def cmd_args(): """Handles command line arguments given.""" parser = argparse.ArgumentParser(description='This is a tool for working' 'with regkey pool on BIG-IQ') parser.add_argument('-d', '--debug', action="store_true", default=False, help='enable debug') parser.add_argument('-v', '--verbose', action="store_true", default=False, help='enable verbose for options that have it') parser.add_argument('-a', '--address', action="store", dest="address", help='IP address of the target host') parser.add_argument('-u', '--username', action="store", dest="username", default='admin', help='username for auth to host') parser.add_argument('-p', '--password', action="store", dest="password", default='admin', help='password for auth to host') parser.add_argument('-l', '--list-pools', action="store_true", default=False, help='list the UUIDs for existing regkey pools, requires no args') parser.add_argument('-o', '--offerings', action="store", dest="pool_uuid", help='take UUID of pool as arg and list the offerings for a pool' ' use -v to also show the active modules') parser.add_argument('-r', '--regkey', action="store", dest="reg_key", help='takes and stores the regkey for use in other options') parser.add_argument('-A', '--add-on-keys', action="store", dest="add_on_key_list", help='takes string of comma sep addon keys for use by other options') parser.add_argument('-i', '--install-offering', action="store", dest="install_pool_uuid", help='takes pool UUID as arg and installs new offering,' 'requires -r, -A can be used to install addon keys at' 'the same time') parser.add_argument('-m', '--modify-offering-addons', action="store", dest="modify_pool_uuid", help='takes pool UUID as arg and installs addon to offering,' 'requires -A [addon_key_list] and -r reg_key') parsed_arguments = parser.parse_args() # debug set print parser info if parsed_arguments.debug is True: print(parsed_arguments) # required args here if parsed_arguments.address is None: parser.error('-a target address is required, ' 'use mgmt for local') if parsed_arguments.install_pool_uuid: if parsed_arguments.reg_key is None: parser.error('-i requires -r') if parsed_arguments.modify_pool_uuid: if parsed_arguments.add_on_key_list is None: parser.error('-m requires -A and -r') elif parsed_arguments.reg_key is None: parser.error('-m requires -A and -r') return parsed_arguments ### END ARGPARSE SECTION ### def get_auth_token(address, user, password, uri='/mgmt/shared/authn/login'): # -> unicode """Get and auth token( to be used but other requests""" credentials_list = [user, ":", password] credentials = ''.join(credentials_list) user_and_pass = b64encode(credentials).decode("ascii") headers = {'Authorization':'Basic {}'.format(user_and_pass), 'Content-Type':'application/json'} post_data = '{"username":"' + user + '","password":"' + password +'"}' url_list = ['https://', address, uri] url = ''.join(url_list) try: request_result = requests.post(url, headers=headers, data=post_data, verify=False) except requests.exceptions.ConnectionError as connection_error: print(connection_error) sys.exit(1) except requests.exceptions.RequestException as request_exception: print(request_exception) sys.exit(1) #returns an instance of unicode that is an auth token with 300 dec timeout return request_result.json()['token']['token'] def get(url, auth_token, debug=False, return_encoding='json'): """Generic GET function. The return_encoding can be:'text', 'json', 'content'(binary), or raw """ headers = {'X-F5-Auth-Token':'{}'.format(auth_token), 'Content-Type':'application/json'} get_result = requests.get(url, headers=headers, verify=False) if debug is True: print('GET request...') print('get_result.encoding: {}'.format(get_result.encoding)) print('get_result.status_code: {}'.format(get_result.status_code)) print('get_result.raise_for_status: {}'.format( get_result.raise_for_status())) if return_encoding == 'json': return get_result.json() elif return_encoding == 'text': return get_result.text elif return_encoding == 'content': return get_result.content elif return_encoding == 'raw': return get_result.raw() # requires 'stream=True' in request def post(url, auth_token, post_data, debug): """ generic POST function """ headers = {'X-F5-Auth-Token':'{}'.format(auth_token), 'Content-Type':'application/json'} #post_data = '{"key":"value"}' try: post_result = requests.post(url, post_data, headers=headers, verify=False, timeout=10) except requests.exceptions.ConnectionError as connection_error: print ("Error Connecting: {}".format(connection_error)) sys.exit(1) except requests.exceptions.RequestException as request_exception: print(request_exception) sys.exit(1) if debug is True: print('POST request...') print('post_result.encoding: {}'.format(post_result.encoding)) print('post_result.status_code: {}'.format(post_result.status_code)) print('post_result.raise_for_status: {}'.format( post_result.raise_for_status())) return post_result.json() def patch(url, auth_token, patch_data, debug): """generic PATCH function""" headers = {'X-F5-Auth-Token':'{}'.format(auth_token), 'Content-Type':'application/json'} #patch_data = '{"key":"value"}' try: patch_result = requests.patch(url, patch_data, headers=headers, verify=False, timeout=10) except requests.exceptions.ConnectionError as connection_error: print ("Error Connecting: {}".format(connection_error)) sys.exit(1) except requests.exceptions.RequestException as request_exception: print(request_exception) sys.exit(1) if debug is True: print('PATCH request...') print('patch_result.encoding: {}'.format(patch_result.encoding)) print('patch_result.status_code: {}'.format(patch_result.status_code)) print('patch_result.raise_for_status: {}'.format( patch_result.raise_for_status())) return patch_result.json() class RegPool: """work with regkey pools""" def __init__(self, bigiq_address, username, password, debug=False): self.address = bigiq_address self.user = username self.password = password self.token = get_auth_token(self.address, self.user, self.password, uri='/mgmt/shared/authn/login') self.debug = debug def list_pool(self): """ Lists existing regkey pools """ uri = '/mgmt/cm/device/licensing/pool/regkey/licenses' url_list = ['https://', self.address, uri] url = ''.join(url_list) pool_list_result = get(url, self.token, self.debug, return_encoding='json') # create a list of list [[ , ]] pool_list = [] for entry in pool_list_result['items']: pool_list.append([entry['id'], entry['name']]) return pool_list def list_offereings(self, regkey_pool_uuid): """Returns a list of offerings for the regkey pool UUID given""" url_list = ['https://', self.address, '/mgmt/cm/device/licensing/pool/regkey/licenses/', regkey_pool_uuid, '/offerings'] url = ''.join(url_list) offering_get_result = get(url, self.token, self.debug, return_encoding='json') offering_list_result = offering_get_result['items'] # returns list of dictionaries of offerings return offering_list_result def install_offering(self, regkey_pool_uuid, new_regkey, add_on_keys): """ :type regkey_pool_uuid: str :type new_regkey: str :type add_on_keys: str comma sep keys This fucntion installs a new base regkey and optional addon keys and installs, and attempts to activate. All status in printed by the function and there is no return statement. If it fails it will show that was well. """ uri = '/mgmt/cm/device/licensing/pool/regkey/licenses/' url_list = ['https://', self.address, uri, regkey_pool_uuid, '/offerings/'] url = ''.join(url_list) if add_on_keys: post_dict = { "regKey": new_regkey, "status": "ACTIVATING_AUTOMATIC", "addOnKeys": add_on_keys.split(','), "description" : "" } else: post_dict = { "regKey": new_regkey, "status": "ACTIVATING_AUTOMATIC", "description" : "" } # format dict to make sure it is json compliant payload = json.dumps(post_dict) try: post(url, self.token, payload, self.debug) print('\nSent base regkey {} to License server status:'.format(new_regkey)) except: print('Post to License server failed') raise # poll for "eulaText" poll_result = {} attempt = 0 # keep track of tries and give up exit script after 10 uri = '/mgmt/cm/device/licensing/pool/regkey/licenses/' url_list = ['https://', self.address, uri, regkey_pool_uuid, '/offerings/', new_regkey] url = ''.join(url_list) while "eulaText" not in poll_result.keys(): try: poll_result = get(url, self.token, self.debug, return_encoding='json') print('\npoll {} for {}'.format(attempt +1, new_regkey)) if "fail" in poll_result['message']: sys.exit(poll_result['message']) print(poll_result['status']) print(poll_result['message']) time.sleep(5) except: print('Poll for eula failed for regkey {}'.format(new_regkey)) raise attempt += 1 if attempt == 5: sys.exit('Giving up after 5 tries to poll for EULA for RegKey') print('Finished polling...') # since we have eula back we need to patch back the eula # update "status" in dict poll_result["status"] = "ACTIVATING_AUTOMATIC_EULA_ACCEPTED" uri = '/mgmt/cm/device/licensing/pool/regkey/licenses/' url_list = ['https://', self.address, uri, regkey_pool_uuid, '/offerings/', new_regkey] url = ''.join(url_list) patch_dict = {"status":poll_result['status'], "eulaText": poll_result['eulaText']} patch_payload = json.dumps(patch_dict) print('sending PATCH to accept EULA for {}'.format(new_regkey)) try: patch_result = patch(url, self.token, patch_payload, self.debug) print('{} for {}'.format(patch_result['message'], new_regkey)) print(patch_result.get('status', 'ERROR: Status Not found in path_result')) except: raise def modify_offering_addon(self, regkey_pool_uuid, new_regkey, add_on_keys): """ :type regkey_pool_uuid: str :type new_regkey: str :type add_on_keys: str comma sep keys Install addon keys to t previously installed Offereing """ uri = '/mgmt/cm/device/licensing/pool/regkey/licenses/' url_list = ['https://', self.address, uri, regkey_pool_uuid, '/offerings/', new_regkey] url = ''.join(url_list) patch_dict = {"status": "ACTIVATING_AUTOMATIC", "addOnKeys": add_on_keys.split(',')} payload = json.dumps(patch_dict) try: patch(url, self.token, payload, self.debug) print('\nAdding {} addons for offering {} to License server status:'.format( add_on_keys.split(','), new_regkey)) except: print('Post to License server failed') raise # poll for "eulaText" poll_result = {} attempt = 0 # keep track of tries and give up exit script after 10 uri = '/mgmt/cm/device/licensing/pool/regkey/licenses/' url_list = ['https://', self.address, uri, regkey_pool_uuid, '/offerings/', new_regkey] url = ''.join(url_list) while not poll_result.get('status'): try: poll_result = get(url, self.token, self.debug, return_encoding='json') print('\npoll {} for {}, Addons: {}'.format( attempt +1, new_regkey, add_on_keys.split(','))) if "fail" in poll_result['message']: sys.exit(poll_result['message']) print(poll_result['status']) print(poll_result['message']) time.sleep(5) except: print('Poll for eula failed for regkey {}'.format(new_regkey)) raise attempt += 1 if attempt == 5: sys.exit('Giving up after 5 tries to poll for EULA for RegKey') print('Reactivation complete') print(poll_result.get('status')) print('Finished polling...') if __name__ == "__main__": SCRIPT_NAME = sys.argv[0] # suppress ssl warning when disbling ssl verification with verify=False urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) OPT = cmd_args() # STATIC GLOBALS ADDRESS = OPT.address RG_OBJECT = RegPool(OPT.address, OPT.username, OPT.password, OPT.debug) # -l if OPT.list_pools: LICENSE_LIST = RG_OBJECT.list_pool for pool in LICENSE_LIST(): print('{} {}'.format(pool[0], pool[1])) # -o, if -v included will also show moodules if OPT.pool_uuid: POOL_OFFERINGS = RG_OBJECT.list_offereings(OPT.pool_uuid) print('{0:35} {1:20} {2:10}'.format('RegKey', 'Status', 'addOnKeys')) print(73 * '-') for offering in POOL_OFFERINGS: if 'addOnKeys' in offering: print('{0:35} {1:20} {2:10}'.format(offering['regKey'], offering['status'], 'YES')) # if verbose given list Active modules if OPT.verbose: active_modules = offering.get('licenseText', 'Not available').splitlines() for line in active_modules: if line.startswith('active module'): print(' {} '.format(line[:80])) else: # -v not given list without active module info print('{0:35} {1:20} {2:10}'.format(offering['regKey'], offering['status'], offering.get('addOnKeys'))) # -i install new offereing with or without an addon keys, requires -r if OPT.install_pool_uuid: RG_OBJECT.install_offering(OPT.install_pool_uuid, OPT.reg_key, OPT.add_on_key_list) # -m requires -r -A if OPT.modify_pool_uuid: RG_OBJECT.modify_offering_addon(OPT.modify_pool_uuid, OPT.reg_key, OPT.add_on_key_list)Big-IQ licensing of Big-IP using REST API - Python OO
Problem this snippet solves: Some may want code that is a little more extendable and will scale to provide more of a workflow structure. In this example, we explore the use of Python class to simplify the code using a call to class method doing the work for licensing a BIGIP for a given License Pool. With the model you can create new methods to extend functionality. For example: class Workflow contains def Discover, def License, def CreateVirtual, def CreatePool, def AddMembers, def RefSecPolicy. Script will look like: workflow.Discover() workflow.License() workflow.CreateVirtual() workflow.CreatePool() workflow.AddMembers() workflow.RefSecPolicy() Code : https://github.com/carldubois/licapiBIG-IQ BIG-IP REST API device licensing - HTTPLIB
Problem this snippet solves: In, bigiq-rest-api-device licensing - requests, we provide an example of how to leverage the rest api for BIG-IQ licensing by adding a device to an existing license pool using python and the requests lib. The example takes advantage of python class inheritance and the requests module python requests. Because the python requests module is supported with python version 2.7.9 and above, the automation cannot run directly within the BIG-IQ shell. Why you ask, because the distribution of Linux used for the BIG-IQ application of rest framework ships with python 2.4.3. For simplification the script attached will use httplib client protocol, python httplib, and will provide a good comprehensive example of licensing a BIGIP running directly on BIG-IQ. How to use this snippet: Log into BIG-IQ as root. Create a directory under /shared/. and call it scripts. Or whatever you'd like. scp license.py to root@<bigiq.ip>:/shared/scripts/. ./license.py <bigip.ip> <base-reg-key> Code : https://github.com/carldubois/bigiq-cm-restapi/blob/master/license.pyBig-IQ bulk licensing of Big-IP using REST API
Problem this snippet solves: Attached is a link to github which provides the user with an comprehensive example of how to license many BIGIP devices via BIGIQ CM REST API into an existing license pool. Script bulkLicensePool.pl is a standalone script installed directly in the BIGIQ shell. Suggested recommendations: 1. Create a /shared/scripts/. directory 2. scp file to BIGIQ, 3. Usage below. This automation will invoke a task to license many BIGIP's as defined in a bulk_license.csv file. This happens sequentially and is very useful when administrator's goal is to license many BIGIP devices in a programmatic manner. ** tested with perl distribution present on bigiq v5.8.8 How to use this snippet: Usage: ./bulkDiscovery -c bulk_discovery.csv Program: bulkLicensePool.pl Version: v2.00.00 ##### License multiple BIG-IP devices. -r Root credentials for every BIG-IP (such as root:default) - overrides root creds in CSV -a Admin credentials for every BIG-IP (such as admin:admin) - overrides any creds in CSV -v Verbose screen output -s Discover ASM -l Discover LTM -p Discover APM -c Path to CSV file with all BIG-IP devices - REQUIRED -u Update framework if needed -h Help -k Keep the CSV file after this finishes (not recommended if it contains creds) -q BIG-IQ admin credentials in form admin:password - REQUIRED if not using default -g access group name if needed -f Discover AFM csv format: ip, user, pw, cluster-name, framework-action, root-user, root-pw ip: ip address of the BigIP to discover. user, pw: username & password of the BigIP. Will be overridden if -a is specified on the command line. configuration csv example format 1.2.3.4, admin, pw, base-reg-key Code : https://github.com/carldubois/bigiq-cm-restapi-bulk Tested this on version: 12.0
