Code to create unreachable ELA license files from BIG-IQ
Problem this snippet solves: *NOTE* if you are upgrading your BIG-IP,please refer to F5 solution:https://support.f5.com/csp/article/K13540950 BIG-IQ traditionally expects to be able to reach any BIG-IP devices it is going to license. This code helps create a license file from the ELA SKU offerings which can be applied on an Unreachable BIG-IP. I've added some troubleshoting steps at the end of the article, Dossier errors seen on the BIG-IP, just in case! How to use this snippet: SSH into the BIG-IP device and run the following command to gain the MAC address of the management interface tmsh show sys mac-address | grep -i interface [root@bigip1:Active:Standalone] config # tmsh show sys mac-address | grep -i interface ll:50:56:xx:xx:36net interfacemgmtmac-address xxxxxxxxxxxxxxxxxnet interface1.3mac-address xxxxxxxxxxxxxxxxxnet interface1.1mac-address xxxxxxxxxxxxxxxxx net interface1.2mac-address In the example above the MAC address we need is “ll:50:56:xx:xx:36” Now SSH into the BIG-IQ Move into the /shared directory (cd /shared) Copy over the Create-license.PY python script and run it by typing python Create-license.py The script runs and will prompt you for the following information [root@Preece-bigiq-cm1:Active:Standalone] shared # python Create-license.py Enter BIG-IQ user ID: admin Enter BIG-IQ Password: Enter Management IP address of BIG-IQ: 44.131.176.101 Enter Management IP address of BIG-IP to be licensed: 44.131.176.22 Enter Management MAC address of BIG-IP to be licensed: ll:50:56:xx:kk:36 Enter the name of the License Pool from which to take BIG-IP license: Load-18 Enter the license name to be assigned to the BIG-IP: F5-BIG-MSP-BT-1GIPIF-LIC-DEV Enter hypervisor used, valid options are: aws, azure, gce, hyperv, kvm, vmware,xen: vmware Optional: Enter chargeback tag if required: Department-A Optional: Enter tenant name if required: Customer-B Once the details have been filled in the script authenticates to the BIG-IQ and generates the license (30 seconds) If everything went well, you will be presented with a success message. The license file is saved as IP-address_bigip.license in the same directory as you run the script Using SCP copy the new license file from the BIG-IQ to your desktop. Copy the license file into the /config directory of the BIG-IP device. Rename the file, copy ip-address.bigip.license bigip.license Reload the license by typing reloadlic Observe the BIG-IP device restart its services and show as active. You can review in the GUI (System—License) and provision modules as needed. Code : import getpass # used to hide the users password input import json import os import requests from time import sleep """ This script uses the BIG-IQ API to license an unreachable (dark site) BIG-IP. The BIG-IQ licensing API needs certain details provided in order to license an appliance, these details can either be provided in a file call lic-data.json or if that file does not exist you will be prompted to enter them. The minimum contents of lic-data.json should be: { "licensePoolName": " -- Enter License Pool Name here. License Pool name can be found in BIG-IQ GUI -- ", "command": "assign", "address": " -- Enter MGMT IP Address of BIG-IP here -- ", "assignmentType": "UNREACHABLE", "macAddress": " -- Enter MAC address of MGMT IP for the BIG-IP here -- ", "hypervisor": " -- Enter hypervisor value here options are; aws, azure, gce, hyperv, kvm, vmware, xen: --", "unitOfMeasure": "yearly", "skuKeyword1": "-- Enter License Name here. License Name (or Offering name) can be found in the BIG-IQ GUI -- " } Additional Optional key:value pairs can be added to the JSON file to afix useful tags to the license. The json file with optional key:value pairs looks like: { "licensePoolName": " -- Enter License Pool Name here. License Pool name can be found in BIG-IQ GUI -- ", "command": "assign", "address": " -- Enter MGMT IP Address of BIG-IP here -- ", "assignmentType": "UNREACHABLE", "macAddress": " -- Enter MAC address of MGMT IP for the BIG-IP here -- ", "hypervisor": " -- Enter hypervisor value here options are; aws, azure, gce, hyperv, kvm, vmware, xen: --", "unitOfMeasure": "yearly", "skuKeyword1": "-- Enter License Name here. License Name (or Offering name) can be found in the BIG-IQ GUI -- ", "chargebackTag": "OPTIONAL: Remove this line if you are not going to use it", "tenant": "OPTIONAL: Remove this line if you are not going to use it" } A completed minimal lic-data.json file will look like this: { "licensePoolName": "byol-pool-utility", "command": "assign", "address": "10.1.1.10", "assignmentType": "UNREACHABLE", "macAddress": "06:ce:c2:43:b3:05", "hypervisor": "kvm", "unitOfMeasure": "yearly", "skuKeyword1": "F5-BIG-MSP-BT-P3-3GF-LIC-DEV" } lic-data.json must reside in the directory from which you execute this python script. """ def bigiqAuth(_bigiqAuthUrl, _bigiqCredentials): """ This function authenticates with BIG-IQ and collects the authentication token provided. Theo token will be used for subsequent calls to BIG-IQ """ _errFlag=0 try: _bigiqAuthInfo=_bigiq_session.post(_bigiqAuthUrl, data=json.dumps(_bigiqCredentials), verify=False) print(_bigiqAuthUrl) _bigiqAuthInfo.raise_for_status() print("Response code: %s" %_bigiqAuthInfo.status_code) except requests.exceptions.HTTPError as err: print(err) _errFlag=1 #end try if _errFlag==0: _bigiqResponse=_bigiqAuthInfo.json() _bigiqToken=_bigiqResponse['token'] for _token in _bigiqToken: if (_token == 'token'): _bigiqAuthToken=(_bigiqToken[_token]) # End if # Next _authHeaders={ "X-F5-Auth-Token": "{_authToken}".format(_authToken=_bigiqAuthToken) } else: _authHeaders=0 #end if print("** Completed Authentication ***") return(_authHeaders); #End Def def extractLicense(_rawLicenseJSON): """ This function pulls the generated license from BIG-IQ """ for _license in _rawLicenseJSON: if (_license=='licenseText'): _extractedLicense=_rawLicenseJSON[_license] #end if if (_license=='status'): if (_rawLicenseJSON[_license]=="FINISHED"): print("***** License has been assigned *****") else: _extractedLicense="FAILED" #end if #end if #next return(_extractedLicense); #End def def licenseData(): """ This function read the lic-data.json file. If it does not exist you will be prompted to enter the necessary values. """ if os.path.exists('lic-data.json'): with open('./lic-data.json') as licfile: _licdata = json.load(licfile) else: _bigipAddress=raw_input("Enter Management IP address of BIG-IP to be licensed: ") _bigipMACaddress=raw_input("Enter Management MAC address of BIG-IP to be licensed: ") _licensePoolName=raw_input("Enter the name of the License Pool from which to take BIG-IP license: ") _licenseSKU=raw_input("Enter the license name to be assigned to the BIG-IP: ") _hypervisorType=raw_input("Enter hypervisor used, valid options are: aws, azure, gce, hyperv, kvm, vmware, xen: ") _chargebackTag=raw_input("Optional: Enter chargeback tag if required: ") _tenantTag=raw_input("Optional: Enter tenant name if required: ") _licdata={ "licensePoolName": "{_licensePool}".format(_licensePool=_licensePoolName), "command": "assign", "address": "{_bigipIP}".format(_bigipIP=_bigipAddress), "assignmentType": "UNREACHABLE", "macAddress": "{_bigipMAC}".format(_bigipMAC=_bigipMACaddress), "hypervisor": "{_hypervisor}".format(_hypervisor=_hypervisorType), "unitOfMeasure": "yearly", "skuKeyword1": "{_license}".format(_license=_licenseSKU), "chargebackTag": "{_chargeback}".format(_chargeback=_chargebackTag), "tenant": "{_tenant}".format(_tenant=_tenantTag) } # End if return(_licdata); def urlConstruction(_bigiqUrl, _bigiqIP): """ This function rewrites the selflink URL returned by BIG-IQ to reflect BIG-IQ management IP address rather than localhost """ count=0 _urlDeConstruct=_bigiqUrl.split("/") _urlReConstruct="" for _urlElement in _urlDeConstruct: #print("%d %s" %(count,_urlElement)) if (_urlElement=="https:"): _urlReConstruct=_urlReConstruct+_urlElement+"//" elif (_urlElement=="localhost"): _urlReConstruct=_urlReConstruct+_bigiqIP else: if (_urlElement!=""): _urlReConstruct=_urlReConstruct+"/"+_urlElement #end if #end if count+=1 #Next return(_urlReConstruct); #End Def _userID=raw_input("Enter BIG-IQ user ID: ") _password=getpass.getpass(prompt="Enter BIG-IQ Password: ") _bigiqAddress=raw_input("Enter Management IP address of BIG-IQ: ") _credPostBody={ "username": "{_uname}".format(_uname=_userID), "password": "{_pword}".format(_pword=_password), "loginProvideriName": "RadiusServer" } _deviceToBeLicensed=licenseData() _bigipAddress=_deviceToBeLicensed['address'] print("BIG-IP Address is: %s" %_bigipAddress) _bigiq_session=requests.session() _bigiq_auth_url="https://{_bigiqIP}/mgmt/shared/authn/login".format(_bigiqIP=_bigiqAddress) # Authenticates with BIG-IQ _bigiqAuthHeader=bigiqAuth(_bigiq_auth_url, _credPostBody) # if _bigiqAuthHeader==0: print("Unable to authenticate with BIG-IQ. Check BIG-IQ reachability and credentials") else: _bigiq_url1="https://{_bigiqIP}/mgmt/cm/device/tasks/licensing/pool/member-management".format(_bigiqIP=_bigiqAddress) # # --- This section requests the license from BIG-IQ. Posting the criteria as laid out in the _deviceToBeLicensed JSON blob # _errFlag=0 try: _bigiqLicenseDevice=_bigiq_session.post(_bigiq_url1, headers=_bigiqAuthHeader, data=json.dumps(_deviceToBeLicensed), verify=False) _bigiqLicenseDevice.raise_for_status() print("Response code: %s" %_bigiqLicenseDevice.status_code) except requests.exceptions.HTTPError as err: print("Issue received, check rquest and or check connectivity %s" %err) _errFlag=1 #end try if _errFlag==0: #print(_bigiqLicenseDevice.status_code) _bigiqResponse=_bigiqLicenseDevice.json() print(_bigiqResponse) print(_bigiqResponse['selfLink']) _bigiqLicenseStatus_url=_bigiqResponse['selfLink'] _bigiqLicenseStatus_url=urlConstruction(_bigiqLicenseStatus_url, _bigiqAddress) print(_bigiqLicenseStatus_url) print("--- Standby for 30 seconds whilst BIG-IQ generates license ---") sleep(30) _errFlag1=0 try: _licenseStatus=_bigiq_session.get(_bigiqLicenseStatus_url, headers=_bigiqAuthHeader, verify=False) _licenseStatus.raise_for_status() print("Response code: %s" %_licenseStatus.status_code) except requests.exceptions.HTTPError as err: print("Issue received, check rquest and or check connectivity %s" %err) _errFlag=1 #end try if _errFlag==0: print(_licenseStatus.content) _licenseStatusDetail=_licenseStatus.json() _licenseOutput=extractLicense(_licenseStatusDetail) if (_licenseOutput=="FAILED"): print("***** License Assignment Failed. Most likely a valid license already exists for device, revoke it before applying a new license *****") else: _licenseFname=(_bigipAddress+"_bigip.license") _licensefile=open(_licenseFname, "w") _licensefile.write("%s" %_licenseOutput) _licensefile.close() print(_licenseOutput) print("***** SUCCESS, the license is stored here %s *****" %_licenseFname) #end if #end if #end if #end if Tested this on version: 13.x, 14.x, 15.x and 16.x Troubleshooting When you apply the license to the BIG-IP you may see an error similar to: License is not operational (expired or digital signature does not match contents) This could simply be that you copy and paste the license file badly, please use MD5SUM on the BIG-IQ to the output license file and compare to the same file on the BIG-IP Example: md5sum 10.2.3.4_bigip.license You can also review the /var/log/ltm file for "Dossier error" messages Dossier error: 1 (MAC address is mismatched) Dossier error: 12 (Hypervisor is mismatched) If this does not help, please open a support case and attach a recent qkview file.Automate Data Group updates on many Big-IP devices using Big-IQ or Ansible or Terraform
Problem this snippet solves: In many cases generated bad ip address lists by a SIEM (ELK, Splunk, IBM QRADAR) need to be uploaded to F5 for to be blocked but the BIG-IQ can't be used to send data group changes to the F5 devices. 1.A workaround to use the BIG-IQ script option to make all the F5 devices to check a file on a source server and to update the information in the external data group. I hope F5 to add the option to BIG-IQ to schedule when the scrpts to be run otherwise a cron job on the BIG-IQ may trigger the script feature that will execute the data group to refresh its data (sounds like the Matrix). https://clouddocs.f5.com/training/community/big-iq-cloud-edition/html/class5/module1/lab6.html Example command to run in the BIG-IQ script feature: tmsh modify sys file data-group ban_ip type ip source-pathhttps://x.x.x.x/files/bad_ip.txt https://support.f5.com/csp/article/K17523 2.You can also set the command with cronjob on the BIG-IP devices if you don't have BIG-IQ as you just need Linux server to host the data group files. 3.Also without BIG-IQ Ansible playbook can be used to manage many groups on the F5 devices as I have added the ansible playbook code below. Now with the windows subsystem you can run Ansible on Windows! 4.If you have AFM then you can use custom feed lists to upload the external data without the need for Ansible or Big-IQ. The ASM supports IP intelligence but no custom feeds can be used: https://techdocs.f5.com/kb/en-us/products/big-ip-afm/manuals/product/big-ip-afm-getting-started-14-1-0/04.html How to use this snippet: I made my code reading: https://docs.ansible.com/ansible/latest/collections/f5networks/f5_modules/bigip_data_group_module.html https://support.f5.com/csp/article/K42420223 If you want to have an automatic timeout then you need to use the irule table command (but you can't edit that with REST-API, so see the article below as a workaround) that writes in the RAM memory that supports automatic timeout and life time for each entry then there is a nice article for that as I added comment about possible bug resolution, so read the comments! https://devcentral.f5.com/s/articles/populating-tables-with-csv-data-via-sideband-connections Another way is on the server where you save the data group info is to add a bash script that with cronjob deletes from time to time old entries. For example (I tested this). Just write each data group line/text entry with for example IP address and next to it the date it was added. cutoff=$(date -d 'now - 30 days' '+%Y-%m-%d') awk -v cutoff="$cutoff" '$2 >= cutoff { print }' <in.txt >out.txt && mv out.txt in.txt Ansible is a great automation tool that makes changes only when the configuration is modified, so even if you run the same playbook 2 times (a playbook is the main config file and it contains many tasks), the second time there will be nothing (the same is true for terraform). Ansible supports "for" loops but calls them "loop" (before time " with_items " was used) and "if else" conditions but it calls them "when" just to confuse us and the conditions and loops are placed at the end of the task not at the start 😀 A loop is good if you want to apply the same config to multiple devices with some variables just being changed and "when" is nice for example to apply different tasks to different versions of the F5 TMOS or F5 devices with different provisioned modules. https://stackoverflow.com/questions/38571524/remove-line-in-text-file-with-bash-if-the-date-is-older-than-30-days Code : --- - name: Create or modify data group hosts: all connection: local vars: provider: password: xxxxx server: x.x.x.x user: xxxxx validate_certs: no server_port: 443 tasks: - name: Create a data group of IP addresses from a file bigip_data_group: name: block_group records_src: /var/www/files/bad.txt type: address provider: "{{ provider }}" notify: - Save the running configuration to disk handlers: - name: Save the running configuration to disk bigip_config: save: yes provider: "{{ provider }}" The "notify" triggers the handler task after the main task is done as there is no point in saving the config before that and the handler runs only on change, Tested this on version: 15.1 Also now F5 has Terraform Provider and together with Visual Studio you can edit your code on Windows and deploy it from the Visual Studio itself! Visual Studio wil even open for you the teminal, where you can select the folder where the terraform code will be saved after you have added the code run terraform init, terraform plan, terraform apply. VS even has a plugin for writting F5 irules.Terraform's files are called "tf" and the terraform providers are like the ansible inventory file (ansible may also have a provider object in the playbook not the inventory file) and are used to make the connection and then to create the resources (like ansible tasks). Usefull links for Visual Studio and Terraform: https://registry.terraform.io/providers/F5Networks/bigip/1.16.0/docs/resources/bigip_ltm_datagroup https://www.youtube.com/watch?v=Z5xG8HLwIh4 For more advanced terafform stuff like for loops and if or count conditions: https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9 Code : You may need to add also this resource below as to save the config and with "depends_on" it wil run after the date group is created. This is like the handler in Ansible that is started after the task is done and also terraform sometimes creates resources at the same time not like Ansible task after task, resource "bigip_command" "save-config" { commands = ["save sys config"] depends_on = [ bigip_ltm_datagroup.terraform-external1 ] } Tested this on version: 16.1 Ansible and Terraform now can be used for AS3 deployments like the BIG-IQ's "applications" as they will push the F5 declarative templates to the F5 device and nowadays even the F5 AWAF/ASM and SSLO (ssl orchestrator) support declarative configurations. For more info: https://www.f5.com/company/blog/f5-as3-and-red-hat-ansible-automation https://clouddocs.f5.com/products/orchestration/ansible/devel/f5_bigip/playbook_tutorial.html https://clouddocs.f5.com/products/orchestration/terraform/latest/userguide/as3-integration.html https://support.f5.com/csp/article/K23449665 https://clouddocs.f5.com/training/fas-ansible-workshop-101/3.3-as3-asm.html https://www.youtube.com/watch?v=Ecua-WRGyJc&t=105s
Big-IQ bulk trust, discovery and import of Big-IP using REST API
Problem this snippet solves: Attached is a link to github which provides the user with an comprehensive example of how to discover and import many BIGIP device via BIGIQ CM REST API. Script bulkDiscovery.pl is a standalone script installed directly in the BIGIQ shell. Suggested recommendations: 1. Create a /shared/scripts/. directory 2. scp file to BIGIQ, 3. Usage below. This automation will invoke a device trust task to negotiate certificate, discover device to population in resolver groups (maintained per module) and import configuration of BIGIP's as defined in bulk_discovery.csv file. This happens sequentially and is very useful when administrator's goal is to discover and import many BIGIP devices in a programmatic manner. ** tested with perl distribution present on bigiq v5.8.8 How to use this snippet: Usage: ./bulkDiscovery -c bulk_discovery.csv Program: bulkDiscovery.pl Version: v2.00.00 ##### Discover multiple BIG-IP devices. -r Root credentials for every BIG-IP (such as root:default) - overrides root creds in CSV -a Admin credentials for every BIG-IP (such as admin:admin) - overrides any creds in CSV -v Verbose screen output -s Discover ASM -l Discover LTM -p Discover APM -c Path to CSV file with all BIG-IP devices - REQUIRED -u Update framework if needed -h Help -k Keep the CSV file after this finishes (not recommended if it contains creds) -q BIG-IQ admin credentials in form admin:password - REQUIRED if not using default -g access group name if needed -f Discover AFM csv format: ip, user, pw, cluster-name, framework-action, root-user, root-pw ip: ip address of the BigIP to discover. user, pw: username & password of the BigIP. Will be overridden if -a is specified on the command line. configuration csv example format: 1.2.3.4 1.2.3.4, admin, pw 1.2.3.4, admin, pw, ha-name 1.2.3.4,,, ha-name 1.2.3.4, admin, pw,, skip 1.2.3.4, admin, pw,, update, root, root-pw Code : https://github.com/carldubois/bigiq-cm-restapi-bulk Tested this on version: 12.0
