Enhance your GenAI chatbot with the power of Agentic RAG and F5 platform
Agentic RAG (Retrieval-Augmented Generation) enhances the capabilities of a GenAI chatbot by integrating dynamic knowledge retrieval into its conversational abilities, making it more context-aware and accurate. In this demo, I will demonstrate an autonomous decision-making GenAI chatbot utilizing Agentic RAG. I will explore what Agentic RAG is and why it's crucial in today's AI landscape. I will also discuss how organizations can leverage GPUaaS (GPU as a Service) or AI Factory providers to accelerate their AI strategy. F5 platform provides robust security features that protect sensitive data while ensuring high availability and performance. They optimize the chatbot by streamlining traffic management and reducing latency, ensuring smooth interactions even during high demand. This integration ensures the GenAI chatbot is not only smart but also reliable and secure for enterprise use.Advanced WAF v16.0 - Declarative API
Since v15.1 (in draft), F5® BIG-IP® Advanced WAF™ canimport Declarative WAF policy in JSON format. The F5® BIG-IP® Advanced Web Application Firewall (Advanced WAF) security policies can be deployed using the declarative JSON format, facilitating easy integration into a CI/CD pipeline. The declarative policies are extracted from a source control system, for example Git, and imported into the BIG-IP. Using the provided declarative policy templates, you can modify the necessary parameters, save the JSON file, and import the updated security policy into your BIG-IP devices. The declarative policy copies the content of the template and adds the adjustments and modifications on to it. The templates therefore allow you to concentrate only on the specific settings that need to be adapted for the specific application that the policy protects. ThisDeclarative WAF JSON policyis similar toNGINX App Protect policy. You can find more information on theDeclarative Policyhere : NAP :https://docs.nginx.com/nginx-app-protect/policy/ Adv. WAF :https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-declarative-security-policy.html Audience This guide is written for IT professionals who need to automate their WAF policy and are familiar with Advanced WAF configuration. These IT professionals can fill a variety of roles: SecOps deploying and maintaining WAF policy in Advanced WAF DevOps deploying applications in modern environment and willing to integrate Advanced WAF in their CI/CD pipeline F5 partners who sell technology or create implementation documentation This article covershow to PUSH/PULL a declarative WAF policy in Advanced WAF: With Postman With AS3 Table of contents Upload Policy in BIG-IP Check the import Apply the policy OpenAPI Spec File import AS3 declaration CI/CD integration Find the Policy-ID Update an existing policy Video demonstration First of all, you need aJSON WAF policy, as below : { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "blocking", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false } } } 1. Upload Policy in BIG-IP There are 2 options to upload a JSON file into the BIG-IP: 1.1 Either youPUSHthe file into the BIG-IP and you IMPORT IT OR 1.2 the BIG-IPPULLthe file froma repository (and the IMPORT is included)<- BEST option 1.1PUSH JSON file into the BIG-IP The call is below. As you can notice, it requires a 'Content-Range' header. And the value is 0-(filesize-1)/filesize. In the example below, the file size is 662 bytes. This is not easy to integrate in a CICD pipeline, so we created the PULL method instead of the PUSH (in v16.0) curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/file-transfer/uploads/policy-api.json' \ --header 'Content-Range: 0-661/662' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --header 'Content-Type: application/json' \ --data-binary '@/C:/Users/user/Desktop/policy-api.json' At this stage,the policy is still a filein the BIG-IP file system. We need toimportit into Adv. WAF. To do so, the next call is required. This call import the file "policy-api.json" uploaded previously. AnCREATEthe policy /Common/policy-api-arcadia curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/javascript' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "filename":"policy-api.json", "policy": { "fullPath":"/Common/policy-api-arcadia" } }' 1.2PULL JSON file from a repository Here, theJSON file is hosted somewhere(in Gitlab or Github ...). And theBIG-IP will pull it. The call is below. As you can notice, the call refers to the remote repo and the body is a JSON payload. Just change the link value with your JSON policy URL. With one call, the policy isPULLEDandIMPORTED. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" } }' Asecond versionof this call exists, and refer to the fullPath of the policy.This will allow you to update the policy, from a second version of the JSON file, easily.One call for the creation and the update. As you can notice below, we add the"policy":"fullPath" directive. The value of the "fullPath" is thepartitionand thename of the policyset in the JSON policy file. This method is VERY USEFUL for CI/CD integrations. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" }, "policy": { "fullPath":"/Common/policy-api-arcadia" } }' 2. Check the IMPORT Check if the IMPORT worked. To do so, run the next call. curl --location --request GET 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ You should see a 200 OK, with the content below (truncated in this example). Please notice the"status":"COMPLETED". { "kind": "tm:asm:tasks:import-policy:import-policy-taskcollectionstate", "selfLink": "https://localhost/mgmt/tm/asm/tasks/import-policy?ver=16.0.0", "totalItems": 11, "items": [ { "isBase64": false, "executionStartTime": "2020-07-21T15:50:22Z", "status": "COMPLETED", "lastUpdateMicros": 1.595346627e+15, "getPolicyAttributesOnly": false, ... From now, your policy is imported and created in the BIG-IP. You can assign it to a VS as usual (Imperative Call or AS3 Call).But in the next session, I will show you how to create a Service with AS3 including the WAF policy. 3. APPLY the policy As you may know, a WAF policy needs to be applied after each change. This is the call. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/apply-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{"policy":{"fullPath":"/Common/policy-api-arcadia"}}' 4. OpenAPI spec file IMPORT As you know,Adv. WAF supports OpenAPI spec (2.0 and 3.0). Now, with the declarative WAF, we can import the OAS file as well. The BEST solution, is toPULL the OAS filefrom a repo. And in most of the customer' projects, it will be the case. In the example below, the OAS file is hosted in SwaggerHub(Github for Swagger files). But the file could reside in a private Gitlab repo for instance. The URL of the projectis :https://app.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3 The URL of the OAS file is :https://api.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3 This swagger file (OpenAPI 3.0 Spec file) includes all the application URL and parameters. What's more, it includes the documentation (for NGINX APIm Dev Portal). Now, it ispretty easy to create a WAF JSON Policy with API Security template, referring to the OAS file. Below, you can notice thenew section "open-api-files"with the link reference to SwaggerHub. And thenew templatePOLICY_TEMPLATE_API_SECURITY. Now, when I upload / import and apply the policy, Adv. WAF will download the OAS file from SwaggerHub and create the policy based on API_Security template. { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "blocking", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false }, "open-api-files": [ { "link": "https://api.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3" } ] } } 5. AS3 declaration Now, it is time to learn how we cando all of these steps in one call with AS3(3.18 minimum). The documentation is here :https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/declarations/application-security.html?highlight=waf_policy#virtual-service-referencing-an-external-security-policy With thisAS3 declaration, we: Import the WAF policy from a external repo Import the Swagger file (if the WAF policy refers to an OAS file) from an external repo Create the service { "class": "AS3", "action": "deploy", "persist": true, "declaration": { "class": "ADC", "schemaVersion": "3.2.0", "id": "Prod_API_AS3", "API-Prod": { "class": "Tenant", "defaultRouteDomain": 0, "API": { "class": "Application", "template": "generic", "VS_API": { "class": "Service_HTTPS", "remark": "Accepts HTTPS/TLS connections on port 443", "virtualAddresses": ["10.1.10.27"], "redirect80": false, "pool": "pool_NGINX_API_AS3", "policyWAF": { "use": "Arcadia_WAF_API_policy" }, "securityLogProfiles": [{ "bigip": "/Common/Log all requests" }], "profileTCP": { "egress": "wan", "ingress": { "use": "TCP_Profile" } }, "profileHTTP": { "use": "custom_http_profile" }, "serverTLS": { "bigip": "/Common/arcadia_client_ssl" } }, "Arcadia_WAF_API_policy": { "class": "WAF_Policy", "url": "http://10.1.20.4/root/as3-waf-api/-/raw/master/policy-api.json", "ignoreChanges": true }, "pool_NGINX_API_AS3": { "class": "Pool", "monitors": ["http"], "members": [{ "servicePort": 8080, "serverAddresses": ["10.1.20.9"] }] }, "custom_http_profile": { "class": "HTTP_Profile", "xForwardedFor": true }, "TCP_Profile": { "class": "TCP_Profile", "idleTimeout": 60 } } } } } 6. CI/CID integration As you can notice, it is very easy to create a service with a WAF policy pulled from an external repo. So, it is easy to integrate these calls (or the AS3 call) into a CI/CD pipeline. Below, an Ansible playbook example. This playbook run the AS3 call above. That's it :) --- - hosts: bigip connection: local gather_facts: false vars: my_admin: "admin" my_password: "admin" bigip: "10.1.1.12" tasks: - name: Deploy AS3 WebApp uri: url: "https://{{ bigip }}/mgmt/shared/appsvcs/declare" method: POST headers: "Content-Type": "application/json" "Authorization": "Basic YWRtaW46YWRtaW4=" body: "{{ lookup('file','as3.json') }}" body_format: json validate_certs: no status_code: 200 7. FIND the Policy-ID When the policy is created, a Policy-ID is assigned. By default, this ID doesn't appearanywhere. Neither in the GUI, nor in the response after the creation. You have to calculate it or ask for it. This ID is required for several actions in a CI/CD pipeline. 7.1 Calculate the Policy-ID Wecreated this python script to calculate the Policy-ID. It is an hash from the Policy name (including the partition). For the previous created policy named"/Common/policy-api-arcadia",the policy ID is"Ar5wrwmFRroUYsMA6DuxlQ" Paste this python codein a newwaf-policy-id.pyfile, and run the commandpython waf-policy-id.py "/Common/policy-api-arcadia" Outcome will beThe Policy-ID for /Common/policy-api-arcadia is: Ar5wrwmFRroUYsMA6DuxlQ #!/usr/bin/python from hashlib import md5 import base64 import sys pname = sys.argv[1] print 'The Policy-ID for', sys.argv[1], 'is:', base64.b64encode(md5(pname.encode()).digest()).replace("=", "") 7.2 Retrieve the Policy-ID and fullPath with a REST API call Make this call below, and you will see in the response, all the policy creations. Find yours and collect thePolicyReference directive.The Policy-ID is in the link value "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0" You can see as well, at the end of the definition, the "fileReference"referring to the JSON file pulled by the BIG-IP. And please notice the"fullPath", required if you want to update your policy curl --location --request GET 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Range: 0-601/601' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ { "isBase64": false, "executionStartTime": "2020-07-22T11:23:42Z", "status": "COMPLETED", "lastUpdateMicros": 1.595417027e+15, "getPolicyAttributesOnly": false, "kind": "tm:asm:tasks:import-policy:import-policy-taskstate", "selfLink": "https://localhost/mgmt/tm/asm/tasks/import-policy/B45J0ySjSJ9y9fsPZ2JNvA?ver=16.0.0", "filename": "", "policyReference": { "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0", "fullPath": "/Common/policy-api-arcadia" }, "endTime": "2020-07-22T11:23:47Z", "startTime": "2020-07-22T11:23:42Z", "id": "B45J0ySjSJ9y9fsPZ2JNvA", "retainInheritanceSettings": false, "result": { "policyReference": { "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0", "fullPath": "/Common/policy-api-arcadia" }, "message": "The operation was completed successfully. The security policy name is '/Common/policy-api-arcadia'. " }, "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" } }, 8 UPDATE an existing policy It is pretty easy to update the WAF policy from a new JSON file version. To do so, collect from the previous call7.2 Retrieve the Policy-ID and fullPath with a REST API callthe"Policy" and"fullPath"directive. This is the path of the Policy in the BIG-IP. Then run the call below, same as1.2 PULL JSON file from a repository,but add thePolicy and fullPath directives Don't forget to APPLY this new version of the policy3. APPLY the policy curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" }, "policy": { "fullPath":"/Common/policy-api-arcadia" } }' TIP : this call, above, can be used in place of the FIRST call when we created the policy "1.2PULL JSON file from a repository". But be careful, the fullPath is the name set in the JSON policy file. The 2 values need to match: "name": "policy-api-arcadia" in the JSON Policy file pulled by the BIG-IP "policy":"fullPath" in the POST call 9 Video demonstration In order to help you to understand how it looks with the BIG-IP, I created this video covering 4 topics explained in this article : The JSON WAF policy Pull the policy from a remote repository Update the WAF policy with a new version of the declarative JSON file Deploy a full service with AS3 and Declarative WAF policy At the end of this video, you will be able to adapt the REST Declarative API calls to your infrastructure, in order to deploy protected services with your CI/CD pipelines. Direct link to the video on DevCentral YouTube channel : https://youtu.be/EDvVwlwEFRwHow to use F5 Distributed Cloud for API Discovery
In today's digital landscape, APIs are crucial for integrating diverse applications and services by enabling seamless communication and data sharing between systems. API discovery involves finding, exploring, and assessing APIs for their suitability in applications, considering their varied sources and functionalities. F5 Distributed provides multiple architectures that make it ridiculously easy to discover APIs.
BIG-IQ Central Management API automation and programmability - Workflow - Python
Summary In conjunction with the announcement of BIG-IQ 5.0 we are excited to bring the field greater flexibility when centrally managing BIG-IP devices utilizing well defined workflows and the BIG-IQ iControl REST API. We understand that automation and programmability is becoming more the norm these days as the network is evolving into a software defined application aware environment. This article will provide a basic example for first establishing trust while adding a device to inventory, second importing device configuration into BIG-IQ's configuration database and finally adding device to a license pool all utilizing the BIG-IQ rest api uri provided by the interface. Response is parsed in JSON format so there are dependencies required. This is intended to be an introduction to leveraging the API directly. We are in the process of developing a python object abstraction for the BIG-IQ CM REST API to provide even greater simplicity when leveraging automation for central management programmability. But we thought this level of detail may be appropriate to get the ball rolling. The BIG-IQ CM SDK will be structured similar to version 0.1.7 BIG-IP iControl REST API described here: BIGIP SDK - https://devcentral.f5.com/s/articles/f5-friday-python-sdk-for-big-ip-18233 BIGIP SDK Docs - https://f5-sdk.readthedocs.io/en/v0.1.7/ Let's get started ... READY ... SET ... GO ... Here are the API URIs' used as reference for both Python and Perl examples below: Trust:https://'bigiq-ip'/mgmt/cm/global/tasks/device-trust Discovery:https://'bigiq-ip'/mgmt/cm/global/tasks/device-discovery Import: ADC - https://'bigiq-ip'/mgmt/cm/adc-core/tasks/declare-mgmt-authority Import: Firewall - https://'bigiq-ip'/mgmt/cm/firewall/tasks/declare-mgmt-authority License: Pools - https://'bigiq-ip'/mgmt/cm/shared/licensing/pools License: Members: - https://'bigiq-ip'/mgmt/cm/shared/licensing/pools/'member_uuid'/members Using a python class abstraction (libraries can be located in /lib/) for Discovery, Import and Licensing, these tasks are a single call from one place which will provide the flexibility to create scripts defined by workflow. For example, lets - 1. Negotiate trust certificates for a newly deployed BIGIP. 2. Discover the BIGIP device and add to inventory. 3. Import ADC configuration and resolve all differences to BIGIQ’s management database. 4. Import Firewall configuration and resolve all differences to BIGIQ’s management database. 5. Add this newly discovered BIGIP to a provisioned license pool using the base-registration-key as a filter. Using the following python libraries, we can accomplish these tasks defined as methods in each class definition: ## Discovery ../lib/discovery.py: class Discover(object) def device_trust(dev_id): def ltm_discover(modules adc_core, firewall, asm, apm etc..): ## Import ../lib/import.py: class Import(object) def import_ltm(dev_id): def import_security(dev_id, afm=True): ## License ../lib/license.py : class License(object) def license_pool(config): Now it’s just a matter of putting the puzzle together in a script that will accomplish the workflow described above. BIG-IQ and BIG-IP device address and credentials are read in from a configuration file (../config/..) the script first and passed into def workflow for processing. Each method is called with argument <config> dictionary to perform the task required. After completion each will return True or False depending on result from rest response code and some evaluation. ## Import libraries to leverage methods described above from discovery import Discover from import_module import Import from license import License ## main def "workflow" def workflow (Discover, License, Import, config): ## Trust "establish certificate exchange" result_trust = Discover.device_trust(config) ## Discover "add a BIGIP to inventory" result = Discover.ltm_discover(config, result_trust) ## Import ADC "import adc configuration into database" result = Import.import_ltm(config, result_trust) ## Import AFM, ASM, APM "import module configuration into database" result = Import.import_sec(config, result_trust, afm=True) ## Add BIGIP member to license pool result = License.license_pool(config) ## Python main body if __name__ == '__main__': # read config file from ../config/.. config = {} with open (file) as infile: config[str(key)] = val.strip('\n') ## create an instance of each imported class defined in the libraries Discover = Discover(config) Import = Import(config) License = License(config) ## call the main program method "workflow" workflow (Discover, Import, License, config) If you are interested in this code for collaboration or solution, search on key words "bigiq" "api" "python" in code share section on dev central or you can refer to the reference link: Device trust, discovery and import using python requests - supported in Python version 2.7.9 and greater. https://devcentral.f5.com/s/articles/big-iq-trust-discovery-and-import-of-big-ip-using-rest-api-python-oo-973 Grants a license to a BIG-IP from an already-activated purchaced pool. https://devcentral.f5.com/s/articles/big-iq-licensing-of-big-ip-using-rest-api-python-oo-974 We will also be adding to github and will update this article once completed. So please come back and visit us soon and often for additional content.BIG-IQ Central Management API automation and programmability - BULK Discovery, Import and Re Import - Perl
Summary In conjunction with the announcement of BIG-IQ 5.0 we are excited to bring the field greater flexibility when centrally managing BIG-IP devices utilizing well defined workflows and the BIG-IQ iControl REST API. We understand that automation and programmability is becoming more the norm these days as the network is evolving into a software defined application aware environment. This article is an addendum to “BIG-IQ Central Management API automation and programmability – Python” and will demonstrate bulk device trust, discovery to populate resolver groups and import bigip configuration of many BIG-IP device as defined in a csv configuration file. This automation can be run as a standalone utilities that will run directly in the BIG-IQ shell for adding devices to inventory in a sequential yet automated fashion. API Reference Trust: https://'bigiq-ip'/mgmt/cm/global/tasks/device-trust Discovery: https://'bigiq-ip'/mgmt/cm/global/tasks/device-discovey Import: ADC - https://'bigiq-ip'/mgmt/cm/adc-core/tasks/declare-mgmt-authority Import: Firewall - https://'bigiq-ip'/mgmt/cm/firewall/tasks/declare-mgmt-authority Let’s get started – When using the BIG-IQ it is suggested to make a directory called scripts under /shared and securely copy this distribution into /shared/scripts/. Contents: ../config/bulk_discovery.csv ../src/perl/bulkDiscovery.pl ../src/perl/bulkReImport.pl Everything is predicated by a main loop which will invoke each task by calling supporting perl subroutines self-contained in the script. All rest calls, using curl (https://curl.haxx.se/), made are highlighted below. Establishment of device trust is completed in the main loop while the process of discovery and import configurations are completed in subroutine blocks within the script. #====================================================== # Main loop # Process trust, discovery, and imports #====================================================== for $bigip (@bigips) { my %postBodyHash = ("address"=>$bigiq, "userName"=>$user, "password"=>$pw,"clusterName"=>"", "useBigiqSync"=>"false", "name"=>"trust_$mip"); my $postBody = encode_json(\%postBodyHash); my $trustCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X POST -d \'$postBody\' https://localhost/mgmt/cm/global/tasks/device-trust"; if (discoverModules($bigiq, $machineId)) { if (importModules($bigiq)) { } } # upgrade the framework if nessasary if (handleFrameworkUpdade ($trustTask, $bigip)) { } } end of all devices #====================================================== # Discover specified modules. #====================================================== sub discoverModules { my %postBodyHash = ("moduleList" => \@moduleList, "status" => "STARTED"); # POST a new discovery task $postBodyHash{"deviceReference"}{"link"} = "cm/system/machineid-resolver/$machineId"; my $newDiscoverTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X POST -d \'$postBodyJson\' \"https://localhost/mgmt/cm/global/tasks/device-discovery\""; } end of discoverModules #====================================================== # A subroutine for importing individual module. #====================================================== sub importModule { # POST a new import task $postBodyHash{"deviceReference"}{"link"} = "cm/system/machineid-resolver/$machineId"; my $postBody = encode_json(\%postBodyHash); my $newImportTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X POST -d \'$postBodyJson\' \"$dmaUrl\""; # if we ecounter conflicts, we mark them to use BigIQ, patch the task back to started, and poll again if (($currentStep eq "PENDING_CONFLICTS") or ($currentStep eq "PENDING_CHILD_CONFLICTS")) if (resolveConflicts($mip, $module, $currentStep, $importSelfLink, @conflicts)) } # end of importModule #========================================================== # sub routine calling importModule to colocate all modules #========================================================== sub importModules { $ltmSuccess = importModule($mip, "ltm", "https://localhost/mgmt/cm/adc-core/tasks/declare-mgmt-authority", %postBodyHash); $asmSuccess = importModule($mip, "asm", "https://localhost/mgmt/cm/asm/tasks/declare-mgmt-authority", %postBodyHash); } And last but not least Re Import of BIGIP configuration objects for greater than one BIGIP device. This script can be run periodically based on Linux cron to ensure your device configurations managed by BIGIQ are up to date. On occasion other Element Management Systems could modify BIGIP object base and BIGIQ should be aware of these changes. If you refer to the below main loop, the discovery and import call's are the same. So two things actually happen that differs from inital bulk discovery and import. 1. Trust establishment is removed as it already contains mutaul certificate trust. 2. We test if the discovery and import tasks exists, if they do we can just PATCH discovery and import tasks to enforce a re import. That's about it. Refer to the code snippet below. #====================================================== # Main loop # Process Re Discovery, and Imports #====================================================== for $bigip (@bigips) { ## get the device properties my $deviceCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X GET https://localhost/mgmt/shared/resolver/device-groups/cm-bigip-allBigIpDevices/devices"; ## call disc routine using ip and machine id. if (discoverModules($bigiq, $machineId)) { ## call import routine using up and machine id. if (importModules($bigiq, $machineId)) } } } # end for devices Just to re iterate the above the discovery and import routines used for Re Import just PATCH the existing task created during inital discovery and import. Here are the PATCH requests. #====================================================== # Discover specified modules. #====================================================== sub discoverModules { ## get the discovery task based on the machineId my $findDiscoverTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X GET \"https://localhost/mgmt/cm/global/tasks/device-discovery?\\\$filter=deviceReference/link+eq+\'*$machineId*\'+and+status+eq+\'FINISHED\'\""; ## If it exists PATCH the task if (defined $discoveryTask->{"items"}[0]) { # PATCH the existing discovery task my $discoveryTaskSelfLink = $discoveryTask->{"items"}[0]->{"selfLink"}; $postBodyJson = encode_json(\%postBodyHash); my $discoverCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X PATCH -d \'$postBodyJson\' $discoveryTaskSelfLink"; } #====================================================== # A subroutine for importing individual module. #====================================================== sub importModule { ## get import task based on the machineid my $findImportTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X GET \"$dmaUrl?\\\$filter=deviceReference/link+eq+\'*$machineId*\'\""; ## If exists PATCH the task if (defined $findImportTask->{"items"}[0]) { # PATCH the existing import task $importTaskLink = $findImportTask->{"items"}[0]->{"selfLink"}; my $importCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X PATCH -d \'$postBodyJson\' $importTaskLink"; } #======================================== # sub routine for calling importModule to collocate all modules. #======================================== sub importModules { $ltmSuccess = importModule($mip, $machineId, "ltm", "https://localhost/mgmt/cm/adc-core/tasks/declare-mgmt-authority", %postBodyHash); $asmSuccess = importModule($mip, machineId, "asm", "https://localhost/mgmt/cm/asm/tasks/declare-mgmt-authority", %postBodyHash); } If you are interested in this code for collaboration or solution, seach on key words "bigiq" "api" "python" or "perl" in code share section on dev central or here is the reference link: https://devcentral.f5.com/s/articles/big-iq-big-ip-rest-api-bulk-device-discovery-perl-972 We will also create a repository on github for easy accessability. Please visit us soon and often for periodic updates.
WordPress REST API Vulnerability: Violating Security’s Rule Zero
It's an API economy. If you don't have an API you're already behind. APIs are the fuel driving organizations' digital transformation. We've all heard something similar to these phrases in the past few years. And while they look like marketing, they taste like truth. Because APIs really are transforming organizations and have taken hold as the de facto method of integration - internally, externally, laterally, and vertically. APIs enable mobile apps and things, web apps and middleware to communicate, collaborate, and extricate digital gold (that's data, by the way). So when there’s a vulnerability discovered in an API, it turns heads. Especially if it’s a vulnerability that is easily exploitable (this one is) and affects a significant number of publicly accessible resources (it does). #WordPress REST API vulnerability is already abused in defacement campaigns https://t.co/5i9pYKW4tH by @danielcid If you’re running WordPress versions 4.7.0 or 4.7.1 you are vulnerable. Stop reading this and go patch it right now. If you can’t for some reason and you’ve got a BIG-IP ASM you can apply these rules right now to protect vulnerable sites. I’m not kidding. I’ll wait right here. Okay, now that we’ve got that out of the way, I want to talk about APIs and security for a moment because there seems to be some general misunderstandings about REST APIs and security that this vulnerability just happens to illustrate perfectly. HTTP: The Foundation on which REST APIs are Built REST stands for Representational State Transfer. Don’t worry too much about what that means, because what you really need to understand is that it’s an architectural style. If you were going to list it along with other methods of achieving the same thing (transfer of data between two endpoints) you might list: RPC, CORBA, and SOAP. A REST API call is an HTTP request where the URI endpoint is typically indistinguishable from a web URI. The request looks the same. Really, which of these two URIs is a call to an API: GET /w/thing/snowmobile/id GET /a/thing/snowmobile/id See what I mean? No difference from outside. There’s no standard out there that requires a REST API contain some identifying attribute or HTTP header that makes it different than a traditional web request. The Content-Type suggested by the JSON specification is application/json but like the Pirate Code, those are more guidelines than actual rules. Thus, one cannot rely solely on Content-Type to identify a REST API from a standard HTTP request. In fact, good old x-www-form-urlencoded is often used to invoke API calls from clients. For example, here’s an HTTP request to an Express-based API I’m working on (for fun, cause I still do that) using Postman: GET /api/user/1 HTTP/1.1 Host: 192.168.0.57:8080 Content-Type: application/x-www-form-urlencoded Cache-Control: no-cache Postman-Token: 6d2247a1-9923-4774-6b86-7ba334bd497e And here’s one that does a POST, to send data to the API endpoint: POST /api/login HTTP/1.1 Host: 192.168.0.57:8080 Content-Type: application/x-www-form-urlencoded Cache-Control: no-cache Postman-Token: 32a29369-5d9d-6952-7de6-8aa4782d694d name=Webmistress&passwd=xxxx Now, these are API calls. And I’m betting that you noticed a couple of things: The REST API uses HTTP verbs like GET, POST, PUT, and DELETE. The URI is – and hold on to your hats now – a standard HTTP URI. The Content-Type does not help us determine whether this is a request to an API or a traditional web app. These first two points are really important because it’s this basic foundation that lets us breathe (a little) easier when it comes to securing APIs. Because we already know a whole lot about HTTP and the myriad ways in which it can be exploited. REST has become the de facto standard for communication because it’s far simpler than its predecessors and it relies solely on well-understood protocols and platforms, primarily HTTP. Building a REST API, then, means using HTTP to define a set of interfaces through which mobile apps, things, and web apps will communicate. These interfaces, in aggregate, comprise an API. And basically, they’re a set of URI endpoints invoked via HTTP and executed on by a server-side application*. The fact that the architectural design is REST, and it’s an API, is irrelevant. The same code could have been written to support a traditional web-based system, because the problem isn’t with the API or REST, it’s with the code that’s processing the input provided. The WordPress vulnerability is not a vulnerability in its API, per se, but in the way the API implementation handles standard, well-understood HTTP mechanisms. API implementations are often accomplished via frameworks (like Express, can you tell I’m a fan?) that take the tedium out of splitting apart the URI and distilling the paths into “routes” that are then used to call the appropriate function. In addition, this code is responsible for grabbing the query parameters (everything that comes after the ? in a URI, including the key-value pairs) and stuffing them into variables that can be used to do things like update databases, retrieve forum posts, and insert new content into the system. In this case, the back-end code for the API improperly handles those variables (including some poorly considered authorization logic), failing to properly validate and sanitize it. In Dungeons and Dragons we have the concept of “Rule Zero” and it comes before every other rule out there. It is foundational. There is a similar Rule Zero in security, and it is this: THOU SHALT NOT TRUST USER INPUT. EVER. It is Rule Zero that has been violated and thus introduces this vulnerability to WordPress. Now, you can certainly dig into the details and walk through the code (I like the type-casting logic that really makes this vulnerability work and the lazy sanitization attempt) but the point here is that REST API security relies a lot on the same, well-understood WEB security long preached by OWASP and every other security vendor out there. In fact, one could say it starts with strong, basic web security, the foundation of which is built upon rule zero. Yes, there are other concerns with APIs with respect to security, and REST introduces some of them because it eschews state. That means it requires a more active or external means of authentication and authorization. And many security folks are still getting up to speed on JSON, as are many of the security services inserted into the data path between endpoints that inspect, scan, and scrub app layer data. But if you’re trying to figure out where to start securing APIs, one of the best places is back to the beginning by laying down a strong set of secure coding practices that begins with Security’s Rule Zero. * Increasingly server-side applications are microservices and in the case of APIs, serverless (Function as a Service) is also gaining traction. And still they rely on endpoint invocation via an HTTP URI. Just sayin. The more things change, the more the stay the same.Generation of Utility Billing Report using BIG-IQ's API
This script generate the utility billing report based of a list of regKey.This script only applies to utility-type licenses used for F5's subscription and/or ELA programs. The regkey listed in the csv file or in the command line are the registration key associated with the license pool you wish to report on. Reports can be automatically submitted to F5 or manually created. /!\ This feature will be included with BIG-IQ 6.1 /!\ Automatic report submission requires BIG-IQ to access api.f5.com. BIG-IQ makes a REST call over SSL to api.f5.com to upload the report. Manual report submission is used in cases where BIG-IQ cannot reach api.f5.com. In this workflow, the customer generates the report, extracts it, then emails it to F5 (/shared/scripts/F5_Billing_License_Report.<date>.json). Pre-Requisist: Make sure you can access api.f5.com port 443 if automatic reporting is chosen (nslookup api.f5.com, ping api.f5.com, telnet api.f5.com 443). The script has to be executed locally from the BIG-IQ. Installation: The script mustbe installed in BIG-IQ under /shared/scripts: # mkdir /shared/scripts # chmod +x /shared/scripts/licenseUtilityReport.pl Allowed command line options: -h Help -c Path to CSV file with all regKey - REQUIRED if not using -k -k regKey(s) separated by , - REQUIRED if not using -c -r Report option automatic or manual - OPTIONAL (default is automatic) Command example automatic report: # cd /shared/scripts # ./licenseUtilityReport.pl -k DRLPZ-JISKU-VPUPT-HZMMV-LERVPYQ,GYCWI-FOUEZ-YMWPX-LYROB-PXTKMTG CSV file example manual report: # ./licenseUtilityReport.pl -c listregkey.csv-r manual # cat listregkey.csv DRLPZ-JISKU-VPUPT-HZMMV-LERVPYQ GYCWI-FOUEZ-YMWPX-LYROB-PXTKMTG Cron table example (every 1st of the month at 10am) for both above examples: # crontab -e (edit: type 'o' to add a new line, then 'ESC' and ':x' to save and quit) # crontab -l (list) 0 10 1 * * /usr/bin/perl /shared/scripts/licenseUtilityReport.pl -k DRLPZ-JISKU-VPUPT-HZMMV-LERVPYQ 0 10 1 * * /usr/bin/perl /shared/scripts/licenseUtilityReport.pl -c /shared/script/listregkey.csv-r manual ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday; │ │ │ │ │ 7 is also Sunday on some systems) │ │ │ │ │ │ │ │ │ │ ** * * * In case you need toobfuscate the Utility Billing Report before sending it, use manual export, then use the following script to mask IP/Macaddresses and Hostnames from the report. Then submit/email report to F5 (Please contact F5 Support for assistance). # ./f5_sanitize_usage_report.sh F5_Billing_License_Report.<date>.json Location of the scriptson GitHub:https://github.com/f5devcentral/f5-big-iq-pm-team
