cancel
Showing results for 
Search instead for 
Did you mean: 
carldubois_1031
Historic F5 Account

Summary

In conjunction with the announcement of BIG-IQ 5.0 we are excited to bring the field greater flexibility when centrally managing BIG-IP devices utilizing well defined workflows and the BIG-IQ iControl REST API. We understand that automation and programmability is becoming more the norm these days as the network is evolving into a software defined application aware environment.

This article is an addendum to “BIG-IQ Central Management API automation and programmability – Python” and will demonstrate bulk device trust, discovery to populate resolver groups and import bigip configuration of many BIG-IP device as defined in a csv configuration file.

This automation can be run as a standalone utilities that will run directly in the BIG-IQ shell for adding devices to inventory in a sequential yet automated fashion.


API Reference

Trust: https://'bigiq-ip'/mgmt/cm/global/tasks/device-trust

Discovery: https://'bigiq-ip'/mgmt/cm/global/tasks/device-discovey

Import: ADC - https://'bigiq-ip'/mgmt/cm/adc-core/tasks/declare-mgmt-authority

Import: Firewall - https://'bigiq-ip'/mgmt/cm/firewall/tasks/declare-mgmt-authority


Let’s get started –

When using the BIG-IQ it is suggested to make a directory called scripts under /shared and securely copy this distribution into /shared/scripts/.


Contents:

../config/bulk_discovery.csv

../src/perl/bulkDiscovery.pl

../src/perl/bulkReImport.pl


Everything is predicated by a main loop which will invoke each task by calling supporting perl subroutines self-contained in the script. All rest calls, using curl (https://curl.haxx.se/), made are highlighted below.

Establishment of device trust is completed in the main loop while the process of discovery and import configurations are completed in subroutine blocks within the script.

#======================================================
# Main loop
# Process trust, discovery, and imports
#======================================================
for $bigip (@bigips) {
   my %postBodyHash = ("address"=>$bigiq, "userName"=>$user, "password"=>$pw,"clusterName"=>"", "useBigiqSync"=>"false", "name"=>"trust_$mip");
  
   my $postBody = encode_json(\%postBodyHash);

   my $trustCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X POST -d \'$postBody\' 
   https://localhost/mgmt/cm/global/tasks/device-trust";

          if (discoverModules($bigiq, $machineId)) {
             if (importModules($bigiq)) {
              }
          }
        
          # upgrade the framework if nessasary
          if (handleFrameworkUpdade ($trustTask, $bigip)) {
          }
} end of all devices
 
#======================================================
# Discover specified modules.
#======================================================
sub discoverModules {
    my %postBodyHash = ("moduleList" => \@moduleList, "status" => "STARTED");

    # POST a new discovery task
    $postBodyHash{"deviceReference"}{"link"} = "cm/system/machineid-resolver/$machineId";

    my $newDiscoverTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X POST  -d \'$postBodyJson\' \"https://localhost/mgmt/cm/global/tasks/device-discovery\"";
} end of discoverModules

#======================================================
# A subroutine for importing individual module.
#======================================================
sub importModule {

# POST a new import task
$postBodyHash{"deviceReference"}{"link"} = "cm/system/machineid-resolver/$machineId";

my $postBody = encode_json(\%postBodyHash);

my $newImportTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X POST  -d \'$postBodyJson\' \"$dmaUrl\"";
   
# if we ecounter  conflicts, we mark them to use BigIQ, patch the task back to started, and poll again

if (($currentStep eq "PENDING_CONFLICTS") or ($currentStep eq "PENDING_CHILD_CONFLICTS"))
    if (resolveConflicts($mip, $module, $currentStep,     $importSelfLink, @conflicts))
} # end of importModule

#==========================================================
# sub routine calling importModule to colocate all modules 
#==========================================================
sub importModules {
  $ltmSuccess = importModule($mip, "ltm", "https://localhost/mgmt/cm/adc-core/tasks/declare-mgmt-authority", %postBodyHash);
  $asmSuccess = importModule($mip, "asm", "https://localhost/mgmt/cm/asm/tasks/declare-mgmt-authority", %postBodyHash);
}

 

And last but not least Re Import of BIGIP configuration objects for greater than one BIGIP device.

This script can be run periodically based on Linux cron to ensure your device configurations managed by BIGIQ are up to date. On occasion other Element Management Systems could modify BIGIP object base and BIGIQ should be aware of these changes.

If you refer to the below main loop, the discovery and import call's are the same. So two things actually happen that differs from inital bulk discovery and import.

1. Trust establishment is removed as it already contains mutaul certificate trust.

2. We test if the discovery and import tasks exists, if they do we can just PATCH discovery and import tasks to enforce a re import.

That's about it. Refer to the code snippet below.

#======================================================
# Main loop
# Process Re Discovery, and Imports
#======================================================
for $bigip (@bigips) {
    ## get the device properties
    my $deviceCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X GET https://localhost/mgmt/shared/resolver/device-groups/cm-bigip-allBigIpDevices/devices";
               ## call disc routine using ip and machine id.
               if (discoverModules($bigiq, $machineId)) {
                 ## call import routine using up and machine id.
                 if (importModules($bigiq, $machineId))         
                 }
               }
} # end for devices 

Just to re iterate the above the discovery and import routines used for Re Import just PATCH the existing task created during inital discovery and import. Here are the PATCH requests.

#======================================================
# Discover specified modules.
#======================================================
sub discoverModules {
    ## get the discovery task based on the machineId
    my $findDiscoverTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X GET \"https://localhost/mgmt/cm/global/tasks/device-discovery?\\\$filter=deviceReference/link+eq+\'*$machineId*\'+and+status+eq+\'FINISHED\'\"";

    ## If it exists PATCH the task 
    if (defined $discoveryTask->{"items"}[0])
    {
        # PATCH the existing discovery task
        my $discoveryTaskSelfLink = $discoveryTask->{"items"}[0]->{"selfLink"};
        $postBodyJson = encode_json(\%postBodyHash);
        my $discoverCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X PATCH -d \'$postBodyJson\' $discoveryTaskSelfLink";
}

#======================================================
# A subroutine for importing individual module.
#======================================================
sub importModule {
     ##  get import task based on the machineid
     my $findImportTaskCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X GET \"$dmaUrl?\\\$filter=deviceReference/link+eq+\'*$machineId*\'\"";


   ## If exists PATCH the task
   if (defined $findImportTask->{"items"}[0])
    {
        # PATCH the existing import task
        $importTaskLink = $findImportTask->{"items"}[0]->{"selfLink"};

        my $importCmd = "curl -s -k -u $bigiqCreds -H \"$contType\" -X PATCH -d \'$postBodyJson\' $importTaskLink";
}

#========================================
# sub routine for calling importModule to collocate all modules.
#========================================
sub importModules {
$ltmSuccess = importModule($mip, $machineId, "ltm", "https://localhost/mgmt/cm/adc-core/tasks/declare-mgmt-authority", %postBodyHash);
 $asmSuccess = importModule($mip, machineId, "asm", "https://localhost/mgmt/cm/asm/tasks/declare-mgmt-authority", %postBodyHash);

}

If you are interested in this code for collaboration or solution, seach on key words "bigiq" "api" "python" or "perl" in code share section on dev central or here is the reference link: 

https://devcentral.f5.com/s/articles/big-iq-big-ip-rest-api-bulk-device-discovery-perl-972

We will also create a repository on github for easy accessability. Please visit us soon and often for periodic updates.

 

Comments
Andrew_Tuttle
Nimbostratus
Nimbostratus

Hi Carl,

 

Bingo on the need for automation. We are looking for automated re-import capability on BIG-IQ version 5.x, so I was happy to see these scripts. Overall the bulk discovery script seems to work well, but the bulk re-import (the one I really need) has a couple of issues I'm trying to work through.

 

  1. Wrong BIG-IP instance is re-imported -- When my BIG-IQ contains several BIG-IP instances, and I want to re-import one of those instances only, I edit the CSV file to contain info for that instance. But when I execute the script, the first BIG-IP instance in the list is re-imported, not the BIG-IP instance I entered into the CSV file.
  2. I want to take advantage of the re-import as a periodic sync mechanism between BIG-IP instances and BIG-IQ. But when I change a rule on the target BIG-IP that is first in the list (I have to use the first because of the problem in 1) and execute the re-import, it doesn't identify the difference between BIG-IP and BIG-IQ. The log says 0 conflicts. However if I use the bulk discovery script to discover this BIG-IP from scratch, I do see the conflict that I generated for this test purpose identified in the bulk discover script's log.

Any assistance would be much appreciated. Thanks!

 

-- Andrew

 

carldubois_1031
Historic F5 Account

Hey Andrew, Just seeing this. Let me work through this and get back to you.

 

carldubois_1031
Historic F5 Account

Yes, it looks like a bug in the code. The way the script manages declare-mgmt-auth tasks that exist for each device instance look to be wrong. I'll fix this and publish a new copy. Sorry Andrew.

 

Andrew_Tuttle
Nimbostratus
Nimbostratus

Carl, thanks a bunch for looking into it!

 

carldubois_1031
Historic F5 Account

Hi Andrew, Just submitted a fix for the bug you described. Can you give it a try and let me know if this works for you? Did some minimal edge testing so your feedback would be great.

 

Thanks again, Carl

 

Andrew_Tuttle
Nimbostratus
Nimbostratus

Hi Carl, Thanks for the update! I am seeing that bulk re-import PERL script is now re-discovering the correct BIG-IP device, but the re-import of the modules (LTM & AFM are of interest to us) is not working. My evidence is that when I look on the BIG-IQ WebUI under services for the BIG-IP of interest, the timestamp for re-discovery is a current time, but for re-import, for both LTM and AFM, the timestamp is far in the past.

 

What additional feedback would be most helpful to you at this time?

 

carldubois_1031
Historic F5 Account

Yes, It looks like things changed a bit. Can you give bulkReImport_51.pl a try. PATCH of the import task was deprecated. Had to add some new code to do the right thing.

 

Hope it works for you, Carl

 

Andrew_Tuttle
Nimbostratus
Nimbostratus

Thanks a million Carl! This one works great! -- Andrew

 

Jason_Adams
F5 Employee
F5 Employee

Hi Carl,

 

This is really a great script, and I see that we have actually implemented the 'bulkImport.pl' script into our official upgrade guide and are hosting it on .

 

Do you have any intention of making bulkReImport.pl work for v5.3?

 

goutham
Nimbostratus
Nimbostratus

0691T000006AqpuQAC.png

 

Hello Carl/Andrew/Jason,

 

Could you guys please help me guide how to run this script successfully? I mean I am able to run the script from the BIGIQ but its failing to discover any modules on BIGIP LTM. Its returning with a failure. Do we need to enable anything on BIGIQ or BIGIP to get make this call happen?

 

Here is the output. It doesnt give much details as to whats causing the discovery failure. And I have no programming/Automation knowledge. My management wants to get this script implemented as early as possible as a cron Job. First I need to test this script successfully before I can get that cron job created. I have attached the output in a screenshot to this comment. Any help is greatly appreciated.

 

I am skipping trust establishment as the devices were already added to BIGIQ manually along time ago.

 

Simon_Lodge
Nimbostratus
Nimbostratus

Hi Carl,

 

We're using the bulk import script to overwrite config changes in the BIGIQ DB with changes made locally on the LTMs themselves (we're still not quite at the point where we can manage all configs through BIGIQ), but I notice your most recent script version doesn't support the option to choose BIGIP/BIGIQ when resolving conflicts - any chance this option could be re-added?

 

Many thanks,

 

Simon

 

RomanJ
F5 Employee
F5 Employee

The perl script isn't supported since 6.1. We have created an Ansible module instead.

https://docs.ansible.com/ansible/latest/modules/bigiq_device_discovery_module.html

 

There are still APIs to do the discovery and import which can be found in the API reference doc or directly in the ansible module python files if you look on the repo. Ansible is using the BIG-IQ APIs anyway.

 

https://github.com/F5Networks/f5-ansible/blob/devel/library/modules/bigiq_device_discovery.py

 

https://clouddocs.f5.com/products/big-iq/mgmt-api/latest

 

Version history
Last update:
‎29-Jun-2016 09:19
Updated by:
Contributors