python and iControl to verify load balancer configurations

Title...

Using python and iControl API to automatically verify F5 LTM LB VIP configurations....

tags: python iControl pyControl suds verify verifying validating configuration settings F5 LTM load balancer VIP automation provisioning

Problem...

If you're in a large company, you might have a shared services model for IT services like networking. You work with several centralized IT teams to implement firewall ACLs, Load Balancer VIPs, storage, etc.. Unless you have a solid provisioning process, and request forms that translate into implementation scripts, then you've likely run into issues resulting from human error. This is unavoidable if new requests are created manually through the UI. Wrong protocols, mismatching ports, incorrect pool members, wrong vlan / subnets, incorrect IPs, or incorrect partitions. The list goes on.

Provisioning errors introduce days to weeks of delays for new requests. They have to go through verification, troubleshooting, additional rounds of approval, scheduling for change control and QA, particularly for compliance environments. This only gets worse if there are many individuals or time zones between the person making the request and the person implementing the change. If you're migrating datacenters, and have multiple environments, this will likely leave your project managers very unhappy.

An automated provisioning process would certainly help, but can be expensive to build and implement. A short-term fix is to provide network engineers (those implementing the request) with a means of automatically verifying the request was implemented correctly. This will significantly reduce the turnaround time for new requests, saving time for both you and your networking team. This will also allow junior staff or interns to verify the work and free up senior staff.

Solution...

What's needed is a script or application that can load and verify a dataset containing new or existing F5 LTM load balancer request definitions. This script will read the dataset, store these in structured data types, and use the iControl API to verify the requests have been implemented properly. The output of the scripts will alert on misconfigured load balancer settings. This can be run by anyone on your ops or networking team, and can even be set up in Jenkins for single-click testing / reporting / alerting / history. Similarly this can be implemented as JUnit / TestNG, or puppet / chef / etc. Your network engineer or intern ops engineer can now verify implementations at the click of a button, immediately after the network engineer completes their work.

NOTE: Once this is in place, you will find that you're already half-way there to automatic provisioning, as the same definition files can be used to generate the implementation scripts.

To do this you'll need the following:

• pyControl + suds (or java + xml libs if you go that route)
• iControl SDK
• a read-only user that is available to networking and operations staff

Here's a list of what our scripts do:

1. read vip names to check from a file (including the LTMs they reside on)
2. connect to each of LTMs referenced in the load balancer requests
3. load the LTM's virtual server and pool member configuration via iControl
4. for each virtual server and pool, verify LTM configuration matches the request and expected values
5. output related information and alert or warn on any mismatches or missing VIPs

Here's a list of the types of things we check for:

... General ...

• check version of iControl modules on the LTM, alert if mismatch (script might need to be updated)

... Virtual Server ...

• exists on the LTM we expect it to
• was created in the correct partition for the BU/FG/application
• name matches the expected pattern (), where is typically fqdn or vip
• ip address of LB VIP matches
• virtual server port matches virtual server name (thisvip_80 is configured for port 80, thisvip_443 is 443, etc.)
• resource pool name matches virtual server (pool name matches virtual server name, including ports)
• state is enabled
• correct protocol
• http/https profile
• connection limits
• source NAT enabled / disabled
• SSL offloading enabled / disabled
• vlan matches (if used)
• all other virtual server settings

... Pools and Pool Members ...

• display pool member IPs and ports in output
• pool member ports match virtual server / vip (80 to 80, 443 to 443)
• pool member IPs match expected for LB VIP (from input)
• pool options match default profile (from input)
• health check matches virtual server type (TCP only)
• timeouts are correct
• minimum number of members matches
• minimum member action matches
• slow ramp time matches
• members are reporting as available

... Other ...

• perform forward and reverse DNS checks on virtual server VIP
• keep script execution history / output for reporting purposes

My first draft of such a script in python served us well, if a bit rudimentary. It can run from Jenkins and a network engineer can click on the job to verify all VIPs in our environment (including the new ones they just added). If they see they made a mistake, they can just fix it and rerun the script (rinse and repeat until no more errors). If I schedule that Jenkins job to run daily, I get an alert if any of the LB VIPs or pool members are misconfigred. The same can be done for firewall requests.

Next step is to add LB VIP profiles and refactor the code so it can be shared.