python and iControl to verify load balancer configurations
Title... Using python and iControl API to automatically verify F5 LTM LB VIP configurations.... tags: python iControl pyControl suds verify verifying validating configuration settings F5 LTM load balancer VIP automation provisioning Problem... If you're in a large company, you might have a shared services model for IT services like networking. You work with several centralized IT teams to implement firewall ACLs, Load Balancer VIPs, storage, etc.. Unless you have a solid provisioning process, and request forms that translate into implementation scripts, then you've likely run into issues resulting from human error. This is unavoidable if new requests are created manually through the UI. Wrong protocols, mismatching ports, incorrect pool members, wrong vlan / subnets, incorrect IPs, or incorrect partitions. The list goes on. Provisioning errors introduce days to weeks of delays for new requests. They have to go through verification, troubleshooting, additional rounds of approval, scheduling for change control and QA, particularly for compliance environments. This only gets worse if there are many individuals or time zones between the person making the request and the person implementing the change. If you're migrating datacenters, and have multiple environments, this will likely leave your project managers very unhappy. An automated provisioning process would certainly help, but can be expensive to build and implement. A short-term fix is to provide network engineers (those implementing the request) with a means of automatically verifying the request was implemented correctly. This will significantly reduce the turnaround time for new requests, saving time for both you and your networking team. This will also allow junior staff or interns to verify the work and free up senior staff. Solution... What's needed is a script or application that can load and verify a dataset containing new or existing F5 LTM load balancer request definitions. This script will read the dataset, store these in structured data types, and use the iControl API to verify the requests have been implemented properly. The output of the scripts will alert on misconfigured load balancer settings. This can be run by anyone on your ops or networking team, and can even be set up in Jenkins for single-click testing / reporting / alerting / history. Similarly this can be implemented as JUnit / TestNG, or puppet / chef / etc. Your network engineer or intern ops engineer can now verify implementations at the click of a button, immediately after the network engineer completes their work. NOTE: Once this is in place, you will find that you're already half-way there to automatic provisioning, as the same definition files can be used to generate the implementation scripts. To do this you'll need the following: pyControl + suds (or java + xml libs if you go that route) iControl SDK a read-only user that is available to networking and operations staff Here's a list of what our scripts do: read vip names to check from a file (including the LTMs they reside on) connect to each of LTMs referenced in the load balancer requests load the LTM's virtual server and pool member configuration via iControl for each virtual server and pool, verify LTM configuration matches the request and expected values output related information and alert or warn on any mismatches or missing VIPs Here's a list of the types of things we check for: ... General ... check version of iControl modules on the LTM, alert if mismatch (script might need to be updated) ... Virtual Server ... exists on the LTM we expect it to was created in the correct partition for the BU/FG/application name matches the expected pattern (), where is typically fqdn or vip ip address of LB VIP matches virtual server port matches virtual server name (thisvip_80 is configured for port 80, thisvip_443 is 443, etc.) resource pool name matches virtual server (pool name matches virtual server name, including ports) state is enabled correct protocol http/https profile connection limits source NAT enabled / disabled SSL offloading enabled / disabled vlan matches (if used) all other virtual server settings ... Pools and Pool Members ... display pool member IPs and ports in output pool member ports match virtual server / vip (80 to 80, 443 to 443) pool member IPs match expected for LB VIP (from input) pool options match default profile (from input) health check matches virtual server type (TCP only) timeouts are correct minimum number of members matches minimum member action matches slow ramp time matches members are reporting as available ... Other ... perform forward and reverse DNS checks on virtual server VIP keep script execution history / output for reporting purposes My first draft of such a script in python served us well, if a bit rudimentary. It can run from Jenkins and a network engineer can click on the job to verify all VIPs in our environment (including the new ones they just added). If they see they made a mistake, they can just fix it and rerun the script (rinse and repeat until no more errors). If I schedule that Jenkins job to run daily, I get an alert if any of the LB VIPs or pool members are misconfigred. The same can be done for firewall requests. Next step is to add LB VIP profiles and refactor the code so it can be shared.
pyControl on pypi.python.org
Hello, we have just recently started to use pyControl in our application, and it works as advertised with no problems so far. We have added pycontrol as a dependency to our application, but that makes it rather difficult to install it, since pyControl is not published at pypi.python.org. Instead, anyone who wishes to install our application has to come to devcentral.f5.com, register an account, find a download page for the right version of pyControl, download it, unpack, fix the setup.py file to actually make it work, copy it to the destination server and install there. That's a lot of work for such a simple application. We considered bundling pyControl to avoid all this hassle and allow for automated installation of our application, but that would deprieve us of any updates and bugfixes. The best solution for us would be to fix the setup.py installtion script and register the package at the Python Package Index, so that it can be automatically downloaded by pip or easy_install. Obviously, this should be done by the original author, if possible, but we are ready to do it by ourselves if you are not interested in doing that. Any thoughts on that?
Smart way to convert a PoolMemberDefinitionSequenceSequence to a IPPortDefinitionSequenceSequence
Hi, I want to dump out (and eventually read in and set) the enabled state of each of my GTM pools pool members. Its easy to get a list of the pools and pool members, but the PoolMember.get_enabled_state requires input parameters of the pools (simple) and a Common.IPPortDefinitionSequenceSequence. I'm having trouble figuring out the best way to programmatically get an IPPortDefinitionSequenceSequnce to pass in. I have a PoolMemberDefinition named poolmembers loaded as follows: wideips = wipobj.get_list() pools = poolobj.get_list() poolmembers = poolobj.get_member(pools) I want to get the enabled state of each of the poolmembers. I should be able to loop through the poolmembers object and strip out each poolmember address and port and save it to another object (which would then be of type IPPortDefinition), but is that the best way? thanks, andy
Setting VLAN on a virtual server
Pulling my hair out trying to enable a VLAN for a virtual server. This is what I'm doing -- "v" is the object reference to LocalLB.VirtualServer: vlan_filter_list = v.typefactory.create('Common.VLANFilterList') enabled_state = v.typefactory.create('Common.EnabledState') vlan_filter_list.state = enabled_state.STATE_ENABLED vlan_filter_list.vlans = ["external"] v.set_vlan([vip_name], [vlan_filter_list]) Unfortunately, it throws an exception: xml.sax._exceptions.SAXParseException: :1:49: syntax error Any ideas? I'm pretty stuck. :("basic auth failed" when creating a simple connection
Hello, When I run the code at the following link from the command line (ie. "python example.py") I get the exception "suds.transport.TransportError: HTTP Error 401: basic auth failed" http://pastebin.com/w60Vb9Ji What makes this so bizarre is that if I set up this same BIGIP instance in the interpreter, it works just fine. So I am unable to run this from a script. What is making this happen? How do I solve this? Thanks
01070621:3: Rule priorities error despite unique priorities !
Fellow pycontrol-freaks :) I've been on a coding marathon for 2 days and have had a fantastic time diving into pycontrol !! I'm trying to use it to dynamically add a maintenance irule and then remove it again based on a ruby script that's out on devcentral. However, I'm running into a wall finally and it smells like a bug or massively misleading API error to me. I know that icontrol is stricter than the gui and won't let me add 2 irules with the same priority except I can prove to you that it does anyway whereas I cannot add a 3rd iRule if 2 exist already no matter what the priorities are. Case in point. I have a test vs with no irules. I can use my pycontrol script to add a redirect rule no problem. *********************** SCENARIO 1 (works when it shouldn't) : [stucky@vito pycontrol]$ ./maintenance.py blablabla The iRule 'api_maintenance' already existed but got refreshed. Successfully added iRule 'api_maintenance' to virtual server '/Common/test' The irule is set to priority 1 : [root@stglbltm01:Active:Standalone] config tmsh show ltm rule api_maintenance --------------------------------------------- Ltm::Rule Event: api_maintenance:HTTP_REQUEST --------------------------------------------- Priority 1 Executions Now as per docs I should not be able do this again if another rule with pri 1 already exists. To test let's remove this rule again and manually add the test rule with pri 1. [root@stglbltm01:Active:Standalone] config tmsh list ltm virtual test ltm virtual test { destination ip:http ip-protocol tcp mask 255.255.255.255 profiles { http { } tcp { } } rules { test_hipri } [root@stglbltm01:Active:Standalone] config tmsh show ltm rule test_hipri ---------------------------------------- Ltm::Rule Event: test_hipri:HTTP_REQUEST ---------------------------------------- Priority 1 Executions Ok now let's try to use pycontrol to add the maintenance rule again: [stucky@vito pycontrol]$ ./maintenance.py blablabla The iRule 'api_maintenance' already existed but got refreshed. Successfully added iRule 'api_maintenance' to virtual server '/Common/test' Let's confirm we have both rules associated with the vs: [root@stglbltm01:Active:Standalone] config tmsh list ltm virtual test ltm virtual test { destination ip:http ip-protocol tcp mask 255.255.255.255 profiles { http { } tcp { } } rules { test_hipri api_maintenance } Now I don't mind if something works that shouldn't as much as I do when something doesn't work that should ! ************************** SCENARIO 2 (Doesn't work when it really should !!) Let's add 2 irules to this vs both with DIFFERENT priorities (other than 1 for sure !): [root@stglbltm01:Active:Standalone] config tmsh list ltm virtual test ltm virtual test { destination ip:http ip-protocol tcp mask 255.255.255.255 profiles { http { } tcp { } } rules { log_persistence_cookie filter_http_methods } We have pri 800 and pri 10. [root@stglbltm01:Active:Standalone] config tmsh show ltm rule log_persistence_cookie ---------------------------------------------------- Ltm::Rule Event: log_persistence_cookie:HTTP_REQUEST ---------------------------------------------------- Priority 800 Executions [root@stglbltm01:Active:Standalone] config tmsh show ltm rule filter_http_methods ------------------------------------------------- Ltm::Rule Event: filter_http_methods:HTTP_REQUEST ------------------------------------------------- Priority 10 Executions According to the docs I should have no problem adding a 3rd irule with priority 1 here via pycontrol so let's try : [stucky@vito pycontrol]$ ./maintenance.py blablab The iRule 'api_maintenance' already existed but got refreshed. No handlers could be found for logger "suds.client" Server raised fault: 'Exception caught in LocalLB::urn:iControl:LocalLB/VirtualServer::add_rule() Exception: Common::OperationFailed primary_error_code : 17237537 (0x01070621) secondary_error_code : 0 error_string : 01070621:3: Rule priorities for virtual server (/Common/test) must be unique.' This totally hoses me cause I cannot finish my maintenance script testing now. It's all moot if I can't get this rule added/removed on the fly. Please tell me I'm doing something wrong ! Keep up the great forum ! thxf5 connection string with pycontrol
I have written a class and defined some methods that work great so far. I am wondering however how I can get around having to pass in the connection string for wsdls on every call though??let me try to explain better: At present if I want to operate on lots of F5's, I have to do something like below and then feed into a for loop where the ip/hostname for the F5 is a varaible that can be iterated over. I have to do this inside each function though and it is ugly. I know this must not be the best way...... b = pc.BIGIP( hostname = 'ip', username = 'user', password = 'pass', fromurl = True, wsdls = ['LocalLB.ProfileClientSSL']) s = b.LocalLB.ProfileClientSSL However, it would be nicer if I could create a function for the connection to the F5 and call this in main. This way I can interate over a bunch of F5's to do different things and dont have to deal with calling the s. every time I want to do something. I cant figure out how to acomplish this because if I make the above a function, the variable s = b.LocalLB.ProfileClientSSL is not available to any other functions in the same class as its not global scope. There must be a better way to do this ... If I am working with a DB for example I can create a connect function and do whatever I want without this headache.... I know icontrol is different and I am still learning as I go, so maybe I am missing something here. thanks for any help.
Can't set 'always-send' property on cookie persistence profile
I'm trying to reproduce a configuration dumped thorugh tmsh using the iControl interface for the LTM. Here's the configuration I'm trying to create: ltm persistence cookie dev-shib-cookie-persistence { always-send enabled app-service none defaults-from /Common/cookie expiration 0 method insert override-connection-limit disabled } Here's what I'm able to create using the iControl API: ltm persistence cookie shib-dev-cookie-persistence { app-service none defaults-from /Common/cookie method insert } Here's my code: lb.ProfilePersistence.create([clean_persistence_name], ['PERSISTENCE_MODE_COOKIE']) lb.ProfilePersistence.set_default_profile([clean_persistence_name], ['/Common/cookie']) lb.ProfilePersistence.set_cookie_persistence_method([clean_persistence_name],[{'value':'COOKIE_PERSISTENCE_METHOD_INSERT', 'default_flag': False}]) I can't find anything on how to configure 'always-send' in the documentation at: https://devcentral.f5.com/wiki/iControl.LocalLB__ProfilePersisten ce.ashx Can you tell me how to set this parameter using the iControl API? I'm using the python bigsuds library if that matters. Thanks!
Adding a iRule to a Virtual Server
Hi, I was trying to write a script to allow me to add an existing iRule to an existing Virtual Server. I am receiving the following error when I run the script: 'Could not find element by name: rule_name'. Any help would be appreciated. Below is the script: !/usr/bin/env python import sys import pycontrol.pycontrol as pc b = pc.BIGIP( hostname = "F5l", username = "username", password = "Password", fromurl = True, wsdls = ['LocalLB.VirtualServer']) c = b.LocalLB.VirtualServer def convert_rule(z,rule): rule_seq = z.LocalLB.VirtualServer.typefactory.create('LocalLB.VirtualServer.VirtualServerRuleSequence') rule_set = z.LocalLB.VirtualServer.typefactory.create('LocalLB.VirtualServer.VirtualServerRule') rule_set.rule_name = rule rule_set.priority = 500 rule_seq = [rule_set] return rule_seq c.add_rule(['VIP'],convert_rule(b,'iRule'))