BIGdiff Version 2
This article is written by, and published on behalf of, DevCentral MVP Leonardo Souza. --- Introduction I wrote the first version of BIGdiff script almost 2 years ago, since that release I have been using the script and keeping a list of new functionalities and improvements I would like to add. BIGdiff Version 2 includes all the things on my wish list which I will describe in this article. First, if you have no idea what is BIGdiff, please read the article I wrote about version 1. https://devcentral.f5.com/s/articles/bigdiff-a-little-help-for-software-upgrades-31933 In this article I will only explain the new functionalities and improvements, so read the above article for the basics about the script. Use Cases When I wrote Version 1, the goal was to help with software upgrades, but I ended up using the script for any kind of change involving F5 devices. In Version 2, the scope is increased to include scenarios where you need to understand if something changed related to an F5 device. Because the script points you to each object that has changed, the change can be anything; for example, a firewall. You can basically, run the script before the firewall change is made, make the firewall change, and run the script after the firewall change is complete. The output will point you to problems for analysis and, if you conclude that the problem is not because of the F5 device, you can report problems to the application team at the time of the change rather than the next day. This immediacy will save you a lot of time troubleshooting and reduce the pain your users may experience. Associative Arrays If you run the Version 2 script while upgrading a device, the script will be extremely fast. As indicated in the article about Version 1, it just takes a couple of minutes to compare 13 thousand objects. That is because the script was optimized for upgrades, where the list of objects before and after have the same order. Internally the script will create an array (by type of object e.g., virtual server) listing objects before the change and another array listing objects after the change. The script then checks if the array index values match; if they have the same object name. If the index values match, like in the case of an upgrade, the search is done. If the index values do not match, the script scans the array with objects from after the change, this is a resource-intensive task and basically exponential. The fix to reduce the 'cost' of the exponential back down to a linear cost is to use an associative array, butassociative arrays were only introduced in Bash version 4. Version 2 of the BIGdiff script, supports versions starting from 11.0.0 but version 11.0.0 uses Bash 3. Somewhere in 11.5.x and 11.6.x, Bash was upgraded to version 4, and 12.0.0 uses Bash 4. As version 11.x.x is becoming less common now, it is time to add associative arrays to the script. However, the script will only use associative arrays if it detects that the current Bash version is 4 or higher. To give you an idea about the improvement in running times I created a 'worst-case scenario' where most of the objects have a new name and the script has to scan the whole array. Note: These times were while using my slow virtual lab, so you will definitely get faster times in a hardware device or a virtual device with more resources. Number of objects: 3398 BIGdiff v1: 1203 seconds BIGdiff v2: 596 seconds Time reduction: around 50% Number of objects: 6648 BIGdiff v1: 2868 seconds BIGdiff v2: 1319 seconds Time reduction: around 54% Number of objects: 9898 BIGdiff v1: 6865 seconds BIGdiff v2: 2219 seconds Time reduction: around 68% As the number of objects increases, so to does the reduction in time (by percentage) to run the script when compared with Version 1. Using associative arrays Bash will internally look for the object using a key, and that key is visible faster than scanning the entire array. DataTables BIGdiff shows the data with HTML tables. Version 1 used TableFilter (https://www.tablefilter.com/) to provide filtering functionality. Version 2 uses DataTables (https://datatables.net/) to provide filtering functionality. DataTables provides more functionality, including breaking the table in multiple pages. Also, now there is no need to download an extra file to provide filtering options because when loading the HTML file the browser will download the required files from CDN networks. Backup Version 2 now generates UCS/QKView/Logs. When you run the script it will automatically generate those files, but you can also opt to run the script without generating those files. You may also choose to generate the log files individually using the script. Object Names LTM does not support creating an object with a space in the name. However, GTM does accept spaces in names for most objects. The Version 2 script now accepts space in the object name for all objects. BIGdiff Remote I created another script called BIGdiff Remote that will automatically upload BIGdiff to the destination devices run the BIGdiff download the files created. You might use BIGdiff Remote when you are running BIGdiff against multiple devices, as the script will automate that task. Also, if you want to get a UCS/QKView/full logs from multiple devices, the script automates that activity as well. Conclusion You should use this BIGdiff script during an upgrade because you don’t lose anything and it will definitely help if things go wrong. I also suggest including the script as part of your change process, if you have one. Normally you (should) generate a UCS file before performing a change to an F5 device, so upgrade that process to use this script and it will not only generate the required UCS file but also QKView and full logs. All of this helps identify whether something was broken during your change. Check out the new version on DevCentrals' Codeshare: https://devcentral.f5.com/s/articles/bigdiff-1160BIGdiff - A Little Help For Software Upgrades
Published on behalf of DevCentral MVP Leonardo Souza If you have been to F5 Agility in Boston and went to my presentation, you should have already an idea of what I will talk about in this article, but you will learn more things, so continue reading. If you haven’t heard of BIGdiff yet, have you been living in Mars? Don’t worry I will explain what that is and how it can help you with software upgrades, and whatever you find useful. It is not an AI that will do the upgrade for you but will help you with the upgrade. Challenges These are the challenges BIGdiff addresses: You are upgrading a F5 device with 1,000 virtual servers and 1,000 wide IPs. How do you know if you have the same number of virtual servers and wide IPs after the upgrade? How do you know if you have the same number of available virtual servers and wide IPs after the upgrade? If the number of available virtual servers or wide IPs changed after upgrade, how can you find what changed? Existing Solutions First Challenge: There are multiple solutions already for this challenge. Both for LTM and GTM, you can take a print screen of the statistics before the upgrade and compare after the upgrade. For LTM, Statistics > Module Statistics > Local Traffic For GTM, Statistics > Module Statistics > DNS > GSLB This in 13.1.0.1, but I think this exist since v9, and will be in similar place in all versions. Qkview and iHealth combination. iHealth will show you configuration totals but is mainly LTM and does not show you GTM objects. Network Map is another option. However, network map is only for LTM. Also, that is a map that start from a virtual server, so if you have a pool that is not linked to a virtual server that will not count in the totals. Second Challenge: The statistics also tell you the status of the objects, so that solution works for both challenges. Third Challenge: There is no automated way to get this. You could run multiple tmsh commands to get the status before the upgrade, or just generate a qkview that will run those commands for you. However, you will still need to compare the objects one by one. If the only slot you got for the software upgrade was 3am in a Sunday, I am sure you will miss some objects or fall asleep. Solution I hope you are thinking about the same, computers don’t need to sleep, and they are better/faster than humans to compare 2 strings or numbers (that is basically 0 or 1, so they are not that smart). So, the conclusion is simple, let the computer do the work comparing objects while you drink another coffee to keep you awake to complete the software upgrade. The idea is simple, get the list of objects, and their respective status, before and after the software upgrade, then compare them and report the result. In this context, object is any entity that has a status in a BIG-IP device that may be affected by the software upgrade. Looking BIG-IP modules, that translate to LTM and GTM objects, for example, virtual servers and wide IPs. That is where BIGdiff script comes to help you and automate that process. You run BIGdiff before the upgrade, upgrade the device, and run again after the upgrade. The script will then generate a HTML file with the results. Technical Bits BIGdiff is a bash script and uses dialog program to generate the graphical menus. Dialog is a common program for CLI menus and is what F5 uses for the config command for example. The script uses snmpwalk to query locally the device for the object status, because so far has been the faster option I tested. That basically generates the same text file before and after the upgrade. Those text files that are used after to compare the objects. The script will generate the results in a HTML format, with tables. If something already exists and do the job well, there is no reason why not to use, so the script uses the TableFilter JavaScript library, that provide filter functionalities for HTML tables. You just need to have the JavaScript library file in the same folder that you have the HTML file, and the magic will happen. If you don’t need the filter functionality, you don’t need to have the JavaScript library, and static tables will be presented. The script is optimized to use mainly bash functionality, to be as faster as possible. I tested the script to compare 13 thousand objects, and it complete the task in a couple minutes. 13K objects is a really big configuration, so even if the device you plan to run the script has a large configuration, that will be just a couple minutes in your change window to run the script. Support The script only supports BIG-IP software, no support for EM or BIG-IQ. The reason is simple, there is no use case for those software. Versions 11.x.x/12.x.x/13.x.x/14.x.x were tested and are supported. As new versions are released, I will be testing to see if any change is needed to support that version. LTM objects are supported and will be listed even if LTM is not provisioned, as majority of the other modules do use LTM internally. GTM objects and partitions are also supported. Using BIGdiff Go to the code share link: BIGdiff Download the tablefilter.js file, if you want to use the table filter functionality as described above. Download the bigdiff.sh that is the script file. In the F5 device, create a folder in /shared/tmp, as /shared is shared between all volumes. Upload the file bigdiff.sh to the F5 device. Change the file permission to run: chmod +x bigdiff.sh Run the script: ./bigdiff.sh Run the script before the upgrade. Upgrade the F5 device. Run the script after the upgrade. Download the file ending in .html from the F5 device. Open the HTML file with your favourite browser. Make sure you have the tablefilter.js in the same folder as the HTML file, if you want the filter functionalities. Other Use Cases The reason I wrote the script was to help with the software upgrades, but you are not limited to software upgrades. You can use the script to compare the objects after you have done something, that can be an upgrade or something else. You can use the software for consolidations, for example 2 devices that will be replaced by a single device. You run the script in the old devices, merge the txt files that are created with the list of objects. Import the configuration in the new device, upload the script and merged txt file you created. Run the script in the new device, and the script will report to you if the objects have the same status as in the old devices. Another use case is for major changes. You can run the script, do the changes, and run the script again. The script will then tell you if you broke something. Silent Mode Silent mode is mainly to be used to integrate with other tools. The image above explains how to use. Conclusion Read the information in the code share page about know issues. I hope you find the script useful.
