21-Apr-2021 22:59 - edited 31-Jan-2023 11:08
Problem this snippet solves:
This is a script which will generate a report of the BIG-IP LTM configuration on all your load balancers making it easy to find information and get a comprehensive overview of virtual servers and pools connected to them.
This information is used to relay information to NOC and developers to give them insight in where things are located and to be able to plan patching and deploys. I also use it myself as a quick way get information or gather data used as a foundation for RFC's, ie get a list of all external virtual servers without compression profiles.
The script has been running on 13 pairs of load balancers, indexing over 1200 virtual servers for several years now and the report is widely used across the company and by many companies and governments across the world.
It's easy to setup and use and only requires auditor (read-only) permissions on your devices.
http://loadbalancing.se/bigipreportdemo/
The device overview:
Certificate details:
How to use this snippet:
This is the only branch we're updating since middle of 2020 and it supports 12.x and upwards (maybe even 11.6).
https://loadbalancing.se/2021/01/05/running-bigipreport-on-docker/
https://loadbalancing.se/2021/04/16/bigipreport-on-kubernetes/
Older version of the report that only runs on Windows and is depending on a Powershell plugin originally written by Joe Pruitt (F5)
https://loadbalancing.se/downloads/bigipreport-5.4.0-beta.zip
https://loadbalancing.se/downloads/f5-icontrol.zip
https://loadbalancing.se/bigip-report/
Written by DevCentral member Shann_P:
https://loadbalancing.se/2018/04/08/protecting-bigip-report-behind-an-apm-by-shannon-poole/
Still have issues? Drop a comment below. We usually reply quite fast. Any bugs found, issues detected or ideas contributed makes the report better for everyone, so it's always appreciated.
---
Join us on Discord: https://discord.gg/7JJvPMYahA
Code :
BigIP Report
Tested this on version:
12, 13, 14, 15, 16
Due to a platform corruption during the 2019 migration I have worked with team to move his original legacy codeshare to this new record (same URL).
The legacy codeshare is temporarily available at https://devcentral.f5.com/s/articles/bigip-report-old
The negative repercussions of this change are:
The positive repercussions of this are:
Thank you for your patience and persistence with Patrik's awesome contribution and thank you for your dedication to our community.
New release v5.5.9. Well done Tim!
I've updated the Kubernetes containers with the new code and also triggered the :latest tag for you Cowboys who likes to use that. 🙂
Got questions/feedback/an insatiable lust for nerd talk? Join us on Discord:
https://discord.gg/W2y2cFX7
Kind regards,
Patrik
Modules directory is missing from the zip file.
Line |
2255 | . .\modules\Get-ExpiredCertificates.ps1
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| The term '.\modules\Get-ExpiredCertificates.ps1' is not recognized as a name of a cmdlet, function, script
| file, or executable program. Check the spelling of the name, or if a path was included, verify that the path is
| correct and try again.
Hi Ben
Apologies, been out enjoying the sun in the afternoon.
Thanks for reporting, will fix the script that creates deploy and get back to you.
/Patrik
Strange. I downloaded the file after updating the build job and I can see the modules just fine:
Please confirm that this is the link you're using?
https://loadbalancing.se/downloads/bigipreport-v5.5.9.zip
/Patrik
That got it. Thanks!
Have you looked at port-lists and/or policies? For port-lists, the VIPs aren't contained in the virtual, but are in a traffic-matching-criteria object instead. I don't know how difficult it would be to add the logic to show the VIPs for those in bigipreport or any associated policies.
Here is an example config for the port-lists.
net port-list /Common/web_443-8443-8080_ports {
description web_443-8443-8080_ports
ports {
443 { }
8080 { }
8443 { }
}
}ltm pool /Common/pool_vip_portlist_example {
load-balancing-mode least-connections-node
members {
/Common/172.1.2.5:0 {
address 172.1.2.5
}
/Common/172.1.2.6:0 {
address 172.1.2.6
}
}
monitor /Common/https_basic_443
}
ltm traffic-matching-criteria /Common/vip_portlist_example_VS_TMC_OBJ {
destination-address-inline 1.2.3.4
destination-port-list /Common/web_443-8443-8080_ports
protocol tcp
source-address-inline 0.0.0.0
}
ltm virtual /Common/vip_portlist_example {
ip-protocol tcp
pool /Common/pool_vip_portlist_example
profiles {
/Common/fastL4 { }
}
serverssl-use-sni disabled
source-address-translation {
type automap
}
traffic-matching-criteria /Common/vip_portlist_example_VS_TMC_OBJ
translate-address enabled
translate-port enabled
vlans {
/Common/proxy-vlan
}
vlans-enabled
}
Hi Ben
Glad it helped. I added a feature request for you on Github. You can go here:
https://github.com/net-utilities/BigIPReport/issues/69
and then watch it for updates.
I'm afraid I can't give any timelines since both me and Tim does this on our spare time. I normally try to squash bugs pretty fast but features happens when time permits. 🙂
/Patrik
Dear all
I got an email from Docker-hub today that they will discontinue the automatic builds due to too many people abusing the service.
I'm not yet sure how this will affect the service. Since we don't build the application that often it might work if the manual build is still free.
Either way, I have applied for the bigipreport project to be approved as an Open Source account. If manual builds won't work and the application is denied we'll have to take it from there.
This is only relevant for those running the report using Docker and I will update here once I know more.
Kind regards,
Patrik
Hello,
I have tried today the Version 5.5.9, but I am not able to run script - there is an issue with device certifcate check:
PS C:\bigipreport-v5.5.9> .\bigipreport-v5.5.9.ps1
2021-07-22 11:34:47 Starting: PSCommandPath=C:\bigipreport-v5.5.9\bigipreport-v5.5.9.ps1 ConfigurationFile=C:\bigipreport-v5.5.9/bigipreportconfig.xml CurrentJob= Location= PSScriptRoot=C:\bigipreport-v5.5.9
2021-07-22 11:34:47 Successfully loaded config file: C:\bigipreport-v5.5.9/bigipreportconfig.xml
2021-07-22 11:34:47 Insecure SkipCertificateCheck enabled, consider using valid certificates and DNS names
ParentContainsErrorRecordException: C:\bigipreport-v5.5.9\bigipreport-v5.5.9.ps1:654
Line |
654 | $PSDefaultParameterValues.Add("Invoke-RestMethod:SkipCertific …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Exception calling "Add" with "2" argument(s): "The key 'Invoke-RestMethod:SkipCertificateCheck' has
| already been added to the dictionary."
PS C:\bigipreport-v5.5.9>
I believe the actual reason behind the error was that the script was executed twice in the same PSSession.
Kind regards,
Patrik
Here's a pre-release:
https://loadbalancing.se/downloads/bigipreport-v5.6.1.zip
If it works fine I'll update the version above.
Heads up to those running the report using Kubernetes.
My Dockerhub account was disabled and my application for the open source version has not been processed yet.
This means that you need to build and store the images yourself.
You can find the manifests here:
https://github.com/net-utilities/BigIPReport-Docker
Will also update the documentation to cover the pagination concept Tim added yesterday.
Good news! Docker has approved my request to include BigIPreport in their open source program.
This means that the docker/k8s containers will be available as soon as it's activated (1-2 weeks).
Kind regards,
Patrik
Dear Patrick,
I used version 5.6.2 and added my code to crawl the policy data as well.
I will try to share the code via github with you.
The F5 statement regarding the translation of the policy logic to readable code was that they cannot share this with me.
(Service Request #C3491339)
I started to create the logic for our most recent used policie configurations already.
Best regards,
Marius
Well done! Just some small things to remark on.
Also good input Tim! I added some comments too.
We're between houses so my lab will be packed into boxes until December so I can't test it myself.
Kind regards,
Patrik
I got an email regarding the kubernetes and docker builds and figured I should update here too. Dockerhub approved the open source project request as stated earlier but then it got stuck because of some internal Dockerhub stuff. Still working on solving this.
Meanwhile, the manifests are available on GitHub. If you are looking for them you probably know how it works but if this is not the case you’re more than welcome to join our discord channel if you have any support questions.
Hi hi Hi Jason
Can you please give me web server config which used in official web site training . I am learning LTM now
No update from Dockerhub regarding the open source project.
First they approved it but then they said there's a problem with my account. After that I have poked them multiple times but nothing happens. I've grown tired and now I pay for it myself from my own pocket.
Kubernetes containers are now available for the current version again.
Kind regards,
Patrik
Hi Ali
Glad you like it!
I'm afraid I don't use BIG IP DNS at work at the moment so I have no opportunity to develop this functionality.
Maybe in the future...
Kind regards,
Patrik
Hello Patrik.
I installed BIGIP Report 5.6.2. But I dont know how to use Get-ExpiredCertificates.ps1 with parameters. Can you explain to me with an example?
Kind Regards
Unal
The modules/* scripts are called as part of the normal run. You should configure the xml file to have your F5s listed, and a user/password credential to log into them. Then run the parent script. It will call each of the modules and output the report.
I just noticed 5.6.2 does not work well with port groups. It stops caching virtual servers as soon as it processes the first one using port groups. I have 1163 virtuals, it only processes the first 136.
2021-11-30 08:02:08 VERBOSE x.x.x.x:Caching Virtual servers
2021-11-30 08:02:10 VERBOSE x.x.x.x:Caching Virtual servers
2021-11-30 08:02:55 ERROR x.x.x.x:Unable to cache virtual servers: Cannot convert value "any6" to type "System.Net.IPAddres
s". Error: "An invalid IP address was specified." (line 1038)
2021-11-30 08:02:56 VERBOSE x.x.x.x:Detecting orphaned pools
2021-11-30 08:02:56 SUCCESS x.x.x.xStats: VS:136 P:937 R:113 DG:7 C:187 M:171 ASM:0 T:151.5807781
Hi there!
There's a pull request from Marius Bauer which includes support for policies. Both Tim and I are old school iRulers (yeah yeah, I know that policies are faster) so we would like to know if there's any users out there willing to test Marius branch?
If there is I'll release a beta package for testing.
Kind regards,
Patrik
Go ahead, we have a pre-prod server running LB-monitor as well. Marius is an ex-colleague of ours 🙂
We are having issues with the script though:
bigip-ext-abc-1.domain.com:Failed to get auth token
bigip-ext-bde-5.domain.com:Failed to get auth token
bigip-cloud-01.domain.com:Failed to get auth token
bigip-ext-bde-1.domain.com does not seem to have been indexed
bigip-ext-abv-5.domain.com does not seem to have been indexed
bigip-ext-abv.domain.com does not seem to have been indexed
These errors are raised is on a daily basis. Probably due to a temporary connection glitch/slow response. Any timeouts we can adjust to alleviate the problem?
Hi there!
If it's inconsistent it could also be poor connection/management provisioning being too small or the config too large. We're on discord if you want to discuss it/get help:
https://discord.gg/RzmjgneW
Kind regards,
Patrik
Well, we see error e-mails couple of times a day.
One of my colleagues is aware of the auth issue, we did some upgrades recently so this seems to be the root cause.
But indexing errors we see pretty much every day at least once, normally the same devices. Config is not big but connection might be interrupted cause they are sitting behind the Great Chinese Firewall and we get all sorts of crap from it 🙂 Is there any way to influence timeouts and retry-periods, eventually on a per-host basis?
Oh darn. I recognize the challenges with the Chinese firewall.
Lived in Wuhan for a year (before Covid) and there were constant challenges with getting outside the famous wall. Afraid that if it is the national firewall that gives you trouble there's not that much to do at the moment except for running a separate BigIPReport within the borders of China.
Since all objects are arrays you could even run a simple script to concatenate the data from the Chinese BigIPReport to the main report. I think this would be a bit more robust that relying on a few hundred API calls through the firewall and frankly very easy to do.
Hi Patrik, I'm using the Kubernetes version 5.6.4 (also tried 5.6.2) and running into a couple of issues:
I'm getting a invalid invite code from your Discord link.
Howdy!
Certificate information missing
Is the user that BigIP report is using auditor or above? Else it won't be able to read the certificate information.
Pair missing data
Sounds like the collector believes both devices are passive in which case it won't collect any data by default. Are you using multiple traffic groups on these devices by any chance?
It's a read only account. I'm only using one traffic group on that F5. I should mentioned that I'm using an older version, on 5.3.1 on a Windows Server and do not have these two issues there and I use the same account. Perhaps it's a SOAP/REST difference?
Perhaps it's a SOAP/REST difference?
Good guess! It is. SOAP got more permissions with a read-only role than REST does.
It's a read only account. I'm only using one traffic group on that F5.
Could you please send me the results from this REST endpoint from both devices?
curl -sku admin:password https://<F5-management>/mgmt/tm/cm/failover-status
Please go ahead and clean the output from sensitive information (if any) before posting. 🙂
In case you haven't worked with the REST API you'll literally need the admin user in order to use basic auth. Otherwise you'd need to a token in order to get the info with a "normal" user. You can check out the authentication troubleshooting in this article to see how to get a token:
I get an authorization error when trying to use the token from the read-only account to query the failover status, but get no error and the expected results when using an admin account. I adjust the permissions of this account and get back.
New version out!
TLDR;
Added buttons to copy the monitor tests instead of the tooltip version
Bunch of other things has happened under the hood:
* Refactored the monitor send string functions * Added unit tests for monitor send string functions
* Added Cypress for integration testing * Refactored and added types for a bunch of functions * Transpiling via Webpack to: - Allow browser friendly module handling while unit tests are still working - Bundling all our code into one bundle instead of multiple files
The main reason for the changes above are for us to be able to protect the project against regression errors and ensure that the code quality reaching the end users is good enough.
For those making local changes, please note that the way the js-src is built has been changed. The contribution guide article has been updated accordingly.
BigIPReport was approved as an Open source project by Dockerhub but upon enrolling the account used to host the Docker images the process got stuck. My contact stopped replying and after multiple reminders and a few months I have now re-submitted the application. Until then I'll keep paying from my own pocket.
The k8s data-collector used to gather data for the report has been pulled over 3500 times so I am guessing at least a few people are using the builds.
Manual installations: https://loadbalancing.se/downloads/bigipreport-v5.6.5.zip
Kubernetes: bigipreport/data-collector-k8s:v5.6.5
I will not publish this version as stable as there's way too much code refactoring. However, if you want to help, please go ahead and download it from here:
https://loadbalancing.se/downloads/bigipreport-v5.6.6-beta.zip
There is also a (half-baked) docker-compose example of how to run the data-collector here:
https://github.com/epacke/bigipreport-docker-example
Working on a video guide on how to get started but my right arm is still not recovered from being broken so it's a bit slow. 🙂
A user reported that there are issues with the live polling function in v5.6.6. Last version it worked on seems to be v5.6.1. If you use this function I'd advise to wait with an upgrade.
Fix can be tracked here:
https://github.com/net-utilities/BigIPReport/issues/104
Found more bugs or want features? Join us on Discord:
https://discord.gg/RzmjgneW
Polling function has been fixed in v5.6.7-beta together with bug fix from Tim related to the ASM policy indexing.
You can download it here (or pull from DockerHub):
https://loadbalancing.se/downloads/bigipreport-v5.6.7-beta.zip
Found more bugs or want features? Join us on Discord:
https://discord.gg/RzmjgneW
Would appreciate some feedback. If you run version above v5.5.0 and use an F5 in front of it, could you please try this iRule? It makes the web server serve the Brotli compressed files instead of the Json files and it should accelerate the loading times quite a lot. Would be interesting to hear load times before and after:
when HTTP_REQUEST {
set has_replaced 0
if {
[HTTP::header Accept-Encoding] contains "br"
&& [HTTP::uri] ends_with ".json"
&& [HTTP::uri] ne "/json/knowndevices.json"
} {
HTTP::uri "[HTTP::uri].br"
set has_replaced 1
}
}
when HTTP_RESPONSE {
if { $has_replaced } {
HTTP::header replace "Content-Encoding" "br"
}
}
If it works well we can include it in the repository and installation instructions.
Small fix released in v5.6.8 where the public IP information was shown even when there are no NAT file configured.
Can be downloaded from here:
https://loadbalancing.se/downloads/bigipreport-v5.6.8-beta.zip
Thanks to Tim for the fix!
/Patrik
If you do not know what Brotli is you can look at it as a much more efficient way to compress css, json and javascript files. It's a bit slower to compress but on the upshot it's much smaller and using it will speed up the BigIPReport application delivery significantly. The last week we've been focusing on making it easier for people to use Brotli with their BigIPReport installations by creating server templates for different web server vendors.
For those with larger BigIPReport installations I'd really really recommend checking this out. It pretty easy and the gain is high.
Before you start though, please note that the report must run over HTTPS for Brotli to be supported!
I've now fixed the nginx configuration in the frontend containers. Pull bigipreport/frontend:v5.6.8 or latest to use the fix. Make sure to empty your cache if you use the tag :latest (not recommended to use this tag btw).
For those serving BigIPReport via an F5 you can grab an iRule which will do the necessary rewrites to use Brotli. The iRule can be found here:
https://github.com/net-utilities/BigIPReport/blob/master/other/ServeBrotliViaF5/serve-brotli.tcl
Still stuck with an old Windows installation? First I'd recommend moving to a Linux based installation instead. If this is not possible Tim has been so kind as to share his IIS web.config here:
https://github.com/net-utilities/BigIPReport/blob/master/other/iis/web.config
For those that uses Apache our superstar Tim has yet again delivered. You can find the Apache config here:
https://github.com/net-utilities/BigIPReport/blob/master/other/apache/brotli.conf
If you prefer to run your own Nginx server you can check out the file used in the frontend container:
https://github.com/net-utilities/BigIPReport-Docker/blob/master/frontend/default.conf
Using curl this is easy enough. Just run the command below and look for "content-encoding: br":
curl -I -H "Accept-Encoding: br" https://bigipreport.xip.se/json/pools.json
HTTP/2 200
server: istio-envoy
date: Wed, 11 May 2022 20:59:36 GMT
content-type: application/json
content-length: 709
last-modified: Wed, 11 May 2022 20:58:19 GMT
etag: "627c236b-2c5"
content-encoding: br
vary: Accept-Encoding
accept-ranges: bytes
x-envoy-upstream-service-time: 0
You can also double check this by opening up the developer tools of Chrome, head over to the Network tab and refresh your BigIPReport page. If Brotli is used as it should you should see "content-encoding: br" in the response headers. See the screenshot below:
If you run into trouble, please go ahead and head over to our Discord channel:
https://discord.gg/fwEaT7Rf
Have a good one!