BIG-IP Report

[Patrik_Jonsson]

Problem this snippet solves:

Overview

This is a script which will generate a report of the BIG-IP LTM configuration on all your load balancers making it easy to find information and get a comprehensive overview of virtual servers and pools connected to them.

This information is used to relay information to NOC and developers to give them insight in where things are located and to be able to plan patching and deploys. I also use it myself as a quick way get information or gather data used as a foundation for RFC's, ie get a list of all external virtual servers without compression profiles.

The script has been running on 13 pairs of load balancers, indexing over 1200 virtual servers for several years now and the report is widely used across the company and by many companies and governments across the world.

It's easy to setup and use and only requires auditor (read-only) permissions on your devices.

Demo/Preview

Interactive demo

http://loadbalancing.se/bigipreportdemo/

Screen shots

The main report:

The device overview:

Certificate details:

How to use this snippet:

Installation instructions

BigipReport REST

This is the only branch we're updating since middle of 2020 and it supports 12.x and upwards (maybe even 11.6).

• Downloads (two latest versions):
  https://loadbalancing.se/downloads/bigipreport-v5.7.0.zip
  https://loadbalancing.se/downloads/bigipreport-v5.6.5.zip
• Documentation, installation instructions and troubleshooting: https://loadbalancing.se/bigipreport-rest/

  Docker support

  https://loadbalancing.se/2021/01/05/running-bigipreport-on-docker/

  Kubernetes support

  https://loadbalancing.se/2021/04/16/bigipreport-on-kubernetes/

  BIG-IP Report (Legacy)

  Older version of the report that only runs on Windows and is depending on a Powershell plugin originally written by Joe Pruitt (F5)

  BIG-IP Report (only download this if you have v10 devices):

  https://loadbalancing.se/downloads/bigipreport-5.4.0-beta.zip

  iControl Snapin

  https://loadbalancing.se/downloads/f5-icontrol.zip

  Documentation and Installation Instructions

  https://loadbalancing.se/bigip-report/

  Upgrade instructions

  Protect the report using APM and active directory

  Written by DevCentral member Shann_P:

  https://loadbalancing.se/2018/04/08/protecting-bigip-report-behind-an-apm-by-shannon-poole/

  Got issues/problems/feedback?

  Still have issues? Drop a comment below. We usually reply quite fast. Any bugs found, issues detected or ideas contributed makes the report better for everyone, so it's always appreciated.

  ---

  Join us on Discord: https://discord.gg/7JJvPMYahA

  Code :

  BigIP Report

  Tested this on version:

  12, 13, 14, 15, 16

[LiefZimmerman]

Due to a platform corruption during the 2019 migration I have worked with   team to move his original legacy codeshare to this new record (same URL).

The legacy codeshare is temporarily available at https://devcentral.f5.com/s/articles/bigip-report-old

The negative repercussions of this change are:

• the nearly 1000 historical comments appear that they must will have to remain on the legacy record.
• the numerous *likes* stayed with the legacy record (smash that like at the top right of this page!! if you like it)
• anyone bookmarking that legacy record will not be notified of changes made here (click that bookmark icon)

The positive repercussions of this are:

• Patrik can now make edits to the entry - making updates more likely and timely. (a deep in the weeds corruption I couldn't ferret out)
• You all only have to deal with one set of comments here - making conversation easier. (another part of the corruption)
• And, looking on the bright side of a negative, I don't have to find out (yet) what happens when the comment counter rolls over to 1000.😩

Thank you for your patience and persistence with Patrik's awesome contribution and thank you for your dedication to our community.

[Patrik_Jonsson]

New release v5.5.9. Well done Tim!

I've updated the Kubernetes containers with the new code and also triggered the :latest tag for you Cowboys who likes to use that. 🙂

Got questions/feedback/an insatiable lust for nerd talk? Join us on Discord:

https://discord.gg/W2y2cFX7

Kind regards,

Patrik

[BenT]

Modules directory is missing from the zip file.

Line |

2255 | . .\modules\Get-ExpiredCertificates.ps1

   |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   | The term '.\modules\Get-ExpiredCertificates.ps1' is not recognized as a name of a cmdlet, function, script

   | file, or executable program. Check the spelling of the name, or if a path was included, verify that the path is

   | correct and try again.

[Patrik_Jonsson]

Hi Ben

Apologies, been out enjoying the sun in the afternoon.

Thanks for reporting, will fix the script that creates deploy and get back to you.

/Patrik

[Patrik_Jonsson]

Please try to download it again?

/Patrik

[BenT]

I've tried a few times over the last few hours, the zip sizes are identical and the modules folder isn't in the zip.

[Patrik_Jonsson]

Strange. I downloaded the file after updating the build job and I can see the modules just fine:

Please confirm that this is the link you're using?

https://loadbalancing.se/downloads/bigipreport-v5.5.9.zip

/Patrik

[Patrik_Jonsson]

Maybe I should have been more clear that I updated something... 😉

/Patrik

[BenT]

That got it. Thanks!

Have you looked at port-lists and/or policies? For port-lists, the VIPs aren't contained in the virtual, but are in a traffic-matching-criteria object instead. I don't know how difficult it would be to add the logic to show the VIPs for those in bigipreport or any associated policies.

Here is an example config for the port-lists.

net port-list /Common/web_443-8443-8080_ports {
    description web_443-8443-8080_ports
    ports {
        443 { }
        8080 { }
        8443 { }
    }
}ltm pool /Common/pool_vip_portlist_example {
    load-balancing-mode least-connections-node
    members {
        /Common/172.1.2.5:0 {
            address 172.1.2.5
        }
        /Common/172.1.2.6:0 {
            address 172.1.2.6
        }
    }
    monitor /Common/https_basic_443
}
ltm traffic-matching-criteria /Common/vip_portlist_example_VS_TMC_OBJ {
    destination-address-inline 1.2.3.4
    destination-port-list /Common/web_443-8443-8080_ports
    protocol tcp
    source-address-inline 0.0.0.0
}
ltm virtual /Common/vip_portlist_example {
    ip-protocol tcp
    pool /Common/pool_vip_portlist_example
    profiles {
        /Common/fastL4 { }
    }
    serverssl-use-sni disabled
    source-address-translation {
        type automap
    }
    traffic-matching-criteria /Common/vip_portlist_example_VS_TMC_OBJ
    translate-address enabled
    translate-port enabled
    vlans {
        /Common/proxy-vlan
    }
    vlans-enabled
}

[Patrik_Jonsson]

Hi Ben

Glad it helped. I added a feature request for you on Github. You can go here:

https://github.com/net-utilities/BigIPReport/issues/69

and then watch it for updates.

I'm afraid I can't give any timelines since both me and Tim does this on our spare time. I normally try to squash bugs pretty fast but features happens when time permits. 🙂

/Patrik

[Patrik_Jonsson]

Dear all

I got an email from Docker-hub today that they will discontinue the automatic builds due to too many people abusing the service.

I'm not yet sure how this will affect the service. Since we don't build the application that often it might work if the manual build is still free.

Either way, I have applied for the bigipreport project to be approved as an Open Source account. If manual builds won't work and the application is denied we'll have to take it from there.

This is only relevant for those running the report using Docker and I will update here once I know more.

Kind regards,

Patrik

[richi3161]

Hello,

I have tried today the Version 5.5.9, but I am not able to run script - there is an issue with device certifcate check:

PS C:\bigipreport-v5.5.9> .\bigipreport-v5.5.9.ps1

2021-07-22 11:34:47 Starting: PSCommandPath=C:\bigipreport-v5.5.9\bigipreport-v5.5.9.ps1 ConfigurationFile=C:\bigipreport-v5.5.9/bigipreportconfig.xml CurrentJob= Location= PSScriptRoot=C:\bigipreport-v5.5.9

2021-07-22 11:34:47 Successfully loaded config file: C:\bigipreport-v5.5.9/bigipreportconfig.xml

2021-07-22 11:34:47 Insecure SkipCertificateCheck enabled, consider using valid certificates and DNS names

ParentContainsErrorRecordException: C:\bigipreport-v5.5.9\bigipreport-v5.5.9.ps1:654

Line |

 654 |     $PSDefaultParameterValues.Add("Invoke-RestMethod:SkipCertific …

   |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   | Exception calling "Add" with "2" argument(s): "The key 'Invoke-RestMethod:SkipCertificateCheck' has

   | already been added to the dictionary."

PS C:\bigipreport-v5.5.9>

[Patrik_Jonsson]

I believe the actual reason behind the error was that the script was executed twice in the same PSSession.

Kind regards,

Patrik

[richi3161]

That´s correct. When I try to run the script twice again, I get exactly this error.

[TimRiker]

 I patched this in 5.6.1 Can you test once Patrik makes a release? or test from git.

[Patrik_Jonsson]

Here's a pre-release:

https://loadbalancing.se/downloads/bigipreport-v5.6.1.zip

If it works fine I'll update the version above.

[Patrik_Jonsson]

Heads up to those running the report using Kubernetes.

My Dockerhub account was disabled and my application for the open source version has not been processed yet.

This means that you need to build and store the images yourself.

You can find the manifests here:

https://github.com/net-utilities/BigIPReport-Docker

Will also update the documentation to cover the pagination concept Tim added yesterday.

[Patrik_Jonsson]

Good news! Docker has approved my request to include BigIPreport in their open source program.

This means that the docker/k8s containers will be available as soon as it's activated (1-2 weeks).

Kind regards,

Patrik

[JRahm]

Nice work , way to go!

[MBauer]

Dear Patrick,

I used version 5.6.2 and added my code to crawl the policy data as well.

I will try to share the code via github with you.

The F5 statement regarding the translation of the policy logic to readable code was that they cannot share this with me.

(Service Request #C3491339)

I started to create the logic for our most recent used policie configurations already.

Best regards,

Marius

[TimRiker]

 I sent back some reviews to your code on github. I've not yet tried it out locally.

[Patrik_Jonsson]

 Well done! Just some small things to remark on.

Also good input Tim! I added some comments too.

We're between houses so my lab will be packed into boxes until December so I can't test it myself.

Kind regards,

Patrik

[Patrik_Jonsson]

I got an email regarding the kubernetes and docker builds and figured I should update here too. Dockerhub approved the open source project request as stated earlier but then it got stuck because of some internal Dockerhub stuff. Still working on solving this.

Meanwhile, the manifests are available on GitHub. If you are looking for them you probably know how it works but if this is not the case you’re more than welcome to join our discord channel if you have any support questions.

https://discord.gg/6hVHur2H

[sandip_kakade]

Hi hi Hi Jason

Can you please give me web server config which used in official web site training . I am learning LTM now

[Patrik_Jonsson]

No update from Dockerhub regarding the open source project.

First they approved it but then they said there's a problem with my account. After that I have poked them multiple times but nothing happens. I've grown tired and now I pay for it myself from my own pocket.

Kubernetes containers are now available for the current version again.

Kind regards,

Patrik

[Ali_Hyder]

Thanks Patrik. Your script works very well and helps me to view my LTM Data. I am looking for same kind of script with BIG IP DNS to map the data of Wide IP and its backend pool and servers.

[Patrik_Jonsson]

Hi Ali

Glad you like it!

I'm afraid I don't use BIG IP DNS at work at the moment so I have no opportunity to develop this functionality.

Maybe in the future...

Kind regards,

Patrik

[Unal_Sahin]

Hello Patrik.

I installed BIGIP Report 5.6.2. But I dont know how to use Get-ExpiredCertificates.ps1 with parameters. Can you explain to me with an example?

Kind Regards

Unal

[TimRiker]

  The modules/* scripts are called as part of the normal run. You should configure the xml file to have your F5s listed, and a user/password credential to log into them. Then run the parent script. It will call each of the modules and output the report.

[BenT]

I just noticed 5.6.2 does not work well with port groups. It stops caching virtual servers as soon as it processes the first one using port groups. I have 1163 virtuals, it only processes the first 136.

2021-11-30 08:02:08 VERBOSE x.x.x.x:Caching Virtual servers

2021-11-30 08:02:10 VERBOSE x.x.x.x:Caching Virtual servers

2021-11-30 08:02:55 ERROR x.x.x.x:Unable to cache virtual servers: Cannot convert value "any6" to type "System.Net.IPAddres

s". Error: "An invalid IP address was specified." (line 1038)

2021-11-30 08:02:56 VERBOSE x.x.x.x:Detecting orphaned pools

2021-11-30 08:02:56 SUCCESS x.x.x.xStats: VS:136 P:937 R:113 DG:7 C:187 M:171 ASM:0 T:151.5807781

[BenT]

Ignore that. For some reason a /0 was added to the destination IPs for the port group VIPs after I reloaded the config from CLI a few days ago. After removing those, the script is completing again.

[Patrik_Jonsson]

Hi there!
There's a pull request from Marius Bauer which includes support for policies. Both Tim and I are old school iRulers (yeah yeah, I know that policies are faster) so we would like to know if there's any users out there willing to test Marius branch?

If there is I'll release a beta package for testing.

Kind regards,
Patrik

[TheGrave]

Go ahead, we have a pre-prod server running LB-monitor as well. Marius is an ex-colleague of ours 🙂

We are having issues with the script though:

bigip-ext-abc-1.domain.com:Failed to get auth token
bigip-ext-bde-5.domain.com:Failed to get auth token
bigip-cloud-01.domain.com:Failed to get auth token
bigip-ext-bde-1.domain.com does not seem to have been indexed
bigip-ext-abv-5.domain.com does not seem to have been indexed
bigip-ext-abv.domain.com does not seem to have been indexed

These errors are raised is on a daily basis. Probably due to a temporary connection glitch/slow response. Any timeouts we can adjust to alleviate the problem?

[Patrik_Jonsson]

Hi there!

• Does it happen consistently?
• Have you read this part of the FAQ?
  https://loadbalancing.se/bigipreport-rest/#Unable_to_get_a_token_from_the_device
• I can also recommend this one (authentication part):
  https://loadbalancing.se/2021/03/28/installing-troubleshooting-and-running-bigip-ingress-controller/...

If it's inconsistent it could also be poor connection/management provisioning being too small or the config too large. We're on discord if you want to discuss it/get help:
https://discord.gg/RzmjgneW

Kind regards,
Patrik

[TheGrave]

Well, we see error e-mails couple of times a day.

One of my colleagues is aware of the auth issue, we did some upgrades recently so this seems to be the root cause.

But indexing errors we see pretty much every day at least once, normally the same devices. Config is not big but connection might be interrupted cause they are sitting behind the Great Chinese Firewall and we get all sorts of crap from it 🙂 Is there any way to influence timeouts and retry-periods, eventually on a per-host basis?

[Patrik_Jonsson]

Oh darn. I recognize the challenges with the Chinese firewall.

Lived in Wuhan for a year (before Covid) and there were constant challenges with getting outside the famous wall. Afraid that if it is the national firewall that gives you trouble there's not that much to do at the moment except for running a separate BigIPReport within the borders of China.

Since all objects are arrays you could even run a simple script to concatenate the data from the Chinese BigIPReport to the main report. I think this would be a bit more robust that relying on a few hundred API calls through the firewall and frankly very easy to do.

[TheGrave]

These devices are about to be decomissioned in the upcoming months so no way we can put money and effort into installing new servers there. I was thinking about some simple mitigation instead.

[Delbrugge]

Hi Patrik, I'm using the Kubernetes version 5.6.4 (also tried 5.6.2) and running into a couple of issues:

• The certificates pane is empty - this error is in the log - Error loading certificates. {"code":400,"message":"\"recursive\" unexpected argument","errorStack":[],"apiError":26214401} (line 1126)
• A certain pair of F5s do not display any data in the webpage. There isn't any error in the data collector log. It oddly will not display their hostname and also says both of them are not active.

I'm getting a invalid invite code from your Discord link.

[Patrik_Jonsson]

Howdy!

Certificate information missing

Is the user that BigIP report is using auditor or above? Else it won't be able to read the certificate information. 

Pair missing data

Sounds like the collector believes both devices are passive in which case it won't collect any data by default. Are you using multiple traffic groups on these devices by any chance?

[Delbrugge]

It's a read only account. I'm only using one traffic group on that F5. I should mentioned that I'm using an older version, on 5.3.1 on a Windows Server and do not have these two issues there and I use the same account. Perhaps it's a SOAP/REST difference?

[Patrik_Jonsson]

Perhaps it's a SOAP/REST difference?

Good guess! It is. SOAP got more permissions with a read-only role than REST does.

It's a read only account. I'm only using one traffic group on that F5.

Could you please send me the results from this REST endpoint from both devices?

curl -sku admin:password https://<F5-management>/mgmt/tm/cm/failover-status

Please go ahead and clean the output from sensitive information (if any) before posting. 🙂

[Patrik_Jonsson]

In case you haven't worked with the REST API you'll literally need the admin user in order to use basic auth. Otherwise you'd need to a token in order to get the info with a "normal" user. You can check out the authentication troubleshooting in this article to see how to get a token:

https://loadbalancing.se/2021/03/28/installing-troubleshooting-and-running-bigip-ingress-controller/...

[Delbrugge]

I get an authorization error when trying to use the token from the read-only account to query the failover status, but get no error and the expected results when using an admin account. I adjust the permissions of this account and get back.

[Patrik_Jonsson]

New version out!

Changes

TLDR;
Added buttons to copy the monitor tests instead of the tooltip version

Bunch of other things has happened under the hood:

* Refactored the monitor send string functions
* Added unit tests for monitor send string functions
* Added Cypress for integration testing
* Refactored and added types for a bunch of functions
* Transpiling via Webpack to:
  - Allow browser friendly module handling while unit tests are still working
  - Bundling all our code into one bundle instead of multiple files

The main reason for the changes above are for us to be able to protect the project against regression errors and ensure that the code quality reaching the end users is good enough.

For those making local changes, please note that the way the js-src is built has been changed. The contribution guide article has been updated accordingly.

DockerHub update

BigIPReport was approved as an Open source project by Dockerhub but upon enrolling the account used to host the Docker images the process got stuck. My contact stopped replying and after multiple reminders and a few months I have now re-submitted the application. Until then I'll keep paying from my own pocket.

The k8s data-collector used to gather data for the report has been pulled over 3500 times so I am guessing at least a few people are using the builds.

Manual installations: https://loadbalancing.se/downloads/bigipreport-v5.6.5.zip 
Kubernetes: bigipreport/data-collector-k8s:v5.6.5 

[Patrik_Jonsson]

v5.6.6-beta has been released

• Added much stricter linting rules
• Added parameter typing to almost all functions
• Added prettier for a more consistent code style
• Handling Virtual server masks with value any6
• Graceful handling of policy processing for optional attributes

I will not publish this version as stable as there's way too much code refactoring. However, if you want to help, please go ahead and download it from here:

https://loadbalancing.se/downloads/bigipreport-v5.6.6-beta.zip

There is also a (half-baked) docker-compose example of how to run the data-collector here:
https://github.com/epacke/bigipreport-docker-example

Working on a video guide on how to get started but my right arm is still not recovered from being broken so it's a bit slow. 🙂

[Patrik_Jonsson]

A user reported that there are issues with the live polling function in v5.6.6. Last version it worked on seems to be v5.6.1. If you use this function I'd advise to wait with an upgrade.

Fix can be tracked here:
https://github.com/net-utilities/BigIPReport/issues/104

Found more bugs or want features? Join us on Discord:
https://discord.gg/RzmjgneW

[Patrik_Jonsson]

Polling function has been fixed in v5.6.7-beta together with bug fix from Tim related to the ASM policy indexing.

You can download it here (or pull from DockerHub):
https://loadbalancing.se/downloads/bigipreport-v5.6.7-beta.zip

Found more bugs or want features? Join us on Discord:
https://discord.gg/RzmjgneW

[Patrik_Jonsson]

Would appreciate some feedback. If you run version above v5.5.0 and use an F5 in front of it, could you please try this iRule? It makes the web server serve the Brotli compressed files instead of the Json files and it should accelerate the loading times quite a lot. Would be interesting to hear load times before and after:

when HTTP_REQUEST {
  set has_replaced 0
  if { 
    [HTTP::header Accept-Encoding] contains "br"
    && [HTTP::uri] ends_with ".json"
    && [HTTP::uri] ne "/json/knowndevices.json"
   } {
    HTTP::uri "[HTTP::uri].br"
    set has_replaced 1
  }
}

when HTTP_RESPONSE {
  if { $has_replaced } {
    HTTP::header replace "Content-Encoding" "br"
  }
}

 If it works well we can include it in the repository and installation instructions. 

[Patrik_Jonsson]

Small fix released in v5.6.8 where the public IP information was shown even when there are no NAT file configured.

Can be downloaded from here:
https://loadbalancing.se/downloads/bigipreport-v5.6.8-beta.zip

Thanks to Tim for the fix!

/Patrik

[Patrik_Jonsson]

Update regarding Brotli support

If you do not know what Brotli is you can look at it as a much more efficient way to compress css, json and javascript files. It's a bit slower to compress but on the upshot it's much smaller and using it will speed up the BigIPReport application delivery significantly. The last week we've been focusing on making it easier for people to use Brotli with their BigIPReport installations by creating server templates for different web server vendors.

For those with larger BigIPReport installations I'd really really recommend checking this out. It pretty easy and the gain is high.

Before you start though, please note that the report must run over HTTPS for Brotli to be supported!

Docker/Kubernetes

I've now fixed the nginx configuration in the frontend containers. Pull bigipreport/frontend:v5.6.8 or latest to use the fix. Make sure to empty your cache if you use the tag :latest (not recommended to use this tag btw).

F5 iRules

For those serving BigIPReport via an F5 you can grab an iRule which will do the necessary rewrites to use Brotli. The iRule can be found here:
https://github.com/net-utilities/BigIPReport/blob/master/other/ServeBrotliViaF5/serve-brotli.tcl

IIS

Still stuck with an old Windows installation? First I'd recommend moving to a Linux based installation instead. If this is not possible Tim has been so kind as to share his IIS web.config here:
https://github.com/net-utilities/BigIPReport/blob/master/other/iis/web.config

Apache

For those that uses Apache our superstar Tim has yet again delivered. You can find the Apache config here:
https://github.com/net-utilities/BigIPReport/blob/master/other/apache/brotli.conf

Nginx

If you prefer to run your own Nginx server you can check out the file used in the frontend container:
https://github.com/net-utilities/BigIPReport-Docker/blob/master/frontend/default.conf

Double checking that it works

Using curl this is easy enough. Just run the command below and look for "content-encoding: br":

 curl -I -H "Accept-Encoding: br" https://bigipreport.xip.se/json/pools.json
HTTP/2 200 
server: istio-envoy
date: Wed, 11 May 2022 20:59:36 GMT
content-type: application/json
content-length: 709
last-modified: Wed, 11 May 2022 20:58:19 GMT
etag: "627c236b-2c5"
content-encoding: br
vary: Accept-Encoding
accept-ranges: bytes
x-envoy-upstream-service-time: 0

  You can also double check this by opening up the developer tools of Chrome, head over to the Network tab and refresh your BigIPReport page. If Brotli is used as it should you should see "content-encoding: br" in the response headers. See the screenshot below:

If you run into trouble, please go ahead and head over to our Discord channel:
https://discord.gg/fwEaT7Rf

Have a good one!