Knowledge sharing: F5 Software Upgrade/RMA process
Here is quick summary about things should be checked before an F5 upgrade. This is the general F5 support article with clips and there is nice info for VIPRION and VCMP systems: https://support.f5.com/csp/article/K41125752 https://support.f5.com/csp/article/K84554955 https://support.f5.com/csp/article/K84205182 This a great community article 7 Steps Checklist before upgrading your F5 BIG-IP https://support.f5.com/csp/article/K11661449 https://support.f5.com/csp/article/K13081744 Extra addition to the DNS upgrade is that it is better upgrade first the LTM devices that the DNS devices monitor and after the upgrade of 1 or 2 DNS systems till the other DNS systems are also upgraded better upgrade the big3d process on the older DNS systems in the DNS sunc group: https://support.f5.com/csp/article/K15844889 https://support.f5.com/csp/article/K45907236 https://support.f5.com/csp/article/K13734 https://support.f5.com/csp/article/K13312 For BIG-IQ upgrade or for BIG-IQ to upgrade f5 devices: https://support.f5.com/csp/article/K51342220 https://techdocs.f5.com/en-us/bigiq-8-0-0/managing-big-ip-devices-from-big-iq/big-ip-software-upgrades.html For F5 devices with the F5 APM module after upgrade check if the installed F5 Edge Client software needs to be upgraded as it may not work with the new F5 APM TMOS version. https://support.f5.com/csp/article/K13757 An issue I have seen is to install the new version in a volume and transferring the configuration from the old volume to the new but without activating it and then to activate it after a week and there would an old configuration during that week many changes were done on the old volume config, so better before an upgrade so save UCS just in case from the old volume/partition: Some workarounds: https://support.f5.com/csp/article/K82463047 https://support.f5.com/csp/article/K14724 F5 RMA process general articles: F5 general articles for RMA with or withour UCS as without UCS the system and network settings may need to be configured manually and the configuration to be synchronized from the active device to the rma device. https://support.f5.com/csp/article/K12880 For F5 DNS/GTM there are special steps: https://support.f5.com/csp/article/K14083 F5 RMA of VIPRION chassis or a blade as for example when the new blade is installed but the active software version on other blades and vcmp quests is missing then the blade will get stuck in quorum for the chassis or vcmp quest as the primary blade will not be able to update it. If there is single blade in the chassis better hope that there is saved UCS expecially if there are vCMP quests as then for every vcmp quest the system and network need to be manually configured and the other config can be synchronized from the other chassis and vcmp quests that are in HA cluster. https://support.f5.com/csp/article/K14302 https://support.f5.com/csp/article/K16992 https://support.f5.com/csp/article/K23795307?utm_source=f5support&utm_medium=RSS https://support.f5.com/csp/article/K40222952 As the F5 VIPRION chassis is most complex (see K14302) if there is no saved master key as the vCMP quests use keys that are signed by the vCMP host master key and if it is lost then it is really complex, this is a nice F5 devcentral procedure how to generate your own master key that can be the same for the different F5 VIPRION Devices: https://community.f5.com/t5/technical-articles/working-with-masterkeys/ta-p/290454 When loading UCS on the RMA device that has containing encrypted passwords or passphrases, you can check(I have never used the second article but it is nice to have if issues are seen on a vCMP system when a chassis is replaced): https://support.f5.com/csp/article/K9420 Working with MasterKeys https://support.f5.com/csp/article/K13408 The new F5 Joutneys tool can be used for migrating to configuration to the new F5 VELOS and rSeries platforms and maybe in the future the F5 NEXT Operational System. https://community.f5.com/t5/technical-articles/welcome-to-the-f5-big-ip-migration-assistant-now-the-f5-journeys/ta-p/279673 https://www.youtube.com/watch?v=lLm5OkJRicw For the F5 imish/zebos routing module it is good to renember that that the config is not synchronized in a HA pair and before an RMA/upgrade to run the "write" command in the module as this is like the F5 command "save sys config" for CLI made changes as because of the reboot of the devices this changes can be lost. Before the license reactivation I suggest using the tool https://secure.f5.com/validate/validate.jsp to check that you have legitimate license and support contract.Knowledge sharing: Containers, Kubernetes, Openshift, F5 Container Connector, NGINX Ingress
For anyone interested about the free traning for "F5 Container Connector for Kubernetes" or "F5 OpenShift Container Integration" at "LearnF5". For NGINX being installed in Kubernetes there is enough info but for F5 Contaner Connector/Container Ingress Services there is not so much: https://docs.nginx.com/nginx-ingress-controller/f5-ingresslink/ https://www.nginx.com/products/nginx-ingress-controller/ https://community.f5.com/t5/technical-articles/better-together-f5-container-ingress-services-and-nginx-plus/ta-p/280471 F5 Devcentral also has youtube channel with usefull info: https://www.youtube.com/c/devcentral If you don't have good knowledge about containers and kubernetes then first check the links below. For Docker containers in youtube you will find a lot of good training for example: you need to learn Kubernetes RIGHT NOW!! - YouTube Docker Tutorial for Beginners [FULL COURSE in 3 Hours] - YouTube Docker overview | Docker Documentation The same is true for Kubernetes and they have a free test lab on their site: Learn Kubernetes Basics | Kubernetes you need to learn Docker RIGHT NOW!! // Docker Containers 101 - YouTube Red Hat has some free training and IBM provides some free labs for Containers, Kubernetes, Openshift etc.: Training and Certification (redhat.com) IBM CloudLabs: Free, Interactive Kubernetes Tutorials | IBM Red Hat OpenShift Tutorials | IBMI invite you to ping my test site for the next few days!
Maybe I'll regret this? haha Today on my Live Stream, I spun up a site - https://community.edge.buulam.net I will leave it up for a couple days and will check back to see what everyone has done to it. I'll post the results later this week! So far:
Knowledge sharing: Advanced Logging and debugging for the F5 modules
For the different F5 issues related to the different F5 modules advanced logging can be enabled. There is an F5 general article for such tasks: https://support.f5.com/csp/article/K97244114 1. F5 BIG-IP LTM For the f5 LTM advanced debug logging can be enabled or F5 iRule logging if the issue is with an irule: https://support.f5.com/csp/article/K15530 https://support.f5.com/csp/article/K5532 https://community.f5.com/t5/technical-articles/the101-logging-and-comments/ta-p/280832 2. F5 BIG-IP GTM/DNS For F5 GTM/DNS if the issue is with bad DNS response from the F5 device the DNS logging profile can be placed to log DNS requests and DNS responses from example the local Bind. If there is Wide IP that has many load balancing options then Wide IP load balancing decisions can be logged globally or better yet just for the affected Wide IP: https://support.f5.com/csp/article/K65762138 https://support.f5.com/csp/article/K25751652 https://support.f5.com/csp/article/K14615 https://techdocs.f5.com/kb/en-us/products/big-ip-dns/manuals/product/bigip-dns-implementations-13-0-0/20.html For iquery DNS communication between the F5 DNS/GTM devices in a cluster iqdump can be used: https://support.f5.com/csp/article/K13690 https://support.f5.com/csp/article/K19451442 Also there are DNS logs(big3d etc.) under global system logs for the f5 device: https://support.f5.com/csp/article/K5532 3. F5 BIG-IP AFM The AFM has a packet tracer utility that may show where is the issue with AFM rules or DDOS protection, also the AFM rules can log when they are matched and even the DDOS layer 3/4 attacks. Also the AFM IPS protocol inspection can log with action set to " Accept+Log". https://support.f5.com/csp/article/K15368 https://support.f5.com/csp/article/K03094407 https://clouddocs.f5.com/training/community/firewall/html/archive/archive1/lab2/step2.html https://support.f5.com/csp/article/K37718515 https://support.f5.com/csp/article/K51266926 https://support.f5.com/csp/article/K25265787 4. F5 BIG-IP ASM/Advanced WAF. The F5 WAF needs a security logging profile to log much of the data needed for investigation (the learning suggestions are not related to the logs and the security logging profile but to the local SQL database) but if the logs will be local better to log just illegal requests and responses. For DDOS or Bot defense the Security Logging profile under F5 Virtual server should have those options enabled. Also generate ASM reports for false postives the Security logging profiles are needed. https://support.f5.com/csp/article/K58073501 https://support.f5.com/csp/article/K37655278 https://clouddocs.f5.com/training/community/firewall/html/archive/archive1/lab3/2a-03.html https://support.f5.com/csp/article/K11412315 https://techdocs.f5.com/kb/en-us/products/big-ip_asm/manuals/product/asm-implementations-13-1-0/13.html 5. F5 BIG-IP APM For the F5 APM reports that show traffic for specfic user was processed and where the issue could be. For SSO or VDI seperate logging options need to be configured. https://support.f5.com/csp/article/K13095148 https://support.f5.com/csp/article/K13394 https://support.f5.com/csp/article/K24826763 https://techdocs.f5.com/kb/en-us/products/big-ip_apm/manuals/product/apm-secure-web-gateway-13-1-0/14.html https://support.f5.com/csp/article/K20320970 https://support.f5.com/csp/article/K44555523 https://support.f5.com/csp/article/K41437771 https://support.f5.com/csp/article/K35932460 https://support.f5.com/csp/article/K24756214 https://support.f5.com/csp/article/K44555523 6. F5 BIG-IP Analytics and BIG-IQ. The Analytics module can help discover web application issues, also BIG-IQ uses this module to provide advanced statistics for applications deployed from the BIG-IQ using AS3. https://techdocs.f5.com/kb/en-us/products/big-ip_analytics/manuals/product/analytics-implementations-12-1-0/5.html https://www.youtube.com/watch?v=0WKFKX9Ulcg&t=4s https://www.youtube.com/watch?v=8qJ4oWDPQiY https://www.youtube.com/watch?v=fscXGPUtkn0
Integration of Azure Sentinel and F5 BIG-IP using TS and AS3
This user guide is all about the configuration and deployment of Telemetry Streaming and Application Service 3 (AS3) on F5 BIG-IP to fetch logs on Azure Sentinel as its consumer. This guide is heavily based on the work performed by Greg_Coward and one can view on here. The purpose of this guide is to document a little more elaborated guide for both learning and deployment aspects and also address the possible issues that could be faced during the process of deployment. Note: More detailed steps along with configuration images can be found on : https://nishalrai.com.np/2023/06/19/integration-of-azure-sentinel-and-f5-big-ip-using-ts-and-as3/ One can leverage the usage of Azure Sentinel to collect and display the data using the Telemetry streaming extension on the F5 BIG-IP device. Azure Sentinel is able to collect the logs from the F5 BIG-IP via Telemetry Streaming regardless of its deployed location – F5 BIG-IP does not need to be on Azure to fetch those logs. A little background about the F5 BIG-IP Application Services 3 and Telemetry Streaming. BIG-IP AS3, the F5 BIG-IP Application Services 3 is an extension that uses a declarative model – JSON declaration instead of a set of imperative commands to create resources on a BIG-IP system. The system’s API endpoint – (https://<BIG-IP>/mgmt/shared/appsvcs/declare) Telemetry streaming (TS) is an iControl LX extension delivered as a TMOS-independent RPM file with the ability to declaratively aggregate, normalize and forward statistics and events from the BIG-IP to a consumer application by posting a single TS JSON declaration to TS’s declarative REST API endpoint. The Telemetry Streaming’s API endpoint – (https://<BIG-IP>>/mgmt/shared/telemetry/declare) Setup of TS and AS3 on F5 BIG-IP to integrate with Azure Sentinel The whole configuration is summarized in the following points: Verify the required modules are enabled Install the TS and AS3 extension on the F5 BIG-IP device Create the required configuration object on F5 BIG-IP Configure the Data connector of Azure with F5 BIG-IP device Verify all the required data types are available on Azure Sentinel The configuration involves both TS and AS3 extensions for different purposes – TS for establishing a connection with Azure Sentinel Data connector and AS3 for creating configuration object in the F5 BIG-IP like Virtual Server, Request Logging profile, log profile, iRule, and others. On the F5 BIG-IP device, the required modules to be enabled are ASM, AVR and iRulesLX. NOTE: The version on which the configuration is carried out is F5 BIG-IP v16.3.3 and v17.0.1 Install the TS and AS3 extension on the F5 BIG-IP device You need to download TS and AS3 extension and upload on your F5 BIG-IP device. Download link of Telemetry Streaming: https://github.com/F5Networks/f5-appsvcs-extension/releases Download link of Application Streaming 3 extension: https://github.com/F5Networks/f5-telemetry-streaming/releases To upload on F5 BIG-IP device: Go to Main Dashboard > iApps > Package Management LX Click on Import and select the file f5-appsvcs v3.45.0 and f5-telemetry v1.33.0 is being used (the latest version available). Create the required configuration object on F5 BIG-IP AS3 and TS extension is used to configure F5 BIG-IP with the necessary resources with a single JSON declaration. In this configuration, Postman is used to configure event listeners for the various deployed modules. The JSON declaration to configure to the following configuration object – Virtual Server, Pool, Node, iRule, Request Logging and Request log. { "class": "ADC", "schemaVersion": "3.45.0", "remark": "Example depicting creation of BIG-IP module log profiles", "Common": { "class": "Tenant", "Shared": { "class": "Application", "template": "shared", "telemetry_local_rule": { "remark": "Only required when TS is a local listener", "class": "iRule", "iRule": "when CLIENT_ACCEPTED {\n node 127.0.0.1 6514\n}" }, "telemetry_local": { "remark": "Only required when TS is a local listener", "class": "Service_TCP", "virtualAddresses": [ "255.255.255.254" ], "virtualPort": 6514, "iRules": [ "telemetry_local_rule" ] }, "telemetry": { "class": "Pool", "members": [{ "enable": true, "serverAddresses": [ "255.255.255.254" ], "servicePort": 6514 }], "monitors": [{ "bigip": "/Common/tcp" }] }, "telemetry_hsl": { "class": "Log_Destination", "type": "remote-high-speed-log", "protocol": "tcp", "pool": { "use": "telemetry" } }, "telemetry_formatted": { "class": "Log_Destination", "type": "splunk", "forwardTo": { "use": "telemetry_hsl" } }, "telemetry_publisher": { "class": "Log_Publisher", "destinations": [{ "use": "telemetry_formatted" }] }, "telemetry_traffic_log_profile": { "class": "Traffic_Log_Profile", "requestSettings": { "requestEnabled": true, "requestProtocol": "mds-tcp", "requestPool": { "use": "telemetry" }, "requestTemplate": "event_source=\"request_logging\",hostname=\"$BIGIP_HOSTNAME\",client_ip=\"$CLIENT_IP\",server_ip=\"$SERVER_IP\",http_method=\"$HTTP_METHOD\",http_uri=\"$HTTP_URI\",virtual_name=\"$VIRTUAL_NAME\",event_timestamp=\"$DATE_HTTP\"" }, "responseSettings": { "responseEnabled": true, "responseProtocol": "mds-tcp", "responsePool": { "use": "telemetry" }, "responseTemplate": "event_source=\"response_logging\",hostname=\"$BIGIP_HOSTNAME\",client_ip=\"$CLIENT_IP\",server_ip=\"$SERVER_IP\",http_method=\"$HTTP_METHOD\",http_uri=\"$HTTP_URI\",virtual_name=\"$VIRTUAL_NAME\",event_timestamp=\"$DATE_HTTP\",http_statcode=\"$HTTP_STATCODE\",http_status=\"$HTTP_STATUS\",response_ms=\"$RESPONSE_MSECS\"" } }, "telemetry_asm_security_log_profile": { "class": "Security_Log_Profile", "application": { "localStorage": false, "remoteStorage": "splunk", "servers": [{ "address": "255.255.255.254", "port": "6514" }], "storageFilter": { "requestType": "all" } } } } } } Tips to mitigate configuration issues Use the visual studio code and add JSON formatter extension to format the JSON code and avoid any indentation error on the code. On the JSON declaration, be careful with the schemaVersion, the version should match with the install The F5 Application Streaming v3 extension, in my case it’s 3.45.0 Launch the postman, enter the API endpoint: https://<BIG-IP>/mgmt/shared/appsvcs/declare Output of the successful deployment: Verify whether the object has been created on F5 BIG-IP Browse to the F5 BIG-IP dashboard and verify whether all the required objects has been created or not. Once all the object has been created, you need to execute the following command on the F5 BIG-IP CLI. This seems to be a bug on the TS listener with the F5 BIG-IP device. The issue was caused by a new db key which by default prohibits loopback addresses in irules. If you have configured a local listener, with an irule such as “when CLIENT_ACCEPTED {\n node 127.0.0.1 6514\n}” Then you need to run the following tmsh command. tmsh modify sys db tmm.tcl.rule.node.allow_loopback_addresses value true For more info: https://github.com/F5Networks/f5-telemetry-streaming/issues/238 Configure the Data connector of Azure Sentinel with F5 BIG-IP device Once all the above configuration has been completed, it’s time to integrate F5 BIG-IP device with Azure Sentinel. Telemetry Streaming extension will be used to establish the connection between the F5 BIG-IP device and data connector of Azure sentinel. The JSON declaration used to establish the connection between the Azure Sentinel – Data Connector and F5 BIG-IP device. { "class": "Telemetry", "controls": { "class": "Controls", "logLevel": "info", "debug": true }, "My_System": { "class": "Telemetry_System", "trace": "/var/tmp/telemetry_trace.log", "systemPoller": { "interval": 60 } }, "My_Listener": { "class": "Telemetry_Listener", "port": 6514 }, "My_Consumer": { "class": "Telemetry_Consumer", "type": "Azure_Log_Analytics", "workspaceId": "<workspace-id>", "passphrase": { "cipherText": "<cipher-text>" }, "useManagedIdentity": false, "region": "<region>" } } You can find the required credentials of the Azure Sentinel on the workspace of the F5 BIG-IP connector page. Once you’ve got all the required credentials then you can carry out the configuration. I will be using Postman to declare the configuration in JSON format on system’s endpoint: https://<BIG-IP>>/mgmt/shared/telemetry/declare then you will get something like this as an output on the successful deployment: Verify all the required data types are available on Azure Sentinel After all the configuration has been completed, you need to login into the Azure Portal. Browse to the Microsoft Sentinel then select the workspace. Search for F5 BIG-IP and open the connector page then you can see the data type available. On the Workspace of the Azure Sentinel, you can browse to the Workbook – F5 BIG-IP ASM, where all the collected logs of ASM (only Application Security logs) are visualized. This is the visualization of the ASM logs on the Azure Sentinel.
RabbitMQ monitor.
We are trying to configure RabbitMQ health monitors to determine which node is primary and send all traffic to it, and failover if the primary goes down. Configuration is 2 nodes in two different Azure regions (total 4) with web services and RabbitMW running on all 4. Any thoughts? Frank
Load Balancing between same application deployed on Virtual clusters and Openshift containers
Hi, What will be a recommended way to load balance between existing GTM setup of Two Data Centers for both Virtual machines and Openshift Containers? My setup is the following: 1: Virtual Machine deployment: With two Data Centers DC1 and DC2 deploying same 80 rest services on separate ports, we have a GTM that distributes to two LTM (LTM1, LTM2), One in DC1 and another in DC2. The two LTMs are configured inside one pool on the GTM. The LTMs use Snat. The LTMs have a pool members pointing to the service endpoints per port. This setup is giving our clients access to all our services found on different ports and virtual machines and executing using the round robin Method. FQDN/WideIP is a.b.c.com DC1.b.c.com dc2.b.c.com 2: Openshift containers: Containers has same services but with a different GTM and LTM setup. FQDN/WideIP is ocp.b.c.com DC1.ocp.b.c.com Dc2.ocp.b.c.com I am wondering what is the recommended way to load balance between the two existing GTMs setup so that the client can reach either the Virtualized or the containerized service endpoints in a round robin fashion. Is it even recommended to load Balance even between to existing GTM configurations?
SAML SESSION VARIABLE AND ATTRIBUTES
HI, I am currently setup on my APM to use SAML single sign on with Azure as my IDP and F5 APM as my SP. I want to assign resources to authenticated users based on their groups in azure. How do i represent this in the Advanced Resource Assign expression in the Visual Policy Editor? Please this is quite urgent.Modify Azure Active Directory application credentials after ARM template deployment
Hello to you all, I deployed an F5 BIG-IP VE Active/Standby Cluster from Github repository: https://github.com/F5Networks/f5-azure-arm-templates/tree/v5.4.0.0/supported/failover/same-net/via-api/n-nic/existing-stack I have trouble with my HA configuration and I think it come from the Azure Active Directory application (that does not call the Azure API properly to reassign all VS IP from old to new Active node). Please find a comment bellow from Peter Silva on DevCentral: "The next 3 fields (Tenant ID, Client ID, Service Principal Secret) have to do with security. Rather than using your own credentials to modify resources in Azure, you can create an Active Directory application and assign permissions to it." I was wondering if there is a way to change the required "Tenant ID, Client ID, Service Principal Secret" fields after ARM template deployment. From WebUI management or either TMOS shell instance. I found nothing about it online. Thank you in advance for your help, Jordan
