F5 IP geolocation database updating from "Edge" to "Pulse"
Hey everyone, not sure if folks have caught this but there's important information regarding the geo IP database found on BIG-IP. Coles notes is that a new (and better) type of database will be used. Please see this article for detailed and up to date info. F5 IP geolocation database migrated from Edge to the Pulse databaseKnowledge sharing: F5 Software Upgrade/RMA process
Here is quick summary about things should be checked before an F5 upgrade. This is the general F5 support article with clips and there is nice info for VIPRION and VCMP systems: https://support.f5.com/csp/article/K41125752 https://support.f5.com/csp/article/K84554955 https://support.f5.com/csp/article/K84205182 This a great community article 7 Steps Checklist before upgrading your F5 BIG-IP https://support.f5.com/csp/article/K11661449 https://support.f5.com/csp/article/K13081744 Extra addition to the DNS upgrade is that it is better upgrade first the LTM devices that the DNS devices monitor and after the upgrade of 1 or 2 DNS systems till the other DNS systems are also upgraded better upgrade the big3d process on the older DNS systems in the DNS sunc group: https://support.f5.com/csp/article/K15844889 https://support.f5.com/csp/article/K45907236 https://support.f5.com/csp/article/K13734 https://support.f5.com/csp/article/K13312 For BIG-IQ upgrade or for BIG-IQ to upgrade f5 devices: https://support.f5.com/csp/article/K51342220 https://techdocs.f5.com/en-us/bigiq-8-0-0/managing-big-ip-devices-from-big-iq/big-ip-software-upgrades.html For F5 devices with the F5 APM module after upgrade check if the installed F5 Edge Client software needs to be upgraded as it may not work with the new F5 APM TMOS version. https://support.f5.com/csp/article/K13757 An issue I have seen is to install the new version in a volume and transferring the configuration from the old volume to the new but without activating it and then to activate it after a week and there would an old configuration during that week many changes were done on the old volume config, so better before an upgrade so save UCS just in case from the old volume/partition: Some workarounds: https://support.f5.com/csp/article/K82463047 https://support.f5.com/csp/article/K14724 F5 RMA process general articles: F5 general articles for RMA with or withour UCS as without UCS the system and network settings may need to be configured manually and the configuration to be synchronized from the active device to the rma device. https://support.f5.com/csp/article/K12880 For F5 DNS/GTM there are special steps: https://support.f5.com/csp/article/K14083 F5 RMA of VIPRION chassis or a blade as for example when the new blade is installed but the active software version on other blades and vcmp quests is missing then the blade will get stuck in quorum for the chassis or vcmp quest as the primary blade will not be able to update it. If there is single blade in the chassis better hope that there is saved UCS expecially if there are vCMP quests as then for every vcmp quest the system and network need to be manually configured and the other config can be synchronized from the other chassis and vcmp quests that are in HA cluster. https://support.f5.com/csp/article/K14302 https://support.f5.com/csp/article/K16992 https://support.f5.com/csp/article/K23795307?utm_source=f5support&utm_medium=RSS https://support.f5.com/csp/article/K40222952 As the F5 VIPRION chassis is most complex (see K14302) if there is no saved master key as the vCMP quests use keys that are signed by the vCMP host master key and if it is lost then it is really complex, this is a nice F5 devcentral procedure how to generate your own master key that can be the same for the different F5 VIPRION Devices: https://community.f5.com/t5/technical-articles/working-with-masterkeys/ta-p/290454 When loading UCS on the RMA device that has containing encrypted passwords or passphrases, you can check(I have never used the second article but it is nice to have if issues are seen on a vCMP system when a chassis is replaced): https://support.f5.com/csp/article/K9420 Working with MasterKeys https://support.f5.com/csp/article/K13408 The new F5 Joutneys tool can be used for migrating to configuration to the new F5 VELOS and rSeries platforms and maybe in the future the F5 NEXT Operational System. https://community.f5.com/t5/technical-articles/welcome-to-the-f5-big-ip-migration-assistant-now-the-f5-journeys/ta-p/279673 https://www.youtube.com/watch?v=lLm5OkJRicw For the F5 imish/zebos routing module it is good to renember that that the config is not synchronized in a HA pair and before an RMA/upgrade to run the "write" command in the module as this is like the F5 command "save sys config" for CLI made changes as because of the reboot of the devices this changes can be lost. Before the license reactivation I suggest using the tool https://secure.f5.com/validate/validate.jsp to check that you have legitimate license and support contract.
Why you have broken DevCentral?
Why oh why have you broken DevCentral completely? None of the google hits to devcentral don't work anymore. None of the internal links in devcentral to other articles don't work. Fonts and the UI is horrible to read. Articles can't be opened to new tabs or windows, because links are not links but some javascript shit. If I still manage to find some article, like this https://devcentral.f5.com/s/articles/irules-101-01-introduction-to-irules, I would like to be able to freaking READ THE CONTENT! Not some 404 page not found pages. Those I have read already too many times. Please, bring the old working site back.
Higher CPU usage due to unused FQDN nodes
Not a question, but some information that might help others. Observed behaviour was higher CPU usage, unknown when it started, beyond the 30 days of on box statistics. Investigation with top (and 1) showed it was the mcpd process jumping around the different management CPUs and raising usage to 100%. This occurred on both the active and standby unit, pretty much ruling out it being due to traffic. On advice of our F5 partner (thanks) we put the logging level for MCP to debug for a short time. This resulted in many of the following log lines: Dec 1 15:00:00 hostname err mcpd[7623]: 01070726:3: Node address (::) in partition (Common) may not reference monitor (/Common/hm_name) It turned out this health monitor (hm_name) was active on a FQDN node which wasn’t used in any pool. Resulting in not looking up an IP for it it seems. Once these unused FQDN nodes were deleted, the CPU usage returned to normal.
Deleting an AS3 Tenant
Wanted to share the below method for deleting AS3 tenant's as it wasn'tdocumented . You can use the HTTP delete method; but if an admin misses the tenant name after /declare/ it would wipe out all tenants! If you POST the below body to the 'https://{{bigip_mgmt}}/mgmt/shared/appsvcs/declare'; as its a blank declaration; AS3 will remove yourpartition / tenant. . { "class": "AS3", "action": "deploy", "declaration": { "class": "ADC", "schemaVersion": "3.1.0", "id": "tenant_name", "label": "tenant_name_via_AS3", "remark": "tenant_name_via_AS3", "CHANGE-ME-TO-TENANT-NAME": { "class": "Tenant" } } }
Devcentral relaunch || Really very confusing || Errors and Moved Stuff ..?!! || Complain
This is very sad that the community which is a very elegant healthy place that differ F5 from anyone else is not available when I needed that :( I know that change is the most truth and change is for all good , but I have complains . Dears , please make an action regarding moved or retired stuff messages as it's very annoying and a lot of help just evaporates ?! I hope you guys understand our concerns and have a plan. Thanks
Mattermost, F5 LTM, and Websockets
I recently worked with a team that wanted to use the F5 Local Traffic Manager (LTM) feature to load balance connections to their new deployment of the Mattermost open source messaging platform within their on-premises datacenter. This application uses both HTTPS and Websockets connections for real-time chat. We ran into a few configuration issues but eventually found the right combination of “nerd knobs” to allow successful ingress traffic. This post is to consolidate these details and hopefully save time to other F5 engineers attempting to do the same. Business Requirements Ingress (client-side) connections TLS 1.1 or higher Support Websockets Only allow customized Mattermost mobile (iOS and Android) applications, provisioned from within the organization and using a custom header, to connect from the public Internet. The same VIP for the mobile traffic should be used by internal desktop or web browsers, which will not include this custom header Virtual Server Configuration The virtual server configuration was fairly straight forward: Protocol Profile (Client) = a mobile optimized TCP profile Protocol Profile (Server) = a LAN optimized TCP profile SSL Profile (Client) = profile with the Option No TLSv1 enabled SSL Profile (Server) = standard serverssl profile or custom one WebSocket Profile = WebSocket (or a custom one with this as the parent) SNAT = custom pool, but AutoMap would work OneConnect = standard oneconnect profile (or a custom one with this as the parent) Default Persistence = cookie Fallback Persistence = source address Pool = Mattermost server pool iRule = custom Mattermost iRule Mattermost iRule To meet some of the business requirements a custom iRule was created to handle some of the conditions outlined. Comments in-line, but this checks to see if the connection was outside of the organization and if so verifies the presence of the custom HTTP header and value. This also checks to see if the connection was requested to upgrade to Websockets, and if it is, change the HTTP filter from full parsing to passthrough mode. when HTTP_REQUEST { if { !(IP:addr [IP::client_addr] equals 192.168.0.0/255.255.255.0]) } { Request from IP outside of organization, check for customer HTTP header if { [HTTP::header x-the-custom-http-header-name] contains "customvalue" } { Custom HTTP header and matching value found if { [string tolower [HTTP::header Upgrade]] contains "websocket" }{ Connection is requesting WebSockets, stop HTTP parsing HTTP::disable } } elseif { [HTTP::cookie exists MMAUTHENTOKEN] && [HTTP::cookie exists MMUSERID] } { Since WebSocket connections do not have HTTP Header, check to see if connection has already authenticated and allow the connection return } else { Connection fails conditions, reject it reject } } else { if { [string tolower [HTTP::header Upgrade]] contains "websocket" }{ Connection is requesting WebSockets, stop HTTP parsing HTTP::disable } } }[Workaound] User required to manually start EPI and VPN in browsers
After upgrading to version 16.1.4 the users need to manually start the End Point Inspector and the Web Initiated VPN by clicking on a "Start" button. This is describe in this KB. I created a user-common.js that will automatically click on the start button for the user. However, please note that this workround works as of 3rd of November 2023, but might stop working in the future in different browsers. In order to activate the workaround you need to have an Access Policy of the Moden type. Then go to Customizations -> Advanced -> Acces Profiles -> <Your Access Profile> -> Common Add the followinf to the file user-common.js define(["require", "exports", "apmui"], function (require, exports, apmui_1) { "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); var app = apmui_1.App.get(); app.subscribe(apmui_1.EventType.EPS_CHECK_PROGRESS, function (_, store) { var btns = document.getElementsByClassName("apmui-button"); if (btns.length == 0) { console.log("Failed to find button..."); return; } btns[0].click(); }); app.subscribe(apmui_1.EventType.DIALOG_OPEN, function (_, store) { setTimeout(function () { var dialog = document.getElementById("sna_auto_start_not_supported"); if (dialog == null) { console.log("Didn't find the right dialog"); return; } var btns = dialog.getElementsByClassName("apmui-button"); if (btns.length == 0) { console.log("Didn't find the start button"); return; } btns[0].click(); }, 100); }); }); If you have a better solution to this, please let me know. This was just what I came up with when asked by customers that the new "Start" button had created confusion among their users.F5 BIG-IP ASM – L7 DDoS attack Protection (TPS, Behavioral & Stress Based detection)
I would like to extend a big thank you to Mr.Mohamed_Ahmed_Kansoh for his invaluable help in creating this article on the F5 BIG-IP ASM - L7 DDoS protection You can also check this article on: https://nishalrai.com.np/2023/07/14/f5-big-ip-asm-l7-ddos-attack-protection/ F5 BIG-IP ASM DoS profile provided two distinct types of detection of DoS protection – TPS-based detection and Behavioral & Stress-based detection. TPS Based detection On threshold mode, if it is configured automatic then the TPS-based DoS protection is triggered, “If the ratio of the transaction rate during the detection interval to the transaction rate during the history interval is greater than the percentage indicated in the TPS increased by setting, the system considers the URL to be under attack, or the IP address to be suspicious.” Reference:Preventing DoS Attacks for Layer 7 Traffic On this post, We will discuss about the DoS protection profile especially the Behavioral and Stress Based detection features available on the F5 BIG-IP ASM module. One can find the DoS security profile protection profile on F5 Main Dashboard > Security > DOS Protection Behavioral & Stress based detection The server-based detection only implements the mitigation when there is significant latency from the backend server responses. In case of the occasional high spike of CPU usage on the backend server, it does trigger the stress-based DoS but when you view the output of the event timestamp. This sums up the significant latency on the server response, that ends up the stress-based DoS. To view the server response latency: Statistics > Dashboard > Behavioral DoS Regarding the behavior DoS protection of F5 BIG-IP ASM, the behavioral method is an intelligent way to identify and mitigate DoS attacks by creating a good baseline from normal traffic – users from normal or legitimate traffic. One needs to allocate sufficient time for F5 BIG-IP to create its baseline of learning good traffic. One can monitor the learned baseline of traffic on F5 BIG-IP: [root@bigipo01:Active:Standalone] config # admd -s vs./Common/vs_hackazon_http+/Common/hackazon_bados.info.learning - /Common/vs_hackazon_http – is the name of the virtual server - /Common/hackazon_bados – is the name of the DoS profile. **It may take several minutes for baseline numbers to be generated** Ref: Base Configuration and Traffic Baseline Behavioral Detection and mitigation stage When the F5 BIG-IP discovers any abnormal pattern of traffic that violates its learned baseline and observes the latency increased in Sever. It starts to create signatures for these attack patterns and starts to slow down / rate limits the request on the basis of the implemented mitigation technique – standard protection, aggressive protection, and others. F5 BIG-IP takes some headers from the violating/attack requests “abnormal requests ones” and stores it in a signature form, then these headers are bundled with each other within a single signature to identify the attacked request. So these are the parameters that F5 BIG-IP ASM DoS protection uses to create the signatures. In the ASM DoS profile, if “Use approved signature only” has been enabled then the administrator needs to open the signature and review its headers and information in order to approve it. In case you feel that it’s an attack then you need to disable it. If “Use approved signature only” is enabled in F5 BIG-IP ASM DoS profile, it will use the created signatures immediately without the administrator’s approval and will start to block any request that does not match the approved signature. Reference: Behavioral and stress-based detection settings Stress-Based detection The stress-based detection can be configured for detection criteria for server latency and transactions-per-second (TPS). F5 BIG-IP ASM marks the IP address as suspicious if any attacks trigger the set latency thresholds by the F5 BIG-IP. Once it has been triggered, the DoS engine starts collecting TPS information for the suspicious IP then analyzes TPS history rate to determine if it is a real DoS attack or a false positive. When the suspicious criteria TPS is reached, the BIG-IP ASM drops connections based on the policy - either by source IP or per URL. If the TPS (per IP address) is reached, the prevention policy switches to Source IP-Based Rate Limiting mode. If the TPS (per URL) is reached, URL-Based Rate Limiting is applied, and connections are reset. If the request is still considered an attack, the second and third policy are attempted with a two-minute interval between them. The BIG-IP ASM allows you to set a Prevention Duration to limit connections during a DoS attack. This defines the F5 BIG-IP ASM to considered the configured time period to either escalate or deescalate the configured mitigation methods – client-side integrity, CAPTCHA challenger or even block the request. Rate-Limit Prevention The rate-limiting is a mechanism used by the F5 BIG-IP ASM to prevent DDoS attacks on the protected applications or services. The F5 BIG-IP ASM calculates the average Transactions per seconds (TPS) for both the Detection Interval that lasts 10 seconds and the Historical Interval (last hour). Then, it updates them every 10 seconds. To rate-limit incoming traffic, F5 BIG-IP ASM uses a simple equation as: (History Interval TPS + Configured Threshold) / 2 For instance, if the absolute threshold is set to 200 TPS and the Historical Interval TPS average is 100 TPS, then the BIG-IP ASM will rate-limit incoming traffic to 150 TPS. The F5 BIG-IP ASM will rate-limit incoming traffic if the TPS exceeds the absolute threshold or if the sum of the relative threshold and the current TPS exceeds the absolute threshold. Note: You can review it from logs when attack was initiated, you can observe the limit which F5 BIG-IP was set and how it violated by crossing those configured threshold when the attack started. You have the option to record the DoS traffic on the DoS protection profile. During a DDoS attack, the recorded packet capture file shows that some traffic receives RST packets from the BIG-IP ASM, while other traffic receives normal responses (200 OK). This indicates that, while rate-limiting can prevent some traffic from passing through the BIG-IP ASM, it is not a foolproof method. Compared to blocking all traffic, rate-limiting allows some traffic to pass through the BIG-IP ASM, while blocking all traffic resets all traffic from a specific source. Bad Actors behavior detection On the bad actor’s behavior detection, F5 BIG-IP monitors and tracks the server response time. Once significant server latency is observed then starts to keep the track of all the ingress traffic request ip address, monitors their request and calculates the value (kind of score either to quarantine or discard the further request). To display the IP greylist: ipidr -l /Common/<vs-profile>+/Common/<dos-profile> To view the current maximum TPS of the DoS profile. tmsh show security dos profile my_dos_profile In-Sight of F5 BIG-IP ASM DoS Profile Sometimes the spike in CPU usage on the backend server triggers the attached ASM DoS profile. So, what would be the F5 BIG-IP approach to learn and detect those spike patterns as a DoS attack like for certain intervals of time - will it ignore those spikes and only after a certain interval of time with a predefined % of CPU usage of the backend server like +90% for 5 minutes or less that triggers the DoS profile. and if so, can we customize those defined parameters on the DoS profile. Such behavior has resulted into a lot of false positive. Stress-based mitigation will not be triggered if only configured TPS was exceeded and high latencies coming from servers are in AND Boolean condition. With such configuration, even if servers encountered spike due to high number of transactions TPS, F5 BIG-IP will not start mitigation if only there is high latency in server side. In short, even if the dos profile is triggered when the server-side latency is automatically spiked without any violation of the configured baseline TPS, the implemented mitigation will not be enforced on the new inbound traffic. Moreover, we cannot manually configure the baseline of those CPU usage and server latency percentage threshold. Even though F5 has released the white paper addressing those changes can be possible in the F5 BIG-IP, those changes are not possible on the major version or may not be possible on the preceding version too. Reference: Intelligent Layer 7 DoS and Brute Force Protection for Web Applications | F5 White Paper dosl7d dosl7d daemon runs in the background which will continuously monitor the two major variables – Memory Usage and TMM count. The services and scripts attached to the dosl7d process All those logics and algorithms to detect those DoS detection based on the behavioral and stress-based detection has been compiled and stored on the dosl7 dosl7d daemon responsible for detection of the DoS attacks (when triggered) and the log are stored on the /var/log/dosl7d In the directory, /etc/bigstart/startup Scripts of dosl7d - /etc/bigstart/scripts Sample of dosl7d logs (CLI & GUI) F5 BIG-IP GUI Why can't we configure SNMP alert for L7 dos attack on F5 BIG-IP, right now? In F5 Big-IP v 11.3.0 and later, messages generated by the dosl7d daemon cannot trigger commands or custom scripts because they are not processed by the alertd SNMP process. This is because the dosl7d daemon writes directly to its log file (/var/log/dosl7d/dosl7d.log) instread of using syslog facilities. As a result, the messages the dosl7d daemon issues do not pass through the syslog pipe. So, the alertd daemon cannot see the dosl7d messages and unable to trigger SNMP traps. The only current options for detecting a DoS attack is through an external logging device. The other workaround which will implicitly solve the existing issue by creating an email alert notification when the configured threshold (manual TPS of the DoS profile or server latency or others) has been triggered on the selected virtual instances, pools or application. However, the limitation of this configuration can be the inability to select the specific virtual instances and pools to configure separate threshold values to trigger. But it can somehow mitigate the issues of implementing dos protection mitigation without being noticed by the F5 administrator on high critical business oriented applications or services.Get Started with F5 NGINX Plus, Ingress, App Protect and Management Suite
These free trials available for F5 NGINX Products will get you started on whats available and help you get certified. Try NGINX Plus & NGINX App Protect Try NGINX Ingress Controller with NGINX App Protect Try NGINX Management Suite - NGINX
