XC Distributed Cloud and how to keep the Source IP from changing with customer edges(CE)!
The best will always be the application to stop tracking users based on something primitive as an ip address and sometimes the issue is in the Load Balancer or ADC after the XC RE and then if the persistence is based on source IP address on the ADC to be changed in case it is BIG-IP to Cookie or Universal or SSL session based if the Load Balancer is doing no decryption and it is just TCP/UDP layer . As an XC Regional Edge (RE) has many ip addresses it can connect to the origin servers adding a CE for the legacy apps is a good option to keep the source IP from changing for the same client HTTP requests during the session/transaction. Before going through this article I recommend reading the links below: F5 Distributed Cloud – CE High Availability Options: A Comparative Exploration | DevCentral F5 Distributed Cloud - Customer Edge | F5 Distributed Cloud Technical Knowledge Create Two Node HA Infrastructure for Load Balancing Using Virtual Sites with Customer Edges | F5 Distributed Cloud Technical Knowledge RE to CE cluster of 3 nodes The new SNAT prefix option under the origin pool allows no mater what CE connects to the origin pool the same IP address to be seen by the origin. Be careful as if you have more than a single IP with /32 then again the client may get each time different IP address. This may cause "inet port exhaustion " ( that is what it is called in F5BIG-IP) if there are too many connections to the origin server, so be careful as the SNAT option was added primary for that use case. There was an older option called "LB source IP persistence" but better not use it as it was not so optimized and clean as this one. RE to 2 CE nodes in a virtual site The same option with SNAT pool is not allowed for a virtual site made of 2 standalone CE. For this we can use the ring hash algorithm. Why this works? Well as Kayvan explained to me the hashing of the origin is taking into account the CE name, so the same origin under 2 different CE will get the same ring hash and the same source IP address will be send to the same CE to access the Origin Server. This will not work for a single 3-node CE cluster as it all 3 nodes have the same name. I have seen 503 errors when ring hash is enabled under the HTTP LB so enable it only under the XC route object and the attached origin pool to it! CE hosted HTTP LB with Advertise policy In XC with CE you can do do HA with 3-cluster CE that can be layer2 HA based on VRRP and arp or Layer 3 persistence based bgp that can work 3 node CE cluster or 2 CE in a virtual site and it's control options like weight, as prepend or local preference options at the router level. For the Layer 2 I will just mention that you need to allow 224.0.0.8 for the VRRP if you are migrating from BIG-IP HA and that XC selects 1 CE to hold active IP that is seen in the XC logs and at the moment the selection for some reason can't be controlled. if a CE can't reach the origin servers in the origin pool it should stop advertising the HTTP LB IP address through BGP. For those options Deploying F5 Distributed Cloud (XC) Services in Cisco ACI - Layer Three Attached Deployment is a great example as it shows ECMP BGP but with the BGP attributes you can easily select one CE to be active and processing connections, so that just one ip address is seen by the origin server. When a CE gets traffic by default it does prefer to send it to the origin as by default "Local Preferred" is enabled under the origin pool. In the clouds like AWS/Azure just a cloud native LB is added In front of the 3 CE cluster and the solution there is simple as to just modify the LB to have a persistence. Public Clouds do not support ARP, so forget about Layer 2 and play with the native LB that load balances between the CE 😉 Hope you enjoyed this article!F5 XC Distributed Cloud HTTP Header/Cookie manipulations and using the client ip/user headers
1 . F5 XC distributed cloud HTTP Header manipulations In the F5 XC Distributed Cloud some client information is saved to variables that can be inserted in HTTP headers similar to how F5 Big-IP saves some data that can after that be used in a iRule or Local Traffic Policy. By default XC will insert XFF header with the client IP address but what if the end servers want an HTTP header with another name to contain the real client IP. Under the HTTP load balancer under "Other Options" under "More Options" the "Header Options" can be found. Then the the predefined variables can be used for this job like in the example below the $[client_address] is used. A list of the predefined variables for F5 XC: https://docs.cloud.f5.com/docs/how-to/advanced-security/configure-http-header-processing There is $[user] variable and maybe in the future if F5 XC does the authentication of the users this option will be insert the user in a proxy chaining scenario but for now I think that this just manipulates data in the XAU (X-Authenticated-User) HTTP header. 2. Matching of the real client ip HTTP headers You can also match a XFF header if it is inserted by a proxy device before the F5 XC nodes for security bypass/blocking or for logging in the F5 XC. For User logging from the XFF Under "Common Security Controls" create a "User Identification Policy". You can also match a regex that matches the ip address and this is in case there are multiple IP addresses in the XFF header as there could have been many Proxy devices in the data path and we want see if just one is present. For Security bypass or blocking based based on XFF Under "Common Security Controls" create a "Trusted Client Rules" or "Client Blocking Rules". Also if you have "User Identification Policy" then you can just use the "User Identifier" but it can't use regex in this case. I have made separate article about User-Identification F5 XC Session tracking and logging with User Identification Policy | DevCentral To match a regex value in the header that is just a single IP address, even when the header has many ip addresses, use the regex (1\.1\.1\.1) as an example to mach address 1.1.1.1. To use the client IP address as a source Ip address to the backend Origin Servers in the TCP packet after going through the F5 XC (similar to removing the SNAT pool or Automap in F5 Big-IP) use the option below: The same way the XAU (X-Authenticated-User) HTTP header can be used in a proxy chaining topology, when there is a proxy before the F5 XC that has added this header. Edit: Keep in mind that in some cases in the XC Regex for example (1\.1\.1\.1) should be written without () as 1\.1\.1\.1 , so test it as this could be something new and I have seen it in service policy regex matches, when making a new custom signature that was not in WAAP WAF XC policy. I could make a seperate article for this 🙂 XC can even send the client certificate attributes to the backend server if Client Side mTLS is enabled but it is configured at the cert tab. 3. F5 XC distributed cloud HTTP Cookie manipulations. Now you can overwrite the XC cookie by keeping the value but modifying the tags and this is big thing as before this was not possible. When combined with cookies this becomes very powerful thing as you can match on User-Agent header and for Mozilla for example to change the flags as if there is bug with the browser etc. The feature changes cookies returned in the Response Set-Cookie header from the origin server as it should.F5 DNS/GTM External Monitor(EAV) with SNI support and response code check
I have used this monitor for XC Distributed Cloud as the HTTP LB share by default the same tenant IP address and SNI support is needed. You can order dedicated public IP addresses for each HTTP LB and enable "Default Load Balancer" ( https://my.f5.com/manage/s/article/K000152902 ) option but it will cost you extra 😉 The script is a modified version of External https health monitor for SNI-enabled pool as to handle response codes and to set the SNI globally for the entire pool and it's members. If you are uploading from Windows machine see External monitor fails to run as you could hit the bug. This could be needed for F5 DNS/GTM below 16.1 that do not support SNI in HTTPS monitors. The only mandatory variable is "SNI" that should be set in the external monitor config that references this uploaded bash script. The "URI" variable by default is set to "/" and "$2" variable by default is empty or 443, the default expected response code 200. #!/bin/sh # External monitoring script for checking HTTP status code # $1 = IP (::ffff:nnn.nnn.nnn.nnn notation or hostname) # $2 = port (optional; defaults to 443 if not provided) # Default SNI to IP if not explicitly provided node_ip=$(echo "$1" | sed 's/::ffff://') # Remove IPv6 compatibility prefix SNI=${SNI:-"$node_ip"} # Assign sanitized IP to SNI # Default variables MON_NAME=${MON_NAME:-"MyExtMon$$"} pidfile="/var/run/$MON_NAME.$1..$2.pid" # PID file path DEBUG=${DEBUG:-0} # Enable debugging if set to 1 EXPECTED_STATUS=${EXPECTED_STATUS:-200} # Default HTTP status code to 200 URI=${URI:-"/"} # Default URI DEFAULT_PORT=443 # Default port (used if $2 is unset) # Set port to default if $2 is not provided if [ -z "${2}" ]; then PORT=${DEFAULT_PORT} else PORT=${2} fi # Kill old process if pidfile exists if [ -f "$pidfile" ]; then kill -9 -$(cat "$pidfile") > /dev/null 2>&1 fi echo "$$" > "$pidfile" # Perform the HTTP(S) request via single curl (fetch status code only) status_code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 5 --resolve "${SNI}:${PORT}:${node_ip}" "https://${SNI}:${PORT}${URI}") # Cleanup rm -f "$pidfile" > /dev/null 2>&1 # Output server status based on HTTP status code match if [ "$status_code" -eq "$EXPECTED_STATUS" ]; then echo "up" else echo "down" fi # Debugging if [ "$DEBUG" -eq 1 ]; then echo "Debugging on..." echo "SNI=${SNI}" echo "URI=${URI}" echo "IP=${node_ip}" echo "PORT=${PORT}" echo "MON_NAME=${MON_NAME}" echo "STATUS_CODE=${status_code}" echo "EXPECTED_STATUS=${EXPECTED_STATUS}" echo "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 5 --resolve ${SNI}:${PORT}:${node_ip} https://${SNI}:${PORT}${URI}" fiF5 XC – Persistence and Resiliency pt. I (persistence)
Since we're talking about distributed environments (multiple PoPs, CEs, etc.) and need to ensure consistent behavior without necessarily being able to form a cluster (e.g., PoP Frankfurt and PoP Amsterdam), we use a method that is suitable for distributed structures: Ring Hashing (Consistent Hashing).
F5 Roles required for Catalog Items
Having difficulty mapping required roles for a group to have proper access to catalog items. If I create a group call Security-Team and I want them to manage the security like WAF (Web App * API Protection) and Bot Defense, Web App Scanning and whatever else the Security Team should be monitoring to keep our environment safe. What Roles are required for management? They don't need access to everything, just what is required for the application security. Then we have a group called Support-Teams that need ReadOnly access to everything so they can log into F5 XC and just view everything with no ability to make changes. Not sure what Roles would get assigned to this group. Both scenarios let's assume all namespaces. Any help or direction is most appreciated.
