F5 XC Distributed Cloud HTTP Header/Cookie manipulations and using the client ip/user headers
1 . F5 XC distributed cloud HTTP Header manipulations In the F5 XC Distributed Cloud some client information is saved to variables that can be inserted in HTTP headers similar to how F5 Big-IP saves some data that can after that be used in a iRule or Local Traffic Policy. By default XC will insert XFF header with the client IP address but what if the end servers want an HTTP header with another name to contain the real client IP. Under the HTTP load balancer under "Other Options" under "More Options" the "Header Options" can be found. Then the the predefined variables can be used for this job like in the example below the $[client_address] is used. A list of the predefined variables for F5 XC: https://docs.cloud.f5.com/docs/how-to/advanced-security/configure-http-header-processing There is $[user] variable and maybe in the future if F5 XC does the authentication of the users this option will be insert the user in a proxy chaining scenario but for now I think that this just manipulates data in the XAU (X-Authenticated-User) HTTP header. 2. Matching of the real client ip HTTP headers You can also match a XFF header if it is inserted by a proxy device before the F5 XC nodes for security bypass/blocking or for logging in the F5 XC. For User logging from the XFF Under "Common Security Controls" create a "User Identification Policy". You can also match a regex that matches the ip address and this is in case there are multiple IP addresses in the XFF header as there could have been many Proxy devices in the data path and we want see if just one is present. For Security bypass or blocking based based on XFF Under "Common Security Controls" create a "Trusted Client Rules" or "Client Blocking Rules". Also if you have "User Identification Policy" then you can just use the "User Identifier" but it can't use regex in this case. I have made separate article about User-Identification F5 XC Session tracking and logging with User Identification Policy | DevCentral To match a regex value in the header that is just a single IP address, even when the header has many ip addresses, use the regex (1\.1\.1\.1) as an example to mach address 1.1.1.1. To use the client IP address as a source Ip address to the backend Origin Servers in the TCP packet after going through the F5 XC (similar to removing the SNAT pool or Automap in F5 Big-IP) use the option below: The same way the XAU (X-Authenticated-User) HTTP header can be used in a proxy chaining topology, when there is a proxy before the F5 XC that has added this header. Edit: Keep in mind that in some cases in the XC Regex for example (1\.1\.1\.1) should be written without () as 1\.1\.1\.1 , so test it as this could be something new and I have seen it in service policy regex matches, when making a new custom signature that was not in WAAP WAF XC policy. I could make a seperate article for this 🙂 XC can even send the client certificate attributes to the backend server if Client Side mTLS is enabled but it is configured at the cert tab. 3. F5 XC distributed cloud HTTP Cookie manipulations. Now you can overwrite the XC cookie by keeping the value but modifying the tags and this is big thing as before this was not possible. When combined with cookies this becomes very powerful thing as you can match on User-Agent header and for Mozilla for example to change the flags as if there is bug with the browser etc. The feature changes cookies returned in the Response Set-Cookie header from the origin server as it should.Home Lab Server Build Using an Intel NUC and Free VMware ESXi 7
If you're like me, despite having cheap or even free access to cloud compute, you still want to have a bit of compute in a home lab. I can create and destroy to my hearts content. Things can get weird and messy - and it's nobodys problem but my own. For the past 10 years, my home lab has consisted of a couple 2U Dell R710 servers. They are were beefy in specs but they are very loud and consume a relatively large amount of power and space. They have served me really well over the years but it is finally time to upgrade. I ordered an Intel NUC last year. It should be able to handle the workload I'm running on my Dell servers with room to spare. Due to supply chain issues, it took a few months but it finally arrived. I was extremely surprised at how small these are. I knew they were small but I did not expect it to fit in the palm of my hand! I threw on VMware ESXi 7 for the hypervisor but I wanted to document the build for anyone who is building up a similar setup as I encountered a couple issues during my installation. Here is my complete parts list: Intel NUC11TNKV7 2x Kingston 32GB DDR4 3200MHz SODIMM 1TB Samsung 970 EVO NVMe I did document this in a video but this article also serves as a companion to that since there is a lot of commands involved. I immediately found out that because the network card on the NUC does not have a compatible driver included on the ESXi 7 image, I had to create an ISO with the Community Network Driver (Fling). The steps are documented here: https://www.virten.net/2021/11/vmware-esxi-7-0-update-3-on-intel-nuc/ however I also came across my own nuances which I'm noting below. First, download the ESXi Offline Bundle and Fling Community Network Driver and place them in a temporary folder. You need to install the vmware.powercli and vmware.imagebuilder modules from the Powershell command line install-module -name vmware.powercli install-module -name vmware.imagebuilder HOWEVER vmware.powercli and vmware.imagebuilder modules for Powershell is not supported on Powershell v6 and above which meant I could not run these commands on my Mac. Luckily, I had a Windows box kicking around with Powershell v5. I was also getting an error in trying to download the VMware.imagebuilder plugin. As it turns out, my version of PowerShell must have been using TLS 1.0/1.1. These intructions configured TLS1.2: https://docs.microsoft.com/en-us/powershell/scripting/gallery/installing-psget?view=powershell-7.2 [Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12 After all that, I was able to proceed with building the image. The steps were pretty close to what is in the Virten article however the version of ESXi they used was pulled and replaced. I ended up with a different build which is reflected with the file names I used. Add-EsxSoftwareDepot .\VMware-ESXi-7.0U3c-19193900-depot.zip Add-EsxSoftwareDepot .\Net-Community-Driver_1.2.2.0-1vmw.700.1.0.15843807_18835109.zip New-EsxImageProfile -CloneProfile "ESXi-7.0U3c-19193900-standard" -name "ESXi-7.0U3c-19193900-NUC" -Vendor "buulam" Add-EsxSoftwarePackage -ImageProfile "ESXi-7.0U3c-19193900-NUC" -SoftwarePackage "net-community" Export-ESXImageProfile -ImageProfile "ESXi-7.0U3c-19193900-NUC" -ExportToISO -filepath ESXi-7.0U3c-19193900-NUC.iso Note: If you encounter the following error: "windowspowershell\modules\vmware.vimautomation.sdk\12.5.0.19093564\vmware.vimautomation.sdk.psm1 cannot be loaded because running scripts is disabled on this system" you may need to enter the following command: Set-ExecutionPolicy -ExecutionPolicy AllSigned Credit to Pawan Jheeta for this find! Now that I have an ISO image with the Fling Community Network Driver, it was time to create the bootable USB installer. I have a Mac and here are the steps I used to create the USB flash drive: https://virtuallywired.io/2020/08/01/create-a-bootable-esxi-7-usb-installer-on-macos/. I did not encounter any issues with these steps so please refer to the linked article to follow them. In case you are running Windows, this appears to be a good guide for creating the USB flash drive: https://www.virten.net/2014/12/howto-create-a-bootable-esxi-installer-usb-flash-drive/ Once you have the bootable USB flash drive created, you can insert that into the Intel NUC and begin your ESXi installation. The remaining steps I will leave to be explained in my video. I accepted all the defaults except for configuring a static IP address for the management address. I hope this helps some of you out and if there are any questions, please reply to this thread. I'd also love to hear about your home labs!What is the Lightning Network?
When I'm thinking of up and coming technologies in terms of how they'd fit into my everyday life, I often forget that there are things I assume for myself that aren't nessecarily true for others. One of these things is the ease at which I can transact with people and businesses. I can move Canadian dollars to other Canadians for free and instantly. I can exchange money for goods and services from a merchant with just a tap of my phone or bank card. But this is simply not the case in third world countries where banking systems are not as mature or trusted. Blockchain technology has enabled a number of distruptive use cases, over and above enabling something like Bitcoin. Now what we're seeing is use cases that enable anybody with internet connectivity to be able to execute transactions with others in a direct manner. A use case that builds on this idea is payment exchange in third world countries and this is built on the Lightning Network. The Lightning Network is a layer 2 payment protocol. It is built on top of the Bitcoin network but instead of waiting up to 10 minutes for transactions to settle, this side-chain or layer 2 network can transact instantly. It's capable of making large and small transactions so it has use cases that can serve C2C, B2C and B2B. Imagine yourself travelling through Vietnam. You bought lunch. It was $2USD. You don't have the benefit of tap to pay like in your home country. You have some local currency but you'd prefer not to keep breaking up your larger bills. Constantly converting the currency in your head to keep track of your holiday spending is taking away from the fun of your vacation. It's also harder and harder as the larger bills get broken down into smaller ones. But if you can pay through Lightning Network, you settle the transaction in Bitcoin and know exactly how much you've spent. Or let's say you're a student in North America. Your parents are back home overseas and they need to send over some money for the year's tuition. Money transfer agents can help you move the money but at a cost. With Lightning Network, the money can be moved immediately and for little cost. Or, let's say you're a business that needs to wire funds to your supplier. Normally, you'd go to the bank, fill out a wire transfer, hope you got all the numbers right and then wait 5 days for the money to show up in the suppliers account. With Lightning Network, that transaction can happen immediately and it can be tracked electronically to show it was received. The market is flooded with a lot of Blockchain based projects that are still finding their way but I am confident that Lightning Network is something that's going to take off in certain parts of the world. I was able to arrange an interview with Albert Buu, the Founder and CEO of Neutronpay, a Lightning Network Service Provider (LSP) and got to deep dive into his insights on this emerging use case!
XC Distributed Cloud and how to keep the Source IP from changing with customer edges(CE)!
The best will always be the application to stop tracking users based on something primitive as an ip address and sometimes the issue is in the Load Balancer or ADC after the XC RE and then if the persistence is based on source IP address on the ADC to be changed in case it is BIG-IP to Cookie or Universal or SSL session based if the Load Balancer is doing no decryption and it is just TCP/UDP layer . As an XC Regional Edge (RE) has many ip addresses it can connect to the origin servers adding a CE for the legacy apps is a good option to keep the source IP from changing for the same client HTTP requests during the session/transaction. Before going through this article I recommend reading the links below: F5 Distributed Cloud – CE High Availability Options: A Comparative Exploration | DevCentral F5 Distributed Cloud - Customer Edge | F5 Distributed Cloud Technical Knowledge Create Two Node HA Infrastructure for Load Balancing Using Virtual Sites with Customer Edges | F5 Distributed Cloud Technical Knowledge RE to CE cluster of 3 nodes The new SNAT prefix option under the origin pool allows no mater what CE connects to the origin pool the same IP address to be seen by the origin. Be careful as if you have more than a single IP with /32 then again the client may get each time different IP address. This may cause "inet port exhaustion " ( that is what it is called in F5BIG-IP) if there are too many connections to the origin server, so be careful as the SNAT option was added primary for that use case. There was an older option called "LB source IP persistence" but better not use it as it was not so optimized and clean as this one. RE to 2 CE nodes in a virtual site The same option with SNAT pool is not allowed for a virtual site made of 2 standalone CE. For this we can use the ring hash algorithm. Why this works? Well as Kayvan explained to me the hashing of the origin is taking into account the CE name, so the same origin under 2 different CE will get the same ring hash and the same source IP address will be send to the same CE to access the Origin Server. This will not work for a single 3-node CE cluster as it all 3 nodes have the same name. I have seen 503 errors when ring hash is enabled under the HTTP LB so enable it only under the XC route object and the attached origin pool to it! CE hosted HTTP LB with Advertise policy In XC with CE you can do do HA with 3-cluster CE that can be layer2 HA based on VRRP and arp or Layer 3 persistence based bgp that can work 3 node CE cluster or 2 CE in a virtual site and it's control options like weight, as prepend or local preference options at the router level. For the Layer 2 I will just mention that you need to allow 224.0.0.8 for the VRRP if you are migrating from BIG-IP HA and that XC selects 1 CE to hold active IP that is seen in the XC logs and at the moment the selection for some reason can't be controlled. if a CE can't reach the origin servers in the origin pool it should stop advertising the HTTP LB IP address through BGP. For those options Deploying F5 Distributed Cloud (XC) Services in Cisco ACI - Layer Three Attached Deployment is a great example as it shows ECMP BGP but with the BGP attributes you can easily select one CE to be active and processing connections, so that just one ip address is seen by the origin server. When a CE gets traffic by default it does prefer to send it to the origin as by default "Local Preferred" is enabled under the origin pool. In the clouds like AWS/Azure just a cloud native LB is added In front of the 3 CE cluster and the solution there is simple as to just modify the LB to have a persistence. Public Clouds do not support ARP, so forget about Layer 2 and play with the native LB that load balances between the CE 😉 CE on Public Cloud (AWS/Azure/GCP) When deploying on a public cloud the CE can be deployed in two ways one is through the XC GUI and adding the AWS credentials but this way you have not a big freedom to be honest as you can't deploy 2 CE and make a virtual site out of them and add cloud LB in-front of them as it always will be 3-CE cluster with preconfigured cloud LB that will use all 3 LB! Using the newer "clickops" method is much better https://docs.cloud.f5.com/docs-v2/multi-cloud-network-connect/how-to/site-management/deploy-site-aws-clickops or using terraform but with manual mode and aws as a provider (not XC/volterra as it is the same as the XC GUI deployment) https://docs.cloud.f5.com/docs-v2/multi-cloud-network-connect/how-to/site-management/deploy-aws-site-terraform This way you can make the Cloud LB to use just one CE or have some client Persistence or if traffic comes from RE to CE to implement the virtual site 2 CE node! There is no Layer 2 ARP support as I mentioned in public cloud with 3-node cluster but there is NAT policy https://docs.cloud.f5.com/docs-v2/multi-cloud-network-connect/how-tos/networking/nat-policies but I haven't tried it myself to comment on it. Hope you enjoyed this article!The sooner the better: Web App Scanning Without Internet Exposure
In the fast-paced world of app development, security often takes a backseat to feature delivery and tight deadlines. Many organizations rely on external teams to perform penetration testing on their web applications and APIs, but this typically happens after the app has been live for some time and is driven by compliance or regulatory requirements. Waiting until this stage can leave vulnerabilities unaddressed during critical early phases, potentially leading to costly fixes, reputational damage, or even breaches. Early-stage application security testing is key to building a strong foundation and mitigating risks before they escalate. Wouldn't it be cool if there was a way you could scan your apps in a proactive, automated way while they are still in beta? Since you are reading this article, here in the F5 community, you probably already know, that F5 Distributed Cloud Web App Scanning allows you to dynamically and continuously scan your external attack surface to uncover exposed web apps and APIs. We all know that exposing apps, which are at an early stage of their development, to the internet is risky because they may contain unfinished or untested code that could lead to unintended data leaks, privacy violations, or other risks. Therefore you want to keep access to your beta-stage apps restricted. Scanning but not exposing your apps At this point in time XC Web App Scanning can only scan apps that are exposed on the internet. But with some configuration tweaks, you can ensure that only WAS has access to your apps. I want to show a real-world example of how you can restrict access to your application solely to the XC WAS scan engine. Let's take a look at the beta-stage application we aim to perform penetration testing on. It is hosted on an EC2 instance in AWS. Of course we don't plan to expose our application directly to the internet with a Web Application Firewall. Hence F5 Distributed Cloud Web App & API Protection (WAAP) will be positioned as a cloud-proxy in front of our app. Therefore we must make sure only traffic from F5 Distributed Cloud Services has access to our app. Next we want to make sure that only the scan engine of F5 Distributed Cloud Web App Scanning can reach our app, again we want to block the rest of the internet from accessing our app. A pictures of says more then words, we want to achieve something like this: How to set it up Let's take a look how we can satisfy our requirements. ... in AWS In AWS Security Groups are used to control which traffic is allowd to reach reach and leave the resources that it is associated with. Since our application is hosted on an EC2 instance, the Security Group controlls the ingress and egress traffic for the instance. One can think of it like a virtual packet filter firewall. Usually protocol, port and a source IP address range in CIDR notation for an inbound Security Group. We want to allow access only from F5 Distributed Cloud Services to our EC2 instance. Creating hundreds of ingress rules inside of a Security Group did not seem very efficient to me. Hence I used a customer-managed prefix lists and added all F5 Regional Edges. Prefix lists are configured in the VPC section of AWS. The IPv4 address list of all F5 Regional Edges is available here: Public IPv4 Subnet Ranges for F5 Regional Edges After you created you prefix list, you can use it in a Security Group This way we met our first goal. Only F5 Regional Edges can reach our app. ... in XC In the F5 Distributed Cloud a similar kind of access control can be achieved by using Service Policies. Service Policies are a mechanism to control and enforce fine-grained access control for applications deployed in XC. I created a Service Policy that allows access only from the list of ephemeral IP addresses associated with XC Web App Scanning, while blocking all other traffic. First create a Service Policy, in the Rules section select Allowed Sources. In the XC Console Service Policies are created in Security > Service Policies > Service Policies. Then add the IP ranges to the IPv4 Prefix List. The list of all IP addresses associated with XC Web App Scanning is available here: Use Known IPs in Web App Scanning The Service Policy is then applied in the section Common Security Controls of a HTTP Load Balancer configuration. Conclusion By combining AWS Security Groups and XC Security Policies, I can ensure that my beta app (or beta API) is accessible exclusively to the scan engine of XC Web App Scanning, while blocking access from malicious actors on the internet.
