F5 XC CE Debug commands through GUI cloud console and API
Why this feature is important and helpful? With this capability if the IPSEC/SSL tunnels are up from the Customer Edge(CE) to the Regional Edge(RE), there is no need to log into the CE, when troubleshooting is needed. This is possible for Secure Mesh(SM) and Secure Mesh V2 (SMv2) CE deployments. As XC CE are actually SDN-based ADC/proxy devices the option to execute commands from the SDN controller that is the XC cloud seems a logical next step. Using the XC GUI to send SiteCLI debug commands. The first example is sending the "netstat" command to "master-3" of a 3-node CE cluster. This is done under Home > Multi-Cloud Network Connect > Overview > Infrastructure > Sites and finding the site, where you want to trigger the commands. In the VPM logs it is possible to see the command that was send in API format by searching for it or for logs starting with "debug", as to automate this task. If you capture and review the full log, you will even see not only the API URL endpoint but also the POST body data that needs to be added. The VPM logs that can also be seen from the web console and API, are the best place to start investigating issues. XC Commands reference: Node Serviceability Commands Reference | F5 Distributed Cloud Technical Knowledge Troubleshooting Guidelines for Customer Edge Site | F5 Distributed Cloud Technical Knowledge Troubleshooting Guide for Secure Mesh Site v2 Deployment | F5 Distributed Cloud Technical Knowledge Using the XC API to send SiteCLI debug commands. The same commands can be send using the XC API and first the commands can be tested and reviewed using the API doc and developer portals. API documentation even has examples of how to run these commands with vesctl that is the XC shell client that can be installed on any computer or curl. Postman can also be used instead of curl but the best option to test commands through the API is the developer portal. Postman can also be used by the "old school" people 😉 Link reference: F5 Distributed Cloud Services API for ves.io.schema.operate.debug | F5 Distributed Cloud Technical Knowledge F5 Distributed Cloud Dev Portal ves-io-schema-operate-debug-CustomPublicAPI-Exec | F5 Distributed Cloud Technical Knowledge Summary: The option to trigger commands though the XC GUI or even the API is really useful if for example there is a need to periodically monitor the cpu or memory jump with commands like "execcli check-mem" or "execcli top" or even automating the tcpdump with "execcli vifdump xxxx". The use cases for this functionality really are endless.F5 XC Distributed Cloud HTTP Header manipulations and matching of the client ip/user HTTP headers
1 . F5 XC distributed cloud HTTP Header manipulations In the F5 XC Distributed Cloud some client information is saved to variables that can be inserted in HTTP headers similar to how F5 Big-IP saves some data that can after that be used in a iRule or Local Traffic Policy. By default XC will insert XFF header with the client IP address but what if the end servers want an HTTP header with another name to contain the real client IP. Under the HTTP load balancer under "Other Options" under "More Options" the "Header Options" can be found. Then the the predefined variables can be used for this job like in the example below the $[client_address] is used. A list of the predefined variables for F5 XC: https://docs.cloud.f5.com/docs/how-to/advanced-security/configure-http-header-processing There is $[user] variable and maybe in the future if F5 XC does the authentication of the users this option will be insert the user in a proxy chaining scenario but for now I think that this just manipulates data in the XAU (X-Authenticated-User) HTTP header. 2. Matching of the real client ip HTTP headers You can also match a XFF header if it is inserted by a proxy device before the F5 XC nodes for security bypass/blocking or for logging in the F5 XC. For User logging from the XFF Under "Common Security Controls" create a "User Identification Policy". You can also match a regex that matches the ip address and this is in case there are multiple IP addresses in the XFF header as there could have been many Proxy devices in the data path and we want see if just one is present. For Security bypass or blocking based based on XFF Under "Common Security Controls" create a "Trusted Client Rules" or "Client Blocking Rules". Also if you have "User Identification Policy" then you can just use the "User Identifier" but it can't use regex in this case. To match a regex value in the header that is just a single IP address, even when the header has many ip addresses, use the regex (1\.1\.1\.1) as an example to mach address 1.1.1.1. To use the client IP address as a source Ip address to the backend Origin Servers in the TCP packet after going through the F5 XC (similar to removing the SNAT pool or Automap in F5 Big-IP) use the option below: The same way the XAU (X-Authenticated-User) HTTP header can be used in a proxy chaining topology, when there is a proxy before the F5 XC that has added this header. Edit: Keep in mind that in some cases in the XC Regex for example (1\.1\.1\.1) should be written without () as 1\.1\.1\.1 , so test it as this could be something new and I have seen it in service policy regex matches, when making a new custom signature that was not in WAAP WAF XC policy. I could make a seperate article for this 🙂 XC can even send the client certificate attributes to the backend server if Client Side mTLS is enabled but it is configured at the cert tab.
