Automate Let's Encrypt Certificates on BIG-IP
To quote the evil emperor Zurg: "We meet again, for the last time!" It's hard to believe it's been six years since my first rodeo with Let's Encrypt and BIG-IP, but (uncompromised) timestamps don't lie. And maybe this won't be my last look at Let's Encrypt, but it will likely be the last time I do so as a standalone effort, which I'll come back to at the end of this article. The first project was a compilation of shell scripts and python scripts and config files and well, this is no different. But it's all updated to meet the acme protocol version requirements for Let's Encrypt. Here's a quick table to connect all the dots: Description What's Out What's In acme client letsencrypt.sh dehydrated python library f5-common-python bigrest BIG-IP functionality creating the SSL profile utilizing an iRule for the HTTP challenge The f5-common-python library has not been maintained or enhanced for at least a year now, and I have an affinity for the good work Leo did with bigrest and I enjoy using it. I opted not to carry the SSL profile configuration forward because that functionality is more app-specific than the certificates themselves. And finally, whereas my initial project used the DNS challenge with the name.com API, in this proof of concept I chose to use an iRule on the BIG-IP to serve the challenge for Let's Encrypt to perform validation against. Whereas my solution is new, the way Let's Encrypt works has not changed, so I've carried forward the process from my previous article that I've now archived. I'll defer to their how it works page for details, but basically the steps are: Define a list of domains you want to secure Your client reaches out to the Let’s Encrypt servers to initiate a challenge for those domains. The servers will issue an http or dns challenge based on your request You need to place a file on your web server or a txt record in the dns zone file with that challenge information The servers will validate your challenge information and notify you You will clean up your challenge files or txt records The servers will issue the certificate and certificate chain to you You now have the key, cert, and chain, and can deploy to your web servers or in our case, to the BIG-IP Before kicking off a validation and generation event, the client registers your account based on your settings in the config file. The files in this project are as follows: /etc/dehydrated/config # Dehydrated configuration file /etc/dehydrated/domains.txt # Domains to sign and generate certs for /etc/dehydrated/dehydrated # acme client /etc/dehydrated/challenge.irule # iRule configured and deployed to BIG-IP by the hook script /etc/dehydrated/hook_script.py # Python script called by dehydrated for special steps in the cert generation process # Environment Variables export F5_HOST=x.x.x.x export F5_USER=admin export F5_PASS=admin You add your domains to the domains.txt file (more work likely if signing a lot of domains, I tested the one I have access to). The dehydrated client, of course is required, and then the hook script that dehydrated interacts with to deploy challenges and certificates. I aptly named that hook_script.py. For my hook, I'm deploying a challenge iRule to be applied only during the challenge; it is modified each time specific to the challenge supplied from the Let's Encrypt service and is cleaned up after the challenge is tested. And finally, there are a few environment variables I set so the information is not in text files. You could also move these into a credential vault. So to recap, you first register your client, then you can kick off a challenge to generate and deploy certificates. On the client side, it looks like this: ./dehydrated --register --accept-terms ./dehydrated -c Now, for testing, make sure you use the Let's Encrypt staging service instead of production. And since I want to force action every request while testing, I run the second command a little differently: ./dehydrated -c --force --force-validation Depicted graphically, here are the moving parts for the http challenge issued by Let's Encrypt at the request of the dehydrated client, deployed to the F5 BIG-IP, and validated by the Let's Encrypt servers. The Let's Encrypt servers then generate and return certs to the dehydrated client, which then, via the hook script, deploys the certs and keys to the F5 BIG-IP to complete the process. And here's the output of the dehydrated client and hook script in action from the CLI: # ./dehydrated -c --force --force-validation # INFO: Using main config file /etc/dehydrated/config Processing example.com + Checking expire date of existing cert... + Valid till Jun 20 02:03:26 2022 GMT (Longer than 30 days). Ignoring because renew was forced! + Signing domains... + Generating private key... + Generating signing request... + Requesting new certificate order from CA... + Received 1 authorizations URLs from the CA + Handling authorization for example.com + A valid authorization has been found but will be ignored + 1 pending challenge(s) + Deploying challenge tokens... + (hook) Deploying Challenge + (hook) Challenge rule added to virtual. + Responding to challenge for example.com authorization... + Challenge is valid! + Cleaning challenge tokens... + (hook) Cleaning Challenge + (hook) Challenge rule removed from virtual. + Requesting certificate... + Checking certificate... + Done! + Creating fullchain.pem... + (hook) Deploying Certs + (hook) Existing Cert/Key updated in transaction. + Done! This results in a deployed certificate/key pair on the F5 BIG-IP, and is modified in a transaction for future updates. This proof of concept is on github in the f5devcentral org if you'd like to take a look. Before closing, however, I'd like to mention a couple things: This is an update to an existing solution from years ago. It works, but probably isn't the best way to automate today if you're just getting started and have already started pursuing a more modern approach to automation. A better path would be something like Ansible. On that note, there are several solutions you can take a look at, posted below in resources. Resources https://github.com/EquateTechnologies/dehydrated-bigip-ansible https://github.com/f5devcentral/ansible-bigip-letsencrypt-http01 https://github.com/s-archer/acme-ansible-f5 https://github.com/s-archer/terraform-modular/tree/master/lets_encrypt_module(Terraform instead of Ansible) https://community.f5.com/t5/technical-forum/let-s-encrypt-with-cloudflare-dns-and-f5-rest-api/m-p/292943(Similar solution to mine, only slightly more robust with OCSP stapling, the DNS instead of HTTP challenge, and with bash instead of python)Prevent BIG-IP Edge Client VPN Driver to roll back (or forward) during PPP/RAS errors
If you (like some of my customers) want to have the BIG-IP Edge Client packaged and distributed as a software package within your corporate infrastructure and therefore have switched off automatic component updates in your connectivity profiles, you might still get the covpn64.sys file upgraded or downgraded to the same version as the one installed on the BIG-IP APM server. Background We discovered that on some Windows clients the file covpn64.sys file got a newer/older timestamp in and started to investigate what caused this. The conclusion was that sometimes after hibernation or sleep, the Edge Client is unable to open the VPN interface and therefore tries to reinstall the driver. However, instead of using a local copy of the CAB file where the covpn64.sys file resides, it downloads it from the APM server regardless of if the version on the server and client match each other or not. In normal circumstances when you have automatic upgrades on the clients, this might not be a problem, however when you need to have full control on which version is being used on each connected client, this behavior can be a bit of a problem. Removing the Installer Component? Now you might be thinking, hey… Why don't you just remove the Component Installer module from the Edge Client and you won't have this issue. Well the simple answer to this is the fact that the Component Installer module is not only used to install/upgrade the client. In fact, it seems like it's also used when performing the Machine Check Info from the Access Policy when authenticating the user. So by removing the Component Installer module result in other issues. The Solution/workaround The Solution I came up with is to store each version of the urxvpn.cab file in an IFile and then use an iRule to deliver the correct version whenever a client tries to fetch the file for reinstallation. What's needed? In order to make this work we need to Grab a copy of urxvpn.cab from each version of the client Create an IFile for each of these versions Install iRule Attach iRule to the Virtual Server that is running the Access Policy Fetching the file from the apmclients ISOs For every version of the APM client that is available within your organization a corresponding iFile needs to be created. To create the iFiles automatically you can do the following on the APM server. Login to the CLI console with SSH Make sure you are in bash by typing bash Create temporary directories mkdir /tmp/apm-urxvpn mkdir /tmp/apm-iso Run the following (still in bash not TMSH) on the BIG-IP APM server to automatically extract the urxvpn.cab file from each installed image and save them in the folder /tmp/apm-urxvpn. for c in /shared/apm/images/apmclients-* do version="$(echo "$c" | awk -F. \ '{gsub(".*apmclients-","");printf "%04d.%04d.%04d.%04d", $1, $2, $3, $4}')" && \ (mount -o ro $c /tmp/apm-iso cp /tmp/apm-iso/sam/www/webtop/public/download/urxvpn.cab \ /tmp/apm-urxvpn/URXVPN.CAB-$version umount /tmp/apm-iso) done Check the files copied ls -al /tmp/apm-urxvpn Import each file either with tmsh or with GUI. We will cover how to import with tmsh below. If you prefer to do it with the GUI, more information abour how to do it can be found in K13423 You can use the following script to automatically import all files cd /tmp/apm-uxrvpn for f in URXVPN.CAB-* do printf "create sys file ifile $f source-path file:$(pwd)/$f\ncreate ltm ifile $f file-name $f\n" | tmsh done Save the new configuration tmsh -c “save sys config” Time to create the iRule when CLIENT_ACCEPTED { ACCESS::restrict_irule_events disable } when HTTP_REQUEST { set uri [HTTP::uri] set ua [HTTP::header "User-Agent"] if {$uri starts_with "/vdesk" || $uri starts_with "/pre"} { set version "" regexp -- {EdgeClient/(\d{4}\.\d{4}\.\d{4}\.\d{4})} $ua var version if {$version != ""} { table set -subtable vpn_client_ip_to_versions [IP::client_addr] $version 86400 86400 } else { log local0.debug "Unable to parse version from: $ua for IP: [IP::client_addr] URI: $uri" } } elseif {$uri == "/public/download/urxvpn.cab"} { set version "" regexp -- {EdgeClient/(\d{4}\.\d{4}\.\d{4}\.\d{4})} $ua var version if {$version == ""} { log local0.warning "Unable to parse version from: $ua, will search session table" set version [table lookup -subtable vpn_client_ip_to_versions [IP::client_addr]] log local0.warning "Version in table: $version" } if {$version == ""} { log local0.warning "Unable to find version session table" HTTP::respond 404 content "Missing version in request" "Content-Type" "text/plain" } else { set out "" catch { set out [ifile get "/Common/URXVPN.CAB-$version"] } if {$out == ""} { log local0.error "Didn't find urxvpn.cab file for Edge Client version: $version" HTTP::respond 404 content "Unable to find requested file for version $version\n" "Content-Type" "text/plain" } else { HTTP::respond 200 content $out "Content-Type" "application/vnd.ms-cab-compressed" } } } } Add the iRule to the APM Virtual Server Known Limitations If multiple clients with different versions of the Edge Client are behind the same IP address, they might download the wrong version. This is due to the fact that the client doesn't present the version when the request for the file urxvpn.cab reaches the iRule. This is why the iRule tries to store IP addresses based on the source IP address of other requests related to the VPN. More information about this problem can be found in K000132735Pwned Passwords Check
Problem this snippet solves: This snippet makes it possible to use Troy Hunt’s ‘Pwned Passwords’ API. By using this API one can check if the password being used was exposed in earlier data breaches. You can use this information to deny access to highly secure resources or to force a user to first change it’s password to one that isn’t known to be exposed to earlier data breaches. Or you could choose to just to inform a user that it would be wise to change it’s password. It’s good to note that the password itself will not be shared while using this API. This snippet uses a mathematical property called k-anonymity. For more information about k-anonymity and Troy Hunt’s ‘Pwned Passwords’ API see: https://www.troyhunt.com/ive-just-launched-pwned-passwords-version-2/ This snippet also uses Patt-tom McDonnell’s hibp-checker node package. How to use this snippet: Prepare the BIG-IP Provision the BIG-IP with iRuleLX. Create LX Workspace: hibp Add iRule: hibp-irule Add Extension: hibp-extension Add LX Plugin: hibp-plugin -> From Workspace: hibp Install the node.js hibp-checker module # cd /var/ilx/workspaces/Common/hibp/extensions/hibp-extension/ # npm install hibp-checker --save /var/ilx/workspaces/Common/hibp/extensions/hibp-extension └── hibp-checker@1.0.0 # irule To make it works, you need to install the irule on the Virtual Server that publish your application with APM authentication. access profile If you already have an existing access profile, you will need to modify it and include some additionnal configuration in your VPE. If you have no access profile, you can starts building your own based on the description we provide below. Configuring the Visual Policy Editor The printscreen below is an example Visual Policy Editor on how you can use the Pnwed Password snippet. VA – Force Password Change This is a Variable Assignment agent that triggers APM to show a Change Password window. Set variable: session.logon.last.change_password to Custom Expression: expr { 1 } VA – Get Password This is a Variable Assignment agent that copies the password to a session variable that can be read by the hibp irule. Set variable: session.custom.hibp.password to Custom Expression: return [mcget -secure {session.logon.last.password}] IE - HIBP This is an irule event with the ID set to ‘hibp’. This will trigger the hibp_irule to come into action. EA – HIBP Verdict This is an Empty Action with two branches. The branch named "Not Pwned" contains the following expression : expr { [mcget -nocache {session.custom.hibp.status} ] == 0 } MB – Exposed Password This is a message box that will inform the user that it’s password was exposed in earlier data breaches and a password change is needed. The message could be something like this: The password you are using was found in %{session.custom.hibp.status} data breaches. In order to be compliant with our security policy, you must change your password. hibp_irule when ACCESS_POLICY_AGENT_EVENT { if { [ACCESS::policy agent_id ] eq "hibp" } { set password [ACCESS::session data get session.custom.hibp.password] set failonerror 0 if { $password eq "" } { log local0. "Error: no password set" ACCESS::session data set session.custom.hibp.status $failonerror return } set rpc_handle [ ILX::init hibp-plugin hibp-extension ] if {[ catch { ILX::call $rpc_handle -timeout 12000 hibpCheck $password } result ] } { log local0. "hibpCheck failed. ILX failure: $result" ACCESS::session data set session.custom.hibp.status $failonerror return } ACCESS::session data set session.custom.hibp.status [expr { $result }] } } Code : var f5 = require('f5-nodejs'); const checkPassword = require('hibp-checker'); // Create a new rpc server for listening to TCL iRule calls. var ilx = new f5.ILXServer(); ilx.addMethod('hibpCheck', function(req, res) { var password = req.params()[0]; var breachCount = checkPassword(password); breachCount.then(function(result) { return res.reply(result); }, function(err) { return res.reply(err); }); }); // Start listening for ILX::call and ILX::notify events. ilx.listen(); Tested this on version: 13.0Protecting APIs with Access Policy Manager and custom iRules
The problem: Unprotected API - Vulnerable to Overload Without Rate-Limiting Enforcement Our customer in the B2B sector is encountering a challenge with their public API. Despite having implemented a custom method for generating long-lived API keys, they find themselves unable to enforce rate-limiting effectively. This absence of rate-limiting mechanisms poses significant challenges, potentially resulting in the overloading of their system due to excessive requests or the exploitation of their API by unauthorized users. Without proper rate-limiting controls in place, the customer faces risks to both the performance and security of their API infrastructure, necessitating a solution to mitigate these concerns and ensure the smooth operation of their services for their clients. Our customers wants to offer two tiers of service level agreements (SLAs) - gold and standard. Complicating matters further, the API key, integral to authentication, is transmitted via a custom HTTP header. The solution: BIG-IP APM and Custom iRules for Effective Rate-Limiting My solution involves leveraging the API Protection feature of BIG-IP APM in conjunction with a custom iRule. By utilizing this combination, our customer can effectively extract the API Keys from HTTP requests and enforce rate limiting on specific API endpoints. As for now they only want to enforce rate limiting on the POST endpoints. This approach empowers the customer to secure their API while efficiently managing and controlling access to critical endpoints, ensuring optimal performance and safeguarding against abuse or overload. With this iRule we can to extract the API key from the HTTP Requests and store it in a variable, that can later be used by the API Protection feature of the APM. API Keys and the associated SLA level are stored in a Data Groupof the typestring. # Enable (1) or disable (0) logging globally when RULE_INIT { set static::debug 1 } # Access and analyze the HTTP header data for SLA value when HTTP_REQUEST { set sla [class lookup [HTTP::header value apikey] dg_apikeys] if { $static::debug } {log local0. "Made it to HTTP_REQUEST event with SLA value $sla."} } # Evaluate SLA value during per-request access policy execution when ACCESS_PER_REQUEST_AGENT_EVENT { set id [ACCESS::perflow get perflow.irule_agent_id] if { $id eq "read-sla" } { if { $static::debug } {log local0. "Made it to iRule agent in perrequest policy with SLA value $sla."} ACCESS::perflow set perflow.custom "$sla" } } And this is how the Per Request Policy in the API Protection profile looks. It uses the value of the API Key (extracted with the help of the the iRule) and the Source IP of the client to enforce Rate Limiting on the POST endpoints, using two different SLAs. In the APM log you should see the following message, once the client exceeds his quota defined in the SLA. Apr 28 20:12:42 ltm-apm-16.mylab.local notice tmm[11094]: 01870075:5: (null):/Common: API request with weight (1) violated the quota in rate limiting config(/Common/demo_api_ratelimiting_auto_rate_limiting_standard). Apr 28 20:12:42 ltm-apm-16.mylab.local notice tmm[11094]: 0187008d:5: /Common/demo_api_ratelimiting_ap:Common:6600283561834577940: Execution of per request access policy (/Common/demo_api_ratelimiting_prp) done with ending type (Reject) Further reading: You can find a more detailed write-up on my GitHub page: https://github.com/webserverdude/f5_APM_API_Protection There you can find the Per Request Policy explained in all details. The Data Group with for the iRule. A demo API for testing my solution. A Postman Collection for working with my demo API.[Workaround]: Prevent iRule Managers to accidentally enter invalid characters into a Data Group
We realized some time ago that the BIG-IP GUI does not handle UTF-8 encoded strings very well in Data Groups which result in ever growing data records for everytime the Data Group is saved. This is an example on how to reproduce the problem and to see it in action: Start by creating an empty new data group named TEST of the type string. Add a line with the key test1 and the value abcåäö123 (copy and paste if you don't have a nordic keyboard) Click finish Check with tmsh list ltm data-group internal TEST You will see that it doesn't handle the UTF-8 data and you will se two characters for each > 7-bit character. Now go to the GUI again and add a second line with the key test2 and the same value abcåäö123 Click update Once again check tmsh and you will see that not only has test2 been added but test1 has also been updated, now including even more wrong characters. This is a screenshot of how it will look: The result of this is that eventually you will receive the following error from the BIG-IP (even if you only update lines that are not containing 8-bit ascii characters). Workaround Luckily we require the iRule Managers to login to an APM portal (webtop) where we have published a Portal Access to the administrative GUI of the BIG-IP, and since we do that, we can inject some JavaScript the will check the data before it saved. iRule PREVENT_INVALID_DATA_IN_DATAGROUP-IRULE This iRule will inject some JavaScript code that will prevent these mistakes from happening when HTTP_REQUEST { set inject_code 0 if {[HTTP::path] ends_with "/tmui/tmui/skins/Default/scripts/skin.js"} { set inject_code 1 STREAM::disable HTTP::header remove "Accept-Encoding" } } when HTTP_RESPONSE { if {$inject_code == 1} { set s1 "this.form.submitError\ =\ formSubmitError;" set r1 "this.form.submitError\ =\ formSubmitError;\n" append r1 "if\ (document.location.href.indexOf('datagroup')\ >\ -1)\ {\n" append r1 "select\ =\ document.getElementById('class_string_item');\n" append r1 "if\ (select\ !=\ null)\ {\n" append r1 "console.log('We\ found\ the\ select');\n" append r1 "for\ (option\ of\ select.children)\ {\n" append r1 "key\ =\ option.value.substr(0,\ option.value.indexOf('\\\\x0a'));\n" append r1 "if\ (!\ /^\[\\x00-\\x7F\]*\$/.test(key))\ {\n" append r1 "alert('String\ \"'+key+'\"\ contains\ invalid\ characters!');\n" append r1 "return\ false;\n" append r1 "}\n" append r1 "if\ (!\ /^\[\\x00-\\x7F\]*\$/.test(option.value))\ {\n" append r1 "alert('Value\ of\ \"'+key+'\"\ contains\ invalid\ characters!');\n" append r1 "return\ false;\n" append r1 "}\n" append r1 "}\n" append r1 "}\n" append r1 "}\n" STREAM::expression "@$s1@$r1@" STREAM::enable } } Now if you add a streaming profile to your APM Virtual Server and add this iRule the user will be presented with a popup informing them about the invalid data and also prevent them from submitting it. Hope this can be of help until F5 fixes this in the BIG-IP
Redirect iRule differences...
We have a situation where we need to redirect users from one domain to another and had been using Method 1 ( shown below ) of redirection via iRule. It was recently brought to our attention by our web team that the way we were doing the redirects for one of their sites in particular was "really bad" for SEO and we ended up making them happy with doing it via Method 2 ( shown below ). While my team officially maintains our BIG-IP's, we are not network/web savvy and don't really understand the difference here. We have a new site that needs to be redirected and we are not sure which method to use. Would someone please explain in what cases you would use one over the other. Thanks. Method 1 when HTTP_REQUEST { if { [HTTP::host] eq "website1.com" } { HTTP::redirect https://websitesite2.com } } Method 2 when HTTP_REQUEST { if { ([string tolower [HTTP::host]] eq "website1.com")} { HTTP::respond 301 Location "http://website2.com" return } }Json parsing with iRules
JSON is now the format of choice for most APIs. It's time we were able to parse JSON with F5 iRules too, as simple string matching is not always good enough. That's why I wrote a simple JSON parser for iRules. It is a validating single pass parser that processes the JSON string char by char until the JsonPath expression matches, no recursion or any other fancy stuff. As I do not wanted to reinvent the wheel, it is basically a rewrite of the JSON parser found in themongoose webserver project in plain TCL. The usage is very simple: set token [call json::json_get_tok { $json $path }] $json is the json string to parse $path is a JsonPath expression, following operators are implemented: Operator Description $ The root element to query. This starts all path expressions. .<name> Dot-notated child. [<number>] Array index. Example Simple JSON: { "aud": "audience \"test\"", "iss": "https://issuer.de/issuer/", "iat": 1701422123, "roles": [ "role1", "role2" ], "obj": { "sub": "adcad2b8", }, "ver": "2.0" } JsonPath expression to parse this simple JSON: JsonPath Return value $.aud "audience \"test\"" $.iat 1701422123 $.obj.sub "adcad2b8" $.roles[0] "role1" To decode the extracted JSON string: set decoded [call json::json_decode_str { $token }] This removes the enclosing quotes from a string and decodes JSON escapes. Code You can find the code and further documentation in my GitHub repository: https://github.com/JuergenMang/f5-irules-json
Clone F5 traffic and forwarded into different Pool
Dear All, I have scenario, to forwarded Splunk F5 VIP traffic to more then one backend pools. upon checking we find Clone Pool (Client) or Clone Pool (Server ) Option but its limited with TCP/HTTP VIP not with stateless VIP. https://my.f5.com/manage/s/article/K8573 Can you please advise way to achieve this scenario through Irule or LTM policy Thanks
Outbound iRule / BGP routing
Hey sirs, I would like to ask a question about the order of precedence/execute of a connection that consumes a forwarding virtual server/routing table. Currently, we have a forwarding any:0 virtual server, which load balances internet outgoing traffic through a pool_default_gateway that has the IP of 3 routers from different ISP associated with it, including some irules that make the SNAT decision based on LAN-segment. We are planning to include the F5 pair in the BGP neighbors of each ASN ISP and receive the default route and advertise the Virtual Server public IP. Does anyone know if the F5 when reads the dynamic routing table obtained via BGP, the traffic that is handled by the virtual servers of forwarding any:0, including those that are manipulated via iRule can show any kind of intermittence? thanks in advance
