Using Distributed Application Security Policies in Secure Multicloud Networking Customer Edge Sites
This tutorial and workflow guide deploys and uses F5 Distributed Cloud App Security policies with apps at local customer edge sites. Deploy a policy in any customer edge site regardless of location in the cloud or on-prem. Manual and automation workflows show how to make this NetOps and DevOps friendly solution come to life.
Failure creating certificate acme challenge 404 error in BIG-IP F5 WAF
We have more than 600 government websites behind the BIG-IP system. We have done almost 60% of certificates created and offloaded.Suddenly we couldn't create any certificate and got the below error. This error not only for one website. Now we can't renew or create a new certificate. We use fanceg/letsencrypt -in GitHub to integrates Let's Encrypt with BigIP (GitHub - fanceg/letsencrypt-bigip). INFO: Using main config file /etc/dehydrated/configProcessing verugal.ds.gov.lk Signing domains... Generating private key... Generating signing request... Requesting new certificate order from CA... Received 1 authorizations URLs from the CA Handling authorization for verugal.ds.gov.lk 1 pending challenge(s) Deploying challenge tokens... Responding to challenge for verugal.ds.gov.lk authorization... Cleaning challenge tokens... Challenge validation has failed : ( ERROR: Challenge is invalid! (returned: invalid) (result: ["type"] "http-01" ["status"] "invalid" ["error","type"] "urn:ietf:params:acme:error:unauthorized" ["error","detail"] "Invalid response fromhttp://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE1[43.224.124.166]: 404" ["error","status"] 403 ["error"] {"type":"urn:ietf:params:acme:error:unauthorized","detail":"Invalid response fromhttp://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE1[43.224.124.166]: 404","status":403} ["url"] "https://acme-v02.api.letsencrypt.org/acme/chall-v3/12995442889/eoq1dQ" ["token"] "CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE" ["validationRecord",0,"url"] "http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE1" ["validationRecord",0,"hostname"] "verugal.ds.gov.lk" ["validationRecord",0,"port"] "80" ["validationRecord",0,"addressesResolved",0] "43.224.124.166" ["validationRecord",0,"addressesResolved"] ["43.224.124.166"] ["validationRecord",0,"addressUsed"] "43.224.124.166" ["validationRecord",0] {"url":"http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE","hostname":"verugal.ds.gov.lk","port":"80","addressesResolved":["43.224.124.166"],"addressUsed":"43.224.124.166"} ["validationRecord"] [{"url":"http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE","hostname":"verugal.ds.gov.lk","port":"80","addressesResolved":["43.224.124.166"],"addressUsed":"43.224.124.166"}] ["validated"] "2021-05-10T04:46:36Z") Processing vrc.bopepoddala.ds.gov.lk Signing domains... Generating private key... Generating signing request... Requesting new certificate order from CA... Received 1 authorizations URLs from the CA Handling authorization for vrc.bopepoddala.ds.gov.lk 1 pending challenge(s) Deploying challenge tokens... Responding to challenge for vrc.bopepoddala.ds.gov.lk authorization... Cleaning challenge tokens... Challenge validation has failed : ( ERROR: Challenge is invalid! (returned: invalid) (result: ["type"] "http-01" ["status"] "invalid" ["error","type"] "urn:ietf:params:acme:error:unauthorized" ["error","detail"] "Invalid response fromhttp://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y[43.224.124.166]: 404" ["error","status"] 403 ["error"] {"type":"urn:ietf:params:acme:error:unauthorized","detail":"Invalid response fromhttp://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y[43.224.124.166]: 404","status":403} ["url"] "https://acme-v02.api.letsencrypt.org/acme/chall-v3/12995448812/pq_1KA" ["token"] "v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y" ["validationRecord",0,"url"] "http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y" ["validationRecord",0,"hostname"] "vrc.bopepoddala.ds.gov.lk" ["validationRecord",0,"port"] "80" ["validationRecord",0,"addressesResolved",0] "43.224.124.166" ["validationRecord",0,"addressesResolved"] ["43.224.124.166"] ["validationRecord",0,"addressUsed"] "43.224.124.166" ["validationRecord",0] {"url":"http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y","hostname":"vrc.bopepoddala.ds.gov.lk","port":"80","addressesResolved":["43.224.124.166"],"addressUsed":"43.224.124.166"} ["validationRecord"] [{"url":"http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y","hostname":"vrc.bopepoddala.ds.gov.lk","port":"80","addressesResolved":["43.224.124.166"],"addressUsed":"43.224.124.166"}] ["validated"] "2021-05-10T04:46:58Z") Can anyone help me out with this issue? Are there any process changes or updates in letsencrypt site or BIG-IP intigrations? Due to this lots of government websites affected!Create a BIG-IP HA Pair in Azure
Use an Azure ARM template to create a high availability (active-standby) pair of BIG-IP Virtual Edition instances in Microsoft Azure. When one BIG-IP VE goes standby, the other becomes active, the virtual server address is reassigned from one external NIC to another. Today, let’s walk through how to create a high availability pair of BIG-IP VE instances in Microsoft Azure. When we’re done, we’ll have an active-standby pair of BIG-IP VEs. To start, go to the F5 Networks Github repository. Click F5-azure-arm-templates. Then go to Supported>failover. You have several options at this point. You can chose which templates to use based on your needs, failing over via API calls, via upstream load balancers, and NIC counts. Read each readme to determine your desired deployment strategy. When you already have your subnets and existing IP addresses defined but to see how it works, let’s deploy a new stack. Click new stack and scroll down to the Deploy button. If you have a trial or production license from F5, you can use the BYOL or BIG-IQ as license server options but in this case we’re going to choose the PAYG option. Click Deploy and the template opens in the Azure portal. Now we simply fill out the fields. We’ll create a new Resource Group and set a password for the BIG-IP VEs. When you get to the questions: The DNS label is used as part of the URL. Instance Name is just the name of the VM in Azure. Instance Type determines how much memory and CPU you’ll have. Image Name determines how many BIG-IP modules you can run (and you can choose the latest BIG-IP version). Licensed Bandwidth determines the maximum throughput of the traffic going through BIG-IP. Select the Number of External IP addresses (we’ll start with one but can add more later). For instance, if you plan on running more than one application behind the BIG-IP, then you’ll need the appropriate external IP addresses. Vnet Address Prefix is for the address ranges of you subnets (we’ll leave at default). The next 3 fields (Tenant ID, Client ID, Service Principal Secret) have to do with security. Rather than using your own credentials to modify resources in Azure, you can create an Active Directory application and assign permissions to it. The last two fields also go together. Managed Routes let you route traffic from other external networks through the BIG-IPs. The Route Table Tag means that anytime this tag is found in the route table, routes that have this destination are updated so that the next hop is the IP address of the active BIG-IP VE. This is useful if you want all outbound traffic to go through the BIG-IP or if you want to send traffic from a bunch of different Vnets through the BIG-IP. We’ll leave the rest as default but the Restricted Src Address is good way to put IP addresses on my network – the ones that are allowed to connect to the BIG-IP. We’ll agree to the terms and click Purchase. We’re redirected to the Dashboard with the Deployment in Progress indicator. This takes about 15 minutes. Once finished we’ll go check all the resources in the Resource Group. Let’s find out where the virtual server address is located since this is associated with one of the external NICs, which have ‘ext’ in the name. Click the one you want. Then click IP Configuration under Settings. When you look at the IP Configuration for these NICs, whenever the NIC has two IP addresses that’s the NIC for the active BIG-IP. The Primary IP address is the BIG-IP Self IP and the Secondary IP is the virtual server address. If we look at the other external NIC we’ll see that it only has one Self IP and that’s the Primary and it doesn’t have the Secondary virtual server address. The virtual server address is assigned to the active BIG-IP. When we force the active BIG-IP to standby, the virtual server address is reassigned from one NIC to the other. To see this, we’ll log into the BIG-IPs and on the active BIG-IP, we’ll click Force to Standby and the other BIG-IP becomes Active. When we go back to Azure, we can see that the virtual server IP is no longer associated with the external NIC. And if we wait a few minutes, we’ll see that the address is now associated with the other NIC. So basically how BIG-IP HA works in the Azure cloud is by reassigning the virtual server address from one BIG-IP to another. Thanks to our TechPubs group and check out the demo video. psLightboard Lessons: Explaining the GitHub DDoS Attack
On Feb 28, 2018 the popular website GitHub was the victim of the largest Distributed Denial of Service (DDoS) attack in recorded history. The attackers used open memcached servers to launch an amplification attack that pushedtraffic at the rate of 1.35 Tbps. Fortunately, GitHub was prepared with a DDoS scrubbing service and was able to mitigate the attack, and the attackers stopped after about 20 minutes. The unfortunate part, though, is that many thousands of open memcached servers are still out there on the Internet and could be used for a similar attack at any time. Watch the video below to learn about the details of the attack and the mitigation steps. Related Resources: F5 Labs Analysis of the GitHub DDoS Attack F5 Silverline DDoS Protection ServicesAutomatically Update your BIG-IP Pool Using the Service Discovery iApp
Let’s look at how to automatically add members to your BIG-IP pool by using the Service Discovery iApp. Whenever you deploy a BIG-IP Virtual Edition by using one of the templates on the F5 Github site, this iApp is installed on the BIG-IP. The idea behind this iApp is you assign a tag to a virtual machine in the cloud and then BIG-IP automatically discovers it and adds it to the pool. By tagging instances in AWS and Azure, and configuring the iApp, the pool is updated based on an interval you specify. This is especially helpful if you auto-scale your application servers because they are then automatically added and removed. Today, we’ll look how to do this in Azure but you can also do this in AWS. First, we’re going to add a tag to the application sever in Azure. You can assign the tag to either the virtual machine or to the NIC. For auto-scaling you’d tag the scale set. For this we’ll simply add it to the virtual machine. When you click through the virtual machine, on the left you’ll get the ‘Tags’ option. This entry can be any name/value pair you want and for this we’ll use ‘mytag’ and ‘addme.’ And we’ll click Save. For this exercise, we have two application servers in the resource group and already added the tags for that one. So at this point, we’re ready to get into the BIG-IP and configure the iApp. Once in, go to Application Services>Applications>Create. Next, we give it a name and choose f5_service_discovery from the list. Scroll down the same page and fill out the open fields. Under Cloud Provider, we select Azure. Depending on your provider, there are additional questions. Add the Azure resource group and the Subscription ID. The next 3 fields (for the Azure selection) are security related: Tenant ID, Client ID and Service Principal Secret. Rather than using your own credentials to create and modify resources in Azure, you can create an Azure Active Directory application and assign permissions to that. Details are included on the Github ReadMe or the Azure documentation about service Principal. Under the Pool area, is where you enter the name/value pair that we used for the tags in Azure. We leave the rest default. In this instance, you may notice the update interval at 60 seconds. By default, 60 seconds is the interval that BIG-IP will query Azure to see if there is a resource with the tags you specified. Under Application Health, select ‘http’ as the health monitor. Click Finished. When complete, we can see we got a pool with two active members in it. If you take the tags off one of the instances, it’ll leave the pool. Of note however, there must be two members in the pool before you remove tags from an instance. If you remove the tags from all the application servers, the pool will not be updated. BIG-IP must see at least one set of tags to update the pool because it doesn’t want to leave you with an empty pool. Here’s the before and after of removing a tag. One final note. This example configuration has the BIG-IP in one resource group and the application servers in another resource group but they are all on the same Vnet. If you have separate networks in Azure, you’ll need to create a peering so they can communicate. Similarly, in AWS, you need to make sure the networking is set up so the BIG-IP can see the application servers. But, once the initial set up is working, there’s no manual intervention required. You can use the Service Discovery method to add and remove application servers all day long without having to manually update the BIG-IP. Again, and as always, thanks to our Technical Communications team for the great material and watch the video demo here. ps Related: Onboarding F5 in Cloud Part 2 - Service Discovery Deploy the Service Discovery iApp on a BigIP Cluster across two Availability ZonesDeploy BIG-IP VE in Microsoft Azure Using an ARM Template
Azure Resource Manager (ARM) templates allow you to repeatedly deploy applications with confidence. The resources are deployed in a consistent state and you can easily manage and visualize resources for your application. ARM templates take the guesswork out of creating repeatable applications and environments. Deploy and deploy again, consistently. Let’s walk through how to deploy a simple, single-NIC configuration of BIG-IP VE in Microsoft Azure using an ARM template. First, go to the F5 Networks Github site where we keep our supported templates. There are other community-based templates at www.github.com/f5devcentral if needed but for F5 supported templates, go to the F5 Networks site. To view Azure templates, click f5-azure-arm-templates. In that folder you’ll see experimental and right under that is supported (the one you want). Then click on the standalone folder and then the 1nic folder, which is the simplest deployment. And as you scroll through and review the ‘Read Me,’ you’ll see the Deploy to Azure button under Installation. Select either Bring Your Own License (BYOL) or Pay As You Go (PAYG), depending on your situation. This will launch the Azure Portal and the only thing you’ll really need is a license key if you chose BYOL. Then simply fill out the template. In this case, we’re going to use an existing resource group that already contains an application. Important: In the Settings section under Admin UN/PW, enter the credentials you want to use to log in to BIG-IP VE. The DNS Label (where you see REQUIRED) will be used to access your BIG-IP VE, for example, if you enter mybigip, the address will be something like ‘mybigip.westus.cloudapp.azure.com.’ Give the Instance Name something familiar for easy finding. There are different Azure Instance Types, which determine CPU and memory for your VM, and F5 licensing (Good/Better/Best), which determines the BIG-IP modules you can deploy. Then, if needed, enter your BYOL license key. In addition, to be more secure, you should enter a range of IP addresses on your network in the Restricted Src Addresses field so it’s locked to your address range. This setting determines who gets access to the BIG-IP instance in Azure, so you’ll want to lock it down. After the tag values, agree to the terms and conditions and click Purchase. Next, you can monitor progress on the deploy status. Keep hitting refresh and you’ll start seeing resources getting populated along with the top blue ‘Deploying’ indicator. When the Deploying bar disappears, you know you’re done. Once complete, you get the notification that the BIG-IP VE was deployed successfully. Next, we’ll navigate to the resource group we selected at the top and then the security group for the BIG-IP. You can see that within the security rules we’ve allowed ports 443 (HTTPS) and 22 (SSH). 22 allows access to the management port; this is the way we’d connect to the BIG-IP to configure and administer. Going back to the resources, the BIG-IP itself is listed at the top. When we click on the Virtual BIG-IP we can get the IP address and using a browser through port 443, we can connect either with the DNS name or the IP address to the config utility. Here you would enter the Azure credentials you specified in the template. And that’s all there is to it. Now you can configure your virtual servers, pool, profiles and anything you’d normally do on BIG-IP VE for your unique requirements. Thanks to Suzanne Selhorn for the basis of this article and catch a video demo here. ps Related: Azure on DevCentral Azure Setup Guide Enabling Azure Active Directory Tenant Restrictions with F5 Securing your applications with F5 Virtual Editions in Microsoft Azure
