F5 Distributed Cloud - Customer Edge Site - Deployment & Routing Options
F5 Distributed Cloud Customer Edge (CE) software deployment models for scale and routing for enterprises deploying multi-cloud infrastructure. Today's service delivery environments are comprised of multiple clouds in a hybrid cloud environment. How your multi-cloud solution attaches to your existing on-prem and cloud networks can be the difference between a successful overlay fabric, and one that leave you wanting more out of your solution. Learn your options with F5 Distributed Cloud Customer Edge software.F5 Distributed Cloud - Regional Decryption with Virtual Sites
In this article we discuss how the F5 Distributed Cloud can be configured to support regulatory demands for TLS termination of traffic to specific regions around the world. The article provides insight into the F5 Distributed Cloud global backbone and application delivery network (ADN). The article goes on to inspect how the F5 Distriubted Cloud is able to achieve these custom topologies in a multi-tenant architecture while adhearing to the "rules of the internet" for route summarization. Read on to learn about the flexibility of F5's SaaS platform providing application delivery and security solutions for your applications.F5 Distributed Cloud - Listener Logic
In a proxy, there is a client-side and server-side connection. In this article, we'll focus on how the proxy "picks-up" or "listens" for traffic on the client-side. There are many options and creative ideas that adapt to enterprises business needs. First, we need to know the mechanics and what is possible, and this article covers those basics.How to get a F5 BIG-IP VE Developer Lab License
(applies to BIG-IP TMOS Edition) To assist operational teams teams improve their development for the BIG-IP platform, F5 offers a low cost developer lab license. This license can be purchased from your authorized F5 vendor. If you do not have an F5 vendor, and you are in either Canada or the US you can purchase a lab license online: CDW BIG-IP Virtual Edition Lab License CDW Canada BIG-IP Virtual Edition Lab License Once completed, the order is sent to F5 for fulfillment and your license will be delivered shortly after via e-mail. F5 is investigating ways to improve this process. To download the BIG-IP Virtual Edition, log into my.f5.com (separate login from DevCentral), navigate down to the Downloads card under the Support Resources section of the page. Select BIG-IP from the product group family and then the current version of BIG-IP. You will be presented with a list of options, at the bottom, select the Virtual-Edition option that has the following descriptions: For VMware Fusion or Workstation or ESX/i: Image fileset for VMware ESX/i Server For Microsoft HyperV: Image fileset for Microsoft Hyper-V KVM RHEL/CentoOS: Image file set for KVM Red Hat Enterprise Linux/CentOS Note: There are also 1 Slot versions of the above images where a 2nd boot partition is not needed for in-place upgrades. These images include _1SLOT- to the image name instead of ALL. The below guides will help get you started with F5 BIG-IP Virtual Edition to develop for VMWare Fusion, AWS, Azure, VMware, or Microsoft Hyper-V. These guides follow standard practices for installing in production environments and performance recommendations change based on lower use/non-critical needs for development or lab environments. Similar to driving a tank, use your best judgement. Deploying F5 BIG-IP Virtual Edition on VMware Fusion Deploying F5 BIG-IP in Microsoft Azure for Developers Deploying F5 BIG-IP in AWS for Developers Deploying F5 BIG-IP in Windows Server Hyper-V for Developers Deploying F5 BIG-IP in VMware vCloud Director and ESX for Developers Note: F5 Support maintains authoritative Azure, AWS, Hyper-V, and ESX/vCloud installation documentation. VMware Fusion is not an official F5-supported hypervisor so DevCentral publishes the Fusion guide with the help of our Field Systems Engineering teams.F5 Distributed Cloud - Service Policy - Header Matching Logic & Processing
Learn about the F5 Distributed Cloud service policy feature and how to apply logic to your match criteria (and/or). This understanding of the logic structures within service policies unlocks endless combinations of application security services.
AS3 Best Practice
Introduction AS3 is a declarative API that uses JSON key-value pairs to describe a BIG-IP configuration. From virtual IP to virtual server, to the members, pools, and nodes required, AS3 provides a simple, readable format in which to describe a configuration. Once you've got the configuration, all that's needed is to POST it to the BIG-IP, where the AS3 extension will happily accept it and execute the commands necessary to turn it into a fully functional, deployed BIG-IP configuration. If you are new to AS3, start reading the following references: Products - Automation and orchestration toolchain (f5.com; Product information) Application Services 3 Extension Documentation (clouddocs; API documentation and guides) F5 Application Services 3 Extension (AS3) (GitHub; Source repository) This article describes some considerations in order to efficiently deploy the AS3 configurations. Architecture In the TMOS space, the services that AS3 provides are processed by a daemon named 'restnoded'. It relies on the existing BIG-IP framework for deploying declarations. The framework consists of httpd, restjavad and icrd_child as depicted below (the numbers in parenthesis are listening TCP port numbers). These processes are also used by other services. For example, restjavad is a gateway for all the iControl REST requests, and is used by a number of services on BIG-IP and BIG-IQ. When an interaction between any of the processes fails, AS3 operation fails. The failures stem from lack of resources, timeouts, data exceeding predefined thresholds, resource contention among the services, and more. In order to complete AS3 operations successfully, it is advised to follow the Best Practice outlined below. Best Practice Your single source of truth is your declaration Refrain from overwriting the AS3-deployed BIG-IP configurations by the other means such as TMSH, GUI or iControl REST calls. Since you started to use the AS3 declarative model, the source of truth for your device's configurations is in your declaration, not the BIG-IP configuration files. Although AS3 tries to weigh BIG-IP locally stored configurations as much as it can do, discrepancy between the declaration and the current configuration on BIG-IP may cause the AS3 to perform less efficiently or error unexpectedly. When you wish to change a section of a tenant (e.g., pool name change), modify the declaration and submit it. Keep the number of applications in one tenant to a minimum AS3 processes each tenant separately. Having too many applications (virtual servers) in a single tenant (partition) results in a lengthy poll when determining the current configuration. In extreme cases (thousands of virtuals), the action may time out. When you want to deploy a thousand or more applications on a single device, consider chunking the work for AS3 by spreading the applications across multiple tenants (say, 100 applications per tenant). AS3 tenant access behavior behaves as BIG-IP partition behavior. A non-Common partition virtual cannot gain access to another partition's pool, and in the same way, an AS3 application does not have access to a pool or profile in another tenant. In order to share configuration across tenants, AS3 allows configuration of the "Shared" application within the "Common" tenant. AS3 avoids race conditions while configuring /Common/Shared by processing additions first and deletions last, as shown below. This dual process may cause some additional delay in declaration handling. Overwrite rather than patching (POSTing is a more efficient practice than PATCHing) AS3 is a stateless machine and is idempotent. It polls BIG-IP for its full configuration, performs a current-vs-desired state comparison, and generates an optimal set of REST calls to fill the differences. When the initial state of BIG-IP is blank, the poll time is negligible. This is why initial configuration with AS3 is often quicker than subsequent changes, especially when the tenant contains a large number of applications. AS3 provides the means to partially modify using PATCH (see AS3 API Methods Details), but do not expect PATCH changes to be performant. AS3 processes each PATCH by (1) performing a GET to obtain the last declaration, (2) patching that declaration, and (3) POSTing the entire declaration to itself. A PATCH of one pool member is therefore slower than a POST of your entire tenant configuration. If you decide to use PATCH, make sure that the tenant configuration is a manageable size. Note: Using PATCH to make a surgical change is convenient, but using PATCH over POST breaks the declarative model. Your declaration should be your single source of truth. If you include PATCH, the source of truth becomes "POST this file, then apply one or more PATCH declarations." Get the latest version AS3 is evolving rapidly with new features that customers have been wishing for along with fixes for known issues. Visit the AS3 section of the F5 Networks Github. Issues section shows what features and fixes have been incorporated. For BIG-IQ, check K54909607: BIG-IQ Centralized Management compatibility with F5 Application Services 3 Extension and F5 Declarative Onboarding for compatibilities with BIG-IQ versions before installation. Use administrator Use a user with the administrator role when you submit your declaration to a target BIG-IP device. Your may find your role insufficient to manipulate BIG-IP objects that are included in your declaration. Even one authorized item will cause the entire operation to fail and role back. See the following articles for more on BIG-IP user and role. Manual Chapter : User Roles (12.x) Manual Chapter : User Roles (13.x) Manual Chapter : User Roles (14.x) Prerequisites and Requirements (clouddocs AS3 document) Use Basic Authentication for a large declaration You can choose either Basic Authentication (HTTP Authorization header) or Token-Based Authentication (F5 proprietary X-F5-Auth-Token) for accessing BIG-IP. While the Basic Authentication can be used any time, a token obtained for the Token-Based Authentication expires after 1,200 seconds (20 minutes). While AS3 does re-request a new token upon expiry, it requires time to perform the operation, which may cause AS3 to slow down. Also, the number of tokens for a user is limited to 100 (since 13.1), hence if you happen to have other iControl REST players (such as BIG-IQ or your custom iControl REST scripts) using the Token-Based Authentication for the same user, AS3 may not be able to obtain the next token, and your request will fail. See the following articles for more on the Token-Based Authentication. Demystifying iControl REST Part 6: Token-Based Authentication (DevCentral article). iControl REST Authentication Token Management (DevCentral article) Authentication and Authorization (clouddocs AS3 document) Choose the best window for deployment AS3 (restnoded daemon) is a Control Plane process. It competes against other Control Plane processes such as monpd and iRules LX (node.js) for CPU/memory resources. AS3 uses the iControl REST framework for manipulating the BIG-IP resources. This implies that its operation is impacted by any processes that use httpd (e.g., GUI), restjavad, icrd_child and mcpd. If you have resource-hungry processes that run periodically (e.g., avrd), you may want to run your AS3 declaration during some other time window. See the following K articles for a list of processes K89999342 BIG-IP Daemons (12.x) K05645522 BIG-IP Daemons (v13.x) K67197865 BIG-IP Daemons (v14.x) K14020: BIG-IP ASM daemons (11.x - 15.x) K14462: Overview of BIG-IP AAM daemons (11.x - 15.x) Workarounds If you experience issues such as timeout on restjavad, it is possible that your AS3 operation had resource issues. After reviewing the Best Practice above but still unable to alleviate the problem, you may be able to temporarily fix it by applying the following tactics. Increase the restjavad memory allocation The memory size of restjavad can be increased by the following tmsh sys db commands tmsh modify sys db provision.extramb value <value> tmsh modify sys db restjavad.useextramb value true The provision.extramb db key changes the maximum Java heap memory to (192 + <value> * 8 / 10) MB. The default value is 0. After changing the memory size, you need to restart restjavad. tmsh restart sys service restjavad See the following article for more on the memory allocation: K26427018: Overview of Management provisioning Increase a number of icrd_child processes restjavad spawns a number of icrd_child processes depending on the load. The maximum number of icrd_child processes can be configured from /etc/icrd.conf. Please consult F5 Support for details. See the following article for more on the icrd_child process verbosity: K96840770: Configuring the log verbosity for iControl REST API related to icrd_child Decrease the verbosity levels of restjavad and icrd_child Writing log messages to the file system is not exactly free of charge. Writing unnecessarily large amount of messages to files would increase the I/O wait, hence results in slowness of processes. If you have changed the verbosity levels of restjavad and/or icrd_child, consider rolling back the default levels. See the following article for methods to change verbosity level: K15436: Configuring the verbosity for restjavad logs on the BIG-IP systemIntroducing the F5 Application Study Tool (AST)
In the ever-evolving world of application delivery and security, gaining actionable insights into your infrastructure and applications has become more critical than ever. The Application Study Tool (AST) is designed to help technical teams and administrators leverage the power of open-source telemetry and visualization tools to enhance their monitoring, diagnostics, and analysis workflows.BIG-IP Scale-out/Multi-tier構成のポイント解説
はじめに BIG-IPは、1台で負荷分散、SSLアクセラレータ、フォワードプロキシ、ネットワーク保護、ウェブアプリケーション保護、リモートアクセス等の複数機能の提供が可能で、システムを集約させることが可能ですが、以下のようなケースで複数のBIG-IPに処理を分散させることも可能です。 - 高スペックな1台のBIG-IPで処理しきれない非常に多量のトラフィックを複数のBIG-IPに分散 - スモールスタートし、トラフィックや利用ユーザー数の増加に合わせて、BIG-IPを追加 しかしながら、複数BIG-IPで処理を行う場合は、以下のことも考慮の必要があります。 - 複数BIG-IPの負荷分散の考慮 - BIG-IPでSNATの構成を利用しない場合、構成の検討が必要 これらの検討事項は、BIG-IPをMulti-tier構成にすることで、実現が可能です。 本ブログでは、Scale-out/Multi-tier構成にする場合のネットワーク構成、HA構成、ライセンスによる注意事項を解説します。 1. ネットワーク構成につきまして Tier1 - Tier2とTier2 - Tier3間のVLANは別にすることをお勧めします。 また、戻りのTrafficのためにTier2とTier3のBIG-IPでは、Auto Last Hopは有効にします。 2. HA構成 Tier1/Tier3に相当する機器は、耐障害性を高めるためにActive-Standbyをお勧めします。 Tier2に相当する機器は、Standaloneを並べることも可能ですが、Active-Activeの構成でもメリットがあると考えられます。 Active-Activeにするメリットとしては以下が考えられます。 - ConfigをSyncさせることによって複数台の設定が容易になること - Active-Activeにすることにより、障害時は、すぐに別のリソースに切り替わること (Tier1でHealth Monitorと併用すれば障害時の切り替えの時に、よりメリットがあると考えられます) Active-Activeにした場合には、Traffic GroupのFailover Methodに以下の設定も追加して、特定のBIG-IPにActiveのリソースが偏らないようにします。 - Failover using Preferred Device Order and then Load Aware - Always Failback to First Device if it is Available 3. ライセンスによる注意事項 APMの場合、Active-Activeはサポートしておりませんので、Tier2をAPMにする場合は、Standaloneの機器を並べることになります。 4．まとめ 本ブログでは、Scale-out/Multi-tier構成を実施する場合の、考慮点等を解説しました。 Multi-tier構成については、Scale-out構成との組み合わせだけではなく、それぞれのtierで別のモジュールを動作させること等柔軟な構成が考えられます。 負荷分散、SSLアクセラレータ、フォワードプロキシ、ネットワーク保護、ウェブアプリケーション保護、リモートアクセス等様々なシーンで適用できますのでBIG-IPをぜひご活用ください。How to Split DNS with Managed Namespace on F5 Distributed Cloud (XC) Part 1 – DNS over HTTPS
Introduction DNS, everyone’s least favorite infrastructure service. So simple, yet so hard. Simple because it’s really just some text files served up, so hard because get it wrong and everything breaks. And it really doesn’t require a ton of resources, so why use a lot? Containers, rulers of our age, everything must be a container! Not really, but we are in a major shift from waterfall to modern architecture, and its handy to have something small that can be spun up in a lot of locations for redundancy and automated for our needs. "Nature is a mutable cloud, which is always and never the same." - Ralph Waldo Emerson We might not wax that philosophically around here, but our heads are in the cloud nonetheless! Join the F5 Distributed Cloud user group today and learn more with your peers and other F5 experts. But also, if I don’t want to spin up servers or hardware, I probably don’t want to spin up a container infrastructure either. So, use F5 XC Managed K8s solutions… F5 XC is a platform that can not only be used to provide native security solutions in any cloud or datacenter but is also a compute platform like any other cloud service provider. Bring us all your containers to host and secure. For this use-case we are going to use our Managed Namespace solution. It’s very similar to our Managed K8s solution, but more of a sandbox with hardened security policies. Part 1 will focus on DNS over HTTPS, Part 2 will cover TCP/UDP, however, the initial deployment will set up all the ports and services needed for Part 2 now. Managed Namespace - Sandbox Policies Architecture For this solution I went with Bind 9.19-dev, which seemed to have some issues with grpc and HTTP/2 conversion, which I was able to resolve by slapping NGINX in front of the DNS over HTTPS listener to proxy grpc to http/2, all other TCP/UDP traffic goes directly to Bind. Hopefully this is patched in future Bind releases. Otherwise, it’s just a standard tcp/udp DNS deployment. An important note about the architecture is that the workloads can be deployed on Customer Edge Nodes or on F5 owned Regions, so if there is no desire to manage a node on-premises, or manage / host k8s whether on-prem or in a 3 rd party Cloud Service Provider, running on our regions works perfectly fine and give a tremendous amount of redundancy. Managed Namespace Deployment From within the XC Console, we need to ensure that there is a Managed Namespace deployed, so click on the Distributed Apps Tile. Under Applications, select Virtual K8s. From here, if there is not already a vk8s deployed in the namespace, deploy one now. We won’t be covering deploying virtual k8s here, but it’s not too complex, click add new, give it a name, select some virtual sites, leave service isolation disabled and choose a default workload flavor. Once the Managed Namespace (virtual k8s) is online, you can download the kubeconfig by clicking the ellipses on the far right and selecting Kubeconfig. For a more detailed walkthrough of Creating a Managed Namespace you can go to the F5 Tech Docs located here: https://docs.cloud.f5.com/docs/how-to/app-management/create-vk8s-obj Click-Ops Deployment Since it would take up a ton of space I will not cover Click-Ops deployment of workloads, while it may be in a future article a detailed walkthrough can be found here today: https://docs.cloud.f5.com/docs/how-to/app-management/vk8s-workload Kubectl Deployment We WILL be covering deployment via kubectl with a manifest in this guide, so now we can actually start getting into it. As detailed in the architecture we are going to proxy requests to bind via NGINX, and to get NGINX set up as a proxy to Bind we need to get it configured. Posting the YAML in the article was a bit long, so all sources are posted in github and the yaml images link to the specific sections, while a full manifest is located near the end of the article. NGINX Config-Map There are a couple of critical or key differences to pay attention to when deploying to Managed Namespaces versus another k8s provider. The main one that we care about now is annotations. In the context we will be using them, they determine where the configurations and workloads will be deployed, and in other scenarios also include things like workload flavors and other internal details. ves.io/sites: determines the sites we are going to want the objects deployed to. This can be a Customer Site, a Virtual Site, or to all F5 XC Owned Regions. In our nginx.conf, All of these configs are standard as well with a location added for a health check and some self signed certs to force a secure channel. If you need a quick command to generate Cert & Key without searching: openssl req -x509 -nodes -subj '/CN=bind9.local' -newkey rsa:4096 -keyout /etc/ssl/private/dns.key -out /etc/ssl/certs/dns.pem -sha256 -days 3650 Server Block & Upstream upstream http2-doh { server 127.0.0.1:80; } server { listen 8080 default_server; listen 4443 ssl http2; server_name _; # TLS certificate chain and corresponding private key ssl_certificate /etc/ssl/certs/dns.pem; ssl_certificate_key /etc/ssl/private/dns.key; location / { grpc_pass grpc://http2-doh; } location /health-check { add_header Content-Type text/plain; return 200 'what is up buttercup?!'; } } Source: config-map.yml Deployment The deployment models in XC are pretty great, deploy to a cloud site, deploy to on-prem datacenters, deploy to our compute, or any combination. The services can then be published to the internet, to a cloud site, or to an on-prem site with all of the same security models for every facility. The deployment is pretty standard as well, the important pieces are ves.io/sites: this is important for the same reasons mentioned previously, but determines where the workloads will reside, with the same options as before. Environment Variables are where we need to tweek the settings a bit. A full listing of the values can be seen here: https://github.com/Mikej81/docker-bind Some of the values should be self-explanatory, but an important setting for a zone / a mapping is DNS_A. If there is a desire to bring in full zone files, it is possible to create that via a FILE value or a config map for Bind and storing the zone file in the proper named path and mapping the volume. **Not covered here** We will also be mapping some example self-signed certificates, which are only required if encryption is desired all the way to the container / pod. apiVersion: apps/v1 kind: Deployment metadata: name: bind-doh-dep labels: app: bind annotations: ves.io/sites: system/coleman-azure,system/coleman-cluster-100,system/colemantest spec: replicas: 1 selector: matchLabels: app: bind template: metadata: labels: app: bind spec: containers: - name: bind image: mcoleman81/bind-doh env: - name: DOCKER_LOGS value: "1" - name: ALLOW_QUERY value: "any" - name: ALLOW_RECURSION value: "any" - name: DNS_FORWARDER value: "8.8.8.8, 8.8.4.4" - name: DNS_A value: domain1.com=68.183.126.197,domain2.com=68.183.126.197 Source: deployment.yml Services We are almost done building out the manifest. We created the config map, we created the deployment, now we just need to expose some services. The targetPorts need to be above 1024 in the managed namespace so if those are changed, just follow that guideline. apiVersion: v1 kind: Service metadata: name: bind-services annotations: ves.io/sites: system/coleman-azure spec: type: ClusterIP selector: app: bind ports: - name: dns-udp port: 53 targetPort: 5553 protocol: UDP - name: dns-tcp port: 53 targetPort: 5353 protocol: TCP - name: dns-http port: 80 targetPort: 8888 protocol: TCP - name: nginx-http-listener port: 8080 targetPort: 8080 protocol: TCP - name: nginx-https-listener port: 4443 targetPort: 4443 protocol: TCP Source: service.yml Based on everything we have done, we know that the service name will be our [servicename].[namespace created previously], in my case it will be bind-services.m-coleman. We will need that value in a few steps when creating our Origin Pool. bind-manifest.yml Putting it all together! Full manifest can be found here: bind-manifest.yml Apply! kubectl apply -f bind-manifest.yml Application Deliver & Load Balancers HTTPS Origin Now we can create an origin pool. Over on the left menu, under Manage, Load Balancers, click Origin Pools. Let’s give our origin pool a name, and add some Origin Servers, so under Origin Servers, click Add Item. In the Origin Server settings, we want to select K8s Service Name of Origin Server on given Sites as our type, and enter our service name, which will be the service name we remembered from earlier and our namespace, so “servicename.namespace”. For the Site, we select one of the sites we deployed the workload to, and under Select Network on the Site, we want to select vK8s Networks on the Site, then click Apply. Do this for each site we deployed to so we have several servers in our Origin Pool. We also need to tweak the TLS settings since it will be encrypted over 4443 to the origin, but we don’t want to validate the certs since they are self-signed certs with low security settings, in my case, so update this as needed. Once everything is set right, click Save and Exit. HTTPS Load Balancer Now we need a load balancer, on the left menu bar, under Manage, select Load Balanacers and click HTTP Load Balancers. Click Add HTTP Load Balancer and lets assign a name. This is another location where configurations will diverge, for ease of deployment I am going to use an HTTPS Load Balancer with Auto Generated Certificates, but you can use HTTPS with Custom certificates as well. Note: For HTTPS with Auto-Certificate, Advertisement is Internet only, Custom Certificate allows Internet and Internal based advertising. HSTS is optional, as are most of the options shown below aside from Load Balancer Type and HTTPS Port. Under Origin, we Add Item and add the origin pool we created previously. Under Other Settings is where we can configure how & where the service is advertised. If we are going to advertise this service to an internal network only, we would select Custom here, then click configure. An example of what that would look like would be to click add item under the Custom Advertise VIP Configuration menu, Select the type of Site to advertise to, the type of interface to advertise on, and the specific site location. Click Apply as needed, then Save and Exit. Moment of Truth There are several ways to test to make sure we have everything up and running, first lets make sure our services are up. kubectl get services -n m-coleman NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE bind-services ClusterIP 192.168.175.193 <none> 53/UDP,53/TCP,80/TCP,8080/TCP,4443/TCP 3h57m Curl has built in DNS over HTTPS support so we can test via curl to see if our sites are resolving so first ill test one of our custom zones / A records. We are in business! We can also test with firefox or chrome or any number of other tools. Dig Dog Kdig Etc In Part 2 we will cover publishing TCP and UDP.
