F5 High Availability - Public Cloud Guidance
This article will provide information about BIG-IP and NGINX high availability (HA) topics that should be considered when leveraging the public cloud. There are differences between on-prem and public cloud such as cloud provider L2 networking. These differences lead to challenges in how you address HA, failover time, peer setup, scaling options, and application state. Topics Covered: Discuss and Define HA Importance of Application Behavior and Traffic Sizing HA Capabilities of BIG-IP and NGINX Various HA Deployment Options (Active/Active, Active/Standby, auto scale) Example Customer Scenario What is High Availability? High availability can mean many things to different people. Depending on the application and traffic requirements, HA requires dual data paths, redundant storage, redundant power, and compute. It means the ability to survive a failure, maintenance windows should be seamless to user, and the user experience should never suffer...ever! Reference: https://en.wikipedia.org/wiki/High_availability So what should HA provide? Synchronization of configuration data to peers (ex. configs objects) Synchronization of application session state (ex. persistence records) Enable traffic to fail over to a peer Locally, allow clusters of devices to act and appear as one unit Globally, disburse traffic via DNS and routing Importance of Application Behavior and Traffic Sizing Let's look at a common use case... "gaming app, lots of persistent connections, client needs to hit same backend throughout entire game session" Session State The requirement of session state is common across applications using methods like HTTP cookies,F5 iRule persistence, JSessionID, IP affinity, or hash. The session type used by the application can help you decide what migration path is right for you. Is this an app more fitting for a lift-n-shift approach...Rehost? Can the app be redesigned to take advantage of all native IaaS and PaaS technologies...Refactor? Reference: 6 R's of a Cloud Migration Application session state allows user to have a consistent and reliable experience Auto scaling L7 proxies (BIG-IP or NGINX) keep track of session state BIG-IP can only mirror session state to next device in cluster NGINX can mirror state to all devices in cluster (via zone sync) Traffic Sizing The cloud provider does a great job with things like scaling, but there are still cloud provider limits that affect sizing and machine instance types to keep in mind. BIG-IP and NGINX are considered network virtual appliances (NVA). They carry quota limits like other cloud objects. Google GCP VPC Resource Limits Azure VM Flow Limits AWS Instance Types Unfortunately, not all limits are documented. Key metrics for L7 proxies are typically SSL stats, throughput, connection type, and connection count. Collecting these application and traffic metrics can help identify the correct instance type. We have a list of the F5 supported BIG-IP VE platforms on F5 CloudDocs. F5 Products and HA Capabilities BIG-IP HA Capabilities BIG-IP supports the following HA cluster configurations: Active/Active - all devices processing traffic Active/Standby - one device processes traffic, others wait in standby Configuration sync to all devices in cluster L3/L4 connection sharing to next device in cluster (ex. avoids re-login) L5-L7 state sharing to next device in cluster (ex. IP persistence, SSL persistence, iRule UIE persistence) Reference: BIG-IP High Availability Docs NGINX HA Capabilities NGINX supports the following HA cluster configurations: Active/Active - all devices processing traffic Active/Standby - one device processes traffic, others wait in standby Configuration sync to all devices in cluster Mirroring connections at L3/L4 not available Mirroring session state to ALL devices in cluster using Zone Synchronization Module (NGINX Plus R15) Reference: NGINX High Availability Docs HA Methods for BIG-IP In the following sections, I will illustrate 3 common deployment configurations for BIG-IP in public cloud. HA for BIG-IP Design #1 - Active/Standby via API HA for BIG-IP Design #2 - A/A or A/S via LB HA for BIG-IP Design #3 - Regional Failover (multi region) HA for BIG-IP Design #1 - Active/Standby via API (multi AZ) This failover method uses API calls to communicate with the cloud provider and move objects (IP address, routes, etc) during failover events. The F5 Cloud Failover Extension (CFE) for BIG-IP is used to declaratively configure the HA settings. Cloud provider load balancer is NOT required Fail over time can be SLOW! Only one device actively used (other device sits idle) Failover uses API calls to move cloud objects, times vary (see CFE Performance and Sizing) Key Findings: Google API failover times depend on number of forwarding rules Azure API slow to disassociate/associate IPs to NICs (remapping) Azure API fast when updating routes (UDR, user defined routes) AWS reliable with API regarding IP moves and routes Recommendations: This design with multi AZ is more preferred than single AZ Recommend when "traditional" HA cluster required or Lift-n-Shift...Rehost For Azure (based on my testing)... Recommend using Azure UDR versus IP failover when possible Look at Failover via LB example instead for Azure If API method required, look at DNS solutions to provide further redundancy HA for BIG-IP Design #2 - A/A or A/S via LB (multi AZ) Cloud LB health checks the BIG-IP for up/down status Faster failover times (depends on cloud LB health timers) Cloud LB allows A/A or A/S Key difference: Increased network/compute redundancy Cloud load balancer required Recommendations: Use "failover via LB" if you require faster failover times For Google (based on my testing)... Recommend against "via LB" for IPSEC traffic (Google LB not supported) If load balancing IPSEC, then use "via API" or "via DNS" failover methods HA for BIG-IP Design #3 - Regional Failover via DNS (multi AZ, multi region) BIG-IP VE active/active in multiple regions Traffic disbursed to VEs by DNS/GSLB DNS/GSLB intelligent health checks for the VEs Key difference: Cloud LB is not required DNS logic required by clients Orchestration required to manage configs across each BIG-IP BIG-IP standalone devices (no DSC cluster limitations) Recommendations: Good for apps that handle DNS resolution well upon failover events Recommend when cloud LB cannot handle a particular protocol Recommend when customer is already using DNS to direct traffic Recommend for applications that have been refactored to handle session state outside of BIG-IP Recommend for customers with in-house skillset to orchestrate (Ansible, Terraform, etc) HA Methods for NGINX In the following sections, I will illustrate 2 common deployment configurations for NGINX in public cloud. HA for NGINX Design #1 - Active/Standby via API HA for NGINX Design #2 - Auto Scale Active/Active via LB HA for NGINX Design #1 - Active/Standby via API (multi AZ) NGINX Plus required Cloud provider load balancer is NOT required Only one device actively used (other device sits idle) Only available in AWS currently Recommendations: Recommend when "traditional" HA cluster required or Lift-n-Shift...Rehost Reference: Active-Passive HA for NGINX Plus on AWS HA for NGINX Design #2 - Auto Scale Active/Active via LB (multi AZ) NGINX Plus required Cloud LB health checks the NGINX Faster failover times Key difference: Increased network/compute redundancy Cloud load balancer required Recommendations: Recommended for apps fitting a migration type of Replatform or Refactor Reference: Active-Active HA for NGINX Plus on AWS, Active-Active HA for NGINX Plus on Google Pros & Cons: Public Cloud Scaling Options Review this handy table to understand the high level pros and cons of each deployment method. Example Customer Scenario #1 As a means to make this topic a little more real, here isa common customer scenario that shows you the decisions that go into moving an application to the public cloud. Sometimes it's as easy as a lift-n-shift, other times you might need to do a little more work. In general, public cloud is not on-prem and things might need some tweaking. Hopefully this example will give you some pointers and guidance on your next app migration to the cloud. Current Setup: Gaming applications F5 Hardware BIG-IP VIRPIONs on-prem Two data centers for HA redundancy iRule heavy configuration (TLS encryption/decryption, payload inspections) Session Persistence = iRule Universal Persistence (UIE), and other methods Biggest app 15K SSL TPS 15Gbps throughput 2 million concurrent connections 300K HTTP req/sec (L7 with TLS) Requirements for Successful Cloud Migration: Support current traffic numbers Support future target traffic growth Must run in multiple geographic regions Maintain session state Must retain all iRules in use Recommended Design for Cloud Phase #1: Migration Type: Hybrid model, on-prem + cloud, and some Rehost Platform: BIG-IP Retaining iRules means BIG-IP is required Licensing: High Performance BIG-IP Unlocks additional CPU cores past 8 (up to 24) extra traffic and SSL processing Instance type: check F5 supported BIG-IP VE platforms for accelerated networking (10Gb+) HA method: Active/Standby and multi-region with DNS iRule Universal persistence only mirrors to only next device, keep cluster size to 2 scale horizontally via additional HA clusters and DNS clients pinned to a region via DNS (on-prem or public cloud) inside region, local proxy cluster shares state This example comes up in customer conversations often. Based on customer requirements, in-house skillset, current operational model, and time frames there is one option that is better than the rest. A second design phase lends itself to more of a Replatform or Refactor migration type. In that case, more options can be leveraged to take advantage of cloud-native features. For example, changing the application persistence type from iRule UIE to cookie would allow BIG-IP to avoid keeping track of state. Why? With cookies, the client keeps track of that session state. Client receives a cookie, passes the cookie to L7 proxy on successive requests, proxy checks cookie value, sends to backend pool member. The requirement for L7 proxy to share session state is now removed. Example Customer Scenario #2 Here is another customer scenario. This time the application is a full suite of multimedia content. In contrast to the first scenario, this one will illustrate the benefits of rearchitecting various components allowing greater flexibility when leveraging the cloud. You still must factor in-house skill set, project time frames, and other important business (and application) requirements when deciding on the best migration type. Current Setup: Multimedia (Gaming, Movie, TV, Music) Platform BIG-IP VIPRIONs using vCMP on-prem Two data centers for HA redundancy iRule heavy (Security, Traffic Manipulation, Performance) Biggest App: oAuth + Cassandra for token storage (entitlements) Requirements for Success Cloud Migration: Support current traffic numbers Elastic auto scale for seasonal growth (ex. holidays) VPC peering with partners (must also bypass Web Application Firewall) Must support current or similar traffic manipulating in data plane Compatibility with existing tooling used by Business Recommended Design for Cloud Phase #1: Migration Type: Repurchase, migration BIG-IP to NGINX Plus Platform: NGINX iRules converted to JS or LUA Licensing: NGINX Plus Modules: GeoIP, LUA, JavaScript HA method: N+1 Autoscaling via Native LB Active Health Checks This is a great example of a Repurchase in which application characteristics can allow the various teams to explore alternative cloud migration approaches. In this scenario, it describes a phase one migration of converting BIG-IP devices to NGINX Plus devices. This example assumes the BIG-IP configurations can be somewhat easily converted to NGINX Plus, and it also assumes there is available skillset and project time allocated to properly rearchitect the application where needed. Summary OK! Brains are expanding...hopefully? We learned about high availability and what that means for applications and user experience. We touched on the importance of application behavior and traffic sizing. Then we explored the various F5 products, how they handle HA, and HA designs. These recommendations are based on my own lab testing and interactions with customers. Every scenario will carry its own requirements, and all options should be carefully considered when leveraging the public cloud. Finally, we looked at a customer scenario, discussed requirements, and design proposal. Fun! Resources Read the following articles for more guidance specific to the various cloud providers. Advanced Topologies and More on Highly Available Services Lightboard Lessons - BIG-IP Deployments in Azure Google and BIG-IP Failing Faster in the Cloud BIG-IP VE on Public Cloud High-Availability Load Balancing with NGINX Plus on Google Cloud Platform Using AWS Quick Starts to Deploy NGINX Plus NGINX on AzureUsing BIG-IP GTM to Integrate with Amazon Web Services
This is the latest in a series of DNS articles that I've been writing over the past couple of months. This article is taken from a fantastic solution that Joe Cassidy developed. So, thanks to Joe for developing this solution, and thanks for the opportunity to write about it here on DevCentral. As a quick reminder, my first six articles are: Let's Talk DNS on DevCentral DNS The F5 Way: A Paradigm Shift DNS Express and Zone Transfers The BIG-IP GTM: Configuring DNSSEC DNS on the BIG-IP: IPv6 to IPv4 Translation DNS Caching The Scenario Let's say you are an F5 customer who has external GTMs and LTMs in your environment, but you are not leveraging them for your main website (example.com). Your website is a zone sitting on your windows DNS servers in your DMZ that round robin load balance to some backend webservers. You've heard all about the benefits of the cloud (and rightfully so), and you want to move your web content to the Amazon Cloud. Nice choice! As you were making the move to Amazon, you were given instructions by Amazon to just CNAME your domain to two unique Amazon Elastic Load Balanced (ELB) domains. Amazon’s requests were not feasible for a few reasons...one of which is that it breaks the RFC. So, you engage in a series of architecture meetings to figure all this stuff out. Amazon told your Active Directory/DNS team to CNAME www.example.com and example.com to two AWS clusters: us-east.elb.amazonaws.com and us-west.elb.amazonaws.com. You couldn't use Microsoft DNS to perform a basic CNAME of these records because of the BIND limitation of CNAME'ing a single A record to multiple aliases. Additionally, you couldn't point to IPs because Amazon said they will be using dynamic IPs for your platform. So, what to do, right? The Solution The good news is that you can use the functionality and flexibility of your F5 technology to easily solve this problem. Here are a few steps that will guide you through this specific scenario: Redirect requests for http://example.com to http://www.example.com and apply it to your Virtual Server (1.2.3.4:80). You can redirect using HTTP Class profiles (v11.3 and prior) or using a policy with Centralized Policy Matching (v11.4 and newer) or you can always write an iRule to redirect! Make www.example.com a CNAME record to example.lb.example.com; where *.lb.example.com is a sub-delegated zone of example.com that resides on your BIG-IP GTM. Create a global traffic pool “aws_us_east” that contains no members but rather a CNAME to us-east.elb.amazonaws.com. Create another global traffic pool “aws_us_west” that contains no members but rather a CNAME to us-west.elb.amazonaws.com. The following screenshot shows the details of creating the global traffic pools (using v11.5). Notice you have to select the "Advanced" configuration to add the CNAME. Create a global traffic Wide IP example.lb.example.com with two pool members “aws_us_east” and “aws_us_west”. The following screenshot shows the details. Create two global traffic regions: “eastern” and “western”. The screenshot below shows the details of creating the traffic regions. Create global traffic topology records using "Request Source: Region is eastern" and "Destination Pool is aws_us_east". Repeat this for the western region using the aws_us_west pool. The screenshot below shows the details of creating these records. Modify Pool settings under Wide IP www.example.com to use "Topology" as load balancing method. See the screenshot below for details. How it all works... Here's the flow of events that take place as a user types in the web address and ultimately receives the correct IP address. External client types http://example.com into their web browser Internet DNS resolution takes place and maps example.com to your Virtual Server address: IN A 1.2.3.4 An HTTP request is directed to 1.2.3.4:80 Your LTM checks for a profile, the HTTP profile is enabled, the redirect request is applied, and redirect user request with 301 response code is executed External client receives 301 response code and their browser makes a new request to http://www.example.com Internet DNS resolution takes place and maps www.example.com to IN CNAME example.lb.example.com Internet DNS resolution continues mapping example.lb.example.com to your GTM configured Wide IP The Wide IP load balances the request to one of the pools based on the configured logic: Round Robin, Global Availability, Topology or Ratio (we chose "Topology" for our solution) The GTM-configured pool contains a CNAME to either us_east or us_west AWS data centers Internet DNS resolution takes place mapping the request to the ELB hostname (i.e. us-west.elb.amazonaws.com) and gives two A records External client http request is mapped to one of the returned IP addresses And, there you have it. With this solution, you can integrate AWS using your existing LTM and GTM technology! I hope this helps, and I hope you can implement this and other solutions using all the flexibility and power of your F5 technology.Building an OpenSSL Certificate Authority - Creating Your Root Certificate
Creating Your Root Certificate Authority In our previous article, Introductions and Design Considerations for Eliptical Curveswe covered the design requirements to create a two-tier ECC certificate authority based on NSA Suite B's PKI requirements. We can now begin creating our CA's root configuration. Creating the root CArequires us to generate a certificate and private key,since this is the first certificate we're creating, it will be self-signed. The root CA will not sign client and server certificates, it's job it only to create intermeidary certificates and act as the root of our chain of trust. This is standard practice across the public and private PKI configurations and so too should your lab environments. Create Your Directory Structure Create a directory to store your root CA pair and config files. # sudo bash # mkdir /root/ca Yep, I did that. This is for a test lab and permissions may not match real world requirements. I sudoed into bash and created everything under root; aka playing with fire. This affects ownership down the line if you chmod private key files and directoriesto user access only so determine for yourself what user/permission will be accessing files for certificate creation. I have a small team and trust them with root within a lab environment (snapshots allow me to be this trusting). Create your CA database to keep track of signed certificates # cd /root/ca # mkdir private certs crl # touch index.txt # echo 1000 > serial We begin by creating a working root directory with sub directories for the various files we'll be creating. This will allow you to apply your preferred security practices should you choose to do so. Since this is a test lab and I am operating as root, I won't be chmod'ing anything today. Create Your OpenSSL Config File OpenSSLuses configuration files to simplify/template the components of a certificate. Copy the GIST openssl_root.cnf file to /root/ca/openssl_root.cnf which is already prepared for this demo. For the root CA certificate creation, the [ CA ] section is required and will gather it's configuration from the [ CA_default ] section. [ ca ] # `man ca` default_ca = CA_default The [CA_default] section in the openssl_root.cnf file contains the variables OpenSSL will use for the root CA. If you're using alternate directory names from this demo, update the file accordingly. Note the long values for default days (10 years) as we don't care about renewing the root certificate anytime soon. [ CA_default ] # Directory and file locations. dir = /root/ca certs = $dir/certs crl_dir = $dir/crl new_certs_dir = $dir/certs database = $dir/index.txt serial = $dir/serial RANDFILE = $dir/private/.rand # The root key and root certificate. private_key = $dir/private/ca.cheese.key.pem certificate = $dir/certs/ca.cheese.crt.pem # For certificate revocation lists. crlnumber = $dir/crlnumber crl = $dir/crl/ca.cheese.crl.pem crl_extensions = crl_ext default_crl_days = 3650 # SHA-1 is deprecated, so use SHA-2 or SHA-3 instead. default_md = sha384 name_opt = ca_default cert_opt = ca_default default_days = 3650 preserve = no policy = policy_strict For the root CA, we define [policy_strict] which will later force the intermediary's certificateto match country, state/province, and organization name fields. [ policy_strict ] The root CA should only sign intermediate certificates that match. # See POLICY FORMAT section of `man ca`. countryName = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName = supplied emailAddress = optional The [ req ] section is used for OpenSSL certificate requests. Some of the values listed will not be used since we are manually specifying them during certificate creation. [ req ] # Options for the `req` tool (`man req`). default_bits = 4096 distinguished_name = req_distinguished_name string_mask = utf8only # SHA-1 is deprecated, please use SHA-2 or greater instead. default_md = sha384 # Extension to add when the -x509 option is used. x509_extensions = v3_ca I pre-populate the [ req_distinguished_name ] section with values I'll commonly used to save typing down the road. [ req_distinguished_name ] countryName = Country Name (2 letter code) stateOrProvinceName = State or Province Name localityName = Locality Name 0.organizationName = Organization Name organizationalUnitName = Organizational Unit Name commonName = Common Name emailAddress = Email Address # Optionally, specify some defaults. countryName_default = US stateOrProvinceName_default = WA localityName_default = Seattle 0.organizationName_default = Grilled Cheese Inc. organizationalUnitName_default = Grilled Cheese Root CA emailAddress_default = grilledcheese@yummyinmytummy.us The [ v3_ca ] section will further define the Suite B PKI requirements, namely basicConstraints and acceptable keyUsage values for a CA certificate. This section will be used for creating the root CA's certificate. [ v3_ca ] # Extensions for a typical CA (`man x509v3_config`). subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer basicConstraints = critical, CA:true keyUsage = critical, digitalSignature, cRLSign, keyCertSign Selecting the Suite B compliant elliptical curve We're creating a Suite B infrastructure so we'll need to pick an acceptable curve following P-256 or P-384. To do this, run the following OpenSSLcommand: openssl ecparam -list_curves This will give you a long list of options but which one to pick? Let's isolate the suites within the 256 & 384 prime fields; we can grep the results for easier curve identification. openssl ecparam -list_curves | grep '256\|384' And we get the following results (your results may vary depending on the version of OpenSSL running): # openssl ecparam -list_curves | grep '256\|384' secp256k1 : SECG curve over a 256 bit prime field secp384r1 : NIST/SECG curve over a 384 bit prime field prime256v1: X9.62/SECG curve over a 256 bit prime field brainpoolP256r1: RFC 5639 curve over a 256 bit prime field brainpoolP256t1: RFC 5639 curve over a 256 bit prime field brainpoolP384r1: RFC 5639 curve over a 384 bit prime field brainpoolP384t1: RFC 5639 curve over a 384 bit prime field I am going to use secp384r1 as my curve of choice. It's good to mention that RFC5480 notes secp256r1 (not listed) is referred to as prime256v1 for this output's purpose. Why not use something larger than 384? Thank Google. People absolutely were using secp521r1 then Google dropped support for it (read Chromium Bug 478225 for more). The theoryis since NSA Suite B PKI did not explicitly call out anything besides 256 or 384, the Chromium team quietly decided it wasn't needed and dropped support for it. Yea... it kinda annoyed a few people. So to avoid future browser issues, we're sticking with what's defined in public standards. Create the Root CA's Private Key Using the names defined in the openssl_root.cnf's private_key value and our selected secp384r1 ECC curve we will create and encrypt the root certificates private key. # openssl ecparam -genkey -name secp384r1 | openssl ec -aes256 -out private/ca.cheese.key.pem read EC key writing EC key Enter PEM pass phrase: ****** Verifying - Enter PEM pass phrase: ****** Note:The ecparam function within OpenSSL does not encrypt the private key like genrsa/gendsa/gendh does. Instead we combined the private key creation (openssl ecparam) with a secondary encryption command (openssl ec) to encrypt private key before it is written to disk. Keep the password safe. Create the Root CA's Certificate Using the new private key, we can now generate our root'sself-signed certificate. We do this because the root has no authority above it to request trust authority from;it is the absolute source of authority in our certificate chain. # openssl req -config openssl_root.cnf -new -x509 -sha384 -extensions v3_ca -key private/ca.cheese.key.pem -out certs/ca.cheese.crt.pem Enter pass phrase for private/ca.cheese.key.pem: ****** You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [US]: State or Province Name [WA]: Locality Name [Seattle]: Organization Name [Grilled Cheese Inc.]: Organizational Unit Name [Grilled Cheese Root CA]: Common Name []:Grilled Cheese Root Certificate Authority Email Address [grilledcheese@yummyinmytummy.us]: Using OpenSSL we can validate the Certificate contents to ensure we're following the NSA Suite B requirements. # openssl x509 -noout -text -in certs/ca.cheese.crt.pem Certificate: Data: Version: 3 (0x2) Serial Number: ff:bd:f5:2f:c5:0d:3d:02 Signature Algorithm: ecdsa-with-SHA384 Issuer: C = US, ST = WA, L = Seattle, O = Grilled Cheese Inc., OU = Grilled Cheese Root CA, CN = Grilled Cheese Inc. Root Certificate Authority, emailAddress = grilledcheese@yummyinmytummy.us Validity Not Before: Aug 22 23:53:05 2017 GMT Not After : Aug 20 23:53:05 2027 GMT Subject: C = US, ST = WA, L = Seattle, O = Grilled Cheese Inc., OU = Grilled Cheese Root CA, CN = Grilled Cheese Inc. Root Certificate Authority, emailAddress = grilledcheese@yummyinmytummy.us Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (384 bit) pub: 04:a6:b7:eb:8b:9f:fc:95:03:02:20:ea:64:7f:13: ea:b7:75:9b:cd:5e:43:ca:19:70:17:e2:0a:26:79: 0a:23:2f:20:de:02:2d:7c:8f:62:6b:74:7d:82:fe: 04:08:38:77:b7:8c:e0:e4:2b:27:0f:47:01:64:38: cb:15:a8:71:43:b2:d9:ff:ea:0e:d1:c8:f4:8f:99: d3:8e:2b:c1:90:d6:77:ab:0b:31:dd:78:d3:ce:96: b1:a0:c0:1c:b0:31:39 ASN1 OID: secp384r1 NIST CURVE: P-384 X509v3 extensions: X509v3 Subject Key Identifier: 27:C8:F7:34:2F:30:81:97:DE:2E:FC:DD:E2:1D:FD:B6:8F:5A:AF:BB X509v3 Authority Key Identifier: keyid:27:C8:F7:34:2F:30:81:97:DE:2E:FC:DD:E2:1D:FD:B6:8F:5A:AF:BB X509v3 Basic Constraints: critical CA:TRUE X509v3 Key Usage: critical Digital Signature, Certificate Sign, CRL Sign Signature Algorithm: ecdsa-with-SHA384 30:65:02:30:77:a1:f9:e2:ab:3a:5a:4b:ce:8d:6a:2e:30:3f: 01:cf:8e:76:dd:f6:1f:03:d9:b3:5c:a1:3d:6d:36:04:fb:01: f7:33:27:03:85:de:24:56:17:c9:1a:e4:3b:35:c4:a8:02:31: 00:cd:0e:6c:e0:d5:26:d3:fb:88:56:fa:67:9f:e9:be:b4:8f: 94:1c:2c:b7:74:19:ce:ec:15:d2:fe:48:93:0a:5f:ff:eb:b2: d3:ae:5a:68:87:dc:c9:2c:54:8d:04:68:7f Reviewing the above we can verify the certificate details: The Suite B Signature Algorithm: ecdsa-with-SHA384 The certificate date validity when we specificed -days 3650: Not Before: Aug 22 23:53:05 2017 GMT Not After : Aug 20 23:53:05 2027 GMT The Public-Key bit length: (384 bit) The Issuer we defined in the openssl_root.cnf: C = US, ST = WA, L = Seattle, O = Grilled Cheese Inc., OU = Grilled Cheese Root CA, CN = Grilled Cheese Inc. Root Certificate Authority The Certificate Subject, since this is self-signed, refers back to itself: Subject: C = US, ST = WA, L = Seattle, O = Grilled Cheese Inc., OU = Grilled Cheese Root CA, CN = Grilled Cheese Inc. Root Certificate Authority The eliptical curve used when we created the private key: NIST CURVE: P-384 Verify the X.509 v3 extensions we defined within the openssl_root.cnf for a Suite B CA use: X509v3 Basic Constraints: critical CA:TRUE X509v3 Key Usage: critical Digital Signature, Certificate Sign, CRL Sign The root certificate and private key are now compete and we have the first part of our CA complete. Step 1 complete! In our next article we willcreate the intermediary certificate to complete the chain of trust in our two-tier hierarchy.Building an OpenSSL Certificate Authority - Creating Your Intermediary Certificate
Creating Your Intermediary Certificate Authority Previously we created the first part of our OpenSSL CA by building our root certificate. We are now ready to complete our CA chain by creating and signing the intermediary certificate.The intermediary will beresponsible for signing client and server certificate requests. It acts as an authoritative proxy for the root certificate hence the name intermediary. The chain of trust will extend from the root certificate to the intermediary certificate down to the certificates you'll deploy within your infrastructure. Create your directory structure Create a new subdirectory under /root/ca to segregate intermediary files our root configuration . # sudo bash # mkdir /root/ca/intermediate We're creating the same directory structure previously used under /root/ca within /root/ca/intermediary . It's your decision if you if you want to do something different. Some of my best friends are flat directory structures and we don't judge personal practices. Create your intermediary CA database to keep track of signed certificates # cd /root/ca/intermediate # mkdir certs crl csr private # touch index.txt # echo 1000 > serial Create a crlnumber file for the intermediary CA to use # echo 1000 > /root/ca/intermediate/crlnumber Similar to the earlier serial statement, this will create the crlnumber file and start the numerical iteration at 1000. This will be used for future certificate revocation needs. Create your OpenSSL intermediary config file Copy the GIST openssl_intermediate.cnf file to /root/ca/intermediate/openssl_intermediate.cnf and modify the contents for your own naming conventions. Similar to the root_ca.cnf , the [CA] is required and will gather it's configuration from the [CA_default] section. Changes to the [int_ca] include: [ CA_default ] # Directory and file locations. dir = /root/ca/intermediate private_key = $dir/private/int.cheese.key.pem certificate = $dir/cers/int.cheese.crt.pem crlnumber = $dir/crlnumber crl = $dir/crl/int.cheese.crl.pem crl_extensions = crl_ext policy = policy_loose We have new certificate names for our intermediary use and define policy_loose so future certificate requests don't have to match country, state/province, or organization. Create the Intermediary's Private Key and Certificate Signing Request Similar to the root certificate, we're following the NSA Suite B requirements and matching the root's elliptical curve, secp384r1. We'll alsocreate the CSR and private key all in one line, making your scripts and life a bit easier. # cd /root/ca # openssl req -config intermediate/openssl_intermediate.cnf -new -newkey ec:<(openssl ecparam -name secp384r1) -keyout intermediate/private/int.cheese.key.pem -out intermediate/csr/int.cheese.csr Generating an EC private key writing new private key to 'intermediate/private/int.cheese.key.pem' Enter PEM pass phrase: ****** Verifying - Enter PEM pass phrase: ****** ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [US]: State or Province Name [WA]: Locality Name [Seattle]: Organization Name [Grilled Cheese Inc.]: Organizational Unit Name [Grilled Cheese Intermediary CA]: Common Name []:Grilled Cheese Inc. Intermediary Certificate Authority Email Address [grilledcheese@yummyinmytummy.us]: Sign the certificate request with the root certificate and use the openssl_intermediate.cnf config file to specify the [v3_intermediate_ca] extension instead of the [v3_ca] as we did for the root. The openssl_intermediate.cnf has a few changes which we need to note. [ v3_intermediate_ca ] # Extensions for a typical intermediate CA (`man x509v3_config`). subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer basicConstraints = critical, CA:true, pathlen:0 keyUsage = critical, digitalSignature, cRLSign, keyCertSign crlDistributionPoints = @crl_info authorityInfoAccess = @ocsp_info [crl_info] URI.0 = http://crl.grilledcheese.us/whoremovedmycheese.crl [ocsp_info] caIssuers;URI.0 = http://ocsp.grilledcheese.us/cheddarcheeseroot.crt OCSP;URI.0 = http://ocsp.grilledcheese.us/ The Certificate Revocation List (crl) and Online Certificate Status Protocol (OCSP) should be included within the intermediary certificate. This lets systems know where check and see if the intermediary certificate was revoked by the root at any given time. We will cover this in detail later and browsers do not necessarily check the intermediary certificates for revocation, but they absolutely do for the site certificates. We're adding CRL and OCSP to the Intermediary CA for best practices purpose. Create the intermediate certificate Sign the csr/int.cheese.cs r with the root's certificate. We are going to drop down to /root/ca so the creation of the intermediary certificate is stored within the root's index.txt and we'll also use the root's OpenSSL Config file openssl_root.cnf . # openssl ca -config openssl_root.cnf -extensions v3_intermediate_ca -days 3600 -md sha384 -in intermediate/csr/int.cheese.csr -out intermediate/certs/int.cheese.crt.pem Using configuration from openssl_root.cnf Enter pass phrase for /root/ca/private/ca.cheese.key.pem: Check that the request matches the signature Signature ok Certificate Details: Serial Number: 4097 (0x1001) Validity Not Before: Aug 24 21:51:07 2017 GMT Not After : Jul 3 21:51:07 2027 GMT Subject: countryName = US stateOrProvinceName = WA organizationName = Grilled Cheese Inc. organizationalUnitName = Grilled Cheese Intermediary CA commonName = Grilled Cheese Inc. Intermediary Certificate Authority emailAddress = grilledcheese@yummyinmytummy.us X509v3 extensions: X509v3 Subject Key Identifier: 7E:2D:A5:D0:9B:70:B9:E3:D2:F7:C0:0A:CF:70:9A:8B:80:38:B1:CD X509v3 Authority Key Identifier: keyid:27:C8:F7:34:2F:30:81:97:DE:2E:FC:DD:E2:1D:FD:B6:8F:5A:AF:BB X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 X509v3 Key Usage: critical Digital Signature, Certificate Sign, CRL Sign X509v3 CRL Distribution Points: Full Name: URI:http://crl.grilledcheese.us/whomovedmycheese.crl Authority Information Access: CA Issuers - URI:http://ocsp.grilledcheese.us/cheddarcheeseroot.crt OCSP - URI:http://ocsp.grilledcheese.us/ Certificate is to be certified until Jul 3 21:51:07 2027 GMT (3600 days) Sign the certificate? [y/n]:y 1 out of 1 certificate requests certified, commit? [y/n]y Write out database with 1 new entries Data Base Updated Validate the Certificate Contents with OpenSSL. # openssl x509 -noout -text -in intermediate/certs/int.cheese.crt.pem Certificate: Data: Version: 3 (0x2) Serial Number: 4097 (0x1001) Signature Algorithm: ecdsa-with-SHA384 Issuer: C = US, ST = WA, L = Seattle, O = Grilled Cheese Inc., OU = Grilled Cheese Root CA, CN = Grilled Cheese Inc. Root Certificate Authority, emailAddress = grilledcheese@yummyinmytummy.us Validity Not Before: Aug 24 21:51:07 2017 GMT Not After : Jul 3 21:51:07 2027 GMT Subject: C = US, ST = WA, O = Grilled Cheese Inc., OU = Grilled Cheese Intermediary CA, CN = Grilled Cheese Inc. Intermediary Certificate Authority, emailAddress = grilledcheese@yummyinmytummy.us Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (384 bit) pub: 04:9b:14:9a:55:6d:db:15:7f:d7:8b:fd:37:4d:ba: e8:50:8e:88:32:99:27:4e:20:36:25:8b:7b:ac:bb: 2f:d6:61:c1:5a:c8:e6:4c:98:20:3f:cf:86:3c:bf: f4:f3:b0:1c:1c:0b:cc:7f:e4:4b:13:59:58:a1:53: 87:cb:4c:17:66:04:21:01:6a:44:5f:22:31:7d:3d: fe:a2:e7:73:c8:77:7c:1a:f9:9c:4a:9d:e7:77:6a: c7:9e:3e:f0:4a:b0:37 ASN1 OID: secp384r1 NIST CURVE: P-384 X509v3 extensions: X509v3 Subject Key Identifier: 7E:2D:A5:D0:9B:70:B9:E3:D2:F7:C0:0A:CF:70:9A:8B:80:38:B1:CD X509v3 Authority Key Identifier: keyid:27:C8:F7:34:2F:30:81:97:DE:2E:FC:DD:E2:1D:FD:B6:8F:5A:AF:BB X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 X509v3 Key Usage: critical Digital Signature, Certificate Sign, CRL Sign X509v3 CRL Distribution Points: Full Name: URI:http://crl.grilledcheese.us/whomovedmycheese.crl Authority Information Access: CA Issuers - URI:http://ocsp.grilledcheese.us/cheddarcheeseroot.crt OCSP - URI:http://ocsp.grilledcheese.us/ Signature Algorithm: ecdsa-with-SHA384 30:65:02:30:74:07:ba:fe:4b:71:78:d8:d2:7f:84:c0:50:b4: b6:df:6c:f6:57:f5:d9:2c:4b:e1:d4:d8:1d:78:fd:7e:bf:0a: 81:86:bb:40:c5:9b:97:6f:83:04:5f:d3:85:36:6c:d6:02:31: 00:d3:08:78:1c:da:6d:ef:1d:bb:27:df:0b:76:eb:ab:84:b2: 91:04:25:1a:85:5b:d5:c3:cd:66:e4:9e:14:b2:c0:ed:9c:59: b7:18:c3:26:eb:df:78:13:68:47:66:b5:43 Similar to the root, we can note the usage and algorithms but we have the addition of: * X509v3 CRL Distribution Points: Full Name: URI:http://crl.grilledcheese.us/whomovedmycheese.crl *Authority Information Access: CA Issuers - URI:http://ocsp.grilledcheese.us/cheddarcheeseroot.crt OCSP - URI:http://ocsp.grilledcheese.us/ Create the certificate chain The root certificate and intermediary certificate must be available to the requesting client/server in order to validate the chain of trust. To complete the trust validation, a certificate chain must be available to the client application. A certificate chain usually takes the form of separate certificates installed into Root and Intermediary containers (as the case for Windows), or bundled together either in a .pfx cert and certchain bundle or a PEM formatted text file. Concatenate the root and intermediate certificates together to create a PEM certificate chain text file. # cd /root/ca # $cat intermediate/certs/int.cheese.crt.pem certs/ca.cheese.crt.pem > intermediate/certs/chain.cheese.crt.pem The file should look similar to this with two separate BEGIN and END statements for each certificate (example condensed for space): # cat intermediate/certs/chain.cheese.crt.pem -----BEGIN CERTIFICATE----- MIID/TCCA4OgAwIBAgICEAEwCgYIKoZIzj0EAwMwgdQxCzAJBgNVBAYTAlVTMQsw CQYDVQQIDAJXQTEQMA4GA1UEBwwHU2VhdHRsZTEcMBoGA1UECgwTR3JpbGxlZCBD ...... hkjOPQQDAwNoADBlAjB0B7r+S3F42NJ/hMBQtLbfbPZX9dksS+HU2B14/X6/CoGG u0DFm5dvgwRf04U2bNYCMQDTCHgc2m3vHbsn3wt266uEspEEJRqFW9XDzWbknhSy wO2cWbcYwybr33gTaEdmtUM= -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDQTCCAsegAwIBAgIJAP+99S/FDT0CMAoGCCqGSM49BAMDMIHUMQswCQYDVQQG EwJVUzELMAkGA1UECAwCV0ExEDAOBgNVBAcMB1NlYXR0bGUxHDAaBgNVBAoME0dy ...... CgYIKoZIzj0EAwMDaAAwZQIwd6H54qs6WkvOjWouMD8Bz4523fYfA9mzXKE9bTYE +wH3MycDhd4kVhfJGuQ7NcSoAjEAzQ5s4NUm0/uIVvpnn+m+tI+UHCy3dBnO7BXS /kiTCl//67LTrlpoh9zJLFSNBGh/ -----END CERTIFICATE----- Note: In the real world hosting application should never have the entire chain available as it defeats a core principle of PKI. It's recommended in test labs to distribute the root certificate to all testing client applications and systems and include only the intermediary along with the server certificate. This way the client can establish the trust between the intermediary and root certificates. Next we'll move on to creating our CLR endpoint list and OCSP certificate. Our intermediary certificate is now created and signed and we are ready to move on. To complete the CAour next articlewe will create our certificate revocation list (CRL) endpointand online certificate status protocol (OCSP) certificate allowing us to revoke certificates. Lab environments rarely need revocation functionalitybut modern clients check for CLR and OCSP URIs so it's nessisary to have the configruation defined at minimum. Let's proceed.
Ingress/Egress VPC inspection with BIG-IP and GWLB
Introduction The previous article in this series reviewed the BIG-IP and AWS Gateway Load Balancer (GWLB) integration, in this article we will focus on a deployment pattern that is used to inspect traffic in and out of a VPC using BIG-IP security services and GWLB. Baseline: In this scenario we will focus on a single existing VPC with EC2 instances and no BIG-IP security services Goal: Inspect traffic in and out of the VPC, scale the BIG-IP deployment as needed Deployment pattern: Considering the requirements and tools available, the deployment pattern will use the following attributes: Separate the security devices (BIG-IP's) from the workloads they are protecting. The BIG-IP's will be deployed in their own VPC – the Security VPC. We will reference the workload VPC as the 'Consumer' VPC as it consumes security services. Use Routing tables in the 'consumer' VPC to send all traffic to the security VPC for inspection. Leverage transparent security services on the BIG-IP for inspection (BIG-IP security features configuration is out of scope for this article) We can dive into each of the individual tasks: The Security VPC Provisioning the 'consumer' VPC network to send traffic through the security VPC Security VPC: Here, we are deploying the BIG-IP fleet and exposing it using GWLB. Some of the considerations when creating this VPC: Deploy in the same region as the 'consumer' VPC Design based on your availability requirements, Number of availability zones(AZ) How many BIG-IP's in each AZ These are the actions we need to take in the provider VPC to inspect all ingress/egress traffic: Deploy a group of BIG-IP's Create a GWLB target group and associate the BIG-IP's to it Create a GWLB and assign the previously created target group to the listener Create a GWLB endpoint service and associate it with the GWLB Configure the BIG-IP's to receive traffic over the tunnel and inspect according to the desired policy Diagram: The security VPC - BIG-IP fleet behind a GWLB, exposed using GWLB service Consumer VPC: In the consumer VPC, the BIG-IP group is abstracted by the GWLB and consumesthe security services from the provider VPC viaa new component:GWLB endpoint. This endpoint acts as bridge between the consumer VPC and the provider VPC. It essentially creates an ENI in one of the consumer's VPC subnet.Please note that a single endpoint belongs to a single availability zone and design accordingly. Inspecting all ingress traffic requires the use of 'Ingress routing' – an AWS feature that allows sending all ingress traffic from the internet gateway to an ENI or to a GWLB endpoint. Here are the actions we need to take in the consumer VPC to inspect all ingress/egress traffic: Create GWLB endpoints in each relevant availability zone that are attached to the 'GWLB endpoint service' from the provider VPC Change the ingress routing table so that all traffic in each AZ will get routed to the respective GWLB endpoint. Change the subnets routing tables so that all traffic in each AZ will get routed to the respective GWLB endpoint. Diagram: Inspecting all ingress/egress in the Security provider VPC Traffic flow Ingress traffic flow between an external user and an EC2 instance in the consumer VPC: Egress traffic flow between an EC2 instance in the consumer VPC and an external user: Summary: With this deployment you can protect your AWS VPC using the robust security services offered by the BIG-IP platform and get the following benefits: Scalability - Deploy as many BIG-IP instances as you need based on performance and availability requirements Transparent inspection - Inspecting the traffic without any address translation Optimized compute - all BIG-IP devices are active and processing traffic Next steps: Test the deployment yourself - Check out our self-service lab that you can deploy in your own AWS account(Fully automated deployment using Terraform): https://github.com/f5devcentral/f5-digital-customer-engagement-center/tree/main/solutions/security/ingress-egress-fwDeploy BIG-IP in AWS with HA across AZ’s - without using EIP’s
Background: The CloudFormation templates that are provided and supported by F5 are an excellent resource for customers to deploy BIG-IP VE in AWS. Along with these templates, documentation guiding your F5 deployment in AWS is an excellent resource. And of course, DevCentral articles are helpful. I recommend reading about HA topologies in AWS to start. I hope my article today can shed more light on an architecture that will suit a specific set of requirements: No Elastic IP's (EIP’s), High Availability (HA)across AZ’s, and multiple VPC’s. Requirements behind this architecture choice: I recently had a requirement to deploy BIG-IP appliances in AWS across AZ’s. I had read the official deployment guide, but I wasn’t clear on how to achieve failover without EIP’s. I was given 3 requirements: HA across AZ’s. In this architecture, we required a pair of BIG-IP devices in Active/Standby, where each device was in a different AZ. I needed to be able to fail over between devices. No EIP’s. This requirement existed because a 3 rd party firewall was already exposed to the Internet with a public IP address. That firewall would forward inbound requests to the BIG-IP VE in AWS, which in turn proxied traffic to a pair of web servers. Therefore, there was no reason to associate an EIP (a public IP address) with the BIG-IP interface. In my demo below I have not exposed a public website through a 3 rd party firewall, but to do so is a simple addition to this demo environment. Multiple VPC’s. This architecture had to allow for multiple VPC’s in AWS. There was already a “Security VPC” which contained firewalls, BIG-IP devices, and other devices, and I had to be able to use these devices for protecting applications that were across 1 or more disparate VPC’s. Meeting the requirements: HA across AZ’s This is the easiest of the requirements to meet because F5 has already provided templates to deploy these in AWS. I personally used a 2-nic device, with a BYOL license, deployed to an existing VPC, so that meant my template was this one. After this deployment is complete, you should have a pair of devices that will sync their configuration. At time of failover The supported F5 templates will deploy with the Advanced HA iApp. It is important that you configure this iApp after you have completed your AWS deployments. The iApp uses IAM permissions deployed with the template to make API calls to AWS at the time of failover. The API calls will update the route tables that you specify within the iApp wizard. Because this iApp is installed on both devices, either device can update the route in your route tables to point to its own interface. Update as of Dec 2019 This article was first written Feb 2019, and in Dec 2019 F5 released the Cloud Failover Extension (CFE), which is a cloud-agnostic, declarative way to configure failover in multiple public clouds. You can use the CFE instead of the Advanced HA iApp to achieve high availability between BIG-IP devices in cloud. Update as of Apr 2020 Your API calls will typically be sent to the public Internet endpoints for AWS EC2 API calls. Optionally, you can use AWS VPC endpoints to keep your API calls out to AWS EC2 from traversing the public Internet. My colleague Arnulfo Hernandez has written an article explaining how to do this. No EIP’s Configure an “alien IP range” I’m recycling another DevCentral solution here. You will need to choose an IP range for your VIP network that does not fall within the CIDR that is assigned to your VPC. Let’s call it an “alien range” because it “doesn’t belong” in your VPC and you couldn’t assign IP addresses from this range to your AWS ENI’s. Despite that, now create a route table within AWS that points this “alien range” to your Active BIG-IP device’s ENI (if you’re using a 2+ nic device, point it to the correct data plane NIC, not the Mgmt. interface). Don’t forget to associate the route table with specific subnets, per your design. Alternatively, you could add this route to the default VPC route table. Create a VIP on your active device Now create a VIP on your active device and configure the IP address as an IP within your alien range. Ensure the config replicates to your standby device. Ensure that source/destination checking is disabled on the ENI’s that your AWS routes are pointing to (on both Standby and Active devices). You should now have a VIP that you can target from other hosts in your VPC, provided that the route you created above is applied to the traffic destined to the VIP. Multiple VPC’s For extra credit, we’ll set up a Transit Gateway. This will allow other VPCs to route traffic to this “alien range” also, provided that the appropriate routes exist in the remote VPC’s, and also are applied to the appropriate Transit Gateway Route Table. Again, I’m recycling ideas that have already been laid out in other DevCentral posts. I won’t re-hash how to set up a transit gateway in AWS, because you can follow the linked post above. Sufficed to say, this is what you will need to set up if you want to route between multiple VPC’s using a transit gateway: 2 or more VPC’s A transit gateway in AWS Multiple transit gateway attachments that attach the transit gateway and each VPC you would like to route between. You will need one attachment per VPC. A transit gateway route table that is associated with each attachment. I will point out that you need to add a route for your “alien range” in your transit gateway route table, and in your remote VPC’s. That way, hosts in your remote VPC’s will forward traffic destined to your alien range (VIP network) to the transit gateway, and the transit gateway will forward it to your VPC, and the route you created in Step A will forward that traffic to your active BIG-IP device. Completed design: After the above configuration, you should have an environment that looks like the diagram below: Tips Internet access for deployments: When you deploy your BIG-IP devices, they will need Internet access to pull down some resources, including the iApp. So if you are deploying devices into your existing VPC, make sure you have a reachable Internet Gateway in AWS so that the devices have Internet access through both their management interface, and their data plane interface(s). Internet access for failover: Remember that an API call to AWS will still use an outbound request to the Internet. Make sure you allow the BIG-IP devices to make outbound connections over HTTPS to the Internet. If this is not available, you will find that your route tables are not updated at time of failover. (If you have a hard requirement that your devices should not have outbound access to internet, you can follow Arnulfo's guide linked to above and use VPC endpoints to keep this traffic on your local VPC) iApp logs: you can enable this in the iApp settings. I have used the logs (in /var/ltm/log) to troubleshoot issues that I have created myself. That’s how I learned not to accidentally cut off Internet access for your devices! Don’t forget about return routes if SNAT is disabled: Just like on-prem environments, if you disable SNAT, the pool member will need to route return traffic back to the BIG-IP device. You will commonly set up a default route (0.0.0.0/0) in AWS, point it at the ENI of the active BIG-IP device, and associate this route table with the subnet containing the pool members. If the pool members are in a remote VPC, you will need to create this route on the transit gw route table also. Don’t accidentally cut off internet access: When you configure the default route of 0.0.0.0/0 to point to eth1 of the BIG-IP device, don’t apply this route to every subnet in your Security VPC. It may be easy to do so accidentally, but remember that it could cause the API calls to update route tables to fail when the Standby device becomes Active. Don’t forget to disable source/dest check on your ENI’s. This is configured by the template, but if you have other devices that require it, remember to check this setting.
F5 in AWS Part 4 - Orchestrating BIG-IP Application Services with Open-Source tools
Updated for Current Versions and Documentation Part 1 : AWS Networking Basics Part 2: Running BIG-IP in an EC2 Virtual Private Cloud Part 3: Advanced Topologies and More on Highly-Available Services Part 4: Orchestrating BIG-IP Application Services with Open-Source Tools Part 5: Cloud-init, Single-NIC, and Auto Scale Out of BIG-IP in v12 The following post references code hosted at F5's Github repository f5networks/aws-deployments. This code provides a demonstration of using open-source tools to configure and orchestrate BIG-IP. Full documentation for F5 BIG-IP cloud work can be found at Cloud Docs: F5 Public Cloud Integrations. So far we have talked above AWS networking basics, how to run BIG-IP in a VPC, and highly-available deployment footprints. In this post, we’ll move on to my favorite topic, orchestration. By this point, you probably have several VMs running in AWS. You’ve lost track of which configuration is setup on which VM, and you have found yourself slowly going mad as you toggle between the AWS web portal and several SSH windows. I call this ‘point-and-click’ purgatory. Let's be blunt, why would you move to cloud without realizing the benefits of automation, of which cloud is a large enabler. If you remember our second article, we mentioned CloudFormation templates as a great way to deploy a standardized set of resources (perhaps BIG-IP + the additional virtualized network resources) in EC2. This is a great start, but we need to configure these resources once they have started, and we need a way to define and execute workflows which will run across a set of hosts, perhaps even hosts which are external to the AWS environment. Enter the use of open-source configuration management and workflow tools that have been popularized by the software development community. Open-source configuration management and AWS APIs Lately, I have been playing with Ansible, which is a python-based, agentless workflow engine for IT automation. By agentless, I mean that you don’t need to install an agent on hosts under management. Ansible, like the other tools, provides a number of libraries (or “modules”) which provide the ability to manage a diverse collection of remote systems. These modules are typically implemented through the use of API calls, often over HTTP. Out of the box, Ansible comes with several modules for managing resources in AWS. While the EC2 libraries provided are useful for basic orchestration use cases, we decided it would be easier to atomically manage sets of resources using the CloudFormation module. In doing so, we were able to deploy entire CloudFormation stacks which would include items like VPCs, networking elements, BIG-IP, app servers, etc. Underneath the covers, the CloudFormation: Ansible module and our own project use the python module to interact with AWS service endpoints. Ansible provides some basic modules for managing BIG-IP configuration resources. These along with libraries for similar tools can be found here: Ansible Puppet SaltStack In the rest of this post, I’ll discuss some work colleagues and I have done to automate BIG-IP deployments in AWS using Ansible. While we chose to use Ansible, we readily admit that Puppet, Chef, Salt and whatever else you use are all appropriate choices for implementing deployment and configuration management workflows for your network. Each have their upsides and downsides, and different tools may lend themselves to different use cases for your infrastructure. Browse the web to figure out which tool is right for you. Using Standardized BIG-IP Interfaces Speaking of APIs, for years F5 has provided the ability to programmatically configure BIG-IP using iControlSOAP. As the audiences performing automation work have matured, so have the weapons of choice. The new hot ticket is REST (Representational State Transfer), and guess what, BIG-IP has a REST interface (you can probably figure out what it is called). Together, iControlSOAP and iControlREST give you the power to manage nearly every configuration element and feature of BIG-IP. These interfaces become extremely powerful when you combine them with your favorite open-source configuration management tool and a cloud that allows you to spin up and down compute and networking resources. In the project described below, we have also made use of iApps using iControlRest as a way to create a standard virtual server configuration with the correct policies and profiles. The documentation in Github describes this in detail, but our approach shows how iApps provide a strongly supported approach for managing network policy across engineering teams. For example, imagine that a team of software engineers has written a framework to deploy applications. You can package the network policy into iApps for various types of apps, and pass these to the teams writing the deployment framework. Implementing a Service Catalog To pull the above concepts together, a colleague and I put together the aws-deployments project.The goal was to build a simple service catalog which would enable a user to deploy a containerized application in EC2 with BIG-IP network services sitting in front. This is example code that is not supported by F5 support but is a proof of concept to show how you can fully automate production-like deployments in AWS. Some highlights of the project include: Use of iControlRest and iControlSoap within Ansible playbooks to setup advanced topologies of BIG-IP in AWS. Automated deployment of a basic ASM web application firewall policy to protect a vulnerable web app (Hackazon. Use of iApps to manage virtual server configurations, including the WAF policy mentioned above. Figure 1 - Generic Architecture for automating application deployments in public or private cloud In examination of the code, you will see that we provide the opportunity to provision all the development models outlined in our earlier post (a single standalone VE, standalones BIG-IP VEs striped availability zones, clusters within an availability zone, etc). We used Ansible and the interfaces on BIG-IP to orchestrate the workflows assoiated with these deployment models. To perform the clustering step, we have used the iControlSoap interface on BIG-IP. The final set of technology used is depicted in Figure 3. Figure 2 - Technologies used in the aws-deployments project on Github Read the Code and Test It Yourself All the code I have mentioned is available at f5networks/aws-deployments. We encourage you to download and run the code for yourself. Instructions for setting up a development environment which includes the necessary dependencies is easy. We have packaged all the dependencies for use with either Vagrant or Docker as development tools. The instructions for either of these approaches can be found in the README.md or in the /docs directory. The following video shows an end-to-end usage example. (Keep in mind that the code has been updated since this video was produced). At the end of the day, our goal for this work was to collect customer feedback. Please provide some by leaving a comment below, or by filing ‘pull requests’ or ‘issues’ in Github. In the next few weeks, we will be updating the project to include the Hackazon app mentioned above, show how to cluster BIG-IP across availability zones, and how to deploy an ASM profile with an iApp. Have fun!
F5 in AWS Part 5 - Cloud-init, Single-NIC, and Auto Scale Out in BIG-IP
Updated for Current Versions and Documentation Part 1 : AWS Networking Basics Part 2: Running BIG-IP in an EC2 Virtual Private Cloud Part 3: Advanced Topologies and More on Highly-Available Services Part 4: Orchestrating BIG-IP Application Services with Open-Source Tools Part 5: Cloud-init, Single-NIC, and Auto Scale Out of BIG-IP in v12 The following article covers features and examplesin the 12.1 AWS Marketplace release, discussed in the following documentation: Amazon Web Services: Single NIC BIG-IP VE Amazon Web Services: Auto Scaling BIG-IP VE You can find the BIG-IP Hourly and BYOL releases in the Amazon marketplace here. BIG-IP utility billing images are available, which makes it a great time to talk about some of the functionality. So far in Chris’s series, we have discussed some of the highly-available deployment footprints of BIG-IP in AWS and how these might be orchestrated. Several of these footprints leverage BIG-IP's Device Service Clustering (DSC) technology for configuration management across devices and also lend themselves to multi-app or multi-tenant configurations in a shared-service model. But what if you want to deploy BIG-IP on a per-app or per-tenant basis, in a horizontally scalable footprint that plays well with the concepts of elasticity and immutability in cloud? Today we have just the option for you. Before highlighting these scalable deployment models in AWS, we need to cover cloud-init and single-NIC configurations; two important additions to BIG-IP that enable an Auto Scaling topology. Elasticity Elastiity is obviously one of the biggest promises/benefits of cloud. By leveraging cloud, we are essentially tapping into the "unlimited" (at least relative to our own datacenters) resources large cloud providers have built. In actual practice, this means adopting new methodologies and approaches to truely deliver this. Immutablity In traditional operational model of datacenters, everything was "actively" managed. Physical Infrastructure still tends to lend itself to active management but even virtualized services and applications running on top of the infrastructure were actively managed. For example, servers were patched, code was live upgraded inplace, etc. However, to achieve true elasticity, where things are spinning up or down and more ephemeral in nature, it required a new approach. Instead of trying to patch or upgrade individual instances, the approach was treating them as disposable which meant focusing more on the build process itself. ex. Netflix's famous Building with Legos approach. Yes, the concept of golden images/snapshots existed since virtualization but cloud, with self-service, automation and auto scale functionality, forced this to a new level. Operations focus shifted more towards a consistent and repeatable "build" or "packaging" effort, with final goal of creating instances that didn't need to be touched, logged into, etc. In the specific context of AWS's Auto Scale groups, that means modifying the Auto Scale Group's "launch config". The steps for creating the new instances involve either referencing an entirely new image ID or maybe modification to a cloud-init config. Cloud-init What is it? First, let’s talk about cloud-init as it is used with most Linux distributions. Most of you who are evaluating or operating in the cloud have heard of it. For those who haven’t, cloud-init is an industry standard for bootstrapping machines at startup. It provides a simple domain specific language for common infrastructure provisioning tasks. You can read the official docs here. For the average linux or systems engineer, cloud-init is used to perform tasks such as installing a custom package, updating yum repositories or installing certificates to perform final customizations to a "base" or “golden” image. For example, the Security team might create an approved hardened base image and various Dev teams would use cloud-init to customize the image so it booted up with an ‘identity’ if you will – an application server with an Apache webserver running or a database server with MySQL provisioned. Let’s start with the most basic "Hello World" use of cloud-init, passing in User Data (in this case a simple bash script). If launching an instance via the AWS Console, on the Configure Instance page, navigate down the “Advanced Details”: Figure 1: User Data input field - bash However, User Data is limited to < 16KBs and one of the real powers of cloud-init came from extending functionality past this boundry and providing a standardized way to provision, or ah humm, "initialize" instances.Instead of using unwieldy bash scripts that probed whether it was an Ubuntu or a Redhat instance and used this OS method or that OS method (ex. use apt-get vs. rpm) to install a package, configure users, dns settings, mount a drive, etc. you could pass a yaml file starting with #cloud-config file that did a lot of this heavy lifting for you. Figure 2: User Data input field - cloud-config Similar to one of the benefits of Chef, Puppet, Salt or Ansible, it provided a reliable OS or distribution abstraction but where those approaches require external orchestration to configure instances, this was internally orchestrated from the very first boot which was more condusive to the "immutable" workflow. NOTE: cloud-init also compliments and helps boot strap those tools for more advanced or sophisticated workflows (ex. installing Chef/Puppet to keep long running non-immutable services under operation/management and preventing configuration drift). This brings us to another important distinction. Cloud-init originated as a project from Canonical (Ubuntu) and was designed for general purpose OSs. The BIG-IP's OS (TMOS) however is a highly customized, hardened OS so most of the modules don't strictly apply. Much of the Big-IP's configuration is consumed via its APIs (TMSH, iControl REST, etc.) and stored in it's database MCPD. We can still achieve some of the benefits of having cloud-init but instead, we will mostly leverage the simple bash processor. So when Auto Scaling BIG-IPs, there are a couple of approaches. 1) Creating a custom image as described in the official documentation. 2) Providing a cloud-init configuration This is a little lighter weight approach in that it doesn't require the customization work above. 3) Using a combination of the two, creating a custom image and leveraging cloud-init. For example, you may create a custom image with ASM provisioned, SSL certs/keys installed, and use cloud-init to configure additional environment specific elements. Disclaimer: Packaging is an art, just look at the rise of Docker and new operating systems. Ideally, the more you bake into the image upfront, the more predictable it will be and faster it deploys. However, the less you build-in, the more flexible you can be. Things like installing libraries, compiling, etc. are usually worth building in the image upfront. However, the BIG-IP is already a hardened image and things like installing libraries is not something required or recommended so the task is more about addressing the last/lighter weight configuration steps. However, depending on your priorities and objectives,installing sensitive keying material, setting credentials, pre-provisioning modules, etc. might make good candidates for investing in building custom images. Using Cloud-init with CloudFormation templates Remember when we talked about how you could use CloudFormation templates in a previous post to setup BIG-IP in a standard way? Because the CloudFormation service by itself only gave us the ability to lay down the EC2/VPC infrastructure, we were still left with remaining 80% of the work to do; we needed an external agent or service (in our case Ansible) to configure the BIG-IP and application services. Now, with cloud-init on the BIG-IP (using version 12.0 or later), we can perform that last 80% for you. Using Cloud-init with BIG-IP As you can imagine, there’s quite a lot you can do with just that one simple bash script shown above. However, more interestingly, we also installed AWS’s Cloudformation Helper scripts. http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-helper-scripts-reference.html to help extend cloud-init and unlock a larger more powerful set of AWS functionality. So when used with Cloudformation, our User Data simply changes to executing the following AWS Cloudformation helper script instead. "UserData": { "Fn::Base64": { "Fn::Join": [ "", [ "#!/bin/bash\n", "/opt/aws/apitools/cfn-init-1.4-0.amzn1/bin/cfn-init -v -s ", { "Ref": "AWS::StackId" }, " -r ", "Bigip1Instance", " --region ", { "Ref": "AWS::Region" }, "\n" ] ] } } This allows us to do things like obtaining variables passed in from Cloudformation environment, grabbing various information from the metadata service, creating or downloading files, running particular sequence of commands, etc. so once BIG-IP has finishing running, our entire application delivery service is up and running. For more information, this page discusses how meta-data is attached to an instance using CloudFormation templates: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html#aws-resource-init-commands. Example BYOL and Utility CloudFormation Templates We’ve posted several examples on github to get you started. https://github.com/f5networks/f5-aws-cloudformation In just a few short clicks, you can have an entire BIG-IP deployment up and running. The two examples belowwill launch an entire reference stack complete with VPCs, Subnets, Routing Tables, sample webserver, etc. andshow the use of cloud-init to bootstrap a BIG-IP. Cloud-init is used to configure interfaces, Self-IPs, database variables, a simple virtual server, and in the case of of the BYOL instance, to license BIG-IP. Let’s take a closer look at the BIG-IP resource created in one of these to see what’s going on here: "Bigip1Instance ": { "Metadata ": { "AWS::CloudFormation::Init ": { "config ": { "files ": { "/tmp/firstrun.config ": { "content ": { "Fn::Join ": [ " ", [ "#!/bin/bash\n ", "HOSTNAME=`curl http://169.254.169.254/latest/meta-data/hostname`\n ", "TZ='UTC'\n ", "BIGIP_ADMIN_USERNAME=' ", { "Ref ": "BigipAdminUsername " }, "'\n ", "BIGIP_ADMIN_PASSWORD=' ", { "Ref ": "BigipAdminPassword " }, "'\n ", "MANAGEMENT_GUI_PORT=' ", { "Ref ": "BigipManagementGuiPort " }, "'\n ", "GATEWAY_MAC=`ifconfig eth0 | egrep HWaddr | awk '{print tolower($5)}'`\n ", "GATEWAY_CIDR_BLOCK=`curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/${GATEWAY_MAC}/subnet-ipv4-cidr-block`\n ", "GATEWAY_NET=${GATEWAY_CIDR_BLOCK%/*}\n ", "GATEWAY_PREFIX=${GATEWAY_CIDR_BLOCK#*/}\n ", "GATEWAY=`echo ${GATEWAY_NET} | awk -F. '{ print $1\ ".\ "$2\ ".\ "$3\ ".\ "$4+1 }'`\n ", "VPC_CIDR_BLOCK=`curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/${GATEWAY_MAC}/vpc-ipv4-cidr-block`\n ", "VPC_NET=${VPC_CIDR_BLOCK%/*}\n ", "VPC_PREFIX=${VPC_CIDR_BLOCK#*/}\n ", "NAME_SERVER=`echo ${VPC_NET} | awk -F. '{ print $1\ ".\ "$2\ ".\ "$3\ ".\ "$4+2 }'`\n ", "POOLMEM=' ", { "Fn::GetAtt ": [ "Webserver ", "PrivateIp " ] }, "'\n ", "POOLMEMPORT=80\n ", "APPNAME='demo-app-1'\n ", "VIRTUALSERVERPORT=80\n ", "CRT='default.crt'\n ", "KEY='default.key'\n " ] ] }, "group ": "root ", "mode ": "000755 ", "owner ": "root " }, "/tmp/firstrun.utils ": { "group ": "root ", "mode ": "000755 ", "owner ": "root ", "source ": "http://cdn.f5.com/product/templates/utils/firstrun.utils " } "/tmp/firstrun.sh ": { "content ": { "Fn::Join ": [ " ", [ "#!/bin/bash\n ", ". /tmp/firstrun.config\n ", ". /tmp/firstrun.utils\n ", "FILE=/tmp/firstrun.log\n ", "if [ ! -e $FILE ]\n ", " then\n ", " touch $FILE\n ", " nohup $0 0<&- &>/dev/null &\n ", " exit\n ", "fi\n ", "exec 1<&-\n ", "exec 2<&-\n ", "exec 1<>$FILE\n ", "exec 2>&1\n ", "date\n ", "checkF5Ready\n ", "echo 'starting tmsh config'\n ", "tmsh modify sys ntp timezone ${TZ}\n ", "tmsh modify sys ntp servers add { 0.pool.ntp.org 1.pool.ntp.org }\n ", "tmsh modify sys dns name-servers add { ${NAME_SERVER} }\n ", "tmsh modify sys global-settings gui-setup disabled\n ", "tmsh modify sys global-settings hostname ${HOSTNAME}\n ", "tmsh modify auth user admin password \ "'${BIGIP_ADMIN_PASSWORD}'\ "\n ", "tmsh save /sys config\n ", "tmsh modify sys httpd ssl-port ${MANAGEMENT_GUI_PORT}\n ", "tmsh modify net self-allow defaults add { tcp:${MANAGEMENT_GUI_PORT} }\n ", "if [[ \ "${MANAGEMENT_GUI_PORT}\ " != \ "443\ " ]]; then tmsh modify net self-allow defaults delete { tcp:443 }; fi \n ", "tmsh mv cm device bigip1 ${HOSTNAME}\n ", "tmsh save /sys config\n ", "checkStatusnoret\n ", "sleep 20 \n ", "tmsh save /sys config\n ", "tmsh create ltm pool ${APPNAME}-pool members add { ${POOLMEM}:${POOLMEMPORT} } monitor http\n ", "tmsh create ltm policy uri-routing-policy controls add { forwarding } requires add { http } strategy first-match legacy\n ", "tmsh modify ltm policy uri-routing-policy rules add { service1.example.com { conditions add { 0 { http-uri host values { service1.example.com } } } actions add { 0 { forward select pool ${APPNAME}-pool } } ordinal 1 } }\n ", "tmsh modify ltm policy uri-routing-policy rules add { service2.example.com { conditions add { 0 { http-uri host values { service2.example.com } } } actions add { 0 { forward select pool ${APPNAME}-pool } } ordinal 2 } }\n ", "tmsh modify ltm policy uri-routing-policy rules add { apiv2 { conditions add { 0 { http-uri path starts-with values { /apiv2 } } } actions add { 0 { forward select pool ${APPNAME}-pool } } ordinal 3 } }\n ", "tmsh create ltm virtual /Common/${APPNAME}-${VIRTUALSERVERPORT} { destination 0.0.0.0:${VIRTUALSERVERPORT} mask any ip-protocol tcp pool /Common/${APPNAME}-pool policies replace-all-with { uri-routing-policy { } } profiles replace-all-with { tcp { } http { } } source 0.0.0.0/0 source-address-translation { type automap } translate-address enabled translate-port enabled }\n ", "tmsh save /sys config\n ", "date\n ", "# typically want to remove firstrun.config after first boot\n ", "# rm /tmp/firstrun.config\n " ] ] }, "group ": "root ", "mode ": "000755 ", "owner ": "root " } }, "commands ": { "b-configure-Bigip ": { "command ": "/tmp/firstrun.sh\n " } } } } }, "Properties ": { "ImageId ": { "Fn::FindInMap ": [ "BigipRegionMap ", { "Ref ": "AWS::Region " }, { "Ref ": "BigipPerformanceType " } ] }, "InstanceType ": { "Ref ": "BigipInstanceType " }, "KeyName ": { "Ref ": "KeyName " }, "NetworkInterfaces ": [ { "Description ": "Public or External Interface ", "DeviceIndex ": "0 ", "NetworkInterfaceId ": { "Ref ": "Bigip1ExternalInterface " } } ], "Tags ": [ { "Key ": "Application ", "Value ": { "Ref ": "AWS::StackName " } }, { "Key ": "Name ", "Value ": { "Fn::Join ": [ " ", [ "BIG-IP: ", { "Ref ": "AWS::StackName " } ] ] } } ], "UserData ": { "Fn::Base64 ": { "Fn::Join ": [ " ", [ "#!/bin/bash\n ", "/opt/aws/apitools/cfn-init-1.4-0.amzn1/bin/cfn-init -v -s ", { "Ref ": "AWS::StackId " }, " -r ", "Bigip1Instance ", " --region ", { "Ref ": "AWS::Region " }, "\n " ] ] } } }, "Type ": "AWS::EC2::Instance " } Above may look like a lot at first but high level, we start by creating some files "inline" as well as “sourcing” some files from a remote location. /tmp/firstrun.config - Here we create a file inline, laying down variables from the Cloudformation Stack deployment itself and even the metadata service (http://169.254.169.254/latest/meta-data/). Take a look at the “Ref” stanzas. When this file is laid down on the BIG-IP disk itself, those variables will be interpolated and contain the actual contents. The idea here is to try to keep config and execution separate. /tmp/firstrun.utils – These are just some helper functions to help with initial provisioning. We use those to determine when the BIG-IP is ready for this particular configuration (ex. after a licensing or provisioning step). Note that instead of creating the file inline like the config file above, we simply “source” or download the file from a remote location. /tmp/firstrun.sh – This file is created inline as well and where it really all comes together. The first thing we do is load config variables from the firstrun.conf file and load the helper functions from firstrun.utils. We then create separate log file (/tmp/firstrun.log) to capture the output of this particular script. Capturing the output of these various commands just helps with debugging runs. Then we run a function called checkF5Ready (that we loaded from that helper function file) to make sure BIG-IP’s database is up and ready to accept a configuration. The rest may look more familiar and where most of the user customization takes place. We use variables from the config file to configure the BIG-IP using familiar methods like TMSH and iControl REST. Technically, you could lay down an entire config file (like SCF) and load it instead. We use tmsh here for simplicity. The possibilities are endless though. Disclaimer: the specific implementation above will certainly be optimized and evolve but the most important take away is we can now leverage cloud-init and AWS's helper libraries to help bootstrap the BIG-IP into a working configuration from the very first boot! Debugging Cloud-init What if something goes wrong? Where do you look for more information? The first place you might look is in various cloud-init logs in /var/log (cloud-init.log, cfn-init.log, cfn-wire.log): Below is an example for the CFTs below: [admin@ip-10-0-0-205:NO LICENSE:Standalone] log # tail -150 cfn-init.log 2016-01-11 10:47:59,353 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com 2016-01-11 10:47:59,353 [DEBUG] Describing resource BigipEc2Instance in stack arn:aws:cloudformation:us-east-1:452013943082:stack/as-testing-byol-bigip/07c962d0-b893-11e5-9174-500c217b4a62 2016-01-11 10:47:59,782 [DEBUG] Not setting a reboot trigger as scheduling support is not available 2016-01-11 10:47:59,790 [INFO] Running configSets: default 2016-01-11 10:47:59,791 [INFO] Running configSet default 2016-01-11 10:47:59,791 [INFO] Running config config 2016-01-11 10:47:59,792 [DEBUG] No packages specified 2016-01-11 10:47:59,792 [DEBUG] No groups specified 2016-01-11 10:47:59,792 [DEBUG] No users specified 2016-01-11 10:47:59,792 [DEBUG] Writing content to /tmp/firstrun.config 2016-01-11 10:47:59,792 [DEBUG] No mode specified for /tmp/firstrun.config 2016-01-11 10:47:59,793 [DEBUG] Writing content to /tmp/firstrun.sh 2016-01-11 10:47:59,793 [DEBUG] Setting mode for /tmp/firstrun.sh to 000755 2016-01-11 10:47:59,793 [DEBUG] Setting owner 0 and group 0 for /tmp/firstrun.sh 2016-01-11 10:47:59,793 [DEBUG] Running command b-configure-BigIP 2016-01-11 10:47:59,793 [DEBUG] No test for command b-configure-BigIP 2016-01-11 10:47:59,840 [INFO] Command b-configure-BigIP succeeded 2016-01-11 10:47:59,841 [DEBUG] Command b-configure-BigIP output: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 40 0 40 0 0 74211 0 --:--:-- --:--:-- --:--:-- 40000 2016-01-11 10:47:59,841 [DEBUG] No services specified 2016-01-11 10:47:59,844 [INFO] ConfigSets completed 2016-01-11 10:47:59,851 [DEBUG] Not clearing reboot trigger as scheduling support is not available [admin@ip-10-0-0-205:NO LICENSE:Standalone] log # If trying out the example templates above, you can inspect the various files mentioned. Ex. In addition to checking for their general presence: /tmp/firstrun.config= make sure variables were passed as you expected. /tmp/firstrun.utils= Make sure exists and was downloaded /tmp/firstrun.log = See if any obvious errors were outputted. It may also be worth checking AWS Cloudformation Console to make sure you passed the parameters you were expecting. Single-NIC Another one of the important building blocks introduced with 12.0 on AWS and Azure Virtual Editions is the ability to run BIG-IP with just a single network interface. Typically, BIG-IPs were deployed in a multi-interface model, where interfaces were attached to an out-of-band management network and one or more traffic (or "data-plane") networks. But, as we know, cloud architectures scale by requiring simplicity, especially at the network level. To this day, some clouds can only support instances with a single IP on a single NIC. In AWS’s case, although they do support multiple NIC/multiple IP, some of their services like ELB only point to first IP address of the first NIC. So this Single-NIC configuration makes it not only possible but also dramatically easier to deploy in these various architectures. How this works: We can now attach just one interface to the instance and BIG-IP will start up, recognize this, use DHCP to configure the necessary settings on that interface. Underneath the hood, the following DB keys will be set: admin@(ip-10-0-1-65)(cfg-sync Standalone)(Active)(/Common)(tmos)# list sys db provision.1nic one-line sys db provision.1nic { value "enable" } admin@(ip-10-0-1-65)(cfg-sync Standalone)(Active)(/Common)(tmos)# list sys db provision.1nicautoconfig one-line sys db provision.1nicautoconfig { value "enable" } provision.1nic = allows both management and data-plane to use the same interface provision.1nicautoconfig = uses address from DHCP to configure a vlan, Self-IP and default gateway. Ex. network objects automatically configured admin@(ip-10-0-1-65)(cfg-sync Standalone)(Active)(/Common)(tmos)# list net vlan net vlan internal if-index 112 interfaces { 1.0 { } } tag 4094 } admin@(ip-10-0-1-65)(cfg-sync Standalone)(Active)(/Common)(tmos)# list net self net self self_1nic { address 10.0.1.65/24 allow-service { default } traffic-group traffic-group-local-only vlan internal } admin@(ip-10-0-1-65)(cfg-sync Standalone)(Active)(/Common)(tmos)# list net route net route default { gw 10.0.1.1 network default } Note: Traffic Management Shell and the Configuration Utility (GUI) are still available on ports 22 and 443 respectively. If you want to run the management GUI on a higher port (for instance if you don’t have the BIG-IPs behind a Port Address Translation service (like ELB) and want to run an HTTPS virtual on 443), use the following commands: tmsh modify sys httpd ssl-port 8443 tmsh modify net self-allow defaults add { tcp:8443 } tmsh modify net self-allow defaults delete { tcp:443 } WARNING: Now that management and dataplane run on the same interface, make sure to modify your Security Groups to restrict access to SSH and whatever port you use for the Mgmt GUI port to trusted networks. UPDATE: On Single-Nic, Device Service Clustering currently only supports Configuration Syncing (Network Failover is restricted for now due to BZ-606032). In general, the single-NIC model lends itself better to single-tenant or per-app deployments, where you need the advanced services from BIG-IP like content routing policies, iRules scripting, WAF, etc. but don’t necessarily care for maintaining a management subnet in the deployment and are just optimizing or securing a single application. By single tenant we also mean single management domain as you're typically running everything through a single wildcard virtual (ex. 0.0.0.0/0, 0.0.0.0/80, 0.0.0.0/443, etc.) vs. giving each tenant its own Virtual Server (usually with its own IP and configuration) to manage. However, you can still technically run multiple applications behind this virtual with a policy or iRule, where you make traffic steering decisions based on L4-L7 content (SNI, hostname headers, URIs, etc.). In addition, if the BIG-IPs are sitting behind a Port Address Translation service, it also possible to stack virtual services on ports instead. Ex. 0.0.0.0:444 = Virtual Service 1 0.0.0.0:445 = Virtual Service 2 0.0.0.0:446 = Virtual Service 3 We’ll let you get creative here…. BIG-IP Auto Scale Finally, the last component of Auto Scaling BIG-IPs involves building scaling policies via Cloudwatch Alarms. In addition to the built-in EC2 metrics in CloudWatch, BIG-IP can report its own set of metrics, shown below: Figure 3: Cloudwatch metrics to scale BIG-IPs based on traffic load. This can be configured with the following TMSH commands on any version 12.0 or later build: tmsh modify sys autoscale-group autoscale-group-id ${BIGIP_ASG_NAME} tmsh load sys config merge file /usr/share/aws/metrics/aws-cloudwatch-icall-metrics-config These commands tell BIG-IP to push the above metrics to a custom “Namespace” on which we can roll up data via standard aggregation functions (min, max, average, sum). This namespace (based on Auto Scale group name) will appear as a row under in the “Custom metrics” dropdown on the left side-bar in CloudWatch (left side of Figure 1). Once these BIG-IP or EC2 metrics have been populated, CloudWatch alarms can be built, and these alarms are available for use in Auto Scaling policies for the BIG-IP Auto Scale group. (Amazon provides some nice instructions here). Auto Scaling Pool Members and Service Discovery If you are scaling your ADC tier, you are likely also going to be scaling your application as well. There are two options for discovering pool members in your application's auto scale group. 1) FQDN Nodes For example, in a typicalsandwichdeployment, your application members might also be sitting behind an internal ELB so you would simply point your FQDN node at the ELB's DNS. For more information, please see: https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm-implementations-12-1-0/25.html?sr=56133323 2) BIG-IP's AWS Auto Scale Pool Member Discovery feature (introduced v12.0) This feature polls the Auto Scale Group via API calls and populates the pool based on its membership. For more information, please see: https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-ve-autoscaling-amazon-ec2-12-1-0/2.html Putting it all together The high-level steps for Auto Scaling BIG-IP include the following: Optionally* creating an ElasticLoadBalancer group which will direct traffic to BIG-IPs in your Auto Scale group once they become operational. Otherwise, will need Global Server Load Balancing (GSLB). Creating a launch configuration in EC2 (referencing either custom image id and/or using Cloud-init scripts as described above) Creating an Auto Scale group using this launch configuration Creating CloudWatch alarms using the EC2 or custom metrics reported by BIG-IP. Creating scaling policies for your BIG-IP Auto Scale group using the alarms above. You will want to create both scale up and scale down policies. Here are some things to keep in mind when Auto Scaling BIG-IP ● BIG-IP must run in a single-interface configuration with a wildcard listener (as we talked about earlier). This is required because we don't know what IP address the BIG-IP will get. ● Auto Scale Groups themselves consist of utility instances of BIG-IP ● The scale up time for BIG-IP is about 12-20 minutes depending on what is configured or provisioned. While this may seem like a long-time, creating the right scaling policies (polling intervals, thresholds, unit of scale) make this a non-issue. ● This deployment model lends itself toward the themes of stateless, horizontal scalability and immutability embraced by the cloud. Currently, the config on each device is updated once and only once at device startup. The only way to change the config will be through the creation of a new image or modification to the launch configuration. Stay tuned for a clustered deployment which is managed in a more traditional operational approach. If interesting in exploring Auto Scale further, we have put together some examples of scaling the BIG-IP tier here: https://github.com/f5devcentral/f5-aws-autoscale * Above github repository also provides some examples of incorporating BYOL instances in the deployment to help leverage BYOL instances for your static load and using Auto Scale instances for your dynamic load. See the READMEs for more information. CloudFormation templates with Auto Scaled BIG-IP and Application Need some ideas on how you might leverage these solutions? Now that you can completely deploy a solution with a single template, how about building service catalog items for your business units that deploy different BIG-IP services (LB, WAF) or a managed service build on top of AWS that can be easily deployed each time you on-board a new customer?
Successfully Deploy Your Application in the AWS Public Cloud: Part 1 of 4
In this series of articles, we're going to walk you through a fairly typical lift-and-shiftdeployment of BIG-IP in AWS, so that: If you’re just starting, you can get an idea of what lies ahead. If you’re already working in the cloud, you can get familiar with a variety of F5 solutions that will help your application and organization be successful. The scenario we've chosen is pretty typical: we have an application in our data center and we want to move it to the public cloud. As part of this move, we want to give development teams access to an agile environment, and we want to ensure that NetOps/SecOps maintains the stability and control they expect. Here is a simple diagram for a starting point. We’re a business that sells our products on the web. Specifically, we sell a bunch of random picnic supplies and candies. Our hot seller this summer is hotdog-flavored lemonade, something you might think is appalling but that really encompasses everything great about a picnic. But back to the scenario: We have a data center, where we have two physical BIG-IPs that function as a web application firewall (WAF), and they load balance traffic securely to three application servers. These application servers get their product information from a product database. Our warehouse uses a separate internal application to manage inventory, and that inventory is stored in an inventory database. In this series of articles, we’ll show you how to move the application to Amazon Web Services (AWS), and discuss the trade-offs that come at different stages in the process. So let’s get started. The challenge Move to the cloud; keep environments in sync The solution Use a CloudFormation Template (CFT) to create a repeatable cloud deployment We’ve been told to move to the cloud, and after a thorough investigation of the options, have decided to move our picnic-supply-selling app to Amazon Web Services. Our organization has several different environments that we maintain. Dev (one environment per developer) Test UAT Performance Production These environments can tend to be out of sync with one another. This frustrates everyone. And when we deploy the app to production, we often see unexpected results. If possible, we don’t want to bring this problem along to the cloud. We want to deploy our application to all of these environments and have the result be the same every time. Even if each developer has different code, all developers should be working in an infrastructure environment that matches all other environments, most importantly, production. Enter the AWS CloudFormation template. We can create a template and use it to consistently spin up the same environment. If we require a change, we can make the modification and save a new version of the CFT, letting everyone on the team know about the change. And it’s version-controlled, so we can always roll back if we mess up. So we use a CFT to create our application servers and deploy the latest code on them. In our scenario, we create an AWS Elastic Load Balancer so we can continue load balancing to the application servers. Our product data has a dependency on inventory data that comes from the warehouse, and we use BIG-IP for authentication (among other things). We use our on-premise BIG-IPs to create an IPSEC VPN tunnel to AWS. This way, our application can maintain a connection to the inventory system. When we get the CFT working the way we want, we can swing the DNS to point to these new AWS instances. Details about the CloudFormation template We put a CFT on github that you can deploy to demonstrate the AWS part of this setup. It may help you visualize this deployment, and in part 2 of this series, we'll be expanding on this initial setup. If you'd like, you can deploy by clicking the following button. Ensure that when you're in the AWS console, you select the region where you want to deploy. And if you're really new to this, just remember that active instances cost money. The CFT creates four Windows servers behind an AWS Elastic Load Balancer (ELB). Three of the servers are running a web app and one is used for the database. Beware, the website is a bit goofy and we were feeling punchy when we created it. Here is a brief explanation of what specific sections of the CFT do. Parameters The Parameters section includes fields you must populate when deploying the CFT. In this case, you’ll have to specify a name for your servers, and the AMI (Amazon Machine Image) ID to build the servers from. In the template, you can see what parameters look like. For example, the field where you enter the AMI ID: "WindowsAMI": { "Description": "Windows Version and Region AMI", "Type": "String", } To find the ID of the AMI you want to use, look in the marketplace, find the product you want, click the Manual Launch tab, and note the AMI ID for the region where you’re going to deploy. We are using Microsoft Windows Server 2016 Base and Microsoft Windows Server 2016 with MSSQL 2016. Note: These IDs can change; check the AWS Marketplace for the latest AMI IDs. Resources The resources section of the CFT performs the legwork. The CFT creates a Virtual Private Cloud (VPC) with three subnets so that the application is redundant across availability zones. It creates a Windows Server instance in each availability zone, and it creates an AWS Elastic Load Balancer (ELB) in front of the application servers. Code that creates the load balancer: "StackELB01": { "Type": "AWS::ElasticLoadBalancing::LoadBalancer", "Properties": { "Subnets" : [ { "Ref": "StackSubnet1" }, { "Ref": "StackSubnet2" }, { "Ref": "StackSubnet3" } ], "Instances": [ { "Ref": "WindowsInstance1" }, { "Ref": "WindowsInstance2" }, { "Ref": "WindowsInstance3" } ], "Listeners": [ { "LoadBalancerPort": "80", "InstancePort": "80", "Protocol": "HTTP" } ], "HealthCheck": { "Target": "HTTP:80/", "HealthyThreshold": "3", "UnhealthyThreshold": "5", "Interval": "30", "Timeout": "5" }, "SecurityGroups":[ { "Ref": "ELBSecurityGroup" } ] } Then the CFT uses Cloud-Init to configure the Windows machines. It installs IIS on each machine, sets the hostname, and creates an index.html file that contains the server name (so that when you load balance to each machine, you will be able to determine which app server is serving the traffic). It also adds your user to the machine’s local Administrators group. Note: This is just part of the code. Look at the CFT itself for details. "install_IIS": { "files": { "C:\\Users\\Administrator\\Downloads\\firstrun.ps1": { "content": { "Fn::Join": [ "", [ "param ( \n", " [string]$password,\n", " [string]$username,\n", " [string]$servername\n", ")\n", "\n", "Add-Type -AssemblyName System.IO.Compression.FileSystem\n", "\n", "## Create user and add to Administrators group\n", "$pass = ConvertTo-SecureString $password -AsPlainText -Force\n", "New-LocalUser -Name $username -Password $pass -PasswordNeverExpires\n", "Add-LocalGroupMember -Group \"Administrators\" -Member $username\n", The CFT then calls PowerShell to run the script. "commands": { "b-configure": { "command": { "Fn::Join": [ " ", [ "powershell.exe -ExecutionPolicy unrestricted C:\\Users\\Administrator\\Downloads\\firstrun.ps1", { "Ref": "adminPassword" }, { "Ref": "adminUsername"}, { "Ref": "WindowsName1"}, "\n" ] Finally, this section includes signaling. You can use the Cloud-Init cfn-signal helper script to pause the stack until resource creation is complete. For more information, see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-signal.html. Sample of signaling: "WindowsInstance1WaitHandle": { "Type": "AWS::CloudFormation::WaitConditionHandle" }, "WindowsInstance1WaitCondition": { "Type": "AWS::CloudFormation::WaitCondition", "DependsOn": "WindowsInstance1", "Properties": { "Handle": { "Ref": "WindowsInstance1WaitHandle" }, "Timeout": "1200" } } Outputs The output includes the URL of the AWS ELB, which you use to connect to your applications. "Outputs": { "ServerURL": { "Description": "The AWS Generated URL.", "Value": { "Fn::Join": [ "", [ "http://", { "Fn::GetAtt": [ "StackELB01", "DNSName" ] } This output is displayed in the AWS console, on the Outputs tab. You can use the link to quickly connect to the ELB. When we're done deploying the app and we’ve fully tested it in AWS, we can swing the DNS from the internal address to the AWS load balancer, and we’re up and running. Come back next week as we implement BIG-IP VE for security in AWS.Onboarding F5 in Cloud Part 1 - Startup Scripts
In a previous article, we discussed how we leveraged cloud-initto pass startup scripts to BIG-IP Virtual Editions (VE)in AWS. Now that v13 is here, letstake this multi-cloud! As you can guess, youcan pass startup scripts inother environmentsas well. The input namesare a *little*different, because, of course, everyone has to be different,but no worries, the effect is still the same. To see what this looks like, lets take a look at this through the lens of some popular orchestration tools. Let's start withTerraform. Terraform is a popular multi-cloud tool. Much like Cloudformation, it aims to provide a declaritive approach to provisioning your entire stack, everything from network to compute to application services. It does this by having it's own state engine and providinga simple yet elegant DSL known as Hashicorp Configuration Language (HCL). Openstack: In Openstack, that's easy. Starting with BIG-IP v13, our images have cloud-init installed aswell and youusethe exact same"user_data" field to pass that startup script. https://www.terraform.io/docs/providers/openstack/r/compute_instance_v2.html#instance-with-user-data-cloud-init- The only thing different is you need toenable something called "config_drive" because openstack places the data on a drive mounted to the instance and cloud-init looks for the user_data there vs. grabbing it over the network. Google Compute Engine (GCE): In GCE, we've installed a bit of custom code on the BIG-IPs to leveragethe obscurely named "startup_script" parameter 🙂 Kudos to GCE to keeping things nice and simple here! https://www.terraform.io/docs/providers/google/r/compute_instance.html#metadata_startup_script However, they did add a tiny twist.You can also pass yourstartup script by placing it ina metadata key with the name "startup_script" so depending on how you pass it, you get a different behavior. If youpass it via a metadata key, any changes cause thescript to bererun, if you pass it via instance parameter, any changes cause theinstance to be recreated. Microsoft Azure: Not necessarily new in v13 but In Azure, we've installed their version of cloud-init called wa-agent and mostly leverage something called the "custom-script" virtual_machine_extension. https://www.terraform.io/docs/providers/azurerm/r/virtual_machine_extension.html Underneath the hood, wa-agentparses this inputa little differently,through a "commandToExecute" field, but the effect is similar.They also have another parameter called custom_data to let you pass a single file like in cloudsabove which we will leverage for consistency. https://www.terraform.io/docs/providers/azurerm/r/virtual_machine.html#custom_data For comparison, let's take a look at Ansible. In this article, we discussed leveraging Ansible to deployand configure BIG-IPs in AWS. However, in that, we onboarded andconfigured Big-IPs externally using REST API calls. However, if we wanted to use start up scripts instead, for instance doing Auto scaling, lets see if we can find the same parameters in Ansible's modules/defintions. user_data: ec2 instance:https://docs.ansible.com/ansible/latest/ec2_module.html Auto Scale Launch Config: http://docs.ansible.com/ansible/latest/ec2_lc_module.html userdata:https://docs.ansible.com/ansible/os_server_module.html startup_script:https://docs.ansible.com/ansible/gce_module.html Now, let's take a look at an actual BIG-IP startup script example. In this Terraform template, https://github.com/f5devcentral/f5-terraform/blob/master/modules/providers/aws/infrastructure/proxy/standalone/1nic/byol/bigip.tf you can see in the instance definition, resource "aws_instance" "bigip" { ami = "${lookup(var.amis, var.region)}" instance_type = "${var.instance_type}" associate_public_ip_address = "${var.create_management_public_ip}" availability_zone = "${var.availability_zone}" subnet_id = "${var.subnet_id}" vpc_security_group_ids = ["${aws_security_group.sg.id}"] iam_instance_profile = "${aws_iam_instance_profile.proxy_service_discovery_profile.name}" key_name = "${var.ssh_key_name}" root_block_device { delete_on_termination = true } tags { Name = "${var.environment}-proxy" environment = "${var.environment}" owner = "${var.owner}" group = "${var.group}" costcenter = "${var.costcenter}" application = "${var.application}" } user_data = "${data.template_file.user_data.rendered}" } the user_data fieldpoints to the"rendered" output of a template file (user_data.tpl). https://github.com/f5devcentral/f5-terraform/blob/master/modules/providers/aws/infrastructure/proxy/standalone/1nic/byol/user_data.tpl NOTE: The user_data can be a static string or file but these orchestration tools provide a handy templating mechanism so we can make this input dynamic based on the inputs or outputs of objects created. In Terraform's case, it interpolates a "${variable}" into whatever variable you pass it via the "vars" parameter. Here is "template_file" objectwe pass the variables in. data "template_file" "user_data" { template = "${file("${path.module}/user_data.tpl")}" vars { admin_username = "${var.admin_username}" admin_password = "${var.admin_password}" management_gui_port = "${var.management_gui_port}" dns_server = "${var.dns_server}" ntp_server = "${var.ntp_server}" timezone = "${var.timezone}" region = "${var.region}" application = "${var.application}" vs_dns_name = "${var.vs_dns_name}" vs_address = "${var.vs_address}" vs_mask = "${var.vs_mask}" vs_port = "${var.vs_port}" pool_member_port = "${var.pool_member_port}" pool_name = "${var.pool_name}" pool_tag_key = "${var.pool_tag_key}" pool_tag_value = "${var.pool_tag_value}" site_ssl_cert = "${var.site_ssl_cert}" site_ssl_key = "${var.site_ssl_key}" license_key = "${var.license_key}" } } NOTE:The ${variable}syntax also happens to bebash variable syntax so if you're generating a bash script like we are and want the end result to remain a bash variable, you need to escape it with another "$", ex. $${my_variable} 🙂 Ansible leverages common jinja templating so your user_data template would have {{ my_variable }} instead. Now, lets lookat input scripts in other environments: Openstack:https://github.com/f5devcentral/f5-terraform/blob/master/modules/providers/openstack/infrastructure/proxy/standalone/1nic/byol/user_data.tpl GCE:https://github.com/f5devcentral/f5-terraform/blob/master/modules/providers/gce/infrastructure/proxy/standalone/1nic/byol/user_data.tpl Azure:https://github.com/f5devcentral/f5-terraform/blob/master/modules/providers/azure/infrastructure/proxy/standalone/1nic/byol/user_data.tpl Notice the pattern here. We can reuse almost the same startup script save for a few small variations. We leverage these sameinstance customizationparametersin the various platform's native orchestration tools like AWS Cloudformation, Azure ARM templates, Google Deployment templates and Openstack HEAT templates. For more examples of leveraging these inputs, see our official templates which leveragemore best practices, intergrations, etc. https://github.com/F5Networks/f5-aws-cloudformation https://github.com/F5Networks/f5-azure-arm-templates https://github.com/F5Networks/f5-google-gdm-templates https://github.com/F5Networks/f5-openstack-hot Like the Terraform templating mechanism or Ansible's jinja templating, these native tools are able to dynamically generate the payloads from input parameters and outputs of objects created as well. And, as you may want to customize the deployments somemore (ex. torename something to your conventions, download or configure some other policy, etc.) we try to point you to a safe section to customize. For instance, search for section to modify by searching for "### START". ex. ### START CUSTOM add your customizations here. ### END CUSTOM You can even deploythese templates through other orchestration tools. https://plugins.jenkins.io/jenkins-cloudformation-plugin http://docs.ansible.com/ansible/latest/cloudformation_module.html and amazingly through other template based tools. https://www.terraform.io/docs/providers/aws/r/cloudformation_stack.html ex. Deploying a cloudformation template through terraform. https://github.com/f5devcentral/f5-terraform/blob/master/modules/providers/aws/infrastructure/proxy/autoscale/1nic-cft/util/bigip.tf By leveraging the same toolingused to deploy your apps, you can begin to make deployments both quicker and repeatable. Onboarding F5 in Cloud Part 2: Service Discovery
