What is HTTP Part VIII - Compression and Caching
In the last article of this What is HTTP? series we covered the nuances of OneConnect on HTTP traffic through the BIG-IP. In this article, we’ll cover caching and compression. We’ll deal with compression first, and then move on to caching. Compression In the very early days of the internet, much of the content was text based. This meant that the majority of resources were very small in nature. As popularity grew, the desire for more rich content filled with images grew as well, and resource sized began to explode. What had not yet exploded yet, however, was the bandwidth available to handle all that rich content (and you could argue that’s still the case in mobile and remote terrestrial networks as well.) This intersection of more resources without more bandwidth led to HTTP development in a few different areas: Methods for getting or sending partial resources Methods for identifying if resources needed to be retrieved at all Methods for reducing resources during transit that could be successfully reproduced after receipt The various rangeheaders were developed to handle the first case, caching, which we will discuss later in this article, was developed to handle the second case, and compression was developed to handle the third case. The basic definition of data compression is simply reducing the bits necessary to accurately represent the resource. This is done not only to save network bandwidth, but also on storage devices to save space. And of course money in both areas as well. In HTTP/1.0, end-to-end compression was possible, but not hop-by-hop as it does not have a distinguishing mechanism between the two. That is addressed in HTTP/1.1, so intermediaries can use complex algorithms unknown to the server or client to compress data between them and translate accordingly when speaking to the clients and servers respectively. In 11.x forward, compression is managed in its own profile. Prior to 11.x, it was included in the http profile. The httpcompression profile overview on AskF5 is very thorough, so I won’t repeat that information here, but you will want to pay attention to the compression level if you are using gzip (default.) The default of level 1 is fast from the perspective of the act of compressing on BIG-IP, but having done minimal compressing, reaps the least amount of benefit on the wire. If a particular application has great need for less bandwidth utilization toward the clientside of the network footprint, bumping up to level 6 will increase the reduction in bandwidth without overly taxing the BIG-IP to perform the operation. Also, it’s best to avoid compressing data that has already been compressed, like images and pdfs. Compressing them actually makes the resource larger, and wastes BIG-IP resources doing it! SVG format would be an exception to that rule. Also, don’t compress small files. The profile default is 1M for minimum content length. For BIG-IP hardware platforms, compression can be performed in hardware to offload that function. There is a database variable that you can configure to select the data compression strategy via sys modify db compression.strategy . The default value is latency, but there are four other strategies you can employ as covered in the manual. Caching Web caching could (and probably should) be its own multi-part series. The complexities are numerous, and the details plentiful. We did a series called Project Acceleration that covered some of the TCP optimization and compression topics, as well as the larger product we used to call Web Accelerator but is now the Application Acceleration Manager or AAM. AAM is caching and application optimization on steroids and we are not going to dive that deep here. We are going to focus specifically on HTTP caching and how the default functionality of the ramcache works on the BIG-IP. Consider the situation where there is no caching: In this scenario, every request from the browser communicates with the web server, no matter how infrequently the content changes. This is a wasteful use of resources on the server, the network, and even the client itself. The most important resource to our short attention span end users is time! The more objects and distance from the server, the longer the end user waits for that page to render. One way to help is to allow local caching in the browser: This way, the first request will hit the web server, and repeat requests for that same resource will be pulled from the cache (assuming the resource is still valid, more on that below.) Finally, there is the intermediary cache. This can live immediately in front of the end users like in an enterprise LAN, in a content distribution network, immediately in front of the servers in a datacenter, or all of the above! In this case, the browser1 client requests an object not yet in the cache serving all the browser clients shown. Once the cache has the object from the server, it will serve it to all the browser clients, which offloads the requests to server, saves the time in doing so, and brings the response closer to the browser clients as well. Given the benefits of a caching solution, let’s talk briefly of the risks. If you take the control of what’s served away from the server and put it in the hands of an intermediary, especially an intermediary the administrators of the origin server might not have authority over, how do you control then what content the browsers ultimately are loading? That’s where the HTTP standards on caching control come into play. HTTP/1.0 introduced the Pragma, If-Modified-Since, Last-Modified, and Expires headers for cache control. The Cache-Control and ETag headers along with a slew of “If-“ conditional headers were introduced in HTTP/1.1, but you will see many of the HTTP/1.0 cache headers in responses alongside the HTTP/1.1 headers for backwards compatibility. Rather than try to cover the breadth of caching here, I’ll leave it to the reader to dig into the quite good resources linked at the bottom (start with "Things Caches Do") for detailed understanding. However, there's a lot to glean from your browser developer tools and tools like Fiddler and HttpWatch. Consider this request from my browser for the globe-sm.svg file on f5.com. Near the bottom of the image, I’ve highlighted the request Cache-Control header, which has a value of no-cache. This isn’t a very intuitive name, but what the client is directing the cache is that it must submit the request to the origin server every time, even if the content is fresh. This assures authentication is respected while still allowing for the cache to be utilized for content delivery. In the response, the Cache-Control header has two values: public and max-age. The max-age here is quite large, so this is obviously an asset that is not expected to change much. The public directive means the resource can be stored in a shared cache. Now that we have a basic idea what caching is, how does the BIG-IP handle it? The basic caching available in LTM is handled in the same profile that AAM uses, but there are some features missing when AAM is not provisioned. It used to be called ramcache, but now is the webacceleration profile. Solution K14903 provides the overview of the webacceleration profile but we’ll discuss the cache size briefly. Unlike the Web Accelerator, there is no disk associated with the ramcache. As the name implies, this is “hot” cache in memory. So if you are memory limited on your BIG-IP, 100MB might be a little too large to keep locally. Managing the items in cache can be done via the tmsh command line with the ltm profile ramcache command. tmsh show/delete operations can be used against this method. An example show on my local test box: root@(ltm3)(cfg-sync Standalone)(Active)(/Common)(tmos)# show ltm profile ramcache webacceleration Ltm::Ramcaches /Common/webacceleration Host: 192.168.102.62 URI : / -------------------------------------- Source Slot/TMM 1/1 Owner Slot/TMM 1/1 Rank 1 Size (bytes) 3545 Hits 5 Received 2017-11-30 22:16:47 Last Sent 2017-11-30 22:56:33 Expires 2017-11-30 23:16:47 Vary Type encoding Vary Count 1 Vary User Agent none Vary Encoding gzip,deflate Again, if you have AAM licensed, you can provision it and then additional fields will be shown in the webacceleration profile above to allow for an acceleration policy to be applied against your virtual server. Resources RFC 2616 - The standard fine print. Things Caches Do- Excellent napkin diagrams that provide simple explanations of caching operations. Caching Tutorial - Comprehensive walk through of caching. HTTP Caching - Brief but informative look at caching from a webdev perspective. HTTP Caching - Google develops page with examples, flowcharts, and advice on caching strategies. Project Acceleration - Our 10 part series on web acceleration technology available on the BIG-IP platform in LTM and/or AAM modules. Solution K5157 - BIG-IP caching and the Vary header Make Your Cache Work For You - Article by Dawn Parzych here on DevCentral on tuning techniquesDNS Caching
I've been writing some DNS articles over the past couple of months, and I wanted to keep the momentum going with a discussion on DNS Caching. As a reminder, my first five articles are: Let's Talk DNS on DevCentral DNS The F5 Way: A Paradigm Shift DNS Express and Zone Transfers The BIG-IP GTM: Configuring DNSSEC DNS on the BIG-IP: IPv6 to IPv4 Translation We all know that caching improves response time and allows for a better user experience, and the good news is that the BIG-IP is the best in the business when it comes to caching DNS requests. When a user requests access to a website, it's obviously faster if the DNS response comes directly from the cache on a nearby machine rather than waiting for a recursive lookup process that spans multiple servers. The BIG-IP is specifically designed to quickly and efficiently respond to DNS requests directly from cache. This cuts down on administrative overhead and provides a better and faster experience for your users. There are three different types of DNS caches on the BIG-IP: Transparent, Resolver, and Validating Resolver. In order to create a new cache, navigate to DNS >> Caches >> Cache List and create a new cache (I'm using v11.5). Let's check out the details of each one! Transparent Resolver When the BIG-IP is configured with a transparent cache, it uses external DNS resolvers to resolve DNS queries and then cache the responses from the resolvers. Then, the next time the BIG-IP receives a query for a response that exists in cache, it immediately returns the response to the user. Transparent cache responses contain messages and resource records. The diagram below shows a transparent cache scenario where the BIG-IP does not yet have the response to a DNS query in its cache. In this example, the user sends a DNS query, but because the BIG-IP does not have a response cached, it transparently forwards the query to the appropriate external DNS resolver. When the BIG-IP receives the response from the resolver, it caches the response for future queries. A big benefit of transparent caching is that it consolidates content that would otherwise be cached across many different external resolvers. This consolidated cache approach produces a much higher cache hit percentage for your users. The following screenshot shows the configuration options for setting up a transparent cache. Notice that when you select the "Transparent" Resolver Type, you simply configure a few DNS Cache settings and you're done! The Message Cache Size (listed in bytes) is the maximum size of the message cache, and the Resource Record Cache Size (also listed in bytes) is the maximum size of the resource record cache. Pretty straightforward stuff. The "Answer Default Zones" setting is not enabled by default; meaning, it will pass along DNS queries for the default zones. When enabled, the BIG-IP will answer DNS queries for the following default zones: localhost, reverse 127.0.0.1 and ::1, and AS112. The "Add DNSSEC OK Bit On Miss" setting is enabled by default., and simply means that the BIG-IP will add the DNSSEC OK bit when it forwards DNS queries. Adding this bit indicates to the server that the resolver is able to accept DNSSEC security resource records. Resolver Whereas the Transparent cache will forward the DNS query to an external resolver, the "Resolver" cache will actually resolve the DNS queries and cache the responses. The Resolver cache contains messages, resource records, and the nameservers that the BIG-IP queries to resolve DNS queries. The screenshot below shows the configuration options for setting up a Resolver cache. When you select the Resolver cache, you will need to select a Route Domain Name from the dropdown list. This specifies the route domain that the resolver uses for outbound traffic. The Message Cache Size and Resource Record Cache Size are the same settings as in Transparent cache. The Name Server Cache Count (listed in entries) is the maximum number of DNS nameservers on which the BIG-IP will cache connection and capability data. The Answer Default Zones is the same setting as described above for the Transparent cache. As expected, the Resolver cache has several DNS Resolver settings. The "Use IPv4, IPv6, UDP, and TCP" checkboxes are fairly straightforward. They are all enabled by default, and they simply specify whether the BIG-IP will answer and issue queries in those specific formats. The "Max Concurrent UDP and TCP Flows" specifies the maximum number of concurrent sessions the BIG-IP supports. The "Max Concurrent Queries" is similar in that it specifies the maximum number of concurrent queries used by the resolver. The "Unsolicited Reply Threshold" specifies the number of replies to DNS queries the BIG-IP will support before generating an SNMP trap and log message as an alert to a potential attack. DNS cache poisoning and other Denial of Service attacks will sometimes use unsolicited replies as part of their attack vectors. The "Allowed Query Time" is listed in milliseconds and specifies the time the BIG-IP will allow a query to remain in the queue before replacing it with a new query when the number of concurrent distinct queries exceeds the limit listed in the "Max Concurrent Queries" setting (discussed above). The "Randomize Query Character Case" is enabled by default and will force the BIG-IP to randomize the character case (upper/lower) in domain name queries issued to the root DNS servers. Finally, the "Root Hints" setting specifies the host information needed to resolve names outside the authoritative DNS domains. Simply input IP addresses of the root DNS servers and hit the "add" button. Validating Resolver The Validating Resolver cache takes things to the next level by recursively querying public DNS servers and validating the identity of the responding server before caching the response. The Validating Resolver uses DNSSEC to validate the responses (which mitigates DNS attacks like cache poisoning). For more on DNSSEC, you can check out my previous article. The Validating Resolver cache contains messages, resource records, the nameservers that the BIG-IP queries to resolve DNS queries, and DNSSEC keys. When an authoritative server signs a DNS response, the Validating Resolver will verify the data prior to storing it in cache. The Validating Resolver cache also includes a built-in filter and detection mechanism that rejects unsolicited DNS responses. The screenshot below shows the configuration options for setting up a Validating Resolver cache. I won't go into detail on all the settings for this page because most of them are identical to the Validating Resolver cache. However, as you would expect, there are a few extra options in the Validating Resolver cache. The first is found in the DNS Cache settings where you will find the "DNSSEC Key Cache Size". This setting specifies the maximum size (in bytes) of the DNSKEY cache. The DNS Resolver settings are identical to the Resolver cache. The only other difference is found in the DNSSEC Validator settings. The "Prefetch Key" is enabled by default and it specifies that the BIG-IP will fetch the DNSKEY early in the validation process. You can disable this setting if you want to reduce the amount of resolver traffic, but also understand that, if disabled, a client might have to wait for the validating resolver to perform a key lookup (which will take some time). The "Ignore Checking Disabled Bit" is disabled by default. If you enable this setting, the BIG-IP will ignore the Checking Disabled setting on client queries and will perform response validation and only return secure answers. Keep in mind that caching is a great tool to use, but it's also good to know how much space you are willing to allocate for caching. If you allocate too much, you might serve up outdated responses, but if you allocate too little, you will force your users to wait while DNS recursively asks a bunch of servers for information you could have been holding all along. In the end, it's a reminder that you should know how often your application data changes, and you should configure these caching values accordingly. Well, that does it for this edition of DNS caching...stay tuned for more DNS goodness in future articles!
