Caching based on Surrogate-Control header
I want to configure a virtual server to use Web Acceleration caching based on the Surrogate-Control response header instead of the Cache-Control response header. Both headers must be preserved, so I can't just overwrite the Cache-Control header with the Surrogate-Control header upon HTTP_RESPONSE. Is there some built in support for this, or is there some nice solution to this?
APM inactivity timeout redirect or notification page for LTM + APM connections
Background on this: Have a customer that is publishing a Microsoft CRM instance behind APM and doing KCD with smart card auth. Access policy works fine, KCD works fine, web app works fine. The only problem we have is the inactivity timeout setting. Once the limit has been reached, the session is removed and content is no longer sent to the user in a very abrupt fashion. This is a problem because ALOT of the page is cached on the clients workstation and all they see is broken JPEGs and incomplete web content. Once they click around they are re authenticated but it is not pretty. I want to find a way to notify the user they have been inactive for a certain amount of time, send a HTTP 200 response with content notifying them with a link to click on to re authenticate. The option of increasing the inactive timeout is not an option due to their access session license limit. There would be alot of abandoned sessions that would aggregate potentially going over this limit. I know with webtop and ssl vpn, you get a notification that you are about to be logged out due to inactivity but this doesn't seem to be available for LTM + APM policies. This is what I have so far, there has to be a more efficient way of doing this though. when ACCESS_SESSION_STARTED { set ::EXPIRE "false" } when ACCESS_SESSION_CLOSED { log local0. "Session has been closed" set ::EXPIRE "true" } when HTTP_RESPONSE { if {$::EXPIRE equals "true"} { HTTP::respond 200 content " You've Been Logged out due to inactivity You have been logged out due to inactivity Thanks for Using the application Click to log back in. " } }
Session size garbage after an AD query, can I remove them?
Hello community, I'm using our BigIP as an IDP and we have about 20 federations to date. They all share the same VP and iRule. One of the federations need to know, if somebody logs in and they are a manager, how many employees do they have. I have found that this generate a lot of session variables, and I worry that I will exhaust the cache. So I wonder if there is any way to discard these session variables or mark them as garbage or something, because while it might be okay with these leftovers for managers with five employees, it's not okay for managers with five hundred! I'm curious if there is any way to discard these variables to exclude from the session? This is what I do: I use the following LDAP filter to get all enabled users that has them in the manager attribute: (&(manager=CN=%{session.logon.last.username},OU=Users,OU=Organization,DC=Corp,DC=net)(!(userAccountControl:1.2.840.113556.1.4.803:=2))) The filter works fine. To not make the query too expensive, I tried limiting the query to only the "c" attribute (country), which contains two letters only. However, dn is included and I can't change that, so I removed the "c" attribute. This generates session output like this: 9543782a.session.ad./Common/ad_query_employees_act_active_directory_query_ag.attr.dn CN=Tobias Anderson,OU=Users,OU=Organization,DC=Corp,DC=net 9543782a.session.ad./Common/ad_query_employees_act_active_directory_query_ag.attr.dn.1 CN=Philippe Hudson,OU=Users,OU=Organization,DC=Corp,DC=net 9543782a.session.ad./Common/ad_query_employees_act_active_directory_query_ag.attr.dn.2 CN=Jonas Gabriel,OU=Users,OU=Organization,DC=Corp,DC=net 9543782a.session.ad./Common/ad_query_employees_act_active_directory_query_ag.attr.dn.3 CN=Ted Miles,OU=Users,OU=Organization,DC=Corp,DC=net 9543782a.session.ad./Common/ad_query_employees_act_active_directory_query_ag.attr.dn.4 CN=Lars Hedin,OU=Users,OU=Organization,DC=Corp,DC=net 9543782a.session.ad./Common/ad_query_employees_act_active_directory_query_ag.attr.dn.5 CN=Tomas Jeffrey,OU=Users,OU=Organization,DC=Corp,DC=net ... Now, I only care about how many employees they have, not who they are. As it stands now, we'll be populating a custom attribute in AD with this information instead, but I'm curious because we might end up with similar applications in the near future that may produce unwanted variables. Surely there must be some garbage collection functions or unset functions to tidy up our sessions? How would you guys implement this? Any ideas?What is HTTP Part VIII - Compression and Caching
In the last article of this What is HTTP? series we covered the nuances of OneConnect on HTTP traffic through the BIG-IP. In this article, we’ll cover caching and compression. We’ll deal with compression first, and then move on to caching. Compression In the very early days of the internet, much of the content was text based. This meant that the majority of resources were very small in nature. As popularity grew, the desire for more rich content filled with images grew as well, and resource sized began to explode. What had not yet exploded yet, however, was the bandwidth available to handle all that rich content (and you could argue that’s still the case in mobile and remote terrestrial networks as well.) This intersection of more resources without more bandwidth led to HTTP development in a few different areas: Methods for getting or sending partial resources Methods for identifying if resources needed to be retrieved at all Methods for reducing resources during transit that could be successfully reproduced after receipt The various range headers were developed to handle the first case, caching, which we will discuss later in this article, was developed to handle the second case, and compression was developed to handle the third case. The basic definition of data compression is simply reducing the bits necessary to accurately represent the resource. This is done not only to save network bandwidth, but also on storage devices to save space. And of course money in both areas as well. In HTTP/1.0, end-to-end compression was possible, but not hop-by-hop as it does not have a distinguishing mechanism between the two. That is addressed in HTTP/1.1, so intermediaries can use complex algorithms unknown to the server or client to compress data between them and translate accordingly when speaking to the clients and servers respectively. In 11.x forward, compression is managed in its own profile. Prior to 11.x, it was included in the http profile. The httpcompression profile overview on AskF5 is very thorough, so I won’t repeat that information here, but you will want to pay attention to the compression level if you are using gzip (default.) The default of level 1 is fast from the perspective of the act of compressing on BIG-IP, but having done minimal compressing, reaps the least amount of benefit on the wire. If a particular application has great need for less bandwidth utilization toward the clientside of the network footprint, bumping up to level 6 will increase the reduction in bandwidth without overly taxing the BIG-IP to perform the operation. Also, it’s best to avoid compressing data that has already been compressed, like images and pdfs. Compressing them actually makes the resource larger, and wastes BIG-IP resources doing it! SVG format would be an exception to that rule. Also, don’t compress small files. The profile default is 1M for minimum content length. For BIG-IP hardware platforms, compression can be performed in hardware to offload that function. There is a database variable that you can configure to select the data compression strategy via sys modify db compression.strategy . The default value is latency, but there are four other strategies you can employ as covered in the manual. Caching Web caching could (and probably should) be its own multi-part series. The complexities are numerous, and the details plentiful. We did a series called Project Acceleration that covered some of the TCP optimization and compression topics, as well as the larger product we used to call Web Accelerator but is now the Application Acceleration Manager or AAM. AAM is caching and application optimization on steroids and we are not going to dive that deep here. We are going to focus specifically on HTTP caching and how the default functionality of the ramcache works on the BIG-IP. Consider the situation where there is no caching: In this scenario, every request from the browser communicates with the web server, no matter how infrequently the content changes. This is a wasteful use of resources on the server, the network, and even the client itself. The most important resource to our short attention span end users is time! The more objects and distance from the server, the longer the end user waits for that page to render. One way to help is to allow local caching in the browser: This way, the first request will hit the web server, and repeat requests for that same resource will be pulled from the cache (assuming the resource is still valid, more on that below.) Finally, there is the intermediary cache. This can live immediately in front of the end users like in an enterprise LAN, in a content distribution network, immediately in front of the servers in a datacenter, or all of the above! In this case, the browser1 client requests an object not yet in the cache serving all the browser clients shown. Once the cache has the object from the server, it will serve it to all the browser clients, which offloads the requests to server, saves the time in doing so, and brings the response closer to the browser clients as well. Given the benefits of a caching solution, let’s talk briefly of the risks. If you take the control of what’s served away from the server and put it in the hands of an intermediary, especially an intermediary the administrators of the origin server might not have authority over, how do you control then what content the browsers ultimately are loading? That’s where the HTTP standards on caching control come into play. HTTP/1.0 introduced the Pragma, If-Modified-Since, Last-Modified, and Expires headers for cache control. The Cache-Control and ETag headers along with a slew of “If-“ conditional headers were introduced in HTTP/1.1, but you will see many of the HTTP/1.0 cache headers in responses alongside the HTTP/1.1 headers for backwards compatibility. Rather than try to cover the breadth of caching here, I’ll leave it to the reader to dig into the quite good resources linked at the bottom (start with "Things Caches Do") for detailed understanding. However, there's a lot to glean from your browser developer tools and tools like Fiddler and HttpWatch. Consider this request from my browser for the globe-sm.svg file on f5.com. Near the bottom of the image, I’ve highlighted the request Cache-Control header, which has a value of no-cache. This isn’t a very intuitive name, but what the client is directing the cache is that it must submit the request to the origin server every time, even if the content is fresh. This assures authentication is respected while still allowing for the cache to be utilized for content delivery. In the response, the Cache-Control header has two values: public and max-age. The max-age here is quite large, so this is obviously an asset that is not expected to change much. The public directive means the resource can be stored in a shared cache. Now that we have a basic idea what caching is, how does the BIG-IP handle it? The basic caching available in LTM is handled in the same profile that AAM uses, but there are some features missing when AAM is not provisioned. It used to be called ramcache, but now is the webacceleration profile. Solution K14903 provides the overview of the webacceleration profile but we’ll discuss the cache size briefly. Unlike the Web Accelerator, there is no disk associated with the ramcache. As the name implies, this is “hot” cache in memory. So if you are memory limited on your BIG-IP, 100MB might be a little too large to keep locally. Managing the items in cache can be done via the tmsh command line with the ltm profile ramcache command. tmsh show/delete operations can be used against this method. An example show on my local test box: root@(ltm3)(cfg-sync Standalone)(Active)(/Common)(tmos)# show ltm profile ramcache webacceleration Ltm::Ramcaches /Common/webacceleration Host: 192.168.102.62 URI : / -------------------------------------- Source Slot/TMM 1/1 Owner Slot/TMM 1/1 Rank 1 Size (bytes) 3545 Hits 5 Received 2017-11-30 22:16:47 Last Sent 2017-11-30 22:56:33 Expires 2017-11-30 23:16:47 Vary Type encoding Vary Count 1 Vary User Agent none Vary Encoding gzip,deflate Again, if you have AAM licensed, you can provision it and then additional fields will be shown in the webacceleration profile above to allow for an acceleration policy to be applied against your virtual server. Resources RFC 2616 - The standard fine print. Things Caches Do - Excellent napkin diagrams that provide simple explanations of caching operations. Caching Tutorial - Comprehensive walk through of caching. HTTP Caching - Brief but informative look at caching from a webdev perspective. HTTP Caching - Google develops page with examples, flowcharts, and advice on caching strategies. Project Acceleration - Our 10 part series on web acceleration technology available on the BIG-IP platform in LTM and/or AAM modules. Solution K5157 - BIG-IP caching and the Vary header Make Your Cache Work For You - Article by Dawn Parzych here on DevCentral on tuning techniques
BIG-IP : deleted data-group/file contents are re-appearing in re-created data-group/file of same name
f5 BIG-IP LTM VE v11.4.0 on ESXi Does BIG-IP somewhere retain contents of deleted data-files - perhaps in a hidden cache file or a database ? Consider the following sequence : Via BIG-IP browser admin , I delete a data-group and then its related data-file. When I peruse BIG-IP's data dirs, the corresponding data-file is no longer present : /config/filestore/files_d/Common_d/data_group_d/:Common:test_data_file_57958_1 This is as expected. Also not present are any other data-files with names of the form : /:Common:test_data_file_* This is also as expected. Next via the iControl API I create a brand new data-group/file of same name test_data_file Via browser admin, I open the newly created data-file, and I'm very surprised to see remnants of the previous deleted data-file’s data ( at the top of the file ), along with some, but not all, of the data I supplied in the new external data-file ( the one that was pointed to with the Browse button during the Create sequence ). This is completely unexpected. Really weird. Somehow BIG-IP is remembering the contents of a deleted data-file and then co-mingling that data with the new data supplied for a new data-file of same name.BIG-IP : iControl LocalLBDataGroupFile.set_local_path()
F5 BIG-IP LTM VE v11.4.0 on ESXi iControl re-cache data-group operation : LocalLBDataGroupFile.set_local_path() For a live prod BIG-IP cluster with a VIP iRule that is actively reading data from the data-group , is the above considered a risky operation ? e.g. attempt live-swap of data-group's data-file, fails due to locks or other systems-level issues, data-group now unavailable to iRule I've attempted live-update of a data-file's contents via BIG-IP browser admin ( copy/paste new contents and click “Update” ) and seen the corresponding data-group become unavailable to the iRule. The paranoid approach is to create a new data-group/file and a copy of the iRule modified to point to this new data-group , and then in the VIP’s iRule list swap old/new iRules – so that never actually update a “live” data-group. So now I am concerned about performing similar operation via iControl LocalLBDataGroupFile.set_local_path() –- although apparently this API is not actually live-updating a cached data-file but rather re-pointing data-group to a new cached data-file ?RAM Cache and POST Requests
Is it possible to store responses to POST requests in RAM Cache using the URI and body as keys? Let's say I have a resource /deals that consumes a json body containing details of a product and returns the available deals for such product. This request doesn't modify any data in my backend. Should I use an IRule to do this? Thanks in advance!F5-RAMcache cache only when the cache headers is present
Hello, I trying to configure the F5 devices to enable cache only when the cache headers is present and caching the time specified in the headers. After done some test I can see the objects without cache headers in the backend are cached with randomly time-to-be-cached (I'm sure is not random but I don't know the algorithm applied) Do anybody how to configure RAMCache to only cache the object when the cache headers is present? In all my tests the field Ignore Headers is set to "none" I like share my lab and test, all of them test in 8950 with 10.2.4 version. http profile applied profile http STATIC-HTTP { defaults from http ... ramcache enable ramcache size 512mb ramcache max entries 10000 ramcache min object size 512 ramcache max object size 102400 ramcache ignore client cache control all ramcache aging rate 9 ramcache insert age header enable ramcache uri exclude "/someobject.gif" ramcache uri include none ramcache uri pinned none ... } F5 runtime memory - tmsh show /ltm profile ramcache STATIC-HTTP .... Host: www.domain.com URI : /file.html -------------------------------------- Source Slot/TMM 0/7 Owner Slot/TMM 0/0 Rank 1 Size (bytes) 274 Hits 0 Received 2015-07-15 06:11:04 Last Sent 2015-07-15 06:11:04 Expires 2015-07-15 06:39:24 Vary Type none Vary Count 1 Vary User Agent none Vary Encoding none Total records returned: 2 Request to F5 - 2 requests, the second one with the origin content changed osmc@osmc:~$ curl --user-agent t7 -H"Host: www.domain.com http://1.2.3.4/file.html -IXGET HTTP/1.1 200 OK Date: Wed, 15 Jul 2015 06:11:04 GMT Server: Apache Last-Modified: Wed, 15 Jul 2015 05:56:54 GMT ETag: "b-51ae39f32c30d" Accept-Ranges: bytes Content-Length: 11 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html osmc@osmc:~$ curl --user-agent t7 -H"Host: www.domain.com http://1.2.3.4/file.html -IXGET HTTP/1.1 200 OK Server: Apache Last-Modified: Wed, 15 Jul 2015 05:56:54 GMT ETag: "b-51ae39f32c30d" Accept-Ranges: bytes Keep-Alive: timeout=5, max=100 Content-Type: text/html Connection: Keep-Alive Date: Wed, 15 Jul 2015 06:11:48 GMT Age: 44 Content-Length: 11 Requests to backend : 2 GETs with apache in foreground and changing the content requested curl -IXGET -H"Host: www.domain.com" http://apacheX/file.html HTTP/1.1 200 OK Date: Wed, 15 Jul 2015 06:11:21 GMT Server: Apache Last-Modified: Wed, 15 Jul 2015 05:56:54 GMT ETag: "b-51ae39f32c30d" Accept-Ranges: bytes Content-Length: 11 Content-Type: text/html 1.2.3.4 vhostX - [15/Jul/2015:08:11:37 +0200] "GET /file.html HTTP/1.1" 200 11 "-" "curl/7.19.5 (ostype) libcurl/7.19.5 OpenSSL/0.9.7d zlib/1.2.3" www.domain.com "-" "-" 1486 date +%s > file.html && cat file.html 1436940694 curl -IXGET -H"Host: www.domain.com" http://apacheX/file.html HTTP/1.1 200 OK Date: Wed, 15 Jul 2015 06:11:37 GMT Server: Apache Last-Modified: Wed, 15 Jul 2015 06:11:34 GMT ETag: "b-51ae3d3a72c54" Accept-Ranges: bytes Content-Length: 11 Content-Type: text/html 1.2.3.4 vhostX - [15/Jul/2015:08:11:37 +0200] "GET /file.html HTTP/1.1" 200 11 "-" "curl/7.19.5 (ostype) libcurl/7.19.5 OpenSSL/0.9.7d zlib/1.2.3" www.domain.com "-" "-" 1486
Bypass caching when a cookie matching a pattern is present
We have created an iRule which should disable caching for any request containing a cookie matching a pattern (starting with SESS). It seems like all the rules we have tried have no effect. When caching is enabled for the pool, requests are always cached. Here is the latest rule we have: when HTTP_REQUEST { set c_cookies [HTTP::cookie names] if {[lsearch -regexp $c_cookies "SESS*"]} { CACHE::disable } } Any ideas on why this wouldn't be working? Also, I'm curious if F5's respect the HTTP Cache-Control max-age header.
