Forum Discussion

Nimbostratus

Feb 20, 2009

content tracking

we found an irule here by Kirk (thanks Kirk) that breaks content down by count. any genius out there that can take this and add total bytes for each category? our ecomm group wants to see the breakdown in the classification of traffic.

thanks!

Content_ Type_ Tracking

Enter a topic name to show or a new topic name to create; then press Enter

Contributed by: kirkbauer - kirk@f5.com

Description

Have you ever wondered what types of content are served by your web servers? Have you wondered how many requests could be offloaded to the BIG-IP if you added the RAM Cache module? Or maybe how much acceleration you may get from WebAccelerator ? Try out this iRule. Most of the time everything other than the HTML pages should be cacheable.

iRule Source

First, create statistics profile named "ContentType" with following entries:

HTML

Images

Scripts

Documents

Stylesheets

Other

Now associate this Statistics Profile to the virtual server. Then apply the following iRule.

To view the results, go to Statistics -> Profiles - Statistics

when HTTP_RESPONSE {

switch -glob [HTTP::header "Content-type"] {

"image/*" { STATS::incr "ContentType" "Images" }

"text/html" { STATS::incr "ContentType" "HTML" }

"text/css" { STATS::incr "ContentType" "Stylesheets" }

"*javascript" { STATS::incr "ContentType" "Scripts" }

"text/vbscript" { STATS::incr "ContentType" "Scripts" }

"application/pdf" { STATS::incr "ContentType" "Documents" }

"application/msword" { STATS::incr "ContentType" "Documents" }

"application/*powerpoint" { STATS::incr "ContentType" "Documents" }

"application/*excel" { STATS::incr "ContentType" "Documents" }

"default" { STATS::incr "ContentType" "Other" }

}

16 Replies

hoolio
Cirrostratus
Feb 26, 2009
If the Content-Length header doesn't exist, [HTTP::header value "Content-Length"] will return a null string and STATS::incr will probably throw a TCL error. You might either wrap it in a catch, or check that the HTTP::header result is an integer.

Out of curiosity, scottsams, do you see the content length of 1.4k on every response? Can you post the response headers from a response that shows 1.4k for the HTTP::payload length value?

Aaron
scott_sams_8256
Nimbostratus
Feb 26, 2009
thanks i will try that. as for the 1.4k when i sniffed the transaction my MTU was set to 1460. it has since changed and went to 1.2k. i did a tcpdump and saw the frame sizes negotiated to 1260 also. it seemed to correlate. oddly, when i hit one page with a 116k jpg, the stat incremented to 1 with a size of 1.2k.

i am attaching a jpg screen shot with various info. irule, log, website i hit.

here is the log

Feb 26 10:19:30 tmm tmm[955]: 01220002:6: Rule http_stat_tracker : images_len is 1212

Feb 26 10:20:52 tmm tmm[955]: 01220002:6: Rule http_stat_tracker : Header content is image/JPEG; Charset=utf-8 payload length is 1212

Feb 26 10:20:52 tmm tmm[955]: 01220002:6: Rule http_stat_tracker : images_len is 1212

Feb 26 10:21:06 tmm tmm[955]: 01220002:6: Rule http_stat_tracker : Header content is image/JPEG; Charset=utf-8 payload length is 1212

attached is irule, jpg page size
Jon_Strabala_46
Nimbostratus
Jul 02, 2011
Looking at this post I guess I can do the below (simple basic example)

Question 1) but what would I have to (or should I) modify it work with a STREAM profile ?

Question 2) I assume that this does NOT impact the basic built in statistics ?

Question 3) When I make a new stats profile does it basically just build a "rrd" infrastructure ?

Question 4) there was alot of discussion in this thread BUT no single best practice was identified "hoolio" could you comment on this ?

As Scott said (above) I assume I have to first define a profile (Profile->Other-Statistics) and create a new named profile "content_and_len", then check "custum) with following entries:

Html

Html_len

Images

Images_len

Css

Css_len

Other

Other_len

Then associate this Statistics Profile to the virtual server.

Next I would have to the following iRule and then also associate that with the same virtual server, this iRULE will also tests for unset or missing length.
when HTTP_RESPONSE { switch -glob [HTTP::header "Content-type"] { "text/html" { STATS::incr "ContentType" "Html" if { [HTTP::header exists "Content-Length"] } { STATS::incr "ContentType" "Html_len" [HTTP::payload length] } } "image/*" { STATS::incr "ContentType" "Images" if { [HTTP::header exists "Content-Length"] } { STATS::incr "ContentType" "Images_len" [HTTP::payload length] } } "text/css" { STATS::incr "ContentType" "Css" if { [HTTP::header exists "Content-Length"] } { STATS::incr "ContentType" "Css_len" [HTTP::payload length] } } "default" { STATS::incr "ContentType" "Other" if { [HTTP::header exists "Content-Length"] } { STATS::incr "ContentType" "Other_len" [HTTP::payload length] } } } }

To view the results, I would then go to Statistics -> Local Traffic -> Profiles/Statistics

I have not yet pumped any traffic yet but I do see the zero counts.
Jon_Strabala_46
Nimbostratus
Jul 02, 2011
What if I don't care to have the above show up in the GUI F5 interface but want to merely keep a count for every five minutes and write it out to a file and zero the stats

(before anyone goes crazy on me - In my use case I will actually track the content_and_len by some key like USER-AGENT)

I imagine I use could use the version 10.1.* "TABLE" syntax for my hashes and counters along with "AFTER" and dump my results to file (preferably named ua_use_yyyymmdd.out). If I can't use iRULEs to write to a file I guess I could use a rolling log.

rolling log: http://devcentral.f5.com/Tutorials/TechTips/tabid/63/articleType/ArticleView/articleId/1084377/Writing-to-and-rotating-custom-log-files.aspx

Here is some simple code using AFTER (I did not use the "table" syntax) to show what I am having an issue getting my head around

when RULE_INIT { set reqs_last_600_sec 0 set new_req_count 0 after 600000 –periodic set reqs_last_600_sec $new_user_count set new_req_count 0 HOW DO I SAVE the reqs_last_600_sec value to a file ? Is it better to send it to a rollign log file or is it better to make an OOB request to some other server like a memcached server.
} } when HTTP_REQUEST { incr new_req_count }

Any help or directions would be appreciated.
Kevin_Davies_40
Nacreous
Jul 04, 2011
when HTTP_RESPONSE {

if { [HTTP::header exists "Content-Length"] } {

if { [catch {STATS::incr "ContentLength" [HTTP::header "Content-Type"] [HTTP::header "Content-Length"]} } {

log local0.info "Counting content-type \" [HTTP::header "Content-Type"]\" failed with an error"

}

}

}

Seems simpler.
hoolio
Cirrostratus
Jul 05, 2011
Question 1) but what would I have to (or should I) modify it work with a STREAM profile ?

What do you want to use the stream profile for with this iRule? Do you mean that you're rechunking the response content? If so, I don't think that would affect the Content-Length header. I think HTTP_RESPONSE would be triggered before the HTTP payload is modified anyhow.

Question 2) I assume that this does NOT impact the basic built in statistics ?

Correct.

Question 3) When I make a new stats profile does it basically just build a "rrd" infrastructure ?

No, I think the stats data is stored in an internal memory structure. I don't think it's ever written to disk.

Question 4) there was alot of discussion in this thread BUT no single best practice was identified "hoolio" could you comment on this ?

Best practice for what part? As far as avoiding a runtime error if the Content-Length header isn't there? I'd just add a check to see if [HTTP::header Content-Length] ne "" rather than using catch.

Aaron