iRule to log HTTP request as Apache combined standard format

Hi - I am being a bit lazy and have not given you everything you neded but this should get you off to a good start;

when CLIENT_ACCEPTED {
   set hsl [HSL::open -proto UDP -pool $syslog_server_pool]    
}
when HTTP_REQUEST {
     Save request information for later logging
    set req_start [clock clicks -milliseconds]
    set clientip [IP::client_addr]
    set method [HTTP::method]
    set uri [HTTP::uri]
    set referer [HTTP::header Referer]
    set ua [HTTP::header User-Agent]
}
when HTTP_RESPONSE {
    set log_string "$clientip [clock format [clock seconds] -format "%d/%m/%Y %H:%M:%S %z"] [expr {[clock clicks -milliseconds] - $req_start}] \"$method $uri\" [HTTP::status] [HTTP::header Content-Length] \"$referer\" \"$ua\""
    log local0. $log_string

    HSL::send $hsl $log_string
}

Have commented out the HSL (to log to an external device) commands, and we are just logging to /var/log/ltm.

Hello,

I used this iRule to log HTTP as Apache combined format. Why i have twice every log? F5 LTM is configuered to log to a remote syslog server.

iRule Irule_log_combined:

when HTTP_REQUEST { set host [HTTP::host] set username [HTTP::username] set client_IP [IP::client_addr] set request "\"[HTTP::method] [HTTP::uri] HTTP/[HTTP::version]\"" set request_time [clock clicks -milliseconds] set referer [HTTP::header "Referer"] set ua [HTTP::header "User-Agent"] textset xff [HTTP::header "X-Forwarded-For"] } when HTTP_RESPONSE { set response_time [expr [clock clicks -milliseconds] - $request_time] set now [clock format [clock seconds] -format "%d/%b/%Y:%H:%M:%S %z"] set contentlength [HTTP::header "Content-Length"] set status [HTTP::status]

log local0. "$host $client_IP $username $now $request $status $contentlength \"$referer\" \"$ua\" \"$xff\" $response_time" 

}

log output: 2014-03-11T10:33:41.000+01:00 172.19.208.17 canal_ip5_4200LB tmm[9451]: Rule /SRO01_OTT/Irule_log_combined : webtv-live-hss-m1.direct-ip5.canal-plus.com 194.4.235.1%201 11/Mar/2014:10:33:41 +0100 "GET /live/hss/canalplus-hd.isml/QualityLevels(96000)/Fragments(vo=244855920800474) HTTP/1.1" 200 25331 "myreferer" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" "myforwardedip" 2 2014-03-11T10:33:41.000+01:00 172.19.208.17 canal_ip5_4200LB info tmm[9451]: Rule /SRO01_OTT/Irule_log_combined : webtv-live-hss-m1.direct-ip5.canal-plus.com 194.4.235.1%201 11/Mar/2014:10:33:41 +0100 "GET /live/hss/canalplus-hd.isml/QualityLevels(96000)/Fragments(vo=244855920800474) HTTP/1.1" 200 25331 "myreferer" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" "myforwardedip" 2

2014-03-11T10:33:41.000+01:00 172.19.208.17 canal_ip5_4200LB tmm6[9457]: Rule /SRO01_OTT/Irule_log_combined : webtv-live-hss-m1.direct-ip5.canal-plus.com 194.4.235.1%201 11/Mar/2014:10:33:41 +0100 "GET /live/hss/canalplus-hd.isml/QualityLevels(3889)/FragmentInfo(captfra=244855920996586) HTTP/1.1" 200 198 "myreferer" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" "myforwardedip" 0 2014-03-11T10:33:41.000+01:00 172.19.208.17 canal_ip5_4200LB info tmm6[9457]: Rule /SRO01_OTT/Irule_log_combined : webtv-live-hss-m1.direct-ip5.canal-plus.com 194.4.235.1%201 11/Mar/2014:10:33:41 +0100 "GET /live/hss/canalplus-hd.isml/QualityLevels(3889)/FragmentInfo(captfra=244855920996586) HTTP/1.1" 200 198 "myreferer" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" "myforwardedip" 0

Advice please?

Just spitballing here, but the two logs you've shown are for different requests:

GET /live/hss/canalplus-hd.isml/QualityLevels(96000)/Fragments(vo=244855920800474) HTTP/1.1

GET /live/hss/canalplus-hd.isml/QualityLevels(3889)/FragmentInfo(captfra=244855920996586) HTTP/1.1

Maybe i was not so clear, if you check, i have log twice, there are 4 logs:

log output:

2014-03-11T10:33:41.000+01:00 172.19.208.17 canal_ip5_4200LB tmm[9451]: Rule /SRO01_OTT/Irule_log_combined : webtv-live-hss-m1.direct-ip5.canal-plus.com 194.4.235.1%201 11/Mar/2014:10:33:41 +0100 "GET /live/hss/canalplus-hd.isml/QualityLevels(96000)/Fragments(vo=244855920800474) HTTP/1.1" 200 25331 "myreferer" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" "myforwardedip" 2 2014-03-11T10:33:41.000+01:00 172.19.208.17 canal_ip5_4200LB info tmm[9451]: Rule /SRO01_OTT/Irule_log_combined : webtv-live-hss-m1.direct-ip5.canal-plus.com 194.4.235.1%201 11/Mar/2014:10:33:41 +0100 "GET /live/hss/canalplus-hd.isml/QualityLevels(96000)/Fragments(vo=244855920800474) HTTP/1.1" 200 25331 "myreferer" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" "myforwardedip" 2

2014-03-11T10:33:41.000+01:00 172.19.208.17 canal_ip5_4200LB tmm6[9457]: Rule /SRO01_OTT/Irule_log_combined : webtv-live-hss-m1.direct-ip5.canal-plus.com 194.4.235.1%201 11/Mar/2014:10:33:41 +0100 "GET /live/hss/canalplus-hd.isml/QualityLevels(3889)/FragmentInfo(captfra=244855920996586) HTTP/1.1" 200 198 "myreferer" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" "myforwardedip" 0 2014-03-11T10:33:41.000+01:00 172.19.208.17 canal_ip5_4200LB info tmm6[9457]: Rule /SRO01_OTT/Irule_log_combined : webtv-live-hss-m1.direct-ip5.canal-plus.com 194.4.235.1%201 11/Mar/2014:10:33:41 +0100 "GET /live/hss/canalplus-hd.isml/QualityLevels(3889)/FragmentInfo(captfra=244855920996586) HTTP/1.1" 200 198 "myreferer" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" "myforwardedip" 0

Ah, I see. So how have you configured syslog to use a remote server? It seems that you may have duplicated configs. One log is going to local0. and one is going to local0.info.

Thank's for the feedback. DO you mean to check on syslog server or on F5 LTM? On F5 LTM, in the system section->Logs->Configuration->Remote Logging i put the Remote IP as the IP address of my remote syslog server and the remote port:514 and the Local IP: is the management IP of my LTM. Is it OK? should i do something on my LTM?

How to avoid the duplicated config for log?

What F5 version? I just tested this with 11.5 and only received one remote syslog entry to local0.info. You still also get the internal /var/log/ltm syslog entry though. Have you modified /etc/syslog-ng/syslog-ng.conf?

version 11.2.1. What should i check in the /etc/syslog-ng/syslog-ng.conf ? The F5 was already configured by an outsourcer that left the company without purchasing any documentation about the LTM...

I modified the log in the iRule but still receive the logs twice...

log local0.info "$host $client_IP $username $now $request $status $contentlength \"$referer\" \"$ua\" \"$xff\" $response_time" }

1. Look for any references in the syslog-ng.conf file for external syslog. These should be overwritten on a reboot, but stranger things have happened. Here are some references for the syslog configuration:

   http://support.f5.com/kb/en-us/solutions/public/13000/300/sol13317.html?sr=35791506

   http://support.f5.com/kb/en-us/solutions/public/7000/300/sol7342.html?sr=35791506

   http://support.f5.com/kb/en-us/solutions/public/13000/000/sol13083.html?sr=35791506

2. Run this command and see what you get:

   tmsh list sys syslog
   

   You should just see the one remote logging config that you created in the GUI. Example:

   sys syslog {
       remote-servers {
           remotesyslog1 {
               host 10.70.0.1
               local-ip 10.70.0.101
           }
       }
   }
   

   Actually what do you get when you run this command:

   tmsh list sys syslog all-properties