writing an irule to log all traffic

Hi,

You can use iRules to log the requests and syslog-ng to parse them. Here are some example rules and syslog-ng changes:

=======================================================

1. HTTP logger rule:

when HTTP_REQUEST {
    set the URL here, log it on the response
   set url [HTTP::header Host][HTTP::uri]
   set vip [IP::local_addr]:[TCP::local_port]
}
when HTTP_RESPONSE {
   set client [IP::client_addr]:[TCP::client_port]
   set node [IP::server_addr]:[TCP::server_port]
   set nodeResp [HTTP::status]
    log connection info
   log local0.info "Client: $client -> VIP:$vip$url -> Node: $node with response $nodeResp"
}

=======================================================

2. TCP logger rule:

when CLIENT_ACCEPTED {  
   set vip [IP::local_addr]:[TCP::local_port]
}
when SERVER_CONNECTED {  
   set client "[IP::client_addr]:[TCP::client_port]"
   set node "[IP::server_addr]:[TCP::server_port]"
}  
when CLIENT_CLOSED {  
    log connection info
   log local0.info "Client $client -> VIP: $vip -> Node: $node"  
}

=======================================================

3. UDP logger rule:

when CLIENT_ACCEPTED {  
   set vip [IP::local_addr]:[UDP::local_port]
}
when SERVER_CONNECTED {  
   set client "[IP::client_addr]:[UDP::client_port]"
   set node "[IP::server_addr]:[UDP::server_port]"
}  
when CLIENT_CLOSED {  
    log connection info
   log local0.info "Client $client -> VIP: $vip -> Node: $node"  
}

=======================================================

Associate the TCP, UDP and HTTP rules with the respective virtual servers that you want to log connections for. You can enable a rule for a virtual server under the Resources tab for each virtual server. You will need to make sure that the rule matches the type for each virtual server. For example, you can use the TCP or HTTP rules on an HTTP virtual server. However, you cannot associate a UDP rule unless there is a UDP profile associated with the virtual server.

These rules will log to syslog-ng's local0 facility with the following format:

Mar 1 08:34:01 tmm tmm[730]: Rule HTTP_logger : Client: 192.168.42.26:4746 VIP:172.25.2.12:80 to server: 172.25.2.233:80 for 172.25.2.12/ with response 200

You can then configure syslog-ng to parse local0.info entries that contain "logger" and send them to a remote syslog server by making the following changes to the /etc/syslog-ng/syslog-ng.conf file.

=======================================================

1. Add: local0.info filter, destination and log statements:

 local0.info                                   send logger entries to remote syslog server
filter f_local0.info {
   facility(local0) and level(info) and match("logger");
};
 destination can be a hostname or IP address
destination d_logger {
   tcp("syslog.myhost.com" port (5000));
};
log {
   source(local);
   filter(f_local0.info);
   destination(d_logger);
};

2. Add: and not match("logger") to local0.* to exclude the logger entries from being written to file

 local0.*                                      /var/log/ltm
filter f_local0 {
   facility(local0) and level(info..emerg) and not match("logger");
};
destination d_ltm {
   file("/var/log/ltm" create_dirs(yes));
};
log {
   source(local);
   filter(f_local0);
   destination(d_ltm);
};

For more complete documentation on syslog-ng, you can refer to their site:

http://www.balabit.com/products/support/syslog-ng/

Or here:

http://www.iso.port.ac.uk/docs/downloaded/syslog-ng.html/book1.html

Aaron

Thanks for the info. I'm now receiving the logging that I needed. I've also discovered that when I'm sending this to a remote syslog server, it's not using the management interface. How do you designate which interface to use when making the connection to a remote syslog server?

I don't think I can keep up, he's on fire!

 

 

Hah... I have a long way to go to catch up to you guys. This forum is a great resource though and I get a lot from the posts here.

Group -

 

Thanks so much for this, it's about 95% of what I need

 

Is there any way to 'timestamp' this sort of connection info?

 

 

I've been requested to determine how much time is spent 'inside' the F5 for certain http(s) requests.

 

 

Thanks !

Here is an example of how you can use clock to get deltas between different points in the rule execution:

when CLIENT_ACCEPTED {
   set tcp_start_time [clock clicks -milliseconds]
}
when HTTP_REQUEST {
   set http_request_time [clock clicks -milliseconds]
}
when HTTP_RESPONSE {
   set http_response_time [ clock clicks -milliseconds ]
}
when CLIENT_CLOSED {
   set tcp_end_time [ clock clicks -milliseconds ]
   log local0. "HTTP request/response difference: $http_response_time - $http_request_time = [expr $http_response_time - $http_request_time]"
   log local0. "Total connection time: $tcp_end_time - $tcp_start_time = [expr ($tcp_end_time - $tcp_start_time)]"
}

Apparently, there was an issue with high CPU usage when using the clock command in versions prior to 9.2. I did some searching but couldn't find any relevant CR's. I would upgrade to 9.2.3+ to use the clock function and would make sure to test this rule during a maintenance window if you're applying it to every connection through the BIG-IP.

Aaron

Thanks, hoolio !

 

As always, this brings up another question.

 

Using this works great, but I'm getting some puzzling results from my testing.

 

Frequently, the BigIP says it took less time (between HTTP_REQUEST, and HTTP_RESPONSE) than IIS says it took to complete the request (as taken from the 'TimeTaken' field in the IIS logs.

 

Perhaps I don't fully understand the HTTP_RESPONSE - Could IIS still be sending data, and the F5 records the moment that it reads the header data, and not when the request is 'complete' ?

 

 

If so, is there a way to capture the completion of the request from the F5 perspective ?

 

 

Management suspicion is that the F5 is adding a high amount of overhead / latency to HTTP traffic, and I'm trying to refute this.

 

 

Thanks !

I suppose you could use the HTTP_RESPONSE_DATA event to trigger the end time for the HTTP request/response delta, but that would require using HTTP::collect to trigger the HTTP_RESPONSE_DATA event. HTTP::collect buffers the HTTP response content. I'm not sure how much load this would add. I would guess that this might increase the latency enough to impact the accuracy of the time measurements.

 

 

Can anyone else comment on the best way to measure the delta between the HTTP request being received and when the BIG-IP sends the response back to the client?

 

 

Thanks,

 

Aaron

Hi Adrian, some question

 

 

If I created 2 logger for logging two different virtual pools, how can I perform logging to 2 different files rather than logging them into the ltm log file?

 

 

below is what I have done, but only loggerA can be logged, can we also do the same to loggerB?

 

 

 

- I created 2 HTTP logger iRule with name "http_A_logger" and "http_B_logger"

 

 

- change the following in the /etc/syslog-ng/syslog-ng.conf file. How can we also do for loggerB which logs to /var/Blogger? Thanks in advance.

 

 

 

local0.info /var/log/Alogger

 

filter f_local0.info {

 

facility(local0) and level(info) and match("http_A_logger");

 

};

 

 

destination d_Alogger {

 

file("/var/log/Alogger" create_dirs(yes));

 

};

 

 

log {

 

source(local);

 

filter(f_local0.info);

 

destination(d_Alogger);

 

};

 

 

 

local0.* /var/log/ltm

 

filter f_local0 {

 

facility(local0) and level(info..emerg) and not match("http_A_logger");

 

};

 

 

destination d_ltm {

 

file("/var/log/ltm" create_dirs(yes));

 

};

 

 

log {

 

source(local);

 

filter(f_local0);

 

destination(d_ltm);

 

};

 

 

 

You should be able to add another set of statements (filter, destination and log) for "Blogger" events:

 local0.info /var/log/Blogger
filter f_local0.info {
   facility(local0) and level(info) and match("http_B_logger");
};
destination d_Blogger {
   file("/var/log/Blogger" create_dirs(yes));
};
log {
   source(local);
   filter(f_local0.info);
   destination(d_Blogger);
};

I haven't tested this, but I think it should work with what you have already.

Aaron