Need assistance with writing an irule to log all traffic flow. Support suggested that this should be done versus making changes to the syslog-ng file. I’ve tried making changes to syslog-ng file with no luck. Please let me know if this is worth pursuing or should I go back to the syslog-ng file.
I’m looking to log source and destination IP addresses along with the corresponding ports.
Thanks
Hi,
You can use iRules to log the requests and syslog-ng to parse them. Here are some example rules and syslog-ng changes:
=======================================================
1. HTTP logger rule:
when HTTP_REQUEST {
set the URL here, log it on the response
set url [HTTP::header Host][HTTP::uri]
set vip [IP::local_addr]:[TCP::local_port]
}
when HTTP_RESPONSE {
set client [IP::client_addr]:[TCP::client_port]
set node [IP::server_addr]:[TCP::server_port]
set nodeResp [HTTP::status]
log connection info
log local0.info "Client: $client -> VIP:$vip$url -> Node: $node with response $nodeResp"
}
=======================================================
2. TCP logger rule:
when CLIENT_ACCEPTED {
set vip [IP::local_addr]:[TCP::local_port]
}
when SERVER_CONNECTED {
set client "[IP::client_addr]:[TCP::client_port]"
set node "[IP::server_addr]:[TCP::server_port]"
}
when CLIENT_CLOSED {
log connection info
log local0.info "Client $client -> VIP: $vip -> Node: $node"
}
=======================================================
3. UDP logger rule:
when CLIENT_ACCEPTED {
set vip [IP::local_addr]:[UDP::local_port]
}
when SERVER_CONNECTED {
set client "[IP::client_addr]:[UDP::client_port]"
set node "[IP::server_addr]:[UDP::server_port]"
}
when CLIENT_CLOSED {
log connection info
log local0.info "Client $client -> VIP: $vip -> Node: $node"
}
=======================================================
Associate the TCP, UDP and HTTP rules with the respective virtual servers that you want to log connections for. You can enable a rule for a virtual server under the Resources tab for each virtual server. You will need to make sure that the rule matches the type for each virtual server. For example, you can use the TCP or HTTP rules on an HTTP virtual server. However, you cannot associate a UDP rule unless there is a UDP profile associated with the virtual server.
These rules will log to syslog-ng’s local0 facility with the following format:
Mar 1 08:34:01 tmm tmm[730]: Rule HTTP_logger : Client: 192.168.42.26:4746 VIP:172.25.2.12:80 to server: 172.25.2.233:80 for 172.25.2.12/ with response 200
You can then configure syslog-ng to parse local0.info entries that contain “logger” and send them to a remote syslog server by making the following changes to the /etc/syslog-ng/syslog-ng.conf file.
=======================================================
1. Add: local0.info filter, destination and log statements:
local0.info send logger entries to remote syslog server
filter f_local0.info {
facility(local0) and level(info) and match("logger");
};
destination can be a hostname or IP address
destination d_logger {
tcp("syslog.myhost.com" port (5000));
};
log {
source(local);
filter(f_local0.info);
destination(d_logger);
};
2. Add: and not match(“logger”) to local0.* to exclude the logger entries from being written to file
local0.* /var/log/ltm
filter f_local0 {
facility(local0) and level(info..emerg) and not match("logger");
};
destination d_ltm {
file("/var/log/ltm" create_dirs(yes));
};
log {
source(local);
filter(f_local0);
destination(d_ltm);
};
For more complete documentation on syslog-ng, you can refer to their site:
http://www.balabit.com/products/support/syslog-ng/
Or here:
http://www.iso.port.ac.uk/docs/downloaded/syslog-ng.html/book1.html
Aaron
Thanks for the info. I’m now receiving the logging that I needed. I’ve also discovered that when I’m sending this to a remote syslog server, it’s not using the management interface. How do you designate which interface to use when making the connection to a remote syslog server?
I don’t think I can keep up, he’s on fire!
Hah… I have a long way to go to catch up to you guys. This forum is a great resource though and I get a lot from the posts here.
Group -
Thanks so much for this, it’s about 95% of what I need
Is there any way to ‘timestamp’ this sort of connection info?
I’ve been requested to determine how much time is spent ‘inside’ the F5 for certain http(s) requests.
Thanks !
Here is an example of how you can use clock to get deltas between different points in the rule execution:
when CLIENT_ACCEPTED {
set tcp_start_time [clock clicks -milliseconds]
}
when HTTP_REQUEST {
set http_request_time [clock clicks -milliseconds]
}
when HTTP_RESPONSE {
set http_response_time [ clock clicks -milliseconds ]
}
when CLIENT_CLOSED {
set tcp_end_time [ clock clicks -milliseconds ]
log local0. "HTTP request/response difference: $http_response_time - $http_request_time = [expr $http_response_time - $http_request_time]"
log local0. "Total connection time: $tcp_end_time - $tcp_start_time = [expr ($tcp_end_time - $tcp_start_time)]"
}
Apparently, there was an issue with high CPU usage when using the clock command in versions prior to 9.2. I did some searching but couldn’t find any relevant CR’s. I would upgrade to 9.2.3+ to use the clock function and would make sure to test this rule during a maintenance window if you’re applying it to every connection through the BIG-IP.
Aaron
Thanks, hoolio !
As always, this brings up another question.
Using this works great, but I’m getting some puzzling results from my testing.
Frequently, the BigIP says it took less time (between HTTP_REQUEST, and HTTP_RESPONSE) than IIS says it took to complete the request (as taken from the ‘TimeTaken’ field in the IIS logs.
Perhaps I don’t fully understand the HTTP_RESPONSE - Could IIS still be sending data, and the F5 records the moment that it reads the header data, and not when the request is ‘complete’ ?
If so, is there a way to capture the completion of the request from the F5 perspective ?
Management suspicion is that the F5 is adding a high amount of overhead / latency to HTTP traffic, and I’m trying to refute this.
Thanks !
I suppose you could use the HTTP_RESPONSE_DATA event to trigger the end time for the HTTP request/response delta, but that would require using HTTP::collect to trigger the HTTP_RESPONSE_DATA event. HTTP::collect buffers the HTTP response content. I’m not sure how much load this would add. I would guess that this might increase the latency enough to impact the accuracy of the time measurements.
Can anyone else comment on the best way to measure the delta between the HTTP request being received and when the BIG-IP sends the response back to the client?
Thanks,
Aaron
Hi Adrian, some question
If I created 2 logger for logging two different virtual pools, how can I perform logging to 2 different files rather than logging them into the ltm log file?
below is what I have done, but only loggerA can be logged, can we also do the same to loggerB?
- I created 2 HTTP logger iRule with name “http_A_logger” and “http_B_logger”
- change the following in the /etc/syslog-ng/syslog-ng.conf file. How can we also do for loggerB which logs to /var/Blogger? Thanks in advance.
local0.info /var/log/Alogger
filter f_local0.info {
facility(local0) and level(info) and match(“http_A_logger”);
};
destination d_Alogger {
file(“/var/log/Alogger” create_dirs(yes));
};
log {
source(local);
filter(f_local0.info);
destination(d_Alogger);
};
local0.* /var/log/ltm
filter f_local0 {
facility(local0) and level(info..emerg) and not match(“http_A_logger”);
};
destination d_ltm {
file(“/var/log/ltm” create_dirs(yes));
};
log {
source(local);
filter(f_local0);
destination(d_ltm);
};
You should be able to add another set of statements (filter, destination and log) for “Blogger” events:
local0.info /var/log/Blogger
filter f_local0.info {
facility(local0) and level(info) and match("http_B_logger");
};
destination d_Blogger {
file("/var/log/Blogger" create_dirs(yes));
};
log {
source(local);
filter(f_local0.info);
destination(d_Blogger);
};
I haven’t tested this, but I think it should work with what you have already.
Aaron
Actually, I would expect the filter names in syslog-ng.conf need to be unique. Can you try something like this?
filter f_local0_http_A_logger {
facility(local0) and level(info) and match("http_A_logger");
};
filter f_local0_http_B_logger {
facility(local0) and level(info) and match("http_B_logger");
};
Aaron
Nice catch. Good to hear you got it working.
Aaron
Can anybody provide the steps to configure / integrate F5 LTM & Firepass with Splunk.
Please allow me to take this question one step further. I have a need to log whenever someone uses the FTP APPEND command. My thought was that I do a TCP::collect in the client_accepted, then a switch statement in the client_data event. I cannot seem to get the order of the TCP::collect and TCP::release right as when I connect with the iRUle, I do not get anything past the connection. It is as if the iRule is waiting on a server response. I know that tcp::collect and release are slightly different than the http counterparts, but does anyone have a hint on how I can monitor basic data without affecting the flow of data between the client and server?
Hi Thomas,
I believe the problem is that the client waits for the server to send a message first. So there isn’t any client data to collect initially. Spark described an option to use the skip_bytes flag on TCP::collect to handle this sort of scenario:
However, there might be a simpler option if all you want to do is look for APPEND in the request payloads. You might be better off using a blank stream profile and iRule which applies the stream filter only on requests and logs in the STREAM_MATCHED event. You could try enabling the stream filter using STREAM::enable in CLIENT_ACCEPTED and then disabling it in LB_SELECTED or SERVER_CONNECTED.
http://devcentral.f5.com/wiki/default.aspx/iRules/stream
Aaron
Shoudl something like this work combining the two rules?
i guess i would need one to log http traffic under http_mx_log and one for tcp under tcp_mx_log?
when HTTP_REQUEST {
set the URL here, log it on the response
set url [HTTP::header Host][HTTP::uri]
set vip [IP::local_addr]:[TCP::local_port]
}
when HTTP_RESPONSE {
set client [IP::client_addr]:[TCP::client_port]
set node [IP::server_addr]:[TCP::server_port]
set nodeResp [HTTP::status]
local0.* /var/log/ltm
filter f_local0 {
facility(local0) and level(info..emerg) and not match("http_mxa_log") and not match("http_mxb_log");
};
destination d_ltm {
file("/var/log/ltm" create_dirs(yes));
};
log {
source(local);
filter(f_local0);
destination(d_ltm);
}
[root@iris:Active] config b virtual bar list
virtual bar {
snat automap
pool foo
destination 172.28.17.33:http
ip protocol tcp
rules myrule
}
[root@iris:Active] config b rule myrule list
rule myrule {
when CLIENT_ACCEPTED {
log local0. "[IP::client_addr]:[TCP::client_port]"
}
}
[root@iris:Active] config b syslog include
SYSLOG - Include Data:
filter f_local0 {
facility(local0) and
not match("myrule");
};
log {
source(s_syslog_pipe);
filter(f_local0);
filter(f_no_audit);
destination(d_ltm);
};
filter f_myrule {
match("myrule");
};
destination d_myrule {
file("/var/log/myrule" create_dirs(yes));
};
log {
source(s_syslog_pipe);
filter(f_myrule);
destination(d_myrule);
};
[root@iris:Active] config cat /var/log/ltm
[root@iris:Active] config cat /var/log/myrule
Oct 18 22:19:40 local/tmm info tmm[4601]: Rule myrule : 192.168.206.102:53447
Oct 18 22:19:42 local/tmm info tmm[4601]: Rule myrule : 192.168.206.102:53449
Oct 18 22:19:45 local/tmm info tmm[4601]: Rule myrule : 192.168.206.102:53450
Oct 18 22:20:10 local/iris notice b[28110]: 012e0045:5: AUDIT - user root - rule myrule list
hope this helps.
I now have another question is it possible to log traffic based on the cookie value of the traffic going to a particular host?
e.g. we are using an external provider for a search which we want tomonitor response times too and the http requests to that external source use a particular cookie value if that could just be logged then that would give us the info we need?
I now have another question is it possible to log traffic based on the cookie value of the traffic going to a particular host?yes, it’s possible. you may check if cookie exists and then log response time for the request/response.
HTTP::cookie
http://devcentral.f5.com/wiki/iRules.HTTP\_\_cookie.ashx
Log Tcp And Http Request Response Info by Aaron
http://devcentral.f5.com/wiki/iRules.LogTcpAndHttpRequestResponseInfo.ashx
hope this helps.