High Speed Logging (HSL:: commands)

Basically we're sending data to a pool of syslog servers, is that correct? Is it always using port 514 (syslog)?

 

Is the data broadcasted to all member in the pool?

Thanks for the info Colin.

 

 

So it's still using standard TCP or UDP syslog. Is there any difference in performance between using log -remote versus HSL? Is there a reason you have to call HSL::open before using HSL::send? If there isn't an HSL::close, couldn't the open command be inferred with an HSL::send?

 

 

Aaron

Actually, can you just answer the performance question? I figured out from the HSL::open wiki page that you specify the log protocol and destination pool of syslog servers.

 

 

Thanks,

 

Aaron

There shouldn't be any performance difference as it's using the same send mechanism, as I understand it. They've just made that mechanism more easily accessible for things outside a standard syslog message. So say you want to save off the HTTP request that got sent to your server but looked suspicious, well, now you could use HSL to dump that off to a system listening for that data, etc.

 

 

Colin

Can someone provide any documentation or examples of creating the syslog_pool?

 

 

Thanks,

 

Dan

Hi Dan,

 

 

That should just be a standard LTM pool, I'd guess.

 

 

b pool my_syslog_pool {members 1.1.1.1:514}; b save

 

 

would create the pool from the command line and save the config from memory to file.

 

 

Aaron

Hi -

 

 

I finally got to upgrade to 10.1.0 today so I could work with HSL. I got my iRule converted to use teh HSL::open and HSL::send commands. What surprised me is the change in the log format from the log command. What I get with HSL::send is a continuous string in my log file with no time stamps and no line feeds. That makes parsing the log file for analysis impossible.

 

 

I have been looking for more information on HSL but have not found much to help.

 

 

Here is my original iRule:

 

when HTTP_REQUEST {

 

Get time for start of HTTP request

 

set http_request_time [clock clicks -milliseconds]

 

HTTP::collect 4096

 

Log the start of a new HTTP request

 

set LogString "[IP::client_addr] [HTTP::host] [HTTP::method] [HTTP::uri]"

 

log 192.168.50.17 local0.info "$LogString"

 

}

 

when HTTP_RESPONSE {

 

Received the response headers from the server. Log the pool name, IP and port, status and time delta

 

log 192.168.50.17 local0.info "$LogString [HTTP::status] pool_info: [LB::server] (ResponseTime: [expr [clock clicks -milliseconds] - $http_request_time]ms)"

 

}

 

Here is what i see in syslog

 

Mar 9 16:22:12 rn-f5-lan-sip-01 tmm1[6906]: Rule WebLog : 173.70.45.182 ev.ib-ibi.com GET /image.ibvx?sessionid=6e61cafe-01ba-4fb7-af26-c574ccc24c36&go=iblx&osid=aac88b74-68f7-4b05-81b2-dba5cce0b6b2&puburl=http%3A//isabellacatalog.com/&advurl=http%3A//www.isabellacatalog.com/cat.cfm/pgc/21300/sbc/21306

 

Mar 9 16:22:12 rn-f5-lan-sip-01 tmm1[6906]: Rule WebLog : 173.70.45.182 ev.ib-ibi.com GET /image.ibvx?sessionid=6e61cafe-01ba-4fb7-af26-c574ccc24c36&go=iblx&osid=aac88b74-68f7-4b05-81b2-dba5cce0b6b2&puburl=http%3A//isabellacatalog.com/&advurl=http%3A//www.isabellacatalog.com/cat.cfm/pgc/21300/sbc/21306 200 pool_info: Prod-443 192.168.50.14 443 (ResponseTime: 1ms)

 

 

Here is my HSL irule:

 

when HTTP_REQUEST {

 

set hsl [HSL::open -proto TCP -pool rn_syslog_server_pool_01]

 

Get time for start of HTTP request

 

set http_request_time [clock clicks -milliseconds]

 

HTTP::collect 4096

 

Log the start of a new HTTP request

 

set LogString "[IP::client_addr] [HTTP::host] [HTTP::method] [HTTP::uri]"

 

HSL::send $hsl "$LogString"

 

}

 

when HTTP_RESPONSE {

 

set hsl [HSL::open -proto TCP -pool rn_syslog_server_pool_01]

 

Received the response headers from the server. Log the pool name, IP and port, status and time delta

 

HSL::send $hsl "$LogString [HTTP::status] pool_info: [LB::server] (ResponseTime: [expr [clock clicks -milliseconds] - $http_request_time]ms)"

 

}

 

 

Here is what i see in syslog

 

204.128.27.8 ev.ib-ibi.com GET /at.js?sessionid=9c547e54-f657-4b5f-8069-0789c6bb6aa8&go=122&osid=dcd5a534-39af-44e1-9ad8-2580a7375d59204.128.27.8 ev.ib-ibi.com GET /at.js?sessionid=9c547e54-f657-4b5f-8069-0789c6bb6aa8&go=122&osid=dcd5a534-39af-44e1-9ad8-2580a7375d59 200 pool_info: Prod-443 192.168.50.21 443 (ResponseTime: 24ms)70.181.136.166 ev.ib-ibi.com GET /at.js?sessionid=232af8d3-4bd4-4795-9d97-342a368eaf17&go=126&osid=8a7e4586-8d65-4e38-b535-89ff4f5afd4370.181.136.166 ev.ib-ibi.com GET /at.js?sessionid=232af8d3-4bd4-4795-9d97-342a368eaf17&go=126&osid=8a7e4586-8d65-4e38-b535-89ff4f5afd43 200 pool_info: Prod-443 192.168.50.13 443 (ResponseTime: 13ms)75.111.39.162 ev.ib-ibi.com GET /lat.js?sessionid=b4d4c59c-fc21-43ff-b6fd-0afcc9744dcf&go=&osid=fc2af33e-6223-4de8-a101-b4e6faaa288675.111.39.162 ev.ib-ibi.com GET /lat.js?sessionid=b4d4c59c-fc21-43ff-b6fd-0afcc9744dcf&go=&osid=fc2af33e-6223-4de8-a101-b4e6faaa2886 200 pool_info: Prod-443 192.168.50.14 443 (ResponseTime: 4ms)170.213.132.253 ev.ib-ibi.com GET /at.js?sessionid=43abdb4c-b356-495c-8967-4c41c68a19c3&go=125&osid=0aae746c-2dc7-49f3-af94-aba96ba850d7170.213.132.253 ev.ib-ibi.com GET /image.ibvx?sessionid=43abdb4c-b356-495c-8967-4c41c68a19c3&go=125&osid=0aae746c-2dc7-49f3-af94-aba96ba850d7&puburl=http%3A//ad.yieldmanager.com/st%3Fad_type%3Diframe%26ad_size%3D160x600%26section%3D816134&advurl=http%3A//ad.yieldmanager.com/iframe3%3FZiAAAAZ0DAB5CEMAAAAAABjIEQAAAAAAAAAEAAoAAAAAAAsAAwADEJwjFwAAAAAAmpEYAAAAAAA

 

Mar 9 16:02:53 rn-f5-lan-sip-01 .4.60: ev.ib-ibi.com GET /at.js?sessionid=1e018768-5847-4712-aaf6-98f420697ff9&go=115&osid=0ebe8305-86a6-46c9-b2d5-e7e23ef737b2209.188.179.1 ev.ib-ibi.com GET /at.js?sessionid=68e5ed77-3f0a-4223-bcb3-6fbb07de1f4c&go=125&osid=1b17cdca-9085-414f-ae9b-b127d43c0053 200 pool_info: Prod-443 192.168.50.13 443 (ResponseTime: 25ms)208.54.4.60 ev.ib-ibi.com GET /at.js?sessionid=1e018768-5847-4712-aaf6-98f420697ff9&go=115&osid=0ebe8305-86a6-46c9-b2d5-e7e23ef737b2 200 pool_info: Prod-443 192.168.50.14 443 (ResponseTime: 14ms)71.207.153.35 ev.ib-ibi.com GET /at.js?sessionid=334fc6c3-4f78-47cd-9324-cffdf71c37ac&go=125&osid=a75b1403-771d-49ce-8bdb-65a9cd30274b71.207.153.35 ev.ib-ibi.com GET /at.js?sessionid=334fc6c3-4f78-47cd-9324-cffdf71c37ac&go=125&osid=a75b1403-771d-49ce-8bdb-65a9cd30274b 200 pool_info: Prod-443 192.168.50.15 443 (ResponseTime: 14ms)

 

 

Any help would be greatly appreciated

 

 

Dan

Thanks for the additional info, Spark.

 

 

Could multiple connections reference the same HSL handle (ie, save the HSL::open handle to a global variable)? Or would that defeat the purpose?

 

 

Also, when the HSL::send wiki page states "...that a specific pool member cannot be specified. Pool members are selected based on an internal algorithm.", does this mean that the syslog message is sent to one pool member or broadcast to all pool members?

 

 

Thanks,

 

Aaron

I don't see any problem with multiple connections using the same handle. It definitely doesn't defeat the purpose.

 

 

The message is sent to one pool member only. The point of that sentence is that you can't directly specify or control which pool member gets a specific message. I should update that Wiki page; it's not "an internal algorithm", exactly, it's the LB method that is specified in the pool.

Do you know of a way to enter a line feed in a iRule? Specifically I want to put a line feed after each of these entries

 

 

when HTTP_RESPONSE {

 

set hsl [HSL::open -proto TCP -pool rn_syslog_server_pool_01]

 

Received the response headers from the server. Log the pool name, IP and port, status and time delta

 

HSL::send $hsl "$LogString [HTTP::status] pool_info: [LB::server] (ResponseTime: [expr [clock clicks -milliseconds] - $http_request_time]ms)"

 

}

 

 

thank you!