Forum Discussion

GavinW_29074's avatar
GavinW_29074
Icon for Nimbostratus rankNimbostratus
Oct 25, 2011

HSL not working?

Hi there,

I'm trying to set-up an iRule to log-out access logs to a splunk syslog server using HSL.

However I'm having issues with the events not getting to the syslog server...

The iRule is: 

 when HTTP_REQUEST {
   
    Save Request Side Information
   
   set hsl [HSL::open -proto UDP -pool splunk]
   set http_request "\"[HTTP::method] [HTTP::uri] HTTP/[HTTP::version]\""
   set http_request_time [clock clicks -milliseconds]
   set http_user_agent "\"[HTTP::header User-Agent]]\""
   set http_host [HTTP::host]
   set http_username [HTTP::username]
   set client_ip [IP::remote_addr]
   set client_port [TCP::remote_port]
   set http_request_uri [HTTP::uri]
   set referer "\"[HTTP::header value referer]\""
}
when HTTP_RESPONSE {
   set response_time [expr [clock clicks -milliseconds] - $http_request_time]
   set timestamp [clock format [clock seconds] -format "%d/%h/%y:%T %Z" -gmt 1 ]
   set virtual [virtual]
   set content_length 0
   if { [HTTP::header exists "Content-Length"] } {
      set content_length [HTTP::header "Content-Length"]
   }
   set lb_server "[LB::server addr]:[LB::server port]"
   if { [string compare "$lb_server" ""] == 0 } {
      set lb_server ""
   }
   set status_code [HTTP::status]
   set content_type [HTTP::header "Content-type"]
   set log_msg ""
   append log_msg "virtual=$virtual "
   append log_msg "$client_ip "
   append log_msg "client_port=$client_port "
   append log_msg "lb_server=$lb_server "
   append log_msg "$http_host "
   append log_msg "$http_username "
   append log_msg "\[$timestamp\] "
   append log_msg "$http_request_uri "
   append log_msg "$http_request "
   append log_msg "$status_code "
   append log_msg "content_type=$content_type "
   append log_msg "$content_length "
   append log_msg "resp_time=$response_time "
   append log_msg "$referer "   
   append log_msg "$http_user_agent\n"
   log local0. "Sending log to HSL"
   HSL::send $hsl $log_msg
}

I can see the 'Sending log to HSL' in the local ltm logs, and on the remote syslog server as the ltm is set to syslog to the same splunk server on a different port.

However the HSL logs never seem to make it... I've tried both TCP and UDP, with no success.

The Splunk pool looks like:

(/Common)(tmos) list ltm pool splunk
ltm pool splunk {
    members {
        10.0.0.83:printer {
            address 10.0.0.83
            session monitor-enabled
            state up
        }
    }
    monitor tcp
}
And is showing a healthy status in the Gui.

Am running LTM on v11 with HF1 applied.

Any ideas???

Cheers

Gavin

application delivery
dev
devops
iRules
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Oct 25, 2011
    i don't have v11 and splunk right now but i tested it in v10 with netcat and it looked fine.

    have you seen packet if running tcpdump? 

    [root@iris:Active] config  b rule myrule list
rule myrule {
   when HTTP_REQUEST {
        
         Save Request Side Information
        
        set hsl [HSL::open -proto UDP -pool splunk]
        set http_request "\"[HTTP::method] [HTTP::uri] HTTP/[HTTP::version]\""
        set http_request_time [clock clicks -milliseconds]
        set http_user_agent "\"[HTTP::header User-Agent]]\""
        set http_host [HTTP::host]
        set http_username [HTTP::username]
        set client_ip [IP::remote_addr]
        set client_port [TCP::remote_port]
        set http_request_uri [HTTP::uri]
        set referer "\"[HTTP::header value referer]\""
}

when HTTP_RESPONSE {
        set response_time [expr {[clock clicks -milliseconds] - $http_request_time}]
        set timestamp [clock format [clock seconds] -format "%d/%h/%Y:%T %Z" -gmt 1 ]
        set virtual [virtual]
        set content_length 0
        if { [HTTP::header exists "Content-Length"] } {
                set content_length [HTTP::header "Content-Length"]
        }

        set lb_server "[LB::server addr]:[LB::server port]"
        if { [string compare "$lb_server" ""] == 0 } {
                set lb_server ""
        }

        set status_code [HTTP::status]
        set content_type [HTTP::header "Content-type"]
        set log_msg ""
        append log_msg "virtual=$virtual "
        append log_msg "$client_ip "
        append log_msg "client_port=$client_port "
        append log_msg "lb_server=$lb_server "
        append log_msg "$http_host "
        append log_msg "$http_username "
        append log_msg "\[$timestamp\] "
        append log_msg "$http_request_uri "
        append log_msg "$http_request "
        append log_msg "$status_code "
        append log_msg "content_type=$content_type "
        append log_msg "$content_length "
        append log_msg "resp_time=$response_time "
        append log_msg "$referer "
        append log_msg "$http_user_agent\n"

        log local0. "Sending log to HSL"
        HSL::send $hsl $log_msg
}
}

[root@iris:Active] config  b pool splunk list
pool splunk {
   members 192.168.206.102:shell {}
}

[root@iris:Active] config  tail /var/log/ltm
Oct 25 22:23:45 local/tmm info tmm[4672]: Rule myrule : Sending log to HSL

C:\>nc -l -u -p 514
172.28.17.30 172.28.17.33  [25/Oct/2011:14:23:45 GMT] "GET / HTTP/1.1" 200 103 "" "curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5]"
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Oct 25, 2011
    I can see the 'Sending log to HSL' in the local ltm logs, and on the remote syslog server as the ltm is set to syslog to the same splunk server on a different port. what interface does bigip send log to remote syslog server? is it mgmt or tmm interface?
  • GavinW_29074's avatar
    GavinW_29074
    Icon for Nimbostratus rankNimbostratus
    Oct 25, 2011
    Bigip is set to send to splunk using the remote-server config item using tmm...

     

     

    After I posted I found a pointer which suggested running tcpdump, and that's confirmed that the packets are indeed being sent to the splunk server.

     

    This then indicated a splunk issue...

     

     

    Having googled a bit more, I found that the suggested solution is to use syslog-ng to receive the events, and log them out to a file that splunk can then monitor...

     

     

    So having done that, i'm now getting logs coming through into splunk...

     

     

    Cheers for pointers...

     

     

    Regards

     

    Gavin
  • cdougall_14195's avatar
    cdougall_14195
    Icon for Cirrus rankCirrus
    Oct 25, 2011
    HSL, because it goes to a pool, shouldn't go out the mgmt interface.

     

     

    I mean no offence, but are you sure that they splunk server is listening on that UDP port? HSL is really a pretty basic thing in that it just sends a simple UDP stream to a server.
  • cdougall_14195's avatar
    cdougall_14195
    Icon for Cirrus rankCirrus
    Oct 25, 2011
    HSL, because it goes to a pool, shouldn't go out the mgmt interface.

     

     

    I mean no offence, but are you sure that they splunk server is listening on that UDP port? HSL is really a pretty basic thing in that it just sends a simple UDP stream to a server.
  • James_124570's avatar
    James_124570
    Icon for Nimbostratus rankNimbostratus
    Mar 19, 2015

    If you log into the shell of the LTM, run traceroute to the your syslog server. If the route goes through your mgmt ip, then you will need to setup a route to a tmm ip address.