Forum Discussion

GavinW_29074's avatar
GavinW_29074
Icon for Nimbostratus rankNimbostratus
Oct 25, 2011

HSL not working?

Hi there,

I'm trying to set-up an iRule to log-out access logs to a splunk syslog server using HSL.

However I'm having issues with the events not getting to the syslog server...

The iRule is:

 when HTTP_REQUEST {
   
    Save Request Side Information
   
   set hsl [HSL::open -proto UDP -pool splunk]
   set http_request "\"[HTTP::method] [HTTP::uri] HTTP/[HTTP::version]\""
   set http_request_time [clock clicks -milliseconds]
   set http_user_agent "\"[HTTP::header User-Agent]]\""
   set http_host [HTTP::host]
   set http_username [HTTP::username]
   set client_ip [IP::remote_addr]
   set client_port [TCP::remote_port]
   set http_request_uri [HTTP::uri]
   set referer "\"[HTTP::header value referer]\""
}
when HTTP_RESPONSE {
   set response_time [expr [clock clicks -milliseconds] - $http_request_time]
   set timestamp [clock format [clock seconds] -format "%d/%h/%y:%T %Z" -gmt 1 ]
   set virtual [virtual]
   set content_length 0
   if { [HTTP::header exists "Content-Length"] } {
      set content_length [HTTP::header "Content-Length"]
   }
   set lb_server "[LB::server addr]:[LB::server port]"
   if { [string compare "$lb_server" ""] == 0 } {
      set lb_server ""
   }
   set status_code [HTTP::status]
   set content_type [HTTP::header "Content-type"]
   set log_msg ""
   append log_msg "virtual=$virtual "
   append log_msg "$client_ip "
   append log_msg "client_port=$client_port "
   append log_msg "lb_server=$lb_server "
   append log_msg "$http_host "
   append log_msg "$http_username "
   append log_msg "\[$timestamp\] "
   append log_msg "$http_request_uri "
   append log_msg "$http_request "
   append log_msg "$status_code "
   append log_msg "content_type=$content_type "
   append log_msg "$content_length "
   append log_msg "resp_time=$response_time "
   append log_msg "$referer "   
   append log_msg "$http_user_agent\n"
   log local0. "Sending log to HSL"
   HSL::send $hsl $log_msg
} 

I can see the 'Sending log to HSL' in the local ltm logs, and on the remote syslog server as the ltm is set to syslog to the same splunk server on a different port.

However the HSL logs never seem to make it... I've tried both TCP and UDP, with no success.

The Splunk pool looks like:

(/Common)(tmos) list ltm pool splunk
ltm pool splunk {
    members {
        10.0.0.83:printer {
            address 10.0.0.83
            session monitor-enabled
            state up
        }
    }
    monitor tcp
}
And is showing a healthy status in the Gui.

Am running LTM on v11 with HF1 applied.

Any ideas???

Cheers

Gavin

  • i don't have v11 and splunk right now but i tested it in v10 with netcat and it looked fine.

    have you seen packet if running tcpdump?

    [root@iris:Active] config  b rule myrule list
    rule myrule {
       when HTTP_REQUEST {
            
             Save Request Side Information
            
            set hsl [HSL::open -proto UDP -pool splunk]
            set http_request "\"[HTTP::method] [HTTP::uri] HTTP/[HTTP::version]\""
            set http_request_time [clock clicks -milliseconds]
            set http_user_agent "\"[HTTP::header User-Agent]]\""
            set http_host [HTTP::host]
            set http_username [HTTP::username]
            set client_ip [IP::remote_addr]
            set client_port [TCP::remote_port]
            set http_request_uri [HTTP::uri]
            set referer "\"[HTTP::header value referer]\""
    }
    
    when HTTP_RESPONSE {
            set response_time [expr {[clock clicks -milliseconds] - $http_request_time}]
            set timestamp [clock format [clock seconds] -format "%d/%h/%Y:%T %Z" -gmt 1 ]
            set virtual [virtual]
            set content_length 0
            if { [HTTP::header exists "Content-Length"] } {
                    set content_length [HTTP::header "Content-Length"]
            }
    
            set lb_server "[LB::server addr]:[LB::server port]"
            if { [string compare "$lb_server" ""] == 0 } {
                    set lb_server ""
            }
    
            set status_code [HTTP::status]
            set content_type [HTTP::header "Content-type"]
            set log_msg ""
            append log_msg "virtual=$virtual "
            append log_msg "$client_ip "
            append log_msg "client_port=$client_port "
            append log_msg "lb_server=$lb_server "
            append log_msg "$http_host "
            append log_msg "$http_username "
            append log_msg "\[$timestamp\] "
            append log_msg "$http_request_uri "
            append log_msg "$http_request "
            append log_msg "$status_code "
            append log_msg "content_type=$content_type "
            append log_msg "$content_length "
            append log_msg "resp_time=$response_time "
            append log_msg "$referer "
            append log_msg "$http_user_agent\n"
    
            log local0. "Sending log to HSL"
            HSL::send $hsl $log_msg
    }
    }
    
    [root@iris:Active] config  b pool splunk list
    pool splunk {
       members 192.168.206.102:shell {}
    }
    
    [root@iris:Active] config  tail /var/log/ltm
    Oct 25 22:23:45 local/tmm info tmm[4672]: Rule myrule : Sending log to HSL
    
    C:\>nc -l -u -p 514
    172.28.17.30 172.28.17.33  [25/Oct/2011:14:23:45 GMT] "GET / HTTP/1.1" 200 103 "" "curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5]"
    
    
  • I can see the 'Sending log to HSL' in the local ltm logs, and on the remote syslog server as the ltm is set to syslog to the same splunk server on a different port. what interface does bigip send log to remote syslog server? is it mgmt or tmm interface?
  • Bigip is set to send to splunk using the remote-server config item using tmm...

     

     

    After I posted I found a pointer which suggested running tcpdump, and that's confirmed that the packets are indeed being sent to the splunk server.

     

    This then indicated a splunk issue...

     

     

    Having googled a bit more, I found that the suggested solution is to use syslog-ng to receive the events, and log them out to a file that splunk can then monitor...

     

     

    So having done that, i'm now getting logs coming through into splunk...

     

     

    Cheers for pointers...

     

     

    Regards

     

    Gavin
  • HSL, because it goes to a pool, shouldn't go out the mgmt interface.

     

     

    I mean no offence, but are you sure that they splunk server is listening on that UDP port? HSL is really a pretty basic thing in that it just sends a simple UDP stream to a server.
  • HSL, because it goes to a pool, shouldn't go out the mgmt interface.

     

     

    I mean no offence, but are you sure that they splunk server is listening on that UDP port? HSL is really a pretty basic thing in that it just sends a simple UDP stream to a server.
  • If you log into the shell of the LTM, run traceroute to the your syslog server. If the route goes through your mgmt ip, then you will need to setup a route to a tmm ip address.