Forum Discussion

Joseph_Lindsly's avatar
Mar 03, 2020

Intermittent errors sending syslogs to Splunk

I have an irule configured on the LTM that I apply to most of the Virtual Servers that sends the traffic logs to a Splunk server. Lately, i have noticed logs not being in Splunk. On the splunk server, i have created several Data Input Ports other than UDP/514. If i go to the server pool that contains the splunk server and change the port number, as soon as it updates, I start seeing the logs in the splunk again. This will work for awhile and then it would happen again. I would go into the server pool and change to a different port and then it would start working. I ran a capture on the splunk server during the time that it wasn't working and i see the logs coming from the LTM, but the protocol is WHO instead of UDP. When it is working, the captures show the protocol as UDP. I thought it was an issue with Splunk, so i opened a case with them. While waiting for a response from Splunk support, the issue happened again and this time i ran a capture on the LTM. The LTM capture also shows the protocol as WHO instead of UDP. Here is a copy of an entry from the capture.

 

11 2020-02-28 08:49:12.408140 10.42.104.11 10.42.104.230 WHO 249 14.82.39:53753 -> SNAT: 10.48.20: 8251108.39 5398359.36 14476452.42

 

Has anyone ever experienced this before? We have LTMs in other environments that are running the same irule and i never had this issue with them.

 

For reference, here is the irule config:

 

when HTTP_REQUEST {

  # set the URL here, log it on the response

  set url [HTTP::header Host][HTTP::uri]

  set vip [IP::local_addr]:[TCP::local_port]

  set hsl [HSL::open -proto UDP -pool splunk]

 

}

 

when HTTP_RESPONSE {

  set client [IP::client_addr]:[TCP::client_port]

  set snat [IP::local_addr]

  set node [IP::server_addr]:[TCP::server_port]

  set nodeResp [HTTP::status]

  set hsl [HSL::open -proto UDP -pool splunk]

 

  # log connection info

  HSL::send $hsl "Client: $client -> SNAT: $snat -> VIP: $vip -> URL: $url -> Node: $node with response $nodeResp"

}

 

Thanks

No RepliesBe the first to reply