Forum Discussion

Christian_15126's avatar
Christian_15126
Icon for Nimbostratus rankNimbostratus
Dec 10, 2012

f5 ltm sends syslog messages with local/ causing indexing issues with SPlunk

We are trying to point all our f5's to Splunk for syslog, but the default remote servers option is sending local/ in the syslog messages causing indexing issues with Splunk (it's reading them as local messages instead of from remote hosts). When I try to use a modified include statement with a template, the syslog messages not only remove the local/, they also remove the hostname and duplicate the severity level.

 

With default syslog logging I get:

 

remote-servers {

 

Splunk {

 

host 10.10.60.111

 

local-ip none

 

remote-port 514

 

}

 

Msg: Dec 10 08:14:48 local/bpeca03-f501 debug snmpd[3542]: error on subcontainer 'ia_addr' insert (-1)\012

 

08:14:48.438475 00:01:d7:d5:ec:41 > 00:23:5e:56:76:bf, ethertype IPv4 (0x0800), length 138: (tos 0x0, ttl 64, id 34987, offset 0, flags [DF], proto: UDP (17), length: 124) 10.12.254.253.37261 > 10.10.60.111.syslog: SYSLOG, length: 96

 

Facility daemon (3), Severity debug (7)

 

Using the include statement:

 

include "template t_remotetmpl { template(\"<$PRI> $DATE $HOST $PRIORITY $MSG\\n\"); template_escape(no);};filter f_remote_loghost { level(debug..emerg);};destination d_remote_loghost { udp(\"10.10.60.111\" port(514) template(t_remotetmpl));};log { source(s_syslog_pipe); filter(f_remote_loghost); destination(d_remote_loghost);};"

 

The local/ is removed, but the $HOST variable isn't read properly (just shows up as local), and the $PRIORITY variable is duplicated in the syslog message:

 

Msg: Dec 10 08:14:48 local debug debug snmpd[3542]: error on subcontainer 'ia_addr' insert (-1)\012

 

08:15:18.439271 00:01:d7:d5:ec:41 > 00:23:5e:56:76:bf, ethertype IPv4 (0x0800), length 144: (tos 0x0, ttl 64, id 34987, offset 0, flags [DF], proto: UDP (17), length: 130) 10.12.254.253.43748 > 10.10.60.111.syslog: SYSLOG, length: 102

 

Facility daemon (3), Severity debug (7)

 

 

Has anyone see n this issue before? Any help would be appreciated. Thanks!

 

CH

 

 

 

management
monitoring

4 Replies

  • hoolio's avatar
    hoolio
    Icon for Cirrostratus rankCirrostratus
    Dec 10, 2012
    Hi Christian,

     

     

    Which LTM version are you seeing this on?

     

     

    Thanks, Aaron
  • Christophe_Thys's avatar
    Christophe_Thys
    Icon for Nimbostratus rankNimbostratus
    Jan 30, 2013

    Hi

     

     

    I've got the same issue on 10.2.3 Build 112. My parsing engine cannot handle the hostname part containing "local/" in the hostname part.

     

    Syslog message example:

     

    Jan 30 15:14:37 local/MyHostname info logger: [ssl_req][30/Jan/2013:15:14:37 +0100] 192.168.0.3 TLSv1 DHE-RSA-AES256-SHA "POST /xxxx.cgi HTTP/1.1" 437

     

    Any way we can remove that "local/" part?

     

    Regards,

     

    Christophe

     

  • hoolio's avatar
    hoolio
    Icon for Cirrostratus rankCirrostratus
    Jan 30, 2013
    Can you try the fix that Nitass suggested here?

     

     

    https://devcentral.f5.com/community/group/aft/2165556/asg/442272721

     

     

    Aaron
  • Christian_15126's avatar
    Christian_15126
    Icon for Nimbostratus rankNimbostratus
    Nov 14, 2013

    So this turned out to be a completely managed solution issue with splunk and the vendor who was managing it. They were doing some crazy regex filters to segregate traffic between indexes, and the regexes couldn't handle the BIG-IP adding in the local/ into the syslog messages and was unable to parse them properly. We were able to manually remove the local/ from the syslog-ng.conf file, but as you guys know everytime you restart the syslog-ng service (or reboot and restart all services), the file gets overwritten and the local/ came back). In the end it was a moot point as once we brought splunk in-house and used a traditional indexing design without funky regexes, the issue went away. Thx for the help though!