Forum Discussion

mike_aws_119486's avatar
mike_aws_119486
Icon for Nimbostratus rankNimbostratus
Jun 29, 2018

Load Balancing Syslog Across DataCentres for nodes supporting single syslog target

First some background.

 

Two datacentres, within each we have a resilient pair of F5 BIG-IP LTM and a single syslog SIEM server.

 

The two Syslog servers are setup in a resilient solution where they can replicate syslogs between themselves and operate active/active.

 

Majority of kit supports multiple syslog targets so is configured to send to both syslog servers and we don't have to worry about any loss of logs.

 

However we have a number of devices (e.g. HP ILO boards and some Apps) which only support a single syslog destination. If we send to just on syslog server it will replicate the logs to the other, but if that server is down for maintenance or fault we lose the logs!

 

So the idea was to host a VIP on the F5 for syslog with a pool containing both Syslog servers using priority in the pool to send to local server normally, such that we end up with:

 

Node in DCA -> F5 VIP DCA -> Syslog Server DCA or Syslog Server DCB Node in DCB -> F5 VIP DCB -> Syslog Server DCB or Syslog Server DCA

 

The Syslog Servers are in a different VLAN to the F5 (e.g. the load Balancer needs to route forwarded packets) and the Nodes sending the syslog servers are all over the network.

 

Results seem to be as follows:

 

If the Node is in a different VLAN to the F5 VIP then we can see (Wireshark) the syslog packet from the Node to the VIP but never see packet leave the F5 to the syslog server in the pool.

 

If the Node is in the same VLAN to the F5 VIP then the syslog messages arrive at the syslog servers.

 

If I enable SNAT messages do arrive at the syslog servers in all scenarios BUT the source IP is shown as the F5 and hence its not possible to know the source node that sent the syslog message!

 

We have other virtual servers that load balance UDP traffic without SNAT (namely SNMP and DNS) where the clients are not on the same VLAN as the F5 but in these instances the destination is on the same VLAN as the F5 but for these we are not load balancing cross-site just within a local pool.

 

So if my testing is correct we have:

 

  • Client (VLAN2) --> F5 VIP (VLAN2) --> Syslog (VLAN3) = Works
  • Client (VLAN1) --> F5 VIP (VLAN2) --> Syslog (VLAN3) = Doesn't work
  • Client (VLAN1) --> F5 VIP (VLAN2) + SNAT --> Syslog (VLAN3) = Works but syslog message source is the SNAT

and

 

  • Client (VLANX) --> F5 VIP (VLAN2) --> SNMP (VLAN2) = Works
  • Client (VLANX) --> F5 VIP (VLAN2) --> DNS (VLAN2) = Works

It seems to me for syslog that if the Source is on the same VLAN as the VIP then the F5 will route the syslog message but if the source is on a different VLAN to the VIP it drops it.

 

And conversely for snmp/dns if the destination is on the same VLAN as the VIP then the F5 will route the traffic regardless of source.

 

I've thought of giving the F5 a Self IP in the same VLAN as the Syslog Servers which would match the DNS/SNMP configuration BUT that wouldn't help as one of the syslog servers in the pool is at the other data centre.

 

If I use DNS name and BIG-IP DNS it still lands traffic at the F5 and fails with the same behaviour.

 

Thoughts?

 

1 Reply

  • Surgeon's avatar
    Surgeon
    Ret. Employee

    If your syslog servers are in different vlan why just not to add static routes on big-ip towards syslog? If it is TCP connection then you need to add route on your syslog server back to your clients via big-ip.