BIGIP unable to send tcp/udp packets to syslog servers

[prajith_nsg]

Hi Team,

We have deployed partner solution https://aws.amazon.com/solutions/partners/f5-big-ip-ve/ in AWS and HA mode has come up as expected, we can access management UI and configure our backend. However, we are unable to see traffic being sent from BIGIP to our syslog server backend. 

We would like to use BIG IP as TCP/UDP HA load balancer and send traffic to our backend (syslog servers) which sends syslogs to Kibana. 

Client request : [ec2-user@ip-10-0-xx-xx ~]$ while date "+INFO Syslog check for F5 BIGIP on HA mode %T" | logger -s -p user.info -n <VIP_Configured_in_BIGIP> -P 514 ; do sleep 1; done

Client --> [udp protocol 514] --> BIGIP (UDP virtual host has been created) --> syslog server (listening on 5514 port)

In kibana, we can see BIG IP syslog however, my test requests are not being sent from BIG IP to syslog server (nsg-agent) 

What are we missing the configuration, we are assuming the HA configuration from https://aws-quickstart.github.io/quickstart-f5-big-ip-virtual-edition-ha/ should be working 

[CA_Valli]

Hello @prajith_nsg , I see that you're using a Stateless virtual server. Port translation setting is not supported and is known to not work as expected if enabled on the stateless virtual server. (Check this KB for reference)

You can run this command to display the connection table and confirm if port translation is happening or not while the connection is active 

tmsh show /sys connection cs-client-addr <your ec2-user client address 10-0-xx-xx> 


tmsh show /sys connection cs-client-addr 10.0.113.100
Sys::Connections
10.0.113.100:49236  10.0.113.200:5555  10.0.113.100:61501  192.168.1.152:80  tcp  4  (tmm: 0)  none  none
Total records returned: 1

In this example, Client information below: 
SRC IP ADDRESS / PORT (Client to F5): 10.0.113.100:49236
DST IP ADDRESS / PORT (Client to F5): 10.0.113.200:5555

BIG-IP will receive this request and after making a load balance decision it will initiate a connection with the pool member with the following details:
SRC IP ADDRESS / PORT (F5 to server) : 10.0.113.100:61501
DST IP ADDRESS / PORT (F5 to server) : 192.168.1.152:80

If you need port translation to be effective, you should run a different type of virtual server -- fastL4 should work IMO.
If it's not mandatory, you can try changing VS port to 5514 and see if client test is being forwarded as intended. 

[prajith_nsg]

Hi @CA_Valli, Thank you so much for taking time to respond. 

I have changed the virtual server type to use fastL4, attached is the screenshot of configuration. 

From my basiton host,  I am able to reach bigIP on VIP address (10.0.10.101) on port 5514 as expected. 

[ec2-user@ip-10-0-10-251 ~]$ nc -v -i 1 -w 1 10.0.10.101 5514
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.10.101:5514.
Ncat: Idle timeout expired (1000 ms).

 Similarly from my BIGIP box, I am reach backend on port 5514 (syslog server port) 

[admin@failover01:Active:In Sync] ~ # nc -v -i 1 -w 1 10.0.12.112 5514
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 10.0.12.112:5514.
Ncat: Idle timeout expired (1000 ms).
[admin@failover01:Active:In Sync] ~ # nc -v -i 1 -w 1 10.0.22.74 5514
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 10.0.22.74:5514.
Ncat: Idle timeout expired (1000 ms).

The client information shows port translation works as expected now. 

[admin@failover01:Active:In Sync] ~ # tmsh show /sys connection cs-client-addr 10.0.10.251
Sys::Connections
10.0.10.251:60134  10.0.10.101:5514  10.0.10.11:60134  10.0.12.112:5514  tcp  0  (tmm: 1)  none  none
Total records returned: 1

 However, when I am using logger command to generate a log message manually from basiton host(ip-10-0-10-251). Targeting VIP configured on bigip : 10.0.10.101 

[ec2-user@ip-10-0-10-251 ~]$ while date "+INFO Syslog check for F5 BIGIP on HA mode %T" | logger -s -n 10.0.10.101 -P 5514; do sleep 1; done

<13>1 2022-12-20T12:03:27.277895+00:00 ip-10-0-10-251.ap-south-1.compute.internal ec2-user - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="6480"] INFO Syslog check for F5 BIGIP on HA mode 12:03:27
<13>1 2022-12-20T12:03:28.283155+00:00 ip-10-0-10-251.ap-south-1.compute.internal ec2-user - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="6980"] INFO Syslog check for F5 BIGIP on HA mode 12:03:28
<13>1 2022-12-20T12:03:29.286621+00:00 ip-10-0-10-251.ap-south-1.compute.internal ec2-user - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="7480"] INFO Syslog check for F5 BIGIP on HA mode 12:03:29

 Capturing Tcpdump on BIG IP box  doesn't show the any packets reaching bigip

#TCPDUMP command to check for traffic from the client to the VIP and from VIP to the backend pool members

[admin@failover01:Active:In Sync] ~ # tcpdump -nni 0.0:nnnp -s 0 host 10.0.10.101 and host 10.0.22.74

and 

[admin@failover01:Active:In Sync] ~ # tmsh show /sys connection cs-client-addr 10.0.10.251
Sys::Connections
Total records returned: 0

Any specific configuration is missing to cause this?

I want my syslogs to be sent to remote-server which in turn will be sent to kibana dashboard.

[admin@failover01:Active:In Sync] ~ # tmsh list sys syslog
sys syslog {
    remote-servers {
        remotesyslog1 {
            host 10.0.12.112
            remote-port 5514
        }
        remotesyslog2 {
            host 10.0.22.74
            remote-port 5514
        }
    }
}

[prajith_nsg]

@CA_Valli  - I got it working after changing the security group to use UDP instead of TCP. However, our agents can take incoming request on port 5514 on UDP as well as TCP. I highly appreciate your inputs, i will take it from here and figure out. Thanks again. 🙂 

[CA_Valli · Accepted solution]

I think you should be able to run the same configuration with "All protocols" and match both TCP+UDP traffic

[prajith_nsg]

Thanks you @CA_Valli, yes that works too. I have tested my requests are landing on nsg-agent and syslogs are populated in kibana as expected. Appreciated your valueable inputs.