F5 LTM and ASM Sentinel integration - works on one cluster, doesn't on another
Hi! I have quite complex trouble with Sentinel integration. I have 2 F5 clusters implemented as IaaS in Azure - Prod and PreProd with LTM logging took from this manual:https://my.f5.com/manage/s/article/K85539421and ASM integration took from this manual:https://community.f5.com/t5/technical-articles/integrating-the-f5-bigip-with-azure-sentinel/ta-p/282868(here only ASM part). The thing is PreProd F5 Cluster sends the logs correctly while Prod does not. The configuration is very similar for both clusters (with MGT interface, external, internal & HA via internal interface) It has been reimplemented multiple times on Prod cluster, including 4 eyes check, focusing to keep the same config on working PreProd cluster. Checking and rechecking again and again FW rules, NSGs - all should work. PreProd is working, Prod is not... Recently I started to take a look though logs, finding thousands of logs on Prod F5: Fri, 23 Jun 2023 13:34:35 GMT - warning: [telemetry] Skipped Data - Category: "LTM" | Consumers: ["My_Consumer"] | Addtl Info: "event_timestamp": "2023-06-23T13:34:35.000Z" In that moment I realized, we've had a problem with NTP, that was not working and after some TShoot we took in into backlog (probably for too long time). So - NTP can't sync to time.windows.com by url or IP (other time servers also do not work). I started to TShoot this thread. it seems NTP service is running correctly (yet I restarted it) - no change: # tmsh show /sys service ntpd * ntpd.service - start and stop ntpd Loaded: loaded (/etc/rc.d/init.d/ntpd; enabled; vendor preset: enabled) Active: active (running) since Mon 2023-06-26 16:53:06 CEST; 17s ago Process: 25697 ExecStop=/etc/rc.d/init.d/ntpd stop (code=exited, status=0/SUCCESS) Process: 25762 ExecStart=/etc/rc.d/init.d/ntpd start (code=exited, status=0/SUCCESS) CGroup: /system.slice/ntpd.service `-25766 ntpd -g What is a bit strange - NTP listens only on IPv6(?) Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 18 mgmt fe80::222:48ff:fe80:cdf4 UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 19 eth0 fe80::222:48ff:fe80:cdf4 UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 20 tmm fc00:f5::1 UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 21 eth4 fe80::6245:bdff:fe8e:24ab UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 22 eth1 fe80::222:48ff:fe80:abc4 UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 23 external fe80::222:48ff:fe80:abc4 UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 24 dev_internal fe80::6245:bdff:fe8e:24ab UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listening on routing socket on fd #41 for interface updates Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: 0.0.0.0 c016 06 restart Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: 0.0.0.0 c012 02 freq_set kernel -10.616 PPM All NTPs service trying to sync have INIT status # ntpq -np remote refid st t when poll reach delay offset jitter ============================================================================== 20.101.57.9 .INIT. 16 u - 64 0 0.000 0.000 0.000 I set time manually on all F5s (didn't solve the problem with Sentinel. PreProd works, Prod doesn't tcpdump shows that my F5s try to reach NTP server with TMM external interface, which is wrong. # tcpdump -i any host 20.101.57.9 and port 123 -vv 18:04:16.677266 IP (tos 0xc0, ttl 64, id 54613, offset 0, flags [DF], proto UDP (17), length 76) 10.10.1.4.123 > 20.101.57.9.123: [bad udp cksum 0x64fd -> 0x380e!] NTPv4, length 48 Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 10 (1024s), precision 32 Root Delay: 0.000000, Root dispersion: 0.054641, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 965835954.533013659 (2066/09/16 00:14:10) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 965835954.533013659 (2066/09/16 00:14:10) out slot1/tmm0 lis= port=1.1 trunk= According tohttps://my.f5.com/manage/s/article/K92145845it should use MGT interface and written there cause is a lack of MGT route. But this is not my example as I do have MGT routes set correctly (I guess). Below there is one route towards Azure service endpoint and 2nd is default route for MGT: # tmsh list /sys management-route sys management-route azure-metadata { gateway 10.0.0.1 network 169.254.169.254/32 } sys management-route default { gateway 10.0.0.1 network default } After this moment I started to get confused, what might be the problem and is it really NTP related. Any ideas?
NTP time difference between Active / Standby appliance
Hello Folks, It has been quite sometime when I post a question on DevCentral. So here we go, I have 2 F5 A/S appliances. Both are pointing to 2 NTP servers as 10.13.1.11 and 10.13.1.12. NTP traffic is passing through other than Management Interface so far. What has been observed is, Active appliance is showing correct time as per the NTP server, however the standby appliance is showing 3 minutes of difference compare to the Active appliance. Even you sync the time manually, it slowly experience delay, and time difference start getting increased between both of the appliances. Following is the result of the NTPQ -PN from Active and standby. [root@Company_F5-3900U1:Active:In Sync] config ntpq -np remote refid st t when poll reach delay offset jitter ============================================================================== 10.13.1.12 10.13.1.11 2 u 19 64 377 2.077 -43725. 11.051 10.13.1.11 .LOCL. 1 u 59 64 [root@Company_F5-3900U2:Standby:In Sync] config ntpq -pn remote refid st t when poll reach delay offset jitter ============================================================================== 10.13.1.12 10.13.1.11 2 u 18 64 377 1.947 -222609 33.485 10.13.1.11 .LOCL. 1 u 21 64 377 1.953 -222588 28.426 If you see the Offset value on Standby appliance, that justifies the fact why there is a time difference between Active and Standby unit. The NTP servers are reachable via a DMZ interface i.e. 1.1. Both of the appliances have same configured of interfaces along with the media settings. Following is the interface property of both of the appliances, which is exactly same. root@(Company_F5-3900U2)(cfg-sync In Sync)(Standby)(/Common)(tmos) list net interface 1.1 net interface 1.1 { if-index 80 mac-address 00:01:d7:e6:6d:44 media-active 1000T-FD media-max 1000T-FD mtu 1800 } Any help? Darshan
NTP synchronization
Hi Guyz, Please I need some help to get time synchronization from NTP servers on my F5 from NTP servers. I have two F5 LTM running version 10.2.4 working in production as redundancy, one of them getting time from NTP server as well and other one failed. Also there is new IPs for NTP server I have added them but getting failed on both F5, port 123 is opened for both F5. Standby F5 name is (riyadh-f5b)(its IP:10.6.140.240) failed on old and new IPs, Active F5 name (riyadh-f5b)(its IP:10.6.140.241) successfully getting time from old NTP servers but also failed on new IPs. Old IPs: 10.1.0.1 & 10.1.0.1 New IPs: 10.1.9.11 & 10.1.9.12 I will attached some info from both F5 and if there is more information required please let me know. Standby F5a: [root@riyadh-f5a:/S1-green-P:Standby] config ntpq -p remote refid st t when poll reach delay offset jitter 10.1.0.1 .INIT. 16 u - 1024 0 0.000 0.000 0.000 10.1.0.2 .INIT. 16 u - 1024 0 0.000 0.000 0.000 10.1.9.11 .INIT. 16 u - 1024 0 0.000 0.000 0.000 10.1.9.12 .INIT. 16 u - 1024 0 0.000 0.000 0.000 slot1 .INIT. 16 u - 1024 0 0.000 0.000 0.000 slot2 .INIT. 16 u - 1024 0 0.000 0.000 0.000 slot3 .INIT. 16 u - 1024 0 0.000 0.000 0.000 slot4 .INIT. 16 u - 1024 0 0.000 0.000 0.000 [root@riyadh-f5a:/S1-green-P:Standby] config ntpdate 30 Jun 15:39:09 ntpdate[30699]: no servers can be used, exiting [root@riyadh-f5a:/S1-green-P:Standby] config ntpstat unsynchronised time server re-starting polling server every 64 s [root@riyadh-f5a:/S1-green-P:Standby] config ntpdate 30 Jun 15:53:23 ntpdate[1060]: no servers can be used, exiting [root@riyadh-f5a:/S1-green-P:Standby] config ntptrace localhost.localdomain: stratum 16, offset 0.000000, synch distance 1.434780 [root@riyadh-f5a:/S1-green-P:Standby] config cat ntp.conf THIS IS AN AUTO-GENERATED FILE -- DO NOT EDIT!!! Use the bigpipe shell utility to make changes to the system configuration. For more information, see bigpipe ntp help. Permit time synchronization with our time source, but do not permit the source to query or modify the service on this system. restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery Permit all access over the loopback interface. This could be tightened as well, but to do so would effect some of the administrative functions. restrict 127.0.0.1 restrict -6 ::1 Allow queries from the TMM and SCCP. restrict 127.1.1.2 nomodify notrap restrict 127.2.0.1 nomodify notrap --- GENERAL CONFIGURATION --- Undisciplined Local Clock. This is a fake driver intended for backup and when no outside source of synchronized time is available. The default stratum is usually 3, but in this case we elect to use stratum 0. Since the server line does not have the prefer keyword, this driver is never used for synchronization, unless no other other synchronization source is available. In case the local host is controlled by some external source, such as an external oscillator or another protocol, the prefer keyword would cause the local host to disregard all other synchronization sources, unless the kernel modifications are in use and declare an unsynchronized condition. server 10.1.0.1 iburst server 10.1.0.2 iburst server 10.1.9.11 iburst server 10.1.9.12 iburst peer 127.3.0.1 peer 127.3.0.2 peer 127.3.0.3 peer 127.3.0.4 Drift file. Put this in a directory which the daemon can write to. No symbolic links allowed, either, since the daemon updates the file by creating a temporary in the same directory and then rename()'ing it to the file. driftfile /var/lib/ntp/drift broadcastdelay 0.008 Keys file. keys /etc/ntp/keys Active F5b: [root@riyadh-f5b:/S1-green-P:Active] config ntpq -np remote refid st t when poll reach delay offset jitter +10.1.0.1 10.64.0.4 4 u 199 1024 377 1.821 -6.901 1.293 *10.1.0.2 10.64.0.4 4 u 293 1024 377 1.849 -4.921 1.882 10.1.9.11 .INIT. 16 u - 1024 0 0.000 0.000 0.000 10.1.9.12 .INIT. 16 u - 1024 0 0.000 0.000 0.000 127.3.0.1 .INIT. 16 u - 1024 0 0.000 0.000 0.000 127.3.0.2 .INIT. 16 u - 1024 0 0.000 0.000 0.000 127.3.0.3 .INIT. 16 u - 1024 0 0.000 0.000 0.000 127.3.0.4 .INIT. 16 u - 1024 0 0.000 0.000 0.000 [root@riyadh-f5b:/S1-green-P:Active] config ntpdate 30 Jun 16:20:48 ntpdate[10040]: no servers can be used, exiting [root@riyadh-f5b:/S1-green-P:Active] config ntpstat synchronised to NTP server (10.1.0.2) at stratum 5 time correct to within 92 ms polling server every 1024 s [root@riyadh-f5b:/S1-green-P:Active] config ntptrace localhost.localdomain: stratum 5, offset 0.005461, synch distance 0.100876 10.1.0.2: stratum 4, offset 0.000676, synch distance 0.279300 10.64.0.4: timed out, nothing received ***Request timed out [root@riyadh-f5b:/S1-green-P:Active] config cat ntp.conf THIS IS AN AUTO-GENERATED FILE -- DO NOT EDIT!!! Use the bigpipe shell utility to make changes to the system configuration. For more information, see bigpipe ntp help. Permit time synchronization with our time source, but do not permit the source to query or modify the service on this system. restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery Permit all access over the loopback interface. This could be tightened as well, but to do so would effect some of the administrative functions. restrict 127.0.0.1 restrict -6 ::1 Allow queries from the TMM and SCCP. restrict 127.1.1.2 nomodify notrap restrict 127.2.0.1 nomodify notrap --- GENERAL CONFIGURATION --- Undisciplined Local Clock. This is a fake driver intended for backup and when no outside source of synchronized time is available. The default stratum is usually 3, but in this case we elect to use stratum 0. Since the server line does not have the prefer keyword, this driver is never used for synchronization, unless no other other synchronization source is available. In case the local host is controlled by some external source, such as an external oscillator or another protocol, the prefer keyword would cause the local host to disregard all other synchronization sources, unless the kernel modifications are in use and declare an unsynchronized condition. server 10.1.0.1 iburst server 10.1.0.2 iburst server 10.1.9.11 iburst server 10.1.9.12 iburst peer 127.3.0.1 peer 127.3.0.2 peer 127.3.0.3 peer 127.3.0.4 Drift file. Put this in a directory which the daemon can write to. No symbolic links allowed, either, since the daemon updates the file by creating a temporary in the same directory and then rename()'ing it to the file. driftfile /var/lib/ntp/drift broadcastdelay 0.008 Keys file. keys /etc/ntp/keys
What is the Difference between Date and Clock?
What is the difference between date and clock command. What would cause the two to me different and how do I sync them up? ltmA# clock Tue 19 Oct 2021 12:13:53 AM UTC-0.001128 seconds ltmA# date Tue Oct 19 13:04:43 UTC 2021 # tmsh list /sys ntp servers sys ntp { servers { ntp.local.domain } remoterefidst t when poll reachdelayoffsetjitter ============================================================================== *10.211.252.9138.236.128.363 u774 102412.682-0.5650.530
NTP fails with Windows Server
Hello guys, I installed two BIP-IP last week (16.0.1), cannot synchronize with NTP servers (Windows Server). [root@LB01:Peer Time Out of Sync:Changes Pending] config # ntpq -np remoterefidst t when poll reachdelayoffsetjitter ============================================================================== 172.16.155.42.INIT.16 u596400.0000.0000.000 172.16.155.45.INIT.16 u586400.0000.0000.000 Came across this article (https://support.f5.com/csp/article/K03294981), did the change on f5 a few hours ago, but still no luck... Any chance someone had the same problem and was able to resolve it? Thank you!
Are NTP and DNS traffic management type or not?
Hello everyone, I'm system engineer in integrator company and currently I have one PoC of AWAF project with a customer. I have little experience of working with f5 devices, so I have one question and it'll help me a lot in future to analyze how BIG-IP devices. I've done some research in documentations but I couldn't find clear answer on topics, which type of traffic is considered as Data Traffic and which one is Management? For example NTP and DNS traffic should use management route or TMM route (I mean the case when there is no direct path to the destination DNS/NTP servers)? I thought that BIG-IP devices will use management route (management gateway) to do DNS queries and time synchronization, so I asked customer to grant access on firewall from management interface to the destination servers, but it didn't work. Then I've captured traffic via tcpdump and I realized that BIG-IP devices try to use TMM default route instead. But I've read in this article - https://support.f5.com/csp/article/K13284 that NTP is management traffic. Also this article - https://support.f5.com/csp/article/K7017 says that during the device boot, ntpd daemon is starting before TMM, so if it has no route via management interface, time synchronization will fail. So, I'm a little confused, what should I ask customer, open access from TMM interface for DNS, NTP, also for Signature Updates? I just do not understand logically, why NTP, DNS and system update do not use management routes? If all of them are considered as a data traffic, than what is management route used for? Only for accessing management GUI and SSH, is that correct? Sorry for a long question, but I really want to understand the platform's logic of traffic routing, to be able to operate it and correctly implement it with the customer. Thanks in advance. // Giorgi
