Deploying BIG-IP Telemetry Streaming with Azure Sentinel as its consumer.
AZURE SENTINEL and BIG-IP ...with Telemetry Streaming! This work was completed as a collaboration of Remo Mattei r.mattei@f5.com and Bill Wester b.wester@f5.com, feel free to email us if you have questions. One of the things that I have discovered recently is how neat it is to be able to leverage Azures new Sentinel to receive and display telemetry data from F5's BIG-IP devices. The devices don't even have to be in Azure, you could have dedicated hardware BIG-IPs and still send via Telemetry Streaming to Sentinel as your destination for statistics and logs. Let us explore a bit more on how to get all of the moving pieces together to a single cohesive implementation. Telemetry Streaming is a way for you to forward events and statistics from the BIG-IP system to your preferred data consumer and visualization application. You can do all of this byPOSTinga single JSON declaration to a declarative REST API endpoint. Telemetry Streaming uses a declarative model, meaning you provide a JSON declaration rather than a set of imperative commands. More info can be found here: https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/userguide/about-telemetry.html BIG-IP allows you to send logs to several external providers. Splunk, awell knownone, is one of the most used out there. However, the new Azure Sentinel, a cloud solution, is something that many customers can take advantages from.This section,will help in understanding on how to setup BIG-IP to get the logs to Azure Sentinel. Setup BIG-IP First of all, this is broken into two parts, one shows the logs of the BIG-IP System Metrics, like what OS, what modules are installed etc. The second, is about themodule ASM. The two have a few things in common. They use the TS RPM file which is added to the BIG-IP, and the declaration, which tells the BIG-IP where to send the stream of data. To send data relate to BIG-IP System Metrics it is required to have AVR provisioned on the device. ASM is not required but we use it here as an example of how to enable another module. Here is a screenshot from the Azure which shows the required modules.One more important thing is that ASM will need to have AFM also enabled otherwise you will not get logs in Azure. ASM Once enabled the required modules it will show System Metrics Common components that you must install for this to work First you need Telemetry Streaming: The TS RPM can be found here on GITHUB: https://github.com/F5Networks/f5-telemetry-streaming/releases/ You can use Visual Studio Code to install the RPM or your favorite way... Here are some screen shots form VS Code, using the F5 Plugin. NOTE: in order to useVSCodeto push AS3, DOetcyou must install the F5 Plugin. Use the command options in Mac it’scommand+shift+P(here you can search for RPM by just typing it in the box) Select AS3and make sure to install both AS3 and TS: Select the version: (probably latest is best here) The Telemetry Streaming declaration looks like this: { "class": "Telemetry", "My_Listener": { "class": "Telemetry_Listener", "port": 6514 }, "Poller": { "class": "Telemetry_System_Poller", "interval": 60, "enable": true, "trace": false, "allowSelfSignedCert": false, "host": "localhost", "port": 8100, "protocol": "http", "actions": [ { "enable": true, "includeData": {}, "locations": { "system": true, "virtualServers": true, "httpProfiles": true, "clientSslProfiles": true, "serverSslProfiles": true } } ] }, "Pull_Consumer": { "class": "Telemetry_Pull_Consumer", "type": "default", "systemPoller": [ "Poller" ] }, "Azure_Consumer": { "class": "Telemetry_Consumer", "type": "Azure_Log_Analytics", "workspaceId": "workspaceID", "passphrase": { "cipherText": "primkey" } }, "schemaVersion": "1.12.0" } NOTE: You will need to get theworkspaceIDand theprimarykey. You can use the azure cli for that: azmonitor log-analytics workspace list --out table CustomerId Location Name ProvisioningState PublicNetworkAccessForIngestion PublicNetworkAccessForQuery ResourceGroup RetentionInDays ------------------------------------ ------------- ---------------------------------------------------------- ------------------- --------------------------------- ----------------------------- ------------------------- ----------------- a05d4bfb-27c8-49a6-96e2-351d2dc78c61 eastus adrianLA Succeeded Enabled Enabled adrian_rg_01 7 63be43ed-b3f5-4e9f-bc92-226bb3393d11 eastus DefaultWorkspace-77c6ebef-d849-4527-a355-742d8d7d3fdc-EUS Succeeded Enabled Enabled defaultresourcegroup-eus 30 2ccbd35a-dfdf-4a5e-ab5f-1d5314f52e4b southeastasia DefaultWorkspace-77c6ebef-d849-4527-a355-742d8d7d3fdc-SEA Succeeded Enabled Enabled defaultresourcegroup-sea 30 9436f742-069a-4e29-aac0-e1258f7b1f87 westus2 calalangakslog Succeeded Enabled Enabled calalang-rg 30 ac071b51-f0c6-43b6-8bef-16b9197fde0f westus2 edgar-log Succeeded Enabled Enabled defaultresourcegroup-eus 31 555ae8d5-75bc-4058-becf-df510c09f8d3 westus2 DefaultWorkspace-77c6ebef-d849-4527-a355-742d8d7d3fdc-WUS2 Succeeded Enabled Enabled defaultresourcegroup-wus2 30 f633bdb1-d560-43cd-a664-cc7a93ed8781 westus2 edgar-log-analytics Succeeded Enabled Enabled edgar-rg 30 9334eb7c-16fc-4db9-a84f-5824a7177ccb centralus DefaultWorkspace-77c6ebef-d849-4527-a355-742d8d7d3fdc-CUS Succeeded Enabled Enabled defaultresourcegroup-cus 30 091c2cf3-853d-4297-9001-41d2109c28ec westus DefaultWorkspace-77c6ebef-d849-4527-a355-742d8d7d3fdc-WUS Succeeded Enabled Enabled defaultresourcegroup-wus 30 52471748-d9c7-46ba-9f9f-72ed8e92a201 westus remo-analytics Succeeded Enabled Enabled remo-telemetry 30 bc8e90ca-f59c-4fbf-a28b-213fe1cfcfda westus wester-log Succeeded Enabled Enabled wester_rg 30 Here you can see the name of the resource group then run the following command: azmonitor log-analytics workspace get-shared-keys --resource-groupwester_rg--workspace-name wester-log Which will print out theprimarykey The workspace isCustomerIdfrom the main table. To install this declaration you can use POSTMAN, curl, or Visual Studio Code; we used Visual Studio Code. Copy the text into a newVScodetab, make sure it’s in json format and then use the command pallet to post it Verify by using the TS version at the bottom ofVSCode, it will execute a GET to the BIG-IP that is connected. ASM In order to use ASM you will need to configure a VIP with the IP of 255.255.255.254, and the port to the 6514, as well as aniRule. This can be done with an AS3 declaration or TMSH. Sample of AS3 declaration { "class":"ADC", "schemaVersion":"3.10.0", "remark":"Example depicting creation of BIG-IP module log profiles", "Common": { "Shared": { "class":"Application", "template":"shared", "telemetry_local_rule": { "remark":"Only required when TS is a local listener", "class":"iRule", "iRule":"when CLIENT_ACCEPTED {\nnode127.0.0.1 6514\n}" }, "telemetry_local": { "remark":"Only required when TS is a local listener", "class":"Service_TCP", "virtualAddresses": [ "255.255.255.254" ], "virtualPort":6514, "iRules": [ "telemetry_local_rule" ] }, "telemetry": { "class":"Pool", "members": [ { "enable":true, "serverAddresses": [ "255.255.255.254" ], "servicePort":6514 } ], "monitors": [ { "bigip":"/Common/tcp" } ] }, "telemetry_hsl": { "class":"Log_Destination", "type":"remote-high-speed-log", "protocol":"tcp", "pool": { "use":"telemetry" } }, "telemetry_formatted": { "class":"Log_Destination", "type":"splunk", "forwardTo": { "use":"telemetry_hsl" } }, "telemetry_publisher": { "class":"Log_Publisher", "destinations": [ { "use":"telemetry_formatted" } ] }, "telemetry_traffic_log_profile": { "class":"Traffic_Log_Profile", "requestSettings": { "requestEnabled":true, "requestProtocol":"mds-tcp", "requestPool": { "use":"telemetry" }, "requestTemplate":"event_source=\"request_logging\",hostname=\"$BIGIP_HOSTNAME\",client_ip=\"$CLIENT_IP\",server_ip=\"$SERVER_IP\",http_method=\"$HTTP_METHOD\",http_uri=\"$HTTP_URI\",virtual_name=\"$VIRTUAL_NAME\",event_timestamp=\"$DATE_HTTP\"" } }, "telemetry_security_log_profile": { "class":"Security_Log_Profile", "application": { "localStorage":false, "remoteStorage":"splunk", "protocol":"tcp", "servers": [ { "address":"255.255.255.254", "port":"6514" } ], "storageFilter": { "requestType":"illegal-including-staged-signatures" } }, "network": { "publisher": { "use":"telemetry_publisher" }, "logRuleMatchAccepts":false, "logRuleMatchRejects":true, "logRuleMatchDrops":true, "logIpErrors":true, "logTcpErrors":true, "logTcpEvents":true } } } } } To post an AS3 declaration like above use Visual Studio Code Use the command menu and select F5 Post an AS3 Declaration from the tab you have pasted the code OUTPUT from the declaration above: iRuleused Assign the Telemetry Policy to the Virtual Service by selecting the option in the advanced menu Once you have the modules installed, and configured the appropriate settings, like above, then you will see data coming in Azure Sentinel. Here is an example: ASM System Metrics For System Metrics to work, you will need to have AVR installed, you do not need an AS3 declaration or aniRule. Once you have AVR installed, and have pushed the declaration to the BIG-IP, you will need to execute the following command in your BIG-IP. tmshmodify analytics global-settings{offbox-protocoltcpoffbox-tcp-addresses add { 127.0.0.1 }offbox-tcp-port 6514 use-offboxenabled } tmshsave /sys config Check the logs in your BIG-IP less /var/log/restnoded/restnoded.log You will see something like: Fri, 18 Sep 2020 06:36:04 GMT - info: [telemetry] Starting systempollerPoller::Poller. Interval = 60 sec. Fri, 18 Sep 2020 06:36:04 GMT - info: [telemetry] 1 consumer plug-in(s) loaded Next you will need to go into the Azure Portal, and you can find a nice pre-defined Sentinel Workbook to view and start to work with: You will select the "template" and then fill out the correct workspace from the dropdown, then select the correct hostname from the dropdown and you will start to see data showing up. Azure Sentinel displaying the workbook As you enable more modules, they will show up in the Azure Sentinel and will show how it’s enabled.You can also add / modify / enhance the workbook to show more data that is in Sentinel sent from the BIG-IP. Remo and I hope you found this article helpful and enjoy using BIG-IPs with Sentinel!F5 LTM and ASM Sentinel integration - works on one cluster, doesn't on another
Hi! I have quite complex trouble with Sentinel integration. I have 2 F5 clusters implemented as IaaS in Azure - Prod and PreProd with LTM logging took from this manual:https://my.f5.com/manage/s/article/K85539421and ASM integration took from this manual:https://community.f5.com/t5/technical-articles/integrating-the-f5-bigip-with-azure-sentinel/ta-p/282868(here only ASM part). The thing is PreProd F5 Cluster sends the logs correctly while Prod does not. The configuration is very similar for both clusters (with MGT interface, external, internal & HA via internal interface) It has been reimplemented multiple times on Prod cluster, including 4 eyes check, focusing to keep the same config on working PreProd cluster. Checking and rechecking again and again FW rules, NSGs - all should work. PreProd is working, Prod is not... Recently I started to take a look though logs, finding thousands of logs on Prod F5: Fri, 23 Jun 2023 13:34:35 GMT - warning: [telemetry] Skipped Data - Category: "LTM" | Consumers: ["My_Consumer"] | Addtl Info: "event_timestamp": "2023-06-23T13:34:35.000Z" In that moment I realized, we've had a problem with NTP, that was not working and after some TShoot we took in into backlog (probably for too long time). So - NTP can't sync to time.windows.com by url or IP (other time servers also do not work). I started to TShoot this thread. it seems NTP service is running correctly (yet I restarted it) - no change: # tmsh show /sys service ntpd * ntpd.service - start and stop ntpd Loaded: loaded (/etc/rc.d/init.d/ntpd; enabled; vendor preset: enabled) Active: active (running) since Mon 2023-06-26 16:53:06 CEST; 17s ago Process: 25697 ExecStop=/etc/rc.d/init.d/ntpd stop (code=exited, status=0/SUCCESS) Process: 25762 ExecStart=/etc/rc.d/init.d/ntpd start (code=exited, status=0/SUCCESS) CGroup: /system.slice/ntpd.service `-25766 ntpd -g What is a bit strange - NTP listens only on IPv6(?) Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 18 mgmt fe80::222:48ff:fe80:cdf4 UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 19 eth0 fe80::222:48ff:fe80:cdf4 UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 20 tmm fc00:f5::1 UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 21 eth4 fe80::6245:bdff:fe8e:24ab UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 22 eth1 fe80::222:48ff:fe80:abc4 UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 23 external fe80::222:48ff:fe80:abc4 UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listen normally on 24 dev_internal fe80::6245:bdff:fe8e:24ab UDP 123 Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: Listening on routing socket on fd #41 for interface updates Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: 0.0.0.0 c016 06 restart Jun 26 16:53:06 bigip-f5-bigip1.local ntpd[25766]: 0.0.0.0 c012 02 freq_set kernel -10.616 PPM All NTPs service trying to sync have INIT status # ntpq -np remote refid st t when poll reach delay offset jitter ============================================================================== 20.101.57.9 .INIT. 16 u - 64 0 0.000 0.000 0.000 I set time manually on all F5s (didn't solve the problem with Sentinel. PreProd works, Prod doesn't tcpdump shows that my F5s try to reach NTP server with TMM external interface, which is wrong. # tcpdump -i any host 20.101.57.9 and port 123 -vv 18:04:16.677266 IP (tos 0xc0, ttl 64, id 54613, offset 0, flags [DF], proto UDP (17), length 76) 10.10.1.4.123 > 20.101.57.9.123: [bad udp cksum 0x64fd -> 0x380e!] NTPv4, length 48 Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 10 (1024s), precision 32 Root Delay: 0.000000, Root dispersion: 0.054641, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 965835954.533013659 (2066/09/16 00:14:10) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 965835954.533013659 (2066/09/16 00:14:10) out slot1/tmm0 lis= port=1.1 trunk= According tohttps://my.f5.com/manage/s/article/K92145845it should use MGT interface and written there cause is a lack of MGT route. But this is not my example as I do have MGT routes set correctly (I guess). Below there is one route towards Azure service endpoint and 2nd is default route for MGT: # tmsh list /sys management-route sys management-route azure-metadata { gateway 10.0.0.1 network 169.254.169.254/32 } sys management-route default { gateway 10.0.0.1 network default } After this moment I started to get confused, what might be the problem and is it really NTP related. Any ideas?
