Forum Discussion
HGS-97-61
Nimbostratus
NimbostratusTelemetry streaming to Elasticsearch
Hi all
I am following a couple of threads since I want to send ASM logging to Elasticsearch like this one from Greg
What I understand is that I need to send an AS3 declaration and a TS declaration.
But there are a couple of things not entirely clear to me.
1. Can I remove the iRule, Service_TCP, Pool, Log_Destination, Log_Publisher and Traffic_Log_profile declarations from the AS3 declaration json?
In the example the telemetry_asm_security_log_profile does not seem to depend on these?
2. In the AS declaration json an IP address is specified 255.255.255.254 (perhaps just an example since it is a subnet mask) and also in the TS declaration where it is 172.16.60.194.
How are the IP in the servers section of the AS3 declaration related to the one in the consumer part in the TS declaration?
3. In telemetry_asm_security_log_profile the field remoteStorage is set to splunk.
According to the reference guide: Reference Guide security-log-profile-application-object the allowed values are
“remote”, “splunk”, “arcsight”, “bigiq”.
I would opt for just remote. Is that the correct choice?
Regards Hans
- Ben_Novak
Employee
Hi Hans. There is a bunch going on with a solution like this. All of the declaration pieces are required. At a high level the AS3 declaration defines the necessary objects to forward ASM logs to the local TS process, which collect data and send to Elastic/Splunk. The ASM logging profile can be configured to send logs to HSL destination so they are not written to the local file system (which can impact performance). The HSL destination points to the "telemetry_local" TCP virtual server, which uses the iRule to point to the TS listener service. I may be a little out of order on this explanation, but it should be close enough.
- Do not remove anything. All pieces are required to get the traffic to Splunk as a single payload
- I'm pretty sure the AS3 virtualAddress of 255.255.255.254 is an internal address listening on anything. This should minimize IP conflicts with any other possible user configurations on the system. It may be related specifically to TS since the listener doesn't define it, just the port 6514. The TS declaration destination of 172.16.60.23 is the Splunk system to send all the data after TS has collected it all.
- You can have ASM logs sent directly to Splunk using the HSL configuration. Many customers do that for dedicated security dashboards. However, this whole solution is bundled as a package. To have all the data come in through TS. Probably so the preconfigured Splunk dashboards know how to find and parse all the data. Remember that TS will also include a ton of other system and application stats, not just the ASM logs.
Alex_MadjeskiAltocumulus
I am trying to answer the same questions. Were you able to get an answer?
HGS-97-62Altostratus
Hello Alex
Yes I was able with F5 support.
The best thing is to work with tmsh
Described here: https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/event-listener.html#requestlog
Start with the irule and work your way down.
After that you only need to create the listener like below
{
"class": "Telemetry",
"controls": {
"class": "Controls",
"logLevel": "debug"
},
"My_System": {
"class": "Telemetry_System",
"systemPoller": {
"interval": "60"
}
},
"My_Listener": {
"class": "Telemetry_Listener",
"port": 6514,
"trace": true
},
"My_Consumer": {
"class": "Telemetry_Consumer",
"type": "Generic_HTTP",
"trace": false,
"host": "10.0.1.111",
"protocol": "http",
"port": 9570,
"path": "/",
"method": "POST",
"headers": [
{
"name": "content-type",
"value": "application/json"
}
],
"outputMode": "processed"
}
}
- Alex_Madjeski
Thank you very much for responding, I really appreciate it.
I got everything in link configured.
I tried pushing the config you pasted above with Postman and got
code: 500
message: Unexpected string in JSON at position 576
referer: restnoded
Not sure if JRahm may be addressing any of this in his upcoming Embracing AS3: Foundations series?
