Forum Discussion
roachet
Nimbostratus
NimbostratusExtended logging to Splunk servers beyond Syslog & Analytic Profiles & iRules.
What has been configured to date.
All syslog traffic is being sent to Splunk destination hosts. The Splunk hosts are external and have to transverse the WAN to eventual destination.
AVR has been provisioned and a Splunk TCP analytic profile has been created. (the traffic is HTTPS) The VIPS also have SNAT configured.
Detailed requirements from network security (Cyberfusion) and performance management groups below.
Task:
o Enable & configure detailed VIP connection event logging that includes Timestamp, F5 Host servicing the connection, Source IP, Initial Destination Port/Protocol, Destination IP, Final Destination Port/Protocol & if URL filters are being used - URL request logging should be enable & configured. This is a minimum requirement.
· Purpose:
o The Cyber Fusion Center analysts require end-to-end visibility in relation to network connection events in order to appropriately track potential threats, potentially malicious activity and assist in attack path validation during Cyber events & Cyber incident investigations. Due to the fact that a good majority of our Web-based traffic utilizes F5 VIPs to frontend infrastructure, it makes it nearly impossible to determine the actual source of traffic when investigating Cyber events & Cyber incidents on victim infrastructure. The source IP that gets logged in the majority of all host logs is the IP of the F5 load balancer. In turn, the Cyber Fusion Center analysts do not have the connection logs available from the F5 load balancers in order to track the activity back to an original source.
· End State:
o All UTC F5 load balancers have detailed VIP connection event logging & URL request logging (where utilized) enabled and configured. All logs are being sent to the xxx Splunk infrastructure for central visibility.
In a nutshell the Cyberfusion group wants to see ALL traffic.
My suggestion to implement network taps, a port aggregator and a netflow collection host has been shot down. The Cyberfusion group want to see if the F5’s are able to perform and satisfy all requirements listed above.
Any suggestions would be appreciated.
I personally do not believe that the F5’s should be part of Cyberfusions security suite relative to the detailed information that is being requested to be sent externally.
Thanks,
et
Simon_BlakelyEmployee
You might start with a Request Logging Profile:
AskF5 | Manual Chapter: Configuring Request Logging
Also look at
Adding the body of requests/responses to the data being logged to Splunk via iRule.
which has a pretty comprehensive irule for sending detailed information formatted for Splunk (ignoring the attempt to send the payload).
> I personally do not believe that the F5’s should be part of Cyberfusions security suite relative to the detailed information that is being requested to be sent externally.
In many cases, the BigIP is supplying required information (such as client IP) via headers and other insertions, and so the logging on webservers should be configured to use things like X-Forwarded-For as the client IP address.
et2Hello,
Thank you for the response.
About the request logging, particularly " From the Pool Name list, select the pool that includes the node log server as a pool member." I take it that the pool that was created (splunk servers receiving the logs) would be the pool selected?
For the environment in question, a HTTP rule would not work well. All VIPS are defined for HTTPS and all pool members in all pools have HTTPS bound to them.
I will also test with x-forwarding enabled.
The requirement " analysts require end-to-end visibility in relation to network connection events in order to appropriately track potential threats, potentially malicious activity and assist in attack path validation during Cyber events & Cyber incident investigations. Due to the fact that a good majority of our Web-based traffic utilizes F5 VIPs to frontend infrastructure it makes it nearly impossible to determine the actual source of traffic when investigating Cyber events & Cyber incidents on victim infrastructure. The source IP that gets logged in the majority of all host logs is the IP of the VIP"
they are asking to see all traffic, end to end with no obfuscation whatsoever. The environment is such that the actual source addresses are from all over the world. IMHO I do not believe that the function of the F5's is to act as an agent to populate a SIEM. As the environment is using APM (replaced Microsoft Threat Management Gateways) there will be added overhead on the F5's as well.
- Simon_Blakely
> About the request logging, particularly " From the Pool Name list, select the pool that includes the node log server as a pool member." I take it that the pool that was created (splunk servers receiving the logs) would be the pool selected?
Yes, that is correct.
> For the environment in question, a HTTP rule would not work well. All VIPS are defined for HTTPS and all pool members in all pools have HTTPS bound to them.
> I will also test with x-forwarding enabled.
If you have client-ssl profiles on the virtual servers, the TLS/SSL traffic is decrypted within the BigIP, and the HTTP events have access to the decrypted traffic, before re-encryption by the server-ssl profile.
And it is this visibility into the traffic that makes the BigIP the ideal data-collection point - the proviso is that data must be logged off-box using the High Speed Logging (HSL) functionality - the BigIP is not optimised for disk-I/O, so local logging at this level of detail will negatively impact performance.
- et2
Splunk destination hosts are defined as Log publishers based on F5 configuration documentation. Splunk destination hosts are in a pool.
Splunk destination hosts are defined as log destinations. As this is APM there are actually 2 destinations defined. One is HSL, which is defined for the Splunk pool.
Another element is syslog instance that forwards to the HSL element.
Access policyies logging created. Access profiles created for each policy created.
Client SSL profiles also created for each instance.
So all syslog traffic is being sent to the Splunk destinations. In fact everything that is defined is sending logs to the Splunk destinations. However, since the beginning of this project the Splunk folks continue to maintain that they "are not seeing all the traffic we would expect to see."
The customer is asking that the F5's perform the heavy lifting in an attempt to avoid spending money on taps, port aggregators and a local flow collecter / netflow host / local Splunk logging host.
- Simon_Blakely
> The customer is asking that the F5's perform the heavy lifting in an attempt to avoid spending money on taps, port aggregators and a local flow collecter / netflow host / local Splunk logging host.
And you appear to be avoiding the fact that because of the privileged position of the BigIP in the packet flow, it is actually uniquely positioned to provide the requested per-connection data that is required, and looking for validation of that position. You are not going to get it here. People who use F5 devices are expecting to use them as a multipurpose tool to patch functionality holes in their network, and discovering that they can do so both capably and elegantly.
If you are dealing with HTTP virtuals or HTTPS virtuals with a client-side SSL profile, the BigIP (via iRules) has complete visibility into the layer-4 connection and layer-7 request/response, and can log all (or part) that information directly to a remote splunk server via High Speed Logging. For virtuals that work at layer-4 (ip-forwarding, performance layer-4, including passthrough TLS/SSL) you still have access to the layer-4 information in the same way.
If you are being asked to provide that sort of visibility from the F5 (without spend large amounts of additional money), you can. Of course, there is a performance cost when using irules, and you will need to monitor the application of logging irules to ensure that the device is not being pushed beyond it's capability.
If you want both visibility and service chaining, talk to F5 about SSL-Orchestrator.
Talk to your F5 Account team/F5 Sales. They can put you in touch with F5 Professional Services who are really good at that sort of thing.