Forum Discussion
former_newbie
Nimbostratus
Nimbostratusunable to question about getting hsl data to be formatted properly in splunk
We're using the following i-rule to craft the data for sending it to the Splunk server:
when CLIENT_ACCEPTED {
set client_address [IP::client_addr]
set vip [IP::local_addr]
}
when HTTP_REQUEST {
set http_host [HTTP::host]:[TCP::local_port]
set http_uri [HTTP::uri]
set http_url $http_host$http_uri
set http_method [HTTP::method]
set http_version [HTTP::version]
set http_user_agent [HTTP::header "User-Agent"]
set http_content_type [HTTP::header "Content-Type"]
set http_referrer [HTTP::header "Referer"]
set tcp_start_time [clock clicks -milliseconds]
set req_start_time [clock format [clock seconds] -format "%Y/%m/%d %H:%M:%S"]
set cookie [HTTP::cookie names]
set user [HTTP::username]
set virtual_server [LB::server]
if { [HTTP::header Content-Length] > 0 } then {
set req_length [HTTP::header "Content-Length"]
} else {
set req_length 0
}
}
when HTTP_RESPONSE {
set res_start_time [clock format [clock seconds] -format "%Y/%m/%d %H:%M:%S"]
set node [IP::server_addr]
set node_port [TCP::server_port]
set http_status [HTTP::status]
set req_elapsed_time [expr {[clock clicks -milliseconds] - $tcp_start_time}]
if { [HTTP::header Content-Length] > 0 } then {
set res_length [HTTP::header "Content-Length"]
} else {
set res_length 0
}
set hsl [HSL::open -proto UDP -pool splunk_hsl_pool]
HSL::send $hsl "<190>,f5_irule=Splunk-iRule-HTTP,src_ip=$client_address,vip=$vip,http_method=$http_method,http_host=$http_host,http_uri=$http_uri,http_url=$http_url,http_version=$http_version,http_user_agent=\"$http_user_agent\",http_content_type=$http_content_type,http_referrer=\"$http_referrer\",req_start_time=$req_start_time,cookie=\"$cookie\",user=$user,virtual_server=\"$virtual_server\",bytes_in=$req_length,res_start_time=$res_start_time,node=$node,node_port=$node_port,http_status=$http_status,req_elapsed_time=$req_elapsed_time,bytes_out=$res_length\r\n"
}
when LB_FAILED {
log local0. "f5_irule=Splunk-iRule-LB_FAILED,src_ip=$client_address,vip=$vip,http_method=$http_method,http_host=$http_host,http_uri=$http_uri,http_url=$http_url,http_version=$http_version,http_user_agent=\"$http_user_agent\",http_content_type=$http_content_type,http_referrer=\"$http_referrer\",req_start_time=$req_start_time,cookie=\"$cookie\",user=$user,virtual_server=\"$virtual_server\",bytes_in=$req_length"
}
We tested it by first sending the data locally and it's getting formatted OK, as per the following:
Dec 11 15:45:10 10.192.156.163 context_name="/Common/fwd_vs",dest_ip="192.168.36.45",dest_port="80",device_product="Advanced Firewall Module",device_vendor="F5",device_version="12.1.0.0.0.1690",errdefs_msgno="23003161", errdefs_msg_name="Firewall NAT",event_name="Session Start",ip_protocol="TCP",duration="",severity="7",route_domain="0",source_ip="10.10.10.168",source_port="33868",timestamp="Dec 11 2015 09:23:01",translated_dest_ip="192.168.36.45",translated_dest_port="80",translated_route_domain="0", translated_source_ip="10.192.156.163",translated_source_port="33868"
However, when we're sending it to the Splunk server, it's not interpreting the variables, just sending the text. For instance, the $client_address variable comes out as $client_address, not the value inside it.
Has anyone come across an issue like this and could shed the light on it?
Thanks.
Hi Forme_Newbie,
To configure F5 LTM field extraction settings in your Splunk data source, In Splunk ,navigate to Settings > Fields > Field extractions within the Splunk interface, where you can define custom extraction rules using regular expressions to identify and isolate specific data fields from your F5 LTM logs, allowing you to analyze them individually within Splunk searches and reports.
Key points about F5 LTM field extraction in Splunk:
Splunk Add-on for F5 BIG-IP:
To effectively extract F5 LTM data, utilize the dedicated Splunk add-on for F5 BIG-IP which provides pre-configured field extractions tailored to F5 log formats.
Props.conf and Inputs.conf:
These configuration files within your Splunk environment are used to define how raw F5 LTM data is parsed and mapped to specific fields.Field extraction methods:
Automatic extraction: The Splunk add-on might automatically identify common fields based on the log format.
Regular expressions (regex): For complex parsing scenarios, use custom regex patterns to extract specific data from the log lines.Important fields to extract:
Virtual Server Name
Client IP Address
Pool Name
HTTP Status Code
Transaction Time
Application NameHow to configure field extraction:
1. Access Field Extractor:
Go to Settings > Fields > Field extractions in Splunk.
2. Select sample data:
Choose a sample log entry from your F5 LTM data source to test your extraction rules.
3. Define extraction rules:
Field names: Specify the names you want to assign to the extracted data fields.
Extraction method: Select "Regex" and provide the appropriate regular expression to match the desired data within the log line.
4. Validate and save:
Review the extracted fields to ensure accuracy and save your configuration.https://docs.splunk.com/Documentation/AddOns/released/F5BIGIP/Install
https://docs.splunk.com/Documentation/AddOns/released/F5BIGIP/About
https://docs.splunk.com/Documentation/AddOns/released/F5BIGIP/Setup
Virtual Server Log Forward to Splunk | DevCentral
Field Extractions in Splunk
https://www.youtube.com/watch?v=BKD-YHBg7iwhttps://medium.com/@mohitrdamke/splunk-fields-extractor-for-beginners-by-mohit-damke-52be06d144b5
https://docs.splunk.com/Documentation/SplunkCloud/latest/Knowledge/ExtractfieldsinteractivelywithIFX
1) Adding Splunk server as pool member.
Navigate to Local Traffic > Pools
Click Create.
Name:-Splunk_HSL_Pool
IP address :-10.10.10.10
Port :-514
Click Finished.
2) Log Destination
Navigate to System > Logs > Configuration > Log Destinations .
Click Create.
Name field :- Splunk_HSL_logging
Type:- Remote High-Speed Log
Pool Name :- Splunk_HSL_Pool
Protocol :- HSL
Click Finished.
3) Log Formatt
Navigate to System > Logs > Configuration > Log Destinations .
Click Create.
Name:- Splunk_Logs
Type:- Splunk.
Forward To :- Splunk_HSL_logging
Click Finished.
4) Publisher
Navigate to System > Logs > Configuration > Log Publishers .
Click Create.
Name :- Splunk_Publisher
Destinations :-Splunk_HSL_logging.
Click Finished.
Irule for virtual Server.
******************************
Goto->Local Traffic-Irule
Name:- Splunk_logging
Copy paste below code.
when CLIENT_ACCEPTED {
set client_address [IP::client_addr]
set vip [IP::local_addr]
set hsl [HSL::open -proto TCP -pool Splunk_HSL_Pool]
}
when HTTP_REQUEST {
set http_host [HTTP::host]:[TCP::local_port]
set http_uri [HTTP::uri]
set http_method [HTTP::method]
set http_version [HTTP::version]
set virtual_server [LB::server]
set http_user_agent [HTTP::header "User-Agent"]
set http_content_type [HTTP::header "Content-Type"]
set tcp_start_time [clock clicks -milliseconds]
set req_start_time [clock format [clock seconds] -format "%Y/%m/%d %H:%M:%S"]
if { [HTTP::header Content-Length] > 0 } then {
set req_length [HTTP::header "Content-Length"]
} else {
set req_length 0
}
}
when HTTP_RESPONSE {
# set res_start_time [clock format [clock seconds] -format "%Y/%m/%d %H:%M:%S"]
set node [IP::server_addr]
set node_port [TCP::server_port]
set http_status [HTTP::status]
set req_elapsed_time [expr {[clock clicks -milliseconds] - $tcp_start_time}]
if { [HTTP::header Content-Length] > 0 } then {
set res_length [HTTP::header "Content-Length"]
} else {
set res_length 0
}
set hsl [HSL::open -proto UDP -pool Splunk_HSL_Pool]
HSL::send $hsl "<514> HSL, CLIENT_IP=$client_address, VIP=$vip, VIP_NAME=\"$virtual_server\", SERVER_NODE=$node, SERVER_NODE_PORT=$node_port, HTTP_URL=$http_url, HTTP_VERSION=$http_version, HTTP_STATUS=$http_status, HTTP_METHOD=$http_method, HTTP_CONTENT_TYPE=$http_content_type, HTTP_USER_AGENT=\"$http_user_agent\", HTTP_REFERRER=\"$http_referrer\",REQUEST_START_TIME=$req_start_time,REQUEST_ELAPSED_TIME=$req_elapsed_time, BYTES_IN=$req_length, BYTES_OUT=$res_length\r\n"
}
Please mark as solution if it help resolve your issue.HTH
F5 Design Engineer
