Improving Log Analysis with Device ID Ratios inside Elasticsearch
Overview Ratios Power Context - Log analysis is less about raw numbers than ratios. Ratios put numbers in context. A security analyst follows multiple ratios andgains a sense of what is a healthy range for each ratio, and follows the trend of the ratio over time, knowing that certain changes indicate trouble. As an additional data field in a SIEM system, F5 Device ID+ adds tremendous security value not so much as a raw number, that is how many Device IDs appear in a log over a period,but rather because it factors into so many useful ratios, which we’ll cover in this article. Moreover, these ratios are actionable. You can use F5 Device ID+ to identify the network requests that require further investigation. Within the context of your organization and your apps, each of these ratios will have an organic range of values within which it fluctuates, likely a narrow range. Changes in these ratios, especially sudden changes over hours or days, should raise alerts and trigger further investigation. Goal & Architecture While I was already using HSL (High Speed Logging) to log BIG-IP LTM / AWF related information to ELK Stack and did correlation with help of different Dashboards, the goal was to enrich that log information with a unique Device Identifier to create simple ratios, that can provide strong intel on sudden fluctuations. Simple ratios that can provide strong intel with sudden fluctuations The overall Architecture - where Device ID+ has been onboarded to BIG-IP using the following documented process: Getting started with F5 Device ID+ Building Ratios Example: Single Device accessing unauthorized accounts - Sudden fluctuations in Users per Device ID In this example, the context to create is the number of times a single device accessing unauthorized accounts. Basically, we measure the login success rate for a single device. The login success rate refers to the percentage of login attempts that succeed out of the total of all login attempts. Applications that offer a login can be instrumented to track this metric. If this is not possible, you might be able to make the determination of success versus failure through the web log looking at the request path and the response status code and headers. The login success rate is quite significant. Every app will have an organic rate. A decline in that rate indicates either an attack, either a credential stuffing or brute force attack, or increased login friction. Analyzing the login success rate per Device ID is even more helpful. Investigating requests from Device IDs with particularly low login success rates is likely to uncover attacks. In other words, the success rate itself tells us whether there is an attack while Device ID helps us to locate the source of attack. The visualization within ELK is showing a Unique Device Identifier as well as a Unique IP Address, however, this single device tried to access multiple accounts on an application. Example: Deliberate use of proxy networks - Sudden fluctuations in IPs per Device ID We expect the ratio of IP address per Device ID to be low for desktop computers, higher forlaptops, and highest for mobile devices. Overall, the ratio across all user devices will not varysignificantly over time, especially not suddenly. A sudden change over a short duration, suchan hour or a day, may indicate that some users are deliberately changing their IP addressesthrough the use of proxy networks, a likely signal of an attack. If discovered, investigate thenetwork requests from Device IDs with high numbers of IP addresses. The visualization within ELK is showing a Unique Device Identifier associated with different IP Addresses, however, this single device tried to access multiple accounts on an application. Example: Unusual Devices accessing user accounts - Sudden fluctuations in Device IDs per User We would expect that many users access our site from more than one device. Like the ratio of IPs to Device ID, we would expect the ratio of Device IDs per user to remain fairlystable. Likewise, a sudden change is a red flag. If suddenly more devices than expected areaccessing a user account, it likely indicates an attack. Most likely, not all of those devicesbelong to real users. The visualization within ELK is showing multiple Device Identifier behind a single IP Address trying to access an application unauthorized by using a user account called "OttoGood". Conclusion By collection data into an ELK Stack and by using F5 Device ID+ to enhance your data, you can build ratios and correlations, which are helping to increase your security posture. If you want to onboard F5 Device ID+ and build your own ratios, please visit Getting started with F5 Device ID+DeviceID with APM and API Server
This article covers the use case to use APM and an external API Server to store DeviceID+. As you learnt in the previous articles in this series, DeviceID+ is a way to "identify" a user. And of course, it is a great solution to combine with F5 APM (Access Policy Manager) - https://www.f5.com/products/security/access-policy-manager The F5 Shape DeviceID+ solution does not store the IDs anywhere. It is up to you to store/use/consume them on your own. To do so, I propose to use an external API Server to store the DeviceID / Username peer. Steps: Built an API server based on loopback.io stack. You can download the project (docker-compose) here : https://github.com/MattDierick/DeviceID-api-server/blob/main/README.md Build an APM Pre Request Policy with HTTP Connector to query the API server (available also on my Github) Created the Postman collection (available also on my Github) First of all, watch the demo video to understand the use case and the outcomes: https://www.youtube.com/watch?v=PVYwh76nGVE The API Server: The API server has a Swagger UI installed so you can "watch" the API (go to /explorer) Structure of the API Below is the structure of the API payload. As you can see, we store the username with the associated DeviceIDs { "username": "dierick", "deviceid": [ "Ac31A2AAAAAA12uwoekcLffhgfjgjghj", "AUq7Xti2Eowm6yVog4CYvt6nRrookqgW" ] } The REST calls used by APM 1. GET DeviceID per username GET /api/deviceids/findOne?filter={"where":{"username":"dierick"}} 2. POST a new DeviceID POST /api/deviceids HTTP/1.1 { "username": "dierick", "deviceid": [ "Ac31A2AAAAAA12uwoekcLffhgfjgjghj", "AUq7Xti2Eowm6yVog4CYvt6nRrookqgW" ] } 3. UPDATE DeviceID per username POST /api/deviceids/upsertWithWhere?where={"username": "dierick"} HTTP/1.1 { "deviceid": [ "Rtdsflkj534565465kenfter" ] } The iRule to collect the DeviceID from the Shape cookie when ACCESS_SESSION_STARTED { log local0. "Access Session Started" if [HTTP::cookie exists _imp_apg_r_] { set deviceid [URI::decode [HTTP::cookie _imp_apg_r_]] log local0. "URL Decoded cookie is $deviceid" set deviceida [lindex [regexp -inline -- (?:"diA":")(.*?)(?:") $deviceid] 1] log local0. "diA = $deviceida" set deviceidb [lindex [regexp -inline -- (?:"diB":")(.*?)(?:") $deviceid] 1] log local0. "diB = $deviceidb" log local0. "IP is [IP::client_addr]" log local0. "Path os [HTTP::path]" ACCESS::session data set session.logon.deviceid $deviceid ACCESS::session data set session.logon.deviceid.A $deviceida ACCESS::session data set session.logon.deviceid.B $deviceidb } else { log local0. "No cookie" } } The APM Per Request Policy Below an example using the APM Per Request Policy. A Per Request Policy is required instead of a Per Session Policy in order to use the HTTP Connector. HTTP Connector is a REST API client. Create the different CALLS, and assign each call in the right branch (new DeviceID, update DeviceID ...)
