X-Forwarded-For Log Filter for Windows Servers
For those that don't know what X-Forwarded-For is, then you might as well close your browser because this post likely will mean nothing to you… A Little Background Now, if you are still reading this, then you likely are having issues with determining the origin client connections to your web servers. When web requests are passed through proxies, load balancers, application delivery controllers, etc, the client no longer has a direct connection with the destination server and all traffic looks like it's coming from the last server in the chain. In the following diagram, Proxy2 is the last hop in the chain before the request hits the destination server. Relying on connection information alone, the server thinks that all connections come from Proxy2, not from the Client that initiated the connection. The only one in the chain here who knows who the client really is (as determined by it's client IP Address, is Proxy1. The problem is that application owners rely on source client information for many reasons ranging from analyzing client demographics to targeting Denial of Service attacks. That's where the X-Forwarded-For header comes in. It is non-RFC standard HTTP request header that is used for identifying the originating IP address of a client connecting to a web server through a proxy. The format of the header is: X-Forwarded-For: client, proxy1, proxy, … X-Forwarded-For header logging is supported in Apache (with mod_proxy) but Microsoft IIS does not have a direct way to support the translation of the X-Forwarded-For value into the client ip (c-ip) header value used in its webserver logging. Back in September, 2005 I wrote an ISAPI filter that can be installed within IIS to perform this transition. This was primarily for F5 customers but I figured that I might as well release it into the wild as others would find value out of it. Recently folks have asked for 64 bit versions (especially with the release of Windows 2008 Server). This gave me the opportunity to brush up on my C skills. In addition to building targets for 64 bit windows, I went ahead and added a few new features that have been asked for. Proxy Chain Support The original implementation did not correctly parse the "client, proxy1, proxy2,…" format and assumed that there was a single IP address following the X-Forwarded-For header. I've added code to tokenize the values and strip out all but the first token in the comma delimited chain for inclusion in the logs. Header Name Override Others have asked to be able to change the header name that the filter looked for from "X-Forwarded-For" to some customized value. In some cases they were using the X-Forwarded-For header for another reason and wanted to use iRules to create a new header that was to be used in the logs. I implemented this by adding a configuration file option for the filter. The filter will look for a file named F5XForwardedFor.ini in the same directory as the filter with the following format: [SETTINGS] HEADER=Alternate-Header-Name The value of "Alternate-Header-Name" can be changed to whatever header you would like to use. Download I've updated the original distribution file so that folks hitting my previous blog post would get the updates. The following zip file includes 32 and 64 bit release versions of the F5XForwardedFor.dll that you can install under IIS6 or IIS7. Installation Follow these steps to install the filter. Download and unzip the F5XForwardedFor.zip distribution. Copy the F5XForwardedFor.dll file from the x86\Release or x64\Release directory (depending on your platform) into a target directory on your system. Let's say C:\ISAPIFilters. Ensure that the containing directory and the F5XForwardedFor.dll file have read permissions by the IIS process. It's easiest to just give full read access to everyone. Open the IIS Admin utility and navigate to the web server you would like to apply it to. For IIS6, Right click on your web server and select Properties. Then select the "ISAPI Filters" tab. From there click the "Add" button and enter "F5XForwardedFor" for the Name and the path to the file "c:\ISAPIFilters\F5XForwardedFor.dll" to the Executable field and click OK enough times to exit the property dialogs. At this point the filter should be working for you. You can go back into the property dialog to determine whether the filter is active or an error occurred. For II7, you'll want to select your website and then double click on the "ISAPI Filters" icon that shows up in the Features View. In the Actions Pane on the right select the "Add" link and enter "F5XForwardedFor" for the name and "C:\ISAPIFilters\F5XForwardedFor.dll" for the Executable. Click OK and you are set to go. I'd love to hear feedback on this and if there are any other feature request, I'm wide open to suggestions. The source code is included in the download distribution so if you make any changes yourself, let me know! Good luck and happy filtering! -Joe
Using "X-Forwarded-For" in Apache or PHP
An issue that often comes up for users of any full proxy-based product is that the original client IP address is often lost to the application or web server. This is because in a full proxy system there are two connections; one between the client and the proxy, and a second one between the proxy and the web server. Essentially, the web server sees the connection as coming from the proxy, not the client. Needless to say, this can cause problems if you want to know the IP address of the real client for logging, for troubleshooting, for tracking down bad guys, or performing IP address specific tasks such as geocoding. Maybe you're just like me and you're nosy, or you're like Don and you want the webalizer graphs to be a bit more interesting (just one host does not a cool traffic graph make, after all!). That's where the "X-Forwarded-For" HTTP header comes into play. Essentially the proxy can, if configured to do so, insert the original client IP address into a custom HTTP header so it can be retrieved by the server for processing. If you've got a BIG-IP you can simply enable the ability to insert the "X-Forwarded-For" header in the http profile. Check out the screen shot below to see just how easy it is. Yeah, it's that easy. If for some reason you can't enable this feature in the HTTP profile, you can write an iRule to do the same thing. when HTTP_REQUEST { HTTP::header insert "X-Forwarded-For" [IP::client_addr]} Yeah, that's pretty easy, too. So now that you're passing the value along, what do you do with it? Modifying Apache's Log Format Well, Joe has a post describing how to obtain this value in IIS. But that doesn't really help if you're not running IIS and like me have chosen to run a little web server you may have heard of called Apache. Configuring Apache to use the X-Forwarded-For instead of (or in conjunction with) the normal HTTP client header is pretty simple. ApacheWeek has a great article on how to incorporate custom fields into a log file, but here's the down and dirty. Open your configuration file (usually in /etc/httpd/conf/) and find the section describing the log formats. Then add the following to the log format you want to modify, or create a new one that includes this to extract the X-Forwarded-For value: %{X-Forwarded-For}i That's it. If you don't care about the proxy IP address, you can simply replace the traditional %h in the common log format with the new value, or you can add it as an additional header. Restart Apache and you're ready to go. Getting the X-Forwarded-For from PHP If you're like me, you might have written an application or site in PHP and for some reason you want the real client IP address, not the proxy IP address. Even though my BIG-IP has the X-Forwarded-For functionality enabled in the http profile, I still need to access that value from my code so I can store it in the database. $headers = apache_request_headers(); $real_client_ip = $headers["X-Forwarded-For"]; That's it, now I have the real IP address of the client, and not just the proxy's address. Happy Coding & Configuring! Imbibing: CoffeeX-Forwarded-For HTTP Module For IIS7, Source Included!
For those who of you that are having problems with logging client addresses in their server logs because you are running your web servers behind a proxy of some sort, never fear, your solution is here. For those that don't, I already discussed in my previous posts about what the X-Forwarded-For header is so feel free to click back into those to read about it. History Back in September, 2005 I wrote and posted a 32-bit ISAPI filter that extracted the X-Forwarded-For header value and replaced the c-ip value (client ip) that is stored in the server logs. Lots of folks have found this useful over time and I was eventually asked for a 64-bit version which I posted about in August, 2009. The Question Well, it looks like it's time for the next generation for this filter… I received an email from a colleague here at F5 telling me that his customer didn't want to deploy any more ISAPI filters in their IIS7 infrastructure. IIS7 introduced the concept of IIS Modules that are more integrated into the whole pipeline and was told that Microsoft is recommending folks move in that direction. I was asked if I had plans to port my ISAPI filter into a HTTP Module. The Answer Well, the answer was "probably not", but now it's changed to a "yes"! The Solution In reading about IIS Module, I found that you can develop in managed (C#/VB) or Native (C++) code. I loaded up the test C# project to see if I could get it working. In a matter of minutes I had a working module that intercepted the event when logging occurs. The only problem was that from managed code, I could find no way to actually modify the values that were passed to the logging processor. This was a problem so I scrapped that and moved to a native C++ module. After a little while of jumping through the documentation, I found the things I needed and pretty soon I had a working HTTP module that implemented the same functionality as the ISAPI filter. Download The new Http Module hasn't had much testing done so please test it out before you roll it out into production. I've made the source available as well if you find an issue and care to fix it. Just make sure you pass back the fixes to me B-). X-Forwarded-For Http Module Binary Distribution X-Forwarded-For Http Module Source Distribution The filter will require installation into IIS in order for you to be able to add it to your applications. Both distributions include a readme.txt file with an example installation procedure. Make sure you use the Release builds for the appropriate platform (x86 or x64) unless you are in need of some troubleshooting as the Debug build will dump a lot of data to a log file. The module supports customizable headers if you are using something other than X-Forwarded-For. Instructions for using that are in the readme.txt file as well. If you have any issues with using this, please let me know on this blog. Keep in mind that this is an unsupported product, but I'll do my best to fix any issues that come up. I'm even open to enhancements if you can think of any. Enjoy! -Joe
Prevent a Spoof of an X-Forwarded-For Request with BIG-IP
Last week, we looked at how to do Selective Compression on BIG-IP with a local traffic policy so this week let’s try something security related using the same procedures. You can associate a BIG-IP local traffic policy to prevent a spoof of an x-forwarded-for request. This is where bad actors might attempt to thwart security by falsifying the IP address in a header, and pass it through the BIG-IP system. Pre-reqs: We’re using BIG-IP v12 and, We already have a Virtual Server configured to manage HTTP traffic with an HTTP profile assigned to it. Let’s log into a BIG-IP The first thing we’ll need to do is create a draft policy. On the main menu select Local Traffic>Policies>Policy List and then the Create or + button. This takes us to the create policy config screen. Type a unique Policy Name like PreventSpoofOfXFF and optionally, add a description. Leave the Strategy at the default of Execute First matching rule. Click Create Policy. We’re then directed to the draft policy’s General Properties page and here we can create the rules for the policy. In the Rules area, click Create. We’ll give the rule a unique name like, StopSpoof and the first condition we need to configure is to match all HTTP traffic with the matching strategy. This means we can use the default setting of All Traffic. Then we’ll tell the policy what to do when the All Traffic condition matches. The new action is to Replace the http header named X-forwarded-for with the value of tcl:[IP::client_addr] (to return the client IP address of the connection) at the request time. Click Save. Also, save the draft. And then select the box next to the draft policy and click Publish. We can now associate the published policy with a virtual server that we’re using to manage http traffic. On the main menu click Local Traffic>Virtual Servers>Virtual Server List and click the name of the virtual server you’d like to associate for the policy. On the menu bar click Resources and next to Policies click Manage. Move PreventSpoofOfXFF to the Enabled list and click Finished. Now, the virtual server with the PreventSpoofOfXFF local traffic policy will prevent any HTTP traffic that attempts to spoof an x-forwarded-for request. Congrats! You’ve easily added additional security to your local traffic policy! You can also watch the full video demo thanks to our TechPubs team. psXFF Universal Persistence iRule
Problem this snippet solves: Simple iRule to read the XFF header on an incoming HTTP Request and use a Universal Persistence ID. Orginal iRule found to have an issue with multiple IP addresses in the XFF header for changed to only pass the first XFF IP. I have updated the iRule line to account for systems where multiple 'X-Forwarded-For' headers have been added. persist uie [lindex [ split [HTTP::header X-Forwarded-For] "," ] 0] to persist uie [lindex [ split [lindex [HTTP::header values X-Forwarded-For] 0] "," ] 0] thanks to the advice from Yann Desmarest. This could also be done with the 'getfield' command see Yann's comments below. How to use this snippet: Create iRule using following code (mine is named 'persist_xff_uie') Create Universal Persistence Profile with iRule set to 'persist_xff_uie' (or what ever name you assign to the iRule) Assign Universal Persistence Profile to Virtual Server (ensure virtual server has HTTP profile assigned) Code : # Name: persist_xff_uie # # To be used with UIE Persistence Profile # # Checks HTTP Request for 'X-Forwarded-For' header and if exists takes the first 'X-Forwarded-For' IP address as sets as # Persist identifier. # If the 'X-Forwarded-For' header does not exist then the client IP address is set as Persist identifier. when HTTP_REQUEST { if {[HTTP::header X-Forwarded-For] != ""} then { persist uie [lindex [ split [lindex [HTTP::header values X-Forwarded-For] 0] "," ] 0] } else { persist uie [IP::client_addr] } } Tested this on version: 11.5
X-Forwarded-For through proxy and F5
My organization recently got the ASM module when they purchased a pair of 5250Vs. I've never used ASM before and we don't have anyone in our organization that has either, so I'm learning on the fly. I have a small SharePoint farm behind the F5. Two web front ends in a pool that uses SNAT. This farm is accessible from within our organization only, not available from the open internet. We have geographically dispersed users and offices, all over the United States. Some offices have a web proxy that all client web traffic goes through as it leaves their local enclave and some sites do not have a proxy like this. For traffic that comes from a site that goes through a proxy, the X-Forwarded-For header shows in ASM as the proxy IP, not the actual client IP. For sites that do not go through a proxy, I see the actual client IP in this header. I contacted the group that manages the proxy at my location and confirmed that the proxy at my site also has X-Forwarded-For enabled, they even sent me a screen shot that showed it was enabled. I'd like to see the IP of the original client through the proxy. From what I read, I thought maybe I'd see multiple IPs in some circumstances, but I admit I've never worked with X-Forwarded-For header before now. Thanks in advance.
Multiple X-Forwarded-For ip address
We have enabled X-Forwarded-For on F5 and in apache we have added following code LogFormat blah...\"user-agent\": \"%{User-agent}i\", \"client\": \"%{X-Forwarded-For}i\",...blah Now i am doing experiment and sending forge X-Forwarded-For using Modify Header plugin on Chrome browser. In apache logs i am seeing two IP addresses. like following 123.123.123.123 is fake IP. "client": "123.123.123.123, 210.76.39.145" Question: is there a way in apache/F5 LogFormat to extract only last IP address which is valid one?X-Forwarded-For: Multiple IP's Chained | Need Original IP | iRules and HTTP Profile?
Hello, I am at a loss. I am a security engineer and one of my jobs is managing the SIEM, which I am building out currently. One of the log sources is, of course, the F5 ASM logs. The problem, however, is that the logs come in with the wrong IP in the source IP field. They come in with IP addresses from our own architecture components such as the ALB or cloudfront instead of correctly parsing the XFF (via the iRule below or otherwise). It definitely has to do with the XFF header situation of our architecture. I have read a boat load of stuff to no avail. Just cannot seem to get the right IP (the ACTUAL origin IP) in the source IP spot. Any help would be greatly appreciated! Architecture: internet/users —> Cloudfront —> ALB —> F5 —> Amazon ECS backend incl. Logrhythm SIEM Also, the profile being used in the LTM is http. We have "insert x-forwarded-for" set to DISABLED, and "Accept XFF" ENABLED, as well as the Alternate XFF box filled in with "X-Forwarded-For" (just to be safe). Desired Solution: We are hoping for the F5 to send its syslog ASM logs to our SIEM (Logrhythm) with the true original IP address listed correctly as the source/client IP address. In other words, We need the SIEM to see the origin IP as the true original IP. Problem: The issues with the xff headers and various IP addresses has created a situation where either Cloudfront, the ALB, the F5, or one of the backend service’s IP address is incorrectly in the original source/client IP spot. The implications of this are far reaching and totally screws things up for our SIEM and security orchestration and all sorts of stuff. iRule Currently Used: We are currently using this iRule which is at the top of the list of execution order. when HTTP_REQUEST { HTTP::header insert X-Forwarded-For [IP::remote_addr] HTTP::header replace X-Forwarded-For [getfield [HTTP::header X-Forwarded-For] "," 1] HTTP::header replace X-Forwarded-For-Raw [getfield [HTTP::header X-Forwarded-For] "," 1] HTTP::header replace CF-Connecting-IP [getfield [HTTP::header X-Forwarded-For] "," 1] } Example Request seen in the F5 ASM Event Logs (to be fwded to SIEM): GET / HTTP/1.1 X-Forwarded-For: AAA.AAA.AAA.AAA, BBB.BBB.BBB.BBB X-Forwarded-Proto: https X-Forwarded-Port: 443 Host: sub.exampledomain.com X-Amzn-Trace-Id: Root=1-5b67cf53-df48e8adfafafdsaff4a726f X-Amz-Cf-Id: PYTrCasdfafgh_5xIO_P4IpHafdsafsdafETZLgQr4CJEfOpQ== User-Agent: Mozilla/5.0 (compatible; Uptime/1.0; http://uptime.com) Via: 1.1 aa9157a1414f639asdfasdfasb7df10.cloudfront.net (CloudFront) CloudFront-Is-Mobile-Viewer: false CloudFront-Is-Tablet-Viewer: false CloudFront-Is-SmartTV-Viewer: false CloudFront-Is-Desktop-Viewer: true CloudFront-Viewer-Country: SG Accept: */* CloudFront-Forwarded-Proto: https x-salt-cloudfront-secret: 6CsSe4cBZwuTBOasdfadUhu3sPKJ5K8SCA X-Forwarded-For: CC.CC.CC.CC X-Forwarded-For-Raw: CC.CC.CC.CC CF-Connecting-IP: CC.CC.CC.CC In this case, the F5 would have CC.CC.CC.CC listed in the source IP Address Thank you very much to anyone who can help!
