Forum Discussion
BIG-IP : redirect loop : debugging techniques
BIG-IP 11.4.0 Build 2384.0 Final
In my hosts file I point
www.mydomain.comhttp://www.mydomain.com/natural-language-url-1and the page is returned as expected..
In BIG-IP , I have configured a single virtual-server
vip-01irule-01when HTTP_REQUEST {
log local0. "HTTP_REQUEST received by irule-01"
log local0. "host = [HTTP::host]"
log local0. "uri = [HTTP::uri]"
switch -glob [string tolower [HTTP::host]] {
"www.mydomain.com" {
switch -glob [string tolower [HTTP::uri]] {
"/api/*" {
pool service-pool-01
}
default {
log local0. "matched default pool"
pool default-pool-01
}
}
}
}
}
Now, in my hosts file I point
www.mydomain.comvip-01http://www.mydomain.com/natural-language-url-1and the client-browser reports an infinite-redirect error
"This webpage has a redirect loop"BIG-IP > System > Logs > Local Traffic
Rule /Common/irule-01 : HTTP_REQUEST received by irule-01`
Rule /Common/irule-01 : host = www.mydomain.com
Rule /Common/irule-01 : uri = /natural-language-url-1
Rule /Common/irule-01 : matched default pool
... repeat 10 times ...
Fiddler shows that 21 302-redirects have occurred :
Result 302
Protocol HTTP
Host www.mydomain.com
URL /natural-language-url-1
Body 185
Caching private
Content-Type text/html; charset=utf-8
Process chrome:7516
Comments
Custom
NOTE 1 : on vip-01, Source Address Translation = Auto-Map
NOTE 2 : I have complete control over the web-server ( remote access with ability to modify configuration, run diagnostics, perform local tests, etc. )
NOTE 3 : The web-server expects the request to include the domain. Sending a request with host = {web-server-host-name} , or host = {web-server-ip} will throw an error because IIS will be unable to locate the virtual app.
What are some debugging techniques I can deploy to further diagnose the underlying cause of the redirect loop ?
The server is redirecting, probably because it does not like something. Check with server admin.
For debugging in any other case use: curl httpwatch tcpdump
In this case these tools will not tell you too much.
Working with server logs is probably best.
HTH
- Thanks for the response John. Could you please elaborate on how the server could be redirecting ? I don't see how that's possible because in my test with hosts pointing www.mydomain.com to the web-server IP, the redirect loop error does not occur. The error only occurs when BIG-IP is introduced into the picture.
- Kevin_Stewart
Employee
I completely agree with John here. There's absolutely nothing in your iRule that could be generating a redirect. I'd be willing to bet that if you did a tcpdump between the F5 and the web server, you'd see this 302 message coming from the server itself. I know you're generalizing the code for forum view, but is there anything different between accessing directly and through the VIP, other than the host name? If not, try adding this to your code:
HTTP::header replace Host "my-real-server-hostname"where "my-real-server-hostname" is the name you use when you're accessing the web server directly.
Do curl -v -H servername.com http://...... To both the virtual and the server. See if either succeeds, and if the results are different.
Post here the entire redirect response So we see the server header.
Check server events log it should tell you why it is redirecting.
It is also possible that the server is not accepting connections from the selfip subnet. You have SNAT automap on the VIP.
- Thanks John. I'm not sure what you mean by "servername.com" ? The web-server FQDN = "web-server-1.devtest.mydomain.com" , but the F5 BIG-IP VIP is just an IP with no corresponding hostname.
- Here is curl request direct to web-server : C:\Windows\System32>curl -v -H web-server-1.devtest.mydomain.com www.mydomain.com/natural-language-url-1 |more * About to connect() to www.mydomain.com port 80 (0) * Trying 10.23.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to www.mydomain.com (10.23.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Host: www.mydomain.com > Accept: */* > < HTTP/1.1 200 OK I'm leaving the rest out because it's a few thousand lines of content ( headers, cookies, html ).
- Here is curl request to virtual-server vip : C:\Windows\System32>curl -v -H 10.25.0.2 www.mydomain.com/natural-language-url-1 |more * About to connect() to www.mydomain.com port 80 (0) * Trying 10.25.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to www.mydomain.com (10.25.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Host: www.mydomain.com > Accept: */* > < HTTP/1.1 302 Found < Date: Fri, 27 Dec 2013 02:42:52 GMT < Server: Microsoft-IIS/6.0 < P3P: CP="CAO PSA OUR" < Etag: < X-mydomainHost: XX-XX-XX-01 < X-Powered-By: ASP.NET < PICS-Label: (PICS-1.0 "http://www.rsac.org/ratingsv01.html" l by "support@mydomain.com" on "2008.12.01T18:21-0800" exp "2035.12.31T12:00-0800" r (v 0 s 0 n 0 l 0)) < X-AspNet-Version: 4.0.30319 < Location: http://www.mydomain.com/natural-language-url-1 < Set-Cookie: lsCKE=ors=otcomqanew; domain=.mydomain.com; path=/ < Cache-Control: private < Content-Type: text/html; charset=utf-8 < Content-Length: 185 < { [data not shown] 100 185 100 185 0 0 2371 0 --:--:-- --:--:-- --:--:-- 2983 * Connection 0 to host www.mydomain.com left intact *Object moved Object moved to here. Closing connection 0
nitass_89166Noctilucent
can you run curl on f5 and post the output?
curl -v -H "Host: www.mydomain.com" http://pool_member_ip/natural-language-url-1 curl -v -H "Host: www.mydomain.com" http://virtual_server_ip/natural-language-url-1- C:\Windows\System32>curl -v -H "Host: www.mydomain.com" http://10.23.0.2/natural-language-url-1 |more * About to connect() to 10.23.0.2 port 80 (0) * Trying 10.23.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to 10.23.0.2 (10.23.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Accept: */* > Host: www.mydomain.com > < HTTP/1.1 200 OK I'm leaving the rest out because it's a few thousand lines of content ( headers, cookies, html ).
- nitass_89166would you mind running curl on f5 command line please?
- C:\Windows\System32>curl -v -H "Host: www.mydomain.com" http://10.25.0.2/natural-language-url-1 |more * About to connect() to 10.25.0.2 port 80 (0) * Trying 10.25.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to 10.25.0.2 (10.25.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Accept: */* > Host: www.mydomain.com > < HTTP/1.1 302 Found < Date: Fri, 27 Dec 2013 03:04:10 GMT < Server: Microsoft-IIS/6.0 < P3P: CP="CAO PSA OUR" < Etag: < X-mydomainHost: XX-XX-XX-01 < X-Powered-By: ASP.NET < PICS-Label: (PICS-1.0 "http://www.rsac.org/ratingsv01.html" l by "support@mydomain.com" on "2008.12.01T18:21-0800" exp "2035.12.31T12:00-0800" r (v 0 s 0 n 0 l 0)) < X-AspNet-Version: 4.0.30319 < Location: http://www.mydomain.com/natural-language-url-1 < Set-Cookie: lsCKE=ors=otcomqanew; domain=.mydomain.com; path=/ < Cache-Control: private < Content-Type: text/html; charset=utf-8 < Content-Length: 185 < { [data not shown] 100 185 100 185 0 0 2371 0 --:--:-- --:--:-- --:--:-- 2936 * Connection 0 to host 10.25.0.2 left intact *Object moved Object moved to here. Closing connection 0
nitasscan you run curl on f5 and post the output?
curl -v -H "Host: www.mydomain.com" http://pool_member_ip/natural-language-url-1 curl -v -H "Host: www.mydomain.com" http://virtual_server_ip/natural-language-url-1- C:\Windows\System32>curl -v -H "Host: www.mydomain.com" http://10.23.0.2/natural-language-url-1 |more * About to connect() to 10.23.0.2 port 80 (0) * Trying 10.23.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to 10.23.0.2 (10.23.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Accept: */* > Host: www.mydomain.com > < HTTP/1.1 200 OK I'm leaving the rest out because it's a few thousand lines of content ( headers, cookies, html ).
- nitasswould you mind running curl on f5 command line please?
- C:\Windows\System32>curl -v -H "Host: www.mydomain.com" http://10.25.0.2/natural-language-url-1 |more * About to connect() to 10.25.0.2 port 80 (0) * Trying 10.25.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to 10.25.0.2 (10.25.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Accept: */* > Host: www.mydomain.com > < HTTP/1.1 302 Found < Date: Fri, 27 Dec 2013 03:04:10 GMT < Server: Microsoft-IIS/6.0 < P3P: CP="CAO PSA OUR" < Etag: < X-mydomainHost: XX-XX-XX-01 < X-Powered-By: ASP.NET < PICS-Label: (PICS-1.0 "http://www.rsac.org/ratingsv01.html" l by "support@mydomain.com" on "2008.12.01T18:21-0800" exp "2035.12.31T12:00-0800" r (v 0 s 0 n 0 l 0)) < X-AspNet-Version: 4.0.30319 < Location: http://www.mydomain.com/natural-language-url-1 < Set-Cookie: lsCKE=ors=otcomqanew; domain=.mydomain.com; path=/ < Cache-Control: private < Content-Type: text/html; charset=utf-8 < Content-Length: 185 < { [data not shown] 100 185 100 185 0 0 2371 0 --:--:-- --:--:-- --:--:-- 2936 * Connection 0 to host 10.25.0.2 left intact *Object moved Object moved to here. Closing connection 0
Sorry ok now I get it : you want curl from f5 command line.
- f5 command line curl direct to web-server pool ip : curl -v -H "Host: www.mydomain.com" http://10.23.0.2/natural-language-url-1 |less * About to connect() to 10.23.0.2 port 80 (0) * Trying 10.23.0.2... connected * Connected to 10.23.0.2 (10.23.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8y zlib/1.2.3 libidn/0.6.5 > Accept: */* > Host: www.mydomain.com > % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 etc.
- f5 command line curl to virtual-server vip : curl -v -H "Host: www.mydomain.com" http://10.25.0.2/natural-language-url-1 |less * About to connect() to 10.25.0.2 port 80 (0) * Trying 10.25.0.2... connected * Connected to 10.25.0.2 (10.25.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8y zlib/1.2.3 libidn/0.6.5 > Accept: */* > Host: www.mydomain.com > % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/1.1 302 Found < Date: Fri, 27 Dec 2013 03:24:06 GMT < Server: Microsoft-IIS/6.0 < P3P: CP="CAO PSA OUR" < Etag: < X-mydomainHost: XX-XX-XX-01 < X-Powered-By: ASP.NET < PICS-Label: (PICS-1.0 "http://www.rsac.org/ratingsv01.html" l by "support@mydomain.com" on "2008.12.01T18:21-0800" exp "2035.12.31T12:00-0800" r (v 0 s 0 n 0 l 0)) < X-AspNet-Version: 4.0.30319 < Location: http://www.mydomain.com/natural-language-url-1 < Set-Cookie: lsCKE=ors=otcomqanew; domain=.mydomain.com; path=/ < Cache-Control: private < Content-Type: text/html; charset=utf-8 < Content-Length: 185 < { [data not shown] 185 185 185 185 0 0 6585 0 --:--:-- --:--:-- --:--:-- 7708* Connection 0 to host 10.25.0.2 left intact * Closing connection 0 Object moved Object moved to here. (END)
- nitass_89166
you want curl from f5 command line. But how to page the output ? because it's overflowing the putty window - even with |more
you can use -I to request only header (i.e. HEAD method).
e.g.
curl -Iv -H "Host: www.mydomain.com" http://ip_address/natural-language-url-1based on the output, requests seem to be same but responses are different. is there any specific configuration on server?
would it be possible to not use snat as John asked?
- I am not a networking engineer, but here is my understanding : This is not an isolated test environment, and to construct one is prohibitively complex & time-consuming. The dev-test F5 ( which I am using to develop the new iRules ) is on the same internal engineering subnet as other non-prod F5s. To be functional, the dev-test website must talk to shared db-servers and web-services, and therefore I have the web-server gateway set to a VIP of one of those other F5s ( otherwise I would have to do a total network configuration on the dev-test F5 - as opposed to just what I need to test natural-language URLs ). I added SNAT to the dev-test F5 virtual-server because the only other option to ensure responses flowed back through the F5 was to add a static route to the web-server, and this was not representative of how it would work in a prod environment. One thing I have noticed is that some of the virtual addresses ( different from those used by the vips the test harness hits ) are red, indicating they are not available, but I don't know why. I am wondering if one of these vips could be the self-ip and this is the underlying cause of the redirect loop error ?
- nitass
you want curl from f5 command line. But how to page the output ? because it's overflowing the putty window - even with |more
you can use -I to request only header (i.e. HEAD method).
e.g.
curl -Iv -H "Host: www.mydomain.com" http://ip_address/natural-language-url-1based on the output, requests seem to be same but responses are different. is there any specific configuration on server?
would it be possible to not use snat as John asked?
- I am not a networking engineer, but here is my understanding : This is not an isolated test environment, and to construct one is prohibitively complex & time-consuming. The dev-test F5 ( which I am using to develop the new iRules ) is on the same internal engineering subnet as other non-prod F5s. To be functional, the dev-test website must talk to shared db-servers and web-services, and therefore I have the web-server gateway set to a VIP of one of those other F5s ( otherwise I would have to do a total network configuration on the dev-test F5 - as opposed to just what I need to test natural-language URLs ). I added SNAT to the dev-test F5 virtual-server because the only other option to ensure responses flowed back through the F5 was to add a static route to the web-server, and this was not representative of how it would work in a prod environment. One thing I have noticed is that some of the virtual addresses ( different from those used by the vips the test harness hits ) are red, indicating they are not available, but I don't know why. I am wondering if one of these vips could be the self-ip and this is the underlying cause of the redirect loop error ?
- Kevin_Stewart
Does data travel the same path to the shared db and web services when accessing directly or via VIP? If all of that is the same, I'm not sure it matters. Have you determined where the 302 is actually coming from (F5 or web server)?
- Kevin_Stewart
The web-server gateway IP is set to a VIP on the internal engineering network F5. This F5 routes traffic to a pool of DNS that resolve the hostnames to the appropriate server IPs.
Does that mean that the data path is the same for both? In other words, assuming the web server is talking to another web service and database in relation to a client request, does the web server communicate with these services in the same way regardless of how the client connects to the web server?
How would I determine whether the 302 is originating from the web-server or the F5 ? As I demonstrated above, curl from the F5 command-line direct to the web-server IP returns the entire page content as expected.
Completely understand that, and perhaps I'm grasping at straws here, but in the absence of anything else that might make sense, it would be helpful to know definitely which device is actually sending the redirect. So, if you run a tcpdump on the interface between the f5 and the web server and use the -Xs0 option, you'll see the HTTP payload coming from the web server (assuming it's not encrypted).
tcpdump -lnni [internal interface] -Xs0 [and any additional filters]This will ultimately tells us where to look for the problem.
