Forum Discussion

Nimbostratus

Sep 08, 2008

youtube.com problem (also with videos.google.com)

Hello everyone,

I'm bit new to F5 technologies and I'm now facing a strange problem that I can't seem to solve so I'm turning to you vets to give me a hand.

I have a customer which has a 1-pair cluster of Link Controllers balancing 2 digital circuits and each belongs to a different ISP (ISP A and ISP B)

The problem is when internal users try to see youtube videos there's a problem that I'll try to explain as detailed as I can.

The original request to the youtube site looks like this:

http://www.youtube.com/watch?v=gnZft1YTcA0

GET /watch?v=gnZft1YTcA0 HTTP/1.1

Host: www.youtube.com

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Proxy-Connection: keep-alive

Referer: http://www.youtube.com/

Cookie: PREF=gl=US&hl=en; GEO=41a13f98e394b5b35e668663d868b749cwwAAAAzUFRVWIfDAEc3wUg=; VISITOR_INFO1_LIVE=oHYwQDby4SA; watched_video_id_list=291ddca53b47f6b60f7db1322a040d03WwEAAABzCwAAAExtNzBKYXVIV2lB; use_hitbox=72c46ff6cbcdb7c5585c36411b6b334edAEAAAAw; BCSI-CS-00BBC18B3F5304DD=2

HTTP/1.x 200 OK

Date: Fri, 05 Sep 2008 14:17:51 GMT

Server: Apache

Expires: Tue, 27 Apr 1971 19:44:06 EST

Cache-Control: no-cache

Content-Type: text/html; charset=utf-8

Content-Length: 25042

Proxy-Connection: Keep-Alive

Connection: Keep-Alive

Content-Encoding: gzip

Set-Cookie: watched_video_id_list=44b7c1c972fe9b974bb039801a5e92f4WwIAAABzCwAAAGduWmZ0MVlUY0EwcwsAAABMbTcwSmF1SFdpQQ==; path=/; domain=.youtube.com

The user receives an HTTP 303 from the site redirecting him to a google video caching server

http://www.youtube.com/get_video?video_id=gnZft1YTcA0&t=OEgsToPDskLxoujug81FFXZ4RRJfUJIZ

GET /get_video?video_id=gnZft1YTcA0&t=OEgsToPDskLxoujug81FFXZ4RRJfUJIZ HTTP/1.1

Host: www.youtube.com

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Proxy-Connection: keep-alive

Cookie: PREF=gl=US&hl=en; GEO=41a13f98e394b5b35e668663d868b749cwwAAAAzUFRVWIfDAEc3wUg=; VISITOR_INFO1_LIVE=oHYwQDby4SA; watched_video_id_list=44b7c1c972fe9b974bb039801a5e92f4WwIAAABzCwAAAGduWmZ0MVlUY0EwcwsAAABMbTcwSmF1SFdpQQ==; use_hitbox=72c46ff6cbcdb7c5585c36411b6b334edAEAAAAw; BCSI-CS-00BBC18B3F5304DD=2

HTTP/1.x 303 See Other

Date: Fri, 05 Sep 2008 14:17:52 GMT

Server: Apache

Expires: Tue, 27 Apr 1971 19:44:06 EST

Cache-Control: no-cache

Location: http://v1.cache.googlevideo.com/get_video?video_id=gnZft1YTcA0&origin=ash-v190.ash.youtube.com&signature=CC4D530B3C7758DC76526042C6AD7D8B54352A4C.5629ADFF90E52BCA9E002A8583ACA62C466BCB8C&ip=85.88.135.195&ipbits=8&expire=1220645872&key=yt1&sver=2

Content-Type: text/html; charset=utf-8

Transfer-Encoding: chunked

Proxy-Connection: Keep-Alive

Connection: Keep-Alive

If you pay close attention to the new redirect site, it includes the original public IP from ISP A that users use to go to the internet in the URL.

However whenever the user tries to leave through ISP B (or vice versa when original request is made through ISP B and redirect goes out through ISP A) he gets an http 403 forbidden because now he was balanced through the other internet link.

This is a simple problem however hard to explain in words but in sum, the client reaches youtube through ISP A, google redirects user to go to a caching video server to get the video, user is balanced to go to this new website through ISP B, and since the redirect url includes the original NAT IP, whenever the user uses the wrong link to get the file he gets an http 403 like in the below example:

http://v1.cache.googlevideo.com/get_video?video_id=gnZft1YTcA0&origin=ash-v190.ash.youtube.com&signature=CC4D530B3C7758DC76526042C6AD7D8B54352A4C.5629ADFF90E52BCA9E002A8583ACA62C466BCB8C&ip=85.88.135.195&ipbits=8&expire=1220645872&key=yt1&sver=2

GET /get_video?video_id=gnZft1YTcA0&origin=ash-v190.ash.youtube.com&signature=CC4D530B3C7758DC76526042C6AD7D8B54352A4C.5629ADFF90E52BCA9E002A8583ACA62C466BCB8C&ip=85.88.135.195&ipbits=8&expire=1220645872&key=yt1&sver=2 HTTP/1.1

Host: v1.cache.googlevideo.com

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Proxy-Connection: keep-alive

HTTP/1.x 403 Forbidden

Content-Type: text/plain

Date: Fri, 05 Sep 2008 14:17:52 GMT

Server: gvs 1.0

Connection: close

I need to find out a way of creating a Universal persistence iRule that is able to associate the original request to youtube.com and the redirect to a video caching server (tipically resident in domain googlevideos.com). I think that the probable best way to get the data to make the persistence is to look at the http 303 redirect url to look for the original NAT IP used by the client when he made the original request through one of the internet links. So if one of the IPs is found the user is forced through the proper link, if the other IP is found, the user is forced through the other link.

The customer is basically using two IPs to go out to the internet based on the link chosen at the time. The IPs are 85.88.135.195 and 213.58.138.182.

Any help you guys can provide me with this would be immensely appreciated.

Best regards,

Pedro Madeira

15 Replies

Pedro_Madeira_7
Nimbostratus
Sep 09, 2008
Hello guys,

I've been searching some iRules code that I could use to solve this problem. I'm still new to this but I was able to come out with this:

when HTTP_RESPONSE {

if { [HTTP::is_redirect] }{

if {[findstr [HTTP::uri]] contains "85.88.135.19"} {

pool pool1

}else {

pool pool2

}

}

I'm sure the syntax is wrong and I was looking forward if you guys could help me fix the iRule.

Basically my idea is to make the Link controller read the HTTP 303 redirect sent from youtube to the user.

Location: http://v1.cache.googlevideo.com/get_video?video_id=gnZft1YTcA0&origin=ash-v190.ash.youtube.com&signature=CC4D530B3C7758DC76526042C6AD7D8B54352A4C.5629ADFF90E52BCA9E002A8583ACA62C466BCB8C&ip=85.88.135.195&ipbits=8&expire=1220645872&key=yt1&sver=2

In the above line, when the HTTP 303 is issued, youtube informs the user of the location of the video he's trying to get. If you play close attention to the URL, it contains the NAT public IP with which the user left to the internet (lets suppose it's ISP A).

So I wanna make an iRule that is able to read this line, find if the IP used is the nat public IP of ISP A or ISP B and based on the findings, force the user to go to that new location through the same IP as of the original request.

Looking forward to hear from you guys.

Best regards,

Pedro Madeira
Pedro_Madeira_7
Nimbostratus
Sep 10, 2008
Based on sample codes I found in dev central I was able to glue some code together to try and read the http 303 response.

I need your help to confirm me if this iRule has the correct syntax and if it will work. This afternoon I'm thinking on going to the end customer facilities and try this code.

The iRule I wanna test is:

when HTTP_RESPONSE {

if { [HTTP::is_redirect] }{

if { [findstr [HTTP::uri]] contains "85.88.135.19"} {

pool link_colt

} elseif { [findstr [HTTP::uri]] contains "213.58.138.182"} {

pool link_oni

}

}

The objective is to read the Location string supplied with the HTTP 303 sent from youtube.com.

Let me remind you the http 303 response sent.

HTTP/1.x 303 See Other

Date: Fri, 05 Sep 2008 14:17:52 GMT

Server: Apache

Expires: Tue, 27 Apr 1971 19:44:06 EST

Cache-Control: no-cache

Location: http://v1.cache.googlevideo.com/get_video?video_id=gnZft1YTcA0&origin=ash-v190.ash.youtube.com&signature=CC4D530B3C7758DC76526042C6AD7D8B54352A4C.5629ADFF90E52BCA9E002A8583ACA62C466BCB8C&ip=85.88.135.195&ipbits=8&expire=1220645872&key=yt1&sver=2Content-Type: text/html; charset=utf-8

Transfer-Encoding: chunked

Proxy-Connection: Keep-Alive

Connection: Keep-Alive

So basically if you look at the Location header field you will see that within that string it's mentioned the public IP with which the client leaves out to the internet.

One of the doubts I have regarding the iRule I built is that I think I'm lacking a code line to make the user leave through the same link when he follows the HTTP 303 indication.

Because the user first makes a request to youtube.com, then he gets the HTTP 303 and then he visits a different domain where google's video caching servers are located. In the iRule I built I think I'm lacking a way to follow the entire session, and when the user gets the http 303, his next request should go through one of either ISPs based on the string found when I read the Location field from the 303 response.
Pedro_Madeira_7
Nimbostratus
Sep 10, 2008
I've been giving it some thought and I think the answer is simpler than I was thinking, instead of reading the http 303 response, I can read the final client request to google's video caching server and I look for the string in the http URI and make a persistence based on my findings of the IP used in the request.

iRule looks like this:

Second attempt to persist on http 303

when HTTP_REQUEST {

if { [findstr [HTTP::uri]] contains "85.88.135.19} {

pool link_colt

} elseif { [findstr [HTTP::uri]] contains "213.58.138.182"} {

pool link_oni

}

}

Will post my findings after I test this on production system.
hooleylist
Cirrostratus
Sep 15, 2008
Hello Pedro,

Do you have the two links defined in the same pool? If so, have you tried adding a source address persistence profile to the VIP? That should ensure that all requests from the same client go out the same link. If that doesn't work, can you describe why? Else, there might be a way to support this using an iRule and the session table.

Aaron
Pedro_Madeira_7
Nimbostratus
Sep 15, 2008
Hello Aaron,

Let me thank you in the first place for your reply. User interactivity is indeed the best way to help each other.

The problem I was facing is already solved. Not by iRule usage because F5 support told me that they do not officially support iRules.

I couldn't use a source based persistence because behind the link controllers were a cluster of Bluecoat proxies and they were hiding all the user space address.

I was able to solve the problem by pinpointing the source of it. Somehow the load balancing algorithm was the cause of the problem. It was set to fastest application (I have to read about this method since I don't know how it works) and changed it to least connections.

As soon as I set the least connections balancing method, the problem was gone.

Somehow fastest application is incompatible to video.google.com and youtube.com architecture.

Regarding the iRule, when I tried to insert the following iRule into the Link Controller:

when HTTP_REQUEST {

if { [findstr [HTTP::uri] contains "85.88.135.19"]} {

pool Router_COLT

} elseif { [findstr [HTTP::uri] contains "213.58.138.182"]} {

pool Router_ONI

}

}

It came out with the following error messages:

01070151:3: Rule [youtube_persist] error:

line 1: [ltm_rule_L7 feature not licensed] [when HTTP_REQUEST {

if { [findstr [HTTP::uri] contains "85.88.135.19"]} {

pool Router_COLT

} elseif { [findstr [HTTP::uri] contains "213.58.138.182"]} {

pool Router_ONI

}

}]

line 2: [ltm_rule_L7 feature not licensed] [HTTP::uri]

line 2: [invalid integer] ["85.88.135.19"]

line 4: [ltm_rule_L7 feature not licensed] [HTTP::uri]

line 4: [invalid integer] ["213.58.138.182"]

It was at this time that I abandoned the iRule solution when F5 support told me they weren't supporting iRules.

Best regards,

Pedro Madeira
hooleylist
Cirrostratus
Sep 15, 2008
Ah, I didn't think about the license limitation with Link Controller. It looks like it's not possible to inspect the URI with the LC license.

I'm not sure that least connections would ensure the client uses the same gateway to reach the destination server. Least connections load balancing would mean a new client connection would be sent to the pool member with the lowest number of existing TCP connections.

Aaron
Nicolas_Menant
Employee
Sep 17, 2008
with LC, you're stuck to L4 capabilities. forget everything about L7.
Pedro_Madeira_7
Nimbostratus
Sep 17, 2008
Yeps.

That's what I realized. Initially I thought the problem was because of lack of persistence but when I changed to least connections (member) and the problem disappeared, then I concluded that Fastest Application load balancing method has some kind of incompatibility with youtube and google.

I sincerely hope F5 devs read this post and investigate the matter.

And yes again, it seems that with link controller, our hands are tied to L4 decisions. Bye bye iRules.

It's a pity.
Pedro_Madeira_7
Nimbostratus
Oct 08, 2008
Hi everyone,

For some weird reason the problem seems to be back. Load balancing method is still set to least connections so that puts aside that the problem was directly related to the fastest application LB method.

I really need someone to help me, if possible, to find a way on how to create persistence in link controller.

It can't be based on source IP because I have a cluster of proxies behind link controller so if I build a persistence based on source address, all the connections from the proxies will be persisted to a single link.

Based on destination IP it's also a problem, since youtube works with http 303 redirects.

It seems that link controller is unable to set persistence based on cookies like we can do with LTM.

Any ideas?

Thanks once again,

Pedro Madeira
hooleylist
Cirrostratus
Oct 08, 2008
Hi Pedro,

I wouldn't expect any default load balancing algorithm to solve this problem as it wouldn't guarantee that a client ever gets sent to the same link. Persistence could solve the problem if you were able to use source address or cookie (or another L7) persistence method to pin a client to the same outgoing link every time. But I can't see a way to handle this if you have proxies between the client and LC, and you can't use L7 iRules or persistence with LC.

Aaron