youtube.com problem (also with videos.google.com)

[Pedro_Madeira_7]

Hello everyone, I'm bit new to F5 technologies and I'm now facing a strange problem that I can't seem to solve so I'm turning to you vets to give me a hand. I have a customer which has a 1-pair cluster of Link Controllers balancing 2 digital circuits and each belongs to a different ISP (ISP A and ISP B) The problem is when internal users try to see youtube videos there's a problem that I'll try to explain as detailed as I can. The original request to the youtube site looks like this: http://www.youtube.com/watch?v=gnZft1YTcA0 GET /watch?v=gnZft1YTcA0 HTTP/1.1 Host: www.youtube.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Proxy-Connection: keep-alive Referer: http://www.youtube.com/ Cookie: PREF=gl=US&hl=en; GEO=41a13f98e394b5b35e668663d868b749cwwAAAAzUFRVWIfDAEc3wUg=; VISITOR_INFO1_LIVE=oHYwQDby4SA; watched_video_id_list=291ddca53b47f6b60f7db1322a040d03WwEAAABzCwAAAExtNzBKYXVIV2lB; use_hitbox=72c46ff6cbcdb7c5585c36411b6b334edAEAAAAw; BCSI-CS-00BBC18B3F5304DD=2 HTTP/1.x 200 OK Date: Fri, 05 Sep 2008 14:17:51 GMT Server: Apache Expires: Tue, 27 Apr 1971 19:44:06 EST Cache-Control: no-cache Content-Type: text/html; charset=utf-8 Content-Length: 25042 Proxy-Connection: Keep-Alive Connection: Keep-Alive Content-Encoding: gzip Set-Cookie: watched_video_id_list=44b7c1c972fe9b974bb039801a5e92f4WwIAAABzCwAAAGduWmZ0MVlUY0EwcwsAAABMbTcwSmF1SFdpQQ==; path=/; domain=.youtube.com The user receives an HTTP 303 from the site redirecting him to a google video caching server http://www.youtube.com/get_video?video_id=gnZft1YTcA0&t=OEgsToPDskLxoujug81FFXZ4RRJfUJIZ GET /get_video?video_id=gnZft1YTcA0&t=OEgsToPDskLxoujug81FFXZ4RRJfUJIZ HTTP/1.1 Host: www.youtube.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Proxy-Connection: keep-alive Cookie: PREF=gl=US&hl=en; GEO=41a13f98e394b5b35e668663d868b749cwwAAAAzUFRVWIfDAEc3wUg=; VISITOR_INFO1_LIVE=oHYwQDby4SA; watched_video_id_list=44b7c1c972fe9b974bb039801a5e92f4WwIAAABzCwAAAGduWmZ0MVlUY0EwcwsAAABMbTcwSmF1SFdpQQ==; use_hitbox=72c46ff6cbcdb7c5585c36411b6b334edAEAAAAw; BCSI-CS-00BBC18B3F5304DD=2 HTTP/1.x 303 See Other Date: Fri, 05 Sep 2008 14:17:52 GMT Server: Apache Expires: Tue, 27 Apr 1971 19:44:06 EST Cache-Control: no-cache Location: http://v1.cache.googlevideo.com/get_video?video_id=gnZft1YTcA0&origin=ash-v190.ash.youtube.com&signature=CC4D530B3C7758DC76526042C6AD7D8B54352A4C.5629ADFF90E52BCA9E002A8583ACA62C466BCB8C&ip=85.88.135.195&ipbits=8&expire=1220645872&key=yt1&sver=2 Content-Type: text/html; charset=utf-8 Transfer-Encoding: chunked Proxy-Connection: Keep-Alive Connection: Keep-Alive If you pay close attention to the new redirect site, it includes the original public IP from ISP A that users use to go to the internet in the URL. However whenever the user tries to leave through ISP B (or vice versa when original request is made through ISP B and redirect goes out through ISP A) he gets an http 403 forbidden because now he was balanced through the other internet link. This is a simple problem however hard to explain in words but in sum, the client reaches youtube through ISP A, google redirects user to go to a caching video server to get the video, user is balanced to go to this new website through ISP B, and since the redirect url includes the original NAT IP, whenever the user uses the wrong link to get the file he gets an http 403 like in the below example: http://v1.cache.googlevideo.com/get_video?video_id=gnZft1YTcA0&origin=ash-v190.ash.youtube.com&signature=CC4D530B3C7758DC76526042C6AD7D8B54352A4C.5629ADFF90E52BCA9E002A8583ACA62C466BCB8C&ip=85.88.135.195&ipbits=8&expire=1220645872&key=yt1&sver=2 GET /get_video?video_id=gnZft1YTcA0&origin=ash-v190.ash.youtube.com&signature=CC4D530B3C7758DC76526042C6AD7D8B54352A4C.5629ADFF90E52BCA9E002A8583ACA62C466BCB8C&ip=85.88.135.195&ipbits=8&expire=1220645872&key=yt1&sver=2 HTTP/1.1 Host: v1.cache.googlevideo.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Proxy-Connection: keep-alive HTTP/1.x 403 Forbidden Content-Type: text/plain Date: Fri, 05 Sep 2008 14:17:52 GMT Server: gvs 1.0 Connection: close I need to find out a way of creating a Universal persistence iRule that is able to associate the original request to youtube.com and the redirect to a video caching server (tipically resident in domain googlevideos.com). I think that the probable best way to get the data to make the persistence is to look at the http 303 redirect url to look for the original NAT IP used by the client when he made the original request through one of the internet links. So if one of the IPs is found the user is forced through the proper link, if the other IP is found, the user is forced through the other link. The customer is basically using two IPs to go out to the internet based on the link chosen at the time. The IPs are 85.88.135.195 and 213.58.138.182. Any help you guys can provide me with this would be immensely appreciated. Best regards, Pedro Madeira

[Pedro_Madeira_7]

Hello guys, I've been searching some iRules code that I could use to solve this problem. I'm still new to this but I was able to come out with this: when HTTP_RESPONSE { if { [HTTP::is_redirect] }{ if {[findstr [HTTP::uri]] contains "85.88.135.19"} { pool pool1 }else { pool pool2 } } I'm sure the syntax is wrong and I was looking forward if you guys could help me fix the iRule. Basically my idea is to make the Link controller read the HTTP 303 redirect sent from youtube to the user. Location: http://v1.cache.googlevideo.com/get_video?video_id=gnZft1YTcA0&origin=ash-v190.ash.youtube.com&signature=CC4D530B3C7758DC76526042C6AD7D8B54352A4C.5629ADFF90E52BCA9E002A8583ACA62C466BCB8C&ip=85.88.135.195&ipbits=8&expire=1220645872&key=yt1&sver=2 In the above line, when the HTTP 303 is issued, youtube informs the user of the location of the video he's trying to get. If you play close attention to the URL, it contains the NAT public IP with which the user left to the internet (lets suppose it's ISP A). So I wanna make an iRule that is able to read this line, find if the IP used is the nat public IP of ISP A or ISP B and based on the findings, force the user to go to that new location through the same IP as of the original request. Looking forward to hear from you guys. Best regards, Pedro Madeira

[Pedro_Madeira_7]

Based on sample codes I found in dev central I was able to glue some code together to try and read the http 303 response. I need your help to confirm me if this iRule has the correct syntax and if it will work. This afternoon I'm thinking on going to the end customer facilities and try this code. The iRule I wanna test is: when HTTP_RESPONSE { if { [HTTP::is_redirect] }{ if { [findstr [HTTP::uri]] contains "85.88.135.19"} { pool link_colt } elseif { [findstr [HTTP::uri]] contains "213.58.138.182"} { pool link_oni } } The objective is to read the Location string supplied with the HTTP 303 sent from youtube.com. Let me remind you the http 303 response sent. HTTP/1.x 303 See Other Date: Fri, 05 Sep 2008 14:17:52 GMT Server: Apache Expires: Tue, 27 Apr 1971 19:44:06 EST Cache-Control: no-cache Location: http://v1.cache.googlevideo.com/get_video?video_id=gnZft1YTcA0&origin=ash-v190.ash.youtube.com&signature=CC4D530B3C7758DC76526042C6AD7D8B54352A4C.5629ADFF90E52BCA9E002A8583ACA62C466BCB8C&ip=85.88.135.195&ipbits=8&expire=1220645872&key=yt1&sver=2Content-Type: text/html; charset=utf-8 Transfer-Encoding: chunked Proxy-Connection: Keep-Alive Connection: Keep-Alive So basically if you look at the Location header field you will see that within that string it's mentioned the public IP with which the client leaves out to the internet. One of the doubts I have regarding the iRule I built is that I think I'm lacking a code line to make the user leave through the same link when he follows the HTTP 303 indication. Because the user first makes a request to youtube.com, then he gets the HTTP 303 and then he visits a different domain where google's video caching servers are located. In the iRule I built I think I'm lacking a way to follow the entire session, and when the user gets the http 303, his next request should go through one of either ISPs based on the string found when I read the Location field from the 303 response.

[Pedro_Madeira_7]

I've been giving it some thought and I think the answer is simpler than I was thinking, instead of reading the http 303 response, I can read the final client request to google's video caching server and I look for the string in the http URI and make a persistence based on my findings of the IP used in the request. iRule looks like this: Second attempt to persist on http 303 when HTTP_REQUEST { if { [findstr [HTTP::uri]] contains "85.88.135.19} { pool link_colt } elseif { [findstr [HTTP::uri]] contains "213.58.138.182"} { pool link_oni } } Will post my findings after I test this on production system.

[hooleylist]

Hello Pedro, Do you have the two links defined in the same pool? If so, have you tried adding a source address persistence profile to the VIP? That should ensure that all requests from the same client go out the same link. If that doesn't work, can you describe why? Else, there might be a way to support this using an iRule and the session table. Aaron

[Pedro_Madeira_7]

Hello Aaron, Let me thank you in the first place for your reply. User interactivity is indeed the best way to help each other. The problem I was facing is already solved. Not by iRule usage because F5 support told me that they do not officially support iRules. I couldn't use a source based persistence because behind the link controllers were a cluster of Bluecoat proxies and they were hiding all the user space address. I was able to solve the problem by pinpointing the source of it. Somehow the load balancing algorithm was the cause of the problem. It was set to fastest application (I have to read about this method since I don't know how it works) and changed it to least connections. As soon as I set the least connections balancing method, the problem was gone. Somehow fastest application is incompatible to video.google.com and youtube.com architecture. Regarding the iRule, when I tried to insert the following iRule into the Link Controller: when HTTP_REQUEST { if { [findstr [HTTP::uri] contains "85.88.135.19"]} { pool Router_COLT } elseif { [findstr [HTTP::uri] contains "213.58.138.182"]} { pool Router_ONI } } It came out with the following error messages: 01070151:3: Rule [youtube_persist] error: line 1: [ltm_rule_L7 feature not licensed] [when HTTP_REQUEST { if { [findstr [HTTP::uri] contains "85.88.135.19"]} { pool Router_COLT } elseif { [findstr [HTTP::uri] contains "213.58.138.182"]} { pool Router_ONI } }] line 2: [ltm_rule_L7 feature not licensed] [HTTP::uri] line 2: [invalid integer] ["85.88.135.19"] line 4: [ltm_rule_L7 feature not licensed] [HTTP::uri] line 4: [invalid integer] ["213.58.138.182"] It was at this time that I abandoned the iRule solution when F5 support told me they weren't supporting iRules. Best regards, Pedro Madeira

[hooleylist]

Ah, I didn't think about the license limitation with Link Controller. It looks like it's not possible to inspect the URI with the LC license. I'm not sure that least connections would ensure the client uses the same gateway to reach the destination server. Least connections load balancing would mean a new client connection would be sent to the pool member with the lowest number of existing TCP connections. Aaron

[Nicolas_Menant]

with LC, you're stuck to L4 capabilities. forget everything about L7.

[Pedro_Madeira_7]

Yeps. That's what I realized. Initially I thought the problem was because of lack of persistence but when I changed to least connections (member) and the problem disappeared, then I concluded that Fastest Application load balancing method has some kind of incompatibility with youtube and google. I sincerely hope F5 devs read this post and investigate the matter. And yes again, it seems that with link controller, our hands are tied to L4 decisions. Bye bye iRules. It's a pity.

[Pedro_Madeira_7]

Hi everyone, For some weird reason the problem seems to be back. Load balancing method is still set to least connections so that puts aside that the problem was directly related to the fastest application LB method. I really need someone to help me, if possible, to find a way on how to create persistence in link controller. It can't be based on source IP because I have a cluster of proxies behind link controller so if I build a persistence based on source address, all the connections from the proxies will be persisted to a single link. Based on destination IP it's also a problem, since youtube works with http 303 redirects. It seems that link controller is unable to set persistence based on cookies like we can do with LTM. Any ideas? Thanks once again, Pedro Madeira

[hooleylist]

Hi Pedro, I wouldn't expect any default load balancing algorithm to solve this problem as it wouldn't guarantee that a client ever gets sent to the same link. Persistence could solve the problem if you were able to use source address or cookie (or another L7) persistence method to pin a client to the same outgoing link every time. But I can't see a way to handle this if you have proxies between the client and LC, and you can't use L7 iRules or persistence with LC. Aaron

[Pedro_Madeira_7]

Hi there, First of all thanks for your replies which I appreciate very much. This iRule you built seems capable of doing the trick as I presume that it doesn't have any L7 attibutes. In the past I came up with a similar iRule but it had L7 features and LC doesn't support L7 decisions. Basing that iRule with dynamic dns resolutions would be better since there are at least 3 different domains related to the supply of videos. But I think I can try to define 3 classes (1 for youtube.com, 1 for googlevideos.com, and another for some other network google works with). Anyway, next week I'll be able to try this iRule and Ill keep you guys updated on how it worked. Thanks once again for your support, Pedro

[hwidjaja_37598]

Yeah, I just 'downgrade' your iRules not to use L7. Just to let you know, the youtube class has all network for youtube.com and googlevideos.com (even google.com).

[hwidjaja_37598]

Another issue with the previous iRules is when the link for youtube is down. There will be no redundancy. I think you can create a new pool with priority enabled, so the LC would be able to re-select the other link when the link for youtube is down.

  
  when CLIENT_ACCEPTED {      
      if { [matchclass [IP::local_addr] equals $::youtube] } {      
      pool   
      } else {      
      pool       
      }      
      }      
  

[Miguel_Santos]

Hello, I'm a colleague from Pedro, I tried the last solution provided by widjaja, and it worked fine. Thank you all for the help you have given us. brgds Miguel Santos

[HailyGibbons]

The Information you shared above is great. I have been reading all you shared here. In this you explained everything very well. If i want any further guideline we will contact you here https://community.f5.com/t5/technical-forum/youtube-com-gosloto-problem-also-with-videos-google-com/td-p/188396.

Thanks