Forum Discussion
Chris_Phillips
Nimbostratus
Nimbostratusstream + content-length + chunking
Hi,
i have an xml application for which i'm using a stream profile to do hostname rewrites. as i've seen from deb's comments on a number of posts, the resulting variance in the payload size and the content-length can be overcome by using rechunking in the attached http profile. seemed great but...
1) there appears to be all sorts of crap added after each chunk according to wireshark (stuff like "10e", "d" and other tiny bits of raw hex)
2) the client app doesn't seem to like the chunking at all, possibly becasue of the above corrupting the xml, maybe just because it doesn't support it.
another option i saw was to change the content-length header manunally. this however causes the client software to hang until a timeout occurs, which is unacceptable.
essentially i really think i need the data to reach the client "correctly", without any hacks in the data (which i can't predict the contents of - so no idea how many replacements may occur). the only other option i can think of is to actually collect the full payload and do a regsub and ditch the stream profile, but this would presumably have a massive performance hit.
any takers?
Cheers
Chris
John_Klemm_4418I may be able to help you out. I had some issues with sharepoint and trying to stream. show me your stream profile as it is now. Also, make sure that you have streaming enabled under you vip.
Chris_Phillipsthe streaming in itself is working fine, it's the fact that it changes the size of the payload i am having issues with.- Hi Chris --
Stream profile is low-overhead, performing replacements on the fly mid-stream, but requires chunking because the resulting payload size cannot be known when the Content-Length header would be sent.
Collecting the entire payload to perform replacement so size may be calculated is high overhead but allows explicit Content-Length declaration in header.
You can certainly use regsub to manipulate the entire payload, but as you suspected, it will carry a considerable performance hit. Search the forum & codeshare for "regsub" + "::payload" and you'll find a number of relevant examples.
/deb - Chris_PhillipsThanks for the replies, i've since got the HTTP_RESPONSE_DATA event firing after the HTTP::collect completes and the data is being rewritten correctly.
But... I guess there is no actual solution to what i'm asking then? just a case of having to live with whichever evil i think is the easiest to cope with, whether it's performance on the box or complexity of iRules..?
if so, that does kinda suck. I can totally appreciate developers wanting to add functionality whever they can, but it does leave a number of really great options which are incompatible with one another... 
hoolioCirrostratus
If I understand Deb's explanation of the process correctly, it's not a limitation in the rule functionality, but a limitation of the scenarios.
The stream profile is fast because it doesn't buffer all of the content. As the full content isn't being collected before the substitution is done, you don't know what the content length will be after the substitution(s) are complete. Therefore, you can't set the content-length. To resolve this you need to use chunking.
If you want to avoid chunking and set the content-length correctly, then you have to buffer the full content. This can be done with TCP/HTTP::collect, but uses more resources to do.
Aaron- Chris_Phillipsno, i'd not class it as a iRule issue, more what the boxes we spent a lot of money on being able to do what we want them to... not that they ever said they could do that of course, but still it's frustrating as a customer. looks like we'll be using some other plan of attack here, like a real http proxy server.
thanks
chris
