Forum Discussion
JamesS_40157
Nimbostratus
NimbostratusHTTP::collect and HTTP::payload corrupt?
Hi,
I am trying to create an iRule that, when a response is sent back to the client (the client initiates a GET request for a file, in this case an image), it will collect the body. The iRule should then POST that content back through the virtual server, with some modified headers.
Unfortunately it seems like for whatever reason the payload in the POST has been truncated (i have confirmed this with wireshark dumps, although i cant work out exactly by how many bytes its been truncated), I can't for the life of me work out why! I've posted the relevant logic below (i have truncated some parts that are irrelevant although if you want to i can post the full irule, it's still a work in progress however!). The image I am testing with is 7737 bytes in size and all my logs indicate that it is getting the correct content length of 7737 bytes...does anyone know what the issue could be?
when HTTP_RESPONSE {
set contentlength [HTTP::header Content-Length]
if { $debug>=3 } { log local0.info "$castoruri found in production castor domain. Collecting file of length $contentlength to POST back to $original_partition environment" }
HTTP::collect $contentlength
}
when HTTP_RESPONSE_DATA {
if { $debug>=3 } { log local0.info "HTTP_RESPONSE_DATA triggered for castoruri $castoruri" }
set payload [HTTP::payload]
This POSTs back through the iRule like a client request, and so the iRule will trigger HTTP_REQUEST again
if { $debug>=3 } {log local0.info "REQUEST IS POST $original_path HTTP/1.1\r\nHost: $original_castordomain\r\nContent-Length: $contentlength\r\n\r\n$payload" }
HTTP::retry "POST $original_path HTTP/1.1\r\nHost: $original_castordomain\r\nX-Forwarded-Host: $original_castordomain\r\nContent-Length: $contentlength\r\n\r\n$payload\r\n"
}
All variables above are present and correct, what am I missing? Any help much appreciated :)
Logs:
: /default/JAMES12.JPG found in production castor domain. Collecting file of length 7737 to POST back to qa environment
: HTTP_RESPONSE_DATA triggered for castoruri /default/JAMES12.JPG
: REQUEST IS POST /JAMES12.JPG HTTP/1.1 Host: castor.qa.rightmove.com Content-Length: 7737 ÿØÿà��020JFIF��001001001��`��`����ÿþ��036LEAD Technologies Inc. V1.01ÿÛ��C��003002002002002002003002002002003003003003004006004004004004004010006006005006011010 011010011011 014017014 013016013011011 021 016017020020021020 014022023022020023017020020020ÿÛ��C001003003003004003004010004004010020013011013020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020020ÿÀ��021010��´��‡003001"��002021001003021001ÿÄ��035����001005001001001001������������������005��003004006007010002001011ÿÄ��B020��002001003002003004007006004003007005������001002003��004021005!0060221007023AQ024"aq�‘¡0102B±Á0253RÑb¢Â026$‚’²áñ4CSƒ”ÿÄ��033001��002002003001����������������������003004002005��001006007ÿÄ��4021��002001003002004004003006007001����������001002��003004021!1005022023A006Qaq"2‘024B¡±Ñð0070253Rb�ÁÿÚ��014003001��002021003021��?��µiœe*C004·021jЬÜØRRP0pzîjÍo¯i2ÜMmtöí$001‹‰l˜025ÁßÖ025Õ—¿g016Íî—Ñ¡³¸·D˧ç005ŽýG°P{ß³g É5Ö¡o¨H¯z¯013«D010034ç031Î|016õ³N–àÂ013Š½Ìç…›†.R6Qf{Â@1Þ•$û025‡¶›ŸIÒ˜1�ïbÇOÁ(ÿ��)006¶035oì©5͵¥…†§hæÆVœ™024ŽñX�W033ÿ��OÖ²n?û,v�l8‡QÐôäŸø†$±ô{•Gþpr1Ì010<™ùb„É�Œ*\·Þ•{ømNM
JamesS_40157I have since modified the HTTP_RESPONSE_DATA to look inside the payload:
when HTTP_RESPONSE_DATA {
if { $debug>=3 } { log local0.info "HTTP_RESPONSE_DATA triggered for castoruri $castoruri" }
set payload [HTTP::payload]
binary scan [HTTP::payload] H* payload_hex
This POSTs back through the iRule like a client request, and so the iRule will trigger HTTP_REQUEST again
if { $debug>=3 } {log local0.info "REQUEST IS POST $original_path HTTP/1.1\r\nHost: $original_castordomain\r\nContent-Length: $contentlength\r\n\r\n$payload" }
HTTP::retry "POST $original_path HTTP/1.1\r\nHost: $original_castordomain\r\nX-Forwarded-Host: $original_castordomain\r\nContent-Length: $contentlength\r\n\r\n$payload\r\n"
HTTP::respond 200 content "
$payload_hex
"
}
I have compared the output in my browser with the hex dump of the original file and they are identical, which is a good sign! So it must be something to do with the POST request that is corrupting it..hmm- JamesS_40157I've stopped playing with this for a while, however I have a question (perhaps it's more for F5 support) - can the HTTP::retry manually formatted POST request include a binary payload? I reckon this is where the problem might lay?
hooleylistCirrostratus
Hi James,
The binary content gets converted to UTF8 if you try to append it to a string. Can you try this?
HTTP::retry "POST $original_path HTTP/1.1\r\nHost: $original_castordomain\r\nX-Forwarded-Host: $original_castordomain\r\nContent-Length: $contentlength\r\n\r\n[HTTP::payload]
Note that per RFC2616 section 5, the payload shouldn't have a CRLF at the end:
http://tools.ietf.org/html/rfc2616section-5
Once you're done testing, you might also want to limit which responses you collect the payload for (based on requested URI, response content-type, etc). And you should limit the amount of data you collect to 4Mb in pre-v11:
SOL6578 - TMM will crash if an iRule collects too much data
https://support.f5.com/kb/en-us/solutions/public/6000/500/sol6578.html
Aaron- JamesS_40157Thanks Aaron for the suggestions. I've tried what you suggest but unfortunately it is still truncating/modifying the payload somehow on the way out, in exactl the same way (i ran a tcpdump to see where the truncation was happening). I'm a bit lost on this one :(
nitassEmployee
where did you capture traffic? was it bigip? have you ever tried to capture on server instead?- JamesS_40157Hi nitass, the tcpdump was run on the server side, looking at the flow between the bigip vip and the server. I think we are going to write the logic in perl on an apache instance instead, i can't seem to work out why it's not posting the payload correctly and we don't really have the time to investigate the issue fully! Thanks for your help though (although if anyone really does know the answer i'd be happy to see it :) )