Forum Discussion

Joe_Hsy_45207's avatar
Joe_Hsy_45207
Icon for Nimbostratus rankNimbostratus
Jan 18, 2008

HTTP:collect 4MB limit bug>?

Hi,

 

 

I just learned that there is a limit to HTTP::collect of 4 mb before it will crash TMM:

 

 

https://support.f5.com/kb/en-us/solutions/public/6000/500/sol6578.html?sr=350501

 

 

This means many of the sample iRles on devcentral (credit card scrubber, for example) could cause TMM to crash! Anyway, we're working on a workaround for our iRule and I have the following question:

 

 

If the content-length for a http response is greater this 4MB, is there a way to collect the end of the content using HTTP::release and HTTP::collect pairs? In other words, in the HTTP_RESPONSE_DATA event, call HTTP::release, then HTTP::collect until the last content is recieved. I don't need to keep all the content, but I need an event to trigger when the last content is received.

 

 

The above seems doable if the content-length header exists and we know ahead of time the size of the content. However, how could this be made to work when content-length header is missing? Is there a way to know when the last byte of content has been received?

 

 

Alternatively, is there an event other than HTTP_RESPONSE_DATA which will trigger when the last byte of HTTP content is received from the server?

 

 

Thanks!

 

 

//Joe
application delivery
dev
devops
iRules

7 Replies

Sort By
  • Deb_Allen_18's avatar
    Deb_Allen_18
    Historic F5 Account
    Jan 19, 2008

    Hi Joe -

     

    The functional limit if you're running any manipulation of payload can actually be as small as 1MB if regex ops are involved.

     

    The workaround is to limit the amount collected to 1MB at a time (or 4MB if you just need to scan the payload).

     

    I just added a codeshare example covering the basics: HTTP Payload Collection

     

    Post back if you have more questions after looking it over.

     

    /deb

     

    • Arie's avatar
      Arie
      Icon for Altostratus rankAltostratus
      Mar 27, 2015

      The above link to the HTTP Payload Collection is not valid. The correct (new) URL is https://devcentral.f5.com/s/articles/http-payload-collection.

    • JRahm's avatar
      JRahm
      Icon for Admin rankAdmin
      Mar 27, 2015
      Thanks Arie, I fixed it in the link above.
  • Joe_Hsy_45207's avatar
    Joe_Hsy_45207
    Icon for Nimbostratus rankNimbostratus
    Jan 21, 2008
    Hi Deb,

     

     

    Thanks for the quick response and the codeshare sample! I will try it and let you know.

     

     

    I do have a question regarding how it would work when the response does not include the content-length header. I don't believe that setting the protocol to 1.0 will force the server to include a content-length header, but maybe I'm mistaken.

     

     

    Also, is there a way to have an event fire when the last byte of the response content is recieved from the server?

     

     

    Thanks again!

     

     

    //Joe
  • Deb_Allen_18's avatar
    Deb_Allen_18
    Historic F5 Account
    Jan 21, 2008
    Hi Joe -

     

     

    HTTP 1.0 spec doesn't support chunking, so the server should not send a chunked response to a v1.0 request, instead will have to include the standard content length header up front.

     

     

    There is no automatic event triggered when the last byte is received, but if you are iteratively collecting and counting as you go, it should be apparent when the DATA event is triggered for the final time for a given payload size. Once that condition has been met, you can either insert the remaining logic right there in the DATA event, or you can use the "TCP::notify response" command (Click here) to trigger the USER_RESPONSE event & add the remaining logic there instead.

     

     

    HTH

     

    /deb
  • Joe_Hsy_45207's avatar
    Joe_Hsy_45207
    Icon for Nimbostratus rankNimbostratus
    Jan 21, 2008
    Hi Deb,

     

     

    Thanks again for the quick response. I think there are cases when the content-length header is missing even when not chunking:

     

     

    http://blogs.msdn.com/david.wang/archive/2006/02/28/Why-IIS6-can-send-invalid-HTTP-Responses.aspx

     

     

    I don't know common this is, but seems to occur in real field situations. In those cases, I am assuming that the response could hang?

     

     

    Thanks for the tip on TCP::notify. The docs indicate that the USER_RESPONSE event may wait until the data is sent - does it wait until all content is sent or just what has been released? In other words, does each call to TCP::notify result in a USER_RESPONSE event? If I call TCP::notify for each of the collect/release iteration, would it fire multiple times or just one final time?

     

     

    Thanks again!

     

     

    //Joe
  • Deb_Allen_18's avatar
    Deb_Allen_18
    Historic F5 Account
    Jan 22, 2008
    The collect mechanism will time out internally after no data has been received for a short time, even if the entire specified content length has not been received. I'm not sure what the timing is there, but it's short enough to keep the connection from stalling and induced latency is minimal.

     

     

    Re: TCP::notify trigger timing: Great question, and I wasn't aware of that, so thanks for bringing it to our attention. Not sure exactly what the sequence would be, but I am trying to find out.

     

     

    Stay tuned for more on this.

     

     

    /deb