Forum Discussion
Reading chunked data in iRule
Hey, I have an iRule in which I read all the payloads in each request and response.
When there iRule receives chunked data, because there is no content-length I just use some big number as a limit on how much to read with HTTP::collect.
There problem is that by doing that I keep the api occupied while waiting for the EOF or to reach the limit in collect, and only then does it send the request/response forward, instead of each chunk reaching the other side when sent.
If I put a small limit to HTTP::collect I will miss out on more data as the payloads will be truncated.
Any way to read the whole chunked payload at the very end, or reading in a loop each chunk while also letting it reach the destination?
- zamroni777Nacreous
you might try to use irules HTML or XML object:
https://clouddocs.f5.com/api/irules/HTML_TAG_MATCHED.html
https://clouddocs.f5.com/api/irules/XML_CONTENT_BASED_ROUTING.htmlor use irules lx if the payload in json becuse iruleslx uses node.js engine that can process json natively.
- Lucas_ThompsonEmployee
Sounds like a fun and challenging project!
To do this the right way you'll have to read the byte count in the chunk header and collect that much OR a fixed value, rather than collecting a fixed value all the time. This way your loop can be sure to exit at the right time. The last line in a chunked response is always "0\r\n", and you can also test for that to make sure the response is completed.
One other gotcha here is that there are architectural oddities about how BIG-IP's HTTP profile works with HTTP chunk mode together with HTTP compression to coordinate traffic between the server-side and client-side halves of the flow. If you find something weird, try turning off chunk mode, or compression, or both.
One trick I've used successfully in the past is to avoid chunked mode altogether by using iRules to remove or modify the "accept-encoding" header so that the HTTP participants assume the other one doesn't support it. Clients and servers will usually fall back to normal Content-Length mode if instructed to.
- GuyGAltocumulus
Thanks for the reply.
This is sort of what I'm trying to do.
I have a simple iRule to start with where I'm trying to log the whole payload (all chunks combined) at the end of the response, while allowing the VS to send each chunk back to the client instead of buffering it.when HTTP_RESPONSE { log local0.debug "Start of HTTP_RESPONSE" set readSoFar 0 catch { HTTP::collect 1 } } when HTTP_RESPONSE_DATA { log local0.debug "Start of HTTP_RESPONSE_DATA" set prev_readSoFar $readSoFar set readSoFar [HTTP::payload length] log local0.debug "readSoFar: |$readSoFar|, prev_readSoFar: |$prev_readSoFar|" if { $readSoFar == $prev_readSoFar } { log local0.debug "Whole body: |[HTTP::payload]|" } else { HTTP::release catch { HTTP::collect [expr {$readSoFar + 1}] } } } when HTTP_RESPONSE_RELEASE { log local0.debug "Start of HTTP_RESPONSE_RELEASE" }
I tried using HTTP::release to allow the VS to send each chunk to the client, then called HTTP::collect again to collect the next chunk, the iRule will stop calling collect when it stopped receiving chunks (that's the job of the if).
But when I do this, HTTP_RESPONSE_RELEASE is only triggered once at the very end of the iRule, and it seems like it isn't triggered when I call HTTP::release.
Logs:Rule /Common/xxxxx <HTTP_RESPONSE>: Start of HTTP_RESPONSE Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |6|, prev_readSoFar: |0| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |17|, prev_readSoFar: |6| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |43|, prev_readSoFar: |17| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |74|, prev_readSoFar: |43| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |116|, prev_readSoFar: |74| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |167|, prev_readSoFar: |116| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |183|, prev_readSoFar: |167| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |207|, prev_readSoFar: |183| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |686|, prev_readSoFar: |207| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |701|, prev_readSoFar: |686| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |712|, prev_readSoFar: |701| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |722|, prev_readSoFar: |712| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |727|, prev_readSoFar: |722| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |727|, prev_readSoFar: |727| Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Whole body: |1 6 { 14 "Status": { 19 "Code": "0", 24 "Desc": "交易æˆåŠŸ", 2d "TrxTime": "2023-12-05 03:04:01" b }, 12 "Body": { 1d8 "key": "sajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvls" a } 6 } 5 0 | Rule /Common/xxxxx <HTTP_RESPONSE_RELEASE>: Start of HTTP_RESPONSE_RELEASE
Any idea why it seems like HTTP::release is not working? Did I misunderstand how to use it?
- GuyGAltocumulusSplitting my reply to a few messages, because it didn't let me reply saying I was "Post Flooding".
Thanks for the reply.This is sort of what I'm trying to do.I have a simple iRule to start with where I'm trying to log the whole payload (all chunks combined) at the end of the response, while allowing the VS to send each chunk back to the client instead of buffering it.- GuyGAltocumulus
when HTTP_RESPONSE { log local0.debug "Start of HTTP_RESPONSE" set readSoFar 0 catch { HTTP::collect 1 } } when HTTP_RESPONSE_DATA { log local0.debug "Start of HTTP_RESPONSE_DATA" set prev_readSoFar $readSoFar set readSoFar [HTTP::payload length] log local0.debug "readSoFar: |$readSoFar|, prev_readSoFar: |$prev_readSoFar|" if { $readSoFar == $prev_readSoFar } { log local0.debug "Whole body: |[HTTP::payload]|" } else { HTTP::release catch { HTTP::collect [expr {$readSoFar + 1}] } } } when HTTP_RESPONSE_RELEASE { log local0.debug "Start of HTTP_RESPONSE_RELEASE" }
- GuyGAltocumulus
* Deleted *
Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com