Forum Discussion

GuyG's avatar
GuyG
Icon for Altocumulus rankAltocumulus
Dec 06, 2023

Reading chunked data in iRule

Hey, I have an iRule in which I read all the payloads in each request and response. When there iRule receives chunked data, because there is no content-length I just use some big number as a limit...
devops
Lucas_Thompson's avatar
Lucas_Thompson
Icon for Employee rankEmployee
Dec 06, 2023

Sounds like a fun and challenging project!

To do this the right way you'll have to read the byte count in the chunk header and collect that much OR a fixed value, rather than collecting a fixed value all the time. This way your loop can be sure to exit at the right time. The last line in a chunked response is always "0\r\n", and you can also test for that to make sure the response is completed. 

One other gotcha here is that there are architectural oddities about how BIG-IP's HTTP profile works with HTTP chunk mode together with HTTP compression to coordinate traffic between the server-side and client-side halves of the flow. If you find something weird, try turning off chunk mode, or compression, or both. 

One trick I've used successfully in the past is to avoid chunked mode altogether by using iRules to remove or modify the "accept-encoding" header so that the HTTP participants assume the other one doesn't support it. Clients and servers will usually fall back to normal Content-Length mode if instructed to. 

GuyG's avatar
GuyG
Icon for Altocumulus rankAltocumulus
Dec 12, 2023

Thanks for the reply.

This is sort of what I'm trying to do.
I have a simple iRule to start with where I'm trying to log the whole payload (all chunks combined) at the end of the response, while allowing the VS to send each chunk back to the client instead of buffering it.

 

 

 

when HTTP_RESPONSE {
    log local0.debug "Start of HTTP_RESPONSE"
    set readSoFar 0
    catch { HTTP::collect 1 }
}

when HTTP_RESPONSE_DATA {
    log local0.debug "Start of HTTP_RESPONSE_DATA"

    set prev_readSoFar $readSoFar
    set readSoFar [HTTP::payload length]
    log local0.debug "readSoFar: |$readSoFar|, prev_readSoFar: |$prev_readSoFar|"
    
    if { $readSoFar == $prev_readSoFar } {
        log local0.debug "Whole body: |[HTTP::payload]|"
    } else {
        HTTP::release

        catch { HTTP::collect [expr {$readSoFar + 1}] }
    }
}

when HTTP_RESPONSE_RELEASE {
    log local0.debug "Start of HTTP_RESPONSE_RELEASE"
}

 

 

 

I tried using HTTP::release to allow the VS to send each chunk to the client, then called HTTP::collect again to collect the next chunk, the iRule will stop calling collect when it stopped receiving chunks (that's the job of the if).
But when I do this, HTTP_RESPONSE_RELEASE is only triggered once at the very end of the iRule, and it seems like it isn't triggered when I call HTTP::release.

Logs:

 

 

 

Rule /Common/xxxxx <HTTP_RESPONSE>: Start of HTTP_RESPONSE
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |6|, prev_readSoFar: |0|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |17|, prev_readSoFar: |6|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |43|, prev_readSoFar: |17|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |74|, prev_readSoFar: |43|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |116|, prev_readSoFar: |74|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |167|, prev_readSoFar: |116|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |183|, prev_readSoFar: |167|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |207|, prev_readSoFar: |183|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |686|, prev_readSoFar: |207|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |701|, prev_readSoFar: |686|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |712|, prev_readSoFar: |701|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |722|, prev_readSoFar: |712|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |727|, prev_readSoFar: |722|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |727|, prev_readSoFar: |727|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Whole body: |1     6      {   14          "Status": {   19              "Code": "0",   24              "Desc": "交易成功",   2d              "TrxTime": "2023-12-05 03:04:01"   b          },   12          "Body": {   1d8              "key": "sajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvls"   a          }   6      }   5         0    |
Rule /Common/xxxxx <HTTP_RESPONSE_RELEASE>: Start of HTTP_RESPONSE_RELEASE

 

 

 

 Any idea why it seems like HTTP::release is not working? Did I misunderstand how to use it?