[iRule] to trim hex value and send it to pool

(I assume SSL payload has to be converted to bin)

You beat me to it. What does it look like in the logs if you do this?

log local0. [SSL::payload]

If the app was truly expecting hex, then you shouldn't have to the binary scan. In any case, if the above doesn't produce a hex output, then you should definitely need to convert back to binary.

So I may be confused on what you need to do. At first I thought you only needed to send the first 7 bytes, and now it sounds like you want to send everything but the first 7 bytes. If the former, the you'd want to strip out those 7 bytes from some amount of consumed payload, and then replace the entire payload with those 7 bytes. If the latter, then I don't think you can't simply have 7 bytes of empty space at the beginning of the packet. You'd necessarily need to grab the entire payload, remove want you want, and then replace the whole thing.

when CLIENTSSL_HANDSHAKE {
    SSL::collect
}
when CLIENTSSL_DATA {
    binary scan [SSL::payload] H* hex
    log local0. $hex

    set hex3 [string range $hex 14 end]
    log local0. $hex3

    set bin [binary format H* $hex3]

    SSL::payload replace 0 [SSL::payload length] ""
    SSL::payload replace 0 0 $bin

    SSL::release
}