Using string map or regsub breaks non-english (UTF-8) characters

[adir_f5]

Hi Dear community ,

I woud appreciate your input. I want to replace a string inside a HTTP::payload with string map or regsub and then HTTP::payload replace. The replacement does take place but it also break the non-english [UTF-8] characters inside the HTML.

I tried implementing all the soultions I found inside DevCentral. Including these:

https://community.f5.com/t5/technical-forum/http-collect-results-in-garbled-http-payload-when-using-...

https://support.f5.com/csp/article/K22406348

https://clouddocs.f5.com/api/irules/HTTP__payload.html

• Note that the argument will be interpreted as a byte array. If it is actually a UTF-8 string with multibyte characters, the output will not be what you expect. In order to prepare a UTF-8 string for use as input to HTTP::payload replace, you should first run ‘binary scan c* throwawayvariable’.

Btw, Stream profile does work fine , but it's not very compatible with APM Virtual Servers.

Best Regards,

Adir

[G-Rob]

Adir,

Thanks for posting your question on DevCentral. Could you post your current iRule for easier diagnosis?

Thanks

[adir_f5]

@G-Rob 

when RULE_INIT {
# Limit payload collection to 5Mb
set static::max_collect_len 5368709
# Max characters to log locally (must be less than 1024 bytes)
set static::max_chars 900
}

when HTTP_REQUEST {
set collect 0
if { [HTTP::path] eq "*" } {
set collect 1
}
}

when HTTP_RESPONSE {
if { $collect eq 1 } {
if {[HTTP::header exists "Content-Length"]}{
set collect_length_resp [HTTP::header "Content-Length"]
log local0. "Content-Length exists [HTTP::header "Content-Length"]"
} else {
HTTP::payload rechunk
set collect_length_resp $static::max_collect_len
}
HTTP::collect $collect_length_resp
}
}

when HTTP_RESPONSE_DATA {
log local0. "Response Collected [HTTP::payload length] bytes"
set newData [string map { * **} [HTTP::payload]]
HTTP::payload replace 0 $collect_length_resp $newData
HTTP::payload rechunk
HTTP::release
}

Edited by @Leslie_Hubertus to tag G-Rob to make sure he sees the follow-up. 🙂

[G-Rob · Accepted solution]

Can you try adding 

binary scan $new_data @0

in the HTTP_RESPONSE_DATA event handler? I think that's the missing piece per https://support.f5.com/csp/article/K22406348.

[adir_f5]

@G-Rob Hi Rob,

It's very weird becasue I already tried this command before I open this thread - You can see I event reference this K article in my original post. But Thanks to you I did try it again, and it looks like now it's working great, Thanks a ton. 

Best Regards,

Adir