Bare byte decoding and French characters

Not sure how granular you can be with Evasion Techniques, not very I believe.

Do you get learning suggestions? Is it triggered on a parameter value? I wonder if you can create an explicit parameter and uncheck "check characters on this parameter value"?

Other option (more complicated I know) is do traffic classification (via http class or local traffic policy dependant on your TMOS version) and divert the french pages, say /france, to a different security policy configured as Western European.

Not sure if you want to allow the violation on the specific entity/page or just stop it from being logged? Not sure if you can stop it being logged.

Hope this is some help,

N

When I create my WAF policies, I always create them with unicode (UTF-8). Violations are usually in the uri. (à...) Whitelisting all current French pages with these characters would take quite some time + I need to know when new ones are introduced.

I understand that there is a check necessary for some non-ascii characters, however, it would have been nice if I could whitelist certain characters. (like à, é, è, ...) My goal is to remove these entries from being logged as illegal in the security event logs, as these are legitimate requests.

In the meantime, I have found why we have this issue:

Suppose we have a page with a non-ASCII character in the url (Like http://www.test.com/identité.html). Clicking a link in this page would place the url in the referer field. Internet Explorer apparently puts the URL as it is. (Other browsers do a translation into ASCII characters) This triggers the log messages we see in our event log.

Do you need to take this fact into account when you want to check bare byte decoding? Do you need to whitelist URLs to achieve this?

Hello Tomy,

I am facing the same trouble with Arabic characters. I did upgrade from 11.2.0 to 11.5.1 and started getting a lot of violation for Evasion Technique Detected, precisely "Bare Bye decoding". While going through the violation I Found that all the violations are triggered for non-ascii characters.

In my case, all the users are using IE only, and previous upgrade, BareByte Decoding was checked and it was working without making any noise.

It seems F5 has changed the ASCII character rendering or something with Bare Byte Decoding. Any comments?

Cheers! Darshan

Hi All,

i am facing the same issue only with IE? it looks like no other option except to open a ticket? any idea?

Thanks in advance.

Regards

Hussein

Hi all,

we have a similar issue here as well, so I decided to use this thread instead of opening a new one.

We are using a UTF-8 policy on 11.4.1 HF8 and the violation "Failed to convert character" is triggered due to French characters (like é) on a parameter value. We tried already "Ignore Value" for this parameter as well as disable "Check characters on this parameter value", but the violation will still be triggered. In the lab we found out that the list of Character Sets includes these French characters if you are using Western European instead of UTF-8. Is this the reason why the F5 can't read such characters correctly, because I always thought that UTF-8 is a universal language to avoid such issues? Also in the learning suggestion section it's not possible to learn this violation/make an exception for it. So right now we disabled blocking for this violation type at all.

But isn't there a much better and granular option to solve this issue?

Thank you!

Ciao Stefan :)

Hi Nathan,

thanks for that solution, it really indicates that this is normal behavior:

If the BIG-IP ASM receives a single-byte sequence when the application language is set
to a multi-byte encoding, such as UTF-8, the character will fail the character decoding process.
The Failed to convert character violation will be triggered [...]

But interesting is the other solution (11491) where it refers to at the end. Is this also available (or could be updated) for version 11.x? I'm wondering because for version 10.x it mentions that these French characters are allowed by default for Rapid Deployments (what we also used when creating the policy). Or has this nothing to do with the above issue?

So would that mean that I have to double check if such "higher" ASCII-character will be used somewhere in the URI or in parameters, and if so that I may not use UTF-8? Or is this also depending how the browser behaves with such characters? I mean e.g. a forum in France, where such characters are included in nearly each POST-request when entering a new comment would be triggered. Isn't there any other method to allow or whitelist such characters?

Thank you!

Ciao Stefan 🙂

Hi Nathan,

I don't think I get your point. What do you mean with "I think by uri is your best bet"? Do you mean I have to disable ASM for the affected URI at all (I don't know at the moment if it affects only one or a few URIs).

I also made some further testing and it seems to be also depending on the used browser. The violation is caused by this parameter:

DBParamValue=Libell%E9%20fonds%20ou%20code%20ISIN

When using IE it doesn't matter if I use Libellé, Libell%E9 or Libell%C3%A9 (this is the correct UTF-8 encoding) the violation triggers all the time. But interesting is that the encoding settings in IE is correctly set to UTF-8.

Whereas when using FF it triggers only when using %E9, the other two options will be encoded "correctly".

Ciao Stefan 🙂

It seems we found the real root cause for this issue. It's related to IE11, which introduced a more granular option for UTF8 encoding (for more details check this website). There you can see that IE11 differentiate URL encoding for the path and for the query-string (the latter one is disabled by default). When enable this option and restart the browser, French characters in the query-part of the URL will be correctly UTF8 encoded and the violation will not be triggered anymore.

According to Western European as an alternative language code for the ASM policy, I don't know if this is really a good option, because then it's the other way round. The ASM expects Western European encoding, but if a browser sends it as UTF8 it will fail again to convert the character.

Here is a list of differences between Western European and UTF8, as you can see everything after %7F will different.

And as UTF8 is some kind of standard in the meanwhile, I would always recommend to use it as ASM policy language and correct the issue on the browser (which might be complicated or nearly impossible for public websites, where you don't know which clients will access it). Alternatively you could write an iRule, which checks the query-part of the URI for such "high" ASCII-characters and respond with an error message describing how to change this setting in IE11 (or other browsers which might be affected from this behavior as well).

Ciao Stefan :)

I know this is an old thread but does anyone know how to solve this issue? I am facing similar issue for following characters Å å Ä ä Ö ö. In my case, when these characters are mentioned in the user input fields ASM is raising a violation. As a temporary solution, disabled the Block Character Convertion option.