ASM out of memory XML/JSON processing

[MVA_60288]

Hi, anyone else run into this error while continuing to up the XML memory allocation? We've gone to 500mb to 700mb (SOL10803) and still having this error re-occur. I have a case open with F5, but curious if anyone has run into this and what your (hopefully) final solution was.

Full error "ASM out of memory error: event code X239 Exceeded maximum memory assigned for XML/JSON processing"

[Torti]

it all depends on the total amount of RAM. use the search, there are some old discussions about that.

[MVA]

We are at 900MB XML memory now and so far all is good. I understand from F5 support we can increase to a maximum of 2000MB. There is an enhancement request (ID380171) to provide tools to monitor memory used by XML parser - for now there is no visibility into XML memory use.

[Torti]

At the moment, we ar at 3GB 😉

Till 11.2.x, there was the possibility to see the usage in the bd log file. But they removed this with a later release.

[MVA]

Wow, 3GB. I suspect the ceiling we're given is due to our hardware. We are running on 3900 with 8GB RAM and have ASM/LTM, both provisioned as nominal. I understand we can gain more memory by re-provisioning LTM as minimal, as our current LTM usage is very light. If we run into this issue again we plan to jump to F5 recommended 2000MB.

[Torti]

oh yeah the 3900. We replaced it by a 5000s with 32gb memory, because of the memory problem. We are running a lot of webservices.

Im not shure, if it is a good idea to run LTM as minimal. F5 allways say, we have to run at nominal, although our LTM usage is very light. With the 3900 the maximum xml memory was at 1.7GB. That was the limit, but it wasn't enough and we had to restart the system after some time. But the 1.7gb was only possible with a custom memory setting!

[MVA]

I heard back from F5 support, they are suspecting a memory leak and have issued me an engineering HF. They identify the issue ID489705, memory leak in bd's XML engine. We're running good at 900MB XML memory, so staying put for now. I'm hoping this fix is included in the Ghost HF.

[Torti]

ah ok, a feature. Waht release version is running on your systems?

[MVA]

Version 11.5.1

[Torti]

nice, we are running the same release on our new systems. Now its clear, why the bd process take 6.3gb ram 🙂

It sounds like they didn't told you the fixed release for this bug?

[swo0sh_gt_13163]

Guys,

How did you see the memory utilization of BD? I need to verify the current Memory utilization of ASM. Hopefully this thread would be helpful.

I have used "top" command to see the BD process Memory utilization, however is there a way to filter any particular PID from the "top" result?

Cheers! Darshan

[Torti]

The PID will change after a restart of the process - i.e. with restart of the the device.

If you want to use a managemant tool with snmp, you have to poll 2 values 1. the names with a walk through 1.3.6.1.2.1.25.4.2.1.2 and look for 'bd' 2. the memory 1.3.6.1.2.1.25.5.1.1.2.x - x is the result of 1

[swo0sh_gt_13163]

Thanks Torti!

[prt1969_120570]

Stupid question, but I am assuming that if you run into this problem, there may be tcp streams that do not get processed by ASM and are just sent thru without being inspected?

[MVA]

When this issue occurs traffic is blocked for that virtual server, I believe error is something like not enough memory to process XML request. It fails close.

[gu3rr3ir0]

Does anyone have a "final" solution for this. I have the exact same case, we up from 500 to 700 but still having issues with the out of memory logs still appearing.

The trial and error of updating the Total XML Memory according to the "excedeed memory allocated" is not a good option, because it seems the more memory you configure the more memory XML resources will consume.

At least the logs of exceeded memory allocated in bd.log should give us the correct value and not only 20, 30 or 40 Mb more than we have configured, and even if you configure more 200MB it still is giving the same messages.

[Lidev]

Hi gu3rr3ir0,

I have followed theses 2 KB below and by modify the values of total_xml_memory (max 1GB) and additional_xml_memory_in_mb . I have no longer an ASM out of memory XML error.

https://support.f5.com/csp/article/K64422353

https://support.f5.com/csp/article/K10803

Regards