Technical Forum
Ask questions. Discover Answers.
cancel
Showing results for 
Search instead for 
Did you mean: 

ASM out of memory XML/JSON processing

MVA_60288
Altocumulus
Altocumulus

Hi, anyone else run into this error while continuing to up the XML memory allocation? We've gone to 500mb to 700mb (SOL10803) and still having this error re-occur. I have a case open with F5, but curious if anyone has run into this and what your (hopefully) final solution was.

 

Full error "ASM out of memory error: event code X239 Exceeded maximum memory assigned for XML/JSON processing"

 

16 REPLIES 16

Torti
Altostratus
Altostratus

it all depends on the total amount of RAM. use the search, there are some old discussions about that.

 

MVA
Nimbostratus
Nimbostratus

We are at 900MB XML memory now and so far all is good. I understand from F5 support we can increase to a maximum of 2000MB. There is an enhancement request (ID380171) to provide tools to monitor memory used by XML parser - for now there is no visibility into XML memory use.

 

Torti
Altostratus
Altostratus

At the moment, we ar at 3GB 😉

 

Till 11.2.x, there was the possibility to see the usage in the bd log file. But they removed this with a later release.

 

MVA
Nimbostratus
Nimbostratus

Wow, 3GB. I suspect the ceiling we're given is due to our hardware. We are running on 3900 with 8GB RAM and have ASM/LTM, both provisioned as nominal. I understand we can gain more memory by re-provisioning LTM as minimal, as our current LTM usage is very light. If we run into this issue again we plan to jump to F5 recommended 2000MB.

 

Torti
Altostratus
Altostratus

oh yeah the 3900. We replaced it by a 5000s with 32gb memory, because of the memory problem. We are running a lot of webservices.

 

Im not shure, if it is a good idea to run LTM as minimal. F5 allways say, we have to run at nominal, although our LTM usage is very light.

 

With the 3900 the maximum xml memory was at 1.7GB. That was the limit, but it wasn't enough and we had to restart the system after some time.

 

But the 1.7gb was only possible with a custom memory setting!

 

MVA
Nimbostratus
Nimbostratus

I heard back from F5 support, they are suspecting a memory leak and have issued me an engineering HF. They identify the issue ID489705, memory leak in bd's XML engine. We're running good at 900MB XML memory, so staying put for now. I'm hoping this fix is included in the Ghost HF.

 

Torti
Altostratus
Altostratus

ah ok, a feature. Waht release version is running on your systems?

 

MVA
Nimbostratus
Nimbostratus

Version 11.5.1

 

Torti
Altostratus
Altostratus

nice, we are running the same release on our new systems. Now its clear, why the bd process take 6.3gb ram 🙂

 

It sounds like they didn't told you the fixed release for this bug?

 

swo0sh_gt_13163
Altostratus
Altostratus

Guys,

 

How did you see the memory utilization of BD? I need to verify the current Memory utilization of ASM. Hopefully this thread would be helpful.

 

I have used "top" command to see the BD process Memory utilization, however is there a way to filter any particular PID from the "top" result?

 

Cheers! Darshan

 

Torti
Altostratus
Altostratus

The PID will change after a restart of the process - i.e. with restart of the the device.

 

If you want to use a managemant tool with snmp, you have to poll 2 values

 

1. the names with a walk through 1.3.6.1.2.1.25.4.2.1.2 and look for 'bd'

 

2. the memory 1.3.6.1.2.1.25.5.1.1.2.x - x is the result of 1

 

swo0sh_gt_13163
Altostratus
Altostratus

Thanks Torti!

 

prt1969_120570
Nimbostratus
Nimbostratus

Stupid question, but I am assuming that if you run into this problem, there may be tcp streams that do not get processed by ASM and are just sent thru without being inspected?

 

When this issue occurs traffic is blocked for that virtual server, I believe error is something like not enough memory to process XML request. It fails close.

 

gu3rr3ir0
Nimbostratus
Nimbostratus

Does anyone have a "final" solution for this. I have the exact same case, we up from 500 to 700 but still having issues with the out of memory logs still appearing.

 

The trial and error of updating the Total XML Memory according to the "excedeed memory allocated" is not a good option, because it seems the more memory you configure the more memory XML resources will consume.

 

At least the logs of exceeded memory allocated in bd.log should give us the correct value and not only 20, 30 or 40 Mb more than we have configured, and even if you configure more 200MB it still is giving the same messages.

Lidev
MVP
MVP

Hi gu3rr3ir0,

I have followed theses 2 KB below and by modify the values of total_xml_memory (max 1GB) and additional_xml_memory_in_mb . I have no longer an ASM out of memory XML error.

 

https://support.f5.com/csp/article/K64422353

https://support.f5.com/csp/article/K10803

 

Regards