Forum Discussion
lnxgeek
MVP
MVPASM Too many concurrent long requests
Hello everyone!
I have come across this error at an ASM customer recently:
Too many concurrent long requests allowed 289 ,current 289 needed request length: 5120168
This led me to this article: https://my.f5.com/manage/s/article/K8292
Where a lot of parameters where mentioned:
request_buffer_size
max_concurrent_long_request
long_request_buffer_size
Can anyone point to somewhere documenting these values properly or can describe the to me? What is the definition of a "long request"?
What is puzzling me is that to my understanding these are all values limiting the amount of data which can be pushed towards ASM and onto the backend. But the application in question doesn't work in this way. So, could it be that it is all about clients having extremely large headers in the GET requests that these value are reached?
This all happend during what looks like an attack on the application.
I think K8292 defines it all as well as you're going to find anywhere, to be honest.
In ASM, a long request is simply any request (headers and content) which is larger than the request_buffer_size (10Kb by default). As soon as the buffered content reaches that size, ASM allocates a "long request buffer" (roughly 10Mb) from the pool and moves the entire request there (freeing the original request buffer)
On any modern (supported!) version the maximum number of long request buffers (10Mb each, remember) is a function of how much memory is available to ASM overall (the calculation is in K01235989 but you can ignore that as your system has already told you 298 is the number it allows).
For each (long) request, the long request buffer is only going to be freed once ASM has processed the transaction and handed it off to the back end system, so what this means is that you had 298 "long" requests (over 10Kb) in flight at that moment and the 299th request that came in was 5Mb in length. The fact that ASM knew that up front implies this wasn't a chunked request or headers (which ASM can only caculate the length of at the end of buffering, after it has allocated the long_request_buffer) but rather body content with a defined Content-Length.
That only applies for that specific request, though - for all we know the other 298 were, as you suggest, GET requests with enormous headers. You'd need logging or captures to figure that part out, unfortunately.
- AaronJB
SIRT
I think K8292 defines it all as well as you're going to find anywhere, to be honest.
In ASM, a long request is simply any request (headers and content) which is larger than the request_buffer_size (10Kb by default). As soon as the buffered content reaches that size, ASM allocates a "long request buffer" (roughly 10Mb) from the pool and moves the entire request there (freeing the original request buffer)
On any modern (supported!) version the maximum number of long request buffers (10Mb each, remember) is a function of how much memory is available to ASM overall (the calculation is in K01235989 but you can ignore that as your system has already told you 298 is the number it allows).
For each (long) request, the long request buffer is only going to be freed once ASM has processed the transaction and handed it off to the back end system, so what this means is that you had 298 "long" requests (over 10Kb) in flight at that moment and the 299th request that came in was 5Mb in length. The fact that ASM knew that up front implies this wasn't a chunked request or headers (which ASM can only caculate the length of at the end of buffering, after it has allocated the long_request_buffer) but rather body content with a defined Content-Length.
That only applies for that specific request, though - for all we know the other 298 were, as you suggest, GET requests with enormous headers. You'd need logging or captures to figure that part out, unfortunately.
- lnxgeek
AaronJB thanks for the very good description, it really was what I was looking for 👍
It is the first time I have come across these thresholds and with what you descripe I'm suprised that I haven't been hit by it before in this context. I have only struggled with large file uploads (long_request_buffer_size) in applications where that was their ligitimate use case.
Btw do you know what "bd" stands for? It is used all over 🙂
- AaronJB
No worries!
As you say, if the app in question isn't expecting file uploads or large POST requests then it does seem like it was probably attack related; it would be interesting to know more about the large requests, if you still have any logs!
As for 'bd' - it's the name of ASMs main daemon, the one responsible for actually enforcing policies against traffic, and what it stands for is a bit of a mystery even to those of us who've been here a very long time (nearly 18 years for me, much of that working with ASM). Legend has it that it stands for 'back defender'..
- LiefZimmerman
Admin
Is this something we can move to the Technical Forum? Or does it need to stay inside the MVP group?
- lnxgeek
It should be general available as it turns out.
I wasn't sure if there were any nda related information hidden in my question, so I played it safe 🙂
Can you move the thread or should I just repost it with somewhere public?
- LiefZimmerman
Always better to be safe than sorry. Thanks lnxgeek.
I'll move it over.
Then - if you get the chance - mark Aaron's answer as solved? That would be great.Thanks.
