Hi Guys!, recently my CTO asked me if there's possible to extract the payload of the HTTP request on the local log (big-ip) to send this information to our remote servers. Today, we can't extract this information to send it to our customers, and i we can't see something equal "payload" or "body" of the HTTP request, there is a way to make it possible using an iRule ? When we receive this request, they extract only the payload (or the complete request with the payload) and send it to our remote servers ?
From a performance perspective it is OK, since it uses HSL for remote logging. Beware to set the limit for the collected payload to 32 MB, if you try to log anything larger than 32 MB your TMM will crash.
But maybe if you explain the use case, there is a better solution than doing this with iRules and also maybe there is a better way than doing this on the BIG-IP.
30-Apr-202102:54 - last edited on 24-Mar-202201:12 by li-migration
,
While you do this, do make sure you are not exposing sensitive confidential information's in the logs or on the remote servers. So depending on the criticality of the application, handle this.
Dear Jaikumar, for now, we're only testing on our own homolog website, for understand what type of information we'll receive and mask all of future sensitive confidential informations. Thank you so much for this information.
Firs of all, thank you so much for your information!
Let me explain a little bit more.
Today, we have 2 remote servers with booth of them are receiving the logs of our F5 Appliances. We are working to future desable the local logs of our appliances for only make troubleshooting with our 2 remote servers that are receiving this logs internal only. The problem is, we're sending to every alert/block the "query_string" field, but in some cases, the field "query_string" don't show anything, the field are there, but without any information, we detected that some type of signatures like XSS and SQL do not send this values of the exacly query string that match with the attack signature, but if we see it in our local log in F5 we can see the string that matches with the Attack Signature.
Maybe I misunderstood your question, but you can select the "request" field in your log profile to log the entire HTTP request (headers + body).
Of course for big requests this will still be limited by the max entry length you also configure in the log profile, but maybe the 64k or whatever the max length for standard log profiles is, is already enough for your use case.
If you do use this, make sure it's the last field in your log format. That way if a request does exceed the limit, you don't lose any other log fields.
Oh also, if you're looking for troubleshooting information, the "violation_details" field is very helpful. That's an XML blob that contains (partially base64 encoded) information about what exactly matched where and why. This one can also get very big, depending on the number of violations.
Hi Gersbah, that's the thing!, we're sending all of fileds thath contains in F5 logging profile, if we can only receive always the field "query_string" with the match parameter or string that matches an a attack signature, that way is sufficient for us. Thinking about exacly what you say, if the request it is to long, we'll have a problema with the lenght, so, there is a way to "fix" when the query_string come without any information ? even though on the local log of our appliances ?
If you try to log literally everything remotely, you will probably always run into length issues.
Remote logging in my opinion is meant to be selective.
Something that could work (but I've never done it myself) is to set multiple remote logging profiles to overcome the length limitations. For example one for your "standard" log with whatever log fields you typically need as your standard "access log". Then a second profile with only support ID (so you can identify the log entry) and request (to have as much of the full HTTP request as possible). And a third profile with only support ID and violation_details (again ID to match and then as much as possible of the violation_details field). And you can send those to the same or different log destinations.
All in all, I'm not convinced it's a good idea to try to completely move away from the local log of the bigip. I don't think you will get the same quality and user-friendliness with remote logging.
