Applicaton security logs stopped

[THE_BLUE]

There is no logs received from ASM, how to solve it ?

And how to check the utilization?

[Daniel_Wolf]

I think we need a bit more details.

Did it work before? Are you logging locally or remote? Both?

Are you logging violations only or all requests?

What kind of utilization do you want to check? Connections to the VS? CPU utilization on the BIG-IP device?

Some easy checks:

• Check if asmlogd is running (K14020: BIG-IP ASM daemons (11.x - 16.x)) Also check if other daemons are stuck.
• If you are logging to a remote syslog server: K86480148: Troubleshooting issues sending logs to a remote syslog server
• Check if log entries are stored in the MySQL db (K06821426: Viewing BIG-IP ASM request logs from MySQL database). Follow the steps for "Viewing the number of requests that exist within MySQL database" the number of COUNT(*) should increase.

Best of luck

[THE_BLUE]

Did it work before? yes, only today has been stopped

Are you logging locally or remote? Both? we have both , but the issue with local logs under Event log.

Are you logging violations only or all requests? we have different profile ( all request , illegal requests)

What kind of utilization do you want to check? Connections to the VS? CPU utilization on the BIG-IP device? i don't know of this issue related to utilization, i mean i wanna check the root cause of logs issue if it is because of memory/cpu or what.

[Daniel_Wolf]

Ok, so remote logging is OK, but local logging is not fully OK.

Check for stuck daemons and check if the event logs are in the MySQL db.

For CPU / Memory utilization follow this KB

K05372587: BIG-IP performance check-list

[THE_BLUE]

Is there any command to check the stuck deamons?

[Daniel_Wolf]

Try

tmsh show sys service

[THE_BLUE]

no asmlogd is listed when above command execute, but for asm is running.

[Daniel_Wolf]

Check like this

ps | grep asmlogd
 
tail -f var/log/ts/asmlogd.log

[THE_BLUE]

I got sth like

 9237 ?       S<  308:12 /usr/bin/perl /usr/share/ts/bin/asmlogd

16505 pts/0   S+    0:00 grep asmlogd

[Daniel_Wolf]

Yes, this looks OK. At least it's running.

Do you see anything odd in the log?

Did you check the steps and values mentioned in K05372587 and K06821426?

[Daniel_Wolf]

Also, since remote logging is working, there are two more simple checks you could perform.

• Did you create qkview and upload to ihealth? If yes, did this reveal something obvious?
• Do you have a cluster? Can you do a failover and check if local logging is working on the other node?

[THE_BLUE]

I have run below command

mysql -uasm -p`perl -MF5::Cfg -e 'print F5::Cfg::get_mysql_password()'` -e "select COUNT(*) from PRX.REQUEST_LOG"

In active node : count 0

and in standby : count 5xxx

so it seems the issue with the active node, i will try to offline the active node to double check.

[THE_BLUE]

when the active node forced to be standby , the logs are display on the other node and works as expected. so the issue with the node, but how to investigate the root cause and how to solve it?

[Daniel_Wolf]

I have mentioned a couple of steps above.

1. Qkview and upload to ihealth
2. Check var/log/ts/asmlogd.log
3. Check if, maybe, you are affected by a known issue like: K93091504: Cannot see any event logs for BIG-IP ASM. Security logging stopped working or similar issues.
4. Open a case with F5

[THE_BLUE]

restart asmlogd will cause any issue ?

pkill -f pabnagd

and pkill -f asmlogd

or better to use tmsh restart /sys service asm ?

[Daniel_Wolf]

If this node is still in standby, it is safe to apply this procedure.

However, I am from the school that would rather like to figure out the root cause before I restart daemons. Did you find any log entry that points to a known bug, that can be worked around by restarting pabnagd and asmlogd?

[THE_BLUE]

according to https://support.f5.com/csp/article/K93091504 it might be the issue ,

but i'm not able to view pabnagd.log to check if there is any warning.

However, in security > Event Logs > Application > Requests it display empty no message is displayed.

[Daniel_Wolf · Accepted solution]

You can't view /var/log/ts/pabnagd.log?

Is it a permission issue, no bash access? Or you did not know where the file is located?

Did you check that your BIG-IP version is matching the versions listed in "Applies to (see versions):" in K93091504?

Also yesterday you mentioned that "mysql -uasm -p`perl -MF5::Cfg -e 'print F5::Cfg::get_mysql_password()'` -e "select COUNT(*) from PRX.REQUEST_LOG"" returned a count of 0. Did you recently update the box? Even in my lab machines I have a count of >1000.

[Daniel_Wolf]

@BLUE did you resolve the issue?

[THE_BLUE]

yes the issue is fixed, many thanks for your support .

[Daniel_Wolf]

Will you please share with us how you resolved the issue?

If any of my answer provided the solution, would you mind to mark it as best answer, so that this thread is marked as answered?