Forum Discussion
David_M
Cirrostratus
Cirrostratus/var/lib/mysql is 100% full
Hi I am seeing this right now, and I am aware of AVR and APM KB articles regarding this but here its impacting ASM mostly, I cant export policies nor view request logs. Any ideas on what we c...
Hi Andrew,
This was a bug: we were previously using the system with VCMP and then it was deprovisioned from that to a normal LTM with ASM provisioned.
In this change the VMCP leaves behind a /var/mysqldb of size 2GB which is enough for VCMP to function but not for ASM.
TAC asked us to reinstall the TMOS into a new boot location and then we got a new /var/mysqldb of size 24GB.
Cheers!
Andrew-F5
Employee
EmployeeWhat software version are you running? There are a few bugs in APM and AVR that can fill up /var/lib/mysql which would impact ASM.
K23345645: AVR statistical data for BIG-IP ASM grows to consume available disk space - v13
You could reset the AVR DB and if that frees up the disk space it may help confirm AVR is the culprit.
K14956: Resetting BIG-IP AVR statistics
What's the output of the following? Specifically looking for entries referencing /var/lib/mysql
# lsof -ws | grep -i 'size\|deleted'
David_MCirrostratus
CirrostratusThis is on v13.1.x.x
Yes this is a definite bug, we have multiple cases with these.
Do we have a fix yet in a future version?
We freed some space by deleting AVR files and got the usage to 89% but in 4 days its back to 94%
The TAC used these commands to fix the table.
mysql -t -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e 'check table PRX.REQUEST_LOG'
mysql -t -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e 'repair table PRX.REQUEST_LOG'- Andrew-F5
DavidMas,
I would still be curious to know what the output of
# lsof -ws | grep -i 'size\|deleted'
looks like since daemons like AVR can hold onto files even after deletion. The only way to remove files listed with lsof is to restart AVRD or reboot.
Best,
Andrew
- David_M
Hi Andrew,
This was a bug: we were previously using the system with VCMP and then it was deprovisioned from that to a normal LTM with ASM provisioned.
In this change the VMCP leaves behind a /var/mysqldb of size 2GB which is enough for VCMP to function but not for ASM.
TAC asked us to reinstall the TMOS into a new boot location and then we got a new /var/mysqldb of size 24GB.
Cheers!