Forum Discussion
David_M
Cirrostratus
Cirrostratus/var/lib/mysql is 100% full
Hi
I am seeing this right now, and I am aware of AVR and APM KB articles regarding this but here its impacting ASM mostly, I cant export policies nor view request logs.
Any ideas on what we can do here to reduce the space or compress the logs?
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg--db--sda-set.1.root
427M 274M 131M 68% /
none 16G 2.4M 16G 1% /dev/shm
/dev/mapper/vg--db--sda-set.1._config
3.2G 562M 2.5G 19% /config
/dev/mapper/vg--db--sda-set.1._usr
4.0G 3.3G 509M 87% /usr
/dev/mapper/vg--db--sda-set.1._var
3.0G 1.4G 1.5G 50% /var
/dev/mapper/vg--db--sda-dat.share.1
60G 3.4G 53G 6% /shared
/dev/mapper/vg--db--sda-dat.log.1
6.9G 216M 6.4G 4% /var/log
none 16G 51M 16G 1% /shared/rrd.1.2
none 16G 39M 16G 1% /var/tmstat
none 16G 1.6M 16G 1% /var/run
prompt 4.0M 28K 4.0M 1% /var/prompt
/var/sam/images/apmclients-7171.2018.808.2011-24.0.iso
295M 295M 0 100% /var/apm/mount/apmclients-7171.2018.808.2011-24.0.iso
none 16G 0 16G 0% /var/loipc
/dev/mapper/vg--db--sda-app.ASWADB.set.1.mysqldb
2.0G 1.9G 4.9M 100% /var/lib/mysql
/dev/mapper/vg--db--sda-app.asm.dat.asmdata1
6.7G 4.7G 1.7G 74% /var/asmdata1
/dev/mapper/vg--db--sda-app.avr.dat.avrdata
7.5G 146M 7.0G 2% /shared/avr
find /var/lib/mysql -xdev -type f -exec du {} \; | sort -rn | head -20
262404 /var/lib/mysql/ib_logfile1
262404 /var/lib/mysql/ib_logfile0
59456 /var/lib/mysql/ibdata1
54740 /var/lib/mysql/PRX/REQUEST_LOG#P#p100001.MYD
45384 /var/lib/mysql/PRX/REQUEST_LOG#P#p200001.MYD
45104 /var/lib/mysql/PLC/PL_POLICY_NEGSIG_SIGNATURES.ibd
41664 /var/lib/mysql/PRX/REQUEST_LOG#P#p1300001.MYD
41616 /var/lib/mysql/PRX/REQUEST_LOG#P#p900001.MYD
40984 /var/lib/mysql/PRX/REQUEST_LOG#P#p800001.MYD
40980 /var/lib/mysql/PRX/REQUEST_LOG#P#p1200001.MYD
40972 /var/lib/mysql/PRX/REQUEST_LOG#P#p1400001.MYD
40948 /var/lib/mysql/PRX/REQUEST_LOG#P#p1100001.MYD
39904 /var/lib/mysql/PRX/REQUEST_LOG#P#p1000001.MYD
38000 /var/lib/mysql/PRX/REQUEST_LOG#P#p700001.MYD
36860 /var/lib/mysql/PRX/REQUEST_LOG#P#p300001.MYD
36312 /var/lib/mysql/PRX/REQUEST_LOG#P#p600001.MYD
34112 /var/lib/mysql/PRX/REQUEST_LOG#P#p500001.MYD
34104 /var/lib/mysql/PRX/REQUEST_LOG#P#p400001.MYD
28884 /var/lib/mysql/AVR/AVR_STAT_VCMP_PG_CPU_USAGE_D#P#D_2019_0414_000001_1555185601.MYD
28704 /var/lib/mysql/PLC/PL_SUGGESTIONS.ibd
Hi Andrew,
This was a bug: we were previously using the system with VCMP and then it was deprovisioned from that to a normal LTM with ASM provisioned.
In this change the VMCP leaves behind a /var/mysqldb of size 2GB which is enough for VCMP to function but not for ASM.
TAC asked us to reinstall the TMOS into a new boot location and then we got a new /var/mysqldb of size 24GB.
Cheers!
Andrew-F5Employee
What software version are you running? There are a few bugs in APM and AVR that can fill up /var/lib/mysql which would impact ASM.
K23345645: AVR statistical data for BIG-IP ASM grows to consume available disk space - v13
You could reset the AVR DB and if that frees up the disk space it may help confirm AVR is the culprit.
K14956: Resetting BIG-IP AVR statistics
What's the output of the following? Specifically looking for entries referencing /var/lib/mysql
# lsof -ws | grep -i 'size\|deleted'
David_MThis is on v13.1.x.x
Yes this is a definite bug, we have multiple cases with these.
Do we have a fix yet in a future version?
We freed some space by deleting AVR files and got the usage to 89% but in 4 days its back to 94%
The TAC used these commands to fix the table.
mysql -t -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e 'check table PRX.REQUEST_LOG' mysql -t -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e 'repair table PRX.REQUEST_LOG'- Andrew-F5
DavidMas,
I would still be curious to know what the output of
# lsof -ws | grep -i 'size\|deleted'
looks like since daemons like AVR can hold onto files even after deletion. The only way to remove files listed with lsof is to restart AVRD or reboot.
Best,
Andrew