Forum Discussion

David_M's avatar
David_M
Icon for Cirrostratus rankCirrostratus
Jul 02, 2019
Solved

/var/lib/mysql is 100% full

Hi I am seeing this right now, and I am aware of AVR and APM KB articles regarding this but here its impacting ASM mostly, I cant export policies nor view request logs. Any ideas on what we c...
application delivery
ASM Advanced WAF
  • David_M's avatar
    David_M
    Aug 25, 2019

    Hi Andrew,

     

    This was a bug: we were previously using the system with VCMP and then it was deprovisioned from that to a normal LTM with ASM provisioned.

     

    In this change the VMCP leaves behind a /var/mysqldb of size 2GB which is enough for VCMP to function but not for ASM.

     

    TAC asked us to reinstall the TMOS into a new boot location and then we got a new /var/mysqldb of size 24GB.

     

    Cheers!

     

Andrew-F5's avatar
Andrew-F5
Icon for Employee rankEmployee
Jul 03, 2019

What software version are you running? There are a few bugs in APM and AVR that can fill up /var/lib/mysql which would impact ASM.

K23345645: AVR statistical data for BIG-IP ASM grows to consume available disk space - v13

 

You could reset the AVR DB and if that frees up the disk space it may help confirm AVR is the culprit.

K14956: Resetting BIG-IP AVR statistics

 

What's the output of the following? Specifically looking for entries referencing /var/lib/mysql

# lsof -ws | grep -i 'size\|deleted'

  • David_M's avatar
    David_M
    Icon for Cirrostratus rankCirrostratus
    Jul 07, 2019

    This is on v13.1.x.x

    Yes this is a definite bug, we have multiple cases with these.

    Do we have a fix yet in a future version?

    We freed some space by deleting AVR files and got the usage to 89% but in 4 days its back to 94%

    The TAC used these commands to fix the table.

    mysql -t -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e 'check table PRX.REQUEST_LOG'
 
mysql -t -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e 'repair table PRX.REQUEST_LOG'
    • Andrew-F5's avatar
      Andrew-F5
      Icon for Employee rankEmployee
      Jul 08, 2019

      DavidMas,

       

      I would still be curious to know what the output of

      # lsof -ws | grep -i 'size\|deleted'

      looks like since daemons like AVR can hold onto files even after deletion. The only way to remove files listed with lsof is to restart AVRD or reboot.

       

      Best,

      Andrew

    • David_M's avatar
      David_M
      Icon for Cirrostratus rankCirrostratus
      Aug 25, 2019

      Hi Andrew,

       

      This was a bug: we were previously using the system with VCMP and then it was deprovisioned from that to a normal LTM with ASM provisioned.

       

      In this change the VMCP leaves behind a /var/mysqldb of size 2GB which is enough for VCMP to function but not for ASM.

       

      TAC asked us to reinstall the TMOS into a new boot location and then we got a new /var/mysqldb of size 24GB.

       

      Cheers!