Forum Discussion

AlgebraicMirror's avatar
AlgebraicMirror
Icon for Altostratus rankAltostratus
Oct 18, 2011

Strange F5 ASM Sawtooth CPU Graph

A couple of my ASM 3600s have odd sawtooth patterns on their CPU performance graphs. Approximately every three minutes, CPU spikes, and then shortly afterword goes back down.

 

 

Using top, I was able to determine that mysqld is causing this. From within mysql, I used the "show processlist;" command to see what was being run. It's not one statement, but several, and the same ones are run in the same order each time. This tells me some kind of a script is using mysql every three minutes, and since there are no stored procedures in the ASM database, it must be a script external to mySQL.

 

 

Has anyone run into anything like this before? I have three ASMs in the same production pool, and all are on 10.2.2 HF3. One was just rebuilt from scratch and does not have this problem, while the two that were simply upgraded do (they had this before the upgrade as well). Here are some of the statements I saw mysql running... any idea what is causing this or how to turn it off? Or even how I might investigate further?

 

 

UPDATE PLC.LRN_NEGSIG_SIGNATURES A

 

LEFT JOIN PLC.PL_OBJECTS AS F

 

ON (A.obj_crc = F.obj_crc

 

 

UPDATE PLC.LRN_NEGSIG_SIGNATURES AS A

 

JOIN PLC.PL_PARAM_NEGSIG_SIGNATURES AS B

 

ON (A.param_at

 

 

UPDATE PLC.LRN_NEGSIG_SIGNATURES AS A

 

LEFT JOIN PLC.PL_GLOBAL_PARAMS AS B

 

ON (((A.param_name

 

 

etc.
config
design
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Oct 19, 2011
    how many traffic learning do you have? is it quite a number of suggestion?

     

     

    there is clean_db.pl script which continuously cleaning database but it is run every 60s (at least on my unit).

     

     

    sol9753: The clean_db.pl script consumes excessive memory

     

    https://support.f5.com/kb/en-us/solutions/public/9000/700/sol9753.html?sr=17147926
  • Algebraic_Mirror's avatar
    Algebraic_Mirror
    Icon for Cirrostratus rankCirrostratus
    Oct 19, 2011
    At this point it doesn't look like I have a lot of traffic learning suggestions. I see maybe a dozen related to attack signatures, but that's about it. My database is probably pretty big though as these are large sites going through our ASM... so maybe it is some part of the learning system that is having a hard with the large amount of data? Any idea if that might be true, and if so what might be struggling?

     

     

    As far as the clean_db.pl script goes, a "find / clean_db.pl" didn't turn that script up on my system, and according to the article, the issue was supposed to be fixed in 10.1.0, so maybe it's something no longer used in my version? I am on 10.2.2 HF3.

     

     

    I really appreciate your time, and any further suggestions would be appreciated!
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Oct 20, 2011
    this is mine.

     

     

    [root@B8900-R12-S40:Active] config b version|grep -iA 1 version

     

    BIG-IP Version 10.2.2 930.0

     

    Hotfix HF3 Edition

     

     

    [root@B8900-R12-S40:Active] config find / -name clean_db.cfg

     

    /shared/ene/qkview/etc/ts/tools/clean_db.cfg

     

    /etc/ts/tools/clean_db.cfg

     

     

    [root@B8900-R12-S40:Active] config find / -name clean_db.pl

     

    /usr/share/ts/bin/clean_db.pl
  • Algebraic_Mirror's avatar
    Algebraic_Mirror
    Icon for Cirrostratus rankCirrostratus
    Oct 25, 2011
    It doesn't look to be the clean_db.pl script, but I think this is very much on the right track! That directory "/usr/share/ts/bin" that contains the clean_db.pl script ALSO contains a huge number of other scripts that operate on the database. I'm pretty sure it's one of them. I guess I'm not going to dig too much deeper, since this is starting to look like normal operation. If it starts hurting performance I'll file a ticket, but since it spikes the CPU for one minute and then backs off for two minutes, things stay caught up and nothing gets too hurt.

     

     

    Also, I need to modify my original post: my newly rebuilt ASM is starting to show this problem again, but to a lesser degree. My guess is that because it has been up for a much shorter period, it has a smaller database and queries execute faster. As the DB grows, the queries must slow down and cause CPU spikes, giving me this sawtooth pattern.