Forum Discussion

Stefan_Klotz's avatar
Stefan_Klotz
Icon for Cumulonimbus rankCumulonimbus
Jan 09, 2023
Solved

BIG-IQ statistics in dashboard missing/interrupted

Dear all,

any idea, why statistics data is missing/interrupted for a few days like in the following screenshot?

 

Or how I can verify any issues? In which logs can I found any hints?

The missing data is visible for device statistics as well as for Local Traffic. Am I right, that these statistics are the ones, which are stored on the DCDs? And why is there no data before 27.11.2022? Is there any limit configured that just the last 30 days (or so) will be stored? Does this indicate to an connectivity issue between the DCDs and the BIG-IPs?

As of now I could only verify that the ucs backups were successfully executed during these days, so connectivity between the BIG-IQ and the BIG-IPs was fine?

Thanks for any useful ideas or hints.

Regards Stefan 🙂

application delivery
BIQ-IQ
statistics
  • Stefan_Klotz's avatar
    Stefan_Klotz
    May 11, 2023

    We could solve the issue in the meanwhile together with official F5-support.

    It's related to ID1044337:

    This can occur intermittently during statistics collection from BIG-IP to BIG-IQ and can happen when the retention processor causes a loss of some retention records (book-keeper-start entries are lost while book-keeper-end entries are retained).”

    From the BIG-IQ command line:

    1) Make a backup of the book-keeper-end index:

    curl -sk http://localhost:9200/_reindex?pretty -H "content-type: application/json" -d '{"source":{"index":"book-keeper-end"},"dest":{"index":"bu-book-keeper-end"}}'

    2) Delete the book-keeper-end index:

    curl -skX DELETE http://localhost:9200/book-keeper-end

    Note:  The book-keeper-end index will be rebuilt automatically and will cause no loss of statistic data.

     

    It took some time (I don't know how long exactly), but on the next day I could see statistics data back to one year again.

    Regards Stefan 🙂

7 Replies

Sort By
  • Stefan_Klotz's avatar
    Stefan_Klotz
    Icon for Cumulonimbus rankCumulonimbus
    May 11, 2023

    We could solve the issue in the meanwhile together with official F5-support.

    It's related to ID1044337:

    This can occur intermittently during statistics collection from BIG-IP to BIG-IQ and can happen when the retention processor causes a loss of some retention records (book-keeper-start entries are lost while book-keeper-end entries are retained).”

    From the BIG-IQ command line:

    1) Make a backup of the book-keeper-end index:

    curl -sk http://localhost:9200/_reindex?pretty -H "content-type: application/json" -d '{"source":{"index":"book-keeper-end"},"dest":{"index":"bu-book-keeper-end"}}'

    2) Delete the book-keeper-end index:

    curl -skX DELETE http://localhost:9200/book-keeper-end

    Note:  The book-keeper-end index will be rebuilt automatically and will cause no loss of statistic data.

     

    It took some time (I don't know how long exactly), but on the next day I could see statistics data back to one year again.

    Regards Stefan 🙂

  • Nikoolayy1's avatar
    Nikoolayy1
    Icon for MVP rankMVP
    Jan 09, 2023

    Maybe check the /var/log/avr and /var/log/restjavad on the BIG-IP as the Analytics AVR is the way to stream logs to DCD and the restjavad is the HTTP API  that BIg-IQ uses to control BIG-IP and also check the BIG-IQ device and system logs on the BIG-IQ for the BIG-IP devices as BIG-IQ also manages the DCD.

     

    Just in case see in GUI if the BIG-IP has log statistics for this period or upload qkview to ihealth and check again the Statistics of the BIG-IP as to exclude BIG-IP issue and check in health for any bugs under diagnostics.

    • Leslie_Hubertus's avatar
      Leslie_Hubertus
      Ret. Employee
      Jan 18, 2023

      Stefan_Klotz  - did Nikoolayy1 's suggestion work for you? Inquiring minds want to know! 🙂

      (also, if yes, don't forget to hit the Accept as Solution button if it did work - and if it didn't, let us know so we can see if there's another solution for you. 

    • Stefan_Klotz's avatar
      Stefan_Klotz
      Icon for Cumulonimbus rankCumulonimbus
      Feb 24, 2023

      Sorry for late reply.

      I don't found any indications in any of the mentioned logs. Since end of December logging for all of our BIG-IPs seems to run stable again as you can see in the following screenshot:

      To be more precise and the reason why I'm asking is, that I wonder if and how recording these statistics data depends on the free disc space of the BIG-IQ itself (not the DCDs). Background: until end of last year the "/shared" partition of the BIG-IQ was full, as there were too much ucs-files from all of our BIG-IPs. After christmas, one BIG-IP cluster was removed, which released some GB (because all corresponding ucs-files where deleted as well). Additionally I increased this partition to 100GB beginning of January. And as you can see in the above screenshot, statistics data is available again since around that date!

      Are there any dependencies or how can I verify, why there are no statistics data before christmas 2022?

      I don't expect any issue with the BIG-IPs or the network. It seems for me like a local issue with the BIG-IQ.

      Are there any additional information/documentation about the availability of the statistics data?

      Thank you!

       

      Regards Stefan 🙂

      • Stefan_Klotz's avatar
        Stefan_Klotz
        Icon for Cumulonimbus rankCumulonimbus
        Mar 06, 2023

        It seems only around the last 30 days are stored? I changed nothing and now also the statistics from January are gone 😞

        I found the following settings, which I assume are default:

        But the "max storage Limit" is not reached:


        Any other ideas, why statistics data are not stored up to one year? Or how can I further investigate this issue?
        Thank you!

        Regards Stefan 🙂

  • Stefan_Klotz's avatar
    Stefan_Klotz
    Icon for Cumulonimbus rankCumulonimbus
    Apr 12, 2023

    It's me again. It definitely seems to be the case that the BIG-IQ is storing max. 2 months of data. Now also the February-values are gone.

    And end of April or latest beginning of May, the March-values will be deleted.

    The question is, which values or parameters are causing this behavior?

    Thank you!

    Regards Stefan 🙂