BIG-IQ statistics in dashboard missing/interrupted

[Stefan_Klotz]

Dear all,

any idea, why statistics data is missing/interrupted for a few days like in the following screenshot?

Or how I can verify any issues? In which logs can I found any hints?

The missing data is visible for device statistics as well as for Local Traffic. Am I right, that these statistics are the ones, which are stored on the DCDs? And why is there no data before 27.11.2022? Is there any limit configured that just the last 30 days (or so) will be stored? Does this indicate to an connectivity issue between the DCDs and the BIG-IPs?

As of now I could only verify that the ucs backups were successfully executed during these days, so connectivity between the BIG-IQ and the BIG-IPs was fine?

Thanks for any useful ideas or hints.

Regards Stefan 🙂

[Nikoolayy1]

Maybe check the /var/log/avr and /var/log/restjavad on the BIG-IP as the Analytics AVR is the way to stream logs to DCD and the restjavad is the HTTP API  that BIg-IQ uses to control BIG-IP and also check the BIG-IQ device and system logs on the BIG-IQ for the BIG-IP devices as BIG-IQ also manages the DCD.

Just in case see in GUI if the BIG-IP has log statistics for this period or upload qkview to ihealth and check again the Statistics of the BIG-IP as to exclude BIG-IP issue and check in health for any bugs under diagnostics.

[Leslie_Hubertus]

@Stefan_Klotz  - did @Nikoolayy1 's suggestion work for you? Inquiring minds want to know! 🙂

(also, if yes, don't forget to hit the Accept as Solution button if it did work - and if it didn't, let us know so we can see if there's another solution for you. 

[Stefan_Klotz]

Sorry for late reply.

I don't found any indications in any of the mentioned logs. Since end of December logging for all of our BIG-IPs seems to run stable again as you can see in the following screenshot:

To be more precise and the reason why I'm asking is, that I wonder if and how recording these statistics data depends on the free disc space of the BIG-IQ itself (not the DCDs). Background: until end of last year the "/shared" partition of the BIG-IQ was full, as there were too much ucs-files from all of our BIG-IPs. After christmas, one BIG-IP cluster was removed, which released some GB (because all corresponding ucs-files where deleted as well). Additionally I increased this partition to 100GB beginning of January. And as you can see in the above screenshot, statistics data is available again since around that date!

Are there any dependencies or how can I verify, why there are no statistics data before christmas 2022?

I don't expect any issue with the BIG-IPs or the network. It seems for me like a local issue with the BIG-IQ.

Are there any additional information/documentation about the availability of the statistics data?

Thank you!

Regards Stefan 🙂

[Stefan_Klotz]

It seems only around the last 30 days are stored? I changed nothing and now also the statistics from January are gone 😞

I found the following settings, which I assume are default:

But the "max storage Limit" is not reached:


Any other ideas, why statistics data are not stored up to one year? Or how can I further investigate this issue?
Thank you!

Regards Stefan 🙂

[Nikoolayy1]

As you mentioned /shared it should in no way be related to statistics from from what I have seen https://my.f5.com/manage/s/article/K30902515 . Maybe BIG-IQ had issues with a full hard disk of UCS files that somehow affected its communication to the DCD. F5 BIG-IQ should only hold some logs but not statisctics

For keeping more statistics the DCD should be the way. If you have 1 DCD add 2 more. 2 more as if you just add 1 more DCD they will replicate between each other an no real increase will there be of the Stats space.

Things that you may check. Outside of that see with support.

https://my.f5.com/manage/s/article/K35115510?utm_source=f5support&utm_medium=RSS

https://my.f5.com/manage/s/article/K10100299

[Stefan_Klotz]

It's me again. It definitely seems to be the case that the BIG-IQ is storing max. 2 months of data. Now also the February-values are gone.

And end of April or latest beginning of May, the March-values will be deleted.

The question is, which values or parameters are causing this behavior?

Thank you!

Regards Stefan 🙂

[Stefan_Klotz · Accepted solution]

We could solve the issue in the meanwhile together with official F5-support.

It's related to ID1044337:

This can occur intermittently during statistics collection from BIG-IP to BIG-IQ and can happen when the retention processor causes a loss of some retention records (book-keeper-start entries are lost while book-keeper-end entries are retained).”

From the BIG-IQ command line:

1) Make a backup of the book-keeper-end index:

curl -sk http://localhost:9200/_reindex?pretty -H "content-type: application/json" -d '{"source":{"index":"book-keeper-end"},"dest":{"index":"bu-book-keeper-end"}}'

2) Delete the book-keeper-end index:

curl -skX DELETE http://localhost:9200/book-keeper-end

Note:  The book-keeper-end index will be rebuilt automatically and will cause no loss of statistic data.

It took some time (I don't know how long exactly), but on the next day I could see statistics data back to one year again.

Regards Stefan 🙂