Forum Discussion
mystery files clogging up disk partition var_log
Version 11.5.2. Here's the message I'm getting every few minutes: 011d0004: Disk partition var_log has only 9% free emerg alertd[6319]: 01100048:0: Log disk usage still higher than 80% after logrotate and 24 times log deletion
According to "df -h", 6.0G of 6.9 G is used in partition var_log. However, "du -h" shows a total of only 62M in the var/log directory. The output of "ls -lRH --block-size=M" matches the output of "du -h", only 62 M of files and directories.
How do I figure out what the offending files are? Thanks in advance...
[imanadmin@LTM-1:/S1-green-P:Active:In Sync] log df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg--db--vda-set.1.root
380M 211M 150M 59% /
/dev/mapper/vg--db--vda-set.1._config
3.0G 76M 2.8G 3% /config
/dev/mapper/vg--db--vda-set.1._usr
2.5G 1.8G 536M 78% /usr
/dev/mapper/vg--db--vda-set.1._var
3.0G 438M 2.4G 16% /var
/dev/mapper/vg--db--vda-dat.share.1
20G 224M 19G 2% /shared
/dev/mapper/vg--db--vda-dat.log.1
**6.9G 6.0G 638M 91% /var/log**
none 5.6G 916K 5.6G 1% /dev/shm
none 5.6G 45M 5.6G 1% /shared/rrd.1.2
none 5.6G 7.8M 5.6G 1% /var/tmstat
none 5.6G 1.6M 5.6G 1% /var/run
prompt 4.0M 28K 4.0M 1% /var/prompt
none 5.6G 0 5.6G 0% /var/loipc
[imanadmin@LTM-1:/S1-green-P:Active:In Sync] log pwd
/var/log
[imanadmin@LTM-1:/S1-green-P:Active:In Sync] log du -h /
8.0K /var/log/ts/temp
8.0K /var/log/ts/units
8.0K /var/log/ts/archive/tmp
16K /var/log/ts/archive
8.0K /var/log/ts/unsent
144K /var/log/ts
8.0K /var/log/dosl7
524K /var/log/avr
18M /var/log/auditd
16K /var/log/cbr
8.0K /var/log/wa/push
8.0K /var/log/wa/stats/system
8.0K /var/log/wa/stats/perfmonitor
8.0K /var/log/wa/stats/tmp
32K /var/log/wa/stats
8.0K /var/log/wa/access
56K /var/log/wa
8.0K /var/log/dwbl
17M /var/log/iprepd
20K /var/log/tomcat
26M /var/log/monitors
396K /var/log/sa
8.0K /var/log/pem
20K /var/log/lost+found
16K /var/log/pam
20K /var/log/httpd
**62M /var/log**
5 Replies
- Mike_O_157132
Nimbostratus
There's a rather long winded explanation at http://linuxshellaccount.blogspot.com/2008/12/why-du-and-df-display-different-values.html that may help. Long story short, the two commands measure completely different values so wouldn't be expected to yield the same results except in an idle system (one that isn't using any files for its own use either).
- drbyrd
Nimbostratus
Thanks Mike for the info. In this case, df and du are nowhere close to each other (6G versus 62M). Something is being measured by df that du can't see. I'm thinking the diskmonitor utility uses df or something similar- the % free in the syslog message always agrees with what df says. Still can't find the offending files.
- drbyrd
Nimbostratus
I found the problem after finding this DevCentral page. I had deleted monitor log files before stopping the "monitor logging" on a few nodes. Basically there were files I couldn't see that were still being written to.
I ran "lsof | grep deleted" to find the files, then I just started/stopped monitor logging for each of the affected nodes. That seemed to clear the unlinked files and I now have 97% free in var/log.
- Mike_O_157132
Nimbostratus
I'll add that to my arsenal. Thanks for the update.
- drbyrd
Nimbostratus
Many thanks, Gustavo.
Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com