We have a recurring problem on our 4.5.6 Network Edition server. The /tmp directory keeps filling up but the space reported with user space tools ("du" for example) is next to nothing.
According to the standard system tools:
Code:
[root@mail tmp]# df -h
Filesystem Size Used Avail Use% Mounted on
...snipped...
/dev/md3 3.0G 1.8G 1.1G 63% /tmp
But looking at the /tmp directory with "du" reveals....
Code:
[root@mail tmp]# du -hs /tmp
820K /tmp
Digging deeper with "lsof" we see the following:
Code:
[root@mail tmp]# lsof | grep "/tmp"
bash 1970 root cwd DIR 9,3 4096 2 /tmp
screen 13507 jgray 3r FIFO 9,3 15747 /tmp/uscreens/S-jgray/13507.pts-0.mail
perl 19932 zimbra 1w REG 9,3 71523 18 /tmp/logswatch.out (deleted)
perl 19932 zimbra 2w REG 9,3 71523 18 /tmp/logswatch.out (deleted)
zmlogger 19937 zimbra 1w REG 9,3 71523 18 /tmp/logswatch.out (deleted)
zmlogger 19937 zimbra 2w REG 9,3 71523 18 /tmp/logswatch.out (deleted)
zmlogger 19937 zimbra 4w REG 9,3 1884374605 26 /tmp/zmlogger.out (deleted)
mysqld_sa 21321 zimbra 1w REG 9,3 70 15 /tmp/zmcontrol.out.20738 (deleted)
mysqld_sa 21321 zimbra 2w REG 9,3 70 15 /tmp/zmcontrol.out.20738 (deleted)
logswatch 21391 zimbra 1w REG 9,3 82 19 /tmp/logswatch.out
logswatch 21391 zimbra 2w REG 9,3 82 19 /tmp/logswatch.out
mysqld 21402 zimbra 5u REG 9,3 0 20 /tmp/ibD4jO0l (deleted)
mysqld 21402 zimbra 6u REG 9,3 0 21 /tmp/ibzy1fOB (deleted)
mysqld 21402 zimbra 7u REG 9,3 0 22 /tmp/ibXukIBR (deleted)
mysqld 21402 zimbra 8u REG 9,3 0 23 /tmp/ibJj1dq7 (deleted)
mysqld 21402 zimbra 12u REG 9,3 0 25 /tmp/iblBE0Vn (deleted)
perl 21423 zimbra 1w REG 9,3 82 19 /tmp/logswatch.out
perl 21423 zimbra 2w REG 9,3 82 19 /tmp/logswatch.out
zmlogger 21446 zimbra 1w REG 9,3 82 19 /tmp/logswatch.out
zmlogger 21446 zimbra 2w REG 9,3 82 19 /tmp/logswatch.out
zmlogger 21446 zimbra 4w REG 9,3 373318 27 /tmp/zmlogger.out
mysqld_sa 21877 zimbra 1w REG 9,3 105248 17 /tmp/zmcontrol.out.20738 (deleted)
mysqld_sa 21877 zimbra 2w REG 9,3 105248 17 /tmp/zmcontrol.out.20738 (deleted)
mysqld 21972 zimbra 5u REG 9,3 3075 28 /tmp/ibagshl0 (deleted)
mysqld 21972 zimbra 6u REG 9,3 0 29 /tmp/ibY1LZ2w (deleted)
mysqld 21972 zimbra 7u REG 9,3 0 30 /tmp/ibakKIK3 (deleted)
mysqld 21972 zimbra 8u REG 9,3 0 31 /tmp/ib4yAKFB (deleted)
mysqld 21972 zimbra 12u REG 9,3 0 32 /tmp/ibk4EXla (deleted)
java 22007 zimbra mem REG 9,3 32768 299138 /tmp/hsperfdata_zimbra/22007
zmtomcatm 22790 root 1w REG 9,3 105248 17 /tmp/zmcontrol.out.20738 (deleted)
zmtomcatm 22790 root 2w REG 9,3 105248 17 /tmp/zmcontrol.out.20738 (deleted)
java 22791 zimbra mem REG 9,3 32768 330626 /tmp/hsperfdata_root/22791
swatch 23099 zimbra 1w REG 9,3 79 34 /tmp/swatch.out
swatch 23099 zimbra 2w REG 9,3 79 34 /tmp/swatch.out
perl 23117 zimbra 1w REG 9,3 79 34 /tmp/swatch.out
perl 23117 zimbra 2w REG 9,3 79 34 /tmp/swatch.out
lsof 27703 root cwd DIR 9,3 4096 2 /tmp
bash 27704 root cwd DIR 9,3 4096 2 /tmp
lsof 27705 root cwd DIR 9,3 4096 2 /tmp
logswatch 28631 zimbra 1w REG 9,3 71523 18 /tmp/logswatch.out (deleted)
logswatch 28631 zimbra 2w REG 9,3 71523 18 /tmp/logswatch.out (deleted)
sshd 32234 jgray 7u unix 0xc67d0380 98791141 /tmp/ssh-JGggs32234/agent.32234
See all the deleted files? Despite restarting Zimbra, the file handles are not being freed. Below is a graph showing how quickly the space is being "consumed" on /tmp - when it is full, Zimbra will crash; it happened a few weeks ago.