Deleted files not freeing their handle in /tmp

[Centurion]

We have a recurring problem on our 4.5.6 Network Edition server. The /tmp directory keeps filling up but the space reported with user space tools ("du" for example) is next to nothing.

According to the standard system tools:

[root@mail tmp]# df -h
Filesystem            Size  Used Avail Use% Mounted on
...snipped...
/dev/md3              3.0G  1.8G  1.1G  63% /tmp

But looking at the /tmp directory with "du" reveals....

[root@mail tmp]# du -hs /tmp
820K    /tmp

Digging deeper with "lsof" we see the following:

[root@mail tmp]# lsof | grep "/tmp"
bash       1970    root  cwd       DIR        9,3       4096          2 /tmp
screen    13507   jgray    3r     FIFO        9,3                 15747 /tmp/uscreens/S-jgray/13507.pts-0.mail
perl      19932  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
perl      19932  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
zmlogger  19937  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
zmlogger  19937  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
zmlogger  19937  zimbra    4w      REG        9,3 1884374605         26 /tmp/zmlogger.out (deleted)
mysqld_sa 21321  zimbra    1w      REG        9,3         70         15 /tmp/zmcontrol.out.20738 (deleted)
mysqld_sa 21321  zimbra    2w      REG        9,3         70         15 /tmp/zmcontrol.out.20738 (deleted)
logswatch 21391  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
logswatch 21391  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
mysqld    21402  zimbra    5u      REG        9,3          0         20 /tmp/ibD4jO0l (deleted)
mysqld    21402  zimbra    6u      REG        9,3          0         21 /tmp/ibzy1fOB (deleted)
mysqld    21402  zimbra    7u      REG        9,3          0         22 /tmp/ibXukIBR (deleted)
mysqld    21402  zimbra    8u      REG        9,3          0         23 /tmp/ibJj1dq7 (deleted)
mysqld    21402  zimbra   12u      REG        9,3          0         25 /tmp/iblBE0Vn (deleted)
perl      21423  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
perl      21423  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
zmlogger  21446  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
zmlogger  21446  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
zmlogger  21446  zimbra    4w      REG        9,3     373318         27 /tmp/zmlogger.out
mysqld_sa 21877  zimbra    1w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
mysqld_sa 21877  zimbra    2w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
mysqld    21972  zimbra    5u      REG        9,3       3075         28 /tmp/ibagshl0 (deleted)
mysqld    21972  zimbra    6u      REG        9,3          0         29 /tmp/ibY1LZ2w (deleted)
mysqld    21972  zimbra    7u      REG        9,3          0         30 /tmp/ibakKIK3 (deleted)
mysqld    21972  zimbra    8u      REG        9,3          0         31 /tmp/ib4yAKFB (deleted)
mysqld    21972  zimbra   12u      REG        9,3          0         32 /tmp/ibk4EXla (deleted)
java      22007  zimbra  mem       REG        9,3      32768     299138 /tmp/hsperfdata_zimbra/22007
zmtomcatm 22790    root    1w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
zmtomcatm 22790    root    2w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
java      22791  zimbra  mem       REG        9,3      32768     330626 /tmp/hsperfdata_root/22791
swatch    23099  zimbra    1w      REG        9,3         79         34 /tmp/swatch.out
swatch    23099  zimbra    2w      REG        9,3         79         34 /tmp/swatch.out
perl      23117  zimbra    1w      REG        9,3         79         34 /tmp/swatch.out
perl      23117  zimbra    2w      REG        9,3         79         34 /tmp/swatch.out
lsof      27703    root  cwd       DIR        9,3       4096          2 /tmp
bash      27704    root  cwd       DIR        9,3       4096          2 /tmp
lsof      27705    root  cwd       DIR        9,3       4096          2 /tmp
logswatch 28631  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
logswatch 28631  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
sshd      32234   jgray    7u     unix 0xc67d0380              98791141 /tmp/ssh-JGggs32234/agent.32234

See all the deleted files? Despite restarting Zimbra, the file handles are not being freed. Below is a graph showing how quickly the space is being "consumed" on /tmp - when it is full, Zimbra will crash; it happened a few weeks ago.

[Centurion]

Looks like I jinxed myself. Zimbra crashed at 4:30am today due to no space (sic) on temp. After shutting down the remaining Zimbra processes (ldap and mysql) with "zmcontrol stop" I then had to kill some PID's that wouldn't let go of deleted files - these PID's belonged to zmlogger and mysql.

After that, the space in /tmp magically dropped to 201MB used and 2.8GB free. Restarted Zimbra (zmcontrol start) and it's working again.

Can someone shed some light on this? The server is running 4.5.6 Network Edition on CentOS 4.4 (i386) running on a Dell PE2650.

[jholder]

What version? Your profile says 4.5.10, but your post has 4.5.6

john

[mmorse]

Because he's running 2
4.5.10_GA UBUNTU6 FOSS and 4.5.6_GA RHEL4 Network Edition <time to upgrade that to 4.5.10 anyway

Originally Posted by Centurion

The server is running 4.5.6 Network Edition on CentOS 4.4 (i386) running on a Dell PE2650.

[mmorse]

what's the filesystem? (df -T)

That 1884374605 26 /tmp/zmlogger.out (deleted) is 1.7GB which immediately reminded me...

There was a problem with logger in 4.5.5, there was a patch, which was mostly rolled into 4.5.6 but unfortunately the debug mode got left on in a couple of the scripts so those files continued to be large.
4.5.7 lowered the logging level back to the normal level & with that release /tmp/zmlogger.out /tmp/logswatch.out moved from /tmp to /opt/zimbra/log
At the same time, logswatch.out became zmlogswatch.out and I believe stuff either rotated or combined into other logs.
Also zmmtaconfig.log becomes huge: [SOLVED] running a version 4.5.6 or prior? Prevent Large Log File
If you stay on that version I would temporarily edit /opt/zimbra/libexec/zmmtaconfig and change the default loglevel (on line 55ish) to something lower -change it from 5 to 3
One step further, this should be in the next 5.0 build: Bug 10398 - clean up logging for swatch, zmmtaconfig
Oh, and also on 4.5.6 the clamav build that was included turned out to be bad: [SOLVED] Clamd.pid - no such file

Not like I'm suggesting you upgrade or anything...

[Centurion]

Yes - I am running two versions 4.5.10 FOSS (for our test/eval system) and the production machine is running 4.5.6 Network Edition.

There is currently a change request "in the system" to upgrade the production system to 4.5.10. The stumbling block for us at the moment was the restriction added in 4.5.9 that limits client filters to 21kB total. We have a number of users (including myself) whose filters are well in excess of this limit. So until the users rationalise their rules, we're holding back the upgrade.

Having been a mail admin for a number of years now (mail is "my thing") I know that until we impose a deadline on this, the users will never rationalise their rules. Such is the nature of users

Looks like system availability will take precedence though over rules. Thanks for the info. I'll update this thread with a success/failure message after the upgrade. You guys have given me the "ammo" I need to force the upgrade. Cheers