Results 1 to 6 of 6

Thread: Deleted files not freeing their handle in /tmp

  1. #1
    Centurion's Avatar
    Centurion is offline Active Member
    Join Date
    Apr 2007
    Location
    NSW, Australia
    Posts
    38
    Rep Power
    8

    Default Deleted files not freeing their handle in /tmp

    We have a recurring problem on our 4.5.6 Network Edition server. The /tmp directory keeps filling up but the space reported with user space tools ("du" for example) is next to nothing.

    According to the standard system tools:
    Code:
    [root@mail tmp]# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    ...snipped...
    /dev/md3              3.0G  1.8G  1.1G  63% /tmp
    But looking at the /tmp directory with "du" reveals....
    Code:
    [root@mail tmp]# du -hs /tmp
    820K    /tmp
    Digging deeper with "lsof" we see the following:
    Code:
    [root@mail tmp]# lsof | grep "/tmp"
    bash       1970    root  cwd       DIR        9,3       4096          2 /tmp
    screen    13507   jgray    3r     FIFO        9,3                 15747 /tmp/uscreens/S-jgray/13507.pts-0.mail
    perl      19932  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    perl      19932  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    zmlogger  19937  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    zmlogger  19937  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    zmlogger  19937  zimbra    4w      REG        9,3 1884374605         26 /tmp/zmlogger.out (deleted)
    mysqld_sa 21321  zimbra    1w      REG        9,3         70         15 /tmp/zmcontrol.out.20738 (deleted)
    mysqld_sa 21321  zimbra    2w      REG        9,3         70         15 /tmp/zmcontrol.out.20738 (deleted)
    logswatch 21391  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
    logswatch 21391  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
    mysqld    21402  zimbra    5u      REG        9,3          0         20 /tmp/ibD4jO0l (deleted)
    mysqld    21402  zimbra    6u      REG        9,3          0         21 /tmp/ibzy1fOB (deleted)
    mysqld    21402  zimbra    7u      REG        9,3          0         22 /tmp/ibXukIBR (deleted)
    mysqld    21402  zimbra    8u      REG        9,3          0         23 /tmp/ibJj1dq7 (deleted)
    mysqld    21402  zimbra   12u      REG        9,3          0         25 /tmp/iblBE0Vn (deleted)
    perl      21423  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
    perl      21423  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
    zmlogger  21446  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
    zmlogger  21446  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
    zmlogger  21446  zimbra    4w      REG        9,3     373318         27 /tmp/zmlogger.out
    mysqld_sa 21877  zimbra    1w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
    mysqld_sa 21877  zimbra    2w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
    mysqld    21972  zimbra    5u      REG        9,3       3075         28 /tmp/ibagshl0 (deleted)
    mysqld    21972  zimbra    6u      REG        9,3          0         29 /tmp/ibY1LZ2w (deleted)
    mysqld    21972  zimbra    7u      REG        9,3          0         30 /tmp/ibakKIK3 (deleted)
    mysqld    21972  zimbra    8u      REG        9,3          0         31 /tmp/ib4yAKFB (deleted)
    mysqld    21972  zimbra   12u      REG        9,3          0         32 /tmp/ibk4EXla (deleted)
    java      22007  zimbra  mem       REG        9,3      32768     299138 /tmp/hsperfdata_zimbra/22007
    zmtomcatm 22790    root    1w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
    zmtomcatm 22790    root    2w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
    java      22791  zimbra  mem       REG        9,3      32768     330626 /tmp/hsperfdata_root/22791
    swatch    23099  zimbra    1w      REG        9,3         79         34 /tmp/swatch.out
    swatch    23099  zimbra    2w      REG        9,3         79         34 /tmp/swatch.out
    perl      23117  zimbra    1w      REG        9,3         79         34 /tmp/swatch.out
    perl      23117  zimbra    2w      REG        9,3         79         34 /tmp/swatch.out
    lsof      27703    root  cwd       DIR        9,3       4096          2 /tmp
    bash      27704    root  cwd       DIR        9,3       4096          2 /tmp
    lsof      27705    root  cwd       DIR        9,3       4096          2 /tmp
    logswatch 28631  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    logswatch 28631  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    sshd      32234   jgray    7u     unix 0xc67d0380              98791141 /tmp/ssh-JGggs32234/agent.32234
    See all the deleted files? Despite restarting Zimbra, the file handles are not being freed. Below is a graph showing how quickly the space is being "consumed" on /tmp - when it is full, Zimbra will crash; it happened a few weeks ago.
    Attached Images Attached Images

  2. #2
    Centurion's Avatar
    Centurion is offline Active Member
    Join Date
    Apr 2007
    Location
    NSW, Australia
    Posts
    38
    Rep Power
    8

    Default 4:30am - Zimbra crashed

    Looks like I jinxed myself. Zimbra crashed at 4:30am today due to no space (sic) on temp. After shutting down the remaining Zimbra processes (ldap and mysql) with "zmcontrol stop" I then had to kill some PID's that wouldn't let go of deleted files - these PID's belonged to zmlogger and mysql.

    After that, the space in /tmp magically dropped to 201MB used and 2.8GB free. Restarted Zimbra (zmcontrol start) and it's working again.

    Can someone shed some light on this? The server is running 4.5.6 Network Edition on CentOS 4.4 (i386) running on a Dell PE2650.

  3. #3
    jholder's Avatar
    jholder is offline Former Zimbran
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    20

    Default

    What version? Your profile says 4.5.10, but your post has 4.5.6

    john

  4. #4
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Because he's running 2
    4.5.10_GA UBUNTU6 FOSS and 4.5.6_GA RHEL4 Network Edition <time to upgrade that to 4.5.10 anyway
    Quote Originally Posted by Centurion View Post
    The server is running 4.5.6 Network Edition on CentOS 4.4 (i386) running on a Dell PE2650.

  5. #5
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    what's the filesystem? (df -T)

    That 1884374605 26 /tmp/zmlogger.out (deleted) is 1.7GB which immediately reminded me...

    There was a problem with logger in 4.5.5, there was a patch, which was mostly rolled into 4.5.6 but unfortunately the debug mode got left on in a couple of the scripts so those files continued to be large.
    4.5.7 lowered the logging level back to the normal level & with that release /tmp/zmlogger.out /tmp/logswatch.out moved from /tmp to /opt/zimbra/log
    At the same time, logswatch.out became zmlogswatch.out and I believe stuff either rotated or combined into other logs.
    Also zmmtaconfig.log becomes huge: [SOLVED] running a version 4.5.6 or prior? Prevent Large Log File
    If you stay on that version I would temporarily edit /opt/zimbra/libexec/zmmtaconfig and change the default loglevel (on line 55ish) to something lower -change it from 5 to 3
    One step further, this should be in the next 5.0 build: Bug 10398 - clean up logging for swatch, zmmtaconfig
    Oh, and also on 4.5.6 the clamav build that was included turned out to be bad: [SOLVED] Clamd.pid - no such file

    Not like I'm suggesting you upgrade or anything...
    Last edited by mmorse; 12-06-2007 at 09:34 PM.

  6. #6
    Centurion's Avatar
    Centurion is offline Active Member
    Join Date
    Apr 2007
    Location
    NSW, Australia
    Posts
    38
    Rep Power
    8

    Default Tell me about it...

    Yes - I am running two versions 4.5.10 FOSS (for our test/eval system) and the production machine is running 4.5.6 Network Edition.

    There is currently a change request "in the system" to upgrade the production system to 4.5.10. The stumbling block for us at the moment was the restriction added in 4.5.9 that limits client filters to 21kB total. We have a number of users (including myself) whose filters are well in excess of this limit. So until the users rationalise their rules, we're holding back the upgrade.

    Having been a mail admin for a number of years now (mail is "my thing") I know that until we impose a deadline on this, the users will never rationalise their rules. Such is the nature of users

    Looks like system availability will take precedence though over rules. Thanks for the info. I'll update this thread with a success/failure message after the upgrade. You guys have given me the "ammo" I need to force the upgrade. Cheers

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. [SOLVED] Error Installing Zimbra on RHEL 5
    By harris7139 in forum Installation
    Replies: 10
    Last Post: 09-25-2007, 11:39 AM
  2. Replies: 8
    Last Post: 02-27-2007, 04:10 AM
  3. Openldap log files not deleted
    By chh in forum Administrators
    Replies: 0
    Last Post: 02-12-2007, 06:59 AM
  4. svn version still won't start
    By kinaole in forum Developers
    Replies: 0
    Last Post: 10-04-2006, 06:47 AM
  5. Replies: 16
    Last Post: 09-07-2006, 06:39 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •