[SOLVED] Free space disappearing

[moneymonk]

Hello-

Last friday my users started receiving warnings on their mail clients (and webmail) that their mail could not be sent due to lack of disk space. Alarming, curious, but it happens. Funny thing was i has just gone thru all my boxes to make sure nothing was short on space probably a week before. there was some stuff i could move onto SAN to make room, and I created about 11G. On Monday the 11G was gone. I made a few G more. Today it is gone again.

Zimbra is the only application running on the physical disc on this machine. everything else is ln -s linked to the SAN. I cannot for the life of me figure out where the space is going. It is not a constant ticking away of space but an ebb and flow for awhile. no change for awhile and then a giant bite. then ebb and flow, stable, giant bite.

My questions are these:

1) any idea what in zimbra could be doing this? Large attachments or messages? some sort of logging feedback loop?

2) is it a definite no no to move /opt to my SAN and ln -s link? At least this way i have enough space to keep mail flowing while i figure out what is happening. I've read the scuttlebutt about updates not working across the links and im not too worried about this. We are going to build a commercial version of ZCS on a new server in the near future and migrate to it. No updates for our version.

3) Also, my logger files are pretty huge, and i could use that space. for the time being. I read that this code works

su - zimbra
stop logger db (mysql processes)
rm -rf /opt/zimbra/logger/db/data
/opt/zimbra/libexec/zmloggerinit

But i want to be clear.

stop logger db (mysql processes)

Is not really code. what i want to do is stop mysql, correct?
Then,

rm -rf /opt/zimbra/logger/db/data

Delete the directory (or in my case move it to the SAN so i can look it over)
Then,

/opt/zimbra/libexec/zmloggerinit

run this script which re-creates the dir structure for logging.
Start mysql, start logger.

is this the gist?

I've already leaked 600MB since this morning, and have about 900MB left on the disk, so please help and thanks in advance. Time is of the essence.

Mike

[Klug]

Go to /opt/zimbra and do a "du -h" with the "one directory level only" parameter (the parameter is different depending on the OS).

This way you'll find where the free space is going...

[mmorse]

"I created about 11G." What did you move? (Log files themselves in /opt/zimbra/log I assume)

Are you still running 4.5.3_GA_733 MAC OS X ?
Is your zmmtaconfig.log really large?
If so you can edit line 55ish of /opt/zimbra/libexec/zmmtaconfig from 5 to 3.
4.5.7 lowered the log intensity level: Bug 18445 - zmmtaconfig default loglevel is set to high
5.0.0 added log rotation: Bug 10398 - clean up logging for swatch, zmmtaconfig

To control the logger DB size a bit:
su - zimbra
zmprov mcf zimbraLogRawLifetime 1d (31d by default)
zmprov mcf zimbraLogSummaryLifetime 7d (730d by default)

If you're curious how this process works examine /opt/zimbra/libexec/zmlogprocess (which is fired off every 10 min from crontab against the zimbra_logger DB.)

To answer your other questions yes that process would empty the DB.

As for upgrading it might be time to go to 4.5.11 if you're still at 4.5.3

[moneymonk]

Thanks so much for the quick response...

Im sorry, I cant find line 55ish of /opt/zimbra/libexec/zmmtaconfig to edit.

can you tell me how the line reads?

Working on the other "logger intensity" edits.

where can i find the upgrade to 4.5.11? i was looking yesterday for OSS upgrades.

thanks again-
Mike

[mmorse]

my %config = (
loglevel => 5,

(We created a zmmtaconfig_log_level attribute later.)

Halfway down this page: /community/downloads_previous.html
Old release notes: /community/documentation_previous_releases.html
Don't forget a backup!

I wasn't sure if you we're using PPC or not (we don't support that in the 5.0.x series so that migration would be a little more involved if you are).
You should still do 4.5.11 first before making the jump to 5.0.10 anyways.
Probably gonna need a some room free first to upgrade - or if you want to risk it comment out the space check in the installer script (I know we check on new installs but don't remember if we check during upgrades).

[moneymonk]

Thanks for all yr help, Mike. Klug too. That du -hd 1 incantation is sweet indeed. (i usually use du -cks * |sort -rn |head -11 which works nice too.) Thanks again.

Enough chatting. TCB.

Strange. "pattern not found" using vim search for loglevel in /opt/zimbra/libexec/zmmtaconfig.

I also ran:

su - zimbra
zmprov mcf zimbraLogRawLifetime
zmprov mcf zimbraLogSummaryLifetime

Will these logs flush themselves, or should I continue with

su - zimbra
/opt/zimbra/mysql/bin mysql stop
rm -rf /opt/zimbra/logger/db/data
/opt/zimbra/libexec/zmloggerinit
/opt/zimbra/mysql/bin mysql stop
zmloggerctl start

Thank you too for the info. I will take a close look at the update documents once I put out this fire and have 11 or 12 beers. We are currently using OS X 10.4.xx, but will migrate to a screamer running RHEL (fingers crossed) for our 5.0.xx migration.

I was hoping to put off the migration until summer '09, but the space issue has become pressing. I wish the guy before me used LVM.

What is the consensus on moving the /opt dir and creating a ln -s until I can piece together this migration?

Mike

[moneymonk]

Seems I lost 20G to /opt/local/var/amavis/tmp

any idea what this file does, and can i purge it?

Mike

[Rich Graves]

You really mean /opt/local not /opt/zimbra? Is that a Macism or a localism?

That would be temporary workspace for antispam/antivirus. It's supposed to be cleaned up when the message is either delivered or rejected, but might not be if something crashes. I think we saw growth there back in the 4.5 days, too.

If ought to be safe to remove anything more than a day old without downtime:

cd /opt/local/var/amavis/tmp
find . -maxdepth 1 -mtime +1 -type d | xargs rm -rf

[moneymonk]

It's def /opt/local ... just double checked.

I didn't set up our Zimbra, so I dont know what the impetus was behind some of the decisions. One of my many predecessors did. I'm just really glad I found where my disk space leak was stemming from.

is amavisd part of the Zimbra package? what is its purpose?

everyone has been super helpful. i really appreciate all the feedback.

Mike

[Rich Graves]

amavisd-new ("a mail anti-virus system") is one of Zimbra's open source components, but I'd expect the path to be /opt/zimbra/amavisd-(version)/tmp (in 5.0.x, this moved to a subdirectory of data/). It's also possible someone that installed their own custom copy.

ls -ld /opt/local/var/amavis/tmp

Is it owned by user zimbra?

Is /opt/local symlinked to /opt/zimbra or vice versa?

In any case, old things in "tmp" most certainly can be purged...