Zimbra uses disk like crazy

[sirio81]

Hi all, I have a server with debian lenny 64 bit running a virtual zimbra machine (centos 5 64 bit) using kvm 0.12.4.

Since yesterday, zimbra is using intensively the disk. Zimbra is using two partitions: hda1 for (/) and sdb1 (/opt).

Using atop in centos, i see hdb busy 99% (read 10, write 526). hdb is used only by zimbra (/opt).
Cpu usage is low for all processes.

From host side (debian), by consequence, I also have a heavy disk access.

I'm trying to isolate the process that uses the disk so much
http://www.xaprb.com/blog/2009/08/23...tics-on-linux/

From CENTOS

# ./monitor_io_disk.sh 
TASK                   PID      TOTAL       READ      WRITE      DIRTY DEVICES
kjournald             1259       2259          0       2259          0 hdb1
pdflush              14945        575          0        575          0 hda1, hdb1
kjournald              416        405          0        405          0 hda1
mysqld                6001        274         21        253          0 hdb1
java                  4175        269          0        269          0 hdb1
mysqld                6597        253         16        237          0 hdb1
mysqld                7089        178         11        167          0 hdb1
mysqld                6008        137          8        129          0 hdb1
mysqld                6224        126         10        116          0 hdb1
mysqld                4150        120          8        112          0 hdb1
mysqld                6205         69          6         63          0 hdb1
java                  4335         57          0         57          0 hdb1
mysqld               12768         50          4         46          0 hdb1
java                  4328         45          0         45          0 hdb1
mysqld                3195         41          0         41          0 hdb1
java                  4327         38          0         38          0 hdb1
java                  4333         29          0         29          0 hdb1
mysqld                4914         28          1         27          0 hdb1
java                  4334         25          0         25          0 hdb1
java                  4331         24          0         24          0 hdb1
mysqld                3211         12          0         12          0 hdb1
java                  4332         10          0         10          0 hdb1
java                  4329         10          0         10          0 hdb1
java                  4336          4          0          4          0 hdb1
syslogd               1769          3          0          3          0 hda1
java                  4330          3          0          3          0 hdb1
mysqld                3193          2          0          2          0 hdb1
auditd                1735          1          0          1          0 hda1

If I stop zimbra (/etc/init.d/zimbra stop), disk usage go back normal.
I tried also to reboot the virtual machine but it didn't help.

Any hint?

[chauvetp]

Is this system in production? If so, how many users do you have?

What kind of storage is the /opt directory on (SAN/local disk/NAS/etc). What RAID configuration is the disk on?

[raj]

whats the zimbra version you are running ?
and whats the size of logger folder

# du -sh /opt/zimbra/logger

I have seen the logger in the older zimbra versions 5.0.x grow to 50GB + and bog down the VM due to disk usage.

stopping logger service will bring things to normal and you can also clean the big logger folder by removeing and re-restting the logger.

Raj

[sirio81]

Is this system in production? If so, how many users do you have?

Yes, it is. Not many users: 14.

I think I found the problem origin.
This zimbra installation has a domain called mail.local domain.
Smtp relies on external smtp.
Users mail are fetched from external pop3 server.
And now the strange thing: we have 2 users that need to watch the mail of the other 13 users.
Call these two user andrew and mattias.
To achive that, on andrew@mail.local user, I added an external imap account for each one of the 13 users.
So the external imap account is otheruser@mail.local.
Logging in zimbra web interface like andrew@mail.local, it's possible to see all the other users mail. The same for mattias.
(Sounds realy bad I know).

Here's another problem: when a new mail is received from one of those external account, zimbra web interface doesn't show it. It's not behaving like a traditional imap client.
To 'refresh' the mail list, I had to shedule

zmprov mds mattias@mail.local OtherZimbraImapAccount zimbraDataSourcePollingInterval 7m

(for all the 13 users x 2)...
Watching /opt/zimbra/log/mailbox.log I notiched it was continuing importing stuff

2010-09-22 16:37:01,256 INFO  [ScheduledTask-2] [name=matias@mail.local;mid=22;ds=cilenti;] datasource - Importing data for data source 'cilenti'
2010-09-22 16:37:01,353 INFO  [ScheduledTask-2] [name=matias@mail.local;mid=22;ds=cilenti;] datasource - Importing from IMAP folder Sent to local folder /cilenti/Sent
2010-09-22 16:37:02,652 INFO  [ScheduledTask-2] [name=matias@mail.local;mid=22;ds=cilenti;] datasource - Importing from IMAP folder Junk to local folder /cilenti/Junk
2010-09-22 16:37:02,657 INFO  [ScheduledTask-2] [name=matias@mail.local;mid=22;ds=cilenti;] datasource - Importing from IMAP folder Contacts to local folder /cilenti/Contacts
2010-09-22 16:37:02,659 INFO  [ScheduledTask-2] [name=matias@mail.local;mid=22;ds=cilenti;] datasource - Importing from IMAP folder Trash to local folder /cilenti/Trash
2010-09-22 16:37:02,663 INFO  [ScheduledTask-2] [name=matias@mail.local;mid=22;ds=cilenti;] datasource - Importing from IMAP folder Drafts to local folder /cilenti/Drafts
2010-09-22 16:37:02,826 INFO  [ScheduledTask-2] [name=matias@mail.local;mid=22;ds=cilenti;] datasource - Importing from IMAP folder Chats to local folder /cilenti/Chats
2010-09-22 16:37:02,831 INFO  [ScheduledTask-2] [name=matias@mail.local;mid=22;ds=cilenti;] datasource - Importing from IMAP folder Emailed Contacts to local folder /

It's sure that polling of the imap accounts overlap and they takes a lot of resources.

I tried to reschedule the polling interval far away (60 minutes) and atop was showing a normal disk activity.

Sometime it still get red (busy 99%) but only sometime, not costantly like before.

Now the questions:
is there a better way to watch several users account from one account?
Is polling the only way to have zimbra web interface updated?

Thank you

[sirio81]

su - zimbra -c 'zmcontrol -v'
Release 5.0.16_GA_2921.RHEL5_64_20090429051405 CentOS5_64 FOSS edition

du -sh /opt/zimbra/logger 
879M    /opt/zimbra/logger

Thank you for the logger hint.
But isn't there a logrotate in zimbra?