Zimbra

*greenrenault* · #1 (**permalink**) 08-31-2006, 04:19 AM

G'day

We're using Zimbra 3.1 on CentOS4 Linux with 1GB RAM. The Zimbra client is quite slow to respond to requests, so I did a quick check of top and got the following unusual stats. High load average and unusual amount of swap being used.

Does anyone know what would cause only 470MB of RAM to be used out of a possible 1GB available RAM and then 512MB of swap? Very weird.

This server is currently supporting 15 users.

Any suggestions appreciated.

[admin@mail ~]$ top
top - 18:47:29 up 1 day, 3:58, 1 user, load average: 4.99, 3.14, 2.61
Tasks: 108 total, 3 running, 105 sleeping, 0 stopped, 0 zombie
Cpu0 : 2.5% us, 13.2% sy, 0.0% ni, 49.1% id, 35.2% wa, 0.1% hi, 0.0% si
Mem: 1043912k total, 469896k used, 574016k free, 992k buffers
Swap: 1052248k total, 513748k used, 538500k free, 37292k cached

phoenix · #2 (**permalink**) 08-31-2006, 07:55 AM

Are you talking about the web gui being slow? I don't see that behaviour on my CentOS 4.4 system.

Is there any other problem with your system? My first suggestion would be an upgrade to the current release 4 of Zimbra as it's considerably improved over recent releases - don't forget a backup before the upgrade.

Have you been affected by this issue ?

*greenrenault* · #3 (**permalink**) 08-31-2006, 05:22 PM

Quote:

Originally Posted by phoenix

Are you talking about the web gui being slow? I don't see that behaviour on my CentOS 4.4 system.

Is there any other problem with your system? My first suggestion would be an upgrade to the current release 4 of Zimbra as it's considerably improved over recent releases - don't forget a backup before the upgrade.

Have you been affected by this issue ?

Thanks for the reply, we actually did that bug fix as part of the install process to get it up and running.

To clarify:
1. This server is running within VMWare server as a VMWare image.
2. CentOS 4.4 OS
3. The server load average is frequently between 1 and 5 with a high amount of swap usage, even though there is plenty of available RAM. This impacts most noticably on the Web UI causing the "server slow warning" to be displayed to users.

This problem has occurred after running the system in production for a week. It did not show up during the install/testing process. It's totally bizarre.

Here's another top snapshot.

Anymore ideas?

[admin@mail ~]$ top
top - 11:15:24 up 1 day, 20:26, 1 user, load average: 1.14, 1.63, 2.18
Tasks: 99 total, 1 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 1.1% sy, 0.0% ni, 94.7% id, 4.2% wa, 0.0% hi, 0.0% si
Mem: 1043912k total, 393024k used, 650888k free, 652k buffers
Swap: 1052248k total, 487688k used, 564560k free, 35216k cached

dijichi2 · #4 (**permalink**) 09-02-2006, 04:30 AM

thats very bizarre indeed. normally you would only see swap and free mem like that for a short period after a 1Gb+ process quit and the vm is still recovering. if you sort top by virt or res, can you see any huge memory hogs?

for sure the swap is causing your slowdowns, check vmstat and confirm the box is thrashing. try lowering the tomcat and mysql mem usage temporarily to see what impact this has.

i have noticed zimbra chaps on the forum saying vmware isn't supported or recommended. are you running the latest kernel?

marcmac · #5 (**permalink**) 09-02-2006, 09:40 AM

We are actually ok with VMware - we've got a couple of ISP customers who have great success with it.

What we try to discourage is "I have an old P3 that I use for testing, and I'm running 8 vmware instances on it, each has 128MB of ram and a 4GB slice of a 5200 RPM disk, and my question is, why is zimbra slow?"

Klug · #6 (**permalink**) 09-02-2006, 11:00 AM

Quote:

Originally Posted by marcmac

We are actually ok with VMware

Marc, what do you exactly mean ?

I was given the "we don't support Network Edition on VMWare/ESX" information a couple weeks ago...

marcmac · #7 (**permalink**) 09-02-2006, 11:13 AM

Yes, we support it, but we don't make performance promises (it works great, but you need enough hardware.)

Klug · #8 (**permalink**) 09-02-2006, 11:36 AM

Quote:

Originally Posted by marcmac

Yes, we support it, but we don't make performance promises (it works great, but you need enough hardware.)

Excellent news !
Thank you.

*greenrenault* · #9 (**permalink**) 09-02-2006, 06:34 PM

Quote:

Originally Posted by dijichi2

thats very bizarre indeed. normally you would only see swap and free mem like that for a short period after a 1Gb+ process quit and the vm is still recovering. if you sort top by virt or res, can you see any huge memory hogs?

for sure the swap is causing your slowdowns, check vmstat and confirm the box is thrashing. try lowering the tomcat and mysql mem usage temporarily to see what impact this has.

i have noticed zimbra chaps on the forum saying vmware isn't supported or recommended. are you running the latest kernel?

Here are some more stats. Looks like JVM and MySQL are the culprits. Any specific suggestions on how to calm this savage memory beast?

[admin@mail ~]$ uname -r
2.6.9-34.0.2.EL

[root@mail ~]# free -m
total used free shared buffers cached
Mem: 1019 356 662 0 1 39
-/+ buffers/cache: 316 703
Swap: 1027 100 926

[admin@mail ~]$ top
top - 12:41:31 up 2 days, 2:46, 1 user, load average: 0.58, 0.43, 0.42
Tasks: 93 total, 1 running, 92 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.8% us, 40.1% sy, 0.0% ni, 46.2% id, 7.9% wa, 0.0% hi, 0.0% si
Mem: 1043912k total, 221440k used, 822472k free, 244k buffers
Swap: 1052248k total, 526044k used, 526204k free, 26320k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6651 zimbra 18 0 569m 95m 6792 S 0.3 9.4 4:08.47 java
21173 zimbra 17 0 52124 26m 2788 S 0.0 2.6 0:02.01 amavisd
5557 zimbra 16 0 55060 15m 704 S 0.0 1.5 0:23.75 clamd
25331 zimbra 16 0 52804 14m 2560 S 0.0 1.4 0:05.73 amavisd
30062 zimbra 20 0 461m 12m 9252 S 0.0 1.2 0:01.50 java
5860 zimbra 16 0 424m 9744 1420 S 0.0 0.9 1:23.89 mysqld
26907 zimbra 16 0 52788 8380 2400 S 0.0 0.8 0:06.20 amavisd
10698 zimbra 16 0 52860 7532 2500 S 0.0 0.7 0:06.39 amavisd
22790 zimbra 16 0 53040 5320 2320 S 0.0 0.5 0:07.32 amavisd
2204 ntp 16 0 4976 4976 3428 S 0.0 0.5 0:12.71 ntpd
6390 zimbra 16 0 51156 4888 2116 S 0.0 0.5 0:00.92 amavisd
5575 zimbra 16 0 105m 3804 1796 S 0.0 0.4 4:44.44 mysqld
21625 zimbra 17 0 52256 3480 1940 S 0.0 0.3 0:03.37 amavisd
2445 zimbra 18 0 45956 2956 1996 S 0.0 0.3 0:28.23 slapd
4878 zimbra 16 0 49908 2556 1840 S 0.0 0.2 0:03.59 amavisd
4215 zimbra 17 0 6444 2372 1580 S 0.3 0.2 1:46.04 zmmtaconfig
451 zimbra 16 0 50636 2372 2132 S 0.0 0.2 0:00.07 amavisd
16178 admin 16 0 8036 2296 1836 S 0.3 0.2 0:00.38 sshd
16176 root 16 0 7880 2232 1784 S 0.0 0.2 0:00.05 sshd
23281 zimbra 16 0 52424 1992 1932 S 0.0 0.2 0:06.33 amavisd
22873 zimbra 16 0 52244 1992 1932 S 0.0 0.2 0:05.31 amavisd
6680 zimbra 16 0 4660 1988 1708 S 0.0 0.2 0:02.66 zmconvertdmon
17795 postfix 16 0 7068 1688 1288 S 0.0 0.2 0:00.04 anvil
17794 postfix 16 0 7228 1680 1284 S 0.0 0.2 0:00.02 proxymap
5537 zimbra 15 0 8932 1612 1608 S 0.0 0.2 0:01.78 perl
7596 zimbra 15 0 9932 1612 1608 S 0.0 0.2 0:01.33 perl
7588 zimbra 17 0 7260 1604 1600 S 0.0 0.2 0:00.20 swatch
5326 zimbra 17 0 6316 1600 1596 S 0.0 0.2 0:00.17 logswatch
17836 postfix 17 0 7716 1572 1236 S 0.0 0.2 0:00.02 showq
4811 root 16 0 6900 1356 1280 S 0.0 0.1 0:00.17 cupsd
7531 postfix 16 0 6812 1352 1236 S 0.0 0.1 0:01.45 qmgr
2290 root 16 0 6268 1324 748 S 0.0 0.1 1:18.71 hald
8596 postfix 16 0 7484 1308 1220 S 0.0 0.1 0:00.81 tlsmgr
7522 root 16 0 6640 1300 1204 S 0.0 0.1 0:11.24 master
29693 postfix 17 0 7020 1280 1228 S 0.0 0.1 0:00.03 pickup
16179 admin 15 0 2428 1260 1044 S 0.0 0.1 0:00.08 bash
18381 admin 16 0 3864 948 748 R 0.3 0.1 0:00.19 top
2153 root 16 0 4940 872 772 S 0.0 0.1 0:00.46 sshd

[admin@mail ~]$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 525432 810712 624 28628 43 25 157 71 35 44 2 13 67 17

[admin@mail ~]$ ps aux

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 3480 456 ? S Sep01 0:02 init [3]
root 2 0.0 0.0 0 0 ? SN Sep01 0:00 [ksoftirqd/0]
root 3 0.0 0.0 0 0 ? S< Sep01 0:04 [events/0]
root 4 0.0 0.0 0 0 ? S< Sep01 0:00 [khelper]
root 5 0.0 0.0 0 0 ? S< Sep01 0:00 [kacpid]
root 18 0.0 0.0 0 0 ? S< Sep01 0:00 [kblockd/0]
root 28 0.0 0.0 0 0 ? S Sep01 0:00 [pdflush]
root 29 0.0 0.0 0 0 ? S Sep01 0:11 [pdflush]
root 31 0.0 0.0 0 0 ? S< Sep01 0:00 [aio/0]
root 19 0.0 0.0 0 0 ? S Sep01 0:00 [khubd]
root 30 0.1 0.0 0 0 ? S Sep01 4:33 [kswapd0]
root 105 0.0 0.0 0 0 ? S Sep01 0:00 [kseriod]
root 178 0.0 0.0 0 0 ? S Sep01 0:00 [scsi_eh_0]
root 190 0.0 0.0 0 0 ? S Sep01 0:25 [kjournald]
root 1025 0.0 0.0 3156 360 ? S<s Sep01 0:00 udevd
root 1178 0.0 0.0 0 0 ? S Sep01 0:00 [shpchpd_event]
root 1236 0.0 0.0 0 0 ? S< Sep01 0:00 [kauditd]
root 1269 0.0 0.0 0 0 ? S< Sep01 0:00 [kmirrord]
root 1270 0.0 0.0 0 0 ? S< Sep01 0:00 [kmir_mon]
root 1896 0.0 0.0 2676 568 ? Ss Sep01 0:07 syslogd -m 0
root 1900 0.0 0.0 1572 392 ? Ss Sep01 0:00 klogd -x
rpc 1918 0.0 0.0 3112 424 ? Ss Sep01 0:00 portmap
rpcuser 1938 0.0 0.0 2900 552 ? Ss Sep01 0:00 rpc.statd
root 1964 0.0 0.0 4776 548 ? Ss Sep01 0:01 rpc.idmapd
root 2030 0.0 0.0 1588 520 ? Ss Sep01 0:23 /usr/sbin/vmware-guestd --background /var/run/vmware-guestd.pid
root 2102 0.0 0.0 3328 456 ? Ss Sep01 0:00 /usr/sbin/acpid
root 2153 0.0 0.0 4940 872 ? Ss Sep01 0:00 /usr/sbin/sshd
root 2189 0.0 0.0 2704 540 ? Ss Sep01 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp 2204 0.0 0.4 4976 4976 ? SLs Sep01 0:12 ntpd -u ntp:ntp -p /var/run/ntpd.pid
root 2213 0.0 0.0 3448 416 ? Ss Sep01 0:00 gpm -m /dev/input/mice -t imps2
root 2222 0.0 0.0 3692 668 ? Ss Sep01 0:02 crond
xfs 2255 0.0 0.0 5868 604 ? Ss Sep01 0:00 xfs -droppriv -daemon
root 2272 0.0 0.0 2656 612 ? Ss Sep01 0:00 /usr/sbin/atd
dbus 2281 0.0 0.0 3808 604 ? Ss Sep01 0:00 dbus-daemon-1 --system
root 2290 0.0 0.1 6268 1324 ? Ss Sep01 1:18 hald
zimbra 2445 0.0 0.2 45956 2956 ? Ssl Sep01 0:28 /opt/zimbra/openldap-2.3.21/libexec/slapd -l LOCAL0 -4 -u zimbra -h ldap://:389/ -f /opt/zimbra/conf/slapd.conf
root 2493 0.0 0.0 2904 344 tty1 Ss+ Sep01 0:00 /sbin/mingetty tty1
root 2494 0.0 0.0 2600 344 tty2 Ss+ Sep01 0:00 /sbin/mingetty tty2
root 2495 0.0 0.0 2968 344 tty3 Ss+ Sep01 0:00 /sbin/mingetty tty3
root 2496 0.0 0.0 2952 344 tty4 Ss+ Sep01 0:00 /sbin/mingetty tty4
root 2497 0.0 0.0 1928 344 tty5 Ss+ Sep01 0:00 /sbin/mingetty tty5
root 2611 0.0 0.0 2944 344 tty6 Ss+ Sep01 0:00 /sbin/mingetty tty6
zimbra 4215 0.0 0.2 6444 2372 ? S Sep01 1:45 /usr/bin/perl /opt/zimbra/libexec/zmmtaconfig
zimbra 4878 0.0 0.2 49908 2556 ? Ss Sep01 0:03 amavisd (master)
zimbra 5294 0.0 0.0 2980 804 ? Ss Sep01 0:00 /opt/zimbra/clamav/bin/freshclam --config-file=/opt/zimbra/conf/freshclam.conf -d --checks=12
zimbra 5307 0.0 0.0 3932 816 ? S Sep01 0:00 /bin/sh /opt/zimbra/logger/mysql/bin/mysqld_safe --defaults-file=/opt/zimbra/conf/my.logger.cnf --ledir=/opt/zimbra/logger/mysql/bin
zimbra 5326 0.0 0.1 6316 1600 ? S Sep01 0:00 /usr/bin/perl /opt/zimbra/libexec/logswatch --config-file=/opt/zimbra/conf/logswatchrc --script-dir=/tmp -t /var/log/zimbra.log
zimbra 5537 0.0 0.1 8932 1612 ? S Sep01 0:01 /usr/bin/perl /tmp/.swatch_script.5326
zimbra 5556 0.0 0.0 3764 420 ? S Sep01 0:01 /usr/bin/tail -n 0 -f /var/log/zimbra.log
zimbra 5557 0.0 1.5 55060 15808 ? Ss Sep01 0:23 /opt/zimbra/clamav/sbin/clamd --config-file /opt/zimbra/conf/clamd.conf
zimbra 5575 0.1 0.3 108476 3800 ? Sl Sep01 4:44 /opt/zimbra/logger/mysql/bin/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-locking --port=7307 --socket=/opt/zimbra/logger/db/mysql.sock
zimbra 5785 0.0 0.0 2300 816 ? S Sep01 0:00 /bin/sh /opt/zimbra/mysql/bin/mysqld_safe --defaults-file=/opt/zimbra/conf/my.cnf --ledir=/opt/zimbra/mysql/bin
zimbra 5860 0.0 0.9 435136 9744 ? Sl Sep01 1:23 /opt/zimbra/mysql/bin/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-locking --port=7306 --socket=/opt/zimbra/db/mysql.sock
zimbra 6651 0.1 9.3 583240 97336 ? Sl Sep01 4:08 /opt/zimbra/jdk1.5.0_06/bin/java -Xms305m -Xmx305m -client -XX:NewRatio=2 -Dcatalina.base=/opt/zimbra/apache-tomcat-5.5.15 -Dcatalina.home=/opt/zimbra/apache-tomcat-5.5.15 -Djava.io.tmpdir=/opt/zimbra/apache-tomcat-5.5.15/temp -Djava.library.path=/opt/zimbra/lib/jars -Djava.endorsed.dirs=/opt/zimbra/apache-tomcat-5.5.15/common/endorsed -classpath /opt/zimbra/apache-tomcat-5.5.15/bin/bootstrap.jar:/opt/zimbra/apache-tomcat-5.5.15/bin/commons-logging-api.jar org.apache.catalina.startup.Bootstrap start
zimbra 6680 0.0 0.1 4660 1988 ? S Sep01 0:02 /usr/bin/perl /opt/zimbra/libexec/zmconvertdmon -c /opt/zimbra/libexec/zmconvertd
root 7522 0.0 0.1 6640 1300 ? Ss Sep01 0:11 /opt/zimbra/postfix-2.2.9/libexec/master
postfix 7531 0.0 0.1 6812 1352 ? S Sep01 0:01 qmgr -l -t fifo -u
zimbra 7565 0.0 0.0 6468 780 ? Ss Sep01 0:00 /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd -r -a zimbra
zimbra 7566 0.0 0.0 6468 780 ? S Sep01 0:00 /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd -r -a zimbra
zimbra 7567 0.0 0.0 6468 780 ? S Sep01 0:00 /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd -r -a zimbra
zimbra 7568 0.0 0.0 6468 780 ? S Sep01 0:00 /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd -r -a zimbra
zimbra 7571 0.0 0.0 6468 780 ? S Sep01 0:00 /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd -r -a zimbra
zimbra 7588 0.0 0.1 7260 1604 ? S Sep01 0:00 /usr/bin/perl /opt/zimbra/libexec/swatch --config-file=/opt/zimbra/conf/swatchrc --script-dir=/tmp -t /var/log/zimbra.log
zimbra 7596 0.0 0.1 9932 1612 ? S Sep01 0:01 /usr/bin/perl /tmp/.swatch_script.7588
zimbra 7597 0.0 0.0 10716 724 ? Ss Sep01 0:01 /opt/zimbra/httpd-2.0.54/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 7598 0.0 0.0 11272 744 ? S Sep01 0:00 /opt/zimbra/httpd-2.0.54/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 7599 0.0 0.0 10716 684 ? S Sep01 0:00 /opt/zimbra/httpd-2.0.54/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 7600 0.0 0.0 10716 684 ? S Sep01 0:00 /opt/zimbra/httpd-2.0.54/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 7601 0.0 0.0 10716 684 ? S Sep01 0:00 /opt/zimbra/httpd-2.0.54/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 7602 0.0 0.0 10716 684 ? S Sep01 0:00 /opt/zimbra/httpd-2.0.54/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 7604 0.0 0.0 2588 416 ? S Sep01 0:02 /usr/bin/tail -n 0 -f /var/log/zimbra.log
postfix 8596 0.0 0.1 7484 1308 ? S Sep01 0:00 tlsmgr -l -t unix -u
zimbra 12081 0.0 0.0 10716 692 ? S Sep01 0:00 /opt/zimbra/httpd-2.0.54/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 23281 0.0 0.1 52424 1992 ? S Sep01 0:06 amavisd (ch9-avail)
zimbra 25331 0.0 1.3 52804 14548 ? S Sep01 0:05 amavisd (ch8-avail)
zimbra 26907 0.0 0.8 52788 8380 ? S Sep01 0:06 amavisd (ch8-avail)
zimbra 22790 0.0 0.5 53040 5320 ? S Sep02 0:07 amavisd (ch8-avail)
zimbra 22873 0.0 0.1 52244 1992 ? S Sep02 0:05 amavisd (ch6-avail)
zimbra 10698 0.0 0.7 52860 7532 ? S Sep02 0:06 amavisd (ch7-avail)
zimbra 21625 0.0 0.3 52256 3480 ? S Sep02 0:03 amavisd (ch4-avail)
zimbra 21173 0.0 2.6 52124 27248 ? S 01:47 0:02 amavisd (ch2-avail)
zimbra 6390 0.0 0.4 51156 4888 ? S 02:35 0:00 amavisd (ch1-avail)
root 4811 0.0 0.1 6900 1356 ? Ss 04:02 0:00 cupsd
zimbra 451 0.0 0.2 50636 2372 ? S 08:52 0:00 amavisd (virgin child)
zimbra 30062 0.0 1.1 472168 12524 ? Sl 10:13 0:01 /opt/zimbra/java/bin/java -client -Xmx256m -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib/jars -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib/jars -Djava.awt.headless=true com.zimbra.cs.convertd.TransformationServer
postfix 29693 0.0 0.1 7020 1280 ? S 11:43 0:00 pickup -l -t fifo -u
root 16176 0.0 0.2 7880 2232 ? Ss 12:34 0:00 sshd: admin [priv]
admin 16178 0.1 0.2 7880 2284 ? S 12:34 0:00 sshd: admin@pts/0
admin 16179 0.0 0.1 2428 1260 pts/0 Ss 12:34 0:00 -bash
postfix 17464 0.0 0.1 7416 1580 ? S 12:38 0:00 showq -t unix -u
admin 17496 0.0 0.0 4308 752 pts/0 R+ 12:38 0:00 ps aux

The server is a XEON 3GHZ, 2GB RAM, 143GB RAID MIRROR 10KRPM, it's a mover and shaker

(well usually anyway).

*greenrenault* · #10 (**permalink**) 09-04-2006, 10:15 PM

Found the solution to this problem. Turns out that it was a CentOS 4.3 kernel problem (or bug).

The latest Kernel update (2.6.9-42.0.2.EL) fixed the problem. Now the SAME server does not use any swap at all and is responding much quicker.

So there appears to be a problem with CentOS 4.3 and the default kernel, 2.6.9-34.0.2.EL which causes performance and swap issues. This may affect just Java applications or Java/VMWare guest OS - who knows.

Just very glad that it's been fixed.

So for anyone else running CentOS 4.3 and experiencing terrible performance and high swap usage, I suggest you run update and get the latest kernel.

Cheers!

Zimbra

Downloads

Product Information

Toolbox

Why Join?