Zimbra running very very slow

[razametal]

Hi,

I've installed Zimbra 7 virtualized vmware machine:

cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz
stepping        : 4
cpu MHz         : 2533.422
cache size      : 8192 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida
bogomips        : 5066.84
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz
stepping        : 4
cpu MHz         : 2533.422
cache size      : 8192 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida
bogomips        : 5066.84
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          3964       2467       1497          0         67       1754
-/+ buffers/cache:        644       3320
Swap:          883          0        883

The Zimbra version installed:

$ zmcontrol -v
Release 7.1.3_GA_3346.UBUNTU10_64 UBUNTU10_64 FOSS edition.

The server is always slow:

$ uptime
 17:05:51 up  5:10,  1 user,  load average: 13.14, 13.59, 13.26

# top -b -n1 | head -20
top - 16:56:57 up  5:01,  1 user,  load average: 12.12, 13.62, 13.09
Tasks: 148 total,   1 running, 147 sleeping,   0 stopped,   0 zombie
Cpu(s): 13.0%us,  0.8%sy,  0.0%ni,  9.9%id, 76.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4059880k total,  3595136k used,   464744k free,    68028k buffers
Swap:   905208k total,        0k used,   905208k free,  1827752k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30842 root      20   0 1245m  15m 7860 S   22  0.4   0:00.12 java
30854 root      20   0 19220 1316  952 R    2  0.0   0:00.02 top
31145 zimbra    20   0 1293m 691m  13m S    2 17.4   3:11.26 java
    1 root      20   0 23580 1864 1292 S    0  0.0   0:01.77 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.01 kthreadd
    3 root      RT   0     0    0    0 S    0  0.0   0:00.08 migration/0
    4 root      20   0     0    0    0 S    0  0.0   0:00.26 ksoftirqd/0
    5 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0
    6 root      RT   0     0    0    0 S    0  0.0   0:00.08 migration/1
    7 root      20   0     0    0    0 S    0  0.0   0:00.27 ksoftirqd/1
    8 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/1
    9 root      20   0     0    0    0 S    0  0.0   0:00.44 events/0
   10 root      20   0     0    0    0 S    0  0.0   0:07.25 events/1

# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  4      0 126304  68796 1837792    0    0   101    65  151  273 13  1 10 76

I've disabled antispam and antivirus services but the performance is the same:

$ zmprov gs zimbra.mydomain.com | grep zimbraServiceEnabled
 zimbraServiceEnabled: mailbox
 zimbraServiceEnabled: memcached
 zimbraServiceEnabled: mta
 zimbraServiceEnabled: stats
 zimbraServiceEnabled: imapproxy
 zimbraServiceEnabled: snmp
 zimbraServiceEnabled: ldap
 zimbraServiceEnabled: logger

The services runing are:

 zmcontrol status
Host zimbra.mydomain.com
        imapproxy               Running
        ldap                    Running
        logger                  Running
        mailbox                 Running
        memcached               Running
        mta                     Running
        snmp                    Running
        stats                   Running
        zmconfigd               Running

Why the system is going slow most all day? What can I do to fix or improve it?

The access with the advanced webmail interface is very slow.

Regards,

[vavai]

Is this multi server installation or single server? For single server, you do not need to install memcached & imapproxy. memcached & imapproxy only required for multi server installation.

[razametal]

Hi Vavai, it is a single server installation.

I've disabled memcached and imapproxy services.

[GWilliams]

Hi razametal,

How many domains and accounts are you serving?

As root:
su - zimbra -c "zmprov gaa | wc -l"

Also, what is your disk configuration. Please note that if you are using some form of shared storage, that it may affect the overall performance/load of your system.

Please indicate the following outputs:

df -h
iostat 4

The df -h will indicate disk usage and drive configuration.
Run the iostat 4 command for about 20 seconds and post the output.

Hope this helps.

[razametal]

Hi GWilliams,

zimbra@zimbra:~$ zmprov gaa | wc -l
ERROR: service.INVALID_REQUEST (invalid request: can only be used with  "zmprov -l/--ldap")
0
zimbra@zimbra:~$ zmprov -l gaa | wc -l
31

zimbra@zimbra:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/zimbra-root
                       19G   17G  1.4G  93% /
none                  2.0G  184K  2.0G   1% /dev
none                  2.0G     0  2.0G   0% /dev/shm
none                  2.0G   36K  2.0G   1% /var/run
none                  2.0G     0  2.0G   0% /var/lock
none                  2.0G     0  2.0G   0% /lib/init/rw
/dev/sda1             228M   79M  137M  37% /boot
/dev/mapper/zimbradata-lv01
                       99G   31G   63G  33% /opt

iostat 4
Linux 2.6.32-34-server (zimbra.mydomain.com)      12/14/11        _x86_64_(2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.15    0.00    0.88   31.21    0.00   59.75

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.48        12.72        17.61     803180    1111756
sdb              17.00       593.49       258.02   37477456   16293344
dm-0             37.08       593.48       258.02   37476514   16293376
dm-1              2.67        11.98        17.61     756666    1111728
dm-2              0.00         0.03         0.00       1640          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          23.26    0.00    2.99   40.30    0.00   33.46

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.00         0.00         0.00          0          0
sdb               1.25         0.00        40.00          0        160
dm-0              4.75         0.00        68.00          0        272
dm-1              0.00         0.00         0.00          0          0
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.00    0.00    0.25   94.26    0.00    3.49

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.00         0.00        36.00          0        144
sdb               3.50         0.00        58.00          0        232
dm-0              2.25         0.00        30.00          0        120
dm-1              4.75         0.00        38.00          0        152
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.37    0.00    0.00   75.22    0.00   24.41

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.75         0.00         6.00          0         24
sdb              17.75         0.00       238.00          0        952
dm-0             40.75         0.00       340.00          0       1360
dm-1              1.50         0.00        12.00          0         48
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.12   86.60    0.00   13.28

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.00         0.00         0.00          0          0
sdb               2.50         0.00        26.00          0        104
dm-0              0.25         0.00         0.00          0          0
dm-1              3.00         0.00        24.00          0         96
dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.86    0.00    0.12   89.30    0.00    9.72

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.50         0.00        34.00          0        136
sdb               5.50         0.00       110.00          0        440
dm-0              7.25         0.00        60.00          0        240
dm-1              0.25         0.00         2.00          0          8
dm-2              0.00         0.00         0.00          0          0

[GWilliams]

Hi razametal,

Well, there's your problem. Your iowait is too high, which is causing read/write delays and therefore causing your system to perform slowly. With only 31 accounts, your system really shouldn't be slow at all, but disk performance is a major contributing factor. Perhaps you could advise your physical disk configuration and how it is setup in the virtual environment?

Regards.

[razametal]

Thank you for your response GWilliams, there is a problem with the high iowait reponse. I've asked to the person that owns de the virtual environment is having high load right now and this is the cause of the high iowait.

The system is runing with acceptable performance right now, I've changed the following:

su - zimbra
zmprov ms zimbra.manta.telconet.net -zimbraServiceEnabled memcached
zmprov ms zimbra.manta.telconet.net -zimbraServiceEnabled imapproxy
zmlocalconfig -e mailboxd_java_options="-server -Djava.awt.headless=true \
-Dsun.net.inetaddr.ttl= -XX:+UseConcMarkSweepGC -XX:PermSize=196m \
-XX:MaxPermSize=196m -XX:SoftRefLRUPolicyMSPerMB=1 \
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps \
-XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow \
-XX:+UseParNewGC -XX:NewRatio=2
zmcontrol stop
zmcontrol start