Performance issue ZCS 7.x and 8.x on ESXi 5.0

[Labsy]

Hi,

I've read a lot of threads where others are talking about Zimbra server performance on ESX/ESXi, but my experience tells me it is confusing.

For example, I have one Zimbra 8.0.2 server on Ubuntu 10.04, running on ESXi 5.0.
ZCS has 8 GB dedicated RAM, and 2GHz reserved and available all 10GHz total power of host, which runs on 2 x Quad XEON 2,5 GHz.
It is single-server all-in-one Zimbra installation, with 400 mailboxes.

Average MTA digestion is approx. 9.000 email messages per day, except a spike of 10.000 up to 100.000 mails sent out by some mailing outburst once per day or two.

Average CPU usage is 7%...
...but during these mailings, where single user sends mail out to 10.000 or sometimes even to 100.000 users, CPU of total host burns oil at 80-100% and chokes the whole ESXi server.

Is such a behavior normal?
Well, I know the mailing application does NOT send 1 single mail with 100.000 recipients, but rather sends out 1-by-1, single mail at the time, so there is really huge traffic. But should that choke my whole ESXi? It becomes unresponsive, mail queues extend to 1000 or more mails in Queue, mail delay is up to 4 hours, and web sites, which are also run at this ESXi, return "Timeout".

What hardware upgrade would help most?
Is it RAM upgrade?
Or should Ultra fast SSD PCI-e adapter ioDrive from Fusion boost performance?
Or should I replace SATA disks in RAID 10 with faster SAS 15k disks?

Ideas welcome.

[lytledd]

Similar setup to yours. Quad core Xenon, ESXi 5.0, dual 15k SCSI drives and around 300 users. The differences are, our outbound for a single mailing usually only hits a couple thousand at once. My VM is also Ubuntu 10.04 LTS, but /opt/ is on a LVM and I don't use RAID at all.

My suggestions would be to:

Disable swap, our VM is using 9GB without swap, but I have 27GB assigned. To date, it hasn't gone above 9.
If you're able, get rid of RAID.

I put the following in my /etc/sysctl.conf:

# Turn off swap (DL)
vm.swappiness=0

# Zimbra wiki suggestions (DL)
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_tw_reuse=1

This is the article I followed:

Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki

Doug

[Labsy]

Well, thanx for advice...which I agre with mostly, except on the part "Get rid of RAID", which is misleading. If we are talking about RAID 5 or 6 type, yes, then it is true - these RAID types are not the performers, but rather optimized for capacity. While on the other hand RAID 1, 10 and even RAID DP are optimized for performance (RAID DP also +capacity) and ARE recommended much more than single-disk operation, be it performance-wise or security-wise.

Now back to my problems.
What bothers me most is understanding the SWAP. I know, there are TWO SWAP layers in my config:
- Zimbra OS SWAP inside virtual machine, swapping to virtual disk
- and ESXi host swapping consumed VM RAM to ESXi host disk

Regarding ESXi SWAP I have all sorted out, I monitor ESX swap all the time and act immediately. Most of the time none of my VM swap, but as I have 2 VMs without proper VMWare Tools, it happens every 2-3 weeks that one of machines begins swapping. I reboot them ASAP to get rid of swap.

While internal OS SWAP... well, my Zimbra shows the following (sorted by SWAP):

top - 11:11:40 up 10 days, 10:29,  1 user,  load average: 0.32, 0.38, 0.30
Tasks: 176 total,   1 running, 175 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.8%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8196528k total,  7862740k used,   333788k free,   217016k buffers
Swap:  9289720k total,   851876k used,  8437844k free,  1707348k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND
 1541 zimbra    20   0 80.3g  54m  19m S    0  0.7  19:56.95  80g slapd
 3226 zimbra    20   0 4115m 3.4g  11m S    0 43.5 314:33.53 632m java
 3140 zimbra    20   0 1286m 707m 5600 S    0  8.8  23:06.48 579m mysqld
21185 zimbra    20   0  731m 314m  14m S    0  3.9   1:54.19 416m java
 3655 zimbra    20   0  322m  580  536 S    0  0.0   3:23.98 321m httpd
 3663 zimbra    20   0  322m  584  540 S    0  0.0   3:23.67 321m httpd
 3686 zimbra    20   0  322m  588  544 S    0  0.0   3:23.09 321m httpd
 3364 zimbra    20   0  454m 184m 6340 S    0  2.3  83:17.62 269m clamd
 3918 zimbra    20   0  240m  14m 1740 S    0  0.2   2:04.35 226m opendkim
20828 syslog    20   0  184m 1644 1032 S    0  0.0   0:02.86 183m rsyslogd
 3279 zimbra    20   0  212m  41m 2684 S    0  0.5   0:59.37 171m amavisd
16312 zimbra    20   0  212m  41m 1096 S    0  0.5   0:00.02 170m amavisd
 7599 zimbra    20   0  224m  84m 4048 S    0  1.1   0:01.02 140m amavisd
11379 zimbra    20   0  221m  82m 4392 S    0  1.0   0:06.43 139m amavisd
 1897 zimbra    20   0  221m  82m 4380 S    0  1.0   0:04.81 139m amavisd
22341 zimbra    20   0  224m  85m 4380 S    0  1.1   0:06.02 139m amavisd
 2925 zimbra    20   0  228m  89m 4376 S    0  1.1   0:04.78 138m amavisd
15339 zimbra    20   0  225m  86m 4392 S    0  1.1   0:17.37 138m amavisd
28392 zimbra    20   0  223m  85m 4400 S    0  1.1   0:11.03 138m amavisd
 7729 zimbra    20   0  231m  92m 4396 S    0  1.2   0:12.78 138m amavisd
17885 zimbra    20   0  230m  91m 4396 S    0  1.1   0:08.07 138m amavisd
 3626 zimbra    20   0  106m  936  884 S    0  0.0   0:47.77 105m httpd
17145 postfix   20   0 99.6m 5944 4620 S    0  0.1   0:00.05  93m smtpd
18192 postfix   20   0 99908 5388 4120 S    0  0.1   0:00.03  92m smtpd
 1440 root      20   0 84468 2532 1576 S    0  0.0   1:17.01  80m miniserv.pl
 1001 root      20   0 82768 1972 1728 S    0  0.0  11:10.58  78m vmtoolsd
11219 labsy     20   0 66484 1704  968 S    0  0.0   0:02.04  63m sshd
11198 root      20   0 66484 3140 2424 S    0  0.0   0:00.50  61m sshd
 1403 postgrey  20   0 58260 1952 1948 S    0  0.0   0:00.05  54m postgrey
 3899 root      20   0 55196 1064  940 S    0  0.0   2:25.31  52m master
 4644 postfix   20   0 55408 2476 2296 S    0  0.0   0:08.73  51m tlsmgr
 3917 zimbra    20   0 53256  456  312 S    0  0.0   0:00.01  51m opendkim
 3901 postfix   20   0 63280  10m 2348 S    0  0.1   1:12.39  51m qmgr
 2702 postfix   20   0 55384 3048 2284 S    0  0.0   0:00.05  51m showq
18189 postfix   20   0 55620 3288 2436 S    0  0.0   0:00.02  51m smtp
 4244 postfix   20   0 55300 2980 2236 S    0  0.0   0:00.02  51m pickup
18193 postfix   20   0 55624 3360 2500 S    0  0.0   0:00.02  51m lmtp
 2904 postfix   20   0 55288 3040 2280 S    0  0.0   0:00.03  51m anvil
14222 postfix   20   0 55304 3064 2284 S    0  0.0   0:00.05  51m trivial-rewrite
18188 postfix   20   0 55424 3188 2396 S    0  0.0   0:00.02  51m cleanup
 3908 zimbra    20   0 55744 3924 2508 S    0  0.0   0:42.68  50m saslauthd
 3909 zimbra    20   0 55744 3924 2508 S    0  0.0   0:44.12  50m saslauthd

First... how come "slapd" process using 80G of swap and only 0.7% of RAM??!! Where I missed the point?
Secondly... are these numbers telling me, that my Zimbra is crying for more RAM?

[ccelis5215]

Hi Labsy,

First... how come "slapd" process using 80G of swap and only 0.7% of RAM??!! Where I missed the point?
Secondly... are these numbers telling me, that my Zimbra is crying for more RAM?

Regarding your first question, this columns show the maximum amount of swap will be required if the whole process was swapped out,
and yes..., maybe you need more RAM.

I know, this answer it's not very helpful.

ccelis

[lytledd]

I've never done anything beyond RAID 5.

As for the swapping, I've noted that if it's enabled, Zimbra will use it, even if there is plenty of memory. That's why I initially gave the VM 27GB. It's settled around 9GB.

Also worth noting, We've been using ASSP for spam filtering and virus scanning for years, before we moved to Zimbra. So, that portion of Zimbra has been disabled for us since the beginning.

Doug

[Labsy]

Hmmm...
So, here's what crosses my mind:

1.) Would it be wise to disable swapping on Zimbra OS-level while at the same time doubling RAM from 8 to 16 GB?
...or should just change "swappiness" for Linux from default 60 to, let's say, 10 or even 0?

2.) We are using our own ASF Antispam for the most important 1/4 domains (business users), so Zimbra is not under pressure. But still it has Antispam and Antivirus enabled for the rest 3/4 of domains.

[ccelis5215]

Hi again,

1.) Would it be wise to disable swapping on Zimbra OS-level while at the same time doubling RAM from 8 to 16 GB?
...or should just change "swappiness" for Linux from default 60 to, let's say, 10 or even 0?

No, it's not wise disable swap in a complex application server as Zimbra.
16 GB will be better than 8.
Yes, turn swappiness to 0

ccelis

[lytledd]

I also agree with ccelis,

16GB and swappiness=0

You can keep an eye on it for a couple weeks to get an idea what it'll actually be using, then you can lower it if necessary. I've left mine at 27GB, since it's the only VM on the machine.

Doug