Results 1 to 8 of 8

Thread: Performance issue ZCS 7.x and 8.x on ESXi 5.0

  1. #1
    Labsy is offline Elite Member
    Join Date
    Nov 2009
    Location
    Ljubljana, Slovenia
    Posts
    268
    Rep Power
    5

    Default Performance issue ZCS 7.x and 8.x on ESXi 5.0

    Hi,

    I've read a lot of threads where others are talking about Zimbra server performance on ESX/ESXi, but my experience tells me it is confusing.

    For example, I have one Zimbra 8.0.2 server on Ubuntu 10.04, running on ESXi 5.0.
    ZCS has 8 GB dedicated RAM, and 2GHz reserved and available all 10GHz total power of host, which runs on 2 x Quad XEON 2,5 GHz.
    It is single-server all-in-one Zimbra installation, with 400 mailboxes.

    Average MTA digestion is approx. 9.000 email messages per day, except a spike of 10.000 up to 100.000 mails sent out by some mailing outburst once per day or two.

    Average CPU usage is 7%...
    ...but during these mailings, where single user sends mail out to 10.000 or sometimes even to 100.000 users, CPU of total host burns oil at 80-100% and chokes the whole ESXi server.

    Is such a behavior normal?
    Well, I know the mailing application does NOT send 1 single mail with 100.000 recipients, but rather sends out 1-by-1, single mail at the time, so there is really huge traffic. But should that choke my whole ESXi? It becomes unresponsive, mail queues extend to 1000 or more mails in Queue, mail delay is up to 4 hours, and web sites, which are also run at this ESXi, return "Timeout".

    What hardware upgrade would help most?
    Is it RAM upgrade?
    Or should Ultra fast SSD PCI-e adapter ioDrive from Fusion boost performance?
    Or should I replace SATA disks in RAID 10 with faster SAS 15k disks?

    Ideas welcome.
    Zimbra on SGH dedicated hosting farm, Slovenia.
    In 2013 we announce new program of low cost SSL server certificates.

  2. #2
    lytledd is offline Elite Member
    Join Date
    Dec 2009
    Location
    Michigan
    Posts
    450
    Rep Power
    5

    Default

    Similar setup to yours. Quad core Xenon, ESXi 5.0, dual 15k SCSI drives and around 300 users. The differences are, our outbound for a single mailing usually only hits a couple thousand at once. My VM is also Ubuntu 10.04 LTS, but /opt/ is on a LVM and I don't use RAID at all.

    My suggestions would be to:

    Disable swap, our VM is using 9GB without swap, but I have 27GB assigned. To date, it hasn't gone above 9.
    If you're able, get rid of RAID.

    I put the following in my /etc/sysctl.conf:

    Code:
    # Turn off swap (DL)
    vm.swappiness=0
    
    # Zimbra wiki suggestions (DL)
    net.ipv4.tcp_fin_timeout=15
    net.ipv4.tcp_tw_reuse=1
    This is the article I followed:

    Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki

    Doug
    Ben Franklin quote:

    "Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety."

  3. #3
    Labsy is offline Elite Member
    Join Date
    Nov 2009
    Location
    Ljubljana, Slovenia
    Posts
    268
    Rep Power
    5

    Default

    Well, thanx for advice...which I agre with mostly, except on the part "Get rid of RAID", which is misleading. If we are talking about RAID 5 or 6 type, yes, then it is true - these RAID types are not the performers, but rather optimized for capacity. While on the other hand RAID 1, 10 and even RAID DP are optimized for performance (RAID DP also +capacity) and ARE recommended much more than single-disk operation, be it performance-wise or security-wise.

    Now back to my problems.
    What bothers me most is understanding the SWAP. I know, there are TWO SWAP layers in my config:
    - Zimbra OS SWAP inside virtual machine, swapping to virtual disk
    - and ESXi host swapping consumed VM RAM to ESXi host disk

    Regarding ESXi SWAP I have all sorted out, I monitor ESX swap all the time and act immediately. Most of the time none of my VM swap, but as I have 2 VMs without proper VMWare Tools, it happens every 2-3 weeks that one of machines begins swapping. I reboot them ASAP to get rid of swap.

    While internal OS SWAP... well, my Zimbra shows the following (sorted by SWAP):
    Code:
    top - 11:11:40 up 10 days, 10:29,  1 user,  load average: 0.32, 0.38, 0.30
    Tasks: 176 total,   1 running, 175 sleeping,   0 stopped,   0 zombie
    Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.8%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:   8196528k total,  7862740k used,   333788k free,   217016k buffers
    Swap:  9289720k total,   851876k used,  8437844k free,  1707348k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND
     1541 zimbra    20   0 80.3g  54m  19m S    0  0.7  19:56.95  80g slapd
     3226 zimbra    20   0 4115m 3.4g  11m S    0 43.5 314:33.53 632m java
     3140 zimbra    20   0 1286m 707m 5600 S    0  8.8  23:06.48 579m mysqld
    21185 zimbra    20   0  731m 314m  14m S    0  3.9   1:54.19 416m java
     3655 zimbra    20   0  322m  580  536 S    0  0.0   3:23.98 321m httpd
     3663 zimbra    20   0  322m  584  540 S    0  0.0   3:23.67 321m httpd
     3686 zimbra    20   0  322m  588  544 S    0  0.0   3:23.09 321m httpd
     3364 zimbra    20   0  454m 184m 6340 S    0  2.3  83:17.62 269m clamd
     3918 zimbra    20   0  240m  14m 1740 S    0  0.2   2:04.35 226m opendkim
    20828 syslog    20   0  184m 1644 1032 S    0  0.0   0:02.86 183m rsyslogd
     3279 zimbra    20   0  212m  41m 2684 S    0  0.5   0:59.37 171m amavisd
    16312 zimbra    20   0  212m  41m 1096 S    0  0.5   0:00.02 170m amavisd
     7599 zimbra    20   0  224m  84m 4048 S    0  1.1   0:01.02 140m amavisd
    11379 zimbra    20   0  221m  82m 4392 S    0  1.0   0:06.43 139m amavisd
     1897 zimbra    20   0  221m  82m 4380 S    0  1.0   0:04.81 139m amavisd
    22341 zimbra    20   0  224m  85m 4380 S    0  1.1   0:06.02 139m amavisd
     2925 zimbra    20   0  228m  89m 4376 S    0  1.1   0:04.78 138m amavisd
    15339 zimbra    20   0  225m  86m 4392 S    0  1.1   0:17.37 138m amavisd
    28392 zimbra    20   0  223m  85m 4400 S    0  1.1   0:11.03 138m amavisd
     7729 zimbra    20   0  231m  92m 4396 S    0  1.2   0:12.78 138m amavisd
    17885 zimbra    20   0  230m  91m 4396 S    0  1.1   0:08.07 138m amavisd
     3626 zimbra    20   0  106m  936  884 S    0  0.0   0:47.77 105m httpd
    17145 postfix   20   0 99.6m 5944 4620 S    0  0.1   0:00.05  93m smtpd
    18192 postfix   20   0 99908 5388 4120 S    0  0.1   0:00.03  92m smtpd
     1440 root      20   0 84468 2532 1576 S    0  0.0   1:17.01  80m miniserv.pl
     1001 root      20   0 82768 1972 1728 S    0  0.0  11:10.58  78m vmtoolsd
    11219 labsy     20   0 66484 1704  968 S    0  0.0   0:02.04  63m sshd
    11198 root      20   0 66484 3140 2424 S    0  0.0   0:00.50  61m sshd
     1403 postgrey  20   0 58260 1952 1948 S    0  0.0   0:00.05  54m postgrey
     3899 root      20   0 55196 1064  940 S    0  0.0   2:25.31  52m master
     4644 postfix   20   0 55408 2476 2296 S    0  0.0   0:08.73  51m tlsmgr
     3917 zimbra    20   0 53256  456  312 S    0  0.0   0:00.01  51m opendkim
     3901 postfix   20   0 63280  10m 2348 S    0  0.1   1:12.39  51m qmgr
     2702 postfix   20   0 55384 3048 2284 S    0  0.0   0:00.05  51m showq
    18189 postfix   20   0 55620 3288 2436 S    0  0.0   0:00.02  51m smtp
     4244 postfix   20   0 55300 2980 2236 S    0  0.0   0:00.02  51m pickup
    18193 postfix   20   0 55624 3360 2500 S    0  0.0   0:00.02  51m lmtp
     2904 postfix   20   0 55288 3040 2280 S    0  0.0   0:00.03  51m anvil
    14222 postfix   20   0 55304 3064 2284 S    0  0.0   0:00.05  51m trivial-rewrite
    18188 postfix   20   0 55424 3188 2396 S    0  0.0   0:00.02  51m cleanup
     3908 zimbra    20   0 55744 3924 2508 S    0  0.0   0:42.68  50m saslauthd
     3909 zimbra    20   0 55744 3924 2508 S    0  0.0   0:44.12  50m saslauthd
    First... how come "slapd" process using 80G of swap and only 0.7% of RAM??!! Where I missed the point?
    Secondly... are these numbers telling me, that my Zimbra is crying for more RAM?
    Zimbra on SGH dedicated hosting farm, Slovenia.
    In 2013 we announce new program of low cost SSL server certificates.

  4. #4
    ccelis5215 is offline Elite Member
    Join Date
    Jun 2011
    Location
    Caracas Venezuela
    Posts
    462
    Rep Power
    4

    Default

    Hi Labsy,

    First... how come "slapd" process using 80G of swap and only 0.7% of RAM??!! Where I missed the point?
    Secondly... are these numbers telling me, that my Zimbra is crying for more RAM?
    Regarding your first question, this columns show the maximum amount of swap will be required if the whole process was swapped out,
    and yes..., maybe you need more RAM.

    I know, this answer it's not very helpful.

    ccelis

  5. #5
    lytledd is offline Elite Member
    Join Date
    Dec 2009
    Location
    Michigan
    Posts
    450
    Rep Power
    5

    Default

    I've never done anything beyond RAID 5.

    As for the swapping, I've noted that if it's enabled, Zimbra will use it, even if there is plenty of memory. That's why I initially gave the VM 27GB. It's settled around 9GB.

    Also worth noting, We've been using ASSP for spam filtering and virus scanning for years, before we moved to Zimbra. So, that portion of Zimbra has been disabled for us since the beginning.

    Doug
    Ben Franklin quote:

    "Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety."

  6. #6
    Labsy is offline Elite Member
    Join Date
    Nov 2009
    Location
    Ljubljana, Slovenia
    Posts
    268
    Rep Power
    5

    Default

    Hmmm...
    So, here's what crosses my mind:

    1.) Would it be wise to disable swapping on Zimbra OS-level while at the same time doubling RAM from 8 to 16 GB?
    ...or should just change "swappiness" for Linux from default 60 to, let's say, 10 or even 0?

    2.) We are using our own ASF Antispam for the most important 1/4 domains (business users), so Zimbra is not under pressure. But still it has Antispam and Antivirus enabled for the rest 3/4 of domains.
    Zimbra on SGH dedicated hosting farm, Slovenia.
    In 2013 we announce new program of low cost SSL server certificates.

  7. #7
    ccelis5215 is offline Elite Member
    Join Date
    Jun 2011
    Location
    Caracas Venezuela
    Posts
    462
    Rep Power
    4

    Default

    Hi again,

    Quote Originally Posted by Labsy View Post
    1.) Would it be wise to disable swapping on Zimbra OS-level while at the same time doubling RAM from 8 to 16 GB?
    ...or should just change "swappiness" for Linux from default 60 to, let's say, 10 or even 0?
    No, it's not wise disable swap in a complex application server as Zimbra.
    16 GB will be better than 8.
    Yes, turn swappiness to 0

    ccelis

  8. #8
    lytledd is offline Elite Member
    Join Date
    Dec 2009
    Location
    Michigan
    Posts
    450
    Rep Power
    5

    Default

    I also agree with ccelis,

    16GB and swappiness=0

    You can keep an eye on it for a couple weeks to get an idea what it'll actually be using, then you can lower it if necessary. I've left mine at 27GB, since it's the only VM on the machine.

    Doug
    Ben Franklin quote:

    "Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety."

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. asking help for performance issue
    By tiger2000 in forum Administrators
    Replies: 0
    Last Post: 10-18-2011, 06:41 PM
  2. ZCS on vmware esxi
    By jerryboi in forum Installation
    Replies: 5
    Last Post: 01-07-2011, 08:30 AM
  3. Very low performance issue
    By Panchux in forum Administrators
    Replies: 10
    Last Post: 10-04-2010, 12:10 AM
  4. performance issue
    By tiger2000 in forum Users
    Replies: 2
    Last Post: 03-07-2010, 12:37 AM
  5. Performance issue
    By andywu in forum Users
    Replies: 5
    Last Post: 10-23-2006, 02:16 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •