[SOLVED] Specs for Next Zimbra Machine

[the_griz]

We have been running Zimbra Network Edition for the last 18 months. We have a company of 50-60 mailboxes. Within the last few months, I have noticed the server load (via 'top') is often pegged at 8-11 and always at least 3. A full backup takes over 14 hours to complete for a 180GB mail store.

The machine response seems to be much slower than it was 6 months ago. The machine (Poweredge R200) runs on a single Xeon 3GHz/6M Cache E3110 dual-core processor. It has 4GB of RAM and 250 GB of disk space.

I am considering replacing the server it is currently on to new machine but am not sure how it should be spec'd out. Anyone else running a similar sized shop with any advice would be appreciated.

Thanks!

[LMStone]

Originally Posted by the_griz

We have been running Zimbra Network Edition for the last 18 months. We have a company of 50-60 mailboxes. Within the last few months, I have noticed the server load (via 'top') is often pegged at 8-11 and always at least 3. A full backup takes over 14 hours to complete for a 180GB mail store.

The machine response seems to be much slower than it was 6 months ago. The machine (Poweredge R200) runs on a single Xeon 3GHz/6M Cache E3110 dual-core processor. It has 4GB of RAM and 250 GB of disk space.

I am considering replacing the server it is currently on to new machine but am not sure how it should be spec'd out. Anyone else running a similar sized shop with any advice would be appreciated.

Thanks!

The trick with top is to determine what (or what combination) of bottlenecks is causing the high load.

Some things to look at in our experience are:

1. Once top is running, press "1" to expand the display to show CPU usage by core. Java and MySQL will typically use only 1-2 cores at a time, so a single, fast dual-core CPU can sometimes be speedier than two slower quad-core CPUs.
2. Look at the "%wa" (percentage wait states), and if it is in the double digits noticeably you have a disk bottleneck; the CPUs are "waiting" for I/O to complete before they can do work.
3. Look at the "%si" (software interrupts) statistic, especially if you are running virtualized. Anything in the high single digits or above indicates a lot of context switching is going on. We would see this on ESXi3.x guests, because ESXi3 is 32-bit. Upgrading to ESXi4 (vSphere) pretty much eliminates this bottleneck.

Better yet, post a screenshot of top and we'll be happy to help you interpret it.

Also, do check the logs, especially the mailbox log.

All the best,
Mark

[the_griz]

Thanks for the information and support.. I definitely seem to have a disk I/O problem. I have attached a screenshot ( or check out: http://immartin.com/docs/screenshot.jpg) the %wa is very high all the time.

For disk on this server, I have 2 250GB Seagate 7.2K disks in RAID 1 mirroring. Is this under-spec'd?

I had another sysadmin look at Zimbra about 8 months ago and he commented how he thought Zimbra loggged too much, that it was always hammering at the disk especially mailbox.log. Is there a way to check to see if I have my logging set too high?

Thanks!

[Klug]

Are you using hardware or software RAID?
If hardware, which controler?

[phoenix]

Originally Posted by the_griz

For disk on this server, I have 2 250GB Seagate 7.2K disks in RAID 1 mirroring. Is this under-spec'd?

The recommended RAID level for Zimbra is RAID10, you'd be better moving the /opt directory structure to that.

[the_griz]

It is a hardware RAID controller specifically -> SAS6iR RAID Controller (SATA/SAS Controller) from Dell.

As far as RAID10 being the recommended RAID level for Zimbaa, I have not seen that recommendation nor do I find it in the Admin guide 5 or 6. Where is that posted? Experience tells me that RAID10 is probably the way to go but I would like to review where that recommendation is documented and discussed.

Thanks!

[Klug]

SAS6i/r controler doesn't have any cache (opposite to PERC5 or PERC6 or the brand new ones). No cache on controler plus 7.2K RPM SATA drives can lead to bad perfs, especially under load.

In you case, "load" might not be related to the number of user but more to number of inbound/outbound messages.
Do you have lots of messages?
Is AV/AS activated on ZCS?
If so, it can be a good idea to use a ramdrive for amavis temp directory as this kills the SATA drives for sure.

RAM seems enough, your server is not swapping (as per your screenshot).

I have several servers running RAID1 only (SAS 15K, not SATA 7.2K) for OS and primary storage, and HSM on another RAID1 (SATA 7.2K), even with more than 200 users on them.
But I'm running a ramdrive for amavis (with 8 GB of RAM and a 2.23GHz QuadCore).

Here's the last mail report on one of these server:

messages

  24264   received
  26914   delivered
      0   forwarded
     57   deferred  (598  deferrals)
     26   bounced
  65935   rejected (71%)
      0   reject warnings
      0   held
      0   discarded (0%)

   1761m  bytes received
   2814m  bytes delivered
   3429   senders
   1998   sending hosts/domains
    628   recipients
    278   recipient hosts/domains

[the_griz]

This is good advice. I will create the ramdrive and put amavisd's tmp directory on it.

Are there any other functions that you have going to the ramdrive?

Do you think it would be wise to lower the logging level in Zimbra? It is quite chatty.

[briceb]

Hi,

Zimbra seems to recommend Hardware Raid, you seem to recommend it to, but I have to say I am a bit confused, what is the point really as Software raid is sometimes just faster and easier to manage ? Also not all hardware raid cards will send you an email when failure occurs, mdadm on the other hand..

http://www.redhat.com/docs/manuals/l...pproaches.html

[LMStone]

Originally Posted by the_griz

Thanks for the information and support.. I definitely seem to have a disk I/O problem. I have attached a screenshot ( or check out: http://immartin.com/docs/screenshot.jpg) the %wa is very high all the time.

For disk on this server, I have 2 250GB Seagate 7.2K disks in RAID 1 mirroring. Is this under-spec'd?

I had another sysadmin look at Zimbra about 8 months ago and he commented how he thought Zimbra loggged too much, that it was always hammering at the disk especially mailbox.log. Is there a way to check to see if I have my logging set too high?

Thanks!

The top screenshot definitely shows your disk subsystem is the bottleneck.

In increasing costs order, I would:

First throw some RAM into the box to give you enough room for a RAM disk for Amavis' temp directory.

Second, get a PERC controller with 512MB of cache RAM and a battery-backed write cache.

If you have some $$$ leftover, put a pair of 300GB 15K 6G SAS disks in the box attached to the PERC as a RAID1 and migrate /opt over there. You can then remount /opt/zimbra/backup and even /opt/zimbra/hsm (or wherever your hsm volume is) on the old disks.

Will take some downtime of course!

Hope that helps,
Mark