Performance impressions?

[mikelcu]

I'm curious about real-world rollouts of Zimbra, how many people they're serving and on what kind of hardware. Of several systems I've eval'd recently, Zimbra is the nicest, but generates by far the heaviest load on the server (this doesn't mean it's actually high, just higher than the others were). Here is what we have running:

Zimbra 5.0.1 i686
P4 3.4gHz
2GB RAM
1 120GB SATA Disk

6 users, using Outlook, misc. IMAP, iSync, and web client
We often exaggerate our movements... i.e. move, copy or delete several thousand mails at once.

System load usually sits low at around 0.01 - 0.10
A large operation, even by only one user, can push it up to 0.50 - 0.70
and a mailbox import was 0.90+


-Mike

[Mark Daniel]

I'm curious about real-world rollouts of Zimbra, how many people they're serving and on what kind of hardware. Of several systems I've eval'd recently, Zimbra is the nicest, but generates by far the heaviest load on the server (this doesn't mean it's actually high, just higher than the others were). Here is what we have running:

Zimbra 5.0.1 i686
P4 3.4gHz
2GB RAM
1 120GB SATA Disk

6 users, using Outlook, misc. IMAP, iSync, and web client
We often exaggerate our movements... i.e. move, copy or delete several thousand mails at once.

System load usually sits low at around 0.01 - 0.10
A large operation, even by only one user, can push it up to 0.50 - 0.70
and a mailbox import was 0.90+


-Mike

Just for reference, I have Mac Pro (quad 2.66 xeon, 5gb RAM, Mac OS X Server 10.4.11, Zimbra 5.0.1 NE [just came from OSE 5 days ago]), with 2 mirrored 160GB SATA drives for the OS and Zimbra executables, and 2 mirrored 400GB SATAs for the mailstores and indexes (so, not a hardware RAID, and certainly no RAID5). 35 accounts, about half of which use IMAP (outlook, apple mail), the other half ZWC, with a mailstore of 40GB (and growing pretty quick). The machine used to only have 2GB RAM, and even then, at the peak of the day, that server hardly showed a *dent* in processor usage. Users report a very responsive experience. In contrast, I have another 35 users on another server using the Mac OS X Server cyrus-based mail server, and that thing fluctuates from 30-80% usage on both cores all throughout the day.

One thing I've noticed, though, is that performing a large operation with an account does seize things up a bit... but only for that account. While doing migrations, I was actually having a few users access their accounts while their old mailstores were being migrated via imapsync on another machine, and while other people noticed nothing unusual/no slowdowns, the user who's mail was being imported certainly noticed, and a single core (just one) on the machine would shoot up to 97-99% usage. Again, though, this did not affect performance for anything *but that account.*

[zen]

We are serving 500+ users at the moment, pure IMAP (no POP3 access). Most users through IMAP, about 10-20% through IMAPS. We do not use web interface extensively, mostly relying on standalone clients
We have separate LDAP server, remaining services sitting on another machine. They are both 8 core Intel(R) Xeon(R) CPU E5320@1.86GHz with 8GiB of memory. Load is usually at 2.5 - 3.0, but we are suffering from memory consumption problems. We have 600 IMAP threads running, while we should really increase it to 1200 (at least) (some users are experiencing long response times)

[mikelcu]

My experience sounds a bit like zen's (though on a much much smaller scale). For the most part, everything runs fine but I am getting some blips here and there (mostly mailboxd) that concern me.

For example, my co-worker just attempted to upload a large (100MB) file to his briefcase. The upload failed with a generic message, and on the backend, the java mailbox process spiked to 101% CPU for about 5-6 seconds. Individually, this is not a big deal but what happens when I unleash a couple hundred users on the system....

Also, we'll be using IMAPS/POPS exclusively. Anyone have any medium-large rollouts using SSL only?

-Mike

[void]

Just to toss in my 2 cents, I am running on a single CPU dual core xeon, Dell PowerEdge 1950.

Zimbra is the servers only task. We have 45 people, about 4 use the web interface, the rest use fat clients like Outlook or Thunderbird.

My system load average is always up around 2, no free memory.

Zimbra is a big slow (horribly documented) resource hog compared to just running plain old Sendmail or Postfix. But with all the features it has in it that make life easier, why would you want to?

[mikelcu]

Just to toss in my 2 cents, I am running on a single CPU dual core xeon, Dell PowerEdge 1950.

Zimbra is the servers only task. We have 45 people, about 4 use the web interface, the rest use fat clients like Outlook or Thunderbird.

My system load average is always up around 2, no free memory.

Zimbra is a big slow (horribly documented) resource hog compared to just running plain old Sendmail or Postfix. But with all the features it has in it that make life easier, why would you want to?

Well, some of it is written in Java I figure that given enough hardware, any software will run well enough. And like you say, I think the feature set outweighs potential performance issues... to a point. However, it _can_ get bad enough to outweigh the benefits, and 45 users crushing a PE 1950 is getting close.

Have you tried performance tuning of any kind?

-Mike

[peng1can]

I think a *lot* of zimbra performance has to do with very careful tuning that isn't entirely documented in the performance guide yet. We have 7500 users on two Dual-CPU, Dual Core Xeon 5140 Blades with 8GB of RAM on a DS4400 SAN, one for Mailbox/LDAP and the other for MTA. On a typical day, we have 300 web sessions and 300 IMAP sessions open at a time.

- When we had only 4GB of RAM we had *major* load issues.
- Before we bumped the IMAP thread count up to 4x the number of IMAP users and the http/https thread count (not documented in the performance guide) to more than the number of web users, we had significant load issues (never less than 6, more like 8-10).
- If we make the mistake of letting a backup run long enough that it overlaps with part of our work day, we have load issues (10-12).
- If someone sends an email to one of our campus-wide mailing lists (managed externally in Sympa), we have major load issues (22+). If 2 or 3 hit us at the same time, we're just screwed.

Otherwise, our load can actually be down as low as .4 in the middle of the day, which is pretty good for a 4-core box. If you add up our old cyrus+amavis+Horde installs, I'm sure they used more CPU than that.

It does bug me at times that the server is one big monolithic black box of java, which makes it very tough for someone like me to tune. But, I've gotten some decent tuning help from some of the Zimbra techs and hopefully that info will eventually make it's way into the performance guides.

[mikelcu]

peng1can:

That's really good info, and sort of what I was hoping. I deal with lots of big fat java apps, and I've often seen little tweaks make a huge difference in performance.

Does this summary sound accurate? (I'm just making up numbers that sound close to what's been described)

• RAM is cheap, don't skimp - (i'm guessing here) perhaps 4GB+ for < 1000 users, 8GB+ for > 1000
• IMAP Threads - 4x # of users
• HTTP/HTTPS Threads - 1.5x # of users (more?)
• Split functions - Put Mailbox on separate machine from MTA
• Backups - don't run backups during peak times (what happens if your peak time is 24/7?)
• Beware mass mailing

I think I would add disk I/O as well... if you can't split services onto separate machines, at least split functions on to separate disks, controllers, interfaces, etc. depending on what your storage backend is like.

peng's issue with sympa concerns me some, seems like a DOS would be a little too easy. Running lists inside zimbra would probably help a lot, but that may not always be an option (I use mailman a lot). I wonder if there is a way to mitigate the effect of external mailing list floods.

-Mike

[peng1can]

A few clarifications...

- I don't know if our RAM usage was related to the number of accounts, or the number of threads/sessions. See my previous post for what we have on both fronts, but we were swapping at 4GB and have about 1GB sitting free now. Recently we were advised to kick our MySQL innodb cache up to 1GB, so there's a huge chunk of memory right there. We're also on RHEL_5.1 x86_64.

- I wasn't given much guidance on the web threads issue (in fact, our tech seemed surprised that increasing that setting helped as much as it did), but the initial count was lower than our number of active sessions, and the load dropped when we kicked that number to something higher than the number of sessions. Maybe 2 or 3 times would be even better. I don't know.

- Every tech that I've asked about splitting the mailbox server into two has disagreed with that approach for my environment. If I add a 3rd box to the mix, they've all said to split LDAP from the mailbox server, but leave the mailboxes on one server. I'm not sure how big you have to get to want multiple mailbox servers.

- The MTA server runs at about .5 load all the time, even with razor, bayes, dspam, and rules_du_jour modifications to amavis/SA.

- In fairness, our backups run to an NFS mounted disk. Obviously that increases latency. The new backup grouping may help with the timing problem.

- Sympa & Mailman don't seem to be designed for large internal lists. Both systems have always caused us to DoS ourselves, and I've never found a good balance for this, whether I put 500 recipients per message or 5. This was just as true for cyrus and courier as it's been for Zimbra. I wouldn't care if it takes longer for those mailing list messages to be delivered... I'd rather have the web/interfaces remain responsive during those times rather than throw every last CPU & disk cycle into delivering the mail. One of these days I'm going to test a campus-wide mailing through zimbra and see how it performs.

Overall, Zimbra can be tuned to be very scalable. You just have to spend some time doing it. And for whatever reason, many of our tweaks disappeared in the 4.5->5.0 upgrade, so watch out for that, too.