Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: ZCS/Dovecot/Cyrus performance comparison

  1. #1
    jernejp is offline New Member
    Join Date
    Aug 2006
    Posts
    4
    Rep Power
    8

    Post ZCS/Dovecot/Cyrus performance comparison

    Hello,

    These are the results of my testing ZCS version 3.1 OS Edition on CentOS 4.3 (updated) on VMWare ESX server having 2 Xeon 2.8GHz, 3.5GB RAM, 100Mb network with vmware tools installed.

    I have tested ZCS vs. Dovecot vs. Cyrus by filling 10 users mailboxes with 2000 mails, where 1700 of these mails were small mails from qmail mailing list and the other 300 were jokes mails (usually from several kB to several MB). There were no special tuning of the system for any of the programs, because all of them were installed via RPM and I used Posftix as MTA for dovecot and Cyrus.

    Testset for IMAP testing was something like this: Courier IMAP test with 10 users. I created scripts in perl using Expect (scripts available if interested).
    Testset for POP3 testing were some perl scripts using Net::Pop3 module and with simple commands.
    Analysis was made with some scripts utilizing gnuplot and if you want detailed results (or scripts), PM me.

    The test was made from P4 machine quite close to the server and no other virtual machines were running on VMWare ESX server at the time of the tests.

    It turned out that Zimbra is quite slow compared to other two systems. We were discussing what might be the reason, but I think that only developers could tell us. At first glance we accused java for not being optimised for VMware, but there are no such explanation to be found on web. We also recorded the I/O state of the machine while in test, but the results are not analysed yet.

    So here are the results:
    IMAP tests: IMAP tests






    to be continued in next post...

  2. #2
    jernejp is offline New Member
    Join Date
    Aug 2006
    Posts
    4
    Rep Power
    8

    Default









    to be continued in next post...

  3. #3
    jernejp is offline New Member
    Join Date
    Aug 2006
    Posts
    4
    Rep Power
    8

    Default

    POP3 tests: POP3 tests









    Sorry for spamming but this is the only reasonable way to show the results and performance comparison with real data.

  4. #4
    Klug's Avatar
    Klug is offline Moderator
    Join Date
    Mar 2006
    Location
    Beaucaire, France
    Posts
    2,316
    Rep Power
    13

    Default

    Hello, thanks for the testing.

    I'm not sure Zimbra was designed to be a competitor of Dovecot, Cyrus or Courier-MTA as IMAP4/POP3 server.
    Just like if you were comparing Zimbra's webmail with Squirrelmail on email opening, moving, searching.

    My feeling is this kind of test might be very interesting for the ISP using Zimbra. But in these case, you're way beyong 10 users concurency, aren't you ?

  5. #5
    redhat's Avatar
    redhat is offline Active Member
    Join Date
    Aug 2006
    Posts
    35
    Rep Power
    8

    Default

    I don't understand the comments from Klug. Can you elaborate, or can someone else explain what he means?

    For example, Zimbra is designed for "state-of-the-art messaging and collaboration solutions" which includes IMAP and POP so it IS a "competitor" to the other IMAP/POP servers you list. I'd be surprised if the Zimbra developers were satisfied with having the slowest IMAP server on the market. I haven't seen anything that recommends an alternate IMAP server for large installations. (Is it even possible to use something else?)

    And the marketing literature specifically says "The Zimbra Collaboration Suite (ZCS) was designed to be deployed by service providers, such as carriers/operators, other broadband providers, and Internet service providers (ISPs)." so someone at Zimbra must think it's suitable for large installations.

    Also, what about SquirrelMail vs. Zimbra? Are you saying that Zimbra is going to be slower, less reliable, or what?

    I ask because I'm evaluating Zimbra for a large company right now and I'd really love some feedback. Much of what I see in these forums is coming from people with very small, or just test, installations.

  6. #6
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,487
    Rep Power
    56

    Default

    The problem with your question (and the tests, IMO) is that you're not comparing like with like. Zimbra is designed to be more than squirrelmail/dovecot/cyrus - it's a complete collaboration suite it's not a direct competitor for those products as they don't provide the same functionality as Zimbra. Those test would only be valid if the other products had the same feature set as zimbra.

    You choose a piece of software for the features it has and the problems it will solve in your environment.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  7. #7
    redhat's Avatar
    redhat is offline Active Member
    Join Date
    Aug 2006
    Posts
    35
    Rep Power
    8

    Default

    Understood, as far as Zimbra vs. SquirrelMail is concerned. The Zimbra user interface has more work to do so it's reasonable to assume it will be somewhat slower. The comments from Klug (not authoritative I assume) seemed to contradict the Zimbra marketing materials that say ZCS is suitable for large installations and ISPs.

    But we're really talking about a test of a very specific feature that's easy to isolate. You/we should be able to compare performance on requests sent directly to the IMAP/POP server from an external client. If Zimbra is significantly slower I'd be curious as to why and whether that indicates a real problem or simply an area for future improvements.

    I don't know if the test was conducted accurately or not. In my limited testing (not very scientific) Zimbra worked fine with every IMAP client. This test was just something I came across while trying to find out more about the IMAP/POP server being used. I didn't see anything that identified a specific open source server this was based on, so I was digging around.

    If IMAP/POP does turn out to be a bottleneck is it possible to substitute a different server to access the message store? (It looks like messages are not stored in a standard format.)

  8. #8
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,487
    Rep Power
    56

    Default

    Wel, I can't offer any comment on the code that surrounds the imap/pop functions so I don't know how well it performs. My other comment about the original test is that it's not mentioned how much memory was given to the VM that Zimbra was run in, Zimbra should really be tested on bare metal not a VM and the more memory it has the better it will perform.

    Zimbra is ready for large deployments, have you seen the press release section, specifically this article? It is in use at many ISPs and Hosted Service providers - they don't take decisions to install a critical product like Zimbra lightly.

    I don't believe that imap or pop are bottlenecks in Zimbra and the answer to your question is no, it's not possible to substitute another server for the Zimbra ones.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  9. #9
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,487
    Rep Power
    56

    Default

    I forgot to mention the benchmark information in the blogs here - you may find that more useful for details of performance in a large deployment.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  10. #10
    dkarp is offline Zimbra Employee
    Join Date
    Aug 2005
    Posts
    1,433
    Rep Power
    11

    Arrow IMAP/POP performance

    Well, yes and no. There really is a difference between writing an IMAP server and writing a server that exports an IMAP interface to its content. Your data storage, caching, and various optimizations are tuned to give you the most bang for your buck on all the interfaces you support; what's a no-brainer speedup for a simple IMAP server may not be a great idea for a server like Zimbra that has to field queries like "find all contacts and/or calendar appointments with tag FOO in any folder" on a regular basis.

    Which is not to say that we don't care about IMAP and POP performance. For instance, in 3.2 we've heavily optimized the features tested by your select1 and select2 IMAP tests (are they the same?). Multi-message STOREs/EXPUNGEs are also significantly faster. And, of course, our search results will continue to be very good, as that's one of the things that we're optimized around.

    The places where our numbers look worst at present are your fetch tests, and this is because the other servers are pre-calculating and caching message structure while we're doing it on the fly. We think we can fix this without too much of a tradeoff, so adding a message structure cache that works with our architecture and transaction system is high on the list for the next major release...
    Bugzilla - Wiki - Downloads - Before posting... Search!

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 11
    Last Post: 12-03-2006, 01:29 PM
  2. Zimbra Hardware performance and upgradation
    By mansuper in forum Administrators
    Replies: 8
    Last Post: 10-25-2006, 07:51 PM
  3. Performance, spam and lot more
    By montievv in forum Administrators
    Replies: 1
    Last Post: 07-24-2006, 11:24 AM
  4. ZCS Connector Performance
    By art_hebert in forum Zimbra Connector for Outlook
    Replies: 1
    Last Post: 03-06-2006, 06:33 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •