Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: 49,700 account deployment

  1. #1
    nate.olmstead is offline Member
    Join Date
    Sep 2008
    Location
    United States
    Posts
    10
    Rep Power
    6

    Default 49,700 account deployment

    Same type of setup as Anthony's previous thread, just 49,700 vs. 22,500 this time around.

    22,500!

    Things are rolling smoothly thus far!

  2. #2
    nate.olmstead is offline Member
    Join Date
    Sep 2008
    Location
    United States
    Posts
    10
    Rep Power
    6

    Default

    Some specs - didn't get to these earlier as it's been a busy day!

    Software: ZCS 5.0.9
    Servers: Dell PowerEdge 1950. 8-16GB of RAM, 1-2 quad-core processors. 2-4 drives.
    Storage: SAN-based, combination of SAS and SATA disks (separate arrays, of course ). All arrays are RAID 10. Disk cache is enabled for some arrays, disabled for other arrays. The storage seems to be keeping up fine.

    Load balanced incoming mail. Load balanced (by least connection) proxy servers for HTTPS, POPS, IMAPS. Using the nginx redirect for HTTP -> HTTPS. POP and IMAP requires TLS.

    Backups are done via NFS to a backup server. There are a couple observations/sharing items on this, for greatly improving performance:
    1) --zip and --zipStore was important for us. It doesn't compress the backups but drastically improves performance due to many fewer small files.
    2) Despite the NFS server running NFSv4, it advertises v3 as well and we mount via v3. Some testing on our end showed a much faster file creation with this. It's probably less necessary with item 1 above but it still seemed to make a good difference.
    3) Staggered backup schedules. Multiple mailbox servers are in this configuration (and our 22,500 configuration) so staggering their backup schedule should help keep the backup server happy.
    4) backup server filesystem is xfs, and is on a SAN-based storage as well, which has write caching and readahead enabled. xfs generally has write barriers enabled - pass 'nobarrier' for the mount, and it greatly improves performance. No joke - close to 400% in our situation. The write cache was between 0-1% before but would get thoroughly utilized after, and it more than quadrupled the speed that data was written out and old backups were removed. (tested locally on the server to verify not an NFS issue - it was decidedly improved with nobarrier in the mount options)
    Last edited by nate.olmstead; 10-03-2008 at 11:25 AM.

  3. #3
    Dirk's Avatar
    Dirk is offline Moderator
    Join Date
    May 2006
    Location
    England.
    Posts
    927
    Rep Power
    10

    Default

    Outstanding! That's a fair few users

    What's your disaster recovery policy, if I may ask? I mean, what would happen if the server room where your Zimbra box(es) are stored burns down?

  4. #4
    mesnyder is offline New Member
    Join Date
    May 2007
    Posts
    3
    Rep Power
    8

    Default

    Quote Originally Posted by nate.olmstead View Post
    Some specs - didn't get to these earlier as it's been a busy day!

    Software: ZCS 5.0.9
    Servers: Dell PowerEdge 1950. 8-16GB of RAM, 1-2 quad-core processors. 2-4 drives.
    Storage: SAN-based, combination of SAS and SATA disks (separate arrays, of course ). All arrays are RAID 10. Disk cache is enabled for some arrays, disabled for other arrays. The storage seems to be keeping up fine.

    Load balanced incoming mail. Load balanced (by least connection) proxy servers for HTTPS, POPS, IMAPS. Using the nginx redirect for HTTP -> HTTPS. POP and IMAP requires TLS.

    Backups are done via NFS to a backup server. There are a couple observations/sharing items on this, for greatly improving performance:
    1) --zip and --zipStore was important for us. It doesn't compress the backups but drastically improves performance due to many fewer small files.
    2) Despite the NFS server running NFSv4, it advertises v3 as well and we mount via v3. Some testing on our end showed a much faster file creation with this. It's probably less necessary with item 1 above but it still seemed to make a good difference.
    3) Staggered backup schedules. Multiple mailbox servers are in this configuration (and our 22,500 configuration) so staggering their backup schedule should help keep the backup server happy.
    4) backup server filesystem is xfs, and is on a SAN-based storage as well, which has write caching and readahead enabled. xfs generally has write barriers enabled - pass 'nobarrier' for the mount, and it greatly improves performance. No joke - close to 400% in our situation. The write cache was between 0-1% before but would get thoroughly utilized after, and it more than quadrupled the speed that data was written out and old backups were removed. (tested locally on the server to verify not an NFS issue - it was decidedly improved with nobarrier in the mount options)
    We've got a fairly small deployment now, less than 600, but planning on growing to several thousand. What size and speed drives are you using for your primary storage on the servers? About how many users are you running per server? Any issues or gotchas using NFS for backups? Thanks!

  5. #5
    nate.olmstead is offline Member
    Join Date
    Sep 2008
    Location
    United States
    Posts
    10
    Rep Power
    6

    Default

    Regarding DR: that's all we need to keep hold of our backups for, thankfully - not individual account restores. Things are physically separated so our Zimbra cluster bonfire won't turn into a backup server bonfire as well. In the not-so-distant future, the backup data will also be replicated to a secondary off site location.

    Regarding size and speed of drives: everything is RAID 10.
    Mailstore volumes: Seagate ES.2 SATA disks
    Database / Index volumes: Seagate Cheetah 15K SAS

    While Zimbra's guidelines do not recommend SATA disks in large deployments, with read ahead and write-back caching on a speedy SAN with good cache size and lots of disks, they can work fine. We have very large quotas so having SAS disks all around wasn't a viable option. The MTBF is high on the ES.2 disks, and with enough disks you can effectively offset the lesser IOPS on SATA - assuming the controller doesn't become your bottleneck. (RAID 10 is a big help)

    Basically, I'm happy to give SAN-based large SATA arrays a thumbs up for mailstore storage, with enough good disks and appropriate caching settings.

    Regarding NFS: my comments in the 2nd post, above, were the most useful items we found.
    1) I neglected to add that 'async' mode over NFS also can give a speed improvement, assuming your NFS server doesn't have a big risk of going offline. That can be useful or dangerous, depending, so you'll have to make that call.
    2) The big ticket items were using NFSv3 when using traditional backup due to the much better file creation speeds than NFSv4 for the millions of small files, and then using --zip and --zipStore to not have to create millions of small files but also not put the overhead of compression on the mailbox servers. That might work as well with NFSv4.
    3) xfs was our filesystem of choice for the backup arrays, but note the 'nobarrier' mount option listed earlier - it made an enormous performance difference in our case.

    Users per server: in the implementations, anywhere from ~5000 to 15,000. The quantity of messages flowing into a server and concurrent connections were the big deciding factors rather than quantity of users per server. Zimbra has a handy sizing spreadsheet available that you can enter various numbers into - incoming messages/user, outgoing messages/user, average message size, attachment size, quantity of users, concurrent users, and so on, and it gives recommendations on IOPS and drive quantities depending on your stats.

  6. #6
    PeterSilva is offline New Member
    Join Date
    Oct 2008
    Posts
    3
    Rep Power
    6

    Default

    This case is extremely interesting, (in English: COOL!) Real data from a real deployment, and not a sales person. So forgive me if I get really curious!

    First thing, there is NFS in the story, and the NFS server is running XFS. So far so good, but it isn´t clear whether the mailbox server or the backup server is the NFS server.

    Second thing, not clear on what multiple mailbox servers means... Does that mean that each mailbox server has a local XFS file system, and that requests for a group of mailboxes are somehow directed to that mailbox server? Are there different URL´s for different subsets of users? Or... is it that there are multiple NFS mount points for all the mailbox groupings, and that all the zimbra servers mount all the mailboxes, and share and access them all using NFS read/write?

    or...

    Is there a picture available?
    I´ll even volunteer to draw one in Dia if I can get a good enough picture...

  7. #7
    nate.olmstead is offline Member
    Join Date
    Sep 2008
    Location
    United States
    Posts
    10
    Rep Power
    6

    Default

    The backup server is the NFS server. Each mailbox server acts as an NFS client, and mounts an export from the backup server into each /opt/zimbra/backup, via NFS. There is tuning to be done on both the NFS server, and the clients (mailbox servers, and LDAP master, in this case).

    NFS server
    The tuning involved filesystem tuning & mount options, caching settings on the storage backend, quantity of nfsd that start (default is 8 which is nowhere near adequate for heavy loads), amount of memory available to nfsd via wmem and rmem values in /proc, sync or async on the exports, what versions of NFS are available, etc. There is some great info in /proc for tuning NFS - specifically the values regarding thread contention.

    NFS clients - mailbox servers, LDAP master
    The adjustments were limited to mount options (mounting with version 3 instead of 4, async if you want, wsize, rsize, noatime, etc) and changing the backups to do --zip and --zipStore to speed things up over NFS. Without --zip and --zipStore, mounting with nfsvers=3 doubled the speed. Some quick benchmarking showed that the file create speed was about twice as fast with NFSv3 than NFSv4, but the delete speed was slower. With --zip and --zipStore, using one version over the other may have lesser (or, zero) difference in performance.

    Regarding multiple mailbox servers: each user account lives on a specific server. Each mailbox server has dedicated storage, on ext3, with a number of mke2fs options like the ones Zimbra recommends on the performance guide for large deployments. Each mailbox server also has dedicated faster storage for the MySQL database and Lucene indexes. To give all users a single point of login, the Zimbra Proxy package was used on multiple dedicated proxy servers. This eliminates the need for mailbox server-specific configuration, like having users A-H use server1, I-M on server2, and so on. The proxy servers do some other good things that are helpful in large deployments: SSL offloading which saves the mailbox servers some effort, memcached helps minimize quantity of content mailbox servers have to serve, and they spoon feed the connection data to users on slower connections. (Note that if you have good load balancers, you can often do the SSL offload on them and avoid having to load the certs and keys on the proxy server(s).)

    Basically, what you end up with with the proxy servers behind load balancers is the ability to have one standard configuration for all users (which from a support standpoint is awesome), and a good way to take stress off of the mailbox servers and speed things up as well.

    Regarding picture: I think we have a dia diagram available with a general overview of user access via pops, imaps, and https. I'll look in the next day or so and upload it.

    Feel free to ask any/all questions, as well. We're happy to share.

  8. #8
    PeterSilva is offline New Member
    Join Date
    Oct 2008
    Posts
    3
    Rep Power
    6

    Default individual mailbox failover...

    OK so you are using physically local disk on each mailbox server, or is there a SAN involved?

    If a mailbox server dies, is there some sort of failover to a second one?
    I´m thinking about a high-availability deployment, so would imagine, say, using a SAN, and having the ´service´ for those mailboxes failover to a second server, which would e2fsck & mount the disks if the active server died.

  9. #9
    Klug's Avatar
    Klug is offline Moderator
    Join Date
    Mar 2006
    Location
    Beaucaire, France
    Posts
    2,320
    Rep Power
    13

    Default

    Quote Originally Posted by PeterSilva View Post
    If a mailbox server dies, is there some sort of failover to a second one?
    I´m thinking about a high-availability deployment, so would imagine, say, using a SAN, and having the ´service´ for those mailboxes failover to a second server, which would e2fsck & mount the disks if the active server died.
    There are big deployments like this one using Xen VMs hosted on SAN : if your hardware node fails, just relocate the VM elsewhere.

  10. #10
    Krishopper is offline Dedicated Member
    Join Date
    Dec 2006
    Location
    Minneapolis MN
    Posts
    777
    Rep Power
    9

    Default

    I don't have a huge deployment but mine is also running in Xen, for the ease of transferring the VM from one machine to another in the event of hardware failure.

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 43
    Last Post: 09-18-2013, 05:13 AM
  2. One account not receving email
    By EnglishDude in forum Administrators
    Replies: 12
    Last Post: 04-30-2010, 06:19 AM
  3. Test Delete Zimbra account coding
    By fsloke in forum Developers
    Replies: 3
    Last Post: 11-14-2008, 08:08 AM
  4. Compartmentalized groups or locations sharing a domain
    By dstoliker in forum Administrators
    Replies: 5
    Last Post: 07-14-2008, 12:06 PM
  5. Replies: 3
    Last Post: 09-18-2007, 06:55 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •