The backup server is the NFS server. Each mailbox server acts as an NFS client, and mounts an export from the backup server into each /opt/zimbra/backup, via NFS. There is tuning to be done on both the NFS server, and the clients (mailbox servers, and LDAP master, in this case).
NFS server
The tuning involved filesystem tuning & mount options, caching settings on the storage backend, quantity of nfsd that start (default is 8 which is nowhere near adequate for heavy loads), amount of memory available to nfsd via wmem and rmem values in /proc, sync or async on the exports, what versions of NFS are available, etc. There is some great info in /proc for tuning NFS - specifically the values regarding thread contention.
NFS clients - mailbox servers, LDAP master
The adjustments were limited to mount options (mounting with version 3 instead of 4, async if you want, wsize, rsize, noatime, etc) and changing the backups to do --zip and --zipStore to speed things up over NFS. Without --zip and --zipStore, mounting with nfsvers=3 doubled the speed. Some quick benchmarking showed that the file create speed was about twice as fast with NFSv3 than NFSv4, but the delete speed was slower. With --zip and --zipStore, using one version over the other may have lesser (or, zero) difference in performance.
Regarding multiple mailbox servers: each user account lives on a specific server. Each mailbox server has dedicated storage, on ext3, with a number of mke2fs options like the ones Zimbra recommends on the performance guide for large deployments. Each mailbox server also has dedicated faster storage for the MySQL database and Lucene indexes. To give all users a single point of login, the Zimbra Proxy package was used on multiple dedicated proxy servers. This eliminates the need for mailbox server-specific configuration, like having users A-H use server1, I-M on server2, and so on. The proxy servers do some other good things that are helpful in large deployments: SSL offloading which saves the mailbox servers some effort, memcached helps minimize quantity of content mailbox servers have to serve, and they spoon feed the connection data to users on slower connections. (Note that if you have good load balancers, you can often do the SSL offload on them and avoid having to load the certs and keys on the proxy server(s).)
Basically, what you end up with with the proxy servers behind load balancers is the ability to have one standard configuration for all users (which from a support standpoint is awesome), and a good way to take stress off of the mailbox servers and speed things up as well.
Regarding picture: I think we have a dia diagram available with a general overview of user access via pops, imaps, and https. I'll look in the next day or so and upload it.
Feel free to ask any/all questions, as well. We're happy to share.  |