Some specs - didn't get to these earlier as it's been a busy day!
Software: ZCS 5.0.9
Servers: Dell PowerEdge 1950. 8-16GB of RAM, 1-2 quad-core processors. 2-4 drives.
Storage: SAN-based, combination of SAS and SATA disks (separate arrays, of course
). All arrays are RAID 10. Disk cache is enabled for some arrays, disabled for other arrays. The storage seems to be keeping up fine.
Load balanced incoming mail. Load balanced (by least connection) proxy servers for HTTPS, POPS, IMAPS. Using the nginx redirect for HTTP -> HTTPS. POP and IMAP requires TLS.
Backups are done via NFS to a backup server. There are a couple observations/sharing items on this, for greatly improving performance:
1) --zip and --zipStore was important for us. It doesn't compress the backups but drastically improves performance due to many fewer small files.
2) Despite the NFS server running NFSv4, it advertises v3 as well and we mount via v3. Some testing on our end showed a much faster file creation with this. It's probably less necessary with item 1 above but it still seemed to make a good difference.
3) Staggered backup schedules. Multiple mailbox servers are in this configuration (and our 22,500 configuration) so staggering their backup schedule should help keep the backup server happy.
4) backup server filesystem is xfs, and is on a SAN-based storage as well, which has write caching and readahead enabled. xfs generally has write barriers enabled - pass 'nobarrier' for the mount, and it greatly improves performance. No joke - close to 400% in our situation. The write cache was between 0-1% before but would get thoroughly utilized after, and it more than quadrupled the speed that data was written out and old backups were removed. (tested locally on the server to verify not an NFS issue - it was decidedly improved with nobarrier in the mount options)