Regarding DR: that's all we need to keep hold of our backups for, thankfully - not individual account restores. Things are physically separated so our Zimbra cluster bonfire won't turn into a backup server bonfire as well. In the not-so-distant future, the backup data will also be replicated to a secondary off site location.
Regarding size and speed of drives: everything is RAID 10.
Mailstore volumes: Seagate ES.2 SATA disks
Database / Index volumes: Seagate Cheetah 15K SAS
While Zimbra's guidelines do not recommend SATA disks in large deployments, with read ahead and write-back caching on a speedy SAN with good cache size and lots of disks, they can work fine. We have very large quotas so having SAS disks all around wasn't a viable option. The MTBF is high on the ES.2 disks, and with enough disks you can effectively offset the lesser IOPS on SATA - assuming the controller doesn't become your bottleneck. (RAID 10 is a big help)
Basically, I'm happy to give SAN-based large SATA arrays a thumbs up for mailstore storage, with enough good disks and appropriate caching settings.
Regarding NFS: my comments in the 2nd post, above, were the most useful items we found.
1) I neglected to add that 'async' mode over NFS also can give a speed improvement, assuming your NFS server doesn't have a big risk of going offline. That can be useful or dangerous, depending, so you'll have to make that call.
2) The big ticket items were using NFSv3 when using traditional backup due to the much better file creation speeds than NFSv4 for the millions of small files, and then using --zip and --zipStore to not have to create millions of small files but also not put the overhead of compression on the mailbox servers. That might work as well with NFSv4.
3) xfs was our filesystem of choice for the backup arrays, but note the 'nobarrier' mount option listed earlier - it made an enormous performance difference in our case.
Users per server: in the implementations, anywhere from ~5000 to 15,000. The quantity of messages flowing into a server and concurrent connections were the big deciding factors rather than quantity of users per server. Zimbra has a handy sizing spreadsheet available that you can enter various numbers into - incoming messages/user, outgoing messages/user, average message size, attachment size, quantity of users, concurrent users, and so on, and it gives recommendations on IOPS and drive quantities depending on your stats.  |