22,000 mailboxes - recommended configuration?

[emacias]

Hi LMStone:

We want to install Zimbra using aprox. 22 thounsand mailboxes with quotes from 300 MB and 1 GB. Our infraestructure has Blades Servers and NetApp's SAN. Blade have one disk with 146 GB but by security the most logic is to use SAN for installing /opt/zimbra.

Layout Disk:

The mailbox server includes the following volumes:
• Message Store. Mail message files are in opt/zimbra/store
• Data Store. The MySQL Database files are in opt/zimbra/db
• Index Store. Index files are in opt/zimbra/index
• Backup Area. Full and incremental backups are in opt/zimbra/backup
• Log files. Each component in the Zimbra Collaboration Suite has log files. Local logs are in /opt/zimbra/log

By these 22.000 mailboxes what's your recomendation:

Create one LUN with all directory /opt/zimbra?, create LUN for each volumen or other option?

Thanks for your help!

[Krishopper]

Hello,

We have it set up such that the entire /opt/zimbra is on the SAN. That way if we lose an entire blade, we can easily re-map those LUN's to another blade, mount up the entire /opt/zimbra volume, and quickly get back online without much hassle.

I'm not sure if that's the best way to do it, or if anyone else is doing it any different, but it works quite well here.

[Klug]

If you go the volume way, add a volume for redologs.

[LMStone]

Originally Posted by emacias

Hi LMStone:

We want to install Zimbra using aprox. 22 thounsand mailboxes with quotes from 300 MB and 1 GB. Our infraestructure has Blades Servers and NetApp's SAN. Blade have one disk with 146 GB but by security the most logic is to use SAN for installing /opt/zimbra.

Layout Disk:

The mailbox server includes the following volumes:
• Message Store. Mail message files are in opt/zimbra/store
• Data Store. The MySQL Database files are in opt/zimbra/db
• Index Store. Index files are in opt/zimbra/index
• Backup Area. Full and incremental backups are in opt/zimbra/backup
• Log files. Each component in the Zimbra Collaboration Suite has log files. Local logs are in /opt/zimbra/log

By these 22.000 mailboxes what's your recomendation:

Create one LUN with all directory /opt/zimbra?, create LUN for each volumen or other option?

Thanks for your help!

How many separate LUNs to create in our experience is really dependent on the quality of the SAN. If you have something like an EMC CX4 or better, there is so much cache in the SAN that the underlying disk configurations become less of a bottleneck and you can get by with fewer LUNs on fewer separate arrays.

One client with a big virtualized LAMP stack had a runaway MySL server writing 1GB bin-log files every three minutes to the CX4, and no one noticed any performance degradation at all (Nagios let us know the bin-log partition was getting filled though...) Very few LUNs here and only two underlying arrays.

We have recently come across JetStor iSCSI SANs; these are basically JBOD shelves fitted with a SAN controller. The customer uses this SAN for SAS files (big, big, big data files and temporary working sets), but the performance is about on par with DASD, so you need more LUNs on separate arrays.

I would however try to save some money by using a RAID6 array for /opt/zimbra/backup (with bigger, slower drives) and one or more RAID10 arrays for everything else, with faster drives.

I don't like single drives in blade servers BTW; if you are comfortable installing the operating system (SLES!?) on a SAN mount I would do so. FWIW the very first generation of IBM blade servers had no on-board disks whatsoever, but too many customers couldn't get their heads around having everything on the SAN. Now, we build a lot of ESXi servers with ESXi on an internal 4GB USB stick and use all the on-board disks for the datastores. Go figure...

So, which Netapp are you using? And what features have you licensed? That will help me to be more specific.

All the best,
Mark

[emacias]

We actually have FAS3020 with protocol FCP or IScsi, By the side Blade, we use CentOS, ZCS totally Open Source free, with connection to SAN by FC 4 Gb multipath.

We have discs trays's NetApp of High performance and discs trays of high capacity

[LMStone]

NetApp have an Exchange 2003 deployment whitepaper that incorporates their best practices regarding LUNs and other configuration options on a FAS3040 with 14,000 Exchange mailboxes in a four-server configuration.

http://media.netapp.com/documents/Ne...3040-fcsan.pdf

It's interesting that in the doc above, everything is DP (RAID6), not RAID10. But, that's what you get with huge cache on the SAN.

Hope that helps,
Mark

[leog21]

Hi Emacias,

Can you please update us on what config you went with and how its all working out?