Some planning questions

[Olorin]

I am planning to deploy a ZCS machine in a remote DC on colocation.
Using the latest Community edition
Hardware:
2x SAS disks for OS in raid1,
8x SAS disks in raid10 for Zimbra,
32Gb RAM
8 CPU cores

Requirements:
- Normal system functionality
- Everyting must be encoded (currently considering the standard LUKS)
- Backup to another machine in the same DC, not using 3rd party software
- The possibility of a fast restore to another machine in case of failure. Replication to a standby machine would be even better

Questions:
- Any comments on the hardware?
- How much of a performance hit would the encryption mean?
- Which of the existing backup scripts is preferable? I need the possibility to restore up to 1-2 months back, weekly full, and daily incrementals would be grand. Brick level - extra plus
- What other DR options are available? What about HA?
- How is the installation done in this case? Simply use /opt as the mountpoint for the raid10, or..?
- Which of the supported OS is the most stable and easy to maintain with a large number of mailboxes? I am kind of disappointed in CentOS, anyone aware of how SL performs?

Any additional comments or anything I might have missed also welcome

[cyberdeath]

Originally Posted by Olorin

Questions:
- Any comments on the hardware?

Hardware seems fine. From my experience, the biggest thing you need with Zimbra (or any MTA for that matter) is RAM...and more RAM. The second thing would probably be Disk I/O since processor power isn't too hard to come by these days.

You are making a good choice for the OS RAID & the Zimbra RAID. For larger deployments, I would say, if you can afford it (in disk space and cost), go for the RAID 10 as it outperforms RAID6, even though RAID6 is cheaper in terms of monetary cost per GB.

Originally Posted by Olorin

- How much of a performance hit would the encryption mean?

Encryption is always going to place an impact on your overall I/O. The trick is in mitigation. You have done well thus far by running the RAID10 for performance. The only other thing you could do would be to look into hardware RAID setups (which tend to be costly and are not as "standardized" as linux is...which also applies for software RAID as well).

I would say your best bet would be to do the following:
Go ahead and install the OS you plan to use with encryption enabled on a box. Then, run a dd test (or some other variant) to see how well reading and writing does on the encrypted volumes. Then, of course, you'll need to install the same OS on a device WITHOUT encryption to see how well it does in comparison. That will give you a real rough estimate of how well your encryption algorithm is going to perform. There are also some great case studies and tests out on the web so you don't have to go through the effort...I would check that out as well.

As far as personal experience...encryption does hurt...quite a bit on I/O. However, it all depends on how big your need is. Yes, it may be painful doing a backup or restore....but let's just hope you never have to do one...and also that you proactively keep a standby box stood up and ready to take over should it happen.

Originally Posted by Olorin

- Which of the existing backup scripts is preferable? I need the possibility to restore up to 1-2 months back, weekly full, and daily incrementals would be grand. Brick level - extra plus

If you're looking for block level, you're likely looking for LVM. There's a lot of controversy on this because some say that LVM causes a performance hit while others disagree. I personally do not know because I've never compared the two side-by-side. However, LVM *will* let you do snapshots at the filesystem level. The only caveat is the amount of disk space necessary to keep all of these snapshots...are you planning on adding a third level of low performance storage drives to keep them on? I would strongly encourage it if you go this route and need to keep many months of data. As you well know... 1-2 months is good...but what happens when your CFO or CEO needs 6 months back?

After speaking with a Zimbra engineer about the backups, my understanding is the best (or at least the lesser complicated) way would be to simply use the backups generated by the primary mail server and copying it over to a spare Zimbra server. That way it's just a matter of recovering and you're up. You can even setup the spare to do this automatically as it receives the file (through scripting, of course).

Assuming that both machines are identical:
I would personally setup your primary server how you want it then disk duplicate to the other server (Through whatever means you prefer). Then you'll need to change a couple things (IP address...etc) and then you'll want to setup the script to run and backup your data. (Example: Backing up and restoring Zimbra (Open Source Version) - Zimbra :: Wiki).

Please note that this method assumes you do not mind losing some of the day's data (based upon the frequency of your interval backups...which I believe is default at 24h).

Originally Posted by Olorin

- What other DR options are available? What about HA?

Using a SAN and disk replication would assuredly be the easiest from a system administrators standpoint. This can be setup however you want to and as frequently...with all the snapshots you want...etc etc. The only catch is the amount of money it will cost to implement.

As far as HA is concerned...I have heard a lot said about DRBD. The only catch is you need a fast network interface between the two devices to successfully make it happen (such as the crossover Cat5e or better cables connecting to a NIC on each server). A good article about this topic can be found here: Ajcody-Notes-HA-Linux-How-To - Zimbra :: Wiki

Originally Posted by Olorin

- How is the installation done in this case? Simply use /opt as the mountpoint for the raid10, or..?

Since you are running a RAID1 for the other partition (/), mounting /opt would likely suffice since the mailstores and the majority of the config is underneath that directory.

Originally Posted by Olorin

- Which of the supported OS is the most stable and easy to maintain with a large number of mailboxes? I am kind of disappointed in CentOS, anyone aware of how SL performs?

I have personally used SLES for two reasons:
1. It seems to perform really well and is stable.
2. It is supported by Novell for a modest licensing fee.

However, I'm sure that RedHat and others are just fine as well. Just a personal preference.

Originally Posted by Olorin

Any additional comments or anything I might have missed also welcome

I believe you are going down the right track. However, you never said how many domains and mailboxes you plan to have on this machine. You also did not mention the size of the SAS drives and their performance levels (SATA, SCSI... 7.2k, 10k, 15k, etc...). By providing this information, it will help us better assist you and at least warn you of any pitfalls with your specific configuration (since your post is rather vague).

NOTE: One final page for great reading is: http://wiki.zimbra.com/wiki/Performa...ge_Deployments

Hope this helps!

[LMStone]

Originally Posted by Olorin

I am planning to deploy a ZCS machine in a remote DC on colocation.
Using the latest Community edition
Hardware:
2x SAS disks for OS in raid1,
8x SAS disks in raid10 for Zimbra,
32Gb RAM
8 CPU cores

Requirements:
- Normal system functionality
- Everyting must be encoded (currently considering the standard LUKS)
- Backup to another machine in the same DC, not using 3rd party software
- The possibility of a fast restore to another machine in case of failure. Replication to a standby machine would be even better

Questions:
- Any comments on the hardware?
- How much of a performance hit would the encryption mean?
- Which of the existing backup scripts is preferable? I need the possibility to restore up to 1-2 months back, weekly full, and daily incrementals would be grand. Brick level - extra plus
- What other DR options are available? What about HA?
- How is the installation done in this case? Simply use /opt as the mountpoint for the raid10, or..?
- Which of the supported OS is the most stable and easy to maintain with a large number of mailboxes? I am kind of disappointed in CentOS, anyone aware of how SL performs?

Any additional comments or anything I might have missed also welcome

Why must everything be encoded?

[Olorin]

Originally Posted by LMStone

Why must everything be encoded?

The machine will be hosted in one of the less stable ex-USSR countries, where the authorities don't need a real reason to expropriate hardware. Nothing illegal, just email traffic, but encryption is a requirement.

[Olorin]

First of all thank you for the concise and thorough reply, I really appreciate your time and effort doing this

Originally Posted by cyberdeath

Hardware seems fine. From my experience, the biggest thing you need with Zimbra (or any MTA for that matter) is RAM...and more RAM. The second thing would probably be Disk I/O since processor power isn't too hard to come by these days.

I was under the impression 32Gb should be quite enough. If I am wrong, I still have time to spec up, but I'd really love to see some sort of a calculation table or something that would justify even more RAM. As for disk IO, I do have some fears there - 8 disks in raid10 are normally quite enough, but with encryption thrown in the middle... I just don't know - never tried it before.

You are making a good choice for the OS RAID & the Zimbra RAID. For larger deployments, I would say, if you can afford it (in disk space and cost), go for the RAID 10 as it outperforms RAID6, even though RAID6 is cheaper in terms of monetary cost per GB.

The plan is raid1 for the OS plus a raid10 for Zimbra. I do wonder whether or not /opt can be local and unencrypted, and I can simply mount a separate raid10 based chunk for the mailstore separately.

Encryption is always going to place an impact on your overall I/O. The trick is in mitigation. You have done well thus far by running the RAID10 for performance. The only other thing you could do would be to look into hardware RAID setups (which tend to be costly and are not as "standardized" as linux is...which also applies for software RAID as well).

This is going to be a brand machine (probably a Dell or HP), so I'll be using a proper raid controller, with a BBU and the max amount of cache I can get for it

I would say your best bet would be to do the following:
Go ahead and install the OS you plan to use with encryption enabled on a box. Then, run a dd test (or some other variant) to see how well reading and writing does on the encrypted volumes. Then, of course, you'll need to install the same OS on a device WITHOUT encryption to see how well it does in comparison. That will give you a real rough estimate of how well your encryption algorithm is going to perform. There are also some great case studies and tests out on the web so you don't have to go through the effort...I would check that out as well.

I don't have the option to play with it unfortunately, once the hardware is spec'd out and in place, I have to bring it all up into production. I've googled a LOT looking for zimbra on LUKS use cases and benchmarks, but couldn't find anything. Seems like I'm the first to do that.

As far as personal experience...encryption does hurt...quite a bit on I/O. However, it all depends on how big your need is. Yes, it may be painful doing a backup or restore....but let's just hope you never have to do one...and also that you proactively keep a standby box stood up and ready to take over should it happen.

There's another point, I do plan to have backups done, and with Zimbra, if I back up /opt restore should be easy. However, for that sort of backup, I'll need to stop zimbra services, and I'd rather not do that. Are there any guides that show how to keep the stdby box sync'd, so real HA can work? I'm pretty sure there are ways to do that with the costlier versions of Zimbra, but I'd rather stick to the OSS builds.

If you're looking for block level, you're likely looking for LVM. There's a lot of controversy on this because some say that LVM causes a performance hit while others disagree. I personally do not know because I've never compared the two side-by-side. However, LVM *will* let you do snapshots at the filesystem level. The only caveat is the amount of disk space necessary to keep all of these snapshots...are you planning on adding a third level of low performance storage drives to keep them on? I would strongly encourage it if you go this route and need to keep many months of data. As you well know... 1-2 months is good...but what happens when your CFO or CEO needs 6 months back?

If the CEO wants more than two months, he will approve the budget for a better storage solution In this case however, I'm not looking for block level, but for _brick_ level (guess I got the term for MSExchange). Meaning, I can restore separate mailboxes in case of need, instead of pulling in the entire setup, which might end up being quite the monster.

The backups are going to be held in another server, hosted elsewhere

After speaking with a Zimbra engineer about the backups, my understanding is the best (or at least the lesser complicated) way would be to simply use the backups generated by the primary mail server and copying it over to a spare Zimbra server. That way it's just a matter of recovering and you're up. You can even setup the spare to do this automatically as it receives the file (through scripting, of course).

What kind of scripting? a simple rsync to sync up /opt or...? Being an old sysadmin, I'm rather professionally lazy, so I really hate reinventing wheels, especially when there's a chance of my new wheel turning out to be square.

Assuming that both machines are identical:
I would personally setup your primary server how you want it then disk duplicate to the other server (Through whatever means you prefer). Then you'll need to change a couple things (IP address...etc) and then you'll want to setup the script to run and backup your data. (Example: Backing up and restoring Zimbra (Open Source Version) - Zimbra :: Wiki).

Identical, hardware-wise? The stdby machine is going to be a VM probably, can't afford to have another large server up there. I could keep a zimbra setup on the backup server though, with the services stopped of course. Not sure how all the SSL certs will behave if the setup needs to failover though...

Please note that this method assumes you do not mind losing some of the day's data (based upon the frequency of your interval backups...which I believe is default at 24h).

Yup, no biggie

Using a SAN and disk replication would assuredly be the easiest from a system administrators standpoint. This can be setup however you want to and as frequently...with all the snapshots you want...etc etc. The only catch is the amount of money it will cost to implement.

It's more about rack space in this case. Can't put a SAN in there unfortunately. Besides, failover, as such, should happen to another geo-location, so a local SAN won't help, and implementing iFCP based solutions for a single MTA is really overkill

As far as HA is concerned...I have heard a lot said about DRBD. The only catch is you need a fast network interface between the two devices to successfully make it happen (such as the crossover Cat5e or better cables connecting to a NIC on each server). A good article about this topic can be found here: Ajcody-Notes-HA-Linux-How-To - Zimbra :: Wiki

Yeah, DRBD is actually working on delivering some kind of geoclustering solution, but they are far from being usable yet. So, so far I'm still thinking in terms of rsync

Since you are running a RAID1 for the other partition (/), mounting /opt would likely suffice since the mailstores and the majority of the config is underneath that directory.

What else should I be backing up, besides /opt, by the way? Anything in /etc or /var ?

I believe you are going down the right track. However, you never said how many domains and mailboxes you plan to have on this machine. You also did not mention the size of the SAS drives and their performance levels (SATA, SCSI... 7.2k, 10k, 15k, etc...). By providing this information, it will help us better assist you and at least warn you of any pitfalls with your specific configuration (since your post is rather vague).

Sorry about that. The plan is to eventually have ~500 users, from ~30 domains.
All disks are 15k SAS, 300Gb for the OS and 900Gb for the raid10.

NOTE: One final page for great reading is: Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki

Hope this helps!

Thanks, I missed that link, and it definitely does provide a lot. So does your post, really appreciated.

[cyberdeath]

Originally Posted by Olorin

First of all thank you for the concise and thorough reply, I really appreciate your time and effort doing this

You're welcome and I hope this somehow helps you in your decision-making process.

Originally Posted by Olorin

I was under the impression 32Gb should be quite enough. If I am wrong, I still have time to spec up, but I'd really love to see some sort of a calculation table or something that would justify even more RAM. As for disk IO, I do have some fears there - 8 disks in raid10 are normally quite enough, but with encryption thrown in the middle... I just don't know - never tried it before.

32GB is actually fine. I was not trying to imply that you *needed* more RAM, but as you likely already know, more RAM doesn't hurt anything. It was more intended to state the core requirements of Zimbra than to say that your system was insufficient because it should not be (read: you have enough). However, keep in mind that your backup system will need to have a similar memory configuration (or at least 16GB) to run adequately should the need arise. This also applies to disk space and disk I/O (again, assuming that you want to quickly fail over to the virtual system and obtain/sustain similar performance benchmarks). In such a situation, having an installed (but offline) copy of Zimbra would do the trick...and then using rsync (or another method) to copy the backups that are created by the live Zimbra (which are not susceptible to op/file locks since they are created and dumped). Then you can even setup a restore procedure on these to have it updated on the "hot spare" virtual environment so it's always ready to go.

Originally Posted by Olorin

The plan is raid1 for the OS plus a raid10 for Zimbra. I do wonder whether or not /opt can be local and unencrypted, and I can simply mount a separate raid10 based chunk for the mailstore separately.

I would suggest configuring your system as follows (or something similar...feel free to break the RAID10 into logical chunks if you already spec'd out space constraints...the way below is simply "playing it safe"):
/ (including /boot) = RAID1
/opt/zimbra/ = RAID10 (or depending on what you have in /opt...you might want to "opt" to simply mount your RAID10 array to /opt).

An option would be to encrypt /opt/zimbra and leave everything else unencrypted...warning...there may be temp files...etc. that are not caught by this method...so it's not best practice. However, encrypting the entire drive will be a hassle as well...I hope you have remote KVM.

Originally Posted by Olorin

This is going to be a brand machine (probably a Dell or HP), so I'll be using a proper raid controller, with a BBU and the max amount of cache I can get for it

I don't have the option to play with it unfortunately, once the hardware is spec'd out and in place, I have to bring it all up into production. I've googled a LOT looking for zimbra on LUKS use cases and benchmarks, but couldn't find anything. Seems like I'm the first to do that.

Well, you may not be able to find a 1:1 comparision. However, as with most mail systems, you have two components: a database (SQL) and a bunch of small files (your actual mail content...message store).

Here's a quote from a website I found:

Originally Posted by http://www.linuxuser.co.uk/reviews/the-best-file-encryption-software-in-open-source/

As with the other encryption technologies on test, LUKS is designed in such a way that the unencrypted data is never written to the disk: instead, it is encrypted and decrypted as it’s read and written. While this means that security is kept at a maximum, there is a performance penalty to pay. Thankfully, on a modern system that shouldn’t be too onerous: while small-file performance took a hit – a test in which we copied 500 128KB files to the target volume – the throughput in copying a large file to the encrypted volume was only slightly slower than using no encryption at all.

As with any software-based encryption system, however, there is a trade-off: as you encrypt and decrypt data, the system CPU will be loaded. If you’re running a slower system – especially one with only a single processing core – you may find general performance impacted as the system works the cryptography engine.

So, it appears from what I am reading (and also what I have seen on other sites) that encryption isn't overly painful. Furthermore, based upon the drives you mentioned later (15k SAS drives that aren't overly large), you should have enough "IOPS" to handle the encryption piece and still come out like you're running a lower ended machine but still sufficient for the need (worst-case).

Originally Posted by Olorin

There's another point, I do plan to have backups done, and with Zimbra, if I back up /opt restore should be easy. However, for that sort of backup, I'll need to stop zimbra services, and I'd rather not do that. Are there any guides that show how to keep the stdby box sync'd, so real HA can work? I'm pretty sure there are ways to do that with the costlier versions of Zimbra, but I'd rather stick to the OSS builds.

If you had the Network Edition you can always just backup /opt/zimbra/backup and /opt/zimbra/redolog to restore. Then setup a cron job to redo the redolog on the other system (as I alluded to earlier).

However, since it sounds like you're looking at the OSS only, here's another option: Open Source Edition Backup Procedure - Zimbra :: Wiki . Scroll down until you get to "Backup Shell Script with Compressed & Encrypted Archives" (according to the notes in the script, it will take the server down for less than 2 minutes to complete the backup).

There's a lot out there on backing up and restoring with the FOSS...so I would just look around until you find the right fit...and I would stand up a test box to test all this before you buy the actual hardware...that's the great part about FOSS, right ;-).

Originally Posted by Olorin

If the CEO wants more than two months, he will approve the budget for a better storage solution In this case however, I'm not looking for block level, but for _brick_ level (guess I got the term for MSExchange). Meaning, I can restore separate mailboxes in case of need, instead of pulling in the entire setup, which might end up being quite the monster.

The backups are going to be held in another server, hosted elsewhere

You might find this forum post to be of interest: Open Source Backup strategy in 7.1.3 (with recover of deleted items)

Note that the procedures around restoring and backing up with the FOSS are not easy...so you may come to a point where you think "Is it worth all this time for me to spend trying to make this work over just paying for a perpetual license and being done with it." Honestly, that's where I would be. However, I also love FOSS and the idea behind it is that you *don't* pay...so I respect both ways of looking at it.

Originally Posted by Olorin

What kind of scripting? a simple rsync to sync up /opt or...? Being an old sysadmin, I'm rather professionally lazy, so I really hate reinventing wheels, especially when there's a chance of my new wheel turning out to be square.

See above for example scripts.

Originally Posted by Olorin

Identical, hardware-wise? The stdby machine is going to be a VM probably, can't afford to have another large server up there. I could keep a zimbra setup on the backup server though, with the services stopped of course. Not sure how all the SSL certs will behave if the setup needs to failover though...

If you make an image of the main server and virtualize it, you shouldn't have any problems except changing a few configuration settings...that's what I was referring to. However, if you can't do that...that's fine too..you can always stand up a separate virtual box and install Zimbra...then follow these instructions:

Transfer SSL certificates between servers - Zimbra :: Wiki
and another good one: [SOLVED] installation of existing ssl certificate new install. Certificate not expor

Originally Posted by Olorin

Yup, no biggie

Perfect! Then one of the solutions above should work. If it was a big deal...you'd be stuck based on your pipe to the other location.

Originally Posted by Olorin

It's more about rack space in this case. Can't put a SAN in there unfortunately. Besides, failover, as such, should happen to another geo-location, so a local SAN won't help, and implementing iFCP based solutions for a single MTA is really overkill

Then I would follow the above (one of the backup methods mentioned above) and copy the differences in the files to the failover location and you should be fine...assuming you have a semi-adequate connection based upon the amount of traffic (mail, files..etc) you have on any given day.

Originally Posted by Olorin

Yeah, DRBD is actually working on delivering some kind of geoclustering solution, but they are far from being usable yet. So, so far I'm still thinking in terms of rsync

Yeah, DRBD still isn't there for me yet, either...at least in the topological scenario you describe. And agreed...the scripts I mentioned above for backup use rsync and if you were to buy the NE, I would use rsync to backup the redologs and the backups folders respectively.

Originally Posted by Olorin

What else should I be backing up, besides /opt, by the way? Anything in /etc or /var ?

Well, that all depends on you. Assuming you have another standby system that you plan to keep up-to-date manually (OS patches..etc), then I don't see a need in backing up anything else...everything that Zimbra uses is in /opt/zimbra by default.

Originally Posted by Olorin

Sorry about that. The plan is to eventually have ~500 users, from ~30 domains.
All disks are 15k SAS, 300Gb for the OS and 900Gb for the raid10.

Sounds good. See above for elaboration.

Originally Posted by Olorin

Thanks, I missed that link, and it definitely does provide a lot. So does your post, really appreciated.

Thanks . Hope this helps!

[Olorin]

Originally Posted by cyberdeath

Thanks . Hope this helps!

It definitely does! Cheers