Results 1 to 7 of 7

Thread: Some planning questions

  1. #1
    Olorin is offline Senior Member
    Join Date
    May 2007
    Posts
    52
    Rep Power
    7

    Default Some planning questions

    I am planning to deploy a ZCS machine in a remote DC on colocation.
    Using the latest Community edition
    Hardware:
    2x SAS disks for OS in raid1,
    8x SAS disks in raid10 for Zimbra,
    32Gb RAM
    8 CPU cores

    Requirements:
    - Normal system functionality
    - Everyting must be encoded (currently considering the standard LUKS)
    - Backup to another machine in the same DC, not using 3rd party software
    - The possibility of a fast restore to another machine in case of failure. Replication to a standby machine would be even better

    Questions:
    - Any comments on the hardware?
    - How much of a performance hit would the encryption mean?
    - Which of the existing backup scripts is preferable? I need the possibility to restore up to 1-2 months back, weekly full, and daily incrementals would be grand. Brick level - extra plus
    - What other DR options are available? What about HA?
    - How is the installation done in this case? Simply use /opt as the mountpoint for the raid10, or..?
    - Which of the supported OS is the most stable and easy to maintain with a large number of mailboxes? I am kind of disappointed in CentOS, anyone aware of how SL performs?

    Any additional comments or anything I might have missed also welcome

  2. #2
    cyberdeath's Avatar
    cyberdeath is offline Active Member
    Join Date
    Jan 2008
    Location
    127.0.0.1, Virginia, USA
    Posts
    42
    Rep Power
    7

    Default

    Quote Originally Posted by Olorin View Post
    Questions:
    - Any comments on the hardware?
    Hardware seems fine. From my experience, the biggest thing you need with Zimbra (or any MTA for that matter) is RAM...and more RAM. The second thing would probably be Disk I/O since processor power isn't too hard to come by these days.

    You are making a good choice for the OS RAID & the Zimbra RAID. For larger deployments, I would say, if you can afford it (in disk space and cost), go for the RAID 10 as it outperforms RAID6, even though RAID6 is cheaper in terms of monetary cost per GB.

    Quote Originally Posted by Olorin View Post
    - How much of a performance hit would the encryption mean?
    Encryption is always going to place an impact on your overall I/O. The trick is in mitigation. You have done well thus far by running the RAID10 for performance. The only other thing you could do would be to look into hardware RAID setups (which tend to be costly and are not as "standardized" as linux is...which also applies for software RAID as well).

    I would say your best bet would be to do the following:
    Go ahead and install the OS you plan to use with encryption enabled on a box. Then, run a dd test (or some other variant) to see how well reading and writing does on the encrypted volumes. Then, of course, you'll need to install the same OS on a device WITHOUT encryption to see how well it does in comparison. That will give you a real rough estimate of how well your encryption algorithm is going to perform. There are also some great case studies and tests out on the web so you don't have to go through the effort...I would check that out as well.

    As far as personal experience...encryption does hurt...quite a bit on I/O. However, it all depends on how big your need is. Yes, it may be painful doing a backup or restore....but let's just hope you never have to do one...and also that you proactively keep a standby box stood up and ready to take over should it happen.


    Quote Originally Posted by Olorin View Post
    - Which of the existing backup scripts is preferable? I need the possibility to restore up to 1-2 months back, weekly full, and daily incrementals would be grand. Brick level - extra plus
    If you're looking for block level, you're likely looking for LVM. There's a lot of controversy on this because some say that LVM causes a performance hit while others disagree. I personally do not know because I've never compared the two side-by-side. However, LVM *will* let you do snapshots at the filesystem level. The only caveat is the amount of disk space necessary to keep all of these snapshots...are you planning on adding a third level of low performance storage drives to keep them on? I would strongly encourage it if you go this route and need to keep many months of data. As you well know... 1-2 months is good...but what happens when your CFO or CEO needs 6 months back?

    After speaking with a Zimbra engineer about the backups, my understanding is the best (or at least the lesser complicated) way would be to simply use the backups generated by the primary mail server and copying it over to a spare Zimbra server. That way it's just a matter of recovering and you're up. You can even setup the spare to do this automatically as it receives the file (through scripting, of course).

    Assuming that both machines are identical:
    I would personally setup your primary server how you want it then disk duplicate to the other server (Through whatever means you prefer). Then you'll need to change a couple things (IP address...etc) and then you'll want to setup the script to run and backup your data. (Example: Backing up and restoring Zimbra (Open Source Version) - Zimbra :: Wiki).

    Please note that this method assumes you do not mind losing some of the day's data (based upon the frequency of your interval backups...which I believe is default at 24h).


    Quote Originally Posted by Olorin View Post
    - What other DR options are available? What about HA?
    Using a SAN and disk replication would assuredly be the easiest from a system administrators standpoint. This can be setup however you want to and as frequently...with all the snapshots you want...etc etc. The only catch is the amount of money it will cost to implement.

    As far as HA is concerned...I have heard a lot said about DRBD. The only catch is you need a fast network interface between the two devices to successfully make it happen (such as the crossover Cat5e or better cables connecting to a NIC on each server). A good article about this topic can be found here: Ajcody-Notes-HA-Linux-How-To - Zimbra :: Wiki

    Quote Originally Posted by Olorin View Post
    - How is the installation done in this case? Simply use /opt as the mountpoint for the raid10, or..?
    Since you are running a RAID1 for the other partition (/), mounting /opt would likely suffice since the mailstores and the majority of the config is underneath that directory.

    Quote Originally Posted by Olorin View Post
    - Which of the supported OS is the most stable and easy to maintain with a large number of mailboxes? I am kind of disappointed in CentOS, anyone aware of how SL performs?
    I have personally used SLES for two reasons:
    1. It seems to perform really well and is stable.
    2. It is supported by Novell for a modest licensing fee.

    However, I'm sure that RedHat and others are just fine as well. Just a personal preference.

    Quote Originally Posted by Olorin View Post
    Any additional comments or anything I might have missed also welcome
    I believe you are going down the right track. However, you never said how many domains and mailboxes you plan to have on this machine. You also did not mention the size of the SAS drives and their performance levels (SATA, SCSI... 7.2k, 10k, 15k, etc...). By providing this information, it will help us better assist you and at least warn you of any pitfalls with your specific configuration (since your post is rather vague).

    NOTE: One final page for great reading is: http://wiki.zimbra.com/wiki/Performa...ge_Deployments

    Hope this helps!
    cyberdeath

  3. #3
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,366
    Rep Power
    10

    Default

    Quote Originally Posted by Olorin View Post
    I am planning to deploy a ZCS machine in a remote DC on colocation.
    Using the latest Community edition
    Hardware:
    2x SAS disks for OS in raid1,
    8x SAS disks in raid10 for Zimbra,
    32Gb RAM
    8 CPU cores

    Requirements:
    - Normal system functionality
    - Everyting must be encoded (currently considering the standard LUKS)
    - Backup to another machine in the same DC, not using 3rd party software
    - The possibility of a fast restore to another machine in case of failure. Replication to a standby machine would be even better

    Questions:
    - Any comments on the hardware?
    - How much of a performance hit would the encryption mean?
    - Which of the existing backup scripts is preferable? I need the possibility to restore up to 1-2 months back, weekly full, and daily incrementals would be grand. Brick level - extra plus
    - What other DR options are available? What about HA?
    - How is the installation done in this case? Simply use /opt as the mountpoint for the raid10, or..?
    - Which of the supported OS is the most stable and easy to maintain with a large number of mailboxes? I am kind of disappointed in CentOS, anyone aware of how SL performs?

    Any additional comments or anything I might have missed also welcome

    Why must everything be encoded?

  4. #4
    Olorin is offline Senior Member
    Join Date
    May 2007
    Posts
    52
    Rep Power
    7

    Default

    Quote Originally Posted by LMStone View Post
    Why must everything be encoded?
    The machine will be hosted in one of the less stable ex-USSR countries, where the authorities don't need a real reason to expropriate hardware. Nothing illegal, just email traffic, but encryption is a requirement.

  5. #5
    Olorin is offline Senior Member
    Join Date
    May 2007
    Posts
    52
    Rep Power
    7

    Default

    First of all thank you for the concise and thorough reply, I really appreciate your time and effort doing this

    Quote Originally Posted by cyberdeath View Post
    Hardware seems fine. From my experience, the biggest thing you need with Zimbra (or any MTA for that matter) is RAM...and more RAM. The second thing would probably be Disk I/O since processor power isn't too hard to come by these days.
    I was under the impression 32Gb should be quite enough. If I am wrong, I still have time to spec up, but I'd really love to see some sort of a calculation table or something that would justify even more RAM. As for disk IO, I do have some fears there - 8 disks in raid10 are normally quite enough, but with encryption thrown in the middle... I just don't know - never tried it before.

    You are making a good choice for the OS RAID & the Zimbra RAID. For larger deployments, I would say, if you can afford it (in disk space and cost), go for the RAID 10 as it outperforms RAID6, even though RAID6 is cheaper in terms of monetary cost per GB.
    The plan is raid1 for the OS plus a raid10 for Zimbra. I do wonder whether or not /opt can be local and unencrypted, and I can simply mount a separate raid10 based chunk for the mailstore separately.

    Encryption is always going to place an impact on your overall I/O. The trick is in mitigation. You have done well thus far by running the RAID10 for performance. The only other thing you could do would be to look into hardware RAID setups (which tend to be costly and are not as "standardized" as linux is...which also applies for software RAID as well).
    This is going to be a brand machine (probably a Dell or HP), so I'll be using a proper raid controller, with a BBU and the max amount of cache I can get for it

    I would say your best bet would be to do the following:
    Go ahead and install the OS you plan to use with encryption enabled on a box. Then, run a dd test (or some other variant) to see how well reading and writing does on the encrypted volumes. Then, of course, you'll need to install the same OS on a device WITHOUT encryption to see how well it does in comparison. That will give you a real rough estimate of how well your encryption algorithm is going to perform. There are also some great case studies and tests out on the web so you don't have to go through the effort...I would check that out as well.
    I don't have the option to play with it unfortunately, once the hardware is spec'd out and in place, I have to bring it all up into production. I've googled a LOT looking for zimbra on LUKS use cases and benchmarks, but couldn't find anything. Seems like I'm the first to do that.

    As far as personal experience...encryption does hurt...quite a bit on I/O. However, it all depends on how big your need is. Yes, it may be painful doing a backup or restore....but let's just hope you never have to do one...and also that you proactively keep a standby box stood up and ready to take over should it happen.
    There's another point, I do plan to have backups done, and with Zimbra, if I back up /opt restore should be easy. However, for that sort of backup, I'll need to stop zimbra services, and I'd rather not do that. Are there any guides that show how to keep the stdby box sync'd, so real HA can work? I'm pretty sure there are ways to do that with the costlier versions of Zimbra, but I'd rather stick to the OSS builds.

    If you're looking for block level, you're likely looking for LVM. There's a lot of controversy on this because some say that LVM causes a performance hit while others disagree. I personally do not know because I've never compared the two side-by-side. However, LVM *will* let you do snapshots at the filesystem level. The only caveat is the amount of disk space necessary to keep all of these snapshots...are you planning on adding a third level of low performance storage drives to keep them on? I would strongly encourage it if you go this route and need to keep many months of data. As you well know... 1-2 months is good...but what happens when your CFO or CEO needs 6 months back?
    If the CEO wants more than two months, he will approve the budget for a better storage solution In this case however, I'm not looking for block level, but for _brick_ level (guess I got the term for MSExchange). Meaning, I can restore separate mailboxes in case of need, instead of pulling in the entire setup, which might end up being quite the monster.

    The backups are going to be held in another server, hosted elsewhere

    After speaking with a Zimbra engineer about the backups, my understanding is the best (or at least the lesser complicated) way would be to simply use the backups generated by the primary mail server and copying it over to a spare Zimbra server. That way it's just a matter of recovering and you're up. You can even setup the spare to do this automatically as it receives the file (through scripting, of course).
    What kind of scripting? a simple rsync to sync up /opt or...? Being an old sysadmin, I'm rather professionally lazy, so I really hate reinventing wheels, especially when there's a chance of my new wheel turning out to be square.

    Assuming that both machines are identical:
    I would personally setup your primary server how you want it then disk duplicate to the other server (Through whatever means you prefer). Then you'll need to change a couple things (IP address...etc) and then you'll want to setup the script to run and backup your data. (Example: Backing up and restoring Zimbra (Open Source Version) - Zimbra :: Wiki).
    Identical, hardware-wise? The stdby machine is going to be a VM probably, can't afford to have another large server up there. I could keep a zimbra setup on the backup server though, with the services stopped of course. Not sure how all the SSL certs will behave if the setup needs to failover though...

    Please note that this method assumes you do not mind losing some of the day's data (based upon the frequency of your interval backups...which I believe is default at 24h).
    Yup, no biggie

    Using a SAN and disk replication would assuredly be the easiest from a system administrators standpoint. This can be setup however you want to and as frequently...with all the snapshots you want...etc etc. The only catch is the amount of money it will cost to implement.
    It's more about rack space in this case. Can't put a SAN in there unfortunately. Besides, failover, as such, should happen to another geo-location, so a local SAN won't help, and implementing iFCP based solutions for a single MTA is really overkill

    As far as HA is concerned...I have heard a lot said about DRBD. The only catch is you need a fast network interface between the two devices to successfully make it happen (such as the crossover Cat5e or better cables connecting to a NIC on each server). A good article about this topic can be found here: Ajcody-Notes-HA-Linux-How-To - Zimbra :: Wiki
    Yeah, DRBD is actually working on delivering some kind of geoclustering solution, but they are far from being usable yet. So, so far I'm still thinking in terms of rsync

    Since you are running a RAID1 for the other partition (/), mounting /opt would likely suffice since the mailstores and the majority of the config is underneath that directory.
    What else should I be backing up, besides /opt, by the way? Anything in /etc or /var ?

    I believe you are going down the right track. However, you never said how many domains and mailboxes you plan to have on this machine. You also did not mention the size of the SAS drives and their performance levels (SATA, SCSI... 7.2k, 10k, 15k, etc...). By providing this information, it will help us better assist you and at least warn you of any pitfalls with your specific configuration (since your post is rather vague).
    Sorry about that. The plan is to eventually have ~500 users, from ~30 domains.
    All disks are 15k SAS, 300Gb for the OS and 900Gb for the raid10.

    NOTE: One final page for great reading is: Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki

    Hope this helps!
    Thanks, I missed that link, and it definitely does provide a lot. So does your post, really appreciated.

  6. #6
    cyberdeath's Avatar
    cyberdeath is offline Active Member
    Join Date
    Jan 2008
    Location
    127.0.0.1, Virginia, USA
    Posts
    42
    Rep Power
    7

    Default

    Quote Originally Posted by Olorin View Post
    First of all thank you for the concise and thorough reply, I really appreciate your time and effort doing this
    You're welcome and I hope this somehow helps you in your decision-making process.

    Quote Originally Posted by Olorin View Post
    I was under the impression 32Gb should be quite enough. If I am wrong, I still have time to spec up, but I'd really love to see some sort of a calculation table or something that would justify even more RAM. As for disk IO, I do have some fears there - 8 disks in raid10 are normally quite enough, but with encryption thrown in the middle... I just don't know - never tried it before.
    32GB is actually fine. I was not trying to imply that you *needed* more RAM, but as you likely already know, more RAM doesn't hurt anything. It was more intended to state the core requirements of Zimbra than to say that your system was insufficient because it should not be (read: you have enough). However, keep in mind that your backup system will need to have a similar memory configuration (or at least 16GB) to run adequately should the need arise. This also applies to disk space and disk I/O (again, assuming that you want to quickly fail over to the virtual system and obtain/sustain similar performance benchmarks). In such a situation, having an installed (but offline) copy of Zimbra would do the trick...and then using rsync (or another method) to copy the backups that are created by the live Zimbra (which are not susceptible to op/file locks since they are created and dumped). Then you can even setup a restore procedure on these to have it updated on the "hot spare" virtual environment so it's always ready to go.

    Quote Originally Posted by Olorin View Post
    The plan is raid1 for the OS plus a raid10 for Zimbra. I do wonder whether or not /opt can be local and unencrypted, and I can simply mount a separate raid10 based chunk for the mailstore separately.
    I would suggest configuring your system as follows (or something similar...feel free to break the RAID10 into logical chunks if you already spec'd out space constraints...the way below is simply "playing it safe"):
    / (including /boot) = RAID1
    /opt/zimbra/ = RAID10 (or depending on what you have in /opt...you might want to "opt" to simply mount your RAID10 array to /opt).

    An option would be to encrypt /opt/zimbra and leave everything else unencrypted...warning...there may be temp files...etc. that are not caught by this method...so it's not best practice. However, encrypting the entire drive will be a hassle as well...I hope you have remote KVM.

    Quote Originally Posted by Olorin View Post
    This is going to be a brand machine (probably a Dell or HP), so I'll be using a proper raid controller, with a BBU and the max amount of cache I can get for it

    I don't have the option to play with it unfortunately, once the hardware is spec'd out and in place, I have to bring it all up into production. I've googled a LOT looking for zimbra on LUKS use cases and benchmarks, but couldn't find anything. Seems like I'm the first to do that.
    Well, you may not be able to find a 1:1 comparision. However, as with most mail systems, you have two components: a database (SQL) and a bunch of small files (your actual mail content...message store).

    Here's a quote from a website I found:
    Quote Originally Posted by http://www.linuxuser.co.uk/reviews/the-best-file-encryption-software-in-open-source/
    As with the other encryption technologies on test, LUKS is designed in such a way that the unencrypted data is never written to the disk: instead, it is encrypted and decrypted as it’s read and written. While this means that security is kept at a maximum, there is a performance penalty to pay. Thankfully, on a modern system that shouldn’t be too onerous: while small-file performance took a hit – a test in which we copied 500 128KB files to the target volume – the throughput in copying a large file to the encrypted volume was only slightly slower than using no encryption at all.

    As with any software-based encryption system, however, there is a trade-off: as you encrypt and decrypt data, the system CPU will be loaded. If you’re running a slower system – especially one with only a single processing core – you may find general performance impacted as the system works the cryptography engine.
    So, it appears from what I am reading (and also what I have seen on other sites) that encryption isn't overly painful. Furthermore, based upon the drives you mentioned later (15k SAS drives that aren't overly large), you should have enough "IOPS" to handle the encryption piece and still come out like you're running a lower ended machine but still sufficient for the need (worst-case).

    Quote Originally Posted by Olorin View Post
    There's another point, I do plan to have backups done, and with Zimbra, if I back up /opt restore should be easy. However, for that sort of backup, I'll need to stop zimbra services, and I'd rather not do that. Are there any guides that show how to keep the stdby box sync'd, so real HA can work? I'm pretty sure there are ways to do that with the costlier versions of Zimbra, but I'd rather stick to the OSS builds.
    If you had the Network Edition you can always just backup /opt/zimbra/backup and /opt/zimbra/redolog to restore. Then setup a cron job to redo the redolog on the other system (as I alluded to earlier).

    However, since it sounds like you're looking at the OSS only, here's another option: Open Source Edition Backup Procedure - Zimbra :: Wiki . Scroll down until you get to "Backup Shell Script with Compressed & Encrypted Archives" (according to the notes in the script, it will take the server down for less than 2 minutes to complete the backup).

    There's a lot out there on backing up and restoring with the FOSS...so I would just look around until you find the right fit...and I would stand up a test box to test all this before you buy the actual hardware...that's the great part about FOSS, right ;-).


    Quote Originally Posted by Olorin View Post
    If the CEO wants more than two months, he will approve the budget for a better storage solution In this case however, I'm not looking for block level, but for _brick_ level (guess I got the term for MSExchange). Meaning, I can restore separate mailboxes in case of need, instead of pulling in the entire setup, which might end up being quite the monster.

    The backups are going to be held in another server, hosted elsewhere
    You might find this forum post to be of interest: Open Source Backup strategy in 7.1.3 (with recover of deleted items)

    Note that the procedures around restoring and backing up with the FOSS are not easy...so you may come to a point where you think "Is it worth all this time for me to spend trying to make this work over just paying for a perpetual license and being done with it." Honestly, that's where I would be. However, I also love FOSS and the idea behind it is that you *don't* pay...so I respect both ways of looking at it.

    Quote Originally Posted by Olorin View Post
    What kind of scripting? a simple rsync to sync up /opt or...? Being an old sysadmin, I'm rather professionally lazy, so I really hate reinventing wheels, especially when there's a chance of my new wheel turning out to be square.
    See above for example scripts.

    Quote Originally Posted by Olorin View Post
    Identical, hardware-wise? The stdby machine is going to be a VM probably, can't afford to have another large server up there. I could keep a zimbra setup on the backup server though, with the services stopped of course. Not sure how all the SSL certs will behave if the setup needs to failover though...
    If you make an image of the main server and virtualize it, you shouldn't have any problems except changing a few configuration settings...that's what I was referring to. However, if you can't do that...that's fine too..you can always stand up a separate virtual box and install Zimbra...then follow these instructions:

    Transfer SSL certificates between servers - Zimbra :: Wiki
    and another good one: [SOLVED] installation of existing ssl certificate new install. Certificate not expor

    Quote Originally Posted by Olorin View Post
    Yup, no biggie
    Perfect! Then one of the solutions above should work. If it was a big deal...you'd be stuck based on your pipe to the other location.

    Quote Originally Posted by Olorin View Post
    It's more about rack space in this case. Can't put a SAN in there unfortunately. Besides, failover, as such, should happen to another geo-location, so a local SAN won't help, and implementing iFCP based solutions for a single MTA is really overkill
    Then I would follow the above (one of the backup methods mentioned above) and copy the differences in the files to the failover location and you should be fine...assuming you have a semi-adequate connection based upon the amount of traffic (mail, files..etc) you have on any given day.

    Quote Originally Posted by Olorin View Post
    Yeah, DRBD is actually working on delivering some kind of geoclustering solution, but they are far from being usable yet. So, so far I'm still thinking in terms of rsync
    Yeah, DRBD still isn't there for me yet, either...at least in the topological scenario you describe. And agreed...the scripts I mentioned above for backup use rsync and if you were to buy the NE, I would use rsync to backup the redologs and the backups folders respectively.

    Quote Originally Posted by Olorin View Post
    What else should I be backing up, besides /opt, by the way? Anything in /etc or /var ?
    Well, that all depends on you. Assuming you have another standby system that you plan to keep up-to-date manually (OS patches..etc), then I don't see a need in backing up anything else...everything that Zimbra uses is in /opt/zimbra by default.

    Quote Originally Posted by Olorin View Post
    Sorry about that. The plan is to eventually have ~500 users, from ~30 domains.
    All disks are 15k SAS, 300Gb for the OS and 900Gb for the raid10.
    Sounds good. See above for elaboration.

    Quote Originally Posted by Olorin View Post
    Thanks, I missed that link, and it definitely does provide a lot. So does your post, really appreciated.
    Thanks . Hope this helps!
    cyberdeath

  7. #7
    Olorin is offline Senior Member
    Join Date
    May 2007
    Posts
    52
    Rep Power
    7

    Default

    Quote Originally Posted by cyberdeath View Post
    Thanks . Hope this helps!
    It definitely does! Cheers

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Questions.....
    By Eclipse in forum Installation
    Replies: 1
    Last Post: 09-20-2011, 11:04 PM
  2. Zimbra Mobile Newbie questions
    By dazi01 in forum Zimbra Mobile
    Replies: 1
    Last Post: 10-11-2007, 12:29 PM
  3. Zimbra 5 RC1 - Bug and IM questions
    By greenrenault in forum Installation
    Replies: 1
    Last Post: 10-08-2007, 08:43 AM
  4. A few other questions
    By dcm in forum Installation
    Replies: 0
    Last Post: 08-29-2006, 03:33 PM
  5. Commercial CA questions
    By cdyer in forum Administrators
    Replies: 3
    Last Post: 07-17-2006, 03:48 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •