Geographically Separated Zimbra Sync

[plutocrat]

Hi all,
This post is a result of a lot of frustrated research. I have found several posts about this topic ranging over the last 6 years or so, and no really satisfactory solutions. I hope to collect all the possible ways of synching two Zimbra installations and invite comment from the community. Maybe (hopefully!) I'm missing something.

Executive Summary
While there are a number of solutions for High Availablilty / Disaster Recovery (active/passive) which work over a LAN, there are no officially supported solutions which work over a WAN, mainly due to high bandwidth required to sync up terabyte sized installations.

Here are the solutions I've identified, with a brief description, and the inherent problems.

1) Use VMware's HA capability. Apparently you need at a minimum Vmware Essentials Plus Kit. Two physical servers are clustered together within Vmware, and the VMs can live on either (or both) servers.
Problem: not realistic over a WAN, due to high bandwidth required

2) Use Red Hat Cluster Suite. This you have to purchase Redhat and the cluster suite, which is about USD 1500 per server per year. This option is now no longer officially supported since version 8 of Zimbra.
Problem: not realistic over a WAN, due to high bandwidth required; not officially supported.

3) Sync the SAN on a block level between datacentres. I'm guessing you'd need the same SANs at both ends.
Problem: Would need a lot of bandwidth and SAN is expensive; hard to find same SAN at both ends if using commercial datacentres.

4) Use OS's DRBD (Distributed Replicated Block Device) functionality to replicate file systems.
Problem: Again data replication bandwidth is problematic. May be ameliorated by using DRBD proxy. This is extra software at a one time fee of USD 2000. Not officially supported.

5) Use zimbra's replaylog functionality and some community supplied scripts.
Problem: Not officially supported. Some problems reported with missing data; some issues with switching back from secondary to primary afterwards.

6) Use Ubuntu clustering File System. GlusterFS.
Problems: Same as DRBD. Bandwidth. Not officially supported.

7) Give up and put both servers in the same datacentre!

So I'm finding it hard to believe this is the case. Exchange and Notes will do this sort of thing out of the box, and indeed it seems like such a basic function for an enterprise mail system to be lacking.

Why is it not just as simple as doing an rsync over ssh of the /opt/zimbra/store followed by an LDAP sync? And maybe a mysql replication. Surely that's all that's needed for an active / passive setup? Or LDAP sync plus IMAPsync of all mailboxes?

Any help, comments, suggestions and reality checks gratefully received. At the least I hope this serves as a current roundup of the options available for syncing Zimbra. Maybe it should become a wiki page.

[plutocrat]

Just for clarity, I'm talking about the Network Professional Edition here. I have a client ready to buy it if we can resolve this problem.

And here is a bugzilla item which seems relevant: https://bugzilla.zimbra.com/show_bug.cgi?id=11423
Apparently this feature has been requested for quite a while.

[plutocrat]

OK, an underwhelming response there. I'm guessing this is one of these topics that people sweep under the carpet. I've also been in touch with VMware customer support, but no replies there either.

So, lets put this another way. Obviously everyone running Zimbra is completely happy about having their servers in a single geographic location. So how do you guard against the possibility of your datacentre being buried by a volcano (or just going out of business?). Well you take a backup of course. And that backup must be offsite. But on an installation with a lot of data (lets just say 1Tb) you still run into the same problems: bandwidth and the sheer amount of data to backup means you can't take frequent backups. What are people's approaches to this?

[Himanshu]

OK, an underwhelming response there. I'm guessing this is one of these topics that people sweep under the carpet. I've also been in touch with VMware customer support, but no replies there either.

So, lets put this another way. Obviously everyone running Zimbra is completely happy about having their servers in a single geographic location. So how do you guard against the possibility of your datacentre being buried by a volcano (or just going out of business?). Well you take a backup of course. And that backup must be offsite. But on an installation with a lot of data (lets just say 1Tb) you still run into the same problems: bandwidth and the sheer amount of data to backup means you can't take frequent backups. What are people's approaches to this?

Once You Define RPO and RTO . You should be able to differentiate between all the options available more Carefully.

We have DRBD implemented for couple of Customers.

[kruon]

While there are a number of solutions for High Availablilty / Disaster Recovery (active/passive) which work over a LAN, there are no officially supported solutions which work over a WAN, mainly due to high bandwidth required to sync up terabyte sized installations.

You walk to the server_1, take the terabyte size snapshot to a removable media, like external drive or tape.
Then you either drive or take a plane and go to the other datacenter, plug in the media to server_2 and restore the snapshot.

I don't understand why the bandwidth is an issue here, surely you're not trying to replicate the entire server over WAN link without doing cold copy first?
That would be just stupid platform design, even if you have proper link on both ends assuming your volumes are real.

5) Use zimbra's replaylog functionality and some community supplied scripts.
Problem: Not officially supported. Some problems reported with missing data; some issues with switching back from secondary to primary afterwards.

This sounds quite sane way to me to do this.
Wait, you want official support for a feature that doesn't exist? Good luck with that..

Why is it not just as simple as doing an rsync over ssh of the /opt/zimbra/store followed by an LDAP sync? And maybe a mysql replication. Surely that's all that's needed for an active / passive setup? Or LDAP sync plus IMAPsync of all mailboxes?

It is?
Since you already moved most of the data using tape/media x, now you can rsync, sql replicate and perform ldap sync to finalize things*.
Most of the data is moved already so all you have to sync is few hundred megabytes to couple of gigs, depending on how much mail volume you have.
If that's too much for your second datacenter link to handle, you've selected wrong datacenter to use for geographic separation.

*Note that the remote system is passive "cold" clone, not active replica.

[plutocrat]

We have DRBD implemented for couple of Customers.

Would you be able to share your experiences of that?
- What volume of data are you talking about? (How big is your mailstore and how much does it grow each day for example?)
- Have you experienced any problems with this? Or does it just run 24/7 without any issues? Any problems with upgrading OSes / Zimbra for example?
- Are you running this on Redhat or Ubuntu?

I'd be quite keen to explore this option, but the client is terrified of any non-supported solution.

[plutocrat]

Hi Kruon. Thanks for your comments.

You walk to the server_1, take the terabyte size snapshot to a removable media, like external drive or tape.
Then you either drive or take a plane and go to the other datacenter, plug in the media to server_2 and restore the snapshot.

Ah yes, that works if the original server is still running. It would also take me the best part of a week to get over there to DC1, and travel to DC2! But I take your point. Physical transfer is the easiest way. The point of having a second data centre is to guard against the possibility (however small), that the first datacenter catches fire / gets hit by a tsunami / is taken over by terrorists or any one of a number of unknowables. So for this scenario the second datacentre must have an up to date copy of the data at all times. Copying from DC one is not an option if its already gone dark.

I don't understand why the bandwidth is an issue here, surely you're not trying to replicate the entire server over WAN link without doing cold copy first?
That would be just stupid platform design, even if you have proper link on both ends assuming your volumes are real.

I don't even mind if the second copy initially takes a few days to sync up; just that the ongoing bandwidth is minimal, copying only changed files. The (unsupported) DRBD option seems to do this. But everyone I've talked to about the (supported) VMware option tells me this can't be done. Microsoft manages it with Exchange somehow. IBM manages it with Notes somehow. So I expected that Zimbra's much touted "High Availability" would actually be as advertised. But they must have a different definition of HA to me.

This sounds quite sane way to me to do this.

Here is the link to the information and script: Server Live sync - Zimbra :: Wiki
I suspect we may end up using a combination of incremental backup + replaylog in the end.

Since you already moved most of the data using tape/media x, now you can rsync, sql replicate and perform ldap sync to finalize things*.
Most of the data is moved already so all you have to sync is few hundred megabytes to couple of gigs, depending on how much mail volume you have.

I hear that rsyncing the datastore doesn't work as it contains locked files. Perhaps if it were mounted on an LVM partition, you could snapshot it first and then sync. So this would actually work then? LDAP sync, SQL sync and then snapshot/rsync?
I imagine the secondary server OS would have to be set up with the same hostname. Would the different IP address present a problem?

[Himanshu]

You can take the support from linbit Depending on the Number of Nodes The support Cost would vary.As Implementer One has an option to Implement Open Source or Paid Support for drbd , Depending on Criticality and RTO/RPO requirements.

In One of the Case we have Paid Support where it has Database Replication on 5 Mbps Link and a 3 Node Implemented with Cluster at Central Site and 3rd Node at DR Site.
At Other Zimbra Deployment we are supporting with OpenSource DRBD where Both Nodes are At Central Site with Data Size of > 1 TB and 700 Users.
At One More Installations with similar Data size and Mailbox we are implementing 2 Node Sync with One at Primary and 2nd at DR Site, and Under Process.

[Himanshu]

Just Came Across Following.
Server Live sync - Zimbra :: Wiki

[j2b]

Sorry, if this thread is considered to be closed, but just wanted to add my 2 cents.

As from my experience DRBD or RSYNC is basically the decision between how "hot" you would like your DC2 server to be up to date. Honestly, I really do not consider bandwidth as a pitfall, as for such large data (say 1TB), it should be considered any way, e.g. the customer has to be used to increased expenses. Initial backup/data copy would be an issue any ways.

I've just rsynced a chunk of data over internet, and within EU datacenters it took me with speed almost 30Gigs/hour. So, at 1TB it should make it 1.5-2 days, which is not bad. I do not think, that media transfer to other DC would take less. And event if you move media to DC2, it again, depends, how you can access your nodes. In most cases you don't, which means, you are going to attach your media to locally connected PC and move data over network (ok, a bit faster, than internet), but it should take time, and it adds complexity.

As for sequential transfers, I'd prefer rsync, as it manages already copied data. So, if for some case your server is loosing connection, you don't start from 0. DRBD in this case is quite robust, but there is another problem - link stability, which to increase, would cost you fortune. DRBD is very nice, when you know, what you do. It's low level, thus not touching anything within Zimbra as a software layer. You have to keep an eye on primary/secondary statuses and if you somehow come into problem and overlook this, you loose very much. (Sorry, but from my own experience). And that is why, I'd prefer to go with less recent, but still full copy of data with rsync.

In addition, there is very nice software to support your operations, called zeXtras Suite. Say manage your backups with zeXtra, and move your data with rsync to whatever locations you need. In another DC you can setup separate ZCS stack, and rework your DNS in case of failure. Still zeXtras does not move everything, but it's very good relief for main e-mail and account data.

For disaster, you do not have to look so deep (hurricane, volcano, etc. ) Even decent ISO certified datacenter has it's weaknesses. Recently we suffered quite a great surprise, when such datacenter went dark almost for a week - fuel pumps were flooded, and Basta. Due to this, working data set of Zimbra outside DC is actually a must. Here I stress Zimbra data set (logically working data within Zimbra instance), as there is not enough only to copy just backup files, nor even whole /opt/zimbra directory.

We are actually finishing allocation of one ZCS multiserver instance in DC1 and first thing next year deploying another working set in another country. Aim #1 - get copy of working Zimbra stack with almost last data. zeXtras on both sides, and regular rsync from DC1 to DC2. If you manage to sync last backup data to DC2 and DC1 goes dark, you still have operational data in DC2, where you just have to reapply last backup. Yes, in this scenario your last window backup is accessible, but not the recent one. But you can backup Zimbra with zextras online, yet count on server resources to actually register latest changes to your accounts. If most recent data is needed, than DRBD or kind of snapshoting would make a case, but I haven't got so far.

Regarding data, there is always an option to create second DC as an archive store, and by receiving new e-mails, just resend them to another. But that would not contain 1:1 copy and can make some trouble to manage. Another option to look for would be Ceph file system (Ceph), but this is not designed for speed, which is needed by Zimbra. And you have to implement your own storage level solution, where tradeoff for proprietary SAN purchase would make a case. By the way, I do not think, that IBM/EMC/NetAPP/HP SAN storages are far from DRBD or network level RAID1 over iSCSI. For sure, guys have spent fortune, and would like earn more, but to be honest, most stable things turn out to be pretty simple. You still have an option, like OpenFiler or FreeNAS, where even you can get support, and can build these storages as a VMs, still managing storage on their own level. Unfortunately, just read about it, but have not tested, as I moved all hardware related issues to DC.

As for supportend/unsuppoerted - anyway, you are at OpenSource world with Zimbra. And in most cases options to introduce custom solutions helped me better, rather than wait for Microsoft to turn over for quite pricy product, still letting me wait in a queue. By this I'd like to say, that even if you deploy complex Microsoft solution (which is licenced and paid service), I doubt, that it would be faster, less complicated and less expensive. In most times it's just marketing, but real situation and life is just an opposite.