Server Migration -- ZCS to ZCS with no/minimal downtime

[jwestta]

Hey folks,

Want to run something by those who are seasoned Zimbra administrators...

I'm currently in the process of planning a migration from our current ZCS NE 6.0.x deployment (CentOS 5.6 x86_64) to a new server hosted in a new location of ours. One of the important factors for this project is to minimize downtime as much as possible, since our single-server deployment currently runs email + calendaring for the entire company (~1000 accounts). Additionally, we're totaling about 3.5TB of content (no quotas).

Because we have 3.5TB of email (etc), shutting down Zimbra and performing a massive rsync/scp to the new system isn't possible as we'd have too much downtime. There's a 100Mbps link between the two sites; would just take too long for our comfort. So, I've been investigating alternative migration practices.

One of the ideas I came up with was to install Zimbra on our new server and join it to our current system as a replica LDAP host and a new mailstore. Then, at my leisure, I could use 'zmmailboxmove' on each account over the course of a week or so, to move each account over to the new system. Our current server would of course have to be reconfigured to be a Web as well as IMAP/SMTP proxy, but from my experiments with Zimbra, this was a rather trivial process and really was only a matter of stopping/starting Zimbra along with a couple commands.

Once all mailboxes have been migrated to the new server, I would stop the old one that now has no remaining users, and promote the new system to become the LDAP master. I'd also adjust DNS at this time such that the new server's IP is officially our mail system and no longer a hidden Zimbra server node behind the old server's proxy services.

What do you think of this migration procedure? Bad idea and you foresee many difficulties on the road ahead? Or have folks used this method time and time again with success?

Or... do you have any other methods of migrating 3.5TB ZCS instances with the least downtime possible? Perhaps doing a backup/restore onto the new server and during our scheduled maintenance window, replaying redo logs?

Any tips and war stories would be most helpful.

Thanks folks.

--Josh West

[bdial]

guess it depends on what an acceptable amount of down time is. you could do an initial rsync of the 3.5 TB over a wee or however long it takes, then do another one when thats finished, which may take half a day given how much has hcanged since you did the initial one, then another one, then shut down the server and do a final one. depending on the volume of mail you process normally and the connection, the final rsync may only take a matter of minutes.

[jwestta]

Hey bdial,

Thanks for the response.

I'm thinking that half of a day for a rsync/downtime would be too much. I think I'll run some internal tests here and rate limit to the speed of what I'd expect to get when the new location is available; perhaps I can get a better idea of how much changes and thus how much time it will take for a second rsync pass. Maybe a weekend has less volume... Hmm...

What do you think of my first option though? Joining a new node, setting the original as a proxy, zmmailboxmove everybody over, and then the final cutover?

Thanks.

--Josh West

[bdial]

the first option will be fine, you'll wind up with a different server name in the end so you'll just have to alias the old address to there and maybe set the publichostname if you wish it to appear as the old address.

the downtime shouldn't be that much really if you do multiple rsyncs. think of it like this

1. you rsync 3.5tb initially, it takes a week
2. you rsync again, and it has to rsync 10gb worth of changes that happened in the week it took for the initial rsync. this takes a day
3. you rsync again, and it has to rsync 500mb worth of changes from the previous day, this takes a couple of hours
4. you rsync again, and it has to rynsc 100mb worth of changes, this takes 20min

(you get the idea).

you keep rsyncing until you get it down to a small time frame, and yeah like you said on a weekend this may be easier

[LMStone]

Originally Posted by jwestta

Hey folks,

Want to run something by those who are seasoned Zimbra administrators...

I'm currently in the process of planning a migration from our current ZCS NE 6.0.x deployment (CentOS 5.6 x86_64) to a new server hosted in a new location of ours. One of the important factors for this project is to minimize downtime as much as possible, since our single-server deployment currently runs email + calendaring for the entire company (~1000 accounts). Additionally, we're totaling about 3.5TB of content (no quotas).

Because we have 3.5TB of email (etc), shutting down Zimbra and performing a massive rsync/scp to the new system isn't possible as we'd have too much downtime. There's a 100Mbps link between the two sites; would just take too long for our comfort. So, I've been investigating alternative migration practices.

One of the ideas I came up with was to install Zimbra on our new server and join it to our current system as a replica LDAP host and a new mailstore. Then, at my leisure, I could use 'zmmailboxmove' on each account over the course of a week or so, to move each account over to the new system. Our current server would of course have to be reconfigured to be a Web as well as IMAP/SMTP proxy, but from my experiments with Zimbra, this was a rather trivial process and really was only a matter of stopping/starting Zimbra along with a couple commands.

Once all mailboxes have been migrated to the new server, I would stop the old one that now has no remaining users, and promote the new system to become the LDAP master. I'd also adjust DNS at this time such that the new server's IP is officially our mail system and no longer a hidden Zimbra server node behind the old server's proxy services.

What do you think of this migration procedure? Bad idea and you foresee many difficulties on the road ahead? Or have folks used this method time and time again with success?

Or... do you have any other methods of migrating 3.5TB ZCS instances with the least downtime possible? Perhaps doing a backup/restore onto the new server and during our scheduled maintenance window, replaying redo logs?

Any tips and war stories would be most helpful.

Thanks folks.

--Josh West

Since you have 100MBps between sites, adding a second server and doing a zmmailboxmove is a viable option to minimize downtime, BUT....

To get rid of the original server you'll need to promote the LDAP replica on the second server; that will cause downtime and it's not a trivial task as you will see from the wiki.

The bigger issue for you however may be disk space. When you move a mailbox to another server, you lose the nice hardlinking of the mailstore. So, if User A sent nine other users a 20MB PowerPoint file, your mailstore increase by just a little over 20MB. After you zmmailboxmove all ten mailboxes, that 20MB PowerPoint now takes 200MB of storage. IOW, you have a 3.5TB mailstore now, but depending on how much your users send intra-company emails with attachments, you could have a mailstore on the new server several times that size after you do the zmmailboxmove for all accounts.

Hope that helps,
Mark

[jwestta]

Hi LMStone,

Ahh interesting... Didn't think about the hardlinking conundrum. My new system will be able to handle de-dudplication on the block level, so its possible I'll be able to still save the necessary space. Granted, now it wont be on a filesystem level, so if I ever move away from this new storage (NetApp), I'll be faced with hardlink expansion.

I was able to perform an LDAP replica-to-master migration without any problems in my test lab. Have you experienced any difficulties yourself with such a task?

The rsync method is definitely preferable to the complexity of an ldap master promotion along with DNS adjustments all at the right time. I'm just thinking its going to take wayyyyyyyyyyy too much time -- especially to build the file list before it sends any new/changed files!

Thanks.

--Josh West

[LMStone]

Originally Posted by jwestta

Hi LMStone,

Ahh interesting... Didn't think about the hardlinking conundrum. My new system will be able to handle de-dudplication on the block level, so its possible I'll be able to still save the necessary space. Granted, now it wont be on a filesystem level, so if I ever move away from this new storage (NetApp), I'll be faced with hardlink expansion.

It's great that SANs do that nowadays, but then unless you thin provision at the SAN level you will be explaining to management why you need more disk shelves for the SAN when the SAN reports it is only 60% utilized. And thin provisioning comes with its own set of challenges... We really try hard to keep the hard links if we can; it makes everything simpler.

Originally Posted by jwestta

I was able to perform an LDAP replica-to-master migration without any problems in my test lab. Have you experienced any difficulties yourself with such a task?

No difficulties, just fear! :-) LDAP is the core of Zimbra; mess it up and your mail blobs become just that: blobs. Virtualization snapshots make recovery easier, but the warning on the wiki page is good advice!

Originally Posted by jwestta

The rsync method is definitely preferable to the complexity of an ldap master promotion along with DNS adjustments all at the right time. I'm just thinking its going to take wayyyyyyyyyyy too much time -- especially to build the file list before it sends any new/changed files!

Thanks.

--Josh West

Yes, building the file list will take some time, I'd hazard a rough guess of at least an hour. But as pointed out above, you are only down for that final rsync; the ones previous can be done with the system running:

rsync -avzH --delete --progress --exclude="/backup" root@OldZimbraServerIP:/opt/zimbra/ /opt/zimbra

Hope that helps,
Mark