Full Server Backup and Restore of “open source version”

[raj]

Hello All:
We been testing Zimbra for almost 1 week now and it looks good.
I have read in few post that “network version / commercial” will have mailbox backup and restore but we are strictly interested to know how “Open Source Version” will be Backed up/Restored. I will try to clearly state scenarios

1) Move Zimbra install from 1 production server to another. Apart form installing Zimbra on new server what we need to do to Backup/copy existing one and Restore on New Target Server

2) What we need to Backup under ZIMBRA only (ie. What folders/files) on regular basis to so we can provide Disaster recovery if production server fails

Any insight is greatly appreciated

Thanks
Raj S Vrach

[KevinH]

Originally Posted by raj

Hello All:
We been testing Zimbra for almost 1 week now and it looks good.
I have read in few post that “network version / commercial” will have mailbox backup and restore but we are strictly interested to know how “Open Source Version” will be Backed up/Restored. I will try to clearly state scenarios

1) Move Zimbra install from 1 production server to another. Apart form installing Zimbra on new server what we need to do to Backup/copy existing one and Restore on New Target Server

If the hostname is the same, and the OS matches. You an just move /opt/zimbra everything lives under there. Assuming you've done at least an install on the new box to configure things like iptables(which by the way will not be needed in the next release)

Originally Posted by raj

2) What we need to Backup under ZIMBRA only (ie. What folders/files) on regular basis to so we can provide Disaster recovery if production server fails

Any insight is greatly appreciated

Thanks
Raj S Vrach

There are really 4 main things that have state: messages, indexes, ldap, and mysql. So a quick backup would be take a copy of the message and index data, then do a mysql and ldap dump. You must stop the server to make these backups since everything must remain in sync.

A guide to the directory structure is here:
http://www.zimbra.com/downloads/zimb...e.html#1036288

[bjquinn]

What if we have several hundred megabytes or several gigs of email? That'll take too long to backup. How long are we supposed to keep the server down to make the nightly backups? We want to switch over to Zimbra from Exchange, but even Exchange will backup without shutting down the server...

There has to be a better way with the open source version... what do you do in your Network version, and why can't we do it in the open source version?

-BJ Quinn

[andreychek]

Originally Posted by bjquinn

What if we have several hundred megabytes or several gigs of email? That'll take too long to backup. How long are we supposed to keep the server down to make the nightly backups? We want to switch over to Zimbra from Exchange, but even Exchange will backup without shutting down the server...

There has to be a better way with the open source version... what do you do in your Network version, and why can't we do it in the open source version?

It's only the Open Source version that has this limitation.

Not only does the Network version have the ability to backup the server while it's running, you can choose to backup the full email box, or just an incremental.

So, paying has it's advantages :-)
-Eric

[raj]

Well yes paying is good and it gives you lot more than just hot backup and restore..i have not got any time to do that yet but logically you don’t need to SHUT DOWN open source version really to BACKUP zimbra..
I am just thinking logically..

MySQL and LDAP and FILE STORE so all you need to figure out is how to do

1) HOT BACKUP of mysql
2) dont know how LDAP store data if we can just COPY
3) email store..for sure its just file copy

* iam talking about full backup..plus if you using a backup software it willl do incremental too so you dont backup 100's of gigs every time..only changed files..

so some day i will pull an all nighter and have this answers...till then if someone else is up in night doing this do post..

Raj

[andreychek]

Originally Posted by raj

MySQL and LDAP and FILE STORE so all you need to figure out is how to do

1) HOT BACKUP of mysql
2) dont know how LDAP store data if we can just COPY
3) email store..for sure its just file copy

Sure. And if you get that working, I'm sure lots of people will put it to good use.

The thing you need to watch out for is that there's a race condition... you have to make sure that MySQL, LDAP, and the filesystem never get out of sync. That's the biggest trick to making it work in the OSS version.

There are tools available which can perform hot backups of LDAP and MySQL, and it's easy enough to copy the files from the filesystem. So, the key is to do all that while keeping all the data in sync.

Maybe there's a way to enable some sort of read-only mode? Don't allow the users to change anything while a backup is being performed, but they can still view what already exists?

On a related note, I ran across this eweek article today:

http://www.eweek.com/article2/0,1895,1887301,00.asp

Which mentions the idea that Zimbra supports replication across a WAN. Is this referencing some tricks we don't know about yet? :-) It'd be nice if there were a way to keep a hot backup in another facility in case one facility were to go down for one reason or another.

Have a good one,
-Eric

[raj]

Q is what you want to do with backup
1) if its for BACKUP and RESTORE on frequent basis..like your company and customer keep asking for restoring the mailboxes and their previously deleted emails and stuff ..then yes sync is really imp and required.

2) if you only want to backup for Disaster recovery..like you running 5000 users and 20 GB email and in case of HW failure or some other %$^%$ happens to your production server you like to go back to last backup..in that case 98% of the time you can live with
”out of sync” and how much out of sync you would go..2 hours of email or less..

My point is to prepare for disaster recover not really into restoring stuff on regular or per user biases

One point is we will integrate ADMIN functions how users can be added and removed in our control panel so we can HALT that for 1 hours or less it takes to backup so USERS DB wont go out of sync

More thoughts when we actually try this..right now main things is to learn and use Zimbra and understand all the issues and good things

Raj
i2k2systems.com

[bjquinn]

Post back in this thread if you get any of that stuff to work... we're trying to do the same thing, all we want it for is disaster recovery. What would the risk be of not being "in sync"? I wonder if it would start erroring out all over the place, not being able to find emails referenced in the mySQL databases... or what if you backed up mySQL first and then the email next? So what if you lose two hours of email... you ALWAYS lose SOME email if the server goes down... now you just lose 2 hours + (amount of time since last backup).

Post back here anything interesting you find out, and we'll do the same. Hope we find something.

Oh, and a bit off subject, but since this is the only other thing I wanted the OSS version to do that only the Network version does... what about the MAPI connector? I wonder if you could buy licenses to just that, or if you could find and integrate an OSS MAPI connector?

-BJ Quinn

[rmvg]

I to am intrested in nightly backups for disasters let me know if there is anything i can do to help. Right now i am using bacula to do a hot backup once per night of a fully running zimbra server. The entire /opt/zimbra dir is being backed up.

Next i am going to do an fresh install on a different server and play around with adding bits of the backup to it.

[tim brennan]

Originally Posted by raj

Well yes paying is good and it gives you lot more than just hot backup and restore..i have not got any time to do that yet but logically you don’t need to SHUT DOWN open source version really to BACKUP zimbra..
I am just thinking logically..

MySQL and LDAP and FILE STORE so all you need to figure out is how to do

1) HOT BACKUP of mysql
2) dont know how LDAP store data if we can just COPY
3) email store..for sure its just file copy

* iam talking about full backup..plus if you using a backup software it willl do incremental too so you dont backup 100's of gigs every time..only changed files..

so some day i will pull an all nighter and have this answers...till then if someone else is up in night doing this do post..

Raj

You forgot to list the per-user search indexes in that list.

Remember, not only do you have to backup all these things, you have to keep them completely in-sync so you get a consistent snapshot of the data. Very Bad Things will happen if someone receives or deletes a message in the time between the backup of MySQL and the backup of the Blob store (messages) and the backup of the index.

Seriously folks -- live backup is a surprisingly hard problem and the issues can be very subtle. You'll probably end up getting something that works most of the time, but you really have no protection over getting a bad snapshot at the wrong time....