Zimbra Open Source Backup without stopping the server

[ruben.mourinho]

Hi!

This week I tried to backup my zimbra server to prepare me for a crash situation.
I'm using ubuntu 8.04 and the zimbra version is the latest freeware.

steps
1- I did an image with clonezila (after creating the users)
2- Send some emails, create some contacts and some meetings (just to test)
3- Backup the folder /opt/zimbra/store and the mysql database with the software Mysql Administrator
4- "Crash it" (or simply restore you harddrive with clonezila's image)
5- After restoring, delete the folder /opt/zimbra/store and replace it with your backup that should be daily (or whatever you want)
6- Restore the mysql database using the software Mysql Administrator (this must be after the step 5)

Ok, so, I know that the free version should not be backed up without stopping the server and I should not touch the database. But in fact this works...Did I have luck or this is suppose to work??

Comment please

[bdial]

when you do step 3 what is your timing like? like do you backup /opt/zimbra/store and then backup the mysql db? How would you handle the instance where after you've backed up a particular user's store foler they receive an e-mail before the db backup runs. In that case the db would have record of the e-mail but your store backup wouldn't have the e-mail

[ruben.mourinho]

Originally Posted by bdial

when you do step 3 what is your timing like? like do you backup /opt/zimbra/store and then backup the mysql db? How would you handle the instance where after you've backed up a particular user's store foler they receive an e-mail before the db backup runs. In that case the db would have record of the e-mail but your store backup wouldn't have the e-mail

That true..I run both at 2am and they are fast (database 30 sec and store 1-2 min)
I understand your point but if that is the only reason why the restore might fail I think that is a good risk to take. And my company doesn't receive many email and barely never at night. So for us is a good solution.

But if there is any other reason that can make the restoring fail, please tell me.

thanks

[bdial]

ok well it sounds like you know what you're getting into.

The 2 things you need to worry about when backing up filesystem only are the mysql db and lucene index. I don't know how mysql administrator works but it probably dumps the db to flatfiles. This is pretty much how zimbra's network edition backup does it too and this is a good way to do things.

I'm not sure if you can do the same kind of thing with the lucene index which is pretty much a db as well. Of course you can always reindex any accounts that have corrupted indexes.

The issues surrounding backing up filesystems with live databases is the event where you copy a db that is in the middle of a transaction. This is why zimbra puts accounts into maintenance mode while it backs up indexes. It sounds like this might nto happen often for you if you don't have much e-mail coming in while your backup is running.

Make sure you're backing up your ldap db as well if you're trying to do a full server disaster recovery backup

[ruben.mourinho]

Originally Posted by bdial

ok well it sounds like you know what you're getting into.

The 2 things you need to worry about when backing up filesystem only are the mysql db and lucene index. I don't know how mysql administrator works but it probably dumps the db to flatfiles. This is pretty much how zimbra's network edition backup does it too and this is a good way to do things.

I'm not sure if you can do the same kind of thing with the lucene index which is pretty much a db as well. Of course you can always reindex any accounts that have corrupted indexes.

The issues surrounding backing up filesystems with live databases is the event where you copy a db that is in the middle of a transaction. This is why zimbra puts accounts into maintenance mode while it backs up indexes. It sounds like this might nto happen often for you if you don't have much e-mail coming in while your backup is running.

Make sure you're backing up your ldap db as well if you're trying to do a full server disaster recovery backup

well..Now I'm a little worried. Where is the ldap db and how often does it change?
And can you tell me a little more about the lucene indexes in zimbra because I didn't find anything about it at the wiki.

Thanks a lot for your time

[zmailcc]

What is the reason for wanting to not stop the server? With the open source version, there is always a finite probability that a message will arrive during the backup of a live server and result in databases and files being out of sync; making a reliable restore point questionable.

The safest and easiest way to backup the open source version is to use one of several backup scripts found in the wiki or these forums. I found the LVM snapshot approach to be the best for our servers. I modified a script to fit our needs. It only requires the servers to be offline for about 80 seconds to stop the server, create an LVM snapshot, and restart the server. Most of that time is in stopping and starting Zimbra, the snapshot only takes a second. The snapshot can then be saved using tar or rsync and it is a perfect copy of the static /opt/zimbra directory. The actual backup of the snapshot to a tar.gz file takes about 2.5 hours on our server. The backup is done at 1:00AM and goes completely unnoticed. I have tested the restore operation several times on a backup server and once needed to use it to recover from a failed DRAM in our primary server -- it worked every time.

If you do not have /opt/zimbra on a logical volume, the rsync approach works good. There are some good scripts in the wiki for doing this. They first do an rsync of /opt/zimbra, then stop zimbra, do a second rsync to get what changed, and restart zimbra. On our server the second rsync on the stopped server takes about 5 to 10 minutes, which shows how much can change on a running server.

[raj]

Originally Posted by zmailcc

On our server the second rsync on the stopped server takes about 5 to 10 minutes, which shows how much can change on a running server.

What you do is you do rsync twice or 3 times before stopping and you will see this 5 min will be < 2 min or less

for us at 2 am everyone is sleeeeeeeeeping even on a busy server ..so 2 min is no big deal ..better than no backup.

Raj

[bdial]

Originally Posted by ruben.mourinho

well..Now I'm a little worried. Where is the ldap db and how often does it change?
And can you tell me a little more about the lucene indexes in zimbra because I didn't find anything about it at the wiki.

Thanks a lot for your time

The ldap data can be dumped easily by the command

/opt/zimbra/libexec/zmslapcat /path/to/ldap/backup/dir

that will dump the ldap db to a flat file just like you do with mysql.

The indexes are held in /opt/zimbra/index. I'm not sure where or if theres more documentation on them.

[zmailcc]

Originally Posted by raj

What you do is you do rsync twice or 3 times before stopping and you will see this 5 min will be < 2 min or less

for us at 2 am everyone is sleeeeeeeeeping even on a busy server ..so 2 min is no big deal ..better than no backup.

Raj

Thanks for the tip Raj. That makes the rsync approach very attractive.

I have used the LVM snapshot backup approach since I setup my first Zimbra server using Zimbra 4.0. The forums and wiki really helped -- many of the posts were yours. I use an rsync based script to make a backup just before applying upgrades and will add the second live rsync to it.

I mainly wanted to show that a lot can change on a live server while files and databases are being copied to a backup; making it risky to make a backup of the open source edition without stopping the server. I imagine the Network Edition has extra logic to keep database and message stores in sync during a live backup.

[Mistoffeles]

Originally Posted by ruben.mourinho

well..Now I'm a little worried. Where is the ldap db and how often does it change?

As far as I know LDAP data only changes when you make changes to the user list, such as adding/editing/deleting user accounts or adding/editing/deleting domains. This shouldn't happen too much if you just have one domain and all edits to the addressbook are done by administrators.