Open Source disaster recovery procedure

[dmcentire]

I help with two Zimbra installations, one NE edition and one open source edition (for two different companies). The NE is great with the backup system, but the OS version seems to be a bit complex when it comes to making backup scripts and such. With that said:

The open source Zimbra box had a hard drive failure. The backup schema was just simply "copy the entire /opt/zimbra directory" to another drive. We thought that would be enough to recover data in the event of a failure. But we seem to be wrong, very wrong.

Is there any kind of procedure written to recover a system like this? Here's what I have tried, and they all have failed:

1. Install the same OS and the same version of Zimbra on a box, then shutdown services, rename the zimbra dir. to something else, copy the /opt/zimbra directory from the backup, do the zmfixperms thing, and reboot. Zimbra gets all kinds of errors and doesn't start properly.

2. Install the same OS, copy the /opt/zimbra dir. from the backup into the proper place, then run an install.sh to "upgrade" the installation to get it working again. Also fails miserably.

Any help on this would be great. We currently have a semi-workaround, we are just using a new install of Zimbra on a new box and we re-setup the accounts so incoming mail will at least come in. But it would be nice to restore our people's faith in Zimbra by recovering their old mailboxes.

Thank you, and we're using Zimbra Open Source Edition 6.0.4

Dennis

[phoenix]

Both of those routines for restoring should work, with the proviso that you have the same host name; the same IP; and DNS A & MX records pointing to your server. I've done that many times with the open source version without problems. Without any details of what errors you've had it's difficult to offer any advice on what the problem was.

This isn't meant as a comment on what you've done but rather for posterity (and anyone else that stumbles in here) that a disaster recover procedure should be tested before you actually need it. Have you read some of the threads in the forums (and the wiki) on disaster recovery?

[dmcentire]

root@postoffice:/opt/zimbra/bin# su zimbra
zimbra@postoffice:~/bin$ ./zmcontrol start
Host postoffice.mydomain.com
Starting ldap...Done.
Starting logger...Done.
Starting mailbox...Failed.

Starting memcached...Done.
Starting imapproxy...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Failed.
Starting zmmtaconfig...failed.
zmmtaconfigctl failed to start
Starting saslauthd...done.

Starting stats...Done.

============================

I'm not sure which log you would like, but let me know. The above happens with procedure #1 outlined in my first post.

Thanks

[dmcentire]

Still not have been able to recover any old data.

I'd like to:

1. Install a new installation on a new server.
2. Copy some directories from the backup.
3. Recover data after restarting.

Does anyone have an idea of which directories I should copy? Here's the ones I think need to move over:

/opt/zimbra/conf/
/opt/zimbra/data/
/opt/zimbra/db/
/opt/zimbra/openldap/
/opt/zimbra/store/

Thanks for any help on this,

Dennis

[dmcentire]

I got the Zimbra setup to start with copies of the directories in the previous post. However, something new has happened.

I was able to start ALL the servers and they do not fail.

However, here's something weird:

1. There are no "java" processes running serving webpages on port 80, 443, and 7071.
2. There is a process called zmjava but it's not listening on any ports.
3. The web clients are not available, of course, nothing's listening.

[dmcentire]

Not sure what to do at this point, here's my next attempt:

1. I am downloading the Open Source 6.0.6 version.
2. I will attempt an "upgrade" of the current system.

Hopefully this will get the system installed and working properly with the old data. I'll update this with any information that might be helpful.

Is anybody reading these posts?

[dmcentire]

Upon upgrading to 6.0.6, this happened:

Restoring existing configuration file from /opt/zimbra/.saveconfig/localconfig.xml...done
Operations logged to /tmp/zmsetup.06092010-162837.log
Upgrading from 6.0.4_GA_2038 to 6.0.6_GA_2330
Stopping zimbra services...done.
Verifying /opt/zimbra/conf/my.cnf
Starting mysql...done.

ERROR 1033 (HY000) at line 1: Incorrect information in file: './zimbra/config.frm'
Wed Jun 9 16:28:55 2010: Error while running '/opt/zimbra/bin/mysql --user=zimbra --password=vgoJEafIBeDjrkCNdAo.C3I --database=zimbra --batch --skip-column-names'.

[dmcentire]

Now trying to go back to 6.0.4 ...

Checking for existing installation...
zimbra-ldap...FOUND zimbra-ldap-6.0.6_GA_2330.UBUNTU8
zimbra-logger...FOUND zimbra-logger-6.0.6_GA_2330.UBUNTU8
zimbra-mta...FOUND zimbra-mta-6.0.6_GA_2330.UBUNTU8
zimbra-snmp...FOUND zimbra-snmp-6.0.6_GA_2330.UBUNTU8
zimbra-store...FOUND zimbra-store-6.0.6_GA_2330.UBUNTU8
zimbra-apache...FOUND zimbra-apache-6.0.6_GA_2330.UBUNTU8
zimbra-spell...FOUND zimbra-spell-6.0.6_GA_2330.UBUNTU8
zimbra-convertd...NOT FOUND
zimbra-memcached...FOUND zimbra-memcached-6.0.6_GA_2330.UBUNTU8
zimbra-proxy...FOUND zimbra-proxy-6.0.6_GA_2330.UBUNTU8
zimbra-archiving...NOT FOUND
zimbra-cluster...NOT FOUND
zimbra-core...FOUND zimbra-core-6.0.6_GA_2330.UBUNTU8

Downgrading to version 6.0.4 from 6.0.6 is not supported.

[dmcentire]

I am surprised there is no answer to the question of:

How do I recover email from a complete backup of the /opt/zimbra directory from a live server?

I guess copying the entire /opt/zimbra directory is not a good option for backing up the open source edition.

[raj]

Hi..i guess no one is replying because you have changed a lot of stuff by upgrading the new install..where as you need to do small steps at a time.

stopping services and full backup of /opt/zimbra folder IS the best way to full disaster recovery backup of zimbra open source or for that matter NE edition too in some cases

Moving ZCS to Another Server » Zimbra :: Blog
Above link shows all the steps you need to do to move to new server be it a simple move or disaster recover from full opt/zimbra backup

* but assumption here is that you did the BACKUP the correct way..by following any of the recommended options
Open Source Edition Backup Procedure - Zimbra :: Wiki

if the orignal backup is no good then the whole moving/recovery of new server install will me a mess

Raj