Disaster recovery test.

[Dirk]

So I figured a good thing to do today would be to take the backup folder from the current Zimbra server and try to build a new server from it.

I moved the current backups from the backup folder and took a fresh full backup to make my life a little easier and once the backup was done I put it somewhere safe and pretended like I could not access the old (live production) server anymore.

I built a new box and installed zimbra on it as per the disaster recovery document in the wiki. It had something of a hiccup when it came to restoring the data due to a mismatched LDAP password. I have no idea what the old LDAP password was so I searched around on these forums and tried a few things, I didnt take notes but eventually the restoreoffline command was accepted and after a delay, everything seemed ok.

One quick reboot of the test server later and I should have a fully functional Zimbra server, right?

Well, not quite.

The mail.log file shows the following:

Oct  1 14:54:52 zimbra postfix/proxymap[9713]: warning: dict_ldap_connect: Unable to bind to server ldap://zimbra.domain.com:389 as uid=zmpostfix,cn=appaccts,cn=zimbra: 49 (Invalid credentials)
Oct  1 14:54:52 zimbra postfix/cleanup[7821]: warning: 6751085AC7: sender_canonical_maps map lookup problem for zimbra@zimbra.domain.com
Oct  1 14:54:52 zimbra postfix/pickup[7817]: 6C01C85AC7: uid=1001 from=<zimbra>
Oct  1 14:54:52 zimbra postfix/proxymap[9713]: warning: dict_ldap_connect: Unable to bind to server ldap://zimbra.domain.com:389 as uid=zmpostfix,cn=appaccts,cn=zimbra: 49 (Invalid credentials)
Oct  1 14:54:52 zimbra postfix/cleanup[7821]: warning: 6C01C85AC7: sender_canonical_maps map lookup problem for zimbra@zimbra.domain.com

This looked like it was linked to the ldap password problem, so I did what I may well do if this were real; I fired up the installer again in the hope that magic would happen and it would all work. There's no point spending hours researching it while the mail is down and users are complaining if the installer may do the trick.

So, the installer runs through, removes packages, installs packages, reads & writes config files and all looks great.

I did see an error in the install process which said:

Setting up CA...sh: openssl: not found
done.

which looks like something that aptitude install openssl may resolve, either way the installer went on to complete.

Once it had finished, it all seemed ok, the mail.log no longer contained those errors so I guess it's a fully functional server. Wooo !! Success


So, ask everyone; have you tried this?
Anyone who is in charge of a Network Edition ZCS should ensure that they have what they need to be able to restore their server from a tape (or whatever /opt/zimbra/backup is stored on)

I feel that the process could be simplified. It's ok doing it as a trial run, but if the mailserver really did go on fire and all I had was a tape, I'd prefer a simpler method of restore.

Can we get some real-world comments from the field, could people with real installs practice their recovery process and chime in with how it went?

[tecnalb]

I have been performing the same thing and not had much luck. I was making preparation to move my existing OLD server over to a new box. I've spent a week with it now and it's still does not flow well or function correctly.

I was able to get through the restore process (without any mailbox.log working, BTW) and also found my various ldap passwords after reading a few notes and digging around in the blogs.

In the end I was able to get the store started but could not send messages. I'm back to square one today. I find it troubling that no Zimbra employee has responded to messages in the forums about this. Like you mentioned, this could be a very troublesome operation in a real failure, and if you haven't tried it... you better!! It's not easy to get working again.

[tecnalb]

I did finally get this to work. What I'm going to do now is take a snapshot of my /opt/zimbra directory, minus the backups and /opt/zimbra/db/data, then drop on on a box and follow the procedure again. I would hope this helps me to have a complete setup with passwords already in place.

[Dirk]

I confess to being a little disappointed that noone else has commented on this subject

I know it's easier to assume the fault tolerant hard drives and dual online servers etc will be enough; and that's very true.... until 'that day' happens!

[tecnalb]

Yes, it is an issue that should be taken a little more seriously. Anyway, I've done three restores now and they seemed to work but I noticed something. When I go into the admin portal, I find that under the TOOLs section, the backup and certificate tools are gone. Also, during my restore I found that ONE mailbox did not restore. The mailbox exists, but no content. I found this in the mailbox log:

I substituted user@domain for the actual account

2008-10-07 09:47:00,962 INFO [main] [name=user@domain;mid=7;] backup - R
estore started for account user@domain (188e2beb-6777-4923-a7fe-33023c6ae
f88) original mailbox id 24
2008-10-07 09:47:00,966 WARN [main] [name=user@domain;mid=7;] backup - N
ot restoring because account user@domain did not exist or has been delete
d
2008-10-07 09:47:00,966 INFO [main] [name=user@domain;mid=7;] backup - A
ccount user@domain: All pending file IO completed (0 out of 0)
2008-10-07 09:47:00,966 INFO [main] [name=user@domain;mid=7;] backup - R
estore finished for user@domain (188e2beb-6777-4923-a7fe-33023c6aef88) or
iginal mailbox 24 to mailbox 24


This happened from two different FULL backups and two different accounts.xml for ldap restore.