complete data loss due to corrupted LDAP BDB files?

[atseog]

Hi folks,

one of my customers' zimbra server was restarted due to power loss recently. The machine runs zimbra "Release 7.1.0_GA_3140.UBUNTU8_64 UBUNTU8_64 FOSS edition." on ubuntu 8.04 LTS. The bad news: there is no backup and there is no UPS.

So much for the unchangeable facts.

Now the symptoms:

Starting zimbra fails at the attempt to start ldap. The zimbra.log says:

Aug 10 16:01:31 zimbra slapd[27737]: @(#) $OpenLDAP: slapd 2.4.23 (Mar  4 2011 19:31:58) $ ^Iroot@zre-ubuntu8-64.eng.vmare.com:/home/build/p4/HELIX/ThirdParty/openldap/openldap-2.4.23.6z/servers/slapd 
Aug 10 16:01:31 zimbra slapd[27737]: str2entry: entry -1 has no dn 
Aug 10 16:01:31 zimbra slapd[27737]: slapd stopped. 
Aug 10 16:01:31 zimbra slapd[27737]: connections_destroy: nothing to destroy.

When I try to start slapd a bit more verbosely an irritating new issue arises:

zimbra@zimbra:~$ sudo /opt/zimbra/libexec/zmslapd -f /opt/zimbra/openldap/etc/openldap/slapd.conf -d 256                 
@(#) $OpenLDAP: slapd 2.4.23 (Mar  4 2011 19:31:58) $
	root@zre-ubuntu8-64.eng.vmare.com:/home/build/p4/HELIX/ThirdParty/openldap/openldap-2.4.23.6z/servers/slapd
Unrecognized database type (bdb)
/opt/zimbra/openldap/etc/openldap/slapd.conf: line 53: <database> failed init (bdb)
slapd stopped.
connections_destroy: nothing to destroy.

Recipes that do *not* work for me:

There are several threads around here, pointing out how to:
* recover from backups (obviously not an option)
* db_recover from zimbras packaged bdb tools happily returns "0" and changes not a single file, when run inside /opt/zimbra/data/ldap/hdb/db
* dumping and regenerating the bdb files with db_dump - db_load (as proposed on a bdb mailing list) seems to modify some files (they slightly shrink in size) but fails to make slapd happy too.
* of course slapd has bdb support _not_ compiled as a module (this is the standard reply from openldap folks) and the configuration is exactly the same as it is in a fresh zimbra installation.

I even tried to reinstall zimbra from the original tarball. The installer finds the existing ldap database, tries to recover, fails and then happily loops over the steps start ldap - fail - try to recover - ignore failure - start ldap - ...

No harm seems to be done to the mailstore. There are dozens of message within the 'store' directory and they seem to be intact. I could even write a script, collecting the files, sorting them by recipient and creating a maildir od maibox structure per user. This could be imported to a newly installed zimbra server. But users would loose their folders within the inbox, since that information is exclusively stored in ldap.

Is it true, that a corrupted bdb file prevents me from using this system ever again? There must be a way to fix that file, or at least extract the remaining data.

Any hints are welcome ...

[atseog]

Well, since nobody seemed to be able to help, I finally solved it myself.

Here is what I found out:

After starting slapd under strace, I saw that it complained about a file named ''/opt/zimbra/data/ldap/config/cn=config.ldif'', which, obviously due to the power loss, was empty.

Checking the contents of the same file on a different Zimbra server, I decided to try and start with a copy of that file on the local machine ...

Then there was another missing file (strace comes in handy some times ;-) ): ''/opt/zimbra/data/ldap/config/cn=config/olcDatabase={2}hdb.ldif''.

This file was also copied from a working zimbra instance and: slapd started!

So, my advice to all of you, who find "str2entry: entry -1 has no dn" in zimbra.log try this at home:

root@zimbra:~# strace -eopen /opt/zimbra/libexec/zmslapd -l LOCAL0 -u zimbra -h "ldapi:///" -F /opt/zimbra/data/ldap/config -d 1

greetings