Zimbra service wont start after shutdown.

[millerdc]

Server details.

OS: RHEL5.4 x64
version of zimbra: 6.0.7_GA_2473.RHEL5_64 RHE5_64 FOSS edition

This saturday night 10-09-2010 I powered off the server for a scheduled power outage. The next morning when I turned the server back on it would not get past the starting ldap. It puts up the following error.

[root@email ~]# service zimbra start
Host email.gat.com
Starting ldap...Done.
Failed.
Failed to start slapd. Attempting debug start to determine error.
bdb_db_close: database "": txn_checkpoint failed: Invalid argument (22).
backend_startup_one (type=hdb, suffix=""): bi_db_open failed! (22)
bdb_db_close: database "": alock_close failed

Here is what the /var/log/zimbra shows when attempting to start

Oct 10 14:04:32 email slapd[10384]: @(#) $OpenLDAP: slapd 2.4.22 (Jun 10 2010 14:02:07) $ build@build11.lab.zimbra.com:/home/build/p4/main/ThirdParty/openldap/openldap-2.4.22.3z/servers/slapd
Oct 10 14:04:32 email slapd[10384]: hdb_db_open: database "" cannot be opened, err 22. Restore from backup!
Oct 10 14:04:32 email slapd[10384]: bdb(): txn_checkpoint interface requires an environment configured for the transaction subsystem
Oct 10 14:04:32 email slapd[10384]: bdb_db_close: database "": txn_checkpoint failed: Invalid argument (22).
Oct 10 14:04:32 email slapd[10384]: backend_startup_one (type=hdb, suffix=""): bi_db_open failed! (22)
Oct 10 14:04:32 email slapd[10384]: bdb_db_close: database "": alock_close failed
Oct 10 14:04:32 email slapd[10384]: slapd stopped.


About backups. I make a nightly snapshot of the server @1:45AM. It is based off a method posted here in the forums. The way the snapshot works is that I have a script that does a /sbin/service zimbra stop. It then does an rsync of the /opt/zimbra to another computer. It also rsyncs the / partition minus /opt /proc /sys and /dev to the same computer under /opt/os-backup. So effectivly I have a complete snapshot of the the server in a working state. The night before the shutdown it stopped the processes synced and restarted them without issue.

I searched the zimbra forums for "bdb_db_close: database "": txn_checkpoint failed: Invalid argument (22)." but everything I find seems to be out of date as the solutions to do is posted. Can anyone help me out here?

My next step is to restore from the snapshot on the 8th.

[phoenix]

The answers in the forums are valid for your problem: site:zimbra.com +solved +"txn_checkpoint failed: Invalid argument" - Yahoo! Search Results

[millerdc]

All the solutions do not match the path structure for Zimbra 6.x. They all assume version 5.x or older. For example the solution in one of the solved threads say to do this.

 cd /opt/zimbra/openldap-data
/opt/zimbra/sleepycat/bin/db_recover

I don't have a "/opt/zimbra/sleepycat/bin/db_recover" Or "/opt/zimbra/openldap-data"

When I do a locate for db_recover I get..

/opt/zimbra/bdb-4.7.25.4/bin/db_recover

When I do a locate for openldap-data I get..

/opt/zimbra/openldap-2.4.22.3z/var/openldap-data

Should I do the following?

su - zimbra

cd /opt/zimbra/openldap-2.4.22.3z/var/openldap-data

/opt/zimbra/bdb-4.7.25.4/bin/db_recover

[millerdc]

It appears that in version 6.x the ldap info is now stored in the following location.

/opt/zimbra/data/ldap/hdb/db

Do I need to move into that directory and then run the db_recover?

I have rsynced over the snapshot from the night before shutdown and I'm still getting the same error when trying to start Zimbra. I don't understand why the backup did not work. Assuming the database was corrupted before the backup I would expect Zimbra to have failed to start after the backup ran and the script started the Zimbra processes again. The data I rsynced over was what Zimbra successfully started with. So the only difference was the computers were shutdown about 15 hours later after the backup. I checked the dates of files on the backup and there is nothing older than the night it did the sync before the power shutdown.

[millerdc]

OK, apparently that /opt/zimbra/data/ldap/hdb/db directory was it. My server is now back up and running. I think I removed about 10 years off my live this weekend. I hope this kicks management in the butt to actually pay for the supported version.