ldap id2entry.bdb has bad LSN

[pixelplumber]

Hi, we have had the RHEL4 32bit OSS zimbra running in an OpenVZ container sucessfully for the last four years, and I've just migrated the virtual machine to a new server. Unfortunately I think there may have been some corruption in transit as zimbra fails to startup with the following repeated in the zimbra.log

Nov 18 20:03:33 mail slapd[8250]: bdb(): file id2entry.bdb has LSN 20/7544382, past end of log at 20/7535686
Nov 18 20:03:33 mail slapd[8250]: bdb(): Commonly caused by moving a database from one database environment
Nov 18 20:03:33 mail slapd[8250]: bdb(): to another without clearing the database LSNs, or by removing all of
Nov 18 20:03:33 mail slapd[8250]: bdb(): the log files from a database environment
Nov 18 20:03:33 mail slapd[8250]: bdb(): /opt/zimbra/data/ldap/hdb/db/id2entry.bdb: unexpected file type or format
Nov 18 20:03:33 mail slapd[8250]: hdb_db_open: database "": db_open(/opt/zimbra/data/ldap/hdb/db/id2entry.bdb) failed: Invalid argument (22).
Nov 18 20:03:33 mail slapd[8250]: backend_startup_one (type=hdb, suffix=""): bi_db_open failed! (22)
Nov 18 20:03:33 mail slapd[8250]: bdb_db_close: database "": alock_close failed
Nov 18 20:03:33 mail slapd[8250]: slapd stopped.

I've looked at a few related posts:
[SOLVED] LDAP / slapd - Database environment corrupt (Issue & Solution)
http://www.zimbra.com/forums/install...n-t-start.html

and the reccommended next step is to run db_recover, however I just want to check if there are any differences in Zimbra 6.x as these posts are several years old. For example I can't find db_recover in the sleepycat directory it is now in the jetty directory.

Any advice you can give is appreciated, I don't mind trying a few things as if I hose the install I can recover the VM and try something else.

I have a week old backup of the VM that I can use as a last resort but bringing over this weeks mail is something I would rather avoid.

[pixelplumber]

I've tried running the zmfixperms -verbose -extended as root, and then doing a db_recover as the zimbra user, however I'm getting errors as follows:

$ /opt/zimbra/bdb-4.7.25.4/bin/db_recover
db_recover: unlink: __db.003: Permission denied
db_recover: unlink: __db.002: Permission denied
db_recover: unlink: __db.004: Permission denied
db_recover: unlink: __db.005: Permission denied
db_recover: unlink: __db.001: Permission denied

My openladp-data directory post zmfixperms looks like this:

$ ls -ld /opt/zimbra/openldap-data
drwxr-xr-x  4 root root 4096 Mar 30  2009 /opt/zimbra/openldap-data

$ ls -l /opt/zimbra/openldap-data
total 69252
-r--r-----  1 zimbra zimbra      445 Jan 12  2008 DB_CONFIG
-rw-------  1 zimbra zimbra    16384 Mar 30  2009 __db.001
-rw-------  1 zimbra zimbra 65544192 Mar 30  2009 __db.002
-rw-------  1 zimbra zimbra  2359296 Mar 30  2009 __db.003
-rw-------  1 zimbra zimbra   991232 Mar 30  2009 __db.004
-rw-------  1 zimbra zimbra    16384 Mar 30  2009 __db.005
-rw-r--r--  1 zimbra zimbra     2048 Sep 11 19:52 alock
-rw-------  1 zimbra zimbra    53248 Sep  7 18:43 cn.bdb
drwxr-xr-x  2 zimbra zimbra     4096 Jan 12  2008 db
-rw-------  1 zimbra zimbra    28672 Sep  7 18:43 displayName.bdb
-rw-------  1 zimbra zimbra    94208 Sep 11 18:57 dn2id.bdb
-rw-------  1 zimbra zimbra     8192 Sep 11 18:57 entryCSN.bdb
-rw-------  1 zimbra zimbra    16384 Sep  7 16:53 entryUUID.bdb
-rw-------  1 zimbra zimbra    16384 Sep  7 18:43 givenName.bdb
-rw-------  1 zimbra zimbra  1064960 Sep 11 18:57 id2entry.bdb
-rw-r-----  1 zimbra zimbra   275015 Mar 30  2009 ldap.bak
drwxr-xr-x  2 zimbra zimbra     4096 Nov 18 20:57 logs
-rw-------  1 zimbra zimbra    69632 Sep 11 19:07 mail.bdb
-rw-------  1 zimbra zimbra    28672 Sep 11 18:57 objectClass.bdb
-rw-------  1 zimbra zimbra    24576 Sep  7 18:43 sn.bdb
-rw-------  1 zimbra zimbra    12288 Sep  7 23:03 uid.bdb
-rw-------  1 zimbra zimbra     8192 Mar 28  2008 zimbraCalResBuilding.bdb
-rw-------  1 zimbra zimbra     8192 Mar 28  2008 zimbraCalResCapacity.bdb
-rw-------  1 zimbra zimbra     8192 Mar 28  2008 zimbraCalResFloor.bdb
-rw-------  1 zimbra zimbra     8192 Mar 28  2008 zimbraCalResRoom.bdb
-rw-------  1 zimbra zimbra     8192 Mar 28  2008 zimbraCalResSite.bdb
-rw-------  1 zimbra zimbra     8192 Sep 11 18:57 zimbraDomainName.bdb
-rw-------  1 zimbra zimbra     8192 Sep 11 18:57 zimbraId.bdb
-rw-------  1 zimbra zimbra    45056 Sep 11 18:57 zimbraMailAlias.bdb
-rw-------  1 zimbra zimbra     8192 Sep 11 18:57 zimbraMailCatchAllAddress.bdb
-rw-------  1 zimbra zimbra    40960 Sep 11 18:57 zimbraMailDeliveryAddress.bdb
-rw-------  1 zimbra zimbra     8192 Sep 10 13:28 zimbraMailForwardingAddress.bdb
-rw-------  1 zimbra zimbra     8192 Aug 21 09:27 zimbraMailTransport.bdb
-rw-------  1 zimbra zimbra     8192 Sep  7 17:43 zimbraVirtualHostname.bdb

Should the ownership/perms be different on the openldap-data directory? The zimbra user doesn't have any write perms to the directory atm.

and my version of db_recover is:
$ /opt/zimbra/bdb-4.7.25.4/bin/db_recover -V
Berkeley DB 4.7.25: (May 15, 2008)

[pixelplumber]

running db_recover with -v I get:

$ /opt/zimbra/bdb-4.7.25.4/bin/db_recover -vv
db_recover: unlink: __db.003: Permission denied
db_recover: unlink: __db.002: Permission denied
db_recover: unlink: __db.004: Permission denied
db_recover: unlink: __db.005: Permission denied
db_recover: unlink: __db.001: Permission denied
Finding last valid log LSN: file: 222 offset 120
Recovery complete at Wed Nov 18 22:15:25 2009
Maximum transaction ID 0 Recovery checkpoint [222][120]

And now slapd & zimbra starts woohoo!

Still if anyone can explain the permission denied errors it would help my understanding.

[ArcaneMagus]

Looks like the recovery tool was trying to delete (unlink) the __db.00x files. Since the parent directory was owned by root, and did not have the "sticky" bit set, the tool running as zimbra did not have rights to delete the files, only modify them.

[pixelplumber]

Are those _db files what it uses to recover the database? I'm glad it can't delete them then.

[j2b]

Just come to the same problem, only db_recover stated, that can not read log.0000... file. Permissions denied.

Did you have any solutions up till now, or you just recovered from backup?