After server crash, zimbra/ldap fails to restart

[billybofh]

The test box I was running on was power-cycled over the weekend (more than once) and now zimbra/ldap doesn't start. I've dug around the forums a little but can't see this exact problem being replicated.

The basic error I'm seeing is :

/etc/init.d/zimbra start
Host xxxxxx.xxx.xxx
        Starting ldap...Done.
FAILED
db_recover: Finding last valid log LSN: file: 1 offset 2792480
db_recover: Recovery starting from [1][2792356]
db_recover: Recovery complete at Tue Jan 31 11:53:38 2006
db_recover: Maximum transaction ID 8000000b Recovery checkpoint [1][2792480]
ERROR - failed to start slapd

From digging around some other threads I tried running slapd by hand with debug on and get the attached output. I'm guessing that the berkeleydb files are corrupt in some way? Maybe....


Billy.

[marcmac]

backend_startup: starting ""
bdb_db_open: dbenv_open(/opt/zimbra/openldap-data)
bdb(): unable to initialize mutex: Function not implemented
bdb(): /opt/zimbra/openldap-data/__db.001: unable to initialize environment lock: Function not implemented
bdb_db_open: dbenv_open failed: Function not implemented (38)
backend_startup: bi_db_open failed! (38)


You're either running on a filesystem that doesn't support locking, or you're linking to the wrong sleepycat library.

ldd the slapd binary, see what happens. Also, have the contents of /etc/ld.so.conf changed?

[billybofh]

ldd gives me :

[root@ ~]# ldd /opt/zimbra/openldap/libexec/slapd
        libsasl2.so.2 => /opt/zimbra/cyrus-sasl/lib/libsasl2.so.2 (0x4000f000)
        libssl.so.4 => /lib/libssl.so.4 (0x00907000)
        libcrypto.so.4 => /lib/libcrypto.so.4 (0x007e3000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x00654000)
        libpthread.so.0 => /lib/i686/libpthread.so.0 (0x40024000)
        libc.so.6 => /lib/i686/libc.so.6 (0x40076000)
        libdl.so.2 => /lib/libdl.so.2 (0x00c53000)
        libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x008f1000)
        libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x0066e000)
        libcom_err.so.2 => /lib/libcom_err.so.2 (0x00669000)
        libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x008ce000)
        libz.so.1 => /usr/lib/libz.so.1 (0x00d32000)
        /lib/ld-linux.so.2 (0x00ae9000)

Nothing else has changed to my knowledge...

[marcmac]

This apply to you?

installed, but won't run FC3

[billybofh]

No - this is a default CentOS workstation install, no virtual machines, no unusual file-systems just regular ext3. It was working fine for about a week before it stopped.

[marcmac]

What changed before it failed? Did you upgrade anything on the host?

[billybofh]

Nothing changed on the system as far as I know. No yum updates or software was installed or removed.

[marcmac]

Something has changed, if it worked before and doesn't now.

What's in /etc/ld.so.conf or /etc/ld.so.conf.d/zimbra?

[billybofh]

# cat /etc/ld.so.conf
include ld.so.conf.d/*.conf

# cat /etc/ld.so.conf.d/zimbra.ld.conf
/opt/zimbra/lib
/opt/zimbra/sleepycat/lib
/opt/zimbra/openldap/lib
/opt/zimbra/cyrus-sasl/lib

If it's any help, the end of a strace on slapd gives :

open("/opt/zimbra/conf/slapd.key", O_RDONLY) = 8
fstat64(8, {st_mode=S_IFREG|0664, st_size=887, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40136000
read(8, "-----BEGIN RSA PRIVATE KEY-----\n"..., 4096) = 887
close(8)                                = 0
munmap(0x40136000, 4096)                = 0
open("/opt/zimbra/conf/slapd.crt", O_RDONLY) = 8
fstat64(8, {st_mode=S_IFREG|0664, st_size=1151, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40136000
read(8, "-----BEGIN CERTIFICATE-----\nMIID"..., 4096) = 1151
getpid()                                = 20582
close(8)                                = 0
munmap(0x40136000, 4096)                = 0
rt_sigaction(SIGSTKFLT, {0x482dc020, [], SA_RESTORER|SA_RESTART, 0x47ffc898}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGSYS, {0x482dc020, [], SA_RESTORER|SA_RESTART, 0x47ffc898}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGHUP, {0x482dc020, [], SA_RESTORER|SA_RESTART, 0x47ffc898}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGINT, {0x482dc020, [], SA_RESTORER|SA_RESTART, 0x47ffc898}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGTERM, {0x482dc020, [], SA_RESTORER|SA_RESTART, 0x47ffc898}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGTRAP, {0x482dc020, [], SA_RESTORER|SA_RESTART, 0x47ffc898}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGCHLD, {0x482dc020, [], SA_RESTORER|SA_RESTART, 0x47ffc898}, {SIG_DFL}, 8) = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
fork()                                  = 20583
exit_group(0)                           = ?
Process 20582 detached
ERROR - failed to start slapd

[marcmac]

Anything in /opt/zimbra/redolog/redo.log? You can try moving that to the side and restart.