ldap doesn't fully start, and slapd totally hangs

[EricOnE]

When starting ZCM with ' zmcontrol start ' the typical service starting messages show up, however the very first one is ldap saying that it's started but this information is only from cache.... then everything else seems to start.

The problem is that ldap doesn't actually start, if you look at the system processes slapd is totally hung and pulling 100% of the processor.

So I have a few questions for anyone who's had ldap issues.

1. Where are the logs for the real zimbra service transactions at startup ? The ones in /opt/zimbra/logs/.... just have that same echoed message about the services all starting but the information is from cache and is basically worthless since it's not ' real time ' start information.

2. Where are the logs for ldap ?

3. Is there a debug mode for starting Zimbra to give verbose startup information ?

4. Is there a way to start ldap in debug mode so that verbose information on the startup is echoed to the terminal ?

[Jesster]

Originally Posted by EricOnE

1. Where are the logs for the real zimbra service transactions at startup ? The ones in /opt/zimbra/logs/.... just have that same echoed message about the services all starting but the information is from cache and is basically worthless since it's not ' real time ' start information.

I believe that that's the correct location

Originally Posted by EricOnE

2. Where are the logs for ldap ?

try /var/log/zimbra.log

Originally Posted by EricOnE

3. Is there a debug mode for starting Zimbra to give verbose startup information ?

There are logging levels with slapd, but i'm pretty certain the defaults are already extremely verbose (these defaults were changed in 5.0.6 i think?)

Originally Posted by EricOnE

4. Is there a way to start ldap in debug mode so that verbose information on the startup is echoed to the terminal ?

I'm not sure

[Jesster]

Originally Posted by EricOnE

The problem is that ldap doesn't actually start, if you look at the system processes slapd is totally hung and pulling 100% of the processor.

I actually saw this on one of my systems, and for us, it eventually "cleared up" after about 15 minutes.


Have you played with linux debugging tools? strace, lsof, et cetera.

[phoenix]

Originally Posted by EricOnE

The problem is that ldap doesn't actually start, if you look at the system processes slapd is totally hung and pulling 100% of the processor.

Some idea of operating system and Zimbra version & release would help. Please update your forum profile with the output of the following:

zmcontrol -v

. Is this a new install or an upgrade or has this 'just started happening'?

Have a look at the LDAP Troubleshooting and try the suggestions there.

[EricOnE]

It's running an older version: 4.5.10_GA on FC4 2.6.17-1.2142_FC4smp

This seems to have just started happening.

There isn't anything in the logs about ldap or slapd. The most that the system says is that it's starting , then says ldap has start but cautions you that it's only getting this information from cache and it may not be accurate... which indeed it's not.

Here's what I tried so far:
I tried /opt/zimbra/sleepycat/bin/db_recover and that just hangs like everything else.

Running as root:
/opt/zimbra/openldap/libexec/slapd -l LOCAL0 -4 -u zimbra -h ldap://mail.<domain>.com:389 -f /opt/zimbra/conf/slapd.conf -d 7

This gave me lots of verbose output ending in:

cn: Write
creatorsName: cn=config
modifiersName: cn=config
createTimestamp: 20081204140257Z
modifyTimestamp: 20081204140257Z
"
>>> dnPrettyNormal: <cn=Write,cn=Waiters,cn=Monitor>
=> ldap_bv2dn(cn=Write,cn=Waiters,cn=Monitor,0)
<= ldap_bv2dn(cn=Write,cn=Waiters,cn=Monitor)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(cn=Write,cn=Waiters,cn=Monitor)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(cn=write,cn=waiters,cn=monitor)=0
<<< dnPrettyNormal: <cn=Write,cn=Waiters,cn=Monitor>, <cn=write,cn=waiters,cn=monitor>
>>> dnNormalize: <cn=config>
=> ldap_bv2dn(cn=config,0)
<= ldap_bv2dn(cn=config)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(cn=config)=0
<<< dnNormalize: <cn=config>
>>> dnNormalize: <cn=config>
=> ldap_bv2dn(cn=config,0)
<= ldap_bv2dn(cn=config)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(cn=config)=0
<<< dnNormalize: <cn=config>
<= str2entry(cn=Write,cn=Waiters,cn=Monitor) -> 0x96291cc
backend_startup_one: starting ""
bdb_db_open:
bdb_db_open: unclean shutdown detected; attempting recovery.
bdb_db_open: dbenv_open(/opt/zimbra/openldap-data)

This looks like it was going to just try and repair the database like the first thing I tried.. so this hangs too.

The main issue is that either the database is so jacked up that the tools can't even deal with it enough to at least quit with an error, or ldap gets stuck on something that's just not being logged or I'm not seeing.

After this problem initially started someone tried an attempt to just upgrade Zimbra ( done before I got involved ), and this just froze at some point I'm told.

Any suggestions ?





I've even tried dumping some of the bdb files from openldap manually using /opt/zimbra/sleepycat/bin/db_dump but that just hangs too.