ZimbraLDAP could not start completely

[mikiso]

Hi,

Our customer reported a problem with starting LDAP.

They restarted LDAP via zmcontrol stop/start in order to backup
BDB on LDAP.
1. zmcontrol stop
2. cp -r /opt/zimbra/open-ldap /opt/zimbra/open-ldap.bak20100902
3. zmcontrol start

It seemed ZimbraLDAP started from logs, however our customer
said the server was unresponsive when executing ldap search.

They also said that when the server was unresponsive, there were
around 100 connections whose status were "ESTABLISHED" with store servers.

Could you figure out the problem?
I think the connections from store server could exclude the ldapsearch,
but I'm not sure why so many connections were established. They
didn't use ldap connection pool.


Here is the zimbra.log:

Aug 22 03:11:51 ldapm-p zimbramon[2756]: 2756:info: Starting services initiated by zmcontrol
Aug 22 03:11:52 ldapm-p slapd[2919]: @(#) $OpenLDAP: slapd 2.3.43 (Apr 13 2009 10:47:58) $ build@build11.lab.zimbra.com:/home/build/p4/FRANKLIN/ThirdParty/openldap/openldap-2.3.43.8z/servers/slapd
Aug 22 03:11:53 ldapm-p slapd[2920]: slapd starting
Aug 22 03:12:31 ldapm-p zimbramon[2756]: 2756:info: Rewriting configs
Aug 22 03:12:31 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: zmmtaconfig started on mz-ld01.abc.ne.jp with loglevel=3 pid=3227
Aug 22 03:12:31 ldapm-p zimbramon[2756]: 2756:info: Starting stats via zmcontrol
Aug 22 03:12:32 ldapm-p sendmail[3455]: NOQUEUE: SYSERR(zimbra): can not chdir(/var/spool/mqueue/): Permission denied
Aug 22 03:12:32 ldapm-p zimbramon[3010]: 3010:info: 2010-08-22 03:12:01, STATUS: mz-ld01.abc.ne.jp: ldap: Running
Aug 22 03:12:32 ldapm-p zimbramon[3010]: 3010:info: 2010-08-22 03:12:01, STATUS: mz-ld01.abc.ne.jp: stats: Stopped
Aug 22 03:13:03 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping Global system configuration update.
Aug 22 03:13:03 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: gacf ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.naming.NamingException LDAP response read timed out, timeout used:30000ms.)
Aug 22 03:13:34 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping All Reverse Proxy URLs update.
Aug 22 03:13:34 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping getAllReverseProxyURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.naming.NamingException LDAP response read timed out, timeout used:30000ms.)
Aug 22 03:14:05 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping All Reverse Proxy Backends update.
Aug 22 03:14:05 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping getAllReverseProxyBackends ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.naming.NamingException LDAP response read timed out, timeout used:30000ms.)
Aug 22 03:14:33 ldapm-p sendmail[5837]: NOQUEUE: SYSERR(zimbra): can not chdir(/var/spool/mqueue/): Permission denied
Aug 22 03:14:33 ldapm-p zimbramon[4835]: 4835:info: 2010-08-22 03:14:01, STATUS: mz-ld01.abc.ne.jp: ldap: Running
Aug 22 03:14:33 ldapm-p zimbramon[4835]: 4835:info: 2010-08-22 03:14:01, STATUS: mz-ld01.abc.ne.jp: stats: Running
Aug 22 03:14:36 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping All Memcached Servers update.
Aug 22 03:14:36 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping getAllMemcachedServers ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.naming.NamingException LDAP response read timed out, timeout used:30000ms.)
Aug 22 03:15:07 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping All MTA Authentication Target URLs update.
Aug 22 03:15:07 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping getAllMtaAuthURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.naming.NamingException LDAP response read timed out, timeout used:30000ms.)
Aug 22 03:15:38 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping Configuration for server mz-ld01.abc.ne.jp update.
Aug 22 03:15:38 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: gs:mz-ld01.abc.ne.jp ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.naming.NamingException LDAP response read timed out, timeout used:30000ms.)
Aug 22 03:15:38 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Sleeping...Key lookup failed.
Aug 22 03:16:18 ldapm-p zimbramon[3227]: 3227:info: zmmtaconfig: Skipping Global system configuration update.

Thanks in advance..

[phoenix]

You need to give some details about the configuration of your server(s), single server, multi-server or what? What's the status of the Zimbra services? What does the following show:

zmcontrol status

Is the ldap search being done on the Zimbra server or from a remote client? Has this worked in the past? Which version of Zimbra is this (is it the one in your signature)?

[mikiso]

Sorry, we are running multi-server, 10-15 stores, and separate LDAP master/replicas, proxies and MTAs.

On the LDAP master server, the result of zmcontrol status was:

ldap : running
stats: running

Ldap search being done on the Zimbra server(ldap master),
and that worked in the past.

And the Zibmra version is :

Release 5.0.16_GA_2921.RHEL5_64_20090429021719 RHEL5_64 NETWORK edition

Could it be helpful?