[SOLVED] Zimbra hangs with 100% CPU load

[veronica]

Hi andrewfn,

Zimbra doesnt support XEN yet for LDAP atleast. See this Bug 23683 – Use posix mutexes on Linux builds to avoid Xen issues.

Can you increase the ldap_log_level for ldap to 256 and start the ldap ? I suspect its a ldap issue.

[andrewfn]

Hi Veronica, I see you are correct about Zimbra not supporting XEN. I have been using them together for two years without a glitch, although I know this doesn't mean it is not the cause of the present problem. Where do I change the setting for the ldap_log_level ?

[veronica]

zmlocalconfig setting. You need to change as zmlocalconfig -e ldap_log_level=256

Please note this will enable logging to high value and lots of information will appear, but would be helpful for debug wen server crashes. Make sure you have enough space in /var/log :-)

[andrewfn]

Update: A new XEN/CentOS Kernel has just been released and I updated both the client and host and rebooted them both. The client has stopped hanging, but goes into 100% cpu bursts every 10 seconds and still will not accept connections.
I have enabled a log level of 256. Here are 50 lines from the log:

Aug 14 13:42:57 mail zimbramon[23803]: 23803:info: zmmtaconfig: gs:mail.cfaw.info ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: java.net.SocketException Broken pipe)
Aug 14 13:42:57 mail zimbramon[23803]: 23803:info: zmmtaconfig: Sleeping...Key lookup failed.
Aug 14 13:43:02 mail slapd[23335]: conn=105 fd=11 ACCEPT from IP=192.168.4.122:58316 (IP=192.168.4.122:389)
Aug 14 13:43:02 mail slapd[23335]: conn=105 op=0 STARTTLS
Aug 14 13:43:02 mail slapd[23335]: conn=105 op=0 RESULT oid= err=0 text=
Aug 14 13:43:02 mail slapd[23335]: conn=105 fd=11 closed (TLS negotiation failure)
Aug 14 13:43:03 mail slapd[23335]: conn=106 fd=11 ACCEPT from IP=192.168.4.122:58317 (IP=192.168.4.122:389)
Aug 14 13:43:03 mail slapd[23335]: conn=106 op=0 STARTTLS
Aug 14 13:43:03 mail slapd[23335]: conn=106 op=0 RESULT oid= err=0 text=
Aug 14 13:43:03 mail slapd[23335]: conn=106 fd=11 closed (TLS negotiation failure)
Aug 14 13:43:03 mail zimbramon[23803]: 23803:info: zmmtaconfig: Skipping Global system configuration update.
Aug 14 13:43:03 mail zimbramon[23803]: 23803:info: zmmtaconfig: gacf ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: java.net.SocketException Broken pipe)
Aug 14 13:43:03 mail slapd[23335]: conn=107 fd=11 ACCEPT from IP=192.168.4.122:58318 (IP=192.168.4.122:389)
Aug 14 13:43:03 mail slapd[23335]: conn=107 op=0 STARTTLS
Aug 14 13:43:03 mail slapd[23335]: conn=107 op=0 RESULT oid= err=0 text=
Aug 14 13:43:04 mail slapd[23335]: conn=107 fd=11 closed (TLS negotiation failure)
Aug 14 13:43:04 mail zimbramon[23803]: 23803:info: zmmtaconfig: Skipping All Reverse Proxy URLs update.
Aug 14 13:43:04 mail zimbramon[23803]: 23803:info: zmmtaconfig: Skipping getAllReverseProxyURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: java.net.SocketException Broken pipe)
Aug 14 13:43:04 mail slapd[23335]: conn=108 fd=11 ACCEPT from IP=192.168.4.122:58319 (IP=192.168.4.122:389)
Aug 14 13:43:04 mail slapd[23335]: conn=108 op=0 STARTTLS
Aug 14 13:43:04 mail slapd[23335]: conn=108 op=0 RESULT oid= err=0 text=
Aug 14 13:43:04 mail slapd[23335]: conn=108 fd=11 closed (TLS negotiation failure)
Aug 14 13:43:04 mail zimbramon[23803]: 23803:info: zmmtaconfig: Skipping All Reverse Proxy Backends update.
Aug 14 13:43:04 mail zimbramon[23803]: 23803:info: zmmtaconfig: Skipping getAllReverseProxyBackends ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: java.net.SocketException Broken pipe)
Aug 14 13:43:05 mail slapd[23335]: conn=109 fd=11 ACCEPT from IP=192.168.4.122:58320 (IP=192.168.4.122:389)
Aug 14 13:43:05 mail slapd[23335]: conn=109 op=0 STARTTLS
Aug 14 13:43:05 mail slapd[23335]: conn=109 op=0 RESULT oid= err=0 text=
Aug 14 13:43:05 mail slapd[23335]: conn=109 fd=11 closed (TLS negotiation failure)
Aug 14 13:43:05 mail zimbramon[23803]: 23803:info: zmmtaconfig: Skipping All Memcached Servers update.
Aug 14 13:43:05 mail zimbramon[23803]: 23803:info: zmmtaconfig: Skipping getAllMemcachedServers ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: java.net.SocketException Broken pipe)
Aug 14 13:43:06 mail slapd[23335]: conn=110 fd=11 ACCEPT from IP=192.168.4.122:58321 (IP=192.168.4.122:389)
Aug 14 13:43:06 mail slapd[23335]: conn=110 op=0 STARTTLS
Aug 14 13:43:06 mail slapd[23335]: conn=110 op=0 RESULT oid= err=0 text=
Aug 14 13:43:06 mail slapd[23335]: conn=110 fd=11 closed (TLS negotiation failure)
Aug 14 13:43:06 mail zimbramon[23803]: 23803:info: zmmtaconfig: Skipping All MTA Authentication Target URLs update.
Aug 14 13:43:06 mail zimbramon[23803]: 23803:info: zmmtaconfig: Skipping getAllMtaAuthURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: java.net.SocketException Broken pipe)
Aug 14 13:43:07 mail slapd[23335]: conn=111 fd=11 ACCEPT from IP=192.168.4.122:58322 (IP=192.168.4.122:389)
Aug 14 13:43:07 mail slapd[23335]: conn=111 op=0 STARTTLS
Aug 14 13:43:07 mail slapd[23335]: conn=111 op=0 RESULT oid= err=0 text=
Aug 14 13:43:07 mail slapd[23335]: conn=111 fd=11 closed (TLS negotiation failure)
Aug 14 13:43:07 mail zimbramon[23803]: 23803:info: zmmtaconfig: Skipping Configuration for server mail.cfaw.info update.
Aug 14 13:43:07 mail zimbramon[23803]: 23803:info: zmmtaconfig: gs:mail.cfaw.info ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: java.net.SocketException Broken pipe)
Aug 14 13:43:07 mail zimbramon[23803]: 23803:info: zmmtaconfig: Sleeping...Key lookup failed.
Aug 14 13:43:12 mail slapd[23335]: conn=112 fd=11 ACCEPT from IP=192.168.4.122:58323 (IP=192.168.4.122:389)
Aug 14 13:43:12 mail slapd[23335]: conn=112 op=0 STARTTLS
Aug 14 13:43:12 mail slapd[23335]: conn=112 op=0 RESULT oid= err=0 text=
Aug 14 13:43:12 mail slapd[23335]: conn=112 fd=11 closed (TLS negotiation failure)
Aug 14 13:43:13 mail slapd[23335]: conn=113 fd=11 ACCEPT from IP=192.168.4.122:58324 (IP=192.168.4.122:389)
Aug 14 13:43:13 mail slapd[23335]: conn=113 op=0 STARTTLS
Aug 14 13:43:13 mail slapd[23335]: conn=113 op=0 RESULT oid= err=0 text=

[andrewfn]

It turns out that the issue was an interaction with XEN which was fixed in a very recent patch. However I compounded the problem by re-installing the certificates using the guide here: Irfan-Notes - Zimbra :: Wiki which actually messed them up.

When I followed this guide: Administration Console and CLI Certificate Tools - Zimbra :: Wiki
using the examples in the section: Single-Node Self-Signed Certificate
there were a couple of error messages but afterwards everything worked.
I also found a much more detailed guide here: Recreating a Self-Signed SSL Certificate - Zimbra :: Wiki
Thanks to the people who helped!

[andrewfn]

One more thing I did: The Zimbra machine was configured to run in level 5 (graphical interface). I changed this to level 3 (command line) since level 5 runs a whole load of process which might have contributed to the instability of the machine.
Can anyone tell me how I mark this thread as [SOLVED] ?

[andrewfn]

It happened again this morning. (Funny it should be a Monday morning again.) The Zimbra VM would run for 7 minutes and then lock up with 100% CPU load. When I tried to look at the console of the VM, the problem became apparent. I got the warning message:

enconsole: Could not read tty from store: No such file or directory

This was solved very simply by running:

/usr/sbin/xenconsoled

on Dom0. It seems that it had taken 7 mins for the console buffer to fill up and nothing was emptying it, so the VM hung.
Now that I can see the console, it is full of the following error messages:

4gb seg fixup, process slapd

I will probably have to sort that out, but in the meantime at least I have a working Zimbra.