[SOLVED] Zimbra Web and Imap not responding

[funkknight]

Greetings,

Release 5.0.9_GA_2533.UBUNTU6 UBUNTU6 ( was 5.0.8 before the start of the problem).

I ran into an issue that I can't get out of. I upgraded to 5.0.8 from 4.5.10 around 8/6/2008 and everything had been fine till today. Today I added an IP to the "MTA Trusted Networks" setting in the domain tab and restarted zimbra (to ensure that it was working) and since then anything related to jetty is unresponsive. The MTA and LDAP works fine, but IMAP, Webmail and the Web Admin console are all unresponsive. The port will connect and then nothing happens (times out).

After some playing I was able to revert the change to the "MTA Trusted Networks" setting via "zmprov -l" but this did not change the broken state.

mailbox.log has not been updated since the first zimbra restart after my initial setting change.

zmmailboxd.out will show:

CompilerOracle: exclude com/zimbra/cs/session/SessionMap putAndPrune
CompilerOracle: exclude com/zimbra/cs/mailbox/MailItem delete
0 INFO [main] log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Zimbra server reserving server socket port=143 bindaddr=null ssl=false
Zimbra server reserving server socket port=993 bindaddr=null ssl=true
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
1882 INFO [main] log - Setting umask=027
1886 INFO [main] log - Setting GID=1001
1886 INFO [main] log - Setting UID=1001
1886 INFO [main] log - jetty-6.1.5
2329 INFO [main] log - No Transaction manager found - if your webapp requires one, please configure one.

When you start the mailbox services either via a "zmcontrol start" or "zmmailboxd start". After stopping the service (either way) it will produce a lengthy stack trace in zmmailboxd.out. Thinking that I could get some useful information, I changed the last 3 variables in /opt/zimbra/conf/log4j.properties to DEBUG. After this the stack trace in zmmailboxd.out would end with something similar to this:

171634 INFO [Shutdown] log - Shutdown hook executing
171727 INFO [main] log - No Transaction manager found - if your webapp requires one, please configure one.
171744 INFO [Shutdown] log - Shutdown hook complete
172092 WARN [main] log - Failed startup of context org.mortbay.jetty.webapp.WebAppContext@1db9852{/zimbraAdmin,/opt/zimbra/jetty-6.1.5/webapps/zimbraAdmin/}
javax.naming.NamingException: This context is immutable; remaining name 'env'
at org.mortbay.naming.NamingContext.createSubcontext( NamingContext.java:434)
at org.mortbay.naming.NamingContext.createSubcontext( NamingContext.java:517)
at org.mortbay.jetty.plus.webapp.EnvConfiguration.cre ateEnvContext(EnvConfiguration.java:52)
at org.mortbay.jetty.plus.webapp.EnvConfiguration.con figureWebApp(EnvConfiguration.java:103)
at org.mortbay.jetty.webapp.WebAppContext.startContex t(WebAppContext.java:1219)
at org.mortbay.jetty.handler.ContextHandler.doStart(C ontextHandler.java:501)
at org.mortbay.jetty.webapp.WebAppContext.doStart(Web AppContext.java:449)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:147)
at org.mortbay.jetty.handler.ContextHandlerCollection .doStart(ContextHandlerCollection.java:161)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.jetty.handler.HandlerCollection.doStar t(HandlerCollection.java:147)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:117)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(H andlerWrapper.java:117)
at org.mortbay.jetty.Server.doStart(Server.java:217)
at org.mortbay.setuid.SetUIDServer.doStart(SetUIDServ er.java:100)
at org.mortbay.component.AbstractLifeCycle.start(Abst ractLifeCycle.java:40)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:979)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.mortbay.start.Main.invokeMain(Main.java:183)
at org.mortbay.start.Main.start(Main.java:497)
at org.mortbay.start.Main.main(Main.java:115)
172161 INFO [main] log - Opened /opt/zimbra/jetty-6.1.5/logs/access_log.2008-08-29
172181 WARN [main] log - insufficient maxThreads configured for SslSelectChannelConnector@0.0.0.0:443
172181 INFO [main] log - Started SslSelectChannelConnector@0.0.0.0:443
172185 WARN [main] log - insufficient maxThreads configured for SslSelectChannelConnector@0.0.0.0:7071
172185 INFO [main] log - Started SslSelectChannelConnector@0.0.0.0:7071
172185 WARN [main] log - insufficient maxThreads configured for SelectChannelConnector@0.0.0.0:7072
172185 INFO [main] log - Started SelectChannelConnector@0.0.0.0:7072

Thinking that maybe a file or something was corrupted, I upgraded the install to 5.0.9 hoping that it would replace a corrupted file. However this did nothing to correct to the problem.

Any ideas/solution to this problem. Mail is coming in, but it's useless since I can't get to it in anyway.

Thanks.

[funkknight]

I ran into this problem again and finally found a solution to it.

The short version is that I was running on 630mb of memory on a virtual server. Once I upped this to 810mb of memory I didn't have any time out problems anymore. And to be fair the physical host that ran this virtual changed as well.

I assumed that if I didn't have enough memory it would go to swap. But the system never showed any swap usage until jetty woke up (see below).

The long version:

After I posted this originally I gave up and left the server in a broken state, where it was running but none of the jetty services would respond. When I used an IMAP client the next day out of habit everything was working. Somewhere around 3am the services started working again.

Fast forward to last weekend and I ran into the same problem again while applying the security patch. I tried upgrading to 5.0.16 and even rebuilt the virtual server using Debian 5, CentOS 5, Ubuntu 6.06 and Ubuntu 8. and they all had the same symptoms. I also tried doing a fresh install (on each) and it would hang for approx 2hrs at "Initializing Documents..." before it went into the running but non responsive state.

And once again I gave up in frustration to have it start working on it's own at 3am.

Today I had the virtual definition upped to have 810 mb of memory and startup runs with no problem and the system seems a lot much faster then it was before.

Besides being the next plan level for my virtual host (linode), I upped the memory by 180 because once the jetty server was operational I saw that swap usage was at about 170mb and steady. So I jumped at the idea that this would be the minimum amount of memory that would be needed for zimbra to startup cleanly.

[Razique]

Awesome, on my lab. (VMWare) I just added more ram to my zimbra instance, and it solved the issue