Tomcat crashes after upgrade to 4.0.2

[demanl]

Hi,

I'm running fc4 on a p4 3.0 mhz machine with 2gb ram....

....I was running a clean install of 4.0.0.....

....I then sucessfully upgraded to 4.0.2....and now I'm finding that tomcat does NOT run at all.....

here is output from the catalina.out file..

Sep 28, 2006 10:23:15 PM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-8001
Sep 28, 2006 10:23:15 PM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-7071
Sep 28, 2006 10:23:15 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1556 ms
Sep 28, 2006 10:23:15 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Sep 28, 2006 10:23:15 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.15
Sep 28, 2006 10:23:15 PM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
Sep 28, 2006 10:23:16 PM org.apache.catalina.startup.HostConfig deployDescriptor
WARNING: A docBase /opt/zimbra/apache-tomcat-5.5.15/webapps/zimbra inside the host appBase has been specified, and will be ignored
log4j:WARN No appenders could be found for logger (org.apache.catalina.session.ManagerBase).
log4j:WARN Please initialize the log4j system properly.
Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Fatal error: terminating: PrivilegedServlet init failed
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSock etChannelImpl.java:119)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:59)
at com.zimbra.cs.util.NetUtil.newBoundServerSocket(Ne tUtil.java:85)
at com.zimbra.cs.util.NetUtil.reserveServerSocket(Net Util.java:108)
at com.zimbra.cs.servlet.PrivilegedServlet.init(Privi legedServlet.java:91)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:211)
at org.apache.catalina.core.StandardWrapper.loadServl et(StandardWrapper.java:1105)
at org.apache.catalina.core.StandardWrapper.load(Stan dardWrapper.java:932)
at org.apache.catalina.core.StandardContext.loadOnSta rtup(StandardContext.java:3915)
at org.apache.catalina.core.StandardContext.start(Sta ndardContext.java:4176)
at org.apache.catalina.core.ContainerBase.addChildInt ernal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(Co ntainerBase.java:739)
at org.apache.catalina.core.StandardHost.addChild(Sta ndardHost.java:524)
at org.apache.catalina.startup.HostConfig.deployDirec tory(HostConfig.java:910)
at org.apache.catalina.startup.HostConfig.deployDirec tories(HostConfig.java:873)
at org.apache.catalina.startup.HostConfig.deployApps( HostConfig.java:474)
at org.apache.catalina.startup.HostConfig.start(HostC onfig.java:1118)
at org.apache.catalina.startup.HostConfig.lifecycleEv ent(HostConfig.java:310)
at org.apache.catalina.util.LifecycleSupport.fireLife cycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1020)
at org.apache.catalina.core.StandardHost.start(Standa rdHost.java:718)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1012)
at org.apache.catalina.core.StandardEngine.start(Stan dardEngine.java:442)
at org.apache.catalina.core.StandardService.start(Sta ndardService.java:450)
at org.apache.catalina.core.StandardServer.start(Stan dardServer.java:700)
at org.apache.catalina.startup.Catalina.start(Catalin a.java:551)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootst rap.java:275)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:406)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.zimbra.cs.launcher.TomcatLauncher.start(Tomcat Launcher.java:52)
at com.zimbra.cs.launcher.TomcatLauncher.main(TomcatL auncher.java:74)


is anybody else having this problem...???

any help with this problem will be greatly appreciated....

TIA...

[jholder]

Looks like something is already running on port 110

Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Fatal error: terminating: PrivilegedServlet init failed
java.net.BindException: Address already in use

Make sure that all zimbra services are stopped.

ps auxww | grep zimbra or ps auxww | grep tomcat
Kill any left over processes,

Then check to see if something is running on port 110
netstat -s -a

or

telnet hostname.domain.com 110

john

[demanl]

Originally Posted by wannabetenor

Looks like something is already running on port 110


Make sure that all zimbra services are stopped.

ps auxww | grep zimbra or ps auxww | grep tomcat
Kill any left over processes,

Then check to see if something is running on port 110
netstat -s -a

or

telnet hostname.domain.com 110

john

Hi...

I did all of the above prior to installing the upgrade....as well when I re-installed the upgrade.......

I then tried something off the wall.....I stopped zimbra using zmcontrol stop...I stopped the portmap service.....and I then started zimbra using the zmcontrol start.....which worked........and I then started the portmap service.....

....So.....what exactly does the portmap service do....and why would it lock port 110.....?????

I don't know....but it worked......more importantly......zmcontrol can now start and stop zimbra and tomcat is now behaving normally......?????

Thanks for replying....

[dijichi2]

portmap is the rpc server, runs on 111 by default so shouldn't touch 110. sure you don't have pop3 running out of inetd?

[robbyt]

I bet you have pop3 and the pop3 proxy running at the same time...

try this (as root):

/opt/zimbra/libexec/zmfixperms
su zimbra
zmperditionctl stop
tomcat restart

if you can get into the web interface, go to the admin panel and turn off the pop/imap proxy service

[math]

How do I ensure that peridition doesnt startup next reboot?

[Plugh]

Originally Posted by math

How do I ensure that peridition doesnt startup next reboot?

I'd like to know this too. I think the implication was to go to the admin console and disable imap/pop3 but that doesn't see to have fixed it for me, every time I stop and restart zimbra I get the problem described in the original post. This started as a result of a power failure, unclean shutdown last night.

As a second point, I'd actually like to use imap/pop3 to access my mail server but I never managed to get that working with Zimbra.

So now my startup sequence consists of (following a reboot, or zmcontrol start)

su - zimbra
zmperditionctl stop
tomcat restart

then all is good. It took me about 4 hours to track this down, googling for reasons that tomcat was failing didn't help and I finally went digging and found the logfile catalina.out under /opt/zimbra/apache-tomcat-5.5.15/logs and the stacktrace similar to the one in the OP which I then googled for to get here. Adding "tomcat/JVM exited twice" to help other posters google their way.

[jholder]

Originally Posted by Plugh

I'd like to know this too. I think the implication was to go to the admin console and disable imap/pop3 but that doesn't see to have fixed it for me, every time I stop and restart zimbra I get the problem described in the original post. This started as a result of a power failure, unclean shutdown last night.

As a second point, I'd actually like to use imap/pop3 to access my mail server but I never managed to get that working with Zimbra.

So now my startup sequence consists of (following a reboot, or zmcontrol start)

su - zimbra
zmperditionctl stop
tomcat restart

then all is good. It took me about 4 hours to track this down, googling for reasons that tomcat was failing didn't help and I finally went digging and found the logfile catalina.out under /opt/zimbra/apache-tomcat-5.5.15/logs and the stacktrace similar to the one in the OP which I then googled for to get here. Adding "tomcat/JVM exited twice" to help other posters google their way.

Can you post your catalina.out? I think you may have a typeo, it should be in /opt/zimbra/tomcat/logs/catalina.out.
If you have an address already in use, it will be the port that's listed just before it crashes. In this thread, it was port 110.

Also, are you using 4.0.2 or 4.5.6? and what OS?

john

[Plugh]

Argh! Got timed out posting my response ... I think.

Anyway ... Hi John, thanks for your reply. First things first, I'm running on OpenSuse 10.2 and sadly I can't figure out what version of Zimbra I'm running, can you tell me where to look? I can tell you it's the open source version.

Second, location of catalina.out ... /opt/zimbra/tomcat is a symbolic link to /opt/zimbra/apache-tomcat-5.5.15 so ... same file really.

The offending port was 110, just like the original post in this thread.

Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Fatal error: terminating: PrivilegedServlet init failed
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java :359)
at java.net.ServerSocket.bind(ServerSocket.java:319)
at com.zimbra.cs.util.NetUtil.newBoundServerSocket(Ne tUtil.java:111)
at com.zimbra.cs.util.NetUtil.bindServerSocket(NetUti l.java:134)
at com.zimbra.cs.util.NetUtil.bindTcpServerSocket(Net Util.java:63)
at com.zimbra.cs.servlet.PrivilegedServlet.init(Privi legedServlet.java:94)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:211)
at org.apache.catalina.core.StandardWrapper.loadServl et(StandardWrapper.java:1105)
at org.apache.catalina.core.StandardWrapper.load(Stan dardWrapper.java:932)
at org.apache.catalina.core.StandardContext.loadOnSta rtup(StandardContext.java:3915)
at org.apache.catalina.core.StandardContext.start(Sta ndardContext.java:4176)
at org.apache.catalina.core.ContainerBase.addChildInt ernal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(Co ntainerBase.java:739)
at org.apache.catalina.core.StandardHost.addChild(Sta ndardHost.java:524)
at org.apache.catalina.startup.HostConfig.deployDirec tory(HostConfig.java:910)
at org.apache.catalina.startup.HostConfig.deployDirec tories(HostConfig.java:873)
at org.apache.catalina.startup.HostConfig.deployApps( HostConfig.java:474)
at org.apache.catalina.startup.HostConfig.start(HostC onfig.java:1118)
at org.apache.catalina.startup.HostConfig.lifecycleEv ent(HostConfig.java:310)
at org.apache.catalina.util.LifecycleSupport.fireLife cycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1020)
at org.apache.catalina.core.StandardHost.start(Standa rdHost.java:718)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1012)
at org.apache.catalina.core.StandardEngine.start(Stan dardEngine.java:442)
at org.apache.catalina.core.StandardService.start(Sta ndardService.java:450)
at org.apache.catalina.core.StandardServer.start(Stan dardServer.java:700)
at org.apache.catalina.startup.Catalina.start(Catalin a.java:551)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootst rap.java:275)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:406)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.zimbra.cs.launcher.TomcatLauncher.start(Tomcat Launcher.java:52)
at com.zimbra.cs.launcher.TomcatLauncher.main(TomcatL auncher.java:74)

Everything is working since doing the disable perdition trick, although looking in the log, I am seeing things like

log4j:WARN No appenders could be found for logger (org.apache.catalina.session.ManagerBase).
log4j:WARN Please initialize the log4j system properly.
Sep 17, 2007 4:16:52 PM org.apache.catalina.core.StandardContext resourcesStart
SEVERE: Error starting static Resources
java.lang.IllegalArgumentException: Document base /opt/zimbra/apache-tomcat-5.5.15/server/webapps/host-
manager does not exist or is not a readable directory
at org.apache.naming.resources.FileDirContext.setDocB ase(FileDirContext.java:140)
at org.apache.catalina.core.StandardContext.resources Start(StandardContext.java:3812)
at org.apache.catalina.core.StandardContext.start(Sta ndardContext.java:3983)
at org.apache.catalina.core.ContainerBase.addChildInt ernal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(Co ntainerBase.java:739)

(two errors there, the warning about log4j plus the host-manager warning, this is followed by another one, same path but file name is manager instead of host-manager).

Thanks!
-Peter

[jholder]

Hi Peter-
The hostmanager error can be ignored.

If the error is on port 110, then you have to stop what ever is already bound to port 110. Did you happen to turn on IMAP/POP proxy? If you turn it on in a single server enviroment, it can cause tomcat to go down. Try:
su - zimbra
zmperditionctl stop
tomcat start

and see if it comes back up. If not, then you need to track down what is binding. Try:
lsof | grep TCP

and look for port 110, and the file associated with it.

You can also find out which Zimbra version you're running by running:
su - zimbra
zmcontrol -v

Best,
john