Tomcat dies after starting

[smcgrath1111]

We have a live Zimbra server, and I am trying to set up a copy of it on a test server.


Release 4.5.6_GA_1044.UBUNTU6 UBUNTU6 FOSS edition
(Ubuntu 6.10)

I did the following:
-Stop Zimbra on live box
-tar /opt/zimbra to zimbra.tar on live box

-Install zimbra on test box using 4.5.6 installer
-Untar zimbra.tar to /opt/zimbra on test box
-start zimbra on test box

Everything appears to start normally, but then tomcat appears to die right after, because doing a zmcontrol status shows:

Host phosphor.leddynamics.com
antispam Running
antivirus Stopped
zmclamdctl is not running
ldap Running
logger Running
mailbox Stopped
tomcat is not running
mta Running
snmp Running
spell Running


Here's what shows up in the tomcat log:

Nov 8, 2007 10:41:12 AM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-80
Nov 8, 2007 10:41:15 AM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-443
Nov 8, 2007 10:41:15 AM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-7071
Nov 8, 2007 10:41:15 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 6501 ms
Nov 8, 2007 10:41:15 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Nov 8, 2007 10:41:15 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.15
Nov 8, 2007 10:41:15 AM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
log4j:WARN No appenders could be found for logger (org.apache.catalina.startup.TldConfig).
log4j:WARN Please initialize the log4j system properly.
Nov 8, 2007 10:41:22 AM org.apache.catalina.core.StandardContext resourcesStart
SEVERE: Error starting static Resources
java.lang.IllegalArgumentException: Document base /opt/zimbra/apache-tomcat-5.5.15/server/webapps/host-manager does not exist or is
not a readable directory
at org.apache.naming.resources.FileDirContext.setDocB ase(FileDirContext.java:140)
at org.apache.catalina.core.StandardContext.resources Start(StandardContext.java:3812)
at org.apache.catalina.core.StandardContext.start(Sta ndardContext.java:3983)
at org.apache.catalina.core.ContainerBase.addChildInt ernal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(Co ntainerBase.java:739)
at org.apache.catalina.core.StandardHost.addChild(Sta ndardHost.java:524)
at org.apache.catalina.startup.HostConfig.deployDescr iptor(HostConfig.java:603)
at org.apache.catalina.startup.HostConfig.deployDescr iptors(HostConfig.java:535)
at org.apache.catalina.startup.HostConfig.deployApps( HostConfig.java:470)
at org.apache.catalina.startup.HostConfig.start(HostC onfig.java:1118)
at org.apache.catalina.startup.HostConfig.lifecycleEv ent(HostConfig.java:310)
at org.apache.catalina.util.LifecycleSupport.fireLife cycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1020)
at org.apache.catalina.core.StandardHost.start(Standa rdHost.java:718)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1012)
at org.apache.catalina.core.StandardEngine.start(Stan dardEngine.java:442)
at org.apache.catalina.core.StandardService.start(Sta ndardService.java:450)
at org.apache.catalina.core.StandardServer.start(Stan dardServer.java:700)
at org.apache.catalina.startup.Catalina.start(Catalin a.java:551)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootst rap.java:275)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:406)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:406)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.zimbra.cs.launcher.TomcatLauncher.start(Tomcat Launcher.java:52)
at com.zimbra.cs.launcher.TomcatLauncher.main(TomcatL auncher.java:74)
Nov 8, 2007 10:41:22 AM org.apache.catalina.core.StandardContext start
SEVERE: Error in resourceStart()
Nov 8, 2007 10:41:22 AM org.apache.catalina.core.StandardContext start
SEVERE: Error getConfigured
Nov 8, 2007 10:41:22 AM org.apache.catalina.core.StandardContext start
SEVERE: Context [/host-manager] startup failed due to previous errors
Nov 8, 2007 10:41:22 AM org.apache.catalina.core.StandardContext stop
INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager] has not been started
Nov 8, 2007 10:41:22 AM org.apache.catalina.startup.HostConfig deployDescriptor
WARNING: A docBase /opt/zimbra/apache-tomcat-5.5.15/webapps/zimbra inside the host appBase has been specified, and will be ignored
log4j:WARN No appenders could be found for logger (org.apache.catalina.session.ManagerBase).
log4j:WARN Please initialize the log4j system properly.
Nov 8, 2007 10:41:23 AM org.apache.catalina.core.StandardContext resourcesStart
SEVERE: Error starting static Resources
java.lang.IllegalArgumentException: Document base /opt/zimbra/apache-tomcat-5.5.15/server/webapps/manager does not exist or is not
a readable directory
at org.apache.naming.resources.FileDirContext.setDocB ase(FileDirContext.java:140)
at org.apache.catalina.core.StandardContext.resources Start(StandardContext.java:3812)
at org.apache.catalina.core.StandardContext.start(Sta ndardContext.java:3983)
at org.apache.catalina.core.ContainerBase.addChildInt ernal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(Co ntainerBase.java:739)
at org.apache.catalina.core.StandardHost.addChild(Sta ndardHost.java:524)
at org.apache.catalina.startup.HostConfig.deployDescr iptor(HostConfig.java:603)
at org.apache.catalina.startup.HostConfig.deployDescr iptors(HostConfig.java:535)
at org.apache.catalina.startup.HostConfig.deployApps( HostConfig.java:470)
at org.apache.catalina.startup.HostConfig.start(HostC onfig.java:1118)
at org.apache.catalina.startup.HostConfig.lifecycleEv ent(HostConfig.java:310)
at org.apache.catalina.util.LifecycleSupport.fireLife cycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1020)
at org.apache.catalina.core.StandardHost.start(Standa rdHost.java:718)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1012)
at org.apache.catalina.core.StandardEngine.start(Stan dardEngine.java:442)
at org.apache.catalina.core.StandardService.start(Sta ndardService.java:450)
at org.apache.catalina.core.StandardServer.start(Stan dardServer.java:700)
at org.apache.catalina.startup.Catalina.start(Catalin a.java:551)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.zimbra.cs.launcher.TomcatLauncher.start(Tomcat Launcher.java:52)
at com.zimbra.cs.launcher.TomcatLauncher.main(TomcatL auncher.java:74)
Nov 8, 2007 10:41:23 AM org.apache.catalina.core.StandardContext start
SEVERE: Error in resourceStart()
Nov 8, 2007 10:41:23 AM org.apache.catalina.core.StandardContext start
SEVERE: Error getConfigured
Nov 8, 2007 10:41:23 AM org.apache.catalina.core.StandardContext start
SEVERE: Context [/manager] startup failed due to previous errors
Nov 8, 2007 10:41:23 AM org.apache.catalina.core.StandardContext stop
INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager] has not been started
log4j:WARN No appenders could be found for logger (org.apache.catalina.session.ManagerBase).
log4j:WARN Please initialize the log4j system properly.
Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Zimbra server reserving server socket port=143 bindaddr=null ssl=false
Zimbra server reserving server socket port=993 bindaddr=null ssl=true
Zimbra server process is running as root, changing to user=zimbra uid=1001 gid=1001
Zimbra server process, after change, is running with uid=1001 euid=1001 gid=1001 egid=1001
Nov 8, 2007 10:41:49 AM org.apache.coyote.http11.Http11BaseProtocol init



Does anyone have any suggestions?

Thanks
-Scott

[jholder]

Hello Scott, and welcome to the Zimbra Forums!

Here are a couple questions/suggestions:
1) Is this a migration from one server to another? If so, did the other server have the same rpm/mpkg/dpkg(s) installed? If you're moving to another server, please follow these steps in the wiki:

2) If #1 is complete, then try running zmfixperms located in the libexec directory.

3) Can you attach your whole /opt/zimbra/tomcat/logs/catalina.out to the thread using the manage attachements button?

Good luck
john

[smcgrath1111]

Originally Posted by jholder

Here are a couple questions/suggestions:
1) Is this a migration from one server to another? If so, did the other server have the same rpm/mpkg/dpkg(s) installed? If you're moving to another server, please follow these steps in the wiki:

2) If #1 is complete, then try running zmfixperms located in the libexec directory.

3) Can you attach your whole /opt/zimbra/tomcat/logs/catalina.out to the thread using the manage attachements button?

Good luck
john

Thanks, John-

I guess you can consider it a migration, although in the end the server will have a different hostname and IP address than the orginal that everything was copied from. The purpose is a non-production server for testing the Zimbra upgrades.

I have attempted to make the test server identical, in the software sense, to the live server. I used the same Ubuntu install, and the same Zimbra installer version.

I ran zmfixperms but it didn't seem to help. Attached is catalina.out.

[jholder]

Ah, okay. Changing the server / host name can be especially tricky. Heres the article on how to do that:
ZmSetServerName - Zimbra :: Wiki

So, before you do that, here's the article on how to move Zimbra:
Open Source Edition Backup Procedure - Zimbra :: Wiki

[smcgrath1111]

I changed the hostname of the test server to the one of the backup installation, to eliminate the possibility of that being a problem. I still got the same tomcat error when I started it.

I did follow that one of the procedures from the wiki page to create the back up, however I read somewhere that you could do the install before the restore.

As the wiki page suggested, I will try starting over, this time copying the backup Zimbra installation to the /opt folder and name it "zimbra" first, then do an "upgrade" with the installer, then set permissions.

Thanks

[mmorse]

Originally Posted by smcgrath1111

I read somewhere that you could do the install before the restore.

I'd imagine you're referring to part 3 of this doc: /blog/archives/2007/10/moving_zcs_to_another_server.html

Part 3 : Create a "dummy" Install Then Remove It
The goal of this step is to get the rpm/dpkg databases correct. When you download ZCS, make sure it's for your newer OS, and the SAME version of ZCS that's rsync'ed.

Run the installer with the -s option. This tells the installer to only install the software, and not to configure the installation.

Once the installer has completed, delete it by rm -rf /opt/zimbra. This wipes any dummy data you have in that location.

Part 4 :
Connect and mount your external hard drive. Then, rsync the backed up data to its new location (rsync -avr /mnt/migration/zimbra /opt).

[smcgrath1111]

Actually, it says it here too:

Open Source Edition Backup Procedure - Zimbra :: Wiki

Paraphrased:

...you can either restore your /opt/zimbra folder to a server that is not currently running Zimbra and download a dummy copy of Zimbra and run an upgrade in order to clean everything up and make it run correctly again...

Have not tried the above yet.

...or you may restore the backup to an existing Zimbra server.

...Simply set up your backup server identical to your production server

Shut down Zimbra on the backup server. Copy /opt/zimbra from your backup to the backup server.

chown -R zimbra:zimbra /opt/zimbra
/opt/zimbra/libexec/zmfixperms

Start Zimbra. Everything should work.

Tried the above and had the tomcat problem.

Question- After I copy the /opt/zimbra structure to the backup server, and I run the installer, it says:

The Zimbra Collaboration Suite does not appear to be installed,
yet there appears to be a ZCS directory structure in /opt/zimbra.

Would you like to delete /opt/zimbra before installing? [N]

But it doesn't ask me specifically if I want to do an "upgrade". Should I continue with the installation (keeping /opt/zimbra)?

[jholder]

Yeah, we wrote the wiki back before I even worked at Zimbra. I was an Open Source user. Didn't know better back then.

Considering I wrote the blog post, and partially contributed/wrote the wiki article; I suggest you go with the blog post.

[smcgrath1111]

Did as the blog post said:

-Performed install.sh -s
-Removed /opt/zimbra
-Restored the backup to /opt/zimbra
-Fixed permissions
-Ran install.sh and chose to upgrade
The upgrade seemed to complete ok, although I did see a java error message at one point:

Setting defaults from saved config in /opt/zimbra/.saveconfig/config.save
   HOSTNAME=phosphor.leddynamics.com
   LDAPHOST=phosphor.LEDdynamics.com
   LDAPPORT=389
   SNMPTRAPHOST=phosphor.LEDdynamics.com
   SMTPSOURCE=admin@phosphor.LEDdynamics.com
   SMTPDEST=admin@phosphor.LEDdynamics.com
   SNMPNOTIFY=yes
   SMTPNOTIFY=yes
   LDAPROOTPW=hzpe60vWKi
   LDAPZIMBRAPW=hzpe60vWKi
Restoring existing configuration file from /opt/zimbra/.saveconfig/config.save...done
Operations logged to /tmp/zmsetup.log.1911
ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException phosphor.LEDdynamics.com:389)
Setting defaults...Done

I think I know what may be causing all of my problems - the capital letters in the ldap host domain name (LEDdynamics.com). I noticed, among other things, that I could not apply the changes when I finished with the upgrade menu until I corrected this. This may also explain why our attempted upgrade from 4.5.6 to 4.5.9 failed on the live server (whole other story). I'm going to attempt to locate all of the places this is present, set to all lowercase, and reattempt the migration. Let me know if you have any suggestions. Thanks for your help thus far.

[smcgrath1111]

I'm just writing with a follow up. After I went through and ensured the following, things started to look up. Please correct me if I am wrong:

-All domain names specified anywhere in the zimbra setup routine (install.sh or zmsetup.pl) are all lowercase

-the linux command "dnsdomainname" must return a domain name, and that domain name should be all lowercase. If this is not the case, check to make sure you have properly configured all DNS records for the host (A, MX, and PTR)

-/etc/hosts should contain the host and domain name given to zimbra setup (logical hostname), and its external IP address

-/etc/hostname should be the name of the host that has a DNS A record in your domain

In particular, I no longer got the java error from the last post. There are a few quirks but I think I just need to rerun the upgrade now that everything is fixed. Thanks again