zimbra dies

[jakebriems]

I get a call from a user today saying that email is down:

[zimbra@[HOSTNAME] root]$ zmcontrol status
Host [HOSTNAME].[DOMAIN].com
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Stopped
mta Running
snmp Running
spell Running
[zimbra@[HOSTNAME] root]$ zmcontrol start mailbox
Host [HOSTNAME].[DOMAIN].com
Starting antispam...Done.
Starting antivirus...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting mta...FAILED
getService: sasl
getService: webxml
getService: perdition
DO: /opt/zimbra/postfix/sbin/postconf -e content_filter='smtp-amavis:[127.0.0.1] :10024'
DO: /opt/zimbra/postfix/sbin/postconf -e myhostname='[HOSTNAME].[DOMAIN].com'
DO: /opt/zimbra/postfix/sbin/postconf -e recipient_delimiter=''
DO: /opt/zimbra/postfix/sbin/postconf -e smtpd_sasl_auth_enable='yes'
DO: /opt/zimbra/postfix/sbin/postconf -e smtpd_tls_auth_only='yes'
DO: /opt/zimbra/postfix/sbin/postconf -e disable_dns_lookups='no'
DO: /opt/zimbra/postfix/sbin/postconf -e message_size_limit='10240000'
DO: /opt/zimbra/postfix/sbin/postconf -e relayhost=''
DO: /opt/zimbra/postfix/sbin/postconf -e smtpd_recipient_restrictions='reject_no n_fqdn_recipient, permit_sasl_authenticated, permit_mynetworks, reject_invalid_h ostname, reject_non_fqdn_sender, reject_unauth_destination, permit'
postsuper: Requeued: 16 messages
saslauthd[6281] :detach_tty : Cannot start saslauthd
saslauthd[6281] :detach_tty : Another instance of saslauthd is currently ru nning


Starting snmp...Done.
Starting spell...Done.
[zimbra@[HOSTNAME] root]$ zmcontrol status
Host [HOSTNAME].[DOMAIN].com
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Running
mta Running
snmp Running
spell Running


Any ideas what's going on? I will check the logs.

jb

[bobby]

if mailbox stops again, try running this to find out exactly what part isn't running (though you may need to post the output here for interpretation):

sh -x ~/bin/zmmailboxctl status

a quick guess is that tomcat died. are there any clues in stacktraces or catalina.outs from whenever mailbox became stopped? check in /opt/zimbra/tomcat/logs/

you can get see when mailbox stopped by checking in /var/log/zimbra.log. the zmcontrol status is logged there pretty frequently

[scott_mcneilage]

I'm getting this same error and symptoms.
Everything was installed and running OK until I did zmcontrol shutdown and startup.
Rebooted the server with same result.
Reinstalled zimbra and it rebooted ok
Added a few domains and accounts and it rebooted ok
Changed some POP settings (to allow clear text) and restart / reboot failed.

Following Zimbra Log and Tomcat Catalina.out:
---------------------------
2006-05-04 10:53:42,285 INFO [main] [] soap - Servlet SoapServlet starting up
2006-05-04 10:53:42,336 INFO [main] [] soap - Adding service AccountService to SoapServlet
2006-05-04 10:53:42,483 INFO [main] [] soap - Adding service MailService to SoapServlet
2006-05-04 10:53:42,645 INFO [main] [] soap - Adding service IMService to SoapServlet
2006-05-04 10:53:42,655 INFO [main] [] misc - version=3.0.1_GA_160.RHEL4 release=20060216172310 builddate=20060216-1726 buildhost=build.liquidsys.com
2006-05-04 10:53:42,656 INFO [main] [] misc - LANG environment is set to: en_AU.UTF-8
2006-05-04 10:53:42,883 INFO [main] [] DbPool - Setting mysql connector property: maxActive=10
2006-05-04 10:53:42,884 INFO [main] [] misc - Setting mysql connection pool size to 10
2006-05-04 10:53:43,874 INFO [main] [] extensions - Loading extensions from /opt/zimbra/lib/ext
2006-05-04 10:53:43,893 INFO [main] [] extensions - extension com.zimbra.clam.ClamScanner found in /opt/zimbra/lib/ext/clamscanner/clamscanner.jar
2006-05-04 10:53:43,894 INFO [main] [] extensions - Initializing extensions
2006-05-04 10:53:43,904 INFO [main] [] ClamScanner - attachment scan is disabled
2006-05-04 10:53:44,162 INFO [main] [] extensions - Initialized extension clamscanner: com.zimbra.clam.ClamScanner@com.zimb...tensio n.ZimbraExtensionClassLoader@119e583
2006-05-04 10:53:44,228 INFO [IndexWritersSweeper] [] MailboxIndex - IndexWritersSweeper thread starting
2006-05-04 10:53:44,361 INFO [main] [] RedoLogManager - Starting pre-startup crash recovery
2006-05-04 10:53:44,419 INFO [main] [] RedoPlayer - No uncommitted transactions to redo
2006-05-04 10:53:44,431 INFO [main] [] RedoLogManager - Finished pre-startup crash recovery
2006-05-04 10:53:44,452 FATAL [main] [] soap - Unable to start servlet
com.zimbra.cs.service.ServiceException: system failure: Could not bind to port=7025 bindaddr=clickmail.click.com.au ssl=false
at com.zimbra.cs.service.ServiceException.FAILURE(Ser viceException.java:131)
at com.zimbra.cs.util.NetUtil.getBoundServerSocket(Ne tUtil.java:54)
at com.zimbra.cs.lmtpserver.LmtpServer.startupLmtpSer ver(LmtpServer.java:195)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:106)
at com.zimbra.soap.SoapServlet.init(SoapServlet.java: 118)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:211)
at org.apache.catalina.core.StandardWrapper.loadServl et(StandardWrapper.java:1068)
at org.apache.catalina.core.StandardWrapper.load(Stan dardWrapper.java:900)
at org.apache.catalina.core.StandardContext.loadOnSta rtup(StandardContext.java:3823)
at org.apache.catalina.core.StandardContext.start(Sta ndardContext.java:4087)
at org.apache.catalina.core.ContainerBase.addChildInt ernal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(Co ntainerBase.java:739)
at org.apache.catalina.core.StandardHost.addChild(Sta ndardHost.java:524)
at org.apache.catalina.startup.HostConfig.deployDirec tory(HostConfig.java:909)
at org.apache.catalina.startup.HostConfig.deployDirec tories(HostConfig.java:872)
at org.apache.catalina.startup.HostConfig.deployApps( HostConfig.java:474)
at org.apache.catalina.startup.HostConfig.start(HostC onfig.java:1106)
at org.apache.catalina.startup.HostConfig.lifecycleEv ent(HostConfig.java:310)
at org.apache.catalina.util.LifecycleSupport.fireLife cycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1019)
at org.apache.catalina.core.StandardHost.start(Standa rdHost.java:718)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1011)
at org.apache.catalina.core.StandardEngine.start(Stan dardEngine.java:440)
at org.apache.catalina.core.StandardService.start(Sta ndardService.java:450)
at org.apache.catalina.core.StandardServer.start(Stan dardServer.java:683)
at org.apache.catalina.startup.Catalina.start(Catalin a.java:537)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootst rap.java:271)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:409)
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSock etChannelImpl.java:119)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:52)
at com.zimbra.cs.util.NetUtil.newBoundServerSocket(Ne tUtil.java:78)
at com.zimbra.cs.util.NetUtil.getBoundServerSocket(Ne tUtil.java:52)
... 30 more
2006-05-04 10:53:44,457 INFO [main] [] [/service] - Marking servlet SoapServlet as unavailable
2006-05-04 10:53:44,457 ERROR [main] [] [/service] - Servlet /service threw load() exception
javax.servlet.UnavailableException: system failure: Could not bind to port=7025 bindaddr=clickmail.click.com.au ssl=false
at com.zimbra.soap.SoapServlet.init(SoapServlet.java: 121)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:211)
at org.apache.catalina.core.StandardWrapper.loadServl et(StandardWrapper.java:1068)
at org.apache.catalina.core.StandardWrapper.load(Stan dardWrapper.java:900)
at org.apache.catalina.core.StandardContext.loadOnSta rtup(StandardContext.java:3823)
at org.apache.catalina.core.StandardContext.start(Sta ndardContext.java:4087)
at org.apache.catalina.core.ContainerBase.addChildInt ernal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(Co ntainerBase.java:739)
at org.apache.catalina.core.StandardHost.addChild(Sta ndardHost.java:524)
at org.apache.catalina.startup.HostConfig.deployDirec tory(HostConfig.java:909)
at org.apache.catalina.startup.HostConfig.deployDirec tories(HostConfig.java:872)
at org.apache.catalina.startup.HostConfig.deployApps( HostConfig.java:474)
at org.apache.catalina.startup.HostConfig.start(HostC onfig.java:1106)
at org.apache.catalina.startup.HostConfig.lifecycleEv ent(HostConfig.java:310)
at org.apache.catalina.util.LifecycleSupport.fireLife cycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1019)
at org.apache.catalina.core.StandardHost.start(Standa rdHost.java:718)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1011)
at org.apache.catalina.core.StandardEngine.start(Stan dardEngine.java:440)
at org.apache.catalina.core.StandardService.start(Sta ndardService.java:450)
at org.apache.catalina.core.StandardServer.start(Stan dardServer.java:683)
at org.apache.catalina.startup.Catalina.start(Catalin a.java:537)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootst rap.java:271)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:409)
2006-05-04 10:53:44,466 INFO [main] [] soap - Servlet AdminServlet starting up
2006-05-04 10:53:44,578 INFO [main] [] soap - Adding service AdminService to AdminServlet
2006-05-04 10:53:44,602 INFO [main] [] soap - Adding service AccountService to AdminServlet
2006-05-04 10:53:44,602 INFO [main] [] soap - Adding service MailService to AdminServlet
2006-05-04 10:53:44,603 INFO [main] [] soap - Adding service IMService to AdminServlet
2006-05-04 10:53:44,603 INFO [main] [] misc - version=3.0.1_GA_160.RHEL4 release=20060216172310 builddate=20060216-1726 buildhost=build.liquidsys.com
2006-05-04 10:53:44,603 INFO [main] [] misc - LANG environment is set to: en_AU.UTF-8
2006-05-04 10:53:44,603 INFO [main] [] extensions - Loading extensions from /opt/zimbra/lib/ext
2006-05-04 10:53:44,604 INFO [main] [] extensions - extension com.zimbra.clam.ClamScanner found in /opt/zimbra/lib/ext/clamscanner/clamscanner.jar
2006-05-04 10:53:44,605 INFO [main] [] extensions - Initializing extensions
2006-05-04 10:53:44,605 INFO [main] [] ClamScanner - attachment scan is disabled
2006-05-04 10:53:44,605 INFO [main] [] extensions - Initialized extension clamscanner: com.zimbra.clam.ClamScanner@com.zimb...tensio n.ZimbraExtensionClassLoader@119e583
2006-05-04 10:53:44,613 INFO [main] [] ClamScanner - attachment scan is disabled
2006-05-04 10:53:44,613 INFO [main] [] extensions - Initialized extension clamscanner: com.zimbra.clam.ClamScanner@com.zimb...tensio n.ZimbraExtensionClassLoader@1a41cc7
2006-05-04 10:53:44,614 INFO [IndexWritersSweeper] [] MailboxIndex - IndexWritersSweeper thread exiting
2006-05-04 10:53:44,614 INFO [main] [] MailboxIndex - Flushing all open index writers
2006-05-04 10:53:44,615 INFO [main] [] RedoLogManager - Starting pre-startup crash recovery
2006-05-04 10:53:44,615 INFO [IndexWritersSweeper] [] MailboxIndex - IndexWritersSweeper thread starting
2006-05-04 10:53:44,619 INFO [main] [] RedoPlayer - No uncommitted transactions to redo
2006-05-04 10:53:44,629 INFO [main] [] RedoLogManager - Finished pre-startup crash recovery
2006-05-04 10:53:44,631 FATAL [main] [] soap - Unable to start servlet
com.zimbra.cs.service.ServiceException: system failure: Could not bind to port=7025 bindaddr=clickmail.click.com.au ssl=false
at com.zimbra.cs.service.ServiceException.FAILURE(Ser viceException.java:131)
at com.zimbra.cs.util.NetUtil.getBoundServerSocket(Ne tUtil.java:54)
at com.zimbra.cs.lmtpserver.LmtpServer.startupLmtpSer ver(LmtpServer.java:195)
at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:106)
at com.zimbra.soap.SoapServlet.init(SoapServlet.java: 118)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:211)
at org.apache.catalina.core.StandardWrapper.loadServl et(StandardWrapper.java:1068)
at org.apache.catalina.core.StandardWrapper.load(Stan dardWrapper.java:900)
at org.apache.catalina.core.StandardContext.loadOnSta rtup(StandardContext.java:3823)
at org.apache.catalina.core.StandardContext.start(Sta ndardContext.java:4087)
at org.apache.catalina.core.ContainerBase.addChildInt ernal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(Co ntainerBase.java:739)
at org.apache.catalina.core.StandardHost.addChild(Sta ndardHost.java:524)
at org.apache.catalina.startup.HostConfig.deployDirec tory(HostConfig.java:909)
at org.apache.catalina.startup.HostConfig.deployDirec tories(HostConfig.java:872)
at org.apache.catalina.startup.HostConfig.deployApps( HostConfig.java:474)
at org.apache.catalina.startup.HostConfig.start(HostC onfig.java:1106)
at org.apache.catalina.startup.HostConfig.lifecycleEv ent(HostConfig.java:310)
at org.apache.catalina.util.LifecycleSupport.fireLife cycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1019)
at org.apache.catalina.core.StandardHost.start(Standa rdHost.java:718)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1011)
at org.apache.catalina.core.StandardEngine.start(Stan dardEngine.java:440)
at org.apache.catalina.core.StandardService.start(Sta ndardService.java:450)
at org.apache.catalina.core.StandardServer.start(Stan dardServer.java:683)
at org.apache.catalina.startup.Catalina.start(Catalin a.java:537)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootst rap.java:271)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:409)
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSock etChannelImpl.java:119)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:52)
at com.zimbra.cs.util.NetUtil.newBoundServerSocket(Ne tUtil.java:78)
at com.zimbra.cs.util.NetUtil.getBoundServerSocket(Ne tUtil.java:52)
... 30 more
2006-05-04 10:53:44,635 INFO [main] [] [/service] - Marking servlet AdminServlet as unavailable
2006-05-04 10:53:44,635 ERROR [main] [] [/service] - Servlet /service threw load() exception
javax.servlet.UnavailableException: system failure: Could not bind to port=7025 bindaddr=clickmail.click.com.au ssl=false
at com.zimbra.soap.SoapServlet.init(SoapServlet.java: 121)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:211)
at org.apache.catalina.core.StandardWrapper.loadServl et(StandardWrapper.java:1068)
at org.apache.catalina.core.StandardWrapper.load(Stan dardWrapper.java:900)
at org.apache.catalina.core.StandardContext.loadOnSta rtup(StandardContext.java:3823)
at org.apache.catalina.core.StandardContext.start(Sta ndardContext.java:4087)
at org.apache.catalina.core.ContainerBase.addChildInt ernal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(Co ntainerBase.java:739)
at org.apache.catalina.core.StandardHost.addChild(Sta ndardHost.java:524)
at org.apache.catalina.startup.HostConfig.deployDirec tory(HostConfig.java:909)
at org.apache.catalina.startup.HostConfig.deployDirec tories(HostConfig.java:872)
at org.apache.catalina.startup.HostConfig.deployApps( HostConfig.java:474)
at org.apache.catalina.startup.HostConfig.start(HostC onfig.java:1106)
at org.apache.catalina.startup.HostConfig.lifecycleEv ent(HostConfig.java:310)
at org.apache.catalina.util.LifecycleSupport.fireLife cycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1019)
at org.apache.catalina.core.StandardHost.start(Standa rdHost.java:718)
at org.apache.catalina.core.ContainerBase.start(Conta inerBase.java:1011)
at org.apache.catalina.core.StandardEngine.start(Stan dardEngine.java:440)
at org.apache.catalina.core.StandardService.start(Sta ndardService.java:450)
at org.apache.catalina.core.StandardServer.start(Stan dardServer.java:683)
at org.apache.catalina.startup.Catalina.start(Catalin a.java:537)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootst rap.java:271)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:409)
2006-05-04 10:53:44,644 INFO [main] [] ContentServlet - Servlet ContentServlet starting up
2006-05-04 10:53:44,647 INFO [main] [] PublicICalServlet - Servlet PublicICalServlet starting up
2006-05-04 10:53:44,653 INFO [main] [] account - Servlet PreAuthServlet starting up
2006-05-04 10:53:44,686 INFO [main] [] mailbox - Servlet UserServlet starting up
2006-05-04 10:53:44,697 INFO [main] [] FileUploadServlet - Servlet FileUploadServlet starting up
2006-05-04 10:53:44,702 INFO [main] [] StatsImageServlet - Servlet StatsImageServlet starting up

---------------------------

catalina.out:

4/05/2006 10:53:37 org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-80
4/05/2006 10:53:38 org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-7071
4/05/2006 10:53:38 org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1861 ms
4/05/2006 10:53:38 org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
4/05/2006 10:53:38 org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.7
4/05/2006 10:53:38 org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
log4j:WARN No appenders could be found for logger (org.apache.catalina.session.ManagerBase).
log4j:WARN Please initialize the log4j system properly.
Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Zimbra server reserving server socket port=143 bindaddr=null ssl=false
Zimbra server reserving server socket port=993 bindaddr=null ssl=true
Zimbra server process is running as root, changing to user=zimbra uid=501 gid=502
Zimbra server process, after change, is running with uid=501 euid=501 gid=502 egid=502
4/05/2006 10:53:45 org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-80
4/05/2006 10:53:45 org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-7071
4/05/2006 10:53:45 org.apache.catalina.startup.Catalina start
INFO: Server startup in 6661 ms

[rsharpe]

When zimbra is being restarted it looks as though it is not kill all the processes properly. There were some lines that indicated it couldn not bing to port 7025 which would mean there is still some mailbox process running.

[fredo]

I have some problems in the sense that we're having some power outage issues from time to time currently. I've noticed that every time after a power failure, the antivirus, mailbox and mta are stopped. I have to re-start all services for them to be up and running again.

Can someone clear that issue up ? Can Zimbra keep it's instances alive when such issues happen ?

[phoenix]

Originally Posted by fredo

I have some problems in the sense that we're having some power outage issues from time to time currently. I've noticed that every time after a power failure, the antivirus, mailbox and mta are stopped. I have to re-start all services for them to be up and running again.

Can someone clear that issue up ? Can Zimbra keep it's instances alive when such issues happen ?

Is there anything in the logs indicating why they haven't restarted? I've had power-failures and Zimbra has restarted OK. If you're able to document it then file it in bugzilla.