Results 1 to 3 of 3

Thread: Zimbra servers shut down every morning at 3am

  1. #1
    snsparrish is offline Member
    Join Date
    Jun 2007
    Location
    Phoenix, AZ USA
    Posts
    13
    Rep Power
    7

    Default Zimbra servers shut down every morning at 3am

    Since upgrading from 4.5.4 to 4.5.6, the zimbra services (all of them) shut down every morning at 3:01.

    Could be the same issue as this forum post

    The server stays up and is accessible via SSH. Uptime shows that it didn't reboot. I issue a simple zmcontrol start and everything comes back online like normal and functions without error all day long.

    I saw in the zimbra user crontab that zmpurgemessages was running at 3am so I changed it to run later to determine if it was the problem - the services still shut down at 3am.

    It's a pretty new machine - Dual Xeon quad cores, 8GB RAM, 925GB RAID.

    This server was running 4.5.4 for several months without any issues. Issue appeared a couple days after the upgrade to 4.5.6.

    The catalina.out log is small.
    Code:
    [root@mailhost log]# ls -lah /opt/zimbra/tomcat/logs/catalina.out
    -rw-rw-rw-  1 zimbra zimbra 259K Jul 20 09:12 /opt/zimbra/tomcat/logs/catalina.out
    /opt/zimbra/tomcat/logs/catalina.out
    Code:
    [tomcat launcher] got signal SIGTERM invoking stop
    Jul 20, 2007 3:01:58 AM org.apache.coyote.http11.Http11BaseProtocol pause
    INFO: Pausing Coyote HTTP/1.1 on http-80
    Jul 20, 2007 3:01:58 AM org.apache.coyote.http11.Http11BaseProtocol pause
    INFO: Pausing Coyote HTTP/1.1 on http-443
    Jul 20, 2007 3:01:58 AM org.apache.coyote.http11.Http11BaseProtocol pause
    INFO: Pausing Coyote HTTP/1.1 on http-7071
    Jul 20, 2007 3:01:59 AM org.apache.catalina.core.StandardService stop
    INFO: Stopping service Catalina
    Jul 20, 2007 3:02:10 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 39 instance(s) to be deallocated
    Jul 20, 2007 3:02:11 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 39 instance(s) to be deallocated
    Jul 20, 2007 3:02:12 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 39 instance(s) to be deallocated
    Jul 20, 2007 3:02:12 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 8 instance(s) to be deallocated
    Jul 20, 2007 3:02:13 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 8 instance(s) to be deallocated
    Jul 20, 2007 3:02:14 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 8 instance(s) to be deallocated
    Jul 20, 2007 3:02:14 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 2 instance(s) to be deallocated
    Jul 20, 2007 3:02:15 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 2 instance(s) to be deallocated
    Jul 20, 2007 3:02:16 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 2 instance(s) to be deallocated
    Jul 20, 2007 3:02:17 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 8 instance(s) to be deallocated
    log4j:ERROR Error occured while converting date.
    java.lang.NullPointerException
            at java.lang.System.arraycopy(Native Method)
            at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:331)
            at java.lang.StringBuffer.getChars(StringBuffer.java:202)
            at org.apache.log4j.helpers.ISO8601DateFormat.format(ISO8601DateFormat.java:120)
            at java.text.DateFormat.format(DateFormat.java:314)
            at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:436)
            at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:56)
            at org.apache.log4j.PatternLayout.format(PatternLayout.java:495)
            at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:292)
            at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:349)
            at org.apache.log4j.WriterAppender.append(WriterAppender.java:150)
            at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:221)
            at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57)
            at org.apache.log4j.Category.callAppenders(Category.java:187)
            at org.apache.log4j.Category.forcedLog(Category.java:372)
            at org.apache.log4j.Category.log(Category.java:864)
            at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:192)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
            at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
            at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:667)
            at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
            at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
            at java.lang.Thread.run(Thread.java:595)
    Jul 20, 2007 3:02:18 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 8 instance(s) to be deallocated
    Jul 20, 2007 3:02:19 AM org.apache.catalina.core.StandardWrapper unload
    INFO: Waiting for 8 instance(s) to be deallocated
    /var/log/zimbra.log
    Code:
    Jul 20 03:01:02 mailhost slapd[4555]: is_entry_objectclass("", "2.5.6.1") no objectClass attribute
    Jul 20 03:01:48 mailhost saslauthd[6786]: server_exit     : master exited: 6786
    Jul 20 03:01:48 mailhost slapd[4555]: daemon: shutdown requested and initiated.
    Jul 20 03:01:48 mailhost slapd[4555]: slapd shutdown: waiting for 0 threads to terminate
    Jul 20 03:01:48 mailhost slapd[4555]: slapd stopped.
    Jul 20 03:01:53 mailhost amavis[7338]: Net::Server: 2007/07/20-03:01:53 Server closing!
    Jul 20 03:01:56 mailhost zmtomcatmgr[10209]: threaddump requested
    Jul 20 03:01:56 mailhost zmtomcatmgr[6263]: sending SIQUIT to tomcat/JVM process 6264
    Jul 20 03:01:58 mailhost zmtomcatmgr[10216]: stop requested
    Jul 20 03:01:58 mailhost zmtomcatmgr[10216]: waiting for manager process 6263 to die
    Jul 20 03:01:58 mailhost zmtomcatmgr[6263]: shutdown requested, sending TERM signal to 6264
    Jul 20 03:02:22 mailhost zmtomcatmgr[6263]: tomcat/JVM process exited (waitpid expected 6264 got 6264)
    Jul 20 03:02:22 mailhost zmtomcatmgr[6263]: manager woke up from wait on tomcat/JVM with pid 6264
    Jul 20 03:02:23 mailhost zmtomcatmgr[10216]: manager process 6263 died, shutdown completed
    Jul 20 03:02:28 mailhost zmtomcatmgr[10277]: status requested
    Jul 20 03:02:28 mailhost zmtomcatmgr[10277]: file /opt/zimbra/log/zmtomcatmgr.pid does not exist
    Jul 20 03:02:28 mailhost zmtomcatmgr[10277]: assuming no other instance is running
    Jul 20 03:02:28 mailhost zmtomcatmgr[10277]: no manager process is running
    Jul 20 03:02:28 mailhost zmtomcatmgr[10278]: status requested
    Jul 20 03:02:28 mailhost zmtomcatmgr[10278]: file /opt/zimbra/log/zmtomcatmgr.pid does not exist
    Jul 20 03:02:28 mailhost zmtomcatmgr[10278]: assuming no other instance is running
    Jul 20 03:02:28 mailhost zmtomcatmgr[10278]: no manager process is running
    Jul 20 03:02:28 mailhost postfix/postfix-script: stopping the Postfix mail system
    Jul 20 03:02:28 mailhost postfix/master[6740]: terminating on signal 15
    Jul 20 03:02:33 mailhost zmtomcatmgr[10432]: threaddump requested
    Jul 20 03:02:33 mailhost zmtomcatmgr[10432]: file /opt/zimbra/log/zmtomcatmgr.pid does not exist
    Jul 20 03:02:33 mailhost zmtomcatmgr[10432]: assuming no other instance is running
    Jul 20 03:02:33 mailhost zmtomcatmgr[10432]: no manager process is running
    Jul 20 03:02:35 mailhost zmtomcatmgr[10440]: stop requested
    Jul 20 03:02:35 mailhost zmtomcatmgr[10440]: file /opt/zimbra/log/zmtomcatmgr.pid does not exist
    Jul 20 03:02:35 mailhost zmtomcatmgr[10440]: assuming no other instance is running
    Jul 20 03:02:35 mailhost zmtomcatmgr[10440]: no manager process is running
    Jul 20 03:02:40 mailhost zmtomcatmgr[10442]: status requested
    Jul 20 03:02:40 mailhost zmtomcatmgr[10442]: file /opt/zimbra/log/zmtomcatmgr.pid does not exist
    Jul 20 03:02:40 mailhost zmtomcatmgr[10442]: assuming no other instance is running
    Jul 20 03:02:40 mailhost zmtomcatmgr[10442]: no manager process is running
    Jul 20 03:02:40 mailhost zmtomcatmgr[10443]: status requested
    Jul 20 03:02:40 mailhost zmtomcatmgr[10443]: file /opt/zimbra/log/zmtomcatmgr.pid does not exist
    Jul 20 03:02:40 mailhost zmtomcatmgr[10443]: assuming no other instance is running
    Jul 20 03:02:40 mailhost zmtomcatmgr[10443]: no manager process is running
    Jul 20 03:02:43 mailhost zimbramon[10219]: 10219:info: 2007-07-20 03:02:01, STATUS: : Cannot: determine
    Jul 20 03:02:43 mailhost sendmail[10248]: l6KA2bdR010248: from=zimbra, size=869, class=0, nrcpts=1, msgid=<200707201002.l6KA2bdR010248@mailhost.fhi.net>, rel
    ay=zimbra@localhost
    Jul 20 03:02:43 mailhost sendmail[10248]: l6KA2bdR010248: to=zimbra, ctladdr=zimbra (502/502), delay=00:00:06, xdelay=00:00:00, mailer=relay, pri=30869, rela
    y=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection refused by [127.0.0.1]
    /var/log/secure
    Code:
    Jul 20 02:50:03 mailhost sudo:   zimbra : TTY=unknown ; PWD=/opt/zimbra ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status
    Jul 20 02:52:03 mailhost sudo:   zimbra : TTY=unknown ; PWD=/opt/zimbra ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status
    Jul 20 02:54:04 mailhost sudo:   zimbra : TTY=unknown ; PWD=/opt/zimbra ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status
    Jul 20 02:56:03 mailhost sudo:   zimbra : TTY=unknown ; PWD=/opt/zimbra ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status
    Jul 20 02:58:04 mailhost sudo:   zimbra : TTY=unknown ; PWD=/opt/zimbra ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status
    Jul 20 03:00:03 mailhost sudo:   zimbra : TTY=unknown ; PWD=/opt/zimbra ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status
    Jul 20 03:01:56 mailhost sudo:   zimbra : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr threaddump
    Jul 20 03:01:58 mailhost sudo:   zimbra : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr stop
    Jul 20 03:02:28 mailhost sudo:   zimbra : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status
    Jul 20 03:02:28 mailhost sudo:   zimbra : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status
    Jul 20 03:02:28 mailhost sudo:   zimbra : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zimbra/postfix-2.2.9/sbin/postalias /etc/aliases
    Jul 20 03:02:28 mailhost sudo:   zimbra : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zimbra/postfix-2.2.9/sbin/postfix stop norewrite
    Jul 20 03:02:33 mailhost sudo:   zimbra : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr threaddump
    Jul 20 03:02:35 mailhost sudo:   zimbra : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr stop
    Jul 20 03:02:40 mailhost sudo:   zimbra : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status
    Jul 20 03:02:40 mailhost sudo:   zimbra : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status
    /var/log/messages
    Code:
    Jul 20 03:01:48 mailhost saslauthd[6786]: server_exit     : master exited: 6786
    Jul 20 03:01:53 mailhost amavis[7338]: Net::Server: 2007/07/20-03:01:53 Server closing!
    Jul 20 03:02:43 mailhost zimbramon[10219]: 10219:info: 2007-07-20 03:02:01, STATUS: : Cannot: determine
    Jul 20 03:04:02 mailhost zimbramon[10445]: 10445:info: 2007-07-20 03:04:01, STATUS: : Cannot: determine
    Jul 20 03:06:03 mailhost zimbramon[10498]: 10498:info: 2007-07-20 03:06:01, STATUS: : Cannot: determine
    Jul 20 03:08:02 mailhost zimbramon[10551]: 10551:info: 2007-07-20 03:08:01, STATUS: : Cannot: determine
    I'm getting tired of jumping out of bed at 3am to restart the services. Any help is warmly welcome.

  2. #2
    snsparrish is offline Member
    Join Date
    Jun 2007
    Location
    Phoenix, AZ USA
    Posts
    13
    Rep Power
    7

    Default Stopgap measure

    Until a resolution is found - I've added another cron job for 3:05am to start the Zimbra service and adjusted my network monitoring software to wait 10 minutes until it alerts me that the Zimbra services are down.

    So I'm getting all my beauty sleep again but am still going to need a resolution to this.

  3. #3
    snsparrish is offline Member
    Join Date
    Jun 2007
    Location
    Phoenix, AZ USA
    Posts
    13
    Rep Power
    7

    Default zmfixperms does the trick

    Perhaps it was just our installation but I've found two things when upgrading from 4.5.4 to 4.5.6 on RedHat4-64bit...

    1. You have to undeploy and redeploy all zimlets to get them to show up and work. (This should be in the documentation)
    2. You have to run zmfixperms as root after the upgrade to keep the zimbra services from dying at 3am every morning.

    Took many hours to find these gems. Hopefully this post will help someone else.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. [SOLVED] Clamav problem ? What's happening ?
    By aNt1X in forum Installation
    Replies: 23
    Last Post: 02-14-2008, 05:43 AM
  2. Services stopped working
    By lilwong in forum Administrators
    Replies: 4
    Last Post: 08-15-2006, 09:19 AM
  3. port 7071 not listening OS X install
    By leeimber in forum Installation
    Replies: 7
    Last Post: 03-21-2006, 10:47 AM
  4. Seeming variety of problems on suse-9.1
    By Crexis in forum Installation
    Replies: 52
    Last Post: 03-04-2006, 12:19 AM
  5. FC3 Install and no zimbra ?
    By aws in forum Installation
    Replies: 10
    Last Post: 10-09-2005, 04:19 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •