[SOLVED] Upgrade from 5.06 to 5.07 - zmmailboxdctl is not running

[PS-Man]

Ran a upgrade this morning and all went well!! Installed some new RAm into the Zimbra server (8Gb) and since the shutdown I am unable to login to my webmail or admin interface.

zmcontrol status shows:

Host mail.xxxx.com.au
antispam Running
antivirus Running
imapproxy Running
ldap Running
logger Running
mailbox Stopped
zmmailboxdctl is not running
mta Running
snmp Running
spell Running
stats Running

A stop & start has the same effect.

/var/log/zimbra.log states:

Jul 7 18:56:07 mail zmmailboxdmgr[11792]: status requested
Jul 7 18:56:07 mail zmmailboxdmgr[11792]: stale pid 10278 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Jul 7 18:56:07 mail zmmailboxdmgr[11792]: assuming no other instance is running
Jul 7 18:56:07 mail zmmailboxdmgr[11792]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jul 7 18:56:07 mail zmmailboxdmgr[11792]: assuming no other instance is running
Jul 7 18:56:07 mail zmmailboxdmgr[11792]: no manager process is running

Thanks in Advance.

[uxbod]

Have a look in your Log Files - Zimbra :: Wiki for any error messages.

[PS-Man]

In the mailbox.log is:

2008-07-07 16:53:33,658 WARN [Shutdown] [] session - exception recording unloaded session's last access time
com.zimbra.common.service.ServiceException: system failure: getting database connection
ExceptionId:Shutdown:1215413613658:4997ba196d4b013 3
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:253)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 231)
at com.zimbra.cs.mailbox.Mailbox$MailboxChange.getCon nection(Mailbox.java:234)
at com.zimbra.cs.mailbox.Mailbox.getOperationConnecti on(Mailbox.java:1093)
at com.zimbra.cs.db.DbMailbox.recordLastSoapAccess(Db Mailbox.java:333)
at com.zimbra.cs.mailbox.Mailbox.recordLastSoapAccess Time(Mailbox.java:930)
at com.zimbra.cs.session.SoapSession.unregister(SoapS ession.java:340)
at com.zimbra.cs.session.SoapSession.unregister(SoapS ession.java:70)
at com.zimbra.cs.session.Session.doCleanup(Session.ja va:280)
at com.zimbra.cs.session.SessionCache.shutdown(Sessio nCache.java:142)
at com.zimbra.cs.util.Zimbra.shutdown(Zimbra.java:269 )
at com.zimbra.soap.SoapServlet.destroy(SoapServlet.ja va:125)
at org.mortbay.jetty.servlet.ServletHolder.destroyIns tance(ServletHolder.java:315)
at org.mortbay.jetty.servlet.ServletHolder.doStop(Ser vletHolder.java:286)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:65)
at org.mortbay.jetty.servlet.ServletHandler.doStop(Se rvletHandler.java:170)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:65)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:129)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:65)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:129)
at org.mortbay.jetty.servlet.SessionHandler.doStop(Se ssionHandler.java:124)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:65)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:129)
at org.mortbay.jetty.handler.ContextHandler.doStop(Co ntextHandler.java:557)
at org.mortbay.jetty.webapp.WebAppContext.doStop(WebA ppContext.java:486)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:65)
at org.mortbay.jetty.handler.HandlerCollection.doStop (HandlerCollection.java:164)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:65)
at org.mortbay.jetty.handler.HandlerCollection.doStop (HandlerCollection.java:164)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:65)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:129)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:65)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:129)
at org.mortbay.jetty.Server.doStop(Server.java:270)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:65)
at org.mortbay.jetty.Server$ShutdownHookThread.run(Se rver.java:548)
Caused by: com.mysql.jdbc.exceptions.MySQLNonTransientConnect ionException: Server connection failure during transaction. Due to underlying exception: 'java.net.SocketException: java.net.ConnectException: Connection refused'.

and in zmconvertd.log:

Mon Jul 7 17:04:27 2008: Ping successful.
Use of uninitialized value in string eq at /opt/zimbra/libexec/zmconvertdmon line 39.
Use of uninitialized value in string eq at /opt/zimbra/libexec/zmconvertdmon line 40.
Mon Jul 7 17:04:27 2008: Monitoring convertd at localhost:7047.
Mon Jul 7 17:04:27 2008: Pinging convertd.
Mon Jul 7 17:04:27 2008: Ping successful.
Mon Jul 7 17:04:42 2008: Pinging convertd.
Mon Jul 7 17:04:42 2008: Ping successful.
Mon Jul 7 17:04:42 2008: Pinging convertd.
Mon Jul 7 17:04:42 2008: Ping successful.
Mon Jul 7 17:04:57 2008: Pinging convertd.
Mon Jul 7 17:04:57 2008: Ping successful.
Mon Jul 7 17:04:57 2008: Pinging convertd.
Mon Jul 7 17:04:57 2008: Ping successful.
Mon Jul 7 17:04:58 2008: Sending quit command to convertd.
Mon Jul 7 17:04:58 2008: Sending quit command to convertd.
Mon Jul 7 17:04:58 2008: Cannot connect to convertd at localhost:7047: Connection refused
Use of uninitialized value in string eq at /opt/zimbra/libexec/zmconvertdmon line 39.
Use of uninitialized value in string eq at /opt/zimbra/libexec/zmconvertdmon line 40.
Mon Jul 7 17:06:02 2008: Monitoring convertd at localhost:7047.
Mon Jul 7 17:06:02 2008: Pinging convertd.
Mon Jul 7 17:06:02 2008: Cannot connect to convertd at localhost:7047: Connection refused
Mon Jul 7 17:06:02 2008: Could not connect to convertd.
Mon Jul 7 17:06:02 2008: Running /opt/zimbra/libexec/zmconvertd
Mon Jul 7 17:06:02 2008: Started convertd.

[uxbod]

Please check the file /opt/zimbra/logger/db/data/_zmhostname_.err to see if any of the MySQL tables are corrupt.

[PS-Man]

The last few entries from _zmhostname_.err:

080705 07:55:31 mysqld started
080705 7:55:32 InnoDB: Started; log sequence number 0 43655
080705 7:55:32 [Note] /opt/zimbra/logger/mysql/libexec/mysqld: ready for connections.
Version: '5.0.51a-log' socket: '/opt/zimbra/logger/db/mysql.sock' port: 7307 Source distribution
080706 18:24:24 [Note] /opt/zimbra/logger/mysql/libexec/mysqld: Normal shutdown

080706 18:24:24 InnoDB: Starting shutdown...
080706 18:24:24 InnoDB: Shutdown completed; log sequence number 0 43655
080706 18:24:24 [Note] /opt/zimbra/logger/mysql/libexec/mysqld: Shutdown complete

080706 18:24:24 mysqld ended

[PS-Man]

Just noticed something of interest. When I execute:

zmprov gs 'mail.concisecontrol.com.au' zimbraAdminUrl

I receive:

ERROR: zclient.IO_ERROR (invoke Connection refused, server: localhost) (cause: java.net.ConnectException Connection refused)

And, cat /opt/zimbra/log/zmmailboxd.out:

Invalid initial heap size: -Xms4873m
The specified size exceeds the maximum representable size.
Could not create the Java virtual machine.

Thanks.

[uxbod]

su - zimbra
zmlocalconfig | grep mailboxd_java_heap_memory_percent

If this is greater than 35% then try the following

su - zimbra
zmlocalconfig -e mailboxd_java_heap_memory_percent=35

and then restart ZCS. I think the problem is when the JVM is being created is > 3.5GB.

Are you running 32 or 64 bit ?

Also, http://www.zimbra.com/forums/announc...r-profile.html please so we know which OS and version of ZCS

[PS-Man]

The mailboxd_java_heap_memory_percent was at 60, set to 35, restarted ZCS.
Same problem, new log items in:

cat /opt/zimbra/log/zmmailboxd.out:

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

Cheers.

[uxbod]

Okay Set it at 30% and restart ZCS. It should then allocate :-

((8*1024)/100)*30 = 2457.6MB

[uxbod]

Well according to Bug 27576 – java_heap_memory_percent wrongly allocated by installer it should be set @ 18% for 32bit systems.