zmmailboxdctl is not running.

[cesarliws]

I had a ldap db corrupted this week, and now after work for 3 days I manage to recover it, but I got a new problem "zmmailboxdctl is not running."
All services start Ok, but after that mailbox is not working anymore.

zimbra@mail:~/log> zmcontrol start
Host mail.myserver.com
Starting ldap...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.

zimbra@mail:~/log> zmcontrol status
Host mail.myserver.com
ldap Running
logger Running
mailbox Stopped
zmmailboxdctl is not running.
mta Running
snmp Running
spell Running
stats Running


I review every possible piece of zimbra and I can't even know what is cause.

- Fix the folder permitions
- I recreate the certificates
- Review DNS, hosts, hostname

Among other things and still got nothing, all logs give no clue about what is happening.

The only info I got is from zmmailboxd.out, after change the log level, where I can see:
...
4912 WARN [main] log - Config error at
4912 WARN [main] log - Config error at
4913 WARN [main] log - EXCEPTION
...

I have no idea what that means, can someone give some directions here?


After that I also can see:
...
4983 WARN [main] log - Nested in java.lang.reflect.InvocationTargetException:
java.net.BindException: Address already in use
...

But what is the address? What is the port?
Would be helpful if the log can be more clear.

Trying to figure out what can be the cause, I telnet all the ports used by zimbra, and for all it seems to be OK

Like this:

mail:/opt/zimbra/log # telnet localhost 25
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

I did the same for all:
25, 110, 81, 995, 993, 143, 7025, 7071 ports.

Here is the content of zmmailboxd.out log file:


0 INFO [main] log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
Total time for which application threads were stopped: 0.0002070 seconds
Total time for which application threads were stopped: 0.0002330 seconds
Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Total time for which application threads were stopped: 0.0005650 seconds
Zimbra server reserving server socket port=143 bindaddr=null ssl=false
Zimbra server reserving server socket port=993 bindaddr=null ssl=true
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
4912 WARN [main] log - Config error at
4912 WARN [main] log - Config error at
4913 WARN [main] log - EXCEPTION
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.xml.XmlConfiguration.call(XmlConfigura tion.java:542)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:252)
at org.mortbay.xml.XmlConfiguration.refObj(XmlConfigu ration.java:646)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:260)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:214)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:974)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSock etChannelImpl.java:119)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:59)
at org.mortbay.jetty.nio.SelectChannelConnector.open( SelectChannelConnector.java:216)
... 18 more
4916 WARN [main] log - Nested in java.lang.reflect.InvocationTargetException:
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSock etChannelImpl.java:119)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:59)
at org.mortbay.jetty.nio.SelectChannelConnector.open( SelectChannelConnector.java:216)

... 18 more
4916 WARN [main] log - Nested in java.lang.reflect.InvocationTargetException:
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSock etChannelImpl.java:119)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:59)
at org.mortbay.jetty.nio.SelectChannelConnector.open( SelectChannelConnector.java:216)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.xml.XmlConfiguration.call(XmlConfigura tion.java:542)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:252)
at org.mortbay.xml.XmlConfiguration.refObj(XmlConfigu ration.java:646)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:260)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:214)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:974)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)
Total time for which application threads were stopped: 0.0361240 seconds
4972 INFO [Shutdown] log - Shutdown hook executing
4972 INFO [Shutdown] log - Shutdown hook complete
Heap
def new generation total 116160K, used 39309K [0x8c710000, 0x94510000, 0x94510000)
eden space 103296K, 38% used [0x8c710000, 0x8ed73708, 0x92bf0000)
from space 12864K, 0% used [0x92bf0000, 0x92bf0000, 0x93880000)
to space 12864K, 0% used [0x93880000, 0x93880000, 0x94510000)
concurrent mark-sweep generation total 388096K, used 0K [0x94510000, 0xac010000, 0xac110000)
concurrent-mark-sweep perm gen total 131072K, used 7707K [0xac110000, 0xb4110000, 0xb4110000)
0 INFO [main] log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
Total time for which application threads were stopped: 0.0002080 seconds
Total time for which application threads were stopped: 0.0002430 seconds
Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Total time for which application threads were stopped: 0.0532970 seconds
Zimbra server reserving server socket port=143 bindaddr=null ssl=false
Zimbra server reserving server socket port=993 bindaddr=null ssl=true
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
4979 WARN [main] log - Config error at
4979 WARN [main] log - Config error at
4980 WARN [main] log - EXCEPTION
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.xml.XmlConfiguration.call(XmlConfigura tion.java:542)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:252)
at org.mortbay.xml.XmlConfiguration.refObj(XmlConfigu ration.java:646)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:260)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:214)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:974)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSock etChannelImpl.java:119)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:59)
at org.mortbay.jetty.nio.SelectChannelConnector.open( SelectChannelConnector.java:216)
... 18 more
4983 WARN [main] log - Nested in java.lang.reflect.InvocationTargetException:
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSock etChannelImpl.java:119)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:59)
at org.mortbay.jetty.nio.SelectChannelConnector.open( SelectChannelConnector.java:216)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.xml.XmlConfiguration.call(XmlConfigura tion.java:542)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:252)
at org.mortbay.xml.XmlConfiguration.refObj(XmlConfigu ration.java:646)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:260)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:214)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:974)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)
Total time for which application threads were stopped: 0.0205460 seconds
5007 INFO [Shutdown] log - Shutdown hook executing
5007 INFO [Shutdown] log - Shutdown hook complete
Heap
def new generation total 116160K, used 39310K [0x8c700000, 0x94500000, 0x94500000)
eden space 103296K, 38% used [0x8c700000, 0x8ed63808, 0x92be0000)
from space 12864K, 0% used [0x92be0000, 0x92be0000, 0x93870000)
to space 12864K, 0% used [0x93870000, 0x93870000, 0x94500000)
concurrent mark-sweep generation total 388096K, used 0K [0x94500000, 0xac000000, 0xac100000)
concurrent-mark-sweep perm gen total 131072K, used 7706K [0xac100000, 0xb4100000, 0xb4100000)

[fultonj]

I had a very similar problem upgrading from 5.0.18 to 6.0.7. The JVM which comes with Zimbra 6 needs a tweak which you can apply to each store server:

zmprov ms $store_server zimbraMailUseDirectBuffer FALSE

[fselendic]

Same issue, same errors in zmmailboxd.out...

Not sure if I understand proposed tweak, it was happily running until filesystem crash this morning.

Can't bind to something, we can't find what.
Zimbra 6.0.4 NE. Heeeelp. Urgent.

[fselendic]

OK, solved for us by nice people from Zimbra support. Fast and really profesional.

I will have a nice long chat with our system admin, though...turns out that something was holding a port after all.

[tuanta]

Can you share the full solution?