latest drama...

[sluggo]

My brand new server trashed a mirror when the controller died with a bang, and that took care of that

While server was being reloaded, I read up and decided on:

Network_Edition_Disaster_Recovery
as my game plan.
I followed each step carefully, and all went well until the end,
where I get to the step to:
zmmailboxdctl start
where I get:
tomkat zmmailboxdmgr[1744]: assuming no other instance is running
Mar 31 14:40:04 tomkat zmmailboxdmgr[1744]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 31 14:40:04 tomkat zmmailboxdmgr[1744]: assuming no other instance is running
Mar 31 14:40:04 tomkat zmmailboxdmgr[1744]: no manager process is running
Mar 31 14:40:14 tomkat zmmailboxdmgr[2622]: status requested
Mar 31 14:40:14 tomkat zmmailboxdmgr[2622]: stale pid 32621 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
And it will never start, and I can’t do the zmrestoreoffline.
I was able to zmrestoreldap from backup
Output of host `hostname -f`
tomkat.dynandu.com has address 173.193.182.138
tomkat.dynandu.com mail is handled by 0 tomkat.dynandu.com.
We are still down, please suggest.
thanks

[ajcody]

Can you update the forum post with the logs mentioned in that section or at least the part that looks relevant?

You'll also note, mailboxd will actually be shutdown for the offline restore, but the fact it can't start probably indicates a problem anyways that should be address before continuing.

I'm actually off shift at this point today but did update the case created by your voicemail. I included in that case this forum url and your contact information. Because you don't have a support contract yet, I can't actually update the case with your personal details.

Regards,
Adam

###
Note, A Common Problem Described Here And A Change In The Order Of The Old DR Steps:

To avoid the "No appenders could be found for logger (zimbra.misc) / Please initialize log4j" and other related problems that happened during the zmrestoreoffline. We'll start and then stop the mailboxd service:

zmmailboxdctl start
This could take a couple of minutes before it comes up. Monitor activity by : tail -f /var/log/zimbra.log
If you think there's an issue, did you confirm your passwords? Look for something like the following in /var/log/zimbra.log :
Oct 20 17:39:33 mail zimbramon[9745]: 9745:info: zmmtaconfig: gacf ERROR: service.FAILURE (system failure: unable to get config) (cause: javax.naming.AuthenticationException [LDAP: error code 49 - Invalid Credentials])
See the section above to fix: "9. Double check your LDAP password"
zmmailboxdctl stop
This will configure the log4j.properties and create the necessary log files, such as mailbox.log .

[sluggo]

zimbra.log:

Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: antivirus: Stopped
Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: convertd: Running
Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: ldap: Running
Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: logger: Stopped
Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: mailbox: Stopped
Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: mta: Stopped
Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: snmp: Stopped
Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: spell: Stopped
Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: stats: Stopped
Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: zmconfigd: Stopped
Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: status requested
Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: stale pid 32621 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: assuming no other instance is running
Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: assuming no other instance is running
Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: no manager process is running
Mar 31 17:30:01 tomkat postfix/postqueue[4145]: fatal: Queue report unavailable - mail system is down
Mar 31 17:30:01 tomkat zimbramon[4125]: 4125:info: 2012-03-31 17:30:01, QUEUE: 0 0
Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: status requested
Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: stale pid 32621 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: assuming no other instance is running
Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: assuming no other instance is running
Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: no manager process is running
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: antispam: Stopped
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: antivirus: Stopped
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: convertd: Running
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: ldap: Running
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: logger: Stopped
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: mailbox: Stopped
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: mta: Stopped
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: snmp: Stopped
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: spell: Stopped
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: stats: Stopped
Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: zmconfigd: Stopped

zmmailbod.out:
concurrent mark-sweep generation total 3047424K, used 0K [0x000000073e000000, 0x00000007f8000000, 0x00000007f8000000)
concurrent-mark-sweep perm gen total 131072K, used 4413K [0x00000007f8000000, 0x0000000800000000, 0x0000000800000000)
CompilerOracle: exclude com/zimbra/cs/session/SessionMap.putAndPrune
CompilerOracle: exclude com/zimbra/cs/mailbox/MailItem.delete
CompilerOracle: exclude org/apache/xerces/impl/XMLDocumentFragmentScannerImpl$FragmentContentDisp atcher.dispatch
0 INFO [main] log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
1 WARN [main] log - EXCEPTION
java.io.FileNotFoundException: /disk3/zimbra/jetty-6.1.22.z6/etc/jetty.properties (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.jav a:120)
at org.mortbay.resource.FileResource.getInputStream(F ileResource.java:275)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:965)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)
Heap
par new generation total 914240K, used 48760K [0x0000000700000000, 0x000000073e000000, 0x000000073e000000)
eden space 812672K, 6% used [0x0000000700000000, 0x0000000702f9e230, 0x00000007319a0000)
from space 101568K, 0% used [0x00000007319a0000, 0x00000007319a0000, 0x0000000737cd0000)
to space 101568K, 0% used [0x0000000737cd0000, 0x0000000737cd0000, 0x000000073e000000)

[ajcody]

can you explain this?

/disk3/zimbra/jetty-6.1.22.z6/etc/jetty.properties (No such file or directory)

The disk3 part? Did you not use /opt/zimbra for the path or have you used symbolic links or something?

Can you double check the permission or as root, run the zmfixperms.

/opt/zimbra/libexec/zmfixperms --verbose --extended

[sluggo]

the permissions are right...
and yes it is a symbolic link, and has been from the start.

What happened was, the system crashed hard as I said. after OS reload, I re-installed zimbra, and restored the original zimbra from backups to an alternate location. Then, as directed from the DR wiki document, I copied /backup/sessions/
to the new install, and did a zmrestoreldap, which worked, as did everything up until running zmmailboxdctl start.

I ha already investigated the line from the log you mentioned, but it was at a point where I had been awake far to long and incorrectly concluded that the file was not on the original config either. I just checked and saw it was indeed on the original, so I copied it to the new config, re-ran
zmmailboxd.out,

which returned the following output to zmmailboxd.out:
CompilerOracle: exclude com/zimbra/cs/session/SessionMap.putAndPrune
CompilerOracle: exclude com/zimbra/cs/mailbox/MailItem.delete
CompilerOracle: exclude org/apache/xerces/impl/XMLDocumentFragmentScannerImpl$FragmentContentDisp atcher.dispatch
0 INFO [main] log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
Total time for which application threads were stopped: 0.0001120 seconds
Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Fatal error: exception while binding to ports
java.net.SocketException: Unbound server sockets not implemented
at javax.net.ServerSocketFactory.createServerSocket(S erverSocketFactory.java:80)
at com.zimbra.common.util.NetUtil.newBoundServerSocke t(NetUtil.java:95)
at com.zimbra.common.util.NetUtil.bindServerSocket(Ne tUtil.java:163)
at com.zimbra.common.util.NetUtil.bindSslTcpServerSoc ket(NetUtil.java:54)
at com.zimbra.common.handlers.PrivilegedHandler.openP orts(PrivilegedHandler.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.xml.XmlConfiguration.call(XmlConfigura tion.java:542)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:252)
at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:214)
at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:974)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)
Heap
par new generation total 914240K, used 16253K [0x0000000700000000, 0x000000073e000000, 0x000000073e000000)
eden space 812672K, 2% used [0x0000000700000000, 0x0000000700fdf5d0, 0x00000007319a0000)
from space 101568K, 0% used [0x00000007319a0000, 0x00000007319a0000, 0x0000000737cd0000)
to space 101568K, 0% used [0x0000000737cd0000, 0x0000000737cd0000, 0x000000073e000000)
concurrent mark-sweep generation total 3047424K, used 0K [0x000000073e000000, 0x00000007f8000000, 0x00000007f8000000)
concurrent-mark-sweep perm gen total 131072K, used 9166K [0x00000007f8000000, 0x0000000800000000, 0x0000000800000000)

If there is any issue with the symlink or any other config aspect, I can mount a larger volume under /opt and re-install there without the symlink, although it was working fine there before the crash, so I concluded this was not the problem.

I'll try any suggestion though...

Thanks!!

Mike

[ajcody]

error did change, let's now continue with..

zmmailboxdctl stop

And then the next section:

Last check before doing system [-sys] and user data restores [-a all]

Before you attempt the zmrestoreoffline below; make sure mailboxd isn't running and that only the necessary services are running. (ldap, mysql.server, convertd [unless MAC w/ ZCS 4.x or 5.x]). To confirm [as zimbra user] :

zmcontrol status note running and those services that aren't.
mysql.server status
ldap status
zmconvertctl status

Then proceed with the zmrestoreoffline.

Remember to do the tail and post any errors/issues you see on this forum thread.

tail -f /opt/zimbra/log/mailbox.log

[sluggo]

while the log output did change, mailboxd never did start,

and ofline restore returned:

zimbra@tomkat [~]# zmrestoreoffline -sys -a all -c -br
mailboxd is not running.
log4j:WARN No appenders could be found for logger (zimbra.system).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See Apache log4j 1.2 - Frequently Asked Technical Questions for more info.

on command line, but could not find this in any log

~m

[ajcody]

make sure this file exists and zimbra is able to write to it

/opt/zimbra/conf/log4j.properties.in

[sluggo]

it is here

-rw-rw-r-- 1 zimbra zimbra 6051 Jan 5 12:17 log4j.properties.in

[ajcody]

That error is related to some type of log file not being present or is not writeable.
Does this one exist?
/var/log/zimbra.log

I've left a message in our support conference room that you have been working through a DR and asked other to lend a hand if they can. I, unfortunately have a prior engagement that I can't cancel - otherwise I would.

You might just have to go through the list of log files and do a touch filename on those that don't exist and restart starting/stopping mailboxd to see if that resolves it. I have the log files listed here:

Ajcody-Logging - Zimbra :: Wiki