Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: latest drama...

  1. #1
    sluggo is offline Intermediate Member
    Join Date
    Mar 2012
    Posts
    17
    Rep Power
    3

    Default latest drama...

    My brand new server trashed a mirror when the controller died with a bang, and that took care of that

    While server was being reloaded, I read up and decided on:

    Network_Edition_Disaster_Recovery
    as my game plan.
    I followed each step carefully, and all went well until the end,
    where I get to the step to:
    zmmailboxdctl start
    where I get:
    tomkat zmmailboxdmgr[1744]: assuming no other instance is running
    Mar 31 14:40:04 tomkat zmmailboxdmgr[1744]: file /opt/zimbra/log/zmmailboxd.pid does not exist
    Mar 31 14:40:04 tomkat zmmailboxdmgr[1744]: assuming no other instance is running
    Mar 31 14:40:04 tomkat zmmailboxdmgr[1744]: no manager process is running
    Mar 31 14:40:14 tomkat zmmailboxdmgr[2622]: status requested
    Mar 31 14:40:14 tomkat zmmailboxdmgr[2622]: stale pid 32621 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
    And it will never start, and I canít do the zmrestoreoffline.
    I was able to zmrestoreldap from backup
    Output of host `hostname -f`
    tomkat.dynandu.com has address 173.193.182.138
    tomkat.dynandu.com mail is handled by 0 tomkat.dynandu.com.
    We are still down, please suggest.
    thanks

  2. #2
    ajcody is offline Zimbra Employee
    Join Date
    Mar 2008
    Posts
    31
    Rep Power
    7

    Default

    Can you update the forum post with the logs mentioned in that section or at least the part that looks relevant?

    You'll also note, mailboxd will actually be shutdown for the offline restore, but the fact it can't start probably indicates a problem anyways that should be address before continuing.

    I'm actually off shift at this point today but did update the case created by your voicemail. I included in that case this forum url and your contact information. Because you don't have a support contract yet, I can't actually update the case with your personal details.

    Regards,
    Adam

    ###
    Note, A Common Problem Described Here And A Change In The Order Of The Old DR Steps:

    To avoid the "No appenders could be found for logger (zimbra.misc) / Please initialize log4j" and other related problems that happened during the zmrestoreoffline. We'll start and then stop the mailboxd service:

    zmmailboxdctl start
    This could take a couple of minutes before it comes up. Monitor activity by : tail -f /var/log/zimbra.log
    If you think there's an issue, did you confirm your passwords? Look for something like the following in /var/log/zimbra.log :
    Oct 20 17:39:33 mail zimbramon[9745]: 9745:info: zmmtaconfig: gacf ERROR: service.FAILURE (system failure: unable to get config) (cause: javax.naming.AuthenticationException [LDAP: error code 49 - Invalid Credentials])
    See the section above to fix: "9. Double check your LDAP password"
    zmmailboxdctl stop
    This will configure the log4j.properties and create the necessary log files, such as mailbox.log .

  3. #3
    sluggo is offline Intermediate Member
    Join Date
    Mar 2012
    Posts
    17
    Rep Power
    3

    Default more log data...

    zimbra.log:

    Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: antivirus: Stopped
    Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: convertd: Running
    Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: ldap: Running
    Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: logger: Stopped
    Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: mailbox: Stopped
    Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: mta: Stopped
    Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: snmp: Stopped
    Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: spell: Stopped
    Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: stats: Stopped
    Mar 31 17:28:07 tomkat zimbramon[2908]: 2908:info: 2012-03-31 17:28:02, STATUS: tomkat.dynandu.com: zmconfigd: Stopped
    Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: status requested
    Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: stale pid 32621 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
    Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: assuming no other instance is running
    Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: file /opt/zimbra/log/zmmailboxd.pid does not exist
    Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: assuming no other instance is running
    Mar 31 17:29:51 tomkat zmmailboxdmgr[3941]: no manager process is running
    Mar 31 17:30:01 tomkat postfix/postqueue[4145]: fatal: Queue report unavailable - mail system is down
    Mar 31 17:30:01 tomkat zimbramon[4125]: 4125:info: 2012-03-31 17:30:01, QUEUE: 0 0
    Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: status requested
    Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: stale pid 32621 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
    Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: assuming no other instance is running
    Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: file /opt/zimbra/log/zmmailboxd.pid does not exist
    Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: assuming no other instance is running
    Mar 31 17:30:04 tomkat zmmailboxdmgr[4610]: no manager process is running
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: antispam: Stopped
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: antivirus: Stopped
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: convertd: Running
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: ldap: Running
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: logger: Stopped
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: mailbox: Stopped
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: mta: Stopped
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: snmp: Stopped
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: spell: Stopped
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: stats: Stopped
    Mar 31 17:30:05 tomkat zimbramon[4133]: 4133:info: 2012-03-31 17:30:01, STATUS: tomkat.dynandu.com: zmconfigd: Stopped

    zmmailbod.out:
    concurrent mark-sweep generation total 3047424K, used 0K [0x000000073e000000, 0x00000007f8000000, 0x00000007f8000000)
    concurrent-mark-sweep perm gen total 131072K, used 4413K [0x00000007f8000000, 0x0000000800000000, 0x0000000800000000)
    CompilerOracle: exclude com/zimbra/cs/session/SessionMap.putAndPrune
    CompilerOracle: exclude com/zimbra/cs/mailbox/MailItem.delete
    CompilerOracle: exclude org/apache/xerces/impl/XMLDocumentFragmentScannerImpl$FragmentContentDisp atcher.dispatch
    0 INFO [main] log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    1 WARN [main] log - EXCEPTION
    java.io.FileNotFoundException: /disk3/zimbra/jetty-6.1.22.z6/etc/jetty.properties (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.jav a:120)
    at org.mortbay.resource.FileResource.getInputStream(F ileResource.java:275)
    at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:965)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.mortbay.start.Main.invokeMain(Main.java:194)
    at org.mortbay.start.Main.start(Main.java:534)
    at org.mortbay.start.Main.start(Main.java:441)
    at org.mortbay.start.Main.main(Main.java:119)
    Heap
    par new generation total 914240K, used 48760K [0x0000000700000000, 0x000000073e000000, 0x000000073e000000)
    eden space 812672K, 6% used [0x0000000700000000, 0x0000000702f9e230, 0x00000007319a0000)
    from space 101568K, 0% used [0x00000007319a0000, 0x00000007319a0000, 0x0000000737cd0000)
    to space 101568K, 0% used [0x0000000737cd0000, 0x0000000737cd0000, 0x000000073e000000)

  4. #4
    ajcody is offline Zimbra Employee
    Join Date
    Mar 2008
    Posts
    31
    Rep Power
    7

    Default

    can you explain this?

    /disk3/zimbra/jetty-6.1.22.z6/etc/jetty.properties (No such file or directory)

    The disk3 part? Did you not use /opt/zimbra for the path or have you used symbolic links or something?

    Can you double check the permission or as root, run the zmfixperms.

    /opt/zimbra/libexec/zmfixperms --verbose --extended

  5. #5
    sluggo is offline Intermediate Member
    Join Date
    Mar 2012
    Posts
    17
    Rep Power
    3

    Default disk3

    the permissions are right...
    and yes it is a symbolic link, and has been from the start.

    What happened was, the system crashed hard as I said. after OS reload, I re-installed zimbra, and restored the original zimbra from backups to an alternate location. Then, as directed from the DR wiki document, I copied /backup/sessions/
    to the new install, and did a zmrestoreldap, which worked, as did everything up until running zmmailboxdctl start.

    I ha already investigated the line from the log you mentioned, but it was at a point where I had been awake far to long and incorrectly concluded that the file was not on the original config either. I just checked and saw it was indeed on the original, so I copied it to the new config, re-ran
    zmmailboxd.out,

    which returned the following output to zmmailboxd.out:
    CompilerOracle: exclude com/zimbra/cs/session/SessionMap.putAndPrune
    CompilerOracle: exclude com/zimbra/cs/mailbox/MailItem.delete
    CompilerOracle: exclude org/apache/xerces/impl/XMLDocumentFragmentScannerImpl$FragmentContentDisp atcher.dispatch
    0 INFO [main] log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    Total time for which application threads were stopped: 0.0001120 seconds
    Zimbra server reserving server socket port=110 bindaddr=null ssl=false
    Zimbra server reserving server socket port=995 bindaddr=null ssl=true
    Fatal error: exception while binding to ports
    java.net.SocketException: Unbound server sockets not implemented
    at javax.net.ServerSocketFactory.createServerSocket(S erverSocketFactory.java:80)
    at com.zimbra.common.util.NetUtil.newBoundServerSocke t(NetUtil.java:95)
    at com.zimbra.common.util.NetUtil.bindServerSocket(Ne tUtil.java:163)
    at com.zimbra.common.util.NetUtil.bindSslTcpServerSoc ket(NetUtil.java:54)
    at com.zimbra.common.handlers.PrivilegedHandler.openP orts(PrivilegedHandler.java:82)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.mortbay.xml.XmlConfiguration.call(XmlConfigura tion.java:542)
    at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:252)
    at org.mortbay.xml.XmlConfiguration.configure(XmlConf iguration.java:214)
    at org.mortbay.xml.XmlConfiguration.main(XmlConfigura tion.java:974)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.mortbay.start.Main.invokeMain(Main.java:194)
    at org.mortbay.start.Main.start(Main.java:534)
    at org.mortbay.start.Main.start(Main.java:441)
    at org.mortbay.start.Main.main(Main.java:119)
    Heap
    par new generation total 914240K, used 16253K [0x0000000700000000, 0x000000073e000000, 0x000000073e000000)
    eden space 812672K, 2% used [0x0000000700000000, 0x0000000700fdf5d0, 0x00000007319a0000)
    from space 101568K, 0% used [0x00000007319a0000, 0x00000007319a0000, 0x0000000737cd0000)
    to space 101568K, 0% used [0x0000000737cd0000, 0x0000000737cd0000, 0x000000073e000000)
    concurrent mark-sweep generation total 3047424K, used 0K [0x000000073e000000, 0x00000007f8000000, 0x00000007f8000000)
    concurrent-mark-sweep perm gen total 131072K, used 9166K [0x00000007f8000000, 0x0000000800000000, 0x0000000800000000)

    If there is any issue with the symlink or any other config aspect, I can mount a larger volume under /opt and re-install there without the symlink, although it was working fine there before the crash, so I concluded this was not the problem.

    I'll try any suggestion though...

    Thanks!!

    Mike

  6. #6
    ajcody is offline Zimbra Employee
    Join Date
    Mar 2008
    Posts
    31
    Rep Power
    7

    Default

    error did change, let's now continue with..

    zmmailboxdctl stop

    And then the next section:

    Last check before doing system [-sys] and user data restores [-a all]

    Before you attempt the zmrestoreoffline below; make sure mailboxd isn't running and that only the necessary services are running. (ldap, mysql.server, convertd [unless MAC w/ ZCS 4.x or 5.x]). To confirm [as zimbra user] :

    zmcontrol status note running and those services that aren't.
    mysql.server status
    ldap status
    zmconvertctl status

    Then proceed with the zmrestoreoffline.

    Remember to do the tail and post any errors/issues you see on this forum thread.

    tail -f /opt/zimbra/log/mailbox.log

  7. #7
    sluggo is offline Intermediate Member
    Join Date
    Mar 2012
    Posts
    17
    Rep Power
    3

    Default contiued

    while the log output did change, mailboxd never did start,

    and ofline restore returned:

    zimbra@tomkat [~]# zmrestoreoffline -sys -a all -c -br
    mailboxd is not running.
    log4j:WARN No appenders could be found for logger (zimbra.system).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See Apache log4j 1.2 - Frequently Asked Technical Questions for more info.

    on command line, but could not find this in any log

    ~m

  8. #8
    ajcody is offline Zimbra Employee
    Join Date
    Mar 2008
    Posts
    31
    Rep Power
    7

    Default

    make sure this file exists and zimbra is able to write to it

    /opt/zimbra/conf/log4j.properties.in

  9. #9
    sluggo is offline Intermediate Member
    Join Date
    Mar 2012
    Posts
    17
    Rep Power
    3

    Default log file

    it is here

    -rw-rw-r-- 1 zimbra zimbra 6051 Jan 5 12:17 log4j.properties.in

  10. #10
    ajcody is offline Zimbra Employee
    Join Date
    Mar 2008
    Posts
    31
    Rep Power
    7

    Default

    That error is related to some type of log file not being present or is not writeable.
    Does this one exist?
    /var/log/zimbra.log

    I've left a message in our support conference room that you have been working through a DR and asked other to lend a hand if they can. I, unfortunately have a prior engagement that I can't cancel - otherwise I would.

    You might just have to go through the list of log files and do a touch filename on those that don't exist and restart starting/stopping mailboxd to see if that resolves it. I have the log files listed here:

    Ajcody-Logging - Zimbra :: Wiki

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Where to get latest source...
    By ronnyek in forum Developers
    Replies: 11
    Last Post: 07-18-2008, 10:06 AM
  2. Replies: 6
    Last Post: 03-06-2008, 06:26 PM
  3. upgrade zcs-4.5.5_GA to latest stable version
    By kmuralidharan in forum Administrators
    Replies: 3
    Last Post: 12-19-2007, 04:45 AM
  4. Where do we get the latest pst importer?
    By kurt in forum Migration
    Replies: 4
    Last Post: 09-12-2007, 09:28 AM
  5. Latest network edition
    By mansuper in forum Installation
    Replies: 6
    Last Post: 09-12-2006, 04:46 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •