Results 1 to 9 of 9

Thread: Total Innodb/MySQL failure

  1. #1
    rotorboy is offline Special Member
    Join Date
    Mar 2008
    Location
    Canada
    Posts
    148
    Rep Power
    7

    Default Total Innodb/MySQL failure

    Greetings.
    So tonight a PDU in our data facility randomly flaked out causing power loss to all attached equipment. Everything came up fine except for Zimbra. It took quite awhile before I was able to locate that mysql was not starting (even though it said it had). I tried following the steps: Mysql Crash Recovery - Zimbra :: Wiki
    but that was a miserable failure. I could not even get MySQL to drop databases it was so badly hosed.

    Fortunately I take a hot backup of the entire /opt/zimbra directory nightly via rsync. I realized I could just swap out the zimbra/db directory for the backup, chown, and restart mysql.

    Now that said... there MUST be a better way than this to handle corrupted databases....

  2. #2
    FritzBrause is offline Partner (VAR/HSP)
    Join Date
    Jan 2008
    Location
    Berlin, Germany
    Posts
    201
    Rep Power
    7

    Default

    Why do you assume that the database was corrupt? It is very unlikely that a simple crash will take out your entire MySQL DB. Did you check the logfiles for errors? I'm just curious since I work with MySQL for over a decade now and haven't seen this before.
    Last edited by FritzBrause; 03-18-2012 at 11:30 AM.
    Best regards

    Andreas Wolske

    managedhosting.de GmbH

    Zimbra Cloud Hosting, Consulting & Support
    VMware vCloud Powered Serviceprovider

  3. #3
    rotorboy is offline Special Member
    Join Date
    Mar 2008
    Location
    Canada
    Posts
    148
    Rep Power
    7

    Default MySQL fixed - but now folders are misreporting contents

    Greetings - here's a better update

    On Friday our server lost power when an APC PDU glitched. Upon reboot the zimbra server was not able to start MySQL. I restarted MySQL and it seemed to be fine so off I went. A couple hours later, zimbra would not startup and MySQL was filled with errors.

    I was able to do the dumping process from: Mysql Crash Recovery - Zimbra :: Wiki
    but I could not get past step 3 - every attempt to drop the databases results in lost connections to MySQL. To get past this I shut down zimbra and mysql, moved the db directory elsewhere, and copied the previous days db directory back to /opt/zimbra/db and chowned it to zimbra. I then was able proceed through the rest of the recovery steps and right now MySQL seems to be happy and running fine.

    So currently mail is coming in and going out and seems normal. The only problem is that on one folder (and a couple users reported issues similar) I have an unread message count of 7 - however I have deleted all messages in that folder - there's nothing there!

    I've re-indexed the account - no change. I've deleted and re-added my only IMAP devices (Android phone) - no change. Other than the android I only use Zimbra webmail.

  4. #4
    FritzBrause is offline Partner (VAR/HSP)
    Join Date
    Jan 2008
    Location
    Berlin, Germany
    Posts
    201
    Rep Power
    7

    Default

    I would suggest to check your Database manually:

    First shut down Zimbra via "zmcontrol stop" and now start the MySQL Server
    by "mysql.server start" (as the zimbra user) Now you can check th DB for errors:

    Code:
    /opt/zimbra/mysql/bin/mysqlcheck --defaults-file=/opt/zimbra/conf/my.cnf -S /opt/zimbra/db/mysql.sock -A -C -s -u root --password=XXXXXXXXXXX
    The MySQL Root password can be found by

    Code:
    zmlocalconfig -s | grep mysql_root_password
    If the check is reporting errors you can repair the reported tables by

    Code:
    show tables;
    check table <for each table reported above and if any fail>
    repair table <table name>
    --

    The problem might be that your datastore is out of sync with the information stored in the database. So running zmblobchk might also be an option to narrow down the problems.

    --

    If you can send /opt/zimbra/log/mysql_error.log via PM I can have a look.
    Best regards

    Andreas Wolske

    managedhosting.de GmbH

    Zimbra Cloud Hosting, Consulting & Support
    VMware vCloud Powered Serviceprovider

  5. #5
    rotorboy is offline Special Member
    Join Date
    Mar 2008
    Location
    Canada
    Posts
    148
    Rep Power
    7

    Default

    Currently the MySQL log files are not generating any errors. I don't see any private information here so I've posted it as a text file to this forum.

    I will check the databases later but for now I have to limit my work to anything that will not result in further outages until after about 9pm (PDT).

    I ran zmblobchk and there are did give me 7942 errors such as:
    Mailbox 3, item 48851, rev 85832, volume 1, /opt/zimbra/store/0/3/msg/11/48851-85832.msg: blob not found.
    Mailbox 8, volume 1, /opt/zimbra/store/0/8/msg/7/31529-110468.msg: unexpected blob. File size is 1829.
    Mailbox 188, item 23300, rev 49803, volume 1, /opt/zimbra/store/0/188/msg/5/23300-49371.msg: incorrect data size. Expected 21532, was 17018. File size is 17018.
    Mailbox 188, item 23322, rev 49901, volume 1, /opt/zimbra/store/0/188/msg/5/23322-49442.msg: file has incorrect revision.
    Attached Files Attached Files

  6. #6
    rotorboy is offline Special Member
    Join Date
    Mar 2008
    Location
    Canada
    Posts
    148
    Rep Power
    7

    Default I found messages... now to get them back to the users

    So I'm new to zmblobcheck and tried this:
    zmblobchk -m 134 --unexpected-blob-list /tmp/134-unexpected-blob-list.txt start >/tmp/zmblobcheck-134.txt

    I checked the list of files in the store and opened a few up such as:
    I then did a more /opt/zimbra/store/0/134/msg/49/200805-366805.msg

    This is indeed a message that should have been in one of my folders but I never saw it in Zimbra.

    So now that I have a list of mails I am on the search for some sort of way to get those indexed, re-delivered, imported... or whatever has to happen.

    BTW - thanks so much for your help!

  7. #7
    FritzBrause is offline Partner (VAR/HSP)
    Join Date
    Jan 2008
    Location
    Berlin, Germany
    Posts
    201
    Rep Power
    7

    Default

    The MySQL "InnoDB: Assertion failure in thread 1232542016 in file btr/btr0btr.c line 2386" suggests that InnoDB Metadata is corrupt. This happens if the underlying Filesystem is becoming corrupt due to a power outage. (RAID Controller Cache issue). The MySQL log shows the creation of new metadata files due to Filesystem errors:

    Code:
    120318  9:56:15  InnoDB: Setting file ./ibdata1 size to 10 MB
    120318  9:56:15  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
    120318  9:56:28  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
    120318 10:01:03  InnoDB: Operating system error number 17 in a file operation.
    I would run a fsck as soon as possible to make sure that the underlying fs is o.k.

    If your Database is O.K. by now I would use zmblobchk to resolve the issues with some mailboxes. There's a goog WiKi Article: Ajcody-Notes-No-Such-Blob - Zimbra :: Wiki

    There's probably not much more you can do about this.

    HTH
    Best regards

    Andreas Wolske

    managedhosting.de GmbH

    Zimbra Cloud Hosting, Consulting & Support
    VMware vCloud Powered Serviceprovider

  8. #8
    rotorboy is offline Special Member
    Join Date
    Mar 2008
    Location
    Canada
    Posts
    148
    Rep Power
    7

    Default

    Tonight I'll do an fsck to be sure.

    I'm reading through Ajcody-Notes-No-Such-Blob - Zimbra :: Wiki and I'm very confused... some seems to apply, some references past scripts, and it seems to be a couple major versions old (We're using 7.1.4).

    I'm trying all sorts of zmblobchk commands and it seems only to report issues, not fix them.

    I'm very nervous about trying:
    --missing-blob-delete-item
    --export-dir
    --incorrect-revision-rename-file

    I really don't want to risk losing these missing mails. Documentation and examples seem pretty sparse on this subject.

  9. #9
    rotorboy is offline Special Member
    Join Date
    Mar 2008
    Location
    Canada
    Posts
    148
    Rep Power
    7

    Default

    Well at this point I've used zmlmtpinject from a perl script to re-send most of the missing messages. Now I can't figure out how to get rid of all the unlinked blob files... do I just delete them with a script or will that bust something?

    zmblobchk doesn't seem to actually touch the files...

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. HELIX 7.1.1 ported to FreeBSD
    By solko in forum Installation
    Replies: 22
    Last Post: 05-21-2013, 04:37 AM
  2. [SOLVED] Re-creating the spam training e-mail account
    By richard-hdd in forum Administrators
    Replies: 21
    Last Post: 03-20-2012, 07:34 AM
  3. Zimbra don't start
    By cadinor in forum Migration
    Replies: 1
    Last Post: 08-04-2011, 05:27 PM
  4. Error loading on Mac OS X 10.4.10 server PPC
    By qprcanada in forum Installation
    Replies: 7
    Last Post: 10-26-2007, 06:25 AM
  5. system failure: getDirectContext
    By avisser in forum Installation
    Replies: 3
    Last Post: 10-12-2005, 05:32 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •