[SOLVED] Zimbra SQL is Dead - Cannot Start ./mysql.server

**cyberdeath** · 07-16-2008, 02:32 PM

I am having a major problem with Zimbra...there isn't a mysql.sock file in the ~/db directory (as user zimbra). Therefore, I cannot access mysql at all.

This all started from someone pulling the power plug to the server from the wall. Once the server rebooted, it had a corrupted reiserfs. I was able to recover that by running the repair tree option. But, now I am stuck with a non-functioning Zimbra server.

I have to leave right now, but I'll provide more details later and if asked.

**bdial** · 07-16-2008, 02:34 PM

mysql.sock is a socket created when the daemon starts. when mysql isn't running, there is no mysql.sock file. so your problem isn't that theres no mysql.sock, it's probably that mysql isn't running. for that you should maybe check your log files and find out why that is.

**y@w** · 07-16-2008, 03:10 PM

How exactly are you trying to start mysql?

This may or may not help you, but worth a quick look:
[SOLVED] Mailbox won't start after power outage

**cyberdeath** · 07-16-2008, 06:28 PM

I am trying the /opt/zimbra/bin/mysql.server start command.

Code:

mail:/opt/zimbra/bin # ./mysql.server start
mail:/opt/zimbra/bin # ./mysql.server status
mysql is not running

Of course, when I try the ./mysql command to access the cli, it gives me the socket error:

Code:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/zimbra/db/mysql.sock' (2)

Is there something with Zimbra that I need to use to fix the sql db error? Or should I be able to use standard MySQL tools? If so, what would you recommend?

Here is the mail.err file that was mentioned by y@w in the link.

Code:

080716 21:21:19  mysqld started
InnoDB: Log scan progressed past the checkpoint lsn 0 1783981612
080716 21:21:20  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 0 1784059982
InnoDB: 2 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 3 row operations to undo
InnoDB: Trx id counter is 0 2706432
080716 21:21:39  InnoDB: Error: table 'mboxgroup67/appointment'
InnoDB: in InnoDB data dictionary has tablespace id 921,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
InnoDB: for how to resolve the issue.
080716 21:21:39  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 080716 21:21:40 - mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=0
read_buffer_size=1044480
max_used_connections=0
max_connections=110
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 224839 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x526a1458, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x818bf60
0x83b6e59
0x83b7a47
0x837a1e6
0x837c2c3
0x8368849
0x839271e
0x82d5968
0xb7e252ab
0xb7d5058e
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
080716 21:21:40  mysqld ended

I apologize for the earlier short post as I had to step out unexpectedly (but I still wanted to get something into the forums).

If you need any more information, please let me know.

Thanks!

**jholder** · 07-16-2008, 06:30 PM

We use innodb, which makes repair very difficult. This is the best route for recovery:
Mysql Crash Recovery - Zimbra :: Wiki

**cyberdeath** · 07-16-2008, 09:06 PM

Well, I got to step 4 of the guide you mentioned, jholder....and then it went downhill:

Code:

mysqldump: Got error: 1033: Incorrect information in file: './zimbra/config.frm' when using LOCK TABLES

Suggestions?

**cyberdeath** · 07-16-2008, 10:36 PM

Okay so I fiddled around with it a bit and was able to get this error message when trying #4 on the page jholder provided:

Code:

080717  1:32:38  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 0 1784059982
InnoDB: 2 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 3 row operations to undo
InnoDB: Trx id counter is 0 2706432
080717  1:32:38  InnoDB: Error: table 'mboxgroup67/appointment'
InnoDB: in InnoDB data dictionary has tablespace id 921,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
InnoDB: for how to resolve the issue.
080717  1:32:38  InnoDB: Error: table 'mboxgroup67/imap_folder'
InnoDB: in InnoDB data dictionary has tablespace id 927,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
InnoDB: for how to resolve the issue.
080717  1:32:38  InnoDB: Error: table 'mboxgroup67/imap_message'
InnoDB: in InnoDB data dictionary has tablespace id 929,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
InnoDB: for how to resolve the issue.
080717  1:32:38  InnoDB: Error: table 'mboxgroup67/pop3_message'
InnoDB: in InnoDB data dictionary has tablespace id 924,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
InnoDB: for how to resolve the issue.
080717  1:32:38  InnoDB: Error: table 'mboxgroup67/tombstone'
InnoDB: in InnoDB data dictionary has tablespace id 922,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
InnoDB: for how to resolve the issue.
080717  1:32:38  InnoDB: Starting an apply batch of log records to the database.

After that, it just dumps the data.

Code:

InnoDB: End of page dump
080717 1:32:39 InnoDB: Page checksum 809267069, prior-to-4.0.14-form checksum 1330400654
InnoDB: stored checksum 1895350281, prior-to-4.0.14-form stored checksum 1488708 828
InnoDB: Page lsn 0 1783877752, low 4 bytes of lsn at page end 1783925701
InnoDB: Page number (if stored to page already) 61,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 504
InnoDB: Page may be an index page where index id is 0 1365
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 61.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery .html
InnoDB: about forcing recovery.
080717 1:32:39 - mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

Any suggestions for this one?

Thanks for the help thus far =).

**jholder** · 07-16-2008, 10:49 PM

I'm gonna be honest with you. This looks bad, and without a backup, you're probably hosed.

Did you start mysql in recovery mode? Also, can you cat that frm file? Do you have a backup of that file?

**cyberdeath** · 07-16-2008, 10:54 PM

I had to set innodb_force_recovery to 6 to get it to boot again. Once in, I noticed that it errors at mboxgroup67 and specifically with the open_conversation table.

Code:

mysql> check table open_conversation;
ERROR 2013 (HY000): Lost connection to MySQL server during query

And when I analyze table:

Code:

mysql> analyze table open_conversation;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    2
Current database: mboxgroup67

+-------------------------------+---------+----------+----------+
| Table                         | Op      | Msg_type | Msg_text |
+-------------------------------+---------+----------+----------+
| mboxgroup67.open_conversation | analyze | status   | OK       |
+-------------------------------+---------+----------+----------+
1 row in set (0.01 sec)

As far as backups are concerned...we don't have any backups of the server...yes, horrible practice...but it wasn't my decision.

Further suggestions would be greatly appreciated...I *really* need to be able to recover this server...since it holds a lot of important information.

Thanks again

*EDIT*

I tried to dump the information yet again....

Code:

for db in `< /tmp/mysql.db.list`;
do
    /opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u root --password=pwhere > /tmp/mysql.sql/$db.sql
    echo -e "Dumped $db\n"
done

...and it worked!

So, here's where I run into another problem.....Step 7:

Code:

mail:/opt/zimbra/bin # ./mysql -e "drop database zimbra"
ERROR 1217 (23000) at line 1: Cannot delete or update a parent row: a foreign key constraint fails

**cyberdeath** · 07-16-2008, 11:40 PM

Turns out, I forgot a step...which I quickly found out by using this command while in ./mysql (and looking under foreign key constraints):

Code:

SHOW INNODB STATUS

Once I did the command to drop the mboxes...that solved the problem. Server is running nicely now.

Thanks to everyone who lent a hand and pointed me in the right direction.

Hopefully this information will be useful to someone else in the future =).

Zimbra

Thread: [SOLVED] Zimbra SQL is Dead - Cannot Start ./mysql.server

LinkBack

Thread Tools

Search Thread

Display

[SOLVED] Zimbra SQL is Dead - Cannot Start ./mysql.server

Thread Information

Users Browsing this Thread

Similar Threads

slapd message error

Tomcat won't start after enabling IMAPproxy + a few other issues

[SOLVED] zmfixperms error when migrating [using John Holder's powertip]

Unable to start tomcat

3.1 on FC4 problems

Posting Permissions