System crash leads to major MySQL corruption

[Glider Guider]

I had a system crash this morning (due to overheating due to a failed fan). Unfortunately several key files in /opt/zimbra were corrupted/truncated by the ensuing mess. The biggest problem is MySQL won't start.

Here is the version: Release 7.1.1_GA_3196.RHEL5_64_20110527011124 CentOS5_64 FOSS edition.


I've tried the various steps for using innodb_force_recovery without success...it just won't start due to the corruptions.

No, I don't have a good mysqldump, unfortunately.

Are there any hopes of recovering? I've seen that the NE has zmrestore, etc...anybody create something like this for the FOSS?

Or am I just 100% lost. The sad part is the drives I intend to use for backups are in the mail as we speak...one week too late.

John

[jduino]

Here is the output in mysql_error.log. I removed the hex dumps for clarity. If someone needs the full info just PM me and I'll email it along.:

110801 04:05:58 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
110801 4:05:58 InnoDB: Initializing buffer pool, size = 603.0M
110801 4:05:58 InnoDB: Completed initialization of buffer pool
110801 4:05:58 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110801 4:05:58 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: space id in fsp header 1224737009, but in the page header 25639
InnoDB: Error: tablespace id 18446744073709551615 in file ./mboxgroup9/open_conversation.ibd is not sensible
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Warning: a page in the doublewrite buffer is not within space
InnoDB: bounds; space id 0 page number 1936, page 2 in doublewrite buf.
InnoDB: Warning: a page in the doublewrite buffer is not within space
InnoDB: bounds; space id 0 page number 2058, page 15 in doublewrite buf.
InnoDB: Warning: a page in the doublewrite buffer is not within space
InnoDB: bounds; space id 204 page number 2863, page 16 in doublewrite buf.
InnoDB: Warning: a page in the doublewrite buffer is not within space
InnoDB: bounds; space id 266 page number 368, page 31 in doublewrite buf.
InnoDB: Warning: a page in the doublewrite buffer is not within space
InnoDB: bounds; space id 266 page number 370, page 32 in doublewrite buf.
InnoDB: Warning: a page in the doublewrite buffer is not within space
InnoDB: bounds; space id 223 page number 358, page 33 in doublewrite buf.
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 1.
InnoDB: You may have to recover from a backup.
110801 4:05:58 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 12231a <EDITED FOR LENGTH>
InnoDB: End of page dump
110801 4:05:58 InnoDB: Page checksum 128194496, prior-to-4.0.14-form checksum 893462745
InnoDB: stored checksum 304290357, prior-to-4.0.14-form stored checksum 893462745
InnoDB: Page lsn 0 2766573610, low 4 bytes of lsn at page end 2766573610
InnoDB: Page number (if stored to page already) 1,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 1.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also MySQL :: MySQL 5.1 Reference Manual :: 13.6.6.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/appointment'
InnoDB: in InnoDB data dictionary has tablespace id 270,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/appointment_dumpster'
InnoDB: in InnoDB data dictionary has tablespace id 171,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/data_source_item'
InnoDB: in InnoDB data dictionary has tablespace id 221,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/imap_folder'
InnoDB: in InnoDB data dictionary has tablespace id 275,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/imap_message'
InnoDB: in InnoDB data dictionary has tablespace id 281,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/mail_item'
InnoDB: in InnoDB data dictionary has tablespace id 234,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/mail_item_dumpster'
InnoDB: in InnoDB data dictionary has tablespace id 169,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/open_conversation'
InnoDB: in InnoDB data dictionary has tablespace id 283,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/pop3_message'
InnoDB: in InnoDB data dictionary has tablespace id 190,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/revision'
InnoDB: in InnoDB data dictionary has tablespace id 198,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/revision_dumpster'
InnoDB: in InnoDB data dictionary has tablespace id 170,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup5/tombstone'
InnoDB: in InnoDB data dictionary has tablespace id 212,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
110801 4:05:58 InnoDB: Error: table 'mboxgroup9/open_conversation'
InnoDB: in InnoDB data dictionary has tablespace id 241,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
InnoDB: for how to resolve the issue.
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 47.
InnoDB: You may have to recover from a backup.
110801 4:05:58 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 7fdd8caf0000002 <EDITED FOR LENGTH>
InnoDB: End of page dump
110801 4:05:58 InnoDB: Page checksum 1456109135, prior-to-4.0.14-form checksum 1801731356
InnoDB: stored checksum 2145225903, prior-to-4.0.14-form stored checksum 160692190
InnoDB: Page lsn 0 2387046152, low 4 bytes of lsn at page end 2387046152
InnoDB: Page number (if stored to page already) 47,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an update undo log page
InnoDB: Page may be an index page where index id is 0 12
InnoDB: (index "FOR_IND" of table "SYS_FOREIGN")
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 47.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also MySQL :: MySQL 5.1 Reference Manual :: 13.6.6.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
InnoDB: Error: trying to access page number 3228459753 in space 0,
InnoDB: space name ./ibdata1,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.
InnoDB: If you get this error at mysqld startup, please check that
InnoDB: your my.cnf matches the ibdata files that you have in the
InnoDB: MySQL server.
110801 4:05:58 InnoDB: Assertion failure in thread 47476784580688 in file fil/fil0fil.c line 4352
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to MySQL Bugs.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.6.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
110801 4:05:58 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=1048576
max_used_connections=0
max_threads=110
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 234605 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x40000
/opt/zimbra/mysql/libexec/mysqld(my_print_stacktrace+0x2e)[0x77384e]
/opt/zimbra/mysql/libexec/mysqld(handle_segfault+0x351)[0x594f71]
/lib64/libpthread.so.0[0x3f2ac0eb10]
/lib64/libc.so.6(gsignal+0x35)[0x3f2a030265]
/lib64/libc.so.6(abort+0x110)[0x3f2a031d10]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd3343e]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd187f1]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd18a6c]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd106a3]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabcfdb8f]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd04a19]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd2b1a5]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd28aad]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd1c5cf]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabdc1f1a]
/opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd4a810]
/opt/zimbra/mysql/libexec/mysqld(_Z24ha_initialize_handlertonP13st_plugin_in t+0x31)[0x68b0e1]
/opt/zimbra/mysql/libexec/mysqld[0x70707d]
/opt/zimbra/mysql/libexec/mysqld(_Z11plugin_initPiPPci+0x81b)[0x70a6db]
/opt/zimbra/mysql/libexec/mysqld[0x595b6f]
/opt/zimbra/mysql/libexec/mysqld(main+0x6b5)[0x59a185]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3f2a01d994]
/opt/zimbra/mysql/libexec/mysqld[0x4cfb49]
The manual page at MySQL :: MySQL 5.5 Reference Manual :: C.5.4.2 What to Do If MySQL Keeps Crashing contains
information that should help you find out what is causing the crash.
110801 04:05:58 mysqld_safe mysqld from pid file /opt/zimbra/db/mysql.pid ended