[SOLVED] mysql innodb corrupt, innodb_force_recovery 6 won't work

[wolvesled]

Hi,

this morning our ZCS server discovered not work, after few reviewing I discover the problem is mysql unable to start. with some test I and google I realize is a db corrupt but not so much info I can find on how to recover in this unlucky situation, what I've been already tried:

1. I stop the whole zcs, only su zimbra and test mysql.server start, it won't work
2. I read the log/mysql_error.log realize the problem is some inconsistency in log and innodb start to recovery but failed at error: "Page directory corruption: supremum not pointed to"
3. I read the zimbra mysql recovery guide, but even I set innodb_force_recovery = 6 I do not have any luck to start mysql.server start with user zimbra

my ZCS is version zcs-6.0.0_GA_1802.RHEL5.20090830140212.
in CentOS in a VM, was running perfect.

so what I have to do next?

[wolvesled]

100322 16:57:52  mysqld started
100322 16:57:52 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
100322 16:57:52 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
100322 16:57:53  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Page directory corruption: supremum not pointed to
100322 16:57:53  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex <0000 many of "0"s>; asc








                                              ;InnoDB: End of page dump
100322 16:57:53  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page directory corruption: supremum not pointed to
100322 16:57:53  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex <0000 many of "0"s>; asc








                                              ;InnoDB: End of page dump
100322 16:57:54  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
100322 16:57:54InnoDB: Error: trying to access a stray pointer 0xd5a23ff8
InnoDB: buf pool start is at 0x55a08000, end at 0xb6908000
InnoDB: Probable reason is database corruption or memory
InnoDB: corruption. If this happens in an InnoDB database recovery, see
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: how to force recovery.
100322 16:57:54InnoDB: Assertion failure in thread 3083704000 in file ./../include/buf0buf.ic line 268
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
100322 16:57:54 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=0
read_buffer_size=1048576
max_used_connections=0
max_connections=110
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225280 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfc667b8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8192065
0x83c91fb
0x8342596
0x834db2b
0x82f40fd
0x82f58e4
0x82e50fe
0x82e3f0e
0x8270c86
0x8265662
0x8191298
0x8196645
0xb7cf0e8c
0x80edf51
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
100322 16:57:54  mysqld ended

[uxbod]

Do you have a backup ?

[wolvesled]

not luckly, was planning to have a backup server, but happens before we get there.

[wolvesled]

still looking for chance of non-or-minium lost, let me know if impossible.

P.S. was was running fine and no power failure, can be some software issues?
P.S.2 tested in the case if remove the ibdata and iblogs, that way mysql start, so means the library has no problem.

[LMStone]

I would open a support ticket with Zimbra on this.

Hope that helps,
Mark

[wolvesled]

Thank you all very much in advance!

I was just wondering the ZCS is storing the emails in database or independently in filesystem organized by postfix? maybe any chance to start imap and pull out all emails at least.

I'm taking searches.

[king0770]

If you are running NE, this may be of interest to you.

King0770-Notes-When innodb force recovery Fails - Zimbra :: Wiki

[wolvesled]

Originally Posted by king0770

If you are running NE, this may be of interest to you.

King0770-Notes-When innodb force recovery Fails - Zimbra :: Wiki

Thanks for the info, but I look into my redolog folder there is only redo.log of 60MB, and the archive folder is empty, seems there isn't any incremental logs.

I tried to find the binlog of mysql, but not luck, anyone every seen it?

I'm looking this article about recovery of the innodb data: How to Recover Data using the InnoDB Recovery Tool Chris on MySQL
the last option is to restore the meta data and use the "store" folder together to reinstall zimbra.

hope not too stupid going this way...

[raj]

did you try innodb_force_recovery = 1 to 6 all six options in an increasing order of 1,2...6 or just tried 6 directly.
if you have not tried all 6 then i suggest you do that in increasing order and see if you can start mysqld service

Raj