[SOLVED] mysql.server is not running!

[russgalleywood]

Hi all,

I have an emergency!

Running on CentOS, Zimbra 5.0.18 GA.

At about 9.20am this morning our Zimbra interface went down.

I did the usual and tried to restart the services, they all came back up but seconds later it went down again.

I now see this when from zmcontrol status:

antispam Running
antivirus Running
ldap Running
logger Running
mailbox Stopped
zmmtaconfig is running.
mysql.server is not running
mailboxd is not running.
zmmailboxdctl is not running
mta Running
snmp Running
spell Running
stats Running

I've tried to restart MySQL as recommended elsewhere on the Forums and looked at other posts on the subject but need a little help as I really don't want to mess up MySQL!
I have had a look at the mysql_error.log and it isn't pretty! I will attach it if requested but just want to get some help quickly as Users are moaning at me!

Hope there is someone out there can give me some advice soon!

Thanks
Russ

[russgalleywood]

Hi again,

now I have got my post on I'll add a little more as I discover it.

Firstly,

according to ./mysql.server status, my MySQL is up and down every few seconds.

Below is the first section of my mysql_error.log:

Number of processes running now: 0
100406 09:25:05 mysqld restarted
100406 9:25:06 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
100406 9:25:06 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
InnoDB: Log scan progressed past the checkpoint lsn 6 514675699
100406 9:25:06 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 6 514823628
100406 9:25:12 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
100406 9:25:13 InnoDB: Started; log sequence number 6 514823628
100406 9:25:13 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
Version: '5.0.67-log' socket: '/opt/zimbra/db/mysql.sock' port: 7306 Source distribution
InnoDB: Dump of the tablespace extent descriptor: len 40; hex 000000000000000200000000039600000000016600000004aa aaaaaaaaaaaaaeaaaaaaaaaaaaaaaa; asc $
InnoDB: Serious error! InnoDB is trying to free page 669
InnoDB: though it is already marked as free in the tablespace!
InnoDB: The tablespace free space info is corrupt.
InnoDB: You may need to dump your InnoDB tables and recreate the whole
InnoDB: database!
InnoDB: Please refer to
InnoDB: MySQL :: MySQL 5.0 Reference Manual :: 13.2.6.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
100406 9:25:15InnoDB: Assertion failure in thread 117029776 in file fsp0fsp.c line 2981
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to MySQL Bugs.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: MySQL :: MySQL 5.0 Reference Manual :: 13.2.6.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
100406 9:25:15 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

Thanks
Russ

[russgalleywood]

Apologies, in my rush to get this fixed I missed phl4kx's post on a similar subject.

However, as mentioned in that post, I cannot get into MySQL using the mysql zimbra command, (think I'm using it in the right place).

Russ

[uxbod]

Mysql Crash Recovery - Zimbra :: Wiki

[russgalleywood]

Hi Uxbod,

I'm poised to do this and was already thinking this is the obvious solution really.

Any obvious pitfalls to watch out for doing this please? I should not lose data if I follow the instructions to the letter I guess?

I'm happy to go ahead as I want to get it back up and running quickly but am reluctant to fiddle with the SQL unless I really have to!

Thanks
Russ

[uxbod]

Make sure you have a good backup first of all

[russgalleywood]

I have a problem with this unfortunately, my SQL is constantly stopping and starting itself every few seconds!

I added the innodb_force_recovery = 1 option and stopped and started MySQL, ran for a few seconds and then reported not running again.
Should this not happen when in recovery mode, have I got the syntax wrong there?

Thanks

[russgalleywood]

I have my nightly backups on the Zimbra server of course, also these are moved to another server every day.
Would you suggest backing up the whole opt/zimbra directory first? This will take a while for our number of Users!

Thanks

[vavai]

Originally Posted by russgalleywood

I have my nightly backups on the Zimbra server of course, also these are moved to another server every day.
Would you suggest backing up the whole opt/zimbra directory first? This will take a while for our number of Users!

Thanks

I would like to suggest to backup up the whole /opt/zimbra to prevent any data loss.

I found this problem last year, on Zimbra 5.0.11 after power failure. At the end, I could restore data by follows the instruction on Zimbra wiki as mention by uxbod. Actually, I must deleted 1-2 crashed mailbox

[russgalleywood]

Thanks Guys