[SOLVED] More innodb corruption

[Insanity5902]

I am still stuck of 5.0.2, I have tried every upgrade (excpet 5.0.8 which I will tackle again this weekend).

Sometimes the upgrade actually fails, sometimes it completes only to watch the corruption happen 30 minutes later and the database go down. (FYI - this is what I used Mysql Crash Recovery - Zimbra :: Wiki)

I have even tried exporting and importing every database and the same result happens in the end, innodb corruption.

I would really like to upgrade to get the new features as well as better support for zdesktop.

Does anybody else have any suggestions?

[phoenix]

Are you absolutely certain that you haven't got a RAM problem or a failing HD?

[Insanity5902]

Yeah pretty sure, 5.0.2 runs just fine every time. I always create a backup before I start, and when the upgrade fails, it copy back the 5.0.2 and re-run the 5.0.2 install and then it runs like a champ.

Currently at 30 days of uptime. DMESG is still clean from the last reboot (thought that might help the install process). Nothing but info messages in the logfiles.

EDIT :: I am checking the HDD's now, and will check the ram later using memtest.
Update1 :: HDD's are fine, each drive is NORMAL, and the raid set is NORMAL.

[phoenix]

I see that you've already tried a recovery when this happens, it works for a while and returns later? There should be no reason why this happens on every upgrade you've done. Did you file anything in bugzilla about this problem? I also assume that this is a plain vanilla install of Zimbra on Ubuntu and there's no funny hacks or modifications? I'm not an expert with innodb so I'd suggest that if it happens again after your next upgrade attempt, you file an entry in bugzilla and see if we can get to the bottom of it.

[Insanity5902]

No I haven't filed a bug, this is a production system, so I haven't been able to leave it long.

It is a plain vanilla install.

I actually have the enviroment from the last failed upgraded, we might be able to use that. I was also going to try and build a virtual enviroment using the same install as the one that is botched, and see if I can 1) reproduce 2) spend more time trying to fix it. I was looking at loading a dev system inside of Sun's Virtualbox or something.

[Insanity5902]

I know shame on me for not filing a bug, but I have had no time to look at it long enough to put together a thoughtful bug report.

I can report though, that I just ran through a test install in VirtualBox and then again on the live server with 0 issues with mysql.

This was an upgrade from 5.0.2 to 5.0.11.

I can say, for those that are wanting to test upgrades but can't afford a test box, VirtualBox work wonderfully. I was able to install Ubuntu 6.06, copy over a tarball of /opt/zimbra and test upgrades from there. I would of known about this a year ago, it would of saved me a lot of trouble with botched upgraded. Just remember to reset the permissions from your copied data to zimbra:zimbra, and then run /opt/zimbra/libexec/zmfixperms.

I am still not sure what was causing the problems, but the latest release fixed it for me.