Ubuntu upgrade woes

[zwvpadmin]

This is a long one, so settle in.

It started with a seemingly stable ubuntu 6.06 lts server install with 5.0.8 OSS zimbra running smooth (for the most part). (note: two symmetrical 400gb drives in an mdadm RAID 1 config)

we wanted to up from 6.06 to 8.04 to get better resource management and stay current. The plan of attack was 6.06 -> 8.04 first, then reboot and run 5.0.8 -> 5.0.11.

I ran ubuntu update from update-manager and it seemed to go smooth. Other than a warning about removing some unused unneeded packages there were no outstanding concerns.

Upgrade completed and it prompted for a reboot. On reboot it froze. After a hard restart it dropped to a busybox prompt in the initramfs. I wasn't able to do anything. Couldnt mount, coulnt read, half the commands weren't working, nothing.

So after much panicing, and trying to boot to the second RAID drive along with a slew of other attempts - no dice. No matter what we tried (recruited a friend who's a linux guru) always back to the busybox prompt.

New plan: format one of the RAID drives with a new install of 8.04 and pull the data over. No dice, couldn't mount the second drive. (kept getting error "unknown file system") basically mdadm was barfing and din't know what to do. We even tried manually mounting the partition... nothing.

We did manage to use grub to boot to the other drive once, but it appeared to be corrupted and fsck threw a bazillion errors and quit after a while. Even e2fsck with a backup superblock did no good.

New plan #2: reformat one of the drives AGAIN with 6.06 and rebuild from backup to get back to square one.

Install went smooth, reinstalled 5.0.8 -s, rsynced to latest /opt/zimbra backup, zmfixperms, and then upgrade with 5.0.8 install.sh.

FAILED. every time on the mysql logger. So I hopped on AIM and hunted down mmorse from the zimbra team (that guy deserves a medal). who late on a sunday was very gracious and helpful and walked me through deleting and rebuilding the logger. but still getting the same fail on install.

more to come with details of what we tried:

-zw

[zwvpadmin]

So here are some of the things we tried. I'd also like to note that the only information in the zmsetup.log was:

upgrading from (blank) to 5.0.8_GA_2462
Stopping zimbra services
Verifying /opt/zimbra/conf/my.cnf
Starting mysql
Starting logger mysql
I can't upgrade version

UPGRADE FAILED - exiting.

which is basically what we saw on std/io as well, except it gave this as the fault:

Starting logger mysql
(DATE): SELECT value FROM config WHERE name='db.version'

I can't upgrade version

UPGRADE FAILED - exiting.

at least twice, we did the following:

zmcontrol status stated that zmlogswatchctl and swatch were not running

(as zimbra)
logmysql zimbra_logger
show tables;
check table *; (for each table)

In my case they all had one warning in them, but showed status as OK.

we decided to just rebuild the log database

(as zimbra)
logmysql.server stop
(as root)
rm -rf /opt/zimbra/logger/db/data/*
chown zimbra:zimbra /opt/zimbra/logger/db/data
chmod 775 /opt/zimbra/logger/db/data
cd /opt/zimbra/libexec
./zmloggerinit
(as zimbra)
logmysql.server start

first time this didn't work well. Error: /opt/zimbra/bin/logmysql.server: line 56: /tmp/zimbra/mysql.server.lock: permission denied

the second it did. (not completely sure why, we did a lot of poking around between attempts)

Reran upgrade 5.0.8 - same error

Next we tried to remove the logger.

zmprov ms `zmhostname` -zimbraServiceEnabled logger
zmprov ms `zmhostname` -zimbraServiceInstalled logger

Reran the upgrade 5.0.8 - same error.

At this point mmorse ssh'd in and redid everything you see plus a few pokes - same error. But the system is up and running atm, just without logging functions.

We plan to continue work again at close of business today/tomorrow.

any ideas?

-zw

[uxbod]

Check permissions on /var/log/zimbra.log that it is owned by zimbra:zimbra. I manually cleared my logfile at the weekend and it ended up being owned by root and the same symptoms were encountered

If that is the case please vote for my RFE :- Bug 33582 – zmfixperms uses incorrect directory for ClamAV DB + some enhancements

[mmorse]

I think the next step (now that it's at least running) will be to test the install.sh/utilfunction.sh/zmsetup.pl after dpkg -r zimbra-logger (then install it again with .11 upgrade).

[zwvpadmin]

Originally Posted by uxbod

Check permissions on /var/log/zimbra.log that it is owned by zimbra:zimbra. I manually cleared my logfile at the weekend and it ended up being owned by root and the same symptoms were encountered

Interesting idea, but since currently there is no logger, there is no /var/log/zimbra.log to even check. I'm assuming you were referring to the permission denied error, but we got passed that.

-zw

[mmorse]

Took another look at this today - just a little /opt/zimbra/.install_history fun but we now have 5.0.8 cleanly though the installer.

[mmorse]

5.0.11 - now for logger

[mmorse]

+Logger

(He's gonna do the Ubuntu 8.04 changeover & RAID fix some other time.)

[zwvpadmin]

As stated, the issue seems to have been completely resolved by correcting the .install_history file.

The upgrade from 5.0.8 without logger to 5.0.11 with out logger went smooth, then 5.0.11 with logger and life was good.

Been up and running 20 hours without incident. I'll report back on the Ubuntu 6.06 lts to 8.04 lts upgrade.

-zw

[zwvpadmin]

Was able to successfully upgrade from Ubuntu 6.06 lts server to Ubuntu 8.04 lts server and migrate my temporary 5.0.11 install.

Ran into only one problem related to zimbra, and that was a repeat of the .isntall_history file as mentioned above. But since I knew right what it was able to contact good 'ol MMorse again and get a corrected copy.

I'm not sure if the .install_history is an actual bug or not, as it seems to be related to the backup process. Just seems odd.

Either way if anyone is experiencing similar issues, I suggest manually backing up /opt/zimbra/.install_history apart from whatever other backup/archive methods you use when performing a migration.

Had I done so, there would have been zero issues this time around.

-ZW