Upgrade attempt from NE 6.0.10 to NE 7.0.0 hung

[goo0h]

On a test system, attempted to upgrade it from 6.0.10 to 7.0.0, and it seems to have hung. It was all progressing fine....

Restoring existing configuration file from /opt/zimbra/.saveconfig/localconfig.xml...done
Installing /opt/zimbra/conf/ZCSLicense.xml
Operations logged to /tmp/zmsetup.02092011-181519.log
Upgrading from 6.0.10_GA_2692 to 7.0.0_GA_3077
Stopping zimbra services...done.
Verifying /opt/zimbra/conf/my.cnf
Starting mysql...done.
This appears to be 6.0.10_GA
Checking ldap status...not running.
Running zmldapapplyldif...done.
Checking ldap status...already running.
Running mysql_upgrade...done.
Redolog version update required.
Wed Feb 9 18:16:14 2011: Verified redolog version 1.28.
Wed Feb 9 18:16:14 2011: Updating Redolog schema version from 1.28 to 1.30.
Redolog version update finished.
Schema upgrade required from version 64 to 65.
Running /opt/zimbra/libexec/scripts/migrate20100926-Dumpster.pl
Wed Feb 9 18:16:20 2011: Verified schema version 64.
Wed Feb 9 18:16:34 2011: Verified schema version 64.
Wed Feb 9 18:16:34 2011: Updating DB schema version from 64 to 65.
Stopping mysql...done.
Updating from 6.0.11_GA
Stopping ldap...done.

but then it hung at this point, and there has been no further apparent activity for hours. This is on SUSE 10, 64-bit. Upgrades have been fine in the past.

> zmcontrol status
Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
Host tzem1.x.org
antispam Stopped
zmamavisdctl is not running
antivirus Stopped
zmamavisdctl is not running
zmclamdctl is not running
convertd Stopped
convertd is not running.
imapproxy Stopped
zmnginxctl is not running
ldap Stopped
logger Stopped
zmlogswatchctl is not running
mailbox Stopped
mysql.server is not running.
zmmailboxdctl is not running.
memcached Stopped
memcached is not running.
mta Stopped
postfix is not running
zmsaslauthdctl is not running
snmp Stopped
zmswatch is not running.
spell Stopped
zmapachectl is not running
stats Stopped
zmconfigd Stopped
zmconfigd is not running.

Fortunately this is only a test system, but would be nice to know how to recover from this. I haven't hit ^C yet, but figure I'll have to eventually.

[goo0h]

I didn't see anything in the logs, at least not anything obvious. I checked to see what was running, and got:

zimbra@tzem1:~/log> ps -ft pts/1
UID PID PPID C STIME TTY TIME CMD
root 6796 18187 0 18:15 pts/1 00:00:00 /usr/bin/perl /opt/zimbra/libexec/zmsetup.pl
root 11026 6796 0 18:17 pts/1 00:00:00 sh -c su - zimbra -c "/opt/zimbra/openldap/sbin/slapindex -b '' -q -F /opt/zimbra/data/ldap/co
zimbra 11027 11026 0 18:17 pts/1 00:00:00 su - zimbra -c /opt/zimbra/openldap/sbin/slapindex -b '' -q -F /opt/zimbra/data/ldap/config zim
zimbra 11028 11027 0 18:17 pts/1 00:00:00 /opt/zimbra/openldap/sbin/slapindex -b -q -F /opt/zimbra/data/ldap/config zimbraMailHost
root 18187 20874 0 18:06 pts/1 00:00:00 /bin/bash ./install.sh -x -l ../ZCSLicense.xml
amos 20822 20815 0 Feb08 pts/1 00:00:00 -bash
root 20866 20822 0 Feb08 pts/1 00:00:00 su -
root 20874 20866 0 Feb08 pts/1 00:00:00 -bash

so I did an strace to see what that process was doing:

zimbra@tzem1:~> strace -p 11028
Process 11028 attached - interrupt to quit
futex(0x6a8e04, FUTEX_WAIT, 1, NULL <unfinished ...>

since it didn't appear that it would ever finish doing what it was waiting for, i killed it

zimbra@tzem1:~> kill 11028

the install then proceeded....

Checking ldap status...not running.
Running zmldapapplyldif...done.
Checking ldap status...already running.
Updating from 7.0.0_BETA1
Updating from 7.0.0_BETA2
Updating from 7.0.0_BETA3
Updating from 7.0.0_RC1
Updating from 7.0.0_GA
Updating global config and COS's with attributes introduced after 6.0.10_GA...done.
Stopping ldap...done.
Upgrade complete.

Running bdb db_recover...done.

.....

Upgrade appears to have finished, and I'm able to login to this test system. Will have to poke around more later to see that everything is working OK.

[y@w]

I did the upgrade last night and had the same thing happen to me (blogged here)

Mine also seemed fine after the upgrade.

After the upgrade was all finished, I did a restore to snapshot on the VM to try it again, but it didn't happen the second time around. Since I couldn't reproduce I didn't file a bug. Perhaps this needs a second look.

[y@w]

Should add - mine was Ubuntu 10.04 x64 from 6.0.9 -> 7.0.

[Labsy]

Hi,
similar problem here with 7.0.0GA_3077 on Ubuntu 8.04 64-bit. Mine box seems to run fine after manually running the ERROR-ed line from /opt/zimbra/log/zmconfigd.log (postfix config line with a lot of config settings), but despite of that, I get this repeated in /opt/zimbra/log/zmconfigd.log every 1 minute:

Starting zmconfigd...zmconfigd is already running.
2011-02-14 09:24:28,861 zmconfigd WARNING [8394-MainThread] Fetching All configs
2011-02-14 09:24:29,107 zmconfigd WARNING [8394-MainThread] All configs fetched in 0.24 seconds
2011-02-14 09:24:29,342 zmconfigd WARNING [8394-MainThread] All rewrite threads completed in 0.01 sec
2011-02-14 09:24:29,342 zmconfigd WARNING [8394-MainThread] All restarts completed in 0.00 sec
Starting zmconfigd...zmconfigd is already running.
2011-02-14 09:25:29,354 zmconfigd WARNING [8394-MainThread] Fetching All configs
2011-02-14 09:25:29,585 zmconfigd WARNING [8394-MainThread] All configs fetched in 0.23 seconds
2011-02-14 09:25:29,786 zmconfigd WARNING [8394-MainThread] All rewrite threads completed in 0.00 sec
2011-02-14 09:25:29,786 zmconfigd WARNING [8394-MainThread] All restarts completed in 0.00 sec
Starting zmconfigd...zmconfigd is already running.
2011-02-14 09:26:29,798 zmconfigd WARNING [8394-MainThread] Fetching All configs
2011-02-14 09:26:30,132 zmconfigd WARNING [8394-MainThread] All configs fetched in 0.34 seconds
2011-02-14 09:26:30,338 zmconfigd WARNING [8394-MainThread] All rewrite threads completed in 0.00 sec
2011-02-14 09:26:30,339 zmconfigd WARNING [8394-MainThread] All restarts completed in 0.00 sec

[phoenix]

Originally Posted by Labsy

Hi,
similar problem here with 7.0.0GA_3077 on Ubuntu 8.04 64-bit. Mine box seems to run fine after manually running the ERROR-ed line from /opt/zimbra/log/zmconfigd.log (postfix config line with a lot of config settings), but despite of that, I get this repeated in /opt/zimbra/log/zmconfigd.log every 1 minute:

Why have you posted this again when I've already answered your other post that these messages are normal and just warnings? What does it have to do with the content of this thread?

[liverpoolfcfan]

I had the same issue with an upgrade of my test box from 7.0 Beta to 7.0 Release on CentOS_64 FOSS.

It hung at the

Stopping ldap...done.

and when I eventually hit Ctrl-C the install proceeded and finished.

After it completed, I re-ran the upgrade install again in-place, and this time it went all the way through without stopping.

Definitely seems to be a pattern here for upgrades.

[Rich Graves]

The potentially long delay for upgrades to 7.0 and 6.0.11 seems to be caused by Bug 54901 - Add index for LDAP zimbraMailHost attribute.

Why that indexing takes forever, I'm not sure, but the index certainly isn't going to be very efficient at sites with only one zimbraMailHost.

[Rich Graves]

Bug 57491 - slapindex of zimbraMailHost might hang during install