[SOLVED] Trouble upgrading from 4.5.6 to 5.0.5: Unbound server sockets not implemente

[abuchbinder]

We're migrating a 4.5.6 (4.5.6_GA_1044.UBUNTU6) installation, and I thought that while testing things out, I would try upgrading to 5.0.5 to see how it works. (We'd appreciate the better commercial-cert handling and mobile access.) The procedure used was as follows:

I installed 4.5.6 on the new server, rsync'd the /opt/zimbra directory from the original server, set zimbra_uid and zimbra_gid as follows:

mail1-new$ /opt/zimbra/bin/zmlocalconfig -e zimbra_uid=`grep ^zimbra /etc/passwd|cut -d: -f3`
mail1-new$ /opt/zimbra/bin/zmlocalconfig -e zimbra_gid=`grep ^zimbra /etc/group|cut -d: -f3`

(This fixed errors like "java.io.FileNotFoundException: /opt/zimbra/log/audit.log (Permission denied)" showing up in catalina.out, as the UID and GID of the zimbra user didn't match on the new system.) I deleted openldap-data/* and used slapcat on the old server and slapadd on the new server (as described here) to import the LDAP database from the old server properly. I checked to make sure I could get to my mail and that everything seemed to be running properly.

I downloaded 5.0.5_GA_2201_UBUNTU6 and ran the upgrade. Two things changed: first, I can't get zmprov to work (though zmprov -l works); this may be related to the second problem, which is more serious: zmmailboxdctl isn't running:

$ zmcontrol status
Host mail1.example.com
        antispam                Running
        antivirus               Running
        ldap                    Running
        logger                  Running
        mailbox                 Stopped
                zmmailboxdctl is not running
        mta                     Running
        snmp                    Running
        spell                   Running
        stats                   Running

I'm seeing this in log/zmmailboxd.out:

CompilerOracle: exclude com/zimbra/cs/session/SessionMap.putAndPrune
0    INFO  [main] log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Fatal error: exception while binding to ports
java.net.SocketException: Unbound server sockets not implemented
        at javax.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:80)
        at com.zimbra.common.util.NetUtil.newBoundServerSocket(NetUtil.java:97)
        at com.zimbra.common.util.NetUtil.bindServerSocket(NetUtil.java:157)
        at com.zimbra.common.util.NetUtil.bindSslTcpServerSocket(NetUtil.java:56)
        at com.zimbra.common.handlers.PrivilegedHandler.openPorts(PrivilegedHandler.java:93)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.mortbay.xml.XmlConfiguration.call(XmlConfiguration.java:534)
        at org.mortbay.xml.XmlConfiguration.configure(XmlConfiguration.java:244)
        at org.mortbay.xml.XmlConfiguration.configure(XmlConfiguration.java:206)
        at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:968)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.mortbay.start.Main.invokeMain(Main.java:183)
        at org.mortbay.start.Main.start(Main.java:497)
        at org.mortbay.start.Main.main(Main.java:115)

I did an "ls -i tcp:995" as root but didn't see anything at all occupying the port.

I saw this same error in bug 20970, but the upgrade log seems to imply that it didn't have that particular problem:

$ grep "Updating from" log/zmsetup.01001900-0000139038500.log 
Mon May 12 10:37:36 2008 Updating from 4.5.6_GA
Mon May 12 10:37:38 2008 Updating from 4.5.7_GA
Mon May 12 10:38:24 2008 Updating from 4.5.8_GA
Mon May 12 10:38:24 2008 Updating from 4.5.9_GA
Mon May 12 10:38:25 2008 Updating from 4.6.1_RC1
Mon May 12 10:38:25 2008 Updating from 4.5.10_GA
Mon May 12 10:38:25 2008 Updating from 4.5.11_GA
Mon May 12 10:38:25 2008 Updating from 5.0.0_BETA1
Mon May 12 10:39:56 2008 Updating from 5.0.0_BETA2
Mon May 12 10:40:04 2008 Updating from 5.0.0_BETA3
Mon May 12 10:41:23 2008 Updating from 5.0.0_BETA4
Mon May 12 10:41:44 2008 Updating from 5.0.0_RC1
Mon May 12 10:41:44 2008 Updating from 5.0.0_RC2
Mon May 12 10:41:47 2008 Updating from 5.0.0_RC3
Mon May 12 10:41:53 2008 Updating from 5.0.0_GA
Mon May 12 10:47:44 2008 Updating from 5.0.1_GA
Mon May 12 10:48:02 2008 Updating from 5.0.2_GA
Mon May 12 10:48:24 2008 Updating from 5.0.3_GA
Mon May 12 10:49:03 2008 Updating from 5.0.4_GA
Mon May 12 10:49:03 2008 Updating from 5.0.5_GA

Any ideas? I'd like to migrate us to version 5, but this is kind of a showstopper for us.

[abuchbinder]

I'm not sure why this didn't provide a more straightforward error message, but importing an actual commercial SSL cert fixed the problem. (The upgrade process seemed to import the self-signed cert we'd been using--we'd had trouble importing a commercial key into 4.5.6--but it apparently failed at some point.)

The crucial diagnostic (suggested here) was:

$ keytool -list -keystore /opt/zimbra/jetty/etc/keystore -storepass `zmlocalconfig -m nokey -s mailboxd_keystore_password`
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect

I had to delete (rather, move out of the way) the existing /opt/zimbra/mailboxd/etc/keystore for zmcertmgr to work; I used zmcertmgr to install our commercial cert, and the mailbox service came up properly. (The above command executes without error now.)

I also changed the proxy ports listed in "zmprov -l gs `zmhostname`|grep Port" to not overlap with the regular ports, but since the proxy service is disabled, I don't think that fixed it, and I don't have time to go back and test both options. (I used "zmprov -l ms `zmhostname` zimbraImapProxyBindPort 7143 zimbraImapSSLProxyBindPort 7993 zimbraPop3ProxyBindPort 7110 zimbraPop3SSLProxyBindPort 7995".)

The system is now running successfully on version 5.0.5; I'm putting my solution here so that it might help someone else.