Help, failed 7.1.1->7.1.3 upgrade

[lindsey]

Last night I tried doing a 7.1.1 to 7.1.3 upgrade (7.1.1_GA_3196.RHEL5_64.20110527011124 to 7.1.3_GA_3346.RHEL5_64.20110928134520).

The first time it told me that my operating system was unknown (I'm running CentOS 5.6), so I re-ran it with --platform-override. This was after it shut down MySQL and tested the database.

Unfortunately, this process broke something. Zimbra wouldn't start at all. I restarted the upgrade with --platform-override and it completed the install, but Zimbra continued to fail:

Host localhost
Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
Starting ldap...Done.
Starting zmconfigd...Done
Starting logger...Failed.
Starting logswatch...ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.naming.CommunicationException localhost:389)
zimbra logger service is not enabled! failed.
Starting mailbox...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.

Then I tried reverting back to my old version, but it said

Downgrading to version 7.1.1 from 7.1.3 is not supported.

I see this in the logs:

IOError: [Errno 2] File not found - /opt/zimbra/zimbramon/pylibs/conf$py.class (Permission denied)

-rw-r----- 1 root root 9149 Dec 6 02:50 /opt/zimbra/zimbramon/pylibs/conf$py.class

That's the same permissions as it had before.

I do have a backup of the mailboxes and the main/config ldap databases.

Here are my questions:

1. Does anyone have any ideas to move past this and get our Zimbra install back up and running?

2. Failing that, should I be able to revert by reinstalling 7.1.1 from scratch, importing the ldap database, and then importing the mailboxes?

Thanks,

Chrsi

[phoenix]

Originally Posted by lindsey

Unfortunately, this process broke something. Zimbra wouldn't start at all. I restarted the upgrade with --platform-override and it completed the install, but Zimbra continued to fail:

Host localhost
Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
Starting ldap...Done.
Starting zmconfigd...Done
Starting logger...Failed.
Starting logswatch...ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.naming.CommunicationException localhost:389)
zimbra logger service is not enabled! failed.
Starting mailbox...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.

The highlighted text in blue looks incorrect to me, you should have the FQDN of your server in that message not localhost.

The highlighted text in red would indicate to me that you've a) got a problem with expired certificates or b) your DNS A or MX record and/or your /etc/host file is incorrect.

You can check the article in the wiki on regenerating the certificates (assuming it's a self-signed certificate) and you can check the hosts file and NDS records by running all the commands in the 'Verify....' section of the Split DNS article.

[lindsey]

That's what I was thinking too.

/etc/hosts is right, there are no self-referential MX records (never have been, I override with mailertable on a separate relay host that kicks the mail to the zimbra server), the A record is right, and the certificate is valid.

Do I have enough with my backups to reinstall the old version and restore?

Thanks,

Chris

[lindsey]

When trying to generate a new certificate (for giggles) I noticed that it also spits out some errors.

/opt/zimbra/bin/zmcertmgr createca -new
** Creating /opt/zimbra/ssl/zimbra/ca/zmssl.cnf...chown: invalid option -- 1
Try `chown --help' for more information.
done
chown: invalid option -- 1
Try `chown --help' for more information.
chown: invalid option -- 1
Try `chown --help' for more information.
chown: invalid option -- 1
Try `chown --help' for more information.
** Creating CA private key /opt/zimbra/ssl/zimbra/ca/ca.key...done.
** Creating CA cert /opt/zimbra/ssl/zimbra/ca/ca.pem...done.

Running zmcertmgr with sh -x prepended gives me this gem:

chown -1:-1 /opt/zimbra/ssl/zimbra/ca/zmssl.cnf

So for some reason it's no longer getting the proper uid/gid for Zimbra either. Trying to track down where that comes from.

[lindsey]

Running zmlocalconfig shows that a bunch of the data is lost.

./zmlocalconfig | egrep 'hostname|gid|uid'
zimbra_gid = -1
zimbra_ldap_userdn = uid=zimbra,cn=admins,cn=zimbra
zimbra_server_hostname = localhost
zimbra_uid = -1

Looking at the actual XML files, for some reason all of them are essentially empty.

Chris

[lindsey]

Fortunately, I had a backup for the conf directory. I copied back any files that where empty (conf/ldap*, localconfig.xml, slapd*), made the zimbramon/pylibs files readable by anyone (maybe a bad move, but)...

With the pylibs files readable, zmconfigd was able to start and changed permissions on a *ton* of files.

The server is now up and running again.

Chris