Zimbra

rwc101010 · #1 (**permalink**) 10-15-2009, 09:45 PM

While testing the migration from ZCS 6.0.1 GA 32-bit to ZCS 6.0.1 GA 64-bit on a development machine, I encountered an issue after deploying a commercial SSL certificate. After what appears to be a successful deploy, restarting ZCS results in the following error:

Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.

After restart, no services are functional, and lots of error messages stack up in the logs.

To build this server, I performed a fresh install of Ubuntu 8 LTS Server, then fully patched the system and installed the required Zimbra libraries.

Code:

Checking for prerequisites...
     FOUND: NPTL
     FOUND: sudo-1.6.9p10-1ubuntu3.5
     FOUND: libidn11-1.1-1
     FOUND: libpcre3-7.4-1ubuntu2.1
     FOUND: libgmp3c2-2:4.2.2+dfsg-1ubuntu2
     FOUND: libexpat1-2.0.1-0ubuntu1
     FOUND: libstdc++6-4.2.4-1ubuntu4
     FOUND: libstdc++5-1:3.3.6-15ubuntu6
     FOUND: libperl5.8-5.8.8-12ubuntu0.4
Checking for suggested prerequisites...
    FOUND: perl-5.8.8
    FOUND: sysstat
Prerequisite check complete.

/etc/hosts and local DNS seem to be setup properly, to provide the internal IP to the system for DNS lookups:

Code:

root@mail:~# hostname
mail
root@mail:~# hostname -f
mail.xxxxxxx.xxx
root@mail:~# dig -t mx xxxxxxx.xxx

; <<>> DiG 9.4.2-P2 <<>> -t mx xxxxxxx.xxx
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62757
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 4

;; QUESTION SECTION:
;xxxxxxx.xxx.			IN	MX

;; ANSWER SECTION:
xxxxxxx.xxx.		86400	IN	MX	15 mx2.xxxxxxx.xxx.
xxxxxxx.xxx.		86400	IN	MX	10 mail.xxxxxxx.xxx.

;; AUTHORITY SECTION:
xxxxxxx.xxx.		86400	IN	NS	ns1.xxxxxxx.xxx.
xxxxxxx.xxx.		86400	IN	NS	ns2.xxxxxxx.xxx.
xxxxxxx.xxx.		86400	IN	NS	ns2.xxxxxxx.yyy.
xxxxxxx.xxx.		86400	IN	NS	ns1.xxxxxxx.yyy.

;; ADDITIONAL SECTION:
mail.xxxxxxx.xxx.	86400	IN	A	192.168.3.5
ns1.xxxxxxx.xxx.	78475	IN	A	~removed external IP~
ns2.xxxxxxx.xxx.	74	IN	A	~removed external IP~
ns1.xxxxxxx.yyy.	86400	IN	A	~removed external IP~
ns2.xxxxxxx.yyy.	86400	IN	A	~removed external IP~

;; Query time: 0 msec
;; SERVER: 192.168.3.5#53(192.168.3.5)
;; WHEN: Fri Oct 16 02:01:16 2009
;; MSG SIZE  rcvd: 217

Once the base platform was built, I performed a fresh install of 6.0.1_GA_1816.UBUNTU8_64 UBUNTU8_64 NETWORK edition, maintaining configuration parity with my 32-bit development system to be migrated - items such as the following were changed during the final install configuration to match the 32 bit system:

Administrative Account Password
SPAM User Account
HAM User Account
LDAP Root Password
LDAP Replication Password
LDAP Postfix Password
LDAP Amavis Password
LDAP Nginx Password
Default COS Settings

After install, all services function and all interfaces are accessible.

The next step on my migration checklist entails moving the Commercial cert from my 32-bit dev system, to my 64 bit dev system.

I copied the original working commercial_ca.crt to /tmp/ssl/

Code:

I copied the original working commercial.crt to /tmp/ssl/

Code:

	-----BEGIN CERTIFICATE-----
	  ~~Data Removed From Post for Security~~
	-----END CERTIFICATE-----

I copied the original working commercial.key to /opt/zimbra/ssl/zimbra/commercial/

Code:

	-----BEGIN RSA PRIVATE KEY-----
	  ~~Data Removed From Post for Security~~
	-----END RSA PRIVATE KEY-----

I placed the original working commercial.csr to /opt/zimbra/ssl/zimbra/commercial/

Code:

	-----BEGIN CERTIFICATE REQUEST-----
	  ~~Data Removed From Post for Security~~
	-----END CERTIFICATE REQUEST-----

I then tested and deployed the Commercial SSL Certificate via the command line:

Code:

	root@mail:~# cd /tmp/ssl

	root@mail:/tmp/ssl# /opt/zimbra/bin/zmcertmgr verifycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.key ./commercial.crt ./commercial_ca.crt
	** Verifying ./commercial.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
	Certificate (./commercial.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
	Valid Certificate: ./commercial.crt: OK

	root@mail:/tmp/ssl# /opt/zimbra/bin/zmcertmgr deploycrt comm ./commercial.crt ./commercial_ca.crt
	** Verifying ./commercial.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
	Certificate (./commercial.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
	Valid Certificate: ./commercial.crt: OK
	** Copying ./commercial.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
	** Appending ca chain ./commercial_ca.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
	** Saving server config key zimbraSSLCertificate...done.
	** Saving server config key zimbraSSLPrivateKey...done.
	** Installing mta certificate and key...done.
	** Installing slapd certificate and key...done.
	** Installing proxy certificate and key...done.
	** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
	** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...done.
	** Installing CA to /opt/zimbra/conf/ca...done.

The Commercial Certificate deployment seemed to complete without issue.

I then restarted ZCS (and this is where the fun begins):

Code:

	zimbra@mail:~$ zmcontrol stop
	Host mail.xxxxxxx.xxx
		Stopping stats...Done.
		Stopping mta...Done.
		Stopping spell...Done.
		Stopping snmp...Done.
		Stopping archiving...Done.
		Stopping antivirus...Done.
		Stopping antispam...Done.
		Stopping imapproxy...Done.
		Stopping memcached...Done.
		Stopping mailbox...Done.
		Stopping convertd...Done.
		Stopping logger...Done.
		Stopping ldap...Done.

	zimbra@mail:~$ zmcontrol start
	Host mail.xxxxxxx.xxx
		Starting ldap...Done.
	Unable to determine enabled services from ldap.
	Enabled services read from cache. Service list may be inaccurate.
		Starting logger...Done.
		Starting convertd...Done.
		Starting mailbox...Done.
		Starting memcached...Done.
		Starting antispam...Done.
		Starting antivirus...Done.
		Starting snmp...Done.
		Starting spell...Done.
		Starting mta...Done.
		Starting stats...Done.

After restart, all services become unavailable and /var/log/zimbra.log fills with errors indicating SSL key issues. Nothing in the log files stands out to me as smoking gun... mostly smells to me like an issue with the deployment of the SSL key pair.

Even though the services appear down, a zmcontrol status results in the following output:

Code:

zimbra@mail:~$ zmcontrol status
Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
Host mail.xxxxxxx.xxx
	antispam                Running
	antivirus               Running
	convertd                Running
	ldap                    Running
	logger                  Running
	mailbox                 Running
	memcached               Running
	mta                     Running
	snmp                    Running
	spell                   Running
	stats                   Running
zimbra@mail:~$

I've performed several clean installs of ZCS, and each ends with the same issue. I've attached the console log from my ZCS install and cert deployment, the zimbra.log, and the zimbra installer log. All logs from this server are available for further review, and I'm willing to try just about anything to resolve this issue.

Thanks!

Robert

y@w · #2 (**permalink**) 10-15-2009, 09:53 PM

Quote:

Oct 16 02:58:16 mail zimbramon[30915]: 30915:info: zmmtaconfig: Skipping getAllMtaAuthURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderE xception: unable to find valid certification path to requested target)

Have a look at the troubleshooting section of this page: Installing a Verisign Test Certificate - Zimbra :: Wiki

Looks like you're hitting the same error. That may help, or at least get you in the right direction..

rwc101010 · #3 (**permalink**) 10-15-2009, 10:00 PM

Quote:

Originally Posted by y@w

Have a look at the troubleshooting section of this page: Installing a Verisign Test Certificate - Zimbra :: Wiki

Looks like you're hitting the same error. That may help, or at least get you in the right direction..

No, that article didn't come up in any of my searching. So, the gist of the wiki page is that the java keystore lacks the proper CA chain to validate the cert?

Looking at the wiki page, it appears the following command performs the import:

Code:

# /opt/zimbra/java/bin/keytool -import -alias (ALIAS) -keystore /opt/zimbra/java/jre/lib/security/cacerts -storepass (PASSWORD) -file /opt/zimbra/conf/ca/commercial_ca.pem

My only question would be what value is substituted for (ALIAS) in the above command, and where I find the cacerts (PASSWORD).

Thanks for the tip!

rwc101010 · #4 (**permalink**) 10-15-2009, 10:53 PM

Thanks to y@w for suggesting the root java keystore.

Running the following command as root, and restarting ZCS resolved the issue:

Code:

/opt/zimbra/java/bin/keytool -import -alias root -keystore /opt/zimbra/java/jre/lib/security/cacerts -storepass changeit -file /opt/zimbra/conf/ca/commercial_ca.pem

My questions is, why would a valid cert chain validate as good when using the zmcertmgr verifycrt comm command, but fail to validate using the java root keychain. Seems like a pretty big disconnect to have two inconsistent CA root keychains in the same application stack....

Thanks again for the pointer!

Robert

Zimbra

Downloads

Product Information

Toolbox

Why Join?