Certs expired servers showing ldap error

[MrGreen]

Hi,

I have been running Zimbra Release 7.1.4_GA_2555.UBUNTU10_64 UBUNTU10_64 FOSS edition for just over a year now without any issues. Since yesterday the users complained that they cannot access the serve. When we started to investigate this morning, we found that the certificates have expired. With the help on the forum we have tried to create new certificates. Unfortunately it seems as if the server does not want to start up again.

We get the following message:
root@mail:/opt/zimbra# service zimbra start
Host localhost
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.

I have spent hours on the forum today to search for ways to fix this. I can confirm that the DNS is setup correctly as this has been running for over a year.

Could someone please help us to resolve this issue?

Thanks

[phoenix]

I have spent hours on the forum today to search for ways to fix this.

Did you follow the Zimbra Certified wiki page on how to recreate certificates?

I can confirm that the DNS is setup correctly as this has been running for over a year.

Unfortunately one doesn't necessarily follow the other in that statement.

Could someone please help us to resolve this issue?

Not really, from that information we can't tell a thing other than your certificates have (probably) expired. You need to give the full steps you've taken (including all the commands you've run) and what the results were including any error output. I'd suggest you go to the article I've mentioned above and follow those instructions and see how you get on.

[MrGreen]

Did you follow the Zimbra Certified wiki page on how to recreate certificates?

Unfortunately one doesn't necessarily follow the other in that statement.

Not really, from that information we can't tell a thing other than your certificates have (probably) expired. You need to give the full steps you've taken (including all the commands you've run) and what the results were including any error output. I'd suggest you go to the article I've mentioned above and follow those instructions and see how you get on.

Hi Bill and thank you for the reply.

I have followed the steps on this link Administration Console and CLI Certificate Tools - Zimbra :: Wiki, under the section Single-Node Self-Signed Certificate. Here are the results:

root@mail:~# /opt/zimbra/bin/zmcertmgr createca -new
** Creating /opt/zimbra/ssl/zimbra/ca/zmssl.cnf...chown: invalid option -- '1'
Try `chown --help' for more information.
done
chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
** Creating CA private key /opt/zimbra/ssl/zimbra/ca/ca.key...done.
** Creating CA cert /opt/zimbra/ssl/zimbra/ca/ca.pem...done.
root@mail:~#

root@mail:~# /opt/zimbra/bin/zmcertmgr createcrt -new -days 365
Validation days: 365
** Creating directory /opt/zimbra/mailboxd/webapps/zimbraAdmin/tmp
chown: invalid option -- '1'
Try `chown --help' for more information.
** Creating /opt/zimbra/conf/zmssl.cnf...chown: invalid option -- '1'
Try `chown --help' for more information.
done
** Backup /opt/zimbra/ssl/zimbra to /opt/zimbra/ssl/zimbra.20130321093054
** Generating a server csr for download self -new -keysize 1024
** Creating /opt/zimbra/conf/zmssl.cnf...chown: invalid option -- '1'
Try `chown --help' for more information.
done
** Backup /opt/zimbra/ssl/zimbra to /opt/zimbra/ssl/zimbra.20130321093054
chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
** Retrieving Commercial CA cert from ldap...failed.
** Creating server cert request /opt/zimbra/ssl/zimbra/server/server.csr...done.
** Saving server config key zimbraSSLPrivateKey...failed.
** Signing cert request /opt/zimbra/ssl/zimbra/server/server.csr...done.
root@mail:~#

root@mail:~# /opt/zimbra/bin/zmcertmgr deploycrt self
** Saving server config key zimbraSSLCertificate...failed.
** Saving server config key zimbraSSLPrivateKey...failed.
** Installing mta certificate and key...chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
done.
** Installing slapd certificate and key...chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
done.
** Installing proxy certificate and key...chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/localhost.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...failed.

Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(Java KeyStore.java:771)
at sun.security.provider.JavaKeyStore$JKS.engineLoad( JavaKeyStore.java:38)
at java.security.KeyStore.load(KeyStore.java:1185)
at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import .java:102)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(Java KeyStore.java:769)
... 3 more

** Installing CA to /opt/zimbra/conf/ca...chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
done.
root@mail:~#

root@mail:~# /opt/zimbra/bin/zmcertmgr deployca
** Importing CA /opt/zimbra/ssl/zimbra/ca/ca.pem into CACERTS...done.
** Copying CA to /opt/zimbra/conf/ca...chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
done.
chown: invalid option -- '1'
Try `chown --help' for more information.
chown: invalid option -- '1'
Try `chown --help' for more information.
root@mail:~#

root@mail:~# /opt/zimbra/bin/zmcertmgr viewdeployedcrt
::service mta::
notBefore=Mar 21 07:30:57 2013 GMT
notAfter=Mar 21 07:30:57 2014 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=localhost
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=localhost
SubjectAltName=
::service proxy::
notBefore=Mar 21 07:30:57 2013 GMT
notAfter=Mar 21 07:30:57 2014 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=localhost
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=localhost
SubjectAltName=
::service mailboxd::
XXXXX ERROR: failed to export /opt/zimbra/mailboxd/etc/mailboxd.pem from keystore.

keytool error: java.lang.RuntimeException: Usage error, /opt/zimbra/mailboxd/etc/keystore is not a legal command

notBefore=Mar 19 13:23:35 2012 GMT
notAfter=Mar 19 13:23:35 2013 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=mail.dekhs.co.za
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=mail.dekhs.co.za
SubjectAltName=
::service ldap::
notBefore=Mar 21 07:30:57 2013 GMT
notAfter=Mar 21 07:30:57 2014 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=localhost
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=localhost
SubjectAltName=
root@mail:~#

I hope this is the instructions that you referred to.

Thanks

Gert

[phoenix]

That error usually indicates a missing configuration file (do you have a backup of your server?), check these threads to confirm if that's the case.

[MrGreen]

I have gone through these threads yesterday and all I can see is that I also have a fairly empty localconfig.xml file. I think I have broken it after I ran a command to change the ssl_allow_untrusted_certs vlaue to true. That is when none of the services would start up with the ldap errors.

Here is my localconfig.xml file:

<?xml version="1.0" encoding="UTF-8"?>

<localconfig>
<key name="zimbra_java_home">
<value>/opt/zimbra/java</value>
</key>
<key name="ssl_allow_untrusted_certs">
<value>true</value>
</key>
</localconfig>

Unfortunately I do not have a proper backup of my localconfig.xml. (This is a hard lesson learned.)

Is there a way to repopulate the localconfig.xml? I was thinking of install Zimbra on a lab machine with the same configuration and then compare the localconfig.xml files.

Thanks for you help thus far.

Regards
Gert

[MrGreen]

Hi Bill,

I have installed a server with the exact config in a lab environment and had a look at the localconfig.xml file and could see that I have missed a whole lot of settings in there. Not sure how they disappeared.
I have then copied the content from the lab localconfig.xml to the broken server.
The only correction from this was the host name that shows up when I try and start the server.

I can see that ldap starts up. I can also connect to the zimbra db in mysql.

This is what happens when I try and create the certificates again:

root@mail:/home/administrator# /opt/zimbra/bin/zmcertmgr createca -new
** Creating /opt/zimbra/ssl/zimbra/ca/zmssl.cnf...done
** Creating CA private key /opt/zimbra/ssl/zimbra/ca/ca.key...done.
** Creating CA cert /opt/zimbra/ssl/zimbra/ca/ca.pem...done.
root@mail:/home/administrator# /opt/zimbra/bin/zmcertmgr createcrt -new -days 3650
Validation days: 3650
** Creating /opt/zimbra/conf/zmssl.cnf...done
** Backup /opt/zimbra/ssl/zimbra to /opt/zimbra/ssl/zimbra.20130321172043
** Generating a server csr for download self -new -keysize 1024
** Creating /opt/zimbra/conf/zmssl.cnf...done
** Backup /opt/zimbra/ssl/zimbra to /opt/zimbra/ssl/zimbra.20130321172043
** Retrieving Commercial CA cert from ldap...failed.
** Creating server cert request /opt/zimbra/ssl/zimbra/server/server.csr...done.
** Saving server config key zimbraSSLPrivateKey...failed.
** Signing cert request /opt/zimbra/ssl/zimbra/server/server.csr...done.
root@mail:/home/administrator# /opt/zimbra/bin/zmcertmgr deploycrt self
** Saving server config key zimbraSSLCertificate...failed.
** Saving server config key zimbraSSLPrivateKey...failed.
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...failed.

Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(Java KeyStore.java:771)
at sun.security.provider.JavaKeyStore$JKS.engineLoad( JavaKeyStore.java:38)
at java.security.KeyStore.load(KeyStore.java:1185)
at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import .java:102)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(Java KeyStore.java:769)
... 3 more

** Installing CA to /opt/zimbra/conf/ca...done.
** Installing CA to /opt/zimbra/conf/ca...done.
root@mail:/home/administrator# /opt/zimbra/bin/zmcertmgr deployca
** Importing CA /opt/zimbra/ssl/zimbra/ca/ca.pem into CACERTS...done.
** Saving global config key zimbraCertAuthorityCertSelfSigned...failed.
** Saving global config key zimbraCertAuthorityKeySelfSigned...failed.
** Copying CA to /opt/zimbra/conf/ca...done.

I am still getting the following:
zimbra@mail:~$ zmcontrol restart
Host myhost.mydomain.com
Stopping zmconfigd...Done.
Stopping stats...Done.
Stopping mta...Done.
Stopping spell...Done.
Stopping snmp...Done.
Stopping cbpolicyd...Done.
Stopping archiving...Done.
Stopping antivirus...Done.
Stopping antispam...Done.
Stopping imapproxy...Done.
Stopping memcached...Done.
Stopping mailbox...Done.
Stopping logger...Done.
Stopping ldap...Done.
Host myhost.mydomain.com
Starting ldap...Done.
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.

Do I need to look at maybe read/write/owner access to the zimbra files or is there a specific log file that I could look at to determine what is causing the server from not starting up?

Where can I look to troubleshoot this any further? Thanks

Regards
Gert

[MrGreen]

Hi Bill,

Also please note that the default admin user was replaced with systemadmin. Is this user referenced in any of the config files?

I really need some help with getting this server up. Do you think that it will be easier to do a new installation of Zimbra and then import the mysql and ldap?

Thanks
Gert