SSL certificate renewal & recover from file system crash

[j2b]

Dear all,
Have been searching forums, and could not find any relevant solution to my issue. Tonight, I suffered file system crash in multi-server installation of our ZCS servers (v6.0.10 OSS in this case 32bit Ubuntu 8.04). After FS recovery, I was trying to start mailbox server, which didn't succeed. Following investigation, noticed, that LDAP server is not started, and turned to that task - start LDAP, which was not successfull too.

$ zmcontrol start
Host ldap.example.com
    Starting ldap...Done.
Failed.
/opt/zimbra/bin/ldap: line 56: kill: (6303) - No such process
/opt/zimbra/openldap/sbin/slapd: /opt/zimbra/cyrus-sasl-2.1.23.3z/lib/libsasl2.so.2: no version information available (required by /usr/lib/libldap_r-2.4.so.2)
/opt/zimbra/bin/ldap: line 56: kill: (6303) - No such process
/opt/zimbra/openldap/sbin/slapd: /opt/zimbra/cyrus-sasl-2.1.23.3z/lib/libsasl2.so.2: no version information available (required by /usr/lib/libldap_r-2.4.so.2)
/opt/zimbra/bin/ldap: line 56: kill: (6303) - No such process
/opt/zimbra/openldap/sbin/slapd: /opt/zimbra/cyrus-sasl-2.1.23.3z/lib/libsasl2.so.2: no version information available (required by /usr/lib/libldap_r-2.4.so.2)
/opt/zimbra/bin/ldap: line 56: kill: (6303) - No such process
/opt/zimbra/openldap/sbin/slapd: /opt/zimbra/cyrus-sasl-2.1.23.3z/lib/libsasl2.so.2: no version information available (required by /usr/lib/libldap_r-2.4.so.2)
/opt/zimbra/bin/ldap: line 56: kill: (6303) - No such process
/opt/zimbra/openldap/sbin/slapd: /opt/zimbra/cyrus-sasl-2.1.23.3z/lib/libsasl2.so.2: no version information available (required by /usr/lib/libldap_r-2.4.so.2)
/opt/zimbra/bin/ldap: line 56: kill: (6303) - No such process
/opt/zimbra/openldap/sbin/slapd: /opt/zimbra/cyrus-sasl-2.1.23.3z/lib/libsasl2.so.2: no version information available (required by /usr/lib/libldap_r-2.4.so.2)
/opt/zimbra/bin/ldap: line 56: kill: (6303) - No such process
/opt/zimbra/openldap/sbin/slapd: /opt/zimbra/cyrus-sasl-2.1.23.3z/lib/libsasl2.so.2: no version information available (required by /usr/lib/libldap_r-2.4.so.2)
Failed to start slapd.  Attempting debug start to determine error.
config error processing cn=config: <olcTLSCRLCheck> handler exited with -1

/var/log/zimbra.log displays similar info, but with slight additional details:

Jan 16 16:24:07 ldap-1 slapd[15235]: @(#) $OpenLDAP: slapd 2.4.23 (Jun 30 2010 12:22:04) $ ^Iroot@build25.lab.zimbra.com:/home/build/p4/GNR/ThirdParty/openldap/openldap-2.4.23.2z/servers/slapd 
Jan 16 16:24:08 ldap-1 slapd[15235]: config error processing cn=config: <olcTLSCRLCheck> handler exited with -1 
Jan 16 16:24:08 ldap-1 slapd[15235]: slapd stopped. 
Jan 16 16:24:08 ldap-1 slapd[15235]: connections_destroy: nothing to destroy.

Investigating further I followed to an issue, that it may be due to invalid self signetd SSL certificates, which are deployed inside our system (e.g. frontside is run on apache with its own commercial SSL). And for time being, that these servers were planned to de-comission, nobody turned attention to these certs, as well as there are no sufficient backup available. (I know, shame on me) But still we have to get these servers back, to access mail archive.

I suspect, that this issue is connected with the fact, that LDAP server can not very CA, which may be due to old certs. But regular recreation of new self signed certs ask for running LDAP server, to be deployed correctly. This turned out on fact, that deploycrt and deployca didn't run without an error:

...deploycrt self error:

** Saving server config key zimbraSSLCertificate...failed.
** Saving server config key zimbraSSLPrivateKey...failed.

...deployca error:

** Saving global config key zimbraCertAuthorityCertSelfSigned...failed.
** Saving global config key zimbraCertAuthorityKeySelfSigned...failed.

Still ...viewdeployedcrt mostly looks OK, despite errors on failed export, as mailbox is running on the other server:

::service mta::
notBefore=Jan 16 13:47:18 2012 GMT
notAfter=Jan 15 13:47:18 2013 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=ldap.example.com
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=ldap.example.com
SubjectAltName= 
::service proxy::
notBefore=Jan 16 13:47:18 2012 GMT
notAfter=Jan 15 13:47:18 2013 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=ldap.example.com
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=ldap.example.com
SubjectAltName= 
::service mailboxd::
XXXXX ERROR: failed to export /opt/zimbra/mailboxd/etc/mailboxd.pem from keystore.

keytool error: java.lang.RuntimeException: Usage error, /opt/zimbra/conf/keystore is not a legal command

XXXXX ERROR: /opt/zimbra/mailboxd/etc/mailboxd.pem does not exist
::service ldap::
notBefore=Jan 16 13:47:18 2012 GMT
notAfter=Jan 15 13:47:18 2013 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=ldap.example.com
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration Suite/CN=ldap.example.com
SubjectAltName=

...keytool command does not run throught too, giving an error, that root alias nor zimbra alias is not correct.

Does anybody had an issue, where new certificates should be installed on non-running ZCS instance? Is it at all possible?

If not, the only data I have is files in /opt/zimbra... folders. Does such situation may lead to server recovery at all?

[j2b]

Please do not bother, as I've solved this issue quite magically. Just run ./install.sh -s over the data (e.g. without configuration and setup), and everything started to work without loss of any data.

To be honest, I did changed certs (self signed) and despite their deployment didn't go without any error, it turned out to be correct from Zimbra Admin UI screen. And in addition, I had to run e2fsck on device. Please be sure, if somebody tries to do the same, that this is not warrantied method, as well as doing e2fsck requires unmounting device and changing system to single user mode (init 1).

Reboot is needed.