[SOLVED] Zimbra stopped working suddenly, need advice

[Chaduke]

The server has been running fine for about a year and yesterday it abrubptly stopped working, the admin page and login page just time out. I'm trying to refresh my memory and get caught up on the documentation as well as searching the forums but no luck yet.

Here are the results of what I've done so far -

su zimbra
cd /opt/zimbra/bin

zmcontrol status 
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.

zmcontrol -v
Release 6.0.1_GA_1816.F11_64_20090911212710 F11_64 FOSS edition

cat /etc/hosts
127.0.0.1       localhost.localdomain   localhost.localdomain   localhost4     localhost4.localdomain4  localhost
::1     localhost.localdomain   localhost.localdomain   localhost6      localhost6.localdomain6 localhost
172.30.44.251   zimbra.mydomain.org     zimbra

cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 172.16.0.142
nameserver 172.16.0.138

dig mydomain.org mx
; <<>> DiG 9.6.1-P1-RedHat-9.6.1-4.P1.fc11 <<>> mydomain.org mx
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58418
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;mydomain.org.          IN      MX

;; ANSWER SECTION:
mydomain.org.   3600    IN      MX      10 zimbra.

;; Query time: 93 msec
;; SERVER: 172.16.0.142#53(172.16.0.142)
;; WHEN: Fri Sep 24 09:44:06 2010
;; MSG SIZE  rcvd: 60

dig zimbra.mydomain.org

; <<>> DiG 9.6.1-P1-RedHat-9.6.1-4.P1.fc11 <<>> zimbra.mydomain.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62551
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;zimbra.mydomain.org.   IN      A

;; ANSWER SECTION:
zimbra.mydomain.org. 3600 IN    A       172.30.44.251

;; Query time: 0 msec
;; SERVER: 172.16.0.142#53(172.16.0.142)
;; WHEN: Fri Sep 24 09:45:51 2010
;; MSG SIZE  rcvd: 61

host `hostname`
zimbra.mydomain.org has address 172.30.44.251

zmcontrol stop
Host zimbra.mydomain.org
	Stopping stats...Done.
	Stopping mta...Done.
	Stopping spell...Done.
	Stopping snmp...Done.
	Stopping archiving...Done.
	Stopping antivirus...Done.
	Stopping antispam...Done.
	Stopping imapproxy...Done.
	Stopping memcached...Done.
	Stopping mailbox...Done.
	Stopping logger...Done.
	Stopping ldap...Done.

zmcontrol startup
Host zimbra.mydomain.org
	Starting ldap...Done.
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.

/var/log/zimbra.log looks like this after the shutdown messages

Sep 24 09:56:57 zimbra zimbramon[11983]: 11983:info: Starting services initiated by zmcontrol
Sep 24 09:56:57 zimbra slapd[12023]: @(#) $OpenLDAP: slapd 2.4.17 (Aug  2 2009 16:50:22) $#012#011build@build29.lab.zimbra.com:/home/build/p4/main/ThirdParty/openldap/openldap-2.4.17.8z/servers/slapd
Sep 24 09:56:57 zimbra slapd[12024]: slapd starting

If someone could point me in the right direction I'd much appreciate.

[soxfan]

Looks like someone else is having the same / similar issue (see Zimbra not running). Not sure, but I'm thinking it could be an issue with self-signed certificates. Search the Zimbra Wiki for more info.

[phoenix]

Originally Posted by Chaduke

The server has been running fine for about a year and yesterday it abrubptly stopped working, the admin page and login page just time out.

You should look in the log files for any errors but, as soxfan suggested, I'd also guess it's the certificates from your highlighted comment.

[Chaduke]

Originally Posted by phoenix

You should look in the log files for any errors but, as soxfan suggested, I'd also guess it's the certificates from your highlighted comment.

Used these instructions to fix, thanks for the replies:

Single-Node Self-Signed Certificate

1. Begin by generating a new Certificate Authority (CA).

zmcertmgr createca -new

2. Then generate a certificate signed by the CA that expires in 365 days.

zmcertmgr createcrt -new -days 365

3. Next deploy the certificate.

zmcertmgr deploycrt self

4. Next deploy the CA.

zmcertmgr deployca

5. To finish, verify the certificate was deployed to all the services.

zmcertmgr viewdeployedcrt

[Chaduke]

Originally Posted by phoenix

You should look in the log files for any errors but, as soxfan suggested, I'd also guess it's the certificates from your highlighted comment.

Also wanted to add a comment about this. Seems like this would be a fairly common problem. I'm a little confused about why this would cause the entire server to fail and then consequently not start up again. The error message you get from zmcontrol and the main logs I check say nothing about a certificate expiring.

Seems like logging could be improved in regards to this, unless I'm missing something.