[SOLVED] Zimbra was working for a few days, then doesn't start anymore

[NxwJeff]

Hi everyone.

I've been playing and then migrating a few users to zimbra since the last few days and everything was going very smoothly.

Since I've been migrating users with huge mailboxes I had disabled the logger service. When I went to reeanable it I tried this:
/opt/zimbra/bin/zmprov ms zimbra.something.com zimbraServiceEnabled logger
Then my admin gui starting writing errors, so I tried/opt/zimbra/bin/zmprov ms zimbra.something.com +zimbraServiceEnabled logger
With the same result.

I then decided to stop zimbra completely and then restart it using zmcontrol.

To my utter surprise, it started ldap, logger, and stop there. No error, nothing. Just stopped like it was normal.

I managed to start apache and others so I could use zmprov again to disable logger using
/opt/zimbra/bin/zmprov ms zimbra.something.com -zimbraServiceEnabled logger

And now it only starts ldap and stops.

I stopped / started zimbra yesterday with no problem, and as far as I can tell nothing what so ever changed on either the zimbra server or on the dns entries.

Needless to say I'm in trouble right now. I'll try to display as much stuff as possible here, but please ask for anything that could help you help me.

Server name is zimbra.something.com
it resolves to 192.168.200.223 (local ip) as it should (unless I'm mistaken)

Here is the service line (so yes, ldap is working):
zmprov -l gs zimbra.somthing.com | grep -i service

zimbraMtaAuthURL: https://zimbra.something.com:443/service/soap/
zimbraServiceHostname: zimbra.something.com
zimbraServiceInstalled: antivirus
zimbraServiceInstalled: antispam
zimbraServiceInstalled: logger
zimbraServiceInstalled: mailbox
zimbraServiceInstalled: mta
zimbraServiceInstalled: convertd
zimbraServiceInstalled: stats
zimbraServiceInstalled: snmp
zimbraServiceInstalled: ldap
zimbraServiceInstalled: spell

One thing, is ldap supposed to be listening only on 127.0.1.1?
tcp 0 0 127.0.1.1:389 0.0.0.0:* LISTEN -

/var/log/zimbra.log shows after a zmcontrol start:

Jul 23 21:23:38 zimbra zimbramon[12309]: 12309:info: Starting services initiated by zmcontrol 
Jul 23 21:23:38 zimbra slapd[12388]: @(#) $OpenLDAP: slapd 2.3.43 (Apr 13 2009 11:14:45) $ ^Iroot@build26.lab.zimbra.com:/home/build/p4/FRANKLIN/ThirdParty/openldap/openldap-2.3.43.8z/servers/slapd 
Jul 23 21:23:39 zimbra slapd[12389]: slapd starting

Then if I do a zmcontrol status

Jul 23 21:24:03 zimbra zimbramon[12434]: 12434:info: 2009-07-23 21:24:01, STATUS: : Cannot: determine services - exiting

Thanks in advance for your help. Again I'm in deep trouble right now (some of the users already migrated are the bosses of the company) so any kind of help would be more than welcome.

[phoenix]

Let's start with the basics, run the following commands on the zimbra server and post the output:

cat /etc/hosts
cat /etc/resolv.conf
dig yourdomain.com mx
dig yourdomain.com any
host `hostname`  <-- use that exact command with backticks not single quotes

Is this server on a LAN or public IP? Do you have a firewall and/or SElinux (or AppArmor) enabled? How much RAM does the server have? Is it in a VM?

[NxwJeff]

Thanks for the fast reply! Here are the results:
cat /etc/hosts

127.0.0.1       localhost
127.0.1.1       zimbra.something.com    zimbra

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

cat /etc/resolv.conf

search something.com
nameserver 192.168.200.254
nameserver 192.168.200.213

dig something.com mx (this is a spit domain, with the old mail server as master)

; <<>> DiG 9.4.2-P2 <<>> something.com mx
;; global options:  printcmd
;; Got answer:  
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11807
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 4

;; QUESTION SECTION:
;something.com.         IN      MX

;; ANSWER SECTION:
something.com.  1200    IN      MX      5 mail.somethingelse.com.

;; AUTHORITY SECTION:
something.com.  1200    IN      NS      ns4.something.com.
something.com.  1200    IN      NS      ns1.something.com.
something.com.  1200    IN      NS      ns2.something.com.

;; ADDITIONAL SECTION:
mail.somethingelse.com. 1200    IN      A       192.168.200.213
ns1.something.com.      1200    IN      A       removed
ns2.something.com.      1200    IN      A       removed
ns4.something.com.      1200    IN      A       removed

;; Query time: 8 msec
;; SERVER: 192.168.200.254#53(192.168.200.254)
;; WHEN: Thu Jul 23 21:52:14 2009
;; MSG SIZE  rcvd: 187

dig something.com any

; <<>> DiG 9.4.2-P2 <<>> something.com any
;; global options:  printcmd    
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64294
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:    
;something.com.         IN      ANY     

;; ANSWER SECTION:
something.com.  1200    IN      SOA     ns1.something.com. ns1.something.com. 2009072000 21600 3600 604800 90000
something.com.  1200    IN      A       removed
something.com.  1200    IN      MX      5 mail.somethingelse.com.
something.com.  1200    IN      NS      ns2.something.com.
something.com.  1200    IN      NS      ns4.something.com.
something.com.  1200    IN      NS      ns1.something.com.

;; ADDITIONAL SECTION:
mail.somethingelse.com. 1200    IN      A       192.168.200.213
ns1.something.com.      1200    IN      A       removed
ns2.something.com.      1200    IN      A       removed
ns4.something.com.      1200    IN      A       removed

;; Query time: 0 msec
;; SERVER: 192.168.200.254#53(192.168.200.254)
;; WHEN: Thu Jul 23 21:52:39 2009
;; MSG SIZE  rcvd: 239

host `hostname`

zimbra.something.com has address 192.168.200.223

Thanks again for your help!

[phoenix]

You do actually need Zimbra to be an a valid LAN IP address, that's not reflected in your /etc/hosts file which should look like the following:

127.0.0.1       localhost.localdomain localhost
192.168.x.x       zimbra.something.com    zimbra

You will also need valid DNS A & MX records pointing to the Zimbra server, it's needed (by postfix) for resolution of the server IP. According to the host command you ran at the end it shows the Zimbra IP as: 192.168.200.223 that's not confirmed by your hosts file. What is the answer to my other questions?

[NxwJeff]

Sorry I missed your other questions, I'm under so much stress right now I don't know what I'm doing half the time :/

I changed the hosts file to

192.168.200.223 zimbra.something.com zimbra

The server is on a LAN, with only a private IP. There is full redirection for a public IP to that server on the firewall so it can be accessed from the outside (2 views on the DNS, internal points to 192.168.200.223, external to the public IP).
There is no SELINUX or firewall running on that machine. It's the bare minimum brand new ubuntu 8.04 install, with what was need by zimbra added. The server has 2 GB of RAM, and it's not a VM.

As for the MX issue, I though that if the server was in a split domain as slave it was not needed? It sure worked correctly until now, with mails being redirected from the old postfix to zimbra for the accounts already migrated, and everything else going though the old server.

[NxwJeff]

I've restarted zmcontrol, and now ldap is listening on 192.168.200.223
tcp 0 0 192.168.200.223:389 0.0.0.0:* LISTEN -

Nothing else changed though.

Doing a strace on zmcontrol status, I see it connecting to the ldap:
25120 getsockname(5, {sa_family=AF_INET6, sin6_port=htons(33274), inet_pton(AF_INET6, "::ffff:192.168.200.223", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [9271332184302551068]) = 0

[phoenix]

So you've followed the wiki article on Split Domain, did you set-up Zimbra as the primary or secondary server? You should also have just one DNS server listed in your resolve.conf unless you have an exact copy of the DNS records for your zimbra server on both DNS servers.

[phoenix]

Shutdown zimbra and kill any remaining processes and restart zimbra and what is the output of:

zmcontrol start
zmcontrol -v

Anything odd in the log files?

[NxwJeff]

Yep I've followed the split domain wiki to the letter as far as I can tell, and everything worked perfectly well until I did that logger command(s).
Zimbra is my secondary server.
Both DNS servers are synchronized for both views, so there won't be any problems with these.

[NxwJeff]

zmcontrol start

Host zimbra.something.com
	Starting ldap...Done.

zmcontrol -v

Release 5.0.16_GA_2921.UBUNTU8_64 UBUNTU8_64 NETWORK edition

/var/log/zimbra.log

Jul 23 22:26:27 zimbra zimbramon[27931]: 27931:info: Starting services initiated by zmcontrol 
Jul 23 22:26:28 zimbra slapd[28010]: @(#) $OpenLDAP: slapd 2.3.43 (Apr 13 2009 11:14:45) $ ^Iroot@build26.lab.zimbra.com:/home/build/p4/FRANKLIN/ThirdParty/openldap/openldap-2.3.43.8z/servers/slapd 
Jul 23 22:26:28 zimbra slapd[28011]: slapd starting

No log files in /opt/zimbra/log are updated when I do a zmcontrol start