Yet another zmmailboxd problem

[Storm16]

Hi all,

I see that I am in good company on having this issue after upgrading to 6.0.x. I upgraded from 5.0.16 and also upgraded to 6.0.1. I backed up, made sure there were no zimbra processes were running, and did the upgrade.

Now, I get messages stating

Zimbra server reserving server socket port=110 bindaddr=null ssl=false
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Total time for which application threads were stopped: 0.5210790 seconds
Fatal error: exception while binding to ports

However, netstat shows there is nothing on those ports:

[root@merrimac ~]# netstat -pan | grep 110
[root@merrimac ~]# netstat -pan | grep 995
[root@merrimac ~]#

I also saw in /var/log/zimbra.log that it is looking for zmmailboxd.pid, which does not exist:

Oct 18 19:46:34 merrimac zmmailboxdmgr[17576]: status requested
Oct 18 19:46:34 merrimac zmmailboxdmgr[17576]: stale pid 15573 found in /opt/zi
mbra/log/zmmailboxd_manager.pid: No such process
Oct 18 19:46:34 merrimac zmmailboxdmgr[17576]: assuming no other instance is ru
nning
Oct 18 19:46:34 merrimac zmmailboxdmgr[17576]: file /opt/zimbra/log/zmmailboxd.
pid does not exist

Nothing in /opt/zimbra/log/mailbox.log since 12 Oct.

And here is my obligatory netstat -pan (zimbra is not running):

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:514           0.0.0.0:*               LISTEN      2452/stunnel4
tcp        0      0 192.168.10.12:53       0.0.0.0:*               LISTEN      2369/named
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      2369/named
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2383/sshd
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      2369/named
tcp        0      0 192.168.10.12:22       192.168.10.1:44431     ESTABLISHED 29282/0
tcp6       0      0 :::22                   :::*                    LISTEN      2383/sshd
tcp6       0      0 ::1:953                 :::*                    LISTEN      2369/named
tcp6       0      0 ::1:57931               ::1:37849               TIME_WAIT   -
udp        0      0 192.168.10.12:32768    192.168.10.61:1514     ESTABLISHED 2416/ossec-agentd
udp        0      0 0.0.0.0:161             0.0.0.0:*                           2437/snmpd
udp        0      0 192.168.10.12:53       0.0.0.0:*                           2369/named
udp        0      0 127.0.0.1:53            0.0.0.0:*                           2369/named

I had to disable exim4, since it had been reenabled on upgrade. After that, I did a reinstall of Zimbra, from 6.0.1 to 6.0.1. The only difference is that when I installed the first time, I installed memcached, but uninstalled the package before reinstalling (though it still tries to start it...) I also tried suggestions from some older threads, including removing the zmmailboxd*.pid files and trying to start zmmailboxctld by hand, to no avail.

Suggestion on what is causing my particular issue?

Thanks,
--storm

[uxbod]

cat /etc/hosts
cat /etc/resolv.conf
dig yourdomain mx
dig yourdomain any
host `hostname` <- note backticks and not single quotes (copy 'n' paste)

[Storm16]

Originally Posted by uxbod

cat /etc/hosts
cat /etc/resolv.conf
dig yourdomain mx
dig yourdomain any
host `hostname` <- note backticks and not single quotes (copy 'n' paste)

Hi uxbod,

Here is the info. Two notes...First, I obfuscated the domain name, but it is uniform, and second, this worked just fine with 5.0.16 and prior. I have been using Zimbra on this same hardware for several years (since 3.x or the early 4.x series?)

[root@merrimac ~]# cat /etc/hosts 
127.0.0.1       localhost         
192.168.10.12   merrimac.foo.homelinux.org      merrimac

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback                        
fe00::0 ip6-localnet                                      
ff00::0 ip6-mcastprefix                                   
ff02::1 ip6-allnodes                                      
ff02::2 ip6-allrouters                                    
ff02::3 ip6-allhosts                                      

[root@merrimac ~]# cat /etc/resolv.conf
search foo.homelinux.org               
nameserver 192.168.10.50               

[root@merrimac ~]# dig foo.homelinux.org mx
; <<>> DiG 9.5.1-P3 <<>> foo.homelinux.org mx
;; global options:  printcmd                 
;; Got answer:                               
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50658
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;foo.homelinux.org.     IN      MX

;; ANSWER SECTION:
foo.homelinux.org. 86400 IN     MX      20 mx1.uslinux.net.
foo.homelinux.org. 86400 IN     MX      10 merrimac.foo.homelinux.org.
foo.homelinux.org. 86400 IN     MX      15 gwyn.tux.org.              

;; AUTHORITY SECTION:
foo.homelinux.org. 86400 IN     NS      merrimac.foo.homelinux.org.
foo.homelinux.org. 86400 IN     NS      picard.foo.homelinux.org.  

;; ADDITIONAL SECTION:
merrimac.foo.homelinux.org. 86400 IN A 192.168.10.12
picard.foo.homelinux.org. 86400 IN A    192.168.10.50

;; Query time: 2 msec
;; SERVER: 192.168.10.50#53(192.168.10.50)
;; WHEN: Mon Oct 19 20:52:25 2009         
;; MSG SIZE  rcvd: 193                    


[root@merrimac ~]# dig foo.homelinux.org any
; <<>> DiG 9.5.1-P3 <<>> foo.homelinux.org any
;; global options:  printcmd                  
;; Got answer:                                
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30515
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;foo.homelinux.org.     IN      ANY

;; ANSWER SECTION:
foo.homelinux.org. 86400 IN     SOA     picard.foo.homelinux.org. postmaster.foo.homelinux.org. 2009091203 10800 3600 604800 86400
foo.homelinux.org. 86400 IN     NS      merrimac.foo.homelinux.org.
foo.homelinux.org. 86400 IN     NS      picard.foo.homelinux.org.
foo.homelinux.org. 86400 IN     MX      15 gwyn.tux.org.
foo.homelinux.org. 86400 IN     MX      20 mx1.uslinux.net.
foo.homelinux.org. 86400 IN     MX      10 merrimac.foo.homelinux.org.

;; ADDITIONAL SECTION:
picard.foo.homelinux.org. 86400 IN A    192.168.10.50
merrimac.foo.homelinux.org. 86400 IN A 192.168.10.12

;; Query time: 2 msec
;; SERVER: 192.168.10.50#53(192.168.10.50)
;; WHEN: Mon Oct 19 20:52:30 2009
;; MSG SIZE  rcvd: 240

[root@merrimac ~]# host `hostname`
merrimac.foo.homelinux.org has address 192.168.10.12

[Storm16]

Anyone know of a solution for this?

[phoenix]

Originally Posted by Storm16

Anyone know of a solution for this?

Is your server actually updated and is it running or stopped?

Your hosts file is incorrect, it should be as follows:

127.0.0.1       localhost.localdomain localhost        
192.168.10.12   merrimac.foo.homelinux.org      merrimac

Do you have the proxy installed an enabled on this server and is it a single-server?

[Storm16]

Originally Posted by phoenix

Is your server actually updated and is it running or stopped?

Everything starts but xmmailboxd.

Your hosts file is incorrect, it should be as follows:

127.0.0.1       localhost.localdomain localhost        
192.168.10.12   merrimac.foo.homelinux.org      merrimac

Do you have the proxy installed an enabled on this server and is it a single-server?

I corrected the issue with /etc/hosts, and still get the same result:

zimbra@merrimac:~$ zmcontrol start                                          
Host merrimac.sonsofthunder.homelinux.org                                   
        Starting ldap...Done.                                               
        Starting logger...Done.                                             
        Starting mailbox...Done.                                            
        Starting memcached...Failed.                                        
Starting memcached.../opt/zimbra/bin/zmmemcachedctl: line 65: /opt/zimbra/memcached/bin/memcached: No such file or directory                                  
failed.                                                                        


        Starting antispam...Done.
        Starting antivirus...Done.
        Starting snmp...Done.
        Starting spell...Done.
        Starting mta...Done.
        Starting stats...Done.
You have new mail in /var/mail/zimbra
zimbra@merrimac:~$ zmcontrol status
Host merrimac.sonsofthunder.homelinux.org
        antispam                Running
        antivirus               Running
        ldap                    Running
        logger                  Running
        mailbox                 Stopped
                zmmailboxdctl is not running.
        memcached               Stopped
                memcached is not running.
        mta                     Running
        snmp                    Running
        spell                   Running
        stats                   Running
zimbra@merrimac:~$

Would re (re-re) installing Zimbra buy me anything? I reinstalled when I first encountered problems, and since I had accidentally installed memcached, I de-installed the deb before I reinstalled, but it still tries to start it.

This is, btw, a single-server install with no proxy installed. (I didn't answer that earlier.)

Thoughts?

[Dirk]

Try this, you've nothing to lose and it worked for me with a similar (but not identical) issue: cert issue 5.0.18GA to 6.0.1GA

[Storm16]

Originally Posted by Dirk

Try this, you've nothing to lose and it worked for me with a similar (but not identical) issue: cert issue 5.0.18GA to 6.0.1GA

Thanks Dirk,

It looks like you are pointing me down the right path. I noticed a couple of things when running /opt/zimbra/bin/zmcertmgr viewdeployedcrt.

1. The upgrade apparently did not update the cert.

::service mta::                                                       
notBefore=Apr 20 17:24:19 2007 GMT                                    
notAfter=Apr 19 17:24:19 2008 GMT

The one in the service proxy has a notBefore/notAfter of May 15 2009/2010, which I suspect was about when I upgraded to 5.0.16

2. The LDAP cert has the same dates as the MTA service.

And finally, (and most damningly, IMHO):

    ::service mailboxd::                                                       
    XXXXX ERROR: failed to export /opt/zimbra/mailboxd/etc/mailboxd.pem from
    keystore.                                                                  

    keytool error: java.lang.Exception: Keystore file does not exist:
    /opt/zimbra/mailboxd/etc/keystore                                          

    XXXXX ERROR: /opt/zimbra/mailboxd/etc/mailboxd.pem does not exist

Does this sound like the cause of my issues? If so, what is the fix? I am researching in the wiki, but thought I would ask as well. I apologize, but my last upgrade went really badly (5.0.12 - 5.0.16), and I ended up losing a lot of email, so I am treading gingerly.

Thanks.

[veronica]

If your upgrade is not proper you might land in many more issues. Try running zmsetup.pl first and then recreate the certs.