[SOLVED] postfix/postqueue fatal: Queue report unavailable - mail system is down

[tempusfugit]

Hi All!

I need help - recently started getting these messages in zimbra.log:

Apr 1 11:14:36 pixserver zmmailboxdmgr[22488]: status OK
Apr 1 11:15:00 pixserver postfix/postqueue[22525]: fatal: Queue report unavailable - mail system is down
Apr 1 11:15:30 pixserver postfix/postqueue[22535]: fatal: Queue report unavailable - mail system is down
Apr 1 11:15:46 pixserver zmmailboxdmgr[22790]: status requested
Apr 1 11:15:46 pixserver zmmailboxdmgr[22790]: status OK
Apr 1 11:15:47 pixserver zmmailboxdmgr[22846]: status requested
Apr 1 11:15:47 pixserver zmmailboxdmgr[22846]: status OK
Apr 1 11:16:00 pixserver postfix/postqueue[22883]: fatal: Queue report unavailable - mail system is down
Apr 1 11:16:03 pixserver zmmailboxdmgr[23028]: status requested
Apr 1 11:16:03 pixserver zmmailboxdmgr[23028]: status OK

However, all services are running:
zimbra@pixserver:~$ zmcontrol status
Host mail.#####.net
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Running
mta Running
snmp Running
spell Running
stats Running

The only clue that I have is that everything was operational until Ubuntu released an openssl (0.9.8g-4ubuntu3.5) update a few days ago. I applied that update, and the server stopped sending mail the next day. I did a reinstall/upgrade, but that did not cure the issue.

Any help would be most appreciated!

[tempusfugit]

Here's extract from mail.log, right around the time I rebooted:

Mar 31 19:46:05 pixserver zmmailboxdmgr[5631]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 31 19:46:05 pixserver zmmailboxdmgr[5631]: assuming no other instance is running
Mar 31 19:46:05 pixserver zmmailboxdmgr[5631]: no manager process is running
Mar 31 19:46:58 pixserver zmmailboxdmgr[7057]: status requested
Mar 31 19:46:58 pixserver zmmailboxdmgr[7057]: file /opt/zimbra/log/zmmailboxd_manager.pid does not exist
Mar 31 19:46:58 pixserver zmmailboxdmgr[7057]: assuming no other instance is running
Mar 31 19:46:58 pixserver zmmailboxdmgr[7057]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 31 19:46:58 pixserver zmmailboxdmgr[7057]: assuming no other instance is running
Mar 31 19:46:58 pixserver zmmailboxdmgr[7057]: no manager process is running
Mar 31 19:46:58 pixserver zmmailboxdmgr[7064]: start requested
Mar 31 19:46:58 pixserver zmmailboxdmgr[7064]: checking if another instance of manager is already running
Mar 31 19:46:58 pixserver zmmailboxdmgr[7064]: file /opt/zimbra/log/zmmailboxd_manager.pid does not exist
Mar 31 19:46:58 pixserver zmmailboxdmgr[7064]: assuming no other instance is running
Mar 31 19:46:58 pixserver zmmailboxdmgr[7064]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 31 19:46:58 pixserver zmmailboxdmgr[7064]: assuming no other instance is running
Mar 31 19:46:58 pixserver zmmailboxdmgr[7065]: wrote manager pid 7065 to /opt/zimbra/log/zmmailboxd_manager.pid
Mar 31 19:46:58 pixserver zmmailboxdmgr[7066]: wrote java pid 7066 to /opt/zimbra/log/zmmailboxd_java.pid
Mar 31 19:46:58 pixserver zmmailboxdmgr[7065]: manager started mailboxd/JVM with pid 7066
Mar 31 19:47:30 pixserver postfix/postqueue[7650]: fatal: Queue report unavailable - mail system is down
Mar 31 19:47:52 pixserver zmmailboxdmgr[7934]: status requested
Mar 31 19:47:52 pixserver zmmailboxdmgr[7934]: status OK
Mar 31 19:47:53 pixserver zmmailboxdmgr[7990]: status requested
Mar 31 19:47:53 pixserver zmmailboxdmgr[7990]: status OK
Mar 31 19:48:00 pixserver postfix/postqueue[8024]: fatal: Queue report unavailable - mail system is down
Mar 31 19:48:03 pixserver zmmailboxdmgr[8173]: status requested

[tempusfugit]

I've sifted through the forums for several hours, but to no avail. I'm sure my DNS is configured correctly, and I've verified that no other MTA is installed. As I stated, it was operational until this openssl update was applied last night - I verified this on the phone with the user.

Would appreciate any assistance soon - user is considering scrapping the system and going with their ISP's mail system.

zimbra@pixserver:~$ zmcontrol -v
Release 5.0.14_GA_2850.UBUNTU8 UBUNTU8 FOSS edition

zimbra@pixserver:~$ netstat -luntp | grep 25
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::7025 :::* LISTEN -

zimbra@pixserver:~$ dpkg -l | grep postfix
zimbra@pixserver:~$

[uxbod]

Stop ZCS and then check to make sure all processes have indeed stopped

ps aux | grep zimbra

if not then kill them off nicely. Leave a couple of minutes to ensure all the sockets have been released. Then try starting again.

[tempusfugit]

Thanks for the reply!

zimbra@pixserver:~$ zmcontrol stop
Host mail.extremexposures.net
Stopping stats...Done
Stopping mta...Done
Stopping spell...Done
Stopping snmp...Done
Stopping archiving...Done
Stopping antivirus...Done
Stopping antispam...Done
Stopping imapproxy...Done
Stopping mailbox...Done
Stopping logger...Done
Stopping ldap...Done
zimbra@pixserver:~$ ps aux | grep zimbra
zimbra 692 0.0 0.0 2364 924 pts/0 R+ 07:20 0:00 ps aux
zimbra 693 0.0 0.0 1784 536 pts/0 R+ 07:20 0:00 grep zimbra
zimbra 32523 0.0 0.1 7136 1924 pts/0 S 07:19 0:00 su - zimbra
zimbra 32524 0.0 0.1 4624 2024 pts/0 R 07:19 0:00 -su
zimbra@pixserver:~$ zmcontrol start
Host mail.extremexposures.net
Starting ldap...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.
zimbra@pixserver:~$

I'm adding the following output also:

zimbra@pixserver:~$ ps -u zimbra
PID TTY TIME CMD
758 ? 00:00:00 slapd
1131 pts/0 00:00:00 mysqld_safe
1137 pts/0 00:00:00 logswatch
1177 pts/0 00:00:00 mysqld
1179 pts/0 00:00:00 perl
1190 pts/0 00:00:00 zmlogger
1298 pts/0 00:00:00 zmmtaconfig
1312 pts/0 00:00:00 mysqld_safe
1421 pts/0 00:00:00 mysqld
2347 ? 00:00:05 java
2394 ? 00:00:00 amavisd
2427 pts/0 00:00:00 freshclam
2434 ? 00:00:00 amavisd
2435 ? 00:00:00 amavisd
2436 ? 00:00:00 amavisd
2437 ? 00:00:00 amavisd
2438 ? 00:00:00 amavisd
2439 ? 00:00:00 amavisd
2440 ? 00:00:00 amavisd
2441 ? 00:00:00 amavisd
2442 ? 00:00:00 amavisd
2443 ? 00:00:00 amavisd
2474 ? 00:00:00 clamd
2481 pts/0 00:00:00 swatch
2492 pts/0 00:00:00 perl
2493 ? 00:00:00 httpd
2494 ? 00:00:00 httpd
2495 ? 00:00:00 httpd
2498 ? 00:00:00 httpd
2500 ? 00:00:00 httpd
2502 ? 00:00:00 httpd
2557 ? 00:00:00 saslauthd
2558 ? 00:00:00 saslauthd
2560 ? 00:00:00 saslauthd
2563 ? 00:00:00 saslauthd
2564 ? 00:00:00 saslauthd
2622 pts/0 00:00:00 zmstat-proc
2624 pts/0 00:00:00 zmstat-cpu
2627 pts/0 00:00:00 zmstat-vm
2633 pts/0 00:00:00 zmstat-fd
2635 pts/0 00:00:00 zmstat-mysql
2639 pts/0 00:00:00 zmstat-mtaqueue
2879 pts/0 00:00:00 vmstat
3287 pts/0 00:00:00 ps
32523 pts/0 00:00:00 su
32524 pts/0 00:00:00 bash

zimbra@pixserver:~$ netstat -luntp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:7780 0.0.0.0:* LISTEN 2493/httpd
tcp 0 0 192.168.0.5:389 0.0.0.0:* LISTEN -
tcp 0 0 127.0.1.1:389 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:10024 0.0.0.0:* LISTEN 2394/amavisd (maste
tcp 0 0 127.0.0.1:7306 0.0.0.0:* LISTEN 1421/mysqld
tcp 0 0 127.0.0.1:7307 0.0.0.0:* LISTEN 1177/mysqld
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:3310 0.0.0.0:* LISTEN 2474/clamd
tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:631 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN -
tcp6 0 0 :::7072 :::* LISTEN -
tcp6 0 0 :::7777 :::* LISTEN -
tcp6 0 0 :::993 :::* LISTEN -
tcp6 0 0 :::9443 :::* LISTEN -
tcp6 0 0 :::995 :::* LISTEN -
tcp6 0 0 :::5222 :::* LISTEN -
tcp6 0 0 :::5223 :::* LISTEN -
tcp6 0 0 :::7335 :::* LISTEN -
tcp6 0 0 :::110 :::* LISTEN -
tcp6 0 0 :::143 :::* LISTEN -
tcp6 0 0 :::7025 :::* LISTEN -
tcp6 0 0 :::5269 :::* LISTEN -
tcp6 0 0 :::53 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 :::631 :::* LISTEN -
tcp6 0 0 :::10015 :::* LISTEN -
tcp6 0 0 :::7071 :::* LISTEN -
udp 0 0 192.168.0.5:137 0.0.0.0:* -
udp 0 0 0.0.0.0:137 0.0.0.0:* -
udp 0 0 192.168.0.5:138 0.0.0.0:* -
udp 0 0 0.0.0.0:138 0.0.0.0:* -
udp 0 0 0.0.0.0:53 0.0.0.0:* -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
udp 0 0 0.0.0.0:39386 0.0.0.0:* -
udp 0 0 0.0.0.0:5353 0.0.0.0:* -
udp 0 0 0.0.0.0:631 0.0.0.0:* -
udp6 0 0 :::53 :::* -

As you can see, there is nothing listening on port 25. Here's the last 20 lines of zimbra.log:

zimbra@pixserver:~$ tail -n 20 /var/log/zimbra.log
Apr 2 07:25:13 pixserver clamd[2474]: Reading databases from /opt/zimbra/data/clamav/db
Apr 2 07:25:16 pixserver clamd[2474]: Database correctly reloaded (538289 signatures)
Apr 2 07:25:26 pixserver zmmailboxdmgr[4144]: status requested
Apr 2 07:25:26 pixserver zmmailboxdmgr[4144]: status OK
Apr 2 07:25:27 pixserver zmmailboxdmgr[4200]: status requested
Apr 2 07:25:27 pixserver zmmailboxdmgr[4200]: status OK
Apr 2 07:25:30 pixserver postfix/postqueue[4237]: fatal: Queue report unavailable - mail system is down
Apr 2 07:26:00 pixserver postfix/postqueue[4244]: fatal: Queue report unavailable - mail system is down
Apr 2 07:26:03 pixserver zmmailboxdmgr[4394]: status requested
Apr 2 07:26:03 pixserver zmmailboxdmgr[4394]: status OK
Apr 2 07:26:04 pixserver zimbramon[4248]: 4248:info: 2009-04-02 07:26:01, STATUS: mail.extremexposures.net: antispam: Running
Apr 2 07:26:04 pixserver zimbramon[4248]: 4248:info: 2009-04-02 07:26:01, STATUS: mail.extremexposures.net: antivirus: Running
Apr 2 07:26:04 pixserver zimbramon[4248]: 4248:info: 2009-04-02 07:26:01, STATUS: mail.extremexposures.net: ldap: Running
Apr 2 07:26:04 pixserver zimbramon[4248]: 4248:info: 2009-04-02 07:26:01, STATUS: mail.extremexposures.net: logger: Running
Apr 2 07:26:04 pixserver zimbramon[4248]: 4248:info: 2009-04-02 07:26:01, STATUS: mail.extremexposures.net: mailbox: Running
Apr 2 07:26:04 pixserver zimbramon[4248]: 4248:info: 2009-04-02 07:26:01, STATUS: mail.extremexposures.net: mta: Running
Apr 2 07:26:04 pixserver zimbramon[4248]: 4248:info: 2009-04-02 07:26:01, STATUS: mail.extremexposures.net: snmp: Running
Apr 2 07:26:04 pixserver zimbramon[4248]: 4248:info: 2009-04-02 07:26:01, STATUS: mail.extremexposures.net: spell: Running
Apr 2 07:26:04 pixserver zimbramon[4248]: 4248:info: 2009-04-02 07:26:01, STATUS: mail.extremexposures.net: stats: Running
Apr 2 07:26:30 pixserver postfix/postqueue[4567]: fatal: Queue report unavailable - mail system is down

As far as Zimbra seems to be concerned - it thinks the MTA is running. I was able to confirm with the user that everything was functioning the night of the 31st. I then applied these patches:

root@pixserver:/home/les# cd /var/cache/apt/archives/
root@pixserver:/var/cache/apt/archives# ls -latr
<deleted extra>
-rw-r--r-- 1 root root 2829630 2009-03-30 19:04 libssl0.9.8_0.9.8g-4ubuntu3.5_i386.deb
-rw-r--r-- 1 root root 385420 2009-03-30 19:04 openssl_0.9.8g-4ubuntu3.5_i386.deb

..and rebooted the system. This is when it stopped sending/receiving mail. The web interfaces all continue to function.

Thanks again for any direction you can give me - it's quite a puzzle to me right now.

[tempusfugit]

See attachment - the message board software doesn't allow me to post all the contents.

[uxbod]

Hmmm, can you post the following please as the update may hosed something on the networking side

cat /etc/hosts
cat /etc/resolv.conf
dig yourdomain mx
dig yourdomain any
host `hostname` <- note backticks and not single quotes
su - zimbra
postfix check

[tempusfugit]

root@pixserver:/home/les# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 mail.extremexposures.net pixserver.localdomain pixserver
192.168.0.5 mail.extremexposures.net mail
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
root@pixserver:/home/les# cat /etc/resolv.conf
nameserver 192.168.0.5
nameserver 192.168.0.1
root@pixserver:/home/les# dig mail.extremexposures.net mx

; <<>> DiG 9.4.2-P2 <<>> mail.extremexposures.net mx
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17951
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;mail.extremexposures.net. IN MX

;; ANSWER SECTION:
mail.extremexposures.net. 0 IN MX 50 mail.extremexposures.net.

;; ADDITIONAL SECTION:
mail.extremexposures.net. 0 IN A 192.168.0.5
mail.extremexposures.net. 0 IN A 127.0.1.1

;; Query time: 1 msec
;; SERVER: 192.168.0.5#53(192.168.0.5)
;; WHEN: Thu Apr 2 09:19:27 2009
;; MSG SIZE rcvd: 114

root@pixserver:/home/les# dig mail.extremexposures.net any

; <<>> DiG 9.4.2-P2 <<>> mail.extremexposures.net any
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46083
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;mail.extremexposures.net. IN ANY

;; ANSWER SECTION:
mail.extremexposures.net. 0 IN A 127.0.1.1
mail.extremexposures.net. 0 IN A 192.168.0.5
mail.extremexposures.net. 0 IN MX 50 mail.extremexposures.net.

;; ADDITIONAL SECTION:
mail.extremexposures.net. 0 IN A 192.168.0.5
mail.extremexposures.net. 0 IN A 127.0.1.1

;; Query time: 0 msec
;; SERVER: 192.168.0.5#53(192.168.0.5)
;; WHEN: Thu Apr 2 09:19:42 2009
;; MSG SIZE rcvd: 146

root@pixserver:/home/les# host `hostname`
pixserver has address 127.0.1.1
pixserver mail is handled by 0 mail.extremexposures.net.
root@pixserver:/home/les# cat /etc/hostname
pixserver
root@pixserver:/home/les# su - zimbra
zimbra@pixserver:~$ postfix check
postfix/postfix-script: warning: not owned by root: /opt/zimbra/postfix-2.4.7.5z/conf/main.cf
postfix/postfix-script: warning: not owned by root: /opt/zimbra/postfix-2.4.7.5z/conf/master.cf
postfix/postfix-script: warning: not owned by root: /opt/zimbra/postfix-2.4.7.5z/conf/master.cf.in

I think I see where they may be going.

[tempusfugit]

I set the permission to rootostfix on those files, but after a zmcontrol stop/start, they reset:
root@pixserver:/opt/zimbra/postfix-2.4.7.5z/conf# su - zimbra
zimbra@pixserver:~$ postfix check
postfix/postfix-script: warning: not owned by root: /opt/zimbra/postfix-2.4.7.5z/conf/main.cf
postfix/postfix-script: warning: not owned by root: /opt/zimbra/postfix-2.4.7.5z/conf/master.cf

[tempusfugit]

-rw-r--r-- 1 zimbra zimbra 2.1K 2009-04-02 09:36 main.cf
-rw-r--r-- 1 root postfix 18K 2008-11-17 20:32 main.cf.default
-rw-r--r-- 1 root postfix 3.4K 2008-11-17 20:32 makedefs.out
-r--r----- 1 zimbra zimbra 5.2K 2009-04-02 09:36 master.cf