Zimbra has started deferring all mail

[pwd]

I am testing Open Source 5.0 zimbra and it was working fine for a few weeks but
when I started working on backup (using a perl script I found in the wiki) all mail is
being deferred. I have restarted zimbra a number of times and have turned off AV
I have plenty of space in my tmpfs device (700M plus 0 used). I only have a single user on and have a split domain with zimbra the slave (is-smtp-1 is the smtp server that has aliases for both post office systems, one running zimbra the other (production) running courier.

the following is the log for a single message)

thanks


Feb 6 11:17:13 is-mail-2 postfix/smtpd[8245]: 2050A3184E9: client=is-smtp-1.mdtsoft.com[209.101.192.25]
Feb 6 11:17:13 is-mail-2 postfix/cleanup[8265]: 2050A3184E9: message-id=<937d27e10802060813o6934561jc0a1d3e428d9ad81@ma il.gmail.com>
Feb 6 11:17:13 is-mail-2 postfix/qmgr[7629]: 2050A3184E9: from=<pgsql-hackers-owner+m113708=pwd=mdtsoft.com@postgresql.org>, size=4157, nrcp
t=1 (queue active)
Feb 6 11:17:13 is-mail-2 postfix/qmgr[7629]: 2050A3184E9: to=<pwd@mdtsoft.com>, orig_to=<pwd@is-mail-2.mdtsoft.com>, relay=none, delay=0.13,
delays=0.12/0.01/0/0, dsn=4.4.2, status=deferred (delivery temporarily suspended: lost connection with 127.0.0.1[127.0.0.1] while sending DA
TA command)
Feb 6 11:17:13 is-mail-2 postfix/smtpd[8245]: disconnect from is-smtp-1.mdtsoft.com[209.101.192.25]
Feb 6 11:17:53 is-mail-2 postfix/smtpd[8245]: connect from is-smtp-1.mdtsoft.com[209.101.192.25]
Feb 6 11:17:53 is-mail-2 postfix/smtpd[8245]: setting up TLS connection from is-smtp-1.mdtsoft.com[209.101.192.25]
Feb 6 11:17:53 is-mail-2 postfix/smtpd[8245]: TLS connection established from is-smtp-1.mdtsoft.com[209.101.192.25]: TLSv1 with cipher DHE-R
SA-AES256-SHA (256/256 bits)
Feb 6 11:17:53 is-mail-2 postfix/smtpd[8245]: 44E0E3184EB: client=is-smtp-1.mdtsoft.com[209.101.192.25]
Feb 6 11:17:53 is-mail-2 postfix/cleanup[8265]: 44E0E3184EB: message-id=<200802061627.m16GR63D024793@is-smtp-1.mdtsoft.com>
Feb 6 11:17:53 is-mail-2 postfix/qmgr[7629]: 44E0E3184EB: from=<nobody@is-mail-2.mdtsoft.com>, size=1302, nrcpt=1 (queue active)
Feb 6 11:17:53 is-mail-2 postfix/qmgr[7629]: warning: connect to transport retry: No such file or directory
Feb 6 11:17:53 is-mail-2 postfix/qmgr[7629]: 44E0E3184EB: to=<pwd@mdtsoft.com>, orig_to=<pwd@is-mail-2.mdtsoft.com>, relay=none, delay=0.08,
delays=0.07/0.01/0/0, dsn=4.3.0, status=deferred (mail transport unavailable)
Feb 6 11:17:53 is-mail-2 postfix/smtpd[8245]: disconnect from is-smtp-1.mdtsoft.com[209.101.192.25]
Feb 6 11:18:04 is-mail-2 zmmailboxdmgr[9253]: status requested
Feb 6 11:18:04 is-mail-2 zmmailboxdmgr[9253]: status OK

[mmorse]

Let's see the output of:
host `hostname` <--backticks not single quotes
dig any mdtsoft.com
your /etc/hosts file
zmprov ga pwd@mdtsoft.com | grep Transport

[pwd]

/etc/hosts

[root@is-mail-2 log]# more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
209.101.192.49 is-mail-2.mdtsoft.com is-mail-2
[root@is-mail-2 log]#


[root@is-mail-2 log]# host `hostname`
is-mail-2.mdtsoft.com has address 209.101.192.49
[root@is-mail-2 log]#

[root@is-mail-2 log]# dig any mdtsoft.com

; <<>> DiG 9.4.2 <<>> any mdtsoft.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12837
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 6

;; QUESTION SECTION:
;mdtsoft.com. IN ANY

;; ANSWER SECTION:
mdtsoft.com. 86400 IN SOA is-net-2.mdtsoft.com. is.mdtsoft.com. 863 1200 240 1209600 3600
mdtsoft.com. 86400 IN NS is-net-1.mdtsoft.com.
mdtsoft.com. 86400 IN NS is-net-2.mdtsoft.com.
mdtsoft.com. 86400 IN NS guam.mdtsoft.com.
mdtsoft.com. 86400 IN NS spock.mdtsoft.com.
mdtsoft.com. 86400 IN MX 10 is-smtp-2.mdtsoft.com.
mdtsoft.com. 86400 IN MX 10 is-smtp-1.mdtsoft.com.

;; ADDITIONAL SECTION:
guam.mdtsoft.com. 86400 IN A 209.101.192.2
is-net-1.mdtsoft.com. 86400 IN A 209.101.192.169
is-net-2.mdtsoft.com. 86400 IN A 209.101.192.168
spock.mdtsoft.com. 86400 IN A 209.101.192.158
is-smtp-1.mdtsoft.com. 86400 IN A 209.101.192.25
is-smtp-2.mdtsoft.com. 86400 IN A 209.101.192.119

;; Query time: 2 msec
;; SERVER: 209.101.192.169#53(209.101.192.169)
;; WHEN: Thu Feb 7 06:39:44 2008
;; MSG SIZE rcvd: 301



NOTE I am (and it was working like a charm) running a "split domain" is-smtp-* are
ONLY smtp servers and run canit anti spam. They are set up to forward to the two
(production, courier and testing, zimbra is-mail-2) server by user name (right now
only 1 user on is-mail-2 and mail is also sent to the courier server).

I have zimbra set to always relay to is-smtp-1

Thanks

P.S. all of these commands where run ON is-mail-2 by the user root. is-mail-2 is a
Fedora 7 (Moonshine) with yum -y update done just before the install of zimbra (version 5.0.0 GR Open Source)

[root@is-mail-2 log]# rpm -qi zimbra-core
Name : zimbra-core Relocations: (not relocatable)
Version : 5.0.0_GA_1869.F7 Vendor: Zimbra, Inc.
Release : 20071218205447 Build Date: Wed 19 Dec 2007 12:01:54 AM EST
Install Date: Wed 09 Jan 2008 08:34:50 AM EST Build Host: build20.re.zimbra.com
Group : Applications/Messaging Source RPM: zimbra-core-5.0.0_GA_1869.F7-20071218205447.src.rpm
Size : 270725388 License: Various
Signature : (none)
Packager : Zimbra, Inc.
URL : http://www.zimbra.com
Summary : Zimbra Core
Description :
Best email money can buy


Hope this helps

[mmorse]

Can you ping/telnet between these ports?
Just wanted to double check my understanding of the setup: Are is-smtp-1 & is-smtp-2 tied into a multi server setup with is-mail-2 -if those 3 are zimbra boxes what's the mail transport?
You haven't said it but looking at your dns records would suggest that smtp1/2 are sendmail boxes?
From above:
mail1 courier
mail2 zimbra
While you have an MX record, try out Pingability.com: Web Site Monitoring and Alert Service it looks to me that you're missing an external A record on your main domain. (www. mdtsoft.com is just a CNAME record pointing to fiji.mdtsoft.com)
Said 5.0 what's the full version string? (zmcontrol -v)

[pwd]

I am running zimbra as a secondary at least if I understand primary vs. secondary I am

I won't try to draw but

1) I have two servers that are the external MX records that do two things
a) antispam
b) route mail to the correct mailbox server systems (some *@mdtsoft.com people
get there mail on external servers.

2) I have two internal post office server 1 is a courier server that we have been
running for years (this is the target to be replaced by zimbra0
second a test server running zimbra, I have one user (myself) who gets his
mail on both server (for now). I have a system to provision the smtp servers (and the
courier systems will be upgraded to provision zimbra) that sets up the necessary
stuff to get the mail where it belongs.

3) I have set up zimbra (and to always forward all outgoing mail to is-smtp-1 (this also appears not to work now but it did)



Anyway here is the output of zmprov.....

[root@is-mail-2 pwd]# su - zimbra
[zimbra@is-mail-2 ~]$ zmprov ga pwd@mdtsoft.com | grep Transport
zimbraMailTransport: lmtp:is-mail-2.mdtsoft.com:7025
[zimbra@is-mail-2 ~]$

[pwd]

This is a single server setup. The mail is getting to is-mail-1 (is-smtp-1 is running sendmail and anti spam only) (from the log) it appears to me that the message is being
received without problems by the zimbra server but for some reason it can't get it to the
correct mailbox now. (the sendmail log on is-smtp-1 bears this out)

also the messages are building up in the deferred que.

[pwd]

As it look safe I did the following as root

su - zimbra

zmcontrol stop

^D

/opt/zimbra/libexec/zmfixperms

su - zimbra

zmcontrol start

(running zmfixperms as the user zimbra gave a bunch of errors so I did as root (see above)

This did not fix the problem

[pwd]

On is-mail-2 as root I did the following

[root@is-mail-2 log]# telnet localhost 7025
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 is-mail-2.mdtsoft.com Zimbra LMTP ready
HELO
500 5.5.2 Syntax error
quit
221 is-mail-2.mdtsoft.com closing connection
Connection closed by foreign host.
[root@is-mail-2 log]# telnet is-mail-2.mdtsoft.com 7025
Trying 209.101.192.49...
Connected to is-mail-2.mdtsoft.com.
Escape character is '^]'.
220 is-mail-2.mdtsoft.com Zimbra LMTP ready
quit
221 is-mail-2.mdtsoft.com closing connection
Connection closed by foreign host.
[root@is-mail-2 log]#

-----------------------------------------
So it looks like it is at least talking (as expected from the logs)

[bobby]

what does zmcontrol status show? Is there anything in the /opt/zimbra/log/mailbox.log when the mta is trying to deliver a message to your mailbox?

[pwd]

[pwd@is-mail-2 ~]$ su
Password:
[root@is-mail-2 pwd]# su zimbra
[zimbra@is-mail-2 pwd]$ zmcontrol status
Host is-mail-2.mdtsoft.com
antispam Running
ldap Running
logger Running
mailbox Running
mta Running
snmp Running
spell Running
stats Running
[zimbra@is-mail-2 pwd]$


There is nothing in the /opt/zimbra/log/mailbox.log at the time of the failed delevery
the only thing that is there is the following (the whole file as of 10:43 EST Sat

[root@is-mail-2 log]# more /opt/zimbra/log/mailbox.log
2008-02-09 00:20:51,145 INFO [Timer-3] [] SessionCache - Removed 1 idle sessions (SOAP). 1 active sessions remain.
2008-02-09 03:41:00,897 INFO [btpool0-12] [name=pwd@mdtsoft.com;mid=3;ip=209.101.192.181;] mailbox - Created folder /Corespo
ndents/t-b-o-h.net, id=33158
2008-02-09 03:50:51,165 INFO [Timer-3] [] SessionCache - Removed 1 idle sessions (SOAP). 1 active sessions remain.
2008-02-09 04:20:51,166 INFO [Timer-3] [] SessionCache - Removed 1 idle sessions (SOAP). 1 active sessions remain.
2008-02-09 08:20:51,191 INFO [Timer-3] [] SessionCache - Removed 1 idle sessions (SOAP). 1 active sessions remain.


(the created folder is from a script that has been running (via cron) from before the mail deferrment started.