All outbound mail is being deferred

[gihrig]

All outbound mail is being deferred

We have had a Zimbra server set up and working since last October. It has been working very well until this morning, all out bound mail is being deferred. Every message sent ends up in the deferred mail queue.

Zimbra release 4.0.2_GA_362.RHEL4
Running on CENTOS
Zimbra is setup under a 'split domain' as discussed in the Zimbra wiki. - We are using our ISP's SMTP and POP servers for external mail delivery and receipt.
We have users who access Zimbra through various clients - all have the same result.

The Zimbra admin UI shows all out bound messages in the 'deferred' queue.
'Requing' those messages seems to work, but they are all deferred again within a few seconds.

A sample from /var/log/zimbra.log show authentication failures when talking to the relay host :

Note: the @'s have been replaced with AT's...

Jun 25 11:36:58 zimbra postfix/smtp[32068]: C89ABA9C80D:
to=<glenATarkadias.com>, relay=marketshareservices.com[72.34.38.4],
delay=1, status=deferred 
(Authentication failed: SASL authentication failed; 
server marketshareservices.com[72.34.38.4] said: 
535 Incorrect authentication data)

I am able set up a local mail client (Thunderbird) and send mail through the relay host. The relay host accepts mail whether TLS is enforced or disabled.
Enabling or disabling TLS in the Zimbra Admin UI has no effect on the error message in the log (or the delivery of mail).

The Server Status screen in the Admin UI shows all services are running.

I have confirmed that the Zimbra server has plenty of free disk space.
I have also rebooted the Zimbra server.

None of this has made any difference in the situation.

marketshareservices.com is our relay host, mail sent to glenATarkadias.com from Thunderbird, via the relay host is delivered.

I am at a loss as to what to do next, any help, suggestions, etc. are sincerely appreciated, the natives are growing restless ;-)

Regards,

-Glen

Full log conversation:

Jun 25 11:36:57 zimbra postfix/smtpd[32059]: connect from zimbra.marketshareservices.com[192.168.0.50]
Jun 25 11:36:57 zimbra postfix/smtpd[32059]: 7064CA9C809: client=zimbra.marketshareservices.com[192.168.0.50]
Jun 25 11:36:57 zimbra postfix/cleanup[32062]: 7064CA9C809: message-id=<18929335.771182796617393.JavaMail.rootATzimbra.marketshareservices.com>
Jun 25 11:36:57 zimbra postfix/qmgr[4672]: 7064CA9C809: from=<techdevATmarketshareservices.com>, size=932, nrcpt=1 (queue active)
Jun 25 11:36:57 zimbra postfix/smtpd[32059]: disconnect from zimbra.marketshareservices.com[192.168.0.50]
Jun 25 11:36:57 zimbra amavis[5275]: (05275-05) ESMTP::10024 /opt/zimbra/amavisd/tmp/amavis-20070625T100147-05275: <techdevATmarketshareservices.com> -> <glenATarkadias.com> SIZE=932 Received: from zimbra.marketshareservices.com ([127.0.0.1]) by localhost (zimbra.marketshareservices.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP for <glenATarkadias.com>; Mon, 25 Jun 2007 11:36:57 -0700 (PDT)
Jun 25 11:36:57 zimbra amavis[5275]: (05275-05) Checking: 9m+tyddYbjiD [192.168.0.50] <techdevATmarketshareservices.com> -> <glenATarkadias.com>
Jun 25 11:36:57 zimbra postfix/smtpd[32067]: connect from localhost.localdomain[127.0.0.1]
Jun 25 11:36:57 zimbra postfix/smtpd[32067]: C89ABA9C80D: client=localhost.localdomain[127.0.0.1]
Jun 25 11:36:57 zimbra postfix/cleanup[32062]: C89ABA9C80D: message-id=<18929335.771182796617393.JavaMail.rootATzimbra.marketshareservices.com>
Jun 25 11:36:57 zimbra postfix/qmgr[4672]: C89ABA9C80D: from=<techdevATmarketshareservices.com>, size=1815, nrcpt=1 (queue active)
Jun 25 11:36:57 zimbra postfix/smtpd[32067]: disconnect from localhost.localdomain[127.0.0.1]
Jun 25 11:36:57 zimbra amavis[5275]: (05275-05) FWD via SMTP: <techdevATmarketshareservices.com> -> <glenATarkadias.com>, 250 2.6.0 Ok, id=05275-05, from MTA([127.0.0.1]:10025): 250 Ok: queued as C89ABA9C80D
Jun 25 11:36:57 zimbra amavis[5275]: (05275-05) Passed CLEAN, LOCAL [192.168.0.50] [192.168.0.50] <techdevATmarketshareservices.com> -> <glenATarkadias.com>, Message-ID: <18929335.771182796617393.JavaMail.rootATzimbra.marketshareservices.com>, mail_id: 9m+tyddYbjiD, Hits: -5.65, queued_as: C89ABA9C80D, 395 ms
Jun 25 11:36:57 zimbra postfix/smtp[32063]: 7064CA9C809: to=<glenATarkadias.com>, relay=127.0.0.1[127.0.0.1], delay=0, status=sent (250 2.6.0 Ok, id=05275-05, from MTA([127.0.0.1]:10025): 250 Ok: queued as C89ABA9C80D)
Jun 25 11:36:57 zimbra postfix/qmgr[4672]: 7064CA9C809: removed
Jun 25 11:36:58 zimbra postfix/smtp[32068]: C89ABA9C80D: to=<glenATarkadias.com>, relay=marketshareservices.com[72.34.38.4], delay=1, status=deferred (Authentication failed: SASL authentication failed; server marketshareservices.com[72.34.38.4] said: 535 Incorrect authentication data)

[gihrig]

Here's an update:

We contacted our ISP to see what their mail server was seeing. They reported:

2007-06-25 15:56:58 fixed_plain authenticator failed for
( zimbra.marketshreservices.com) [72.25.121.211]:36914 
I=[72.34.38.4]: 25: 535 Incorrect authentication data
( set_id=zcollector+marketshareservices.com)

The 'zcollector' account was setup in the original Zimbra install in October 2006 to retrieve mail from the ISP's mail server for all accounts in a single session. That scheme did not work for us, so the zcollector account was abandoned, but never deleted from the Zimbra server, fetchmail, or the ISP's server.

The zcollector account was deleted from all three locations, Friday afternoon while doing maintenance on the Zimbra system. Discussions with users seem to confirm that the probable time of deleting the zcollector account is when out bound mail stopped working.

It appears that the zcollector account was possibly involved in an smtp receive-before-send protocol and that I caused this problem by deleting that account.

I recreated the zcollector account, in all three locations and this seemed to result in a mail loop. Any message sent from the Zimbra server was bounced to the user with a '554 Error: too many hops' message in the mail header.

After deleting the zcollector account followed by a bunch other unsuccessful attempts to glean a little insight, I recreated the zcollector account and the mail loop does not exist. But outbound messages still end up being deferred, just as before.

I'm guessing that I have created a problem between SASL on the Zimbra server and the ISP's SMTP server. Beyond that I'm lost.

Any suggestions of even guesses are sincerely appreciated.

Best,

-Glen