Zimbra - mail stuck in "deferred" - antispam / antivirus stopped

[billinvegas]

Hello,

Last night I got a call from the owner, frantic about an e-mail not being sent out (he cc'd his personal account as well)

Came into work (have another question about that) since it's only a few minutes from my house.

Zimbra Admin console > mail queues showed 15 "deferred" e-mails,
Server status showed red x's on antispam and antivirus.

ran "zmcontrol status" as the user zimbra, and it also confirmed antispam and antivirus not running.

Since he was desperate to get his e-mail out, I started and stopped the servers, and restarted them, which seemed to start up the queue.

here's my setup
Release 7.1.1_GA_3196.RHEL5_64_20110527011124 CentOS5_64 FOSS edition.
Dell PowerEdge 830, dual core 3.0 gig, 4 gig RAM, 500 gigs storage
RAID 1.
5 users on the system (+ built-in accounts)

First, can someone please point me to which relevant logs I should check for information regarding this event? I will then post the results in this thread.

Secondly, it would seem that without the antispam / antivirus active, no mail will be sent. Is this accurate? or could there be some other culprit?

More info: at my arrival last night, while a/v a/s were stopped, there was connectivity to / from the server both on the LAN and out to the real world.

thank you all for your help...

[billinvegas]

ok, found the location for the log files
/var/opt/zimbra/log

will go through them once I get a little free time here...

any pointers as to what keywords I should be looking for?

thx

[billinvegas]

searching for "error" brought up this entry, prior to the time that the problem was reported...

a .pdf was attached to the emails that got deferred...

could this be the problem?

# grep error mailbox.log.2011-07-05 | more

2011-07-05 22:43:29,561 WARN  [btpool0-0://10.0.0.1/home/my_username@mydomain.com/Bill-Shared/Atomic_CALL_SHEET_Shoot_7_7.pdf?callback=Zm
PreviewView._errorCallback&fmt=native&view=html] [] misc - native formatter exception
ExceptionId:btpool0-0://10.0.0.1/home/my_username@mydomain.com/Bill-Shared/Atomic_CALL_SHEET_Shoot_7_7.pdf?callback=ZmPreviewView._errorC
allback&fmt=native&view=html:1309931009559:1a21722554032ee6

[billinvegas]

The problem was reported to me around 10:30pm
here's the output of the clamd.log,
there's something in here that appears to be logged about the time the inability to send mail was reported...

I did start/stop the servers at 23:56:58, so I'm assuming that's the
stopped / start entries...

Tue Jul  5 22:06:03 2011 -> SelfCheck: Database status OK.
Tue Jul  5 22:16:03 2011 -> SelfCheck: Database status OK.
Tue Jul  5 22:26:03 2011 -> SelfCheck: Database status OK.
Tue Jul  5 22:32:58 2011 -> Pid file removed.
Tue Jul  5 22:32:58 2011 -> --- Stopped at Tue Jul  5 22:32:58 2011
Tue Jul  5 22:33:22 2011 -> +++ Started at Tue Jul  5 22:33:22 2011
Tue Jul  5 22:33:22 2011 -> clamd daemon 0.97-broken-compiler (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Tue Jul  5 22:33:22 2011 -> Log file size limited to 20971520 bytes.
Tue Jul  5 22:33:22 2011 -> Reading databases from /opt/zimbra/data/clamav/db
Tue Jul  5 22:33:22 2011 -> Not loading PUA signatures.
LibClamAV Warning: Detected duplicate databases /opt/zimbra/data/clamav/db/daily.cvd and /opt/zimbra/data/clamav/db/daily.cld, please manuall
y remove one of them
Tue Jul  5 22:33:30 2011 -> Loaded 846254 signatures.
Tue Jul  5 22:33:32 2011 -> TCP: Bound to address 127.0.0.1 on port 3310
Tue Jul  5 22:33:32 2011 -> TCP: Setting connection queue length to 200
Tue Jul  5 22:33:32 2011 -> Limits: Global size limit set to 10240000 bytes.
Tue Jul  5 22:33:32 2011 -> Limits: File size limit set to 10240000 bytes.
Tue Jul  5 22:33:32 2011 -> Limits: Recursion level limit set to 16.
Tue Jul  5 22:33:32 2011 -> Limits: Files limit set to 10000.
Tue Jul  5 22:33:32 2011 -> Archive support enabled.
Tue Jul  5 22:33:32 2011 -> Archive: Blocking encrypted archives.
Tue Jul  5 22:33:32 2011 -> Algorithmic detection enabled.
Tue Jul  5 22:33:32 2011 -> Portable Executable support enabled.
Tue Jul  5 22:33:32 2011 -> ELF support enabled.
Tue Jul  5 22:33:32 2011 -> Mail files support enabled.
Tue Jul  5 22:33:32 2011 -> OLE2 support enabled.
Tue Jul  5 22:33:32 2011 -> PDF support enabled.
Tue Jul  5 22:33:32 2011 -> HTML support enabled.
Tue Jul  5 22:33:32 2011 -> Self checking every 600 seconds.
Tue Jul  5 22:45:37 2011 -> No stats for Database check - forcing reload
Tue Jul  5 22:45:37 2011 -> Reading databases from /opt/zimbra/data/clamav/db
Tue Jul  5 22:45:45 2011 -> Database correctly reloaded (846254 signatures)
Tue Jul  5 22:59:53 2011 -> SelfCheck: Database status OK.
Tue Jul  5 23:10:30 2011 -> SelfCheck: Database status OK.
Tue Jul  5 23:25:07 2011 -> SelfCheck: Database status OK.
Tue Jul  5 23:39:19 2011 -> SelfCheck: Database status OK.
Tue Jul  5 23:50:39 2011 -> SelfCheck: Database status OK.
Tue Jul  5 23:56:58 2011 -> Pid file removed.
Tue Jul  5 23:56:58 2011 -> --- Stopped at Tue Jul  5 23:56:58 2011
Tue Jul  5 23:59:14 2011 -> +++ Started at Tue Jul  5 23:59:14 2011
Tue Jul  5 23:59:14 2011 -> clamd daemon 0.97-broken-compiler (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Tue Jul  5 23:59:14 2011 -> Log file size limited to 20971520 bytes.
Tue Jul  5 23:59:14 2011 -> Reading databases from /opt/zimbra/data/clamav/db
Tue Jul  5 23:59:14 2011 -> Not loading PUA signatures.
LibClamAV Warning: Detected duplicate databases /opt/zimbra/data/clamav/db/daily.cvd and /opt/zimbra/data/clamav/db/daily.cld, please manuall
y remove one of them
Tue Jul  5 23:59:31 2011 -> Loaded 846254 signatures.
Tue Jul  5 23:59:33 2011 -> TCP: Bound to address 127.0.0.1 on port 3310
Tue Jul  5 23:59:33 2011 -> TCP: Setting connection queue length to 200
Tue Jul  5 23:59:33 2011 -> Limits: Global size limit set to 10240000 bytes.
Tue Jul  5 23:59:33 2011 -> Limits: File size limit set to 10240000 bytes.
Tue Jul  5 23:59:33 2011 -> Limits: Recursion level limit set to 16.
Tue Jul  5 23:59:33 2011 -> Limits: Files limit set to 10000.
Tue Jul  5 23:59:33 2011 -> Archive support enabled.
Tue Jul  5 23:59:33 2011 -> Archive: Blocking encrypted archives.
Tue Jul  5 23:59:33 2011 -> Algorithmic detection enabled.
Tue Jul  5 23:59:33 2011 -> Portable Executable support enabled.
Tue Jul  5 23:59:33 2011 -> ELF support enabled.
Tue Jul  5 23:59:33 2011 -> Mail files support enabled.
Wed Jul  6 00:11:10 2011 -> No stats for Database check - forcing reload
Wed Jul  6 00:11:11 2011 -> Reading databases from /opt/zimbra/data/clamav/db
Wed Jul  6 00:11:18 2011 -> Database correctly reloaded (846254 signatures)

[billinvegas]

Things were running well until last night.
Happened to be at the office at that time, so it wasn't
such a crisis (for the owners)

Antispam stopped, Antivirus stopped...

this information was found in an individual message that was stuck in the "deferred queue"

Content filter: smtp-amavis:[127.0.0.1]10024
Size: 7144
Reason: connect to 127.0.0.1[127.0.0.1]:10024:connection refused

I googled that phrase, and found a post from 2005 on this forum, from the
user andreycheck

Connection refused (port 10024)

"I discovered this can happen if Amavis doesn't understand one's FQDN. I solved it by setting the $myhostname var in /opt/zimbra/conf/amavisd.conf and /opt/zimbra/conf/amavisd.conf.in, as well as myhostname in /opt/zimbra/postfix/conf/main.cf.
-Eric"

here are the variables from the three listed configuration files that
andreycheck mentioned above

/var/opt/zimbra/conf/amavisd.conf

$myhostname = 'mail.mydomainname.com';  # must be a fully-qualified domain name!


/var/opt/zimbra/conf/amavisd.conf.in

$myhostname = '@@zimbra_server_hostname@@';  # must be a fully-qualified domain name!


/opt/zimbra/postfix/conf/main.cf

non_smtpd_milters =
setgid_group = postdrop
alias_maps = hash:/etc/aliases
mydestination = localhost
myhostname = mail.mydomainname.com
message_size_limit = 10240000
recipient_delimiter =
in_flow_delay = 1s

Q: in the above config line
$myhostname = '@@zimbra_server_hostname@@';
it's not "mail.mydomainname.com"
should I change it to reflect that?
(i should remove both the leading and trailing "@" symbols in this file, correct?)


Q: that's the "default", I have never edited that file before...
this service seems to run for several days at a time, but
suddenly seems to stop, at which time all mail is held in deferred queue
(seems to be both send and receive)
why does it seem to function for a whie, and then suddenly stop?
if anyone can point me to where I can check for this, it would be greatly
most appreciated.

I have grepped "amavis" and "amavisd" and "amavis*" in the
/opt/zimbra/log/mailbox.log.(date of error) file, and it returns no results.

grepping "WARN" doesn't return any results regarding amavisd or anti-spam or anti-virus...

any assistance greatly appreciated...

thank you

Bill

[phoenix]

The information in this configuration file is correct and should remain untouched:

/var/opt/zimbra/conf/amavisd.conf.in

$myhostname = '@@zimbra_server_hostname@@';  # must be a fully-qualified domain name!

That file is rewritten to amavisd.conf and the correct value(s) substituted as you can see from the output in your post above.

The following line (on a recent clean install) shows as just the FQDN of my server):

mydestination = localhost

I do have, however, on my normal and upgraded server both 'localhost FQDN' in that entry and both server are working correctly.

I would question why your amavisd.conf files are in a different location to the default install location, is there a specific reason for that?

[billinvegas]

I would question why your amavisd.conf files are in a different location to the default install location, is there a specific reason for that?

I installed ZCS on CentOS 5.6 64-bit, with the appropriate zcs install file for that distro.

I made no modifications to the install, other than ip addresses of the server, dns entries, etc - the required info. Everything else was done via the script.

I would like to know the path to the amavisd.conf in which you are referring.
Can you please specify the path for me where you are expecting the file to reside?

Perhaps the CentOS and it's zcs install use a different directory than
say the Debian based distros?

thank you very much

Bill

[billinvegas]

Also,

still confused as to why amavis / anti-spam / anti-virus would stop.

I'm thinking that it should be logged somewhere, either by Zimbra,
or by CentOS, but I'm still having difficulties in determining which
logfile(s) to check...

thx

[phoenix]

I would like to know the path to the amavisd.conf in which you are referring.

The path you have above is this "/var/opt/zimbra/".

Can you please specify the path for me where you are expecting the file to reside?

All Zimbra files are located in "/opt/zimbra", no other location is used.

Perhaps the CentOS and it's zcs install use a different directory than
say the Debian based distros?

No, it doesn't. All versions of Zimbra, no matter what the distribution, install their files in the same location. I use CentOS myself and there's no circumstances it installs Zimbra in anything than /opt/zimbra - that's on CentOS 5 & 6 on ESXi, bare metal & VM Workstation.

Have you checked the DNS configuration? I'd suggest you go to the Split DNS article and run all the commands in the 'Veify ...' section, post the output here if you want confirmation that it's correct.

[firemike]

Hi billinvegas,

still confused as to why amavis / anti-spam / anti-virus would stop.

Some weeks ago i had a very similar issue:
antispam and anti-virus stopped working suddenly and so the mailqueue was blown up very quickly.
After some exitement i found the following reasen for this behavior:
- In admincosole go to "global config" :: "AS/AV"
- Change thresholds for AS/AV
- Save values
Doing so in my zcs-7.1.0 antivir/antispam stopps working
and needs a manual restart thereafter.

Regards
Mike