Is Bayes working? It's not in headers...

[rfoster]

I'm not getting any contribution from the Bayesian scoring in my received email headers. I've read lots of forum posts, but haven't seen anything to point me in the right direction yet.

I have loaded lots of (several hundred) ham and spam via zmtrainsa. I do get an error message as follows, but since it no longer is learning from this folder, that seems to indicate it had successfully learned earler (right?):

zimbra@mail:$ zmtrainsa bfoster@mydomain.com ham inbox
[] INFO: Total messages processed: 189
netset: illegal network address given: 'mydomain.com'
Learned from 0 message(s) (189 message(s) examined).
netset: illegal network address given: 'mydomain.com'

Here's an email header without any sign of a Bayes score:

Received: from localhost (localhost.localdomain [127.0.0.1])
by mail.mydomain.com (Postfix) with ESMTP id A268416CC5AD
for <bfoster@mydomain.com>; Mon, 22 Oct 2007 13:28:02 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Score: 1.984
X-Spam-Level: *
X-Spam-Status: No, score=1.984 tagged_above=-10 required=6.6
tests=[AWL=-0.320, DNS_FROM_RFC_ABUSE=0.479, DNS_FROM_RFC_WHOIS=0.879,
HTML_10_20=0.945, HTML_MESSAGE=0.001]
Received: from mail.mydomain.com ([127.0.0.1])
by localhost (mail.mydomain.com [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id mv4pL0-5LQIO for <bfoster@mydomain.com>;
Mon, 22 Oct 2007 13:27:58 -0700 (PDT)
Received: from web38903.mail.mud.yahoo.com (web38903.mail.mud.yahoo.com [209.191.125.109])
by mail.mydomain.com (Postfix) with SMTP id AE2C316CC592
for <bfoster@mydomain.com>; Mon, 22 Oct 2007 13:27:56 -0700 (PDT)
Received: (qmail 81127 invoked by uid 60001); 22 Oct 2007 20:28:24 -0000
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
s=s1024; d=yahoo.com;
h=X-YMail-OSG:Receivedate:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
b=v/5PkPEZpWNNHSgT1umO/huHZn3wmUstGmO1HznS4ySemP3U3d7TJiROMNmCDyC5EO9y6kj zxANVp3ACBARTKZqDrUr0p7ol+Q7D7VEukG+TC+qGR4dTO6e3F oVmmYVagUJEvdMf4qof6Nmhe67dnDRchUEzpbjsrbhEx1/w+I0=;
X-YMail-OSG: 3sXRWPwVM1npFt3rzI2NvgHPJ2it2F7o9F7.gi03_dMXSGS0im LJf4dxW_YNhAPx1PTnozNDs8pYoEkmfPqK5zm8ThbE_L.tc3dY gvKNakG57T4-
Received: from [208.200.250.162] by web38903.mail.mud.yahoo.com via HTTP; Mon, 22 Oct 2007 13:28:24 PDT
Date: Mon, 22 Oct 2007 13:28:24 -0700 (PDT)
From: Robert Foster <me@yahoo.com>
Subject: Fwd: ubuntu-security-announce Digest, Vol 37, Issue 8
To: bfoster@mydomain.com
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1845997220-1193084904=:80819"
Content-Transfer-Encoding: 8bit
Message-ID: <612519.80819.qm@web38903.mail.mud.yahoo.com>

--0-1845997220-1193084904=:80819
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit

[dwmtractor]

You are right that your header shows that Bayesian filtering is not happening. That error you're getting in zmtrainsa has nothing to do with it; I believe that is a bug that showed up in 4.5.7 or 4.5.8 but is (AFAIK) harmless. At least I get the same error and it does not harm the action of Bayesian filtering or spam learning on my system.

Check if you have Bayesian filtering turned off. It's on by default, but can be overridden and turned off. Have a look at /opt/zimbra/conf/spamassassin/local.cf

Has this line:

use_bayes 1

got a 0 at the end instead of a 1 (and is it uncommented?). This would turn Bayesian filtering off, and you'll need to change it back to a 1 to get it to run.

The default installation has this line commented out even though it's set to 1; what I don't know is if it's ever been set to 0, whether simply commenting it out is enough to get bayes started again.

In any case, once you've verified that the setting is correct, stop and restart your server (zmcontrol stop and then zmcontrol start as zimbra user) and see if it starts filtering.

[rfoster]

I thought that I should be seeing Bayes scores in the header. Thanks for the confirmation that it isn't working.

Yes, "use_bayes" was set to 1 and it was commented out in /opt/zimbra/conf/spamassassin/local.cf. Probably it's not the problem since the default is 1. For lack of anything better, I uncommented it and restarted spamassassin prior to receiving the mail header I posted. Should I need to restart everything with zmcontrol start/stop?

The spam filters are catching just a small percentage of the spam and I need to do something about it ... One user gets 100+ spams per day in his inbox!

Regards,

Bob

[dwmtractor]

Originally Posted by rfoster

Should I need to restart everything with zmcontrol start/stop?

The honest answer, Bob, is that I'm not sure. I figure for no more time than it takes, I like to have the services started in the order Zimbra is used to having them started, in case a change of pid or something screws it up. Belt & suspenders, I suspect, but I'm that way. . .

Originally Posted by rfoster

The spam filters are catching just a small percentage of the spam and I need to do something about it ...

I would recommend you enable the RBLs to increase the spam detection. Turning them on will catch a great deal of the most egregious junk, bayes or no. The clearest documentation of how to do that is in the Zimbra Administrators' Guide, page 38. Lowering the spam threshold from the default 33% to 29% helped me too (GUI, Global Settings, AS/AV); it changes the raw score that defines a message as spam. . .for me the default of 6.6 points wasn't enough with the scores of the RBLs, but 29% lowered it to 5.8 which, for me, has not increased false positives but significantly lowered false negatives. This wiki article will also give you a variety of useful pointers.

None of this changes your Bayesian problem. You still want to get that fixed, I understand, but a lot of these other fixes will also help. The best Bayesian filter in the world isn't enough to handle the spam problem on its own either.

I see from the announcements section that 4.5.9 has been released. 4.5.6 didn't have trouble with Bayes at least in my experience, and I installed it on Ubuntu LTS, so that shouldn't be the cause of your problems; however, there is a chance that doing the upgrade would fix whatever's broken on your filtering. It's worth a shot. Beyond that, I'm afraid I have to defer to the real gurus!

Good luck,

Dan

[rfoster]

Thanks for the help Dan. I have implemented the RBL's and am planning an upgrade to 4.5.9 this weekend. Hopefully that will get the Bayes straightened out. I will post the result here early next week.

Bob