[SOLVED] RBL -- updates

[padraig]

Hi All,

i use the following RBL:
zimbraMtaRestriction: reject_rbl_client dnsbl.njabl.org
zimbraMtaRestriction: reject_rbl_client opm.blitzed.org
zimbraMtaRestriction: reject_rbl_client relays.ordb.org
zimbraMtaRestriction: reject_rbl_client cbl.abuseat.org
zimbraMtaRestriction: reject_rbl_client bl.spamcop.net
zimbraMtaRestriction: reject_rbl_client dnsbl.sorbs.net

and SPAM still leaks thro'

can anybody recommend any extra rbl sites
TIA.
Padraig.

[padraig]

is there any real advantage to adding RBL

does SA use these anyway (/opt/zimbra/conf/spamassassin/20_dnsbl_tests.cf)
in 4.5.6

[phoenix]

Do you mean adding RBLs in general? Spamassasssin does have that check as you've already said and I'm not a big fan of them and have never used them so, my answer would be.... no. There are many users that swear by them so I guess it's a case of YMMV.

I guess you do things like reject_unlisted_recipients and some of the other techniques in the wiki on Improving Anti-Spam System?

[mmorse]

If you still wanted another one I don't see zen.spamhaus.org in your first post.

You really should look at some other options too (that wiki article), if the RBL's aren't up and you're fully depending on them, well you're out of luck.

Host checks:
reject_invalid_hostname
reject_non_fqdn_hostname
reject_non_fqdn_sender

DNS checks:
reject_unknown_client
reject_unknown_hostname
reject_unknown_sender_domain

-Be careful with the reject_unknown_client & reject_unknown_hostname DNS checks, as they can block more than you think sometimes...

You can also change the entry in /opt/zimbra/conf/zmmta.cf for smtpd_reject_unlisted_recipients to 'yes', save the file & then do a 'postfix reload'.

[dwmtractor]

You might actually take a look at (or post for us to take a look at) the spam headers for a couple of the messages that are getting through. There could be some very revealing stuff in them. Two of the worst offenders in my short experience have been something called the auto-whitelist (a negative AWL score in the header) and the bonded sender program (bsp or bondedsender in the header). A negative score from either of these can ruin all the good work you have done tuning your other filters.

The other thing I had to do was to increase the Bayes scores above the defaults--my philosophy being that I don't really care what other people think is a legitimate use of the term "Spam:" if my users think it's spam and they tell my filters it's spam, I'm bloody well gonna treat it as spam unless it comes from (1) my boss, (2) me, or (3) our vendor.

But then I'm an ornery cuss. . .

Cheers!

Dan

[padraig]

thanks for the excellent feedback, here are some samples:

X-Virus-Scanned: amavisd-new at 
X-Spam-Score: 2.384
X-Spam-Level: **
X-Spam-Status: No, score=2.384 tagged_above=-10 required=4
	tests=[BAYES_50=0.001, EXTRA_MPART_TYPE=1.091, HTML_50_60=0.134,
	HTML_IMAGE_ONLY_20=1.157, HTML_MESSAGE=0.001]
Received: from tdev179-177.codetel.net.do (tdev179-177.codetel.net.do [200.88.179.177] (may be forged))
Received: from [200.88.179.177] by mx.corp.mail.ru; Mon, 5 Nov 2007 00:36:38 +0100
Message-ID: <01c81f43$ed2e6610$b1b358c8@news>
From: "Isaac Roman" <news@corp.mail.ru>
Date: Mon, 5 Nov 2007 00:36:38 +0100
MIME-Version: 1.0
Content-Type: multipart/related;
	type="multipart/alternative";
	boundary="----=_NextPart_000_0006_01C81F43.ED2E6610"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1158
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1158
X-Virus-Scanned: by amavisd-new
Subject: [news #30808] Toolbox for a womanizer

From: "Isaac Roman" <news@corp.mail.ru>
Date: Mon, 5 Nov 2007 00:36:38 +0100
MIME-Version: 1.0
Content-Type: multipart/related;
	type="multipart/alternative";
	boundary="----=_NextPart_000_0006_01C81F43.ED2E6610"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1158
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1158
X-Virus-Scanned: by amavisd-new
X-Spam-Score: 3.963
X-Spam-Level: ***
X-Spam-Status: No, score=3.963 tagged_above=-10 required=4 tests=[BAYES_80=2,
	EXTRA_MPART_TYPE=1.091, HTML_30_40=0.374, HTML_IMAGE_ONLY_16=0.497,
	HTML_MESSAGE=0.001]
This is a multi-part message in MIME format.
Content-Transfer-Encoding: base64

Received: from localhost (localhost.localdomain [127.0.0.1])
X-Virus-Scanned: amavisd-new at 
X-Spam-Score: 0.001
X-Spam-Level: 
X-Spam-Status: No, score=0.001 tagged_above=-10 required=4
	tests=[BAYES_50=0.001]
Received: ([127.0.0.1])
	by localhost [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id 3kl2Y+oyXPva for <>;
	Fri,  2 Nov 2007 13:22:28 +0000 (GMT)
Received: by i (Postfix, from userid 101)
	id BEB9C1729419; Fri,  2 Nov 2007 13:22:28 +0000 (GMT)
Received: from (Postfix) with ESMTP id 9C44E17293DC
	for <>; Fri,  2 Nov 2007 13:22:28 +0000 (GMT)
Received: from localhost (adsl-218-211-17-69.NH.dynamic.sparqnet.net [218.211.17.69] (may be forged))
Message-ID: <000001c81d52$c9c95d80$0100007f@localhost>
From: "Susumu Weber" <unwarlikeness@siobhangraham.com>
Subject: Mlcrosoft W|ndows Sof+ware for $2O
Date: Fri, 02 Nov 2007 21:22:06 +0800
Content-Type: text/plain;
    charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook, Build 10.0.3416
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.150
X-Virus-Scanned: by amavisd-new

V!sit realnewsoft . com

i currently use the av/as settings 66/20

zmprov gacf | grep zimbraMtaRestriction
zimbraMtaRestriction: reject_invalid_hostname
zimbraMtaRestriction: reject_unknown_sender_domain
zimbraMtaRestriction: reject_non_fqdn_sender
zimbraMtaRestriction: reject_unknown_client
zimbraMtaRestriction: reject_rbl_client dnsbl.njabl.org
zimbraMtaRestriction: reject_rbl_client opm.blitzed.org
zimbraMtaRestriction: reject_rbl_client relays.ordb.org
zimbraMtaRestriction: reject_rbl_client cbl.abuseat.org
zimbraMtaRestriction: reject_rbl_client bl.spamcop.net
zimbraMtaRestriction: reject_rbl_client dnsbl.sorbs.net
zimbraMtaRestriction: reject_rbl_client zen.spamhaus.org

[dwmtractor]

Padraig,

I see you have the RBLs enabled, so these messages must be coming from non-RBLed sources. Have you noticed any other messages that ARE getting an RBL score? (perhaps ones that actually DID make it into your junk folders?)

The biggest things I'm seeing in this sample are

(1) the BAYES scores of 50% to 80% mean that the Bayesian filters are not identifying them as spam. When you train the Bayesian filters more effectively, you'll see these messages getting a BAYES_99 score, which is the highest you can get. You'll need to run zmtrainsa on some known and trusted spam and ham folders to get enough data for the filter to perform more effectively.

(2) Even with a well-trained BAYES filter you may or may not catch the spam with your present settings, at least until you increase the point value for strong Bayes hits. You may want to increase the Bayes scores for 80, 95, and 99%

(3) You have lowered your tag threshold significantly, since the required point value for spam is only 4 points. This may actually be too low and result in messages that you want, being tagged as junk. Your actual mileage may vary, of course, but you may find you want to raise that value a little higher than 20 and then just raise the point value of either your RBLs or Bayes or both. It's somewhat a question of surgical targetting vs. nuking. . .

But I think your biggest issue may in fact be that your Bayesian database hasn't had much training. . .it is hard for me to believe that a "toolbox for a womanizer" from a Russian source isn't a strong hit for BOTH Bayes and the RBLs.

Which brings up my other question; you might try your zmprov gacf | grep zimbraMtaRestriction again and see if your RBLs are still active. My own server inexplicably blows them away sometimes (I'm gonna file a separate thread on this, but it's at least in part related to bug 8146).

[padraig]

Thanks dwmtractor,
ran /opt/zimbra/bin/zmtrainsa user@domain.com spam folderName
from CLI zmtrainsa - Zimbra :: Wiki

manually & learned 30 messages from 34.

i see zmtrainsa in in the zimbra crontab
0 23 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1

does this mean the system would learn these anyway

TIA

[dwmtractor]

Originally Posted by padraig

Thanks dwmtractor,
ran /opt/zimbra/bin/zmtrainsa user@domain.com spam folderName
from CLI zmtrainsa - Zimbra :: Wiki

manually & learned 30 messages from 34.

i see zmtrainsa in in the zimbra crontab
0 23 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1

does this mean the system would learn these anyway

TIA

Yes and no. According to everything I've read on these forums, if you drag a message into your junk folder using an IMAP client, it will never hit spam training. So whatever the cron'ed version of zmtrainsa is doing, it is apparently not that (although I have never understood why it couldn't).

Any message you mark as junk using your webclient will be used to train your filters. However, any message that gets to the junk folder through other means (it gets a high enough score on the RBLs for example) is not going to influence your Bayesian filters at all. The only other way to train the filters is to forward the spam messages AS ATTACHMENTS to your automatically-created spam training account. This is the only way for POP clients.

Did you (at the time of setup or since) also train your filters with some ham? Conventional wisdom is that you need to have trained the system with at least 200 messages of each spam and ham before the filters have enough to go on to really make a difference (in my case that was easy, I have two users who between them get over 300 spam messages a day). How long has your server (with spam filtering activated) been operational?

The most important messages for you to get into your Bayesian filters, of course, are any that are not getting recognized as spam anyway. Be sure that your users know to either forward these false negatives to the spam training account, or put them in a folder upon which you can run zmtrainsa, NOT just delete them. In my installations it only took a couple of days to get reliably-trained filters by these methods.

Dan

[phoenix]

Don't forget that DSPAM is disabled in recent versions of Zimbra, you need to manually enable it.