[SOLVED] RBL -- updates

[dwmtractor]

Originally Posted by phoenix

Don't forget that DSPAM is disabled in recent versions of Zimbra, you need to manually enable it.

Bill, I have never used DSPAM. What of these features (if any) are dependent on it? Or is it just one more set of features to add to all the rest?

Thanks

Dan

[mmorse]

zmtrainsa just goes through the spam/ham accounts
Anything in junk would already be trained...so to fix what happens when you move to junk:
Bug 9532 - IMAP/Outlook move to junk doesn't train anti-spam (RFE ended up covering imap & web-ui moves I think)

Originally Posted by mmorse

yup dspam in 4.0.3 had some faults & so did 4.0.5 so it was finally turned off for 4.5.1
Bug 12185 - disable dspam by default
Bug 12079 - turn down or turn off dspam logging
Bug 13962 - DSPAM Blocks devliery when enabled

The moderator JoshuaPrismon (aka Lostknight) was the one who influenced dspam being included in the first place - it was some excellent work for for those who remember (all the way back to dspam 3.6.1)
josh, you been keeping abreast of the dspam world lately?

well i meant 'a long time ago' because zcs4.5.9 actually only has 3.6.2
though I do not know it's status for v5Beta4, but after one year gap we had a new dspam version:
3.6.8 - June 7, 2006
3.8.0 - March 19, 2007
His original directions are here: Using DSPAM for Spam Filtering - Zimbra :: Wiki
Though now that it's built-in, you can enable/disable via a line in amavisd.conf.in Though you may feel more comfortable waiting till it's updated in the zcs releases/enabled by default. If you choose to enable it monitor closely, but I'm sure you'll have no problem with that
Also see: Nuclear Elephant: The DSPAM Project

[dwmtractor]

As you already know, Mike, I'm actually having excellent success with a combination of RBLs, Bayes with the scores enhanced, and tweaks to get rid of the negative effects of certain whitelisting agencies I don't trust.

What advantage(s) does DSPAM bring to the table in comparison to these other entities?

[phoenix]

Originally Posted by dwmtractor

Bill, I have never used DSPAM. What of these features (if any) are dependent on it? Or is it just one more set of features to add to all the rest?

It's 'just another' anti-spam tool really but I think it's far superior to spamassassin and I was quite keen to implement it in the early days. The best description is this (from the DSPAM freshmeat page):

DSPAM is a server-side statistical anti-spam agent for Unix email servers. It masquerades as the email server's local delivery agent and effectively filters spam using a combination of de-obfuscation techniques, specialized algorithms, and statistical analysis. The result is an administratively maintenance-free, self-learning anti-spam tool. DSPAM has yielded real-world success rates beyond 99.9% accuracy with less than a 0.01% chance of false positives.

[mmorse]

Same feeling, but I currently do not enable it for the people I support because I can't always monitor their system's as closely/react as fast if there are problems or something.

To enable:

uncomment a line in amavisd.conf.in
#$dspam = '/opt/zimbra/dspam/bin/dspam';

restart zimbra or as/av

check the crontab -e to see if this got uncommented (if not uncomment)
-it's the training
# Dspam cleanup
#
#0 1 * * * find /opt/zimbra/dspam/var/dspam/data/z/i/zimbra/zimbra.sig/ -type f -name \*sig -mtime +7 -exec rm {} \; > /dev/null 2>&1
#8 4 * * * /opt/zimbra/dspam/bin/dspam_logrotate -a 60 /opt/zimbra/dspam/var/dspam/system.log
#8 8 * * * /opt/zimbra/dspam/bin/dspam_logrotate -a 60 /opt/zimbra/dspam/var/dspam/data/z/i/zimbra/zimbra.log

Originally Posted by hlribeiro

If you zimbra version is <= 4.5.5, you will also need to uncomment the dspam lines in /opt/zimbra/bin/zmtrainsa. The new zmtrainsa script in 4.5.6 have a bug and doesn't learn from dspam.

Bug 18153 - zmtrainsa not learning from dspam when it's enabled

a comment on zmtrainsa script lines 45 and 54 was enough for me for the cron jobs to work. Unafortunely, didn't work for manual training

that might be 35/45 now though i'm not sure

[padraig]

Originally Posted by dwmtractor

Yes and no. According to everything I've read on these forums, if you drag a message into your junk folder using an IMAP client, it will never hit spam training. So whatever the cron'ed version of zmtrainsa is doing, it is apparently not that (although I have never understood why it couldn't).

did a quick & dirty script to do this could be cron'ed:

su - zimbra -c 'zmprov gaa' | grep -v ham | grep -v spam | grep -v wiki> user_list.txt
rm -f update_spam.sh
touch update_spam.sh
chmod u+x update_spam.sh

for SEARCH_ACCOUNT in `cat user_list.txt`;
do

    echo 'su - zimbra -c '\''zmtrainsa '$SEARCH_ACCOUNT' spam junk'\' >> update_spam.sh

done

./update_spam.sh

Originally Posted by dwmtractor

Any message you mark as junk using your webclient will be used to train your filters. However, any message that gets to the junk folder through other means (it gets a high enough score on the RBLs for example) is not going to influence your Bayesian filters at all. The only other way to train the filters is to forward the spam messages AS ATTACHMENTS to your automatically-created spam training account. This is the only way for POP clients.

found an excellent article on how this works sa-learn - train SpamAssassin's Bayesian classifier

Originally Posted by dwmtractor

Did you (at the time of setup or since) also train your filters with some ham? Conventional wisdom is that you need to have trained the system with at least 200 messages of each spam and ham before the filters have enough to go on to really make a difference (in my case that was easy, I have two users who between them get over 300 spam messages a day). How long has your server (with spam filtering activated) been operational?

just used the default "JUNK"/"NOT JUNK" icons will train on accounts now

[padraig]

Originally Posted by dwmtractor

As you already know, Mike, I'm actually having excellent success with a combination of RBLs, Bayes with the scores enhanced, and tweaks to get rid of the negative effects of certain whitelisting agencies I don't trust.

How do you create your own whitelist/blacklist & adjust the bayes scores
Thanks Again,
Padraig.

[dwmtractor]

Originally Posted by padraig

How do you create your own whitelist/blacklist & adjust the bayes scores
Thanks Again,
Padraig.

I have not done a white/blacklist of my own; thought I was going to have to, but once I adjusted my Bayes scores to higher defaults, (such that if Bayes thinks it's spam that's enough to call it spam), I didn't need to. I did this by appending the following lines to /opt/zimbra/conf/spamassassin/local.cf:

#My tweaks to the Bayes scoring system - DWM
score BAYES_00 0.0001 0.0001 -2.312 -2.599
score BAYES_05 0.0001 0.0001 -1.110 -1.110
score BAYES_20 0.0001 0.0001 -0.740 -0.740
score BAYES_40 0.0001 0.0001 -0.185 -0.185
score BAYES_50 0.0001 0.0001 0.001 0.001
score BAYES_60 0.0001 0.0001 1.0 1.0
score BAYES_80 0.0001 0.0001 2.5 2.5
score BAYES_95 0.0001 0.0001 5.5 5.5
score BAYES_99 0.0001 0.0001 6.5 6.5

After that, most of my messages were getting recognized properly, but I found two more irritations that were adding NEGATIVE scores to stuff I'd already classified as junk. One was the so-called Auto-whitelist (AWL), and the other was the Bonded Sender Program, or BSP, now known as ReturnPath's SenderScoreCertified program. Both of them were causing me to get crap in my inbox that I or my users didn't want to see and had classed as spam. . .BAYES_99 scoring in the header. So, I added the following to local.cf:

# Score to eliminate Bonded Sender Program (BSP) whitelisting
score RCVD_IN_BSP_TRUSTED 0
score RCVD_IN_BSP_OTHER 0
score RCVD_IN_BONDEDSENDER 0

# Turn off 'Auto-Whitelist'
use_auto_whitelist 0

Presto, changeo, my inbox is clean!

by the way, for those who want to read more, these subjects are discussed (along with many others) in the "Improving Anti-spam system" wiki.

[dwmtractor]

Originally Posted by padraig

did a quick & dirty script to do this could be cron'ed:

Your mileage may vary, but in my experience so far, I have very little need to train anything more as ham. My problem is false negatives, not false positives. The very LAST thing I want is for any automated process, because it's not quite right, to over-train my system with bad stuff that is mistakenly recognized as ham. So unless you're having a lot of false-positive problems (that can't be addressed more effectively with a little score-tweaking) I would stay away from automated ham analysis altogether.

[padraig]

thanks Dan, what is your opinion on automated spam analysis
or does the same as ham apply