Improving spam filtering

[KevinH]

Originally Posted by inigoml

We didn't change it. Defaults for zimbra, that is, 6.6/15.
You are right. Lowering to 5.5 could improve recall, but would damage precision...
Anyway this is our first week with all company "zimbraed", we will continue adjusting these values.

For the record we run with 4 so even lower and that works very well. I wonder if you'd get same/better spam protection if you just trained Zimbra a little and then slowly lowered the values.

[jholder]

That's what i did. I subscribed to a SPAM list, and just spent a month training it, rather than having the users train it.

I get about 150 spam e-mails a day, and only about 1 or 2 end up in my inbox.

[claros]

You have english as mother language. Most of the filters have been studied for that language, so works very well. It is very different when you receive the most part of your messages in another language... In my experience, Zimbra SA have failed a lot expecially with phishing messages translted into italian. These messages are not recognized by Clamav as a threat, so simply pass the filter. The great adventage to have a lot of check is a big differece in score between valid messages and spam.

[inigoml]

Originally Posted by KevinH

For the record we run with 4 so even lower and that works very well. I wonder if you'd get same/better spam protection if you just trained Zimbra a little and then slowly lowered the values.

My system has been trained with thousands of spam messages with a set of spam samples, but 4 is very low for us. Too many spam messages would enter without been tagged.
In fact, I rely more in blacklist than in bayesian filters...

This is the spam repository I have used. There are hundred of thousand samples...
http://www.untroubled.org/spam/

Trained via sa-learn (/opt/zimbra/libexec)

[claros]

It' very simple changing amavis config:

put in /opt/zimbra/conf/amavis.conf*

read_hash(\%whitelist_sender, '/etc/zimbra/whitelist');
read_hash(\%blacklist_sender, '/etc/zimbra/blacklist');
read_hash(\%spam_lovers, '/etc/zimbra/spamlovers');

In /etc/zimbra/* put sender address or domain, one per line. Wildcards allowed.
Eg:
hotstuff@sexnzen.com
getassistedgroup.com

A spamlovers list is for that accounts that always need to receive all messages, even if spam. According to rfc 2822 postmaster, abuse and other account of this kind should be spam lovers.

Ciao,
Claudio

[scalper]

Originally Posted by inigoml

My system has been trained with thousands of spam messages with a set of spam samples, but 4 is very low for us. Too many spam messages would enter without been tagged.
In fact, I rely more in blacklist than in bayesian filters...

This is the spam repository I have used. There are hundred of thousand samples...
http://www.untroubled.org/spam/

Trained via sa-learn (/opt/zimbra/libexec)

I have downloaded the spam samples, ran sa-learn.. receive these error..

[zimbra@pluto ~]$ /opt/zimbra/libexec/sa-learn --spam /opt/spam/2006/09/
config: invalid regexp for rule TW_QE: /[-<!\s]\w{0,10}qe[abceghjkmopquwxyz]\w{0,: missing or invalid delimiters
bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /opt/zimbra/.spamassassin/bayes.lock.pluto.cnc.net.my.6233 for /opt/zimbra/.spamassassin/bayes.lock: No such file or directory

is my command incorrect?

tq

[inigoml]

Your command should be something kile this (zmtrainsa inspired):

From your spam folder, create a spam index file:
ls > spam.index

And then, as zimbra user::
sa-learn -p /opt/zimbra/conf/salocal.cf --dbpath ${zimbra_home}/amavisd/.spamassassin/ -L --no-sync --spam -f spam.index --shotdots

[scalper]

thanx! it worked flawlessly.. is there any way to check the sa/bayes database contents? just to make sure that it learned the messages properly..

[Klug]

As zimbra user :
sa-learn --dump magic --dbpath ${zimbra_home}/amavisd/.spamassassin/

[k3rmit]

thanks for the tips, but at the moment i can't understand if the changes in the configuration in conf/spamassassin/local.cf are considered, since the conf file for SA in zimbra si conf/salocal.cf: how can i determine it? Is there a way to see what's loaded as rules at startup?

anyway modifying salocal.cf doesn't work since it's recreated at restart from salocal.cf.in . i wanted to touch as little as possible simbra inner workings and yet customize it.

will w have an administration console interface to handle all amavisd - spamassassin -clamd -freshclam configuration settings? In fact i've modified every freshclam configuration i've found to set the http proxy and make it update definitions but none worked. were am i wrong?

thanks in advance.


albe