Improving spam filtering

[inigoml]

Hi.

For the last three weeks, my IT team has been deploying a new Zimbra NE in our network for a 300 users group.
In general, people has appreciated the improvement over our old cyrus+postfix+squirrellmail system, but we have experienced several complaints about spam filtering. We used spamassassin, but with zimbra our "hunting rate" has been specially low.

So, we have started working with zimbra spamassasin in order to improve it. This is what we have done:

1- First of all, zimbra spamassassin has no SPF enabled. Since perl enviroment is system way integrated, adding SPF support is fair simple. If you use RedHat or CentOS, you can "yum-it" from dag.wieers.com repositories by executing
> yum install perl-Mail-SPF-Query

2- Second, we added Razor2 in order to improve score.
As root:
Get razor-agents-sdk from razor.sourceforge.net, untar it and
>perl Makefile.PL
>make
>make install

Get also razor-agents from razor.sourceforge.net, untar it and
>perl Makefile.PL
>make
>make install

Open your firewall ports for razor2 (TCP/2703 outgoing).
Create .razor folder in /opt/zimbra and give zimbra user permissions
>mkdir /opt/zimbra/.razor; chown -Rf zimbra:zimbra /opt/zimbra/.razor

As zimbra user, create your razor account:
>razor-admin -create
>razor-admin -discover
>razor-admin -register

3- Now we are going to add pyzor support for increase (again) spam score
As root, install python support
>yum install python

Get pyzor package from pyzor.sourceforge.net, untar it and:
>python setup.py build
>python setup.py install

Set perms according with pyzor readme.
>chmod -R a+rX /usr/share/doc/pyzor \ >/usr/lib/python2.3/site-packages/pyzor \
>/usr/bin/pyzor /usr/bin/pyzord

Create .pyzor folder into zimbra home and set perms
>mkdir /opt/zimbra/.pyzor; chown zimbra:zimbra /opt/zimbra/.pyzor

Open your firewall ports for pyzor (TCP/24441 outgoing)

And ready to go, as zimbra user, with:
>pyzor discover

4- Now we have PYZOR + RAZOR + SPF. But it would be advisable to enable it and give SPF a higher score. Those admins with wrong SPF entries should be punished since it's not mandatory and so, if you enable it, do it well.
So open your spamassassin config at
/opt/zimbra/conf/spamassassin/local.cf and add this rules at the end (customize it at your own):

----
ok_languages en es
ok_locales en es
trusted_networks 127. 10.70. 192.168.
use_bayes 1
skip_rbl_checks 0
use_razor2 1
#use_dcc 1 <<< WORK IN PROGRESS
use_pyzor 1

dns_available yes

## Optional Score Increases
score DCC_CHECK 4.000
score SPF_FAIL 10.000
score SPF_HELO_FAIL 10.000
score SPF_SOFTFAIL 10.000
score RAZOR2_CHECK 2.500
score PYZOR_CHECK 2.500
score BAYES_99 4.300
score BAYES_90 3.500
score BAYES_80 3.000

bayes_ignore_header Received: from mail3.vectorsf.com
bayes_ignore_header Received: from localhost
bayes_ignore_header Received: from mail1.vectorsf.com
bayes_ignore_header Received: from mail2.vectorsf.com
-----

5- Some notes about this:
In zimbra, by default, spam with 15 score of higher is discarded by amavisd. If you want your user receive these mails, you have to modify amavisd.conf settings (/opt/zimbra/conf/amavisd.conf) in order to pass this email.

[claros]

Originally Posted by inigoml

So open your spamassassin config at
/opt/zimbra/conf/spamassassin/local.cf and add this rules at the end (customize it at your own):

----
ok_languages en es
ok_locales en es
trusted_networks 127. 10.70. 192.168.
use_bayes 1
skip_rbl_checks 0
use_razor2 1
#use_dcc 1 <<< WORK IN PROGRESS
use_pyzor 1

dns_available yes

Nice post, I'd like to contribute.

At least with Zimbra Version 4.xx, you should also edit
/opt/zimbra/conf/spamassassin/v310.pre
to uncomment the loadplugin config in this way

# Razor2 - perform Razor2 message checks.
#
# Razor2 is disabled here because it is not available for unlimited free
# use.  It is currently free for personal use, subject to capacity
# constraints.  See the Cloudmark SpamNet Service Policy for more details.
#
loadplugin Mail::SpamAssassin::Plugin::Razor2

I've the same setup, including DCC e RulesdoJour
I think you increased a lot your spam score values, do you have false positive?
We too have an amavis setup to pass spam with
$final_spam_destiny = D_PASS


To setup DCC:
Download dcc from DCC Site

I compile on different system to build an rpm to install in production environment.
Use the attached spec file (rename it to .spec) to build an rpm with the command:
rpmbuild -ba /usr/src/redhat/SPECS/dcc.spec

install it on the production server:
rpm -ivh dcc-x.y.z.rpm

Change /etc/dcc/dcc_conf to read:

DCCUID=zimbra
DCCD_ENABLE=off

Change /opt/zimbra/conf/spamassassin/v310.pre to enable the DCC plugin:

# DCC - perform DCC message checks.
#
# DCC is disabled here because it is not open source.  See the DCC
# license for more details.
#
loadplugin Mail::SpamAssassin::Plugin::DCC

Have fun.
I use sqlgrey as greylist server, so I don't need another one. As to me the standard value DCC == 2.5 Spamassassin point is ok, so I do not change it.
With SA 3.xx you do not need to use enable_dcc il local.cf. That's the same for razor2 indeed...

I think we should prepare a script to save and restore this config changes upon zimbra updates...

Ciao

[inigoml]

Thank you, claros!
I forgot to comment razor2 module activation.

By the way, your rpm .spec for DCC is very useful for me. In fact, I'm gonna include DCC right now.

I think these modifications should be included in zimbra default configuration... at least SPF and pyzor with some options in the admin interface...

[claros]

Originally Posted by inigoml

Thank you, claros!
I forgot to comment razor2 module activation.

By the way, your rpm .spec for DCC is very useful for me. In fact, I'm gonna include DCC right now.

I think these modifications should be included in zimbra default configuration... at least SPF and pyzor with some options in the admin interface...

I do agree.
And I'd like a Whitelist/Blacklist administration in the admin interface too. Actually I've enabled it in amavis as a textfile...

We should just start a new wiki page with how-to, then submit a new feature request in bugzilla and let the people vote for it

Ciao

[inigoml]

Uploaded to wiki, improved with claros comments and fixed some errors.

http://wiki.zimbra.com/index.php?tit...ti-spam_system

[KevinH]

Nice post and thanks for putting this in the Wiki. Did you change your default tag and kill values? In our experience just turning those down a little after the system has been running for sometime gives you very good spam filtering. What are your tag/kill vales set to now?

[claros]

Originally Posted by inigoml

Uploaded to wiki, improved with claros comments and fixed some errors.

http://wiki.zimbra.com/index.php?tit...ti-spam_system

Just a question: you used dag rpm repo to get "perl-Mail-SPF-Query" but not to get pyzor/agent? Maybe the wiki would be easier using the same repo...

I've got also some other antispam checks: Domainkeys, DKIM, RDJ, White/Blacklist. If someone is interested I can post guidelines/wiki.

Ciao

[claros]

Originally Posted by KevinH

Nice post and thanks for putting this in the Wiki. Did you change your default tag and kill values? In our experience just turning those down a little after the system has been running for sometime gives you very good spam filtering. What are your tag/kill vales set to now?

Mine is 35-100 (Spamassassin 7 -20) but $final_spam_destiny=D_PASS according to our law. Very good spam filtering with some custom values (eg. DK_VERIFIED -2.500 DK_SIGNED 1.000)

Are you planning to improve antispam checks in future releases?
And a native domainkeys support?

[inigoml]

Originally Posted by claros

Just a question: you used dag rpm repo to get "perl-Mail-SPF-Query" but not to get pyzor/agent? Maybe the wiki would be easier using the same repo...

I tried with razor... but.... broken dependency with razor-agents-sdk. No package available. :-(
There is no pyzor available.

I've found them in atrpms.net, but there is also dependencies with proper atrpms custom packages...

I've got also some other antispam checks: Domainkeys, DKIM, RDJ, White/Blacklist. If someone is interested I can post guidelines/wiki.
Ciao

Do it, please.

[inigoml]

Originally Posted by KevinH

Nice post and thanks for putting this in the Wiki. Did you change your default tag and kill values? In our experience just turning those down a little after the system has been running for sometime gives you very good spam filtering. What are your tag/kill vales set to now?

We didn't change it. Defaults for zimbra, that is, 6.6/15.
You are right. Lowering to 5.5 could improve recall, but would damage precision...
Anyway this is our first week with all company "zimbraed", we will continue adjusting these values.