Page 1 of 7 123 ... LastLast
Results 1 to 10 of 68

Thread: Improving spam filtering

  1. #1
    inigoml's Avatar
    inigoml is offline Project Contributor
    Join Date
    Aug 2006
    Location
    Madrid, Spain
    Posts
    124
    Rep Power
    9

    Lightbulb Improving spam filtering

    Hi.

    For the last three weeks, my IT team has been deploying a new Zimbra NE in our network for a 300 users group.
    In general, people has appreciated the improvement over our old cyrus+postfix+squirrellmail system, but we have experienced several complaints about spam filtering. We used spamassassin, but with zimbra our "hunting rate" has been specially low.

    So, we have started working with zimbra spamassasin in order to improve it. This is what we have done:

    1- First of all, zimbra spamassassin has no SPF enabled. Since perl enviroment is system way integrated, adding SPF support is fair simple. If you use RedHat or CentOS, you can "yum-it" from dag.wieers.com repositories by executing
    > yum install perl-Mail-SPF-Query

    2- Second, we added Razor2 in order to improve score.
    As root:
    Get razor-agents-sdk from razor.sourceforge.net, untar it and
    >perl Makefile.PL
    >make
    >make install

    Get also razor-agents from razor.sourceforge.net, untar it and
    >perl Makefile.PL
    >make
    >make install

    Open your firewall ports for razor2 (TCP/2703 outgoing).
    Create .razor folder in /opt/zimbra and give zimbra user permissions
    >mkdir /opt/zimbra/.razor; chown -Rf zimbra:zimbra /opt/zimbra/.razor

    As zimbra user, create your razor account:
    >razor-admin -create
    >razor-admin -discover
    >razor-admin -register

    3- Now we are going to add pyzor support for increase (again) spam score
    As root, install python support
    >yum install python

    Get pyzor package from pyzor.sourceforge.net, untar it and:
    >python setup.py build
    >python setup.py install

    Set perms according with pyzor readme.
    >chmod -R a+rX /usr/share/doc/pyzor \ >/usr/lib/python2.3/site-packages/pyzor \
    >/usr/bin/pyzor /usr/bin/pyzord

    Create .pyzor folder into zimbra home and set perms
    >mkdir /opt/zimbra/.pyzor; chown zimbra:zimbra /opt/zimbra/.pyzor

    Open your firewall ports for pyzor (TCP/24441 outgoing)

    And ready to go, as zimbra user, with:
    >pyzor discover

    4- Now we have PYZOR + RAZOR + SPF. But it would be advisable to enable it and give SPF a higher score. Those admins with wrong SPF entries should be punished since it's not mandatory and so, if you enable it, do it well.
    So open your spamassassin config at
    /opt/zimbra/conf/spamassassin/local.cf and add this rules at the end (customize it at your own):

    ----
    ok_languages en es
    ok_locales en es
    trusted_networks 127. 10.70. 192.168.
    use_bayes 1
    skip_rbl_checks 0
    use_razor2 1
    #use_dcc 1 <<< WORK IN PROGRESS
    use_pyzor 1

    dns_available yes

    ## Optional Score Increases
    score DCC_CHECK 4.000
    score SPF_FAIL 10.000
    score SPF_HELO_FAIL 10.000
    score SPF_SOFTFAIL 10.000
    score RAZOR2_CHECK 2.500
    score PYZOR_CHECK 2.500
    score BAYES_99 4.300
    score BAYES_90 3.500
    score BAYES_80 3.000

    bayes_ignore_header Received: from mail3.vectorsf.com
    bayes_ignore_header Received: from localhost
    bayes_ignore_header Received: from mail1.vectorsf.com
    bayes_ignore_header Received: from mail2.vectorsf.com
    -----

    5- Some notes about this:
    In zimbra, by default, spam with 15 score of higher is discarded by amavisd. If you want your user receive these mails, you have to modify amavisd.conf settings (/opt/zimbra/conf/amavisd.conf) in order to pass this email.

  2. #2
    claros's Avatar
    claros is offline Project Contributor
    Join Date
    Mar 2006
    Location
    L'Aquila, ITALIA
    Posts
    59
    Rep Power
    9

    Default

    Quote Originally Posted by inigoml

    So open your spamassassin config at
    /opt/zimbra/conf/spamassassin/local.cf and add this rules at the end (customize it at your own):

    ----
    ok_languages en es
    ok_locales en es
    trusted_networks 127. 10.70. 192.168.
    use_bayes 1
    skip_rbl_checks 0
    use_razor2 1
    #use_dcc 1 <<< WORK IN PROGRESS
    use_pyzor 1

    dns_available yes
    Nice post, I'd like to contribute.

    At least with Zimbra Version 4.xx, you should also edit
    /opt/zimbra/conf/spamassassin/v310.pre
    to uncomment the loadplugin config in this way
    Code:
    # Razor2 - perform Razor2 message checks.
    #
    # Razor2 is disabled here because it is not available for unlimited free
    # use.  It is currently free for personal use, subject to capacity
    # constraints.  See the Cloudmark SpamNet Service Policy for more details.
    #
    loadplugin Mail::SpamAssassin::Plugin::Razor2
    I've the same setup, including DCC e RulesdoJour
    I think you increased a lot your spam score values, do you have false positive?
    We too have an amavis setup to pass spam with
    $final_spam_destiny = D_PASS


    To setup DCC:
    Download dcc from DCC Site

    I compile on different system to build an rpm to install in production environment.
    Use the attached spec file (rename it to .spec) to build an rpm with the command:
    rpmbuild -ba /usr/src/redhat/SPECS/dcc.spec

    install it on the production server:
    rpm -ivh dcc-x.y.z.rpm

    Change /etc/dcc/dcc_conf to read:

    DCCUID=zimbra
    DCCD_ENABLE=off

    Change /opt/zimbra/conf/spamassassin/v310.pre to enable the DCC plugin:

    Code:
    # DCC - perform DCC message checks.
    #
    # DCC is disabled here because it is not open source.  See the DCC
    # license for more details.
    #
    loadplugin Mail::SpamAssassin::Plugin::DCC
    Have fun.
    I use sqlgrey as greylist server, so I don't need another one. As to me the standard value DCC == 2.5 Spamassassin point is ok, so I do not change it.
    With SA 3.xx you do not need to use enable_dcc il local.cf. That's the same for razor2 indeed...

    I think we should prepare a script to save and restore this config changes upon zimbra updates...

    Ciao
    Attached Files Attached Files
    Last edited by claros; 10-16-2006 at 03:57 AM.

  3. #3
    inigoml's Avatar
    inigoml is offline Project Contributor
    Join Date
    Aug 2006
    Location
    Madrid, Spain
    Posts
    124
    Rep Power
    9

    Default DDC too. Great!

    Thank you, claros!
    I forgot to comment razor2 module activation.

    By the way, your rpm .spec for DCC is very useful for me. In fact, I'm gonna include DCC right now.

    I think these modifications should be included in zimbra default configuration... at least SPF and pyzor with some options in the admin interface...

  4. #4
    claros's Avatar
    claros is offline Project Contributor
    Join Date
    Mar 2006
    Location
    L'Aquila, ITALIA
    Posts
    59
    Rep Power
    9

    Default

    Quote Originally Posted by inigoml
    Thank you, claros!
    I forgot to comment razor2 module activation.

    By the way, your rpm .spec for DCC is very useful for me. In fact, I'm gonna include DCC right now.

    I think these modifications should be included in zimbra default configuration... at least SPF and pyzor with some options in the admin interface...
    I do agree.
    And I'd like a Whitelist/Blacklist administration in the admin interface too. Actually I've enabled it in amavis as a textfile...

    We should just start a new wiki page with how-to, then submit a new feature request in bugzilla and let the people vote for it

    Ciao

  5. #5
    inigoml's Avatar
    inigoml is offline Project Contributor
    Join Date
    Aug 2006
    Location
    Madrid, Spain
    Posts
    124
    Rep Power
    9

    Default Uploaded to wiki

    Uploaded to wiki, improved with claros comments and fixed some errors.

    http://wiki.zimbra.com/index.php?tit...ti-spam_system

  6. #6
    KevinH's Avatar
    KevinH is offline Expert Member
    Join Date
    Aug 2005
    Location
    San Mateo, CA
    Posts
    4,789
    Rep Power
    18

    Default

    Nice post and thanks for putting this in the Wiki. Did you change your default tag and kill values? In our experience just turning those down a little after the system has been running for sometime gives you very good spam filtering. What are your tag/kill vales set to now?
    Looking for new beta users -> Co-Founder of Acompli. Previously worked at Zimbra (and Yahoo! & VMware) since 2005.

  7. #7
    claros's Avatar
    claros is offline Project Contributor
    Join Date
    Mar 2006
    Location
    L'Aquila, ITALIA
    Posts
    59
    Rep Power
    9

    Default

    Quote Originally Posted by inigoml
    Uploaded to wiki, improved with claros comments and fixed some errors.

    http://wiki.zimbra.com/index.php?tit...ti-spam_system
    Just a question: you used dag rpm repo to get "perl-Mail-SPF-Query" but not to get pyzor/agent? Maybe the wiki would be easier using the same repo...

    I've got also some other antispam checks: Domainkeys, DKIM, RDJ, White/Blacklist. If someone is interested I can post guidelines/wiki.

    Ciao

  8. #8
    claros's Avatar
    claros is offline Project Contributor
    Join Date
    Mar 2006
    Location
    L'Aquila, ITALIA
    Posts
    59
    Rep Power
    9

    Default

    Quote Originally Posted by KevinH
    Nice post and thanks for putting this in the Wiki. Did you change your default tag and kill values? In our experience just turning those down a little after the system has been running for sometime gives you very good spam filtering. What are your tag/kill vales set to now?
    Mine is 35-100 (Spamassassin 7 -20) but $final_spam_destiny=D_PASS according to our law. Very good spam filtering with some custom values (eg. DK_VERIFIED -2.500 DK_SIGNED 1.000)

    Are you planning to improve antispam checks in future releases?
    And a native domainkeys support?

  9. #9
    inigoml's Avatar
    inigoml is offline Project Contributor
    Join Date
    Aug 2006
    Location
    Madrid, Spain
    Posts
    124
    Rep Power
    9

    Thumbs up

    Quote Originally Posted by claros
    Just a question: you used dag rpm repo to get "perl-Mail-SPF-Query" but not to get pyzor/agent? Maybe the wiki would be easier using the same repo...
    I tried with razor... but.... broken dependency with razor-agents-sdk. No package available. :-(
    There is no pyzor available.

    I've found them in atrpms.net, but there is also dependencies with proper atrpms custom packages...

    I've got also some other antispam checks: Domainkeys, DKIM, RDJ, White/Blacklist. If someone is interested I can post guidelines/wiki.
    Ciao
    Do it, please.

  10. #10
    inigoml's Avatar
    inigoml is offline Project Contributor
    Join Date
    Aug 2006
    Location
    Madrid, Spain
    Posts
    124
    Rep Power
    9

    Default

    Quote Originally Posted by KevinH
    Nice post and thanks for putting this in the Wiki. Did you change your default tag and kill values? In our experience just turning those down a little after the system has been running for sometime gives you very good spam filtering. What are your tag/kill vales set to now?
    We didn't change it. Defaults for zimbra, that is, 6.6/15.
    You are right. Lowering to 5.5 could improve recall, but would damage precision...
    Anyway this is our first week with all company "zimbraed", we will continue adjusting these values.

Page 1 of 7 123 ... LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Spam filtering on POP accounts?
    By dwmtractor in forum Administrators
    Replies: 8
    Last Post: 01-09-2009, 10:43 AM
  2. Order of operations - SPAM and filtering
    By SpEnTBoY in forum Administrators
    Replies: 0
    Last Post: 04-24-2007, 06:01 AM
  3. Replies: 2
    Last Post: 12-20-2006, 08:07 AM
  4. suggestion for improving spam controls on zimbra
    By pheonix1t in forum Administrators
    Replies: 2
    Last Post: 11-10-2006, 02:31 PM
  5. Disabling Spam Filtering
    By plan9 in forum Administrators
    Replies: 1
    Last Post: 10-25-2006, 07:10 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •