Improving spam filtering

[jdell]

Originally Posted by Ericx

Wow! It took me a few different tries to get the setup going described in the beginning of this forum - or the one on the wiki. The examples do not include instructions for Redhat.

I'm using RedHat RHEL4 - an actual paid for licensed copy!

My first problem was finding all of the packages to get spf and other modules working.
Including
* Net-CIDR-Lite
* perl-Sys-Hostname-Long-1.4-1.2.el4.rf.noarch.rpm
* python-elementtree
* python-urlgrabber-2.9.6-1.2.el4.rf.noarch.rpm
* sqlite - don't know which one actually worked I downloaded about 3 of them.

If someone asks I'll zip them up and post them here.

It took me about 2 hours to do all the work, test some email's, compare headers and adjust scores. Worth every minute. I have to say one of the biggest complaints I've been getting from my users is the increase in spam since we have moved from Exchange. We used GFI mail essentials before.

Zimbra is amazing and I like it, so don't get me wrong, but I think the Zimbra staff should spend more time on documenting how we can improve spam and or creating an admin plug in that can help us better manage spam, including whitelist manager, blacklist manager, and well just take a look at the GFI configuration manager.

I use both CentOS4 and RHEL4. Admittedly, Yum really is much easier to use than up2date, but you can get those rpmforge packages via up2date and save yourself all the headaches and make sure you have the right RPM's.

You just need to add them in /etc/sysconfig/rhn/sources

There are examples in there for how to add Yum repos which work for up2date.

The only other trick is that you need to remember to import the GPG key for the repository.

rpm --import http://dag.wieers.com/packages/RPM-GPG-KEY.dag.txt

Of course, rpmforge has even gone so far as to create an RPM package that installs the GPG keys and updates the sources for you.

Just install this RPM, and RHEL4 will have access to all RPMForge RPM's

rpm -Uhv http://dag.wieers.com/packages/rpmforge-release/rpmforge-release-0.3.4-1.el4.rf.i386.rpm

Check here to ensure that you have the latest goods, although, once you install that RPM, if there is an update to it, you should now get it through up2date

http://dag.wieers.com/rpm/FAQ.php

[inigoml]

Originally Posted by Ericx

Wow! It took me a few different tries to get the setup going described in the beginning of this forum - or the one on the wiki. The examples do not include instructions for Redhat.

Hi Ericx.
Samples were made with CentOS 4.4, that is a clon of RedHat 4.4
Yes, you are right wiki could have been made better, but each user has his own distro with some packages depending what type of installation selected first time set up the machine.

Sorry if the wiki is not as complete as it should be. I'm not zimbra employee, and I'm not get paid for this work. So perhaps the test is not as deep as it should have been. Anyway, I invite you to complete wiki article with your experience.

And now, speaking about spam, I suggest you install SQLGrey. This will indeed increase your spam catching (or spam bouncing in this case) to a really high levels. Of course, your zimbra should be receiving mails directy for internet. If not, don't waste your time. But greylisting is at this moment the most efficient way to avoid spam. In my installation it has about 80-90% of spam bounced, and the 5% pass is caught by other techniques as blacklist, bayesian filter, SPF an others explained in that wiki.

[Ericx]

INIGOML, Thank you for the post on SQLGrey. I'll make that my priority project for Monday. I want to personally thank you or whoever made the wiki for improving spam. It is very well done. I'm sorry if I left the wrong impression.

Since the wiki and forums are community supported, I was nervous about making those changes to my production server. I'm using the commercial version of Zimbra on a Commercial Version of Redhat; which Zimbra recommends. I even received a discount from Zimbra for using RHEL4 instead of Suse, which I normally use. My comments are more for the Zimbra team to officially put something out there for spam management.

Spam is killing me and my users. I've spent more time trying to tame spam than I have all the other tasks combined - including testing and migrating from Exchange (which went extremely well!)

When I did the sales pitch to management for the move to Zimbra from Exchange, the money equation included a comparison of our current Spam Filtering Expenses compared to the built in Spam and Virus filtering in Zimbra. I have never used spamassassian before, but I felt confident that I could figure it out. Wrong! If it weren't for the posts made by the Zimbra community, I would have never figured it out. The spamassasian distro is configured differently enough from the generic configuration that the spamassassian docs were of little help. I would like to see the Zimbra staff put out an official document stating where the config files are stored and how to modify them for basic tasks such as whitelisting. Even a simple admin zimlet that will allow System Admins to manually edit some of the config files would be great! That should not be too hard to do. I'll even try to stub one out and test it and post it on the forums if someone can give me a list of files.

Everything in Zimbra has exceeded my expectations. We are starting to do some very cool integration with workflows and our Project Management system. Everything except spam.... I really want Zimbra to succeed as a company. We need more diversity in the Enterprise. I hope that my comments will be taken as strong need for an area of improvement (spam) and not complaining.

Posts live for a long time! If you are looking at migrating to Zimbra please don't let my comments negatively influence your decision. I still would have migrated knowing this problem existed and chances are the problem will be solved very soon.

[phoenix]

If you wish to see any changes or enhancements in Zimbra then the best place for them is in bugzilla, they get the most attention from the development team. While Zimbra staff do look at the forums they may not always see a request in a post such as yours but they do see all the enhancement requests.

[inigoml]

Originally Posted by Ericx

INIGOML, Thank you for the post on SQLGrey. I'll make that my priority project for Monday. I want to personally thank you or whoever made the wiki for improving spam. It is very well done. I'm sorry if I left the wrong impression.

I uploaded first version, but it has been enriched, corrected and improved by several forum members. This is a colaborative working, so if you can share your experiences, they will be welcomed.

And SQLGrey (or greylisting in general)... ufff... at this moment the holy grial.
Anyway, you have to understand perfectly how greylisting works and some related minor problems. There is only to minor (for me) problems:

1- First mail from a user to a local user takes longer than before. Depending your greylisting configuration and remote MTA retry time, from 5 minutes to several hours. Tipically, no more than 30 minutes. But ONLY first time.

2- Some old MTAs do not work well with "temporaly unavailable" response from your server and treat it like an "unavailable" and mail is bounced to sender. But this is very unsual.

However benefits are evident, and avoiding most of spam is wonderful.

Since the wiki and forums are community supported, I was nervous about making those changes to my production server. I'm using the commercial version of Zimbra on a Commercial Version of Redhat; which Zimbra recommends. I even received a discount from Zimbra for using RHEL4 instead of Suse, which I normally use. My comments are more for the Zimbra team to officially put something out there for spam management.

Zimbra employees read everyday this forum, and they review documents created by members. But I understand perfectly you. When working with commercial software you are expecting some "formal" support for these kind of modifications. But Zimbra is commercial... and an opensource development, so there is a shared way to make things better you will not find in commercial software. For example, Microsoft wouldn't let you customize exchange the way Zimbra let's you customize its product.

Spam is killing me and my users. I've spent more time trying to tame spam than I have all the other tasks combined - including testing and migrating from Exchange (which went extremely well!)

Yes, it's probably one area where zimbra development team should improve the product. However is quite easy to do it. I'm sure for next release (5.0) we will see some changes in spam administration and functionality.

I have never used spamassassian before, but I felt confident that I could figure it out. Wrong! If it weren't for the posts made by the Zimbra community, I would have never figured it out.

Do you remember McAfee antispam? It's spamassasin based.

Even a simple admin zimlet that will allow System Admins to manually edit some of the config files would be great! That should not be too hard to do. I'll even try to stub one out and test it and post it on the forums if someone can give me a list of files.

Ask for it! I assure you that bugzilla works! I have requested some new functionality for Zimbra and, sometimes before, sometimes later, it's finally added. For example, http proxy support for Outlook Connector will be included in next minor release (4.5.1), and I asked for it one month and a half ago.

[Ericx]

I've started a new thread for outlining and possibly building a zimlet to manage spam.

http://www.zimbra.com/forums/showthr...ed=1#post35743

[brained]

I've followed the wiki guidelines for sqlgrey and it has worked well.

I've also implemented an additional method for reducing spam. First some caveats:

This could lead to mass blocking of legitimate mail.
This may not be a good fit for some organizations.
This may not reduce the amount of spam reaching the Inbox's but it will reduce the server load.

Here it is:

After reviewing weeks worth of logs I was able to confirm that the kill level was set high enough that no legitimate email was ever blocked. Some legitimate mail was tagged as spam and passed on but never blocked. Also, it appears that companies that do virtual hosting are not a source of the blocked spam - all of it appears to come from bots and foreign countries.
Based on that I decided there's no reason to even accept email connections for computers that send email that's going to be blocked from delivery anyway.

Enter fail2ban - it scans the zimbra logs for a blocked email and then using iptables drops any further packets from that servers ip address.
That worked so successfully I also added rules for servers that connect and have no dns record (they show up as 'unknown').
I have had for a few years several honeypot email addresses and added them to the rules also (send email to a hidden email address and you don't get to send again).
IP's are banned for just a few days and then allowed to connect again.

The results are that no one is missing legitimate messages and the server is processing 1/6th the number of emails it used to

[dlochart]

I followed the WIKI and enabled SPF, pyzor, and the lot for ZCS 4.0.5 and it worked fine. I have since upgraded to 4.5.1 and just today 4.5.4. It seemed to work in 4.5.1 and I expect it will still work today but my question is that I am seeing entries like in 25_razor2.cf and other files (which I still don't know how they are used) that seem to indicate that some of the WIKI article is now part of the base distribution. Can someone comment on this? Do I have to undo what I did for the WIKI and do it a different way?

One reason I ask is that for the score_SPF_FAIL entry the WIKI has a value of 10. The 50_scores.cf file contains this line:

score SPF_FAIL 0 1.333 0 1.142

So I see 4 values (not 1)

any help/understanding will be appreciated

thanks

Doug

[LMStone]

We use the built-in RBL feature on our SonicWall PRO 2040 to block connection attempts before they even get to the Zimbra server.

On the Zimbra server we then implement:
reject_non_fqdn_sender
reject_unknown_client
reject_unknown_sender_domain

The latter two checks do block some email from legitimate senders with horribly configured email servers, but once we show the end users and the senders the domain reports from dnsreport.com, they "get it" and the sender's email domain administrator then gets the "motivation" to fix their broken configs.

We also implemented Rules du Jour on our Zimbra server, using mostly just the 0 and 1 rulesets (the 2 sets generate too many false positives for us).

The SonicWall has really reduced the workload on our server as you might imagine, and the remaining checks have most users getting only a few spams per week, with only one documented false positive in the past few years we have been using this technique, even before switching our mail system to Zimbra.

Hope that helps,
Mark

[dgamez]

Originally Posted by claros

Just a question: you used dag rpm repo to get "perl-Mail-SPF-Query" but not to get pyzor/agent? Maybe the wiki would be easier using the same repo...

I've got also some other antispam checks: Domainkeys, DKIM, RDJ, White/Blacklist. If someone is interested I can post guidelines/wiki.

Ciao

Hi Claros!

I'm interested basically into implement DKIM.. not for Antispam support, but for signs and/or verifies emails.

Could you post some guidelines/wiki about it ?

Thanks.