Improving spam filtering

[inigoml]

Originally Posted by Klug

The idea is to train a "clean" amavisd with both the "SPAM Archive" and some french spams (collected from several of my own mailboxes) to be able to deploy them by customers setups.

Training with spam is a good policy. However, spammers are changing the way to send spam. At this moment, you will receive a lot of spam containing images. Training is not useful for this spam, since bayesian filters cannot be trained with images. There is a plugin for "OCRing" images, but it's very CPU intensive.
I suggest to implement a greylist based system. It's really IMPRESSIVE how spam will decrease. There is a SQLGrey tutorial at wiki. I've not tested with SQLGrey because our perimeter mail servers uses postfix-gld service, but it's the same thing.

[Klug]

Originally Posted by inigoml

Training with spam is a good policy. However, spammers are changing the way to send spam. At this moment, you will receive a lot of spam containing images. Training is not useful for this spam, since bayesian filters cannot be trained with images. There is a plugin for "OCRing" images, but it's very CPU intensive.

But working nicely, I'm using it (with MailCleaner) for my non-zimbra mail accounts.

Originally Posted by inigoml

I suggest to implement a greylist based system. It's really IMPRESSIVE how spam will decrease. There is a SQLGrey tutorial at wiki. I've not tested with SQLGrey because our perimeter mail servers uses postfix-gld service, but it's the same thing.

Using it with MailCleaner too.

And I'll add it to the Zimbra servers at the same moment than the trained .spammassassin

[dlochart]

Originally Posted by inigoml

Yes, It also uses DSPAM, but I've not worked with it before, so I didn't get any action to improve score. However I've worked with spamassassin for years and has been easier for me to improve it.

Ok thanks

Originally Posted by inigoml

Yes, they do.



Zimbra uses both systems for spam catching. You could improve both.

Since all of this is new to me and my deadline is coming faster that santa I'll stick with SA for now

Originally Posted by inigoml

Check /opt/zimbra/conf/spamassassin/v310.pre. Here SPF plugin should be loaded (that is, without #). But by default it should be enabled.
You have also to restart amavisd (zmamavisdctl restart)

I did a full zimbra restart several times. I checked the file you mentioned and it is uncommented.

Is there a way to increase logging to find out if its working?

I tried doing the mail as you suggested but I did not see anything. I did it from the mta machine would that make any difference. Remember I can only access this thing from within our own network.

What about for Pyzor? Still no activity in the tcpdump

[dlochart]

Originally Posted by inigoml

Training with spam is a good policy. However, spammers are changing the way to send spam. At this moment, you will receive a lot of spam containing images. Training is not useful for this spam, since bayesian filters cannot be trained with images. There is a plugin for "OCRing" images, but it's very CPU intensive.
I suggest to implement a greylist based system. It's really IMPRESSIVE how spam will decrease. There is a SQLGrey tutorial at wiki. I've not tested with SQLGrey because our perimeter mail servers uses postfix-gld service, but it's the same thing.

I posted to the postfix-users group (good bunch of people) a question on how to send saved off mail messages (full headers and all) through my mail system when they reside in a text file. All of them discouraged me from doing this as it may screw things up. Now they say doing it with your own spam is good but not from a spam archive as it was not intended for your domain and you may create problems

[Klug]

Originally Posted by dlochart

Now they say doing it with your own spam is good but not from a spam archive as it was not intended for your domain and you may create problems

I'm not sure I understand what kind of problem it could create.

Spam is spam (as long as you're not a penis enlarger + ****** customer or looking for opportunities in investment in low risks companies 8)).

But I understand that training with "not that spam", such as mailing lists you subscribed but are too lazy to unsubscribe could lead to issues.

[dlochart]

Below is my post and response from the postfix group:

Doug Lochart wrote:
> This is probably way simple but I am unable to get good results
> searching and I want this to be automated as much as possible.
>
> I would like to test and then eventually train my postfix/amavisd/sa
> setup. I found a stie http://www.untroubled.org/spam/ that has archives
> of spam messages. So I downloaded an archive. The spam mailss contain
> the full message including headers. I know the basic telnet/nc way of
> sending mail but these already have the headers and such I would just
> like to inject them into my postfix and watch what happens. I know I
> will need to change the rcpt to address so that I accept the mail.
>
> Is there a simple way to do this?

Yes. Don't. The nice thing about SpamAssassin compared to pure bayesian
filters is that it already comes with spam identification, based on spam
in the wild. It will use these patterns to reliably identify spam and
add tokens to the bayesian database. Once you have tokenized a certain
number of ham & spam messages (200 of each, by default), it starts
scoring messages with it.

If you train your database with messages that aren't aimed at your
site/users, you are likely to negatively affect performance & accuracy.
Since tokens will be expired eventually, there is no advantage gained by
filling the database with tokens that will never be used or may be
classified incorrectly. For example, I have clients that are in the
healthcare industry that *cannot* block a message just because it
contains brand names of "life-enhancing" drugs (that sentence alone
might score some spam points on some systems).

It's actually more useful to run sa-update daily, so that you get the
latest patterns. And you can also train the db on spam that still makes
it to your account (or into the junk folders of users you can trust). I
do this once a week for the few that get by, and I've never needed to
train any messages as ham.

[Klug]

Originally Posted by dlochart

Since tokens will be expired eventually

I did not know that at all...
I might have to check a couple of things and change the policies here...

Edit a couple minutes later
Found this : https://secure.renaissoft.com/maia/wiki/ExpireBayes
It seems expiring tokens are tokens with only one (or very little) occurence, it's not about "age".

[dlochart]

I solved my problem. First of all pyzor just started working. I resolved the SPF issues as well. It turns out that SA will not use SPF or DUL if the ALL_TRUSTED rule fires. I did not have any value set for ALL_TRUSTED (I am in a prototype/test phase) so I specifically set it to my MTA address and sent a mail from another internal address and pow ... there was an SPF header.

[Ericx]

Wow! It took me a few different tries to get the setup going described in the beginning of this forum - or the one on the wiki. The examples do not include instructions for Redhat.

I'm using RedHat RHEL4 - an actual paid for licensed copy!

My first problem was finding all of the packages to get spf and other modules working.
Including
* Net-CIDR-Lite
* perl-Sys-Hostname-Long-1.4-1.2.el4.rf.noarch.rpm
* python-elementtree
* python-urlgrabber-2.9.6-1.2.el4.rf.noarch.rpm
* sqlite - don't know which one actually worked I downloaded about 3 of them.

If someone asks I'll zip them up and post them here.

It took me about 2 hours to do all the work, test some email's, compare headers and adjust scores. Worth every minute. I have to say one of the biggest complaints I've been getting from my users is the increase in spam since we have moved from Exchange. We used GFI mail essentials before.

Zimbra is amazing and I like it, so don't get me wrong, but I think the Zimbra staff should spend more time on documenting how we can improve spam and or creating an admin plug in that can help us better manage spam, including whitelist manager, blacklist manager, and well just take a look at the GFI configuration manager.

[jholder]

Hi Ericx,
Thanks so much for your hard work!

We very much value your suggestions, and I have forwarded them on to several of the team members.

Thanks again,
john