Some questions about Zimbra and spam

[fhouston]

Mods: Hopefully this is the right area of the forum, if not please move to where ever is appropriate.


We are looking at migrating to zimbra from a mish-mash of open source programs running on a debian machine (postfix, spamassassin, procmail, UW IMAP, etc...). So far those helping me w/ the test server like what we see w/ Zimbra but we have some questions about the spam processing. Most of this is based on trying to replicate our current layout as much as possible and some of it may not carry over, but if anyone has any advice it would be appreciated.

To give you an idea of the current layout, spamassassin marks anything above the score of 5 as spam. The server passes ALL mail, aka no matter what the spam score it is passed on, to the end user who then decides what to do w/ it. We haven't stipulated a particular client be used but I'd say the majority of people are running Thunderbird, some are on some Mac applications (Mac mail being the most popular) and I have a few Outlook hold outs (that I'm hoping to move them away from if/when we switch ). Procmail is used to pre-filter some of the mail, mainly marked spam to a "junk" folder which local/client filters then look at to pull out ham. Users manually move all spam to a folder called "spam" which is processed by the server every 4 hours to add the spam to the bayes DB. People can move ham to a ham folder which does the same and recovers the original email. Finally every morning the server runs 'sa-update' to update the rules.

So the questions are:

- Is there a way to remove the subject marking when something is marked "not junk"?

- When is the email actually processed/added to the bayes DB (as soon as it is put in the junk folder or sometime later)? How about ham, when things are marked "not junk", is that added right away?

- Is there something equivalent to the "sa-update" command, or does the happen automatically anyway? I didn't see anything in the admin app about it but maybe I overlooked it. Can/should I just carry my script over from my current machine and run that via cron on the zimbra server?

- I see where individual white/black lists can be created in user preferences, but what is the preferred way to add a global whitelist/blacklist? I found something that said modify the salocal.cf.in, I think it was, but my customizations there were lost at some point. After a server or system restart or something, not entirely sure when.

- Is there a way to turn off the auto kill feature and allow all email to be passed on?

- The filters in the user preferences only work on mail that hits the Inbox right? Is there any way to have them apply to the Junk box as well? For example, how I personally process mail on our current server, I have a filter that looks in the "junk" folder populated by procmail w/ all the server marked spam. Anything from someone in one of my address books goes back into my Inbox. Anything w/ a score of 5 to 10 goes into a second junk box that I take a closer look at to see if something creeped into the spam range, everything else is dumped into the "spam" folder for processing by the server. I'd like to replicate this as much as possible but at min. having people that are in my address books not go into the junk folder (or extracted from it if they are) would be nice.

EDIT: On this last issue I tried creating a filter "address from in my contacts" "file into folder inbox" and running that on the "junk" folder but it didn't recover anything, even though I purposefully put an email from a contact in my address book into the junk folder to test it. Any idea why this didn't work?


I think that's it for now... hints/clues/suggestions are appreciated.

[fhouston]

No input


Also another quick question... if I understand the setup correct the default internal base score is 10 (although I don't recall where I read that so it might be wrong), kill % is 75 (so a score of 7.5) and mark % is 33 (score of 3.3), does that sound right? If that is the case can someone explain this line from one of the spam emails I've received recently:

X-Spam-Status: No, score=6.351 tagged_above=-10 required=6.6

This email was not marked and based on 6.6 I guess the score is 20 not 10? But then what is the "tagged above=-10" mean?

[uxbod]

[SOLVED] How setting the spamassassin scrore limit ?

[fhouston]

So it is 20... cool thx. What's the "-10" thing about then? Any thoughts on the questions in the OP?

[ewilen]

I believe that -10 is the score above which the scoring header will be added.

Let me look over your OP and see if I can help with any of those other questions.

[ewilen]

Originally Posted by fhouston

To give you an idea of the current layout, spamassassin marks anything above the score of 5 as spam. The server passes ALL mail, aka no matter what the spam score it is passed on, to the end user who then decides what to do w/ it. We haven't stipulated a particular client be used but I'd say the majority of people are running Thunderbird, some are on some Mac applications (Mac mail being the most popular) and I have a few Outlook hold outs (that I'm hoping to move them away from if/when we switch ). Procmail is used to pre-filter some of the mail, mainly marked spam to a "junk" folder which local/client filters then look at to pull out ham. Users manually move all spam to a folder called "spam" which is processed by the server every 4 hours to add the spam to the bayes DB. People can move ham to a ham folder which does the same and recovers the original email. Finally every morning the server runs 'sa-update' to update the rules.

Replicating all of this exactly as written seems kind of inefficient given how Zimbra works, but that may just be me.

- Is there a way to remove the subject marking when something is marked "not junk"?

Zimbra's antispam setup doesn't rewrite subjects by default, although you can enable that if you wish. It sounds like you would be best served by simply not having it rewrite subjects.

Zimbra deals with spam in two layers, both of which are optional. First, there's MTA-level screening that rejects email during the SMTP transaction, based on configurable tests such as whether the hostname used in the SMTP greeting is valid, whether the sender address is valid, whether the sender's domain exists in DNS, and whether the delivering SMTP server's IP address is in any DNS blacklist that you choose to test against.

Second, there's amavisd, which calls spamassassin, clamav (for antivirus), and (optionally) other tests such DSPAM. The email gets a spam score which is then used against the kill/tag percent to decide what to do with the mail in terms of delivery. I.e., does the mail get dropped, is it passed but put in the user's Junk folder, or does it get delivered normally? The score is also used to decide if the mail is automatically used to retrain Bayesian scoring (apparently...see More Spam after upgrading to 6.0.5).

- When is the email actually processed/added to the bayes DB (as soon as it is put in the junk folder or sometime later)? How about ham, when things are marked "not junk", is that added right away?

If the mail is automatically used for retraining Bayes (see above), I don't know when it gets processed. Otherwise, email is ONLY used to retrain Bayes if an action is taken that reclassifies it. That is, if the mail is moved TO or FROM the Junk folder, a copy is automatically sent to a special ham or spam account, whose mailbox is then used to retrain Bayes once/day (or as often as you like if you set it up in cron, plus manual invocation).

Note that if you have any kind of client-level classification (such as Mac Mail's Junkmail feature, or even a user-level filter in Zimbra Web Client) then it can be used as a second line of defense to move emails to the Junk folder. When this happens, the mail is submitted just as if the user moved the mail.

In short, mail that gets classified as spam or ham by Zimbra isn't automatically re-fed into the Bayes training unless it's above or below a certain threshold. You should be able to modify that threshold but I haven't looked into it myself. Mail that's reclassified by user action does get re-fed into Bayes.

- Is there something equivalent to the "sa-update" command, or does the happen automatically anyway? I didn't see anything in the admin app about it but maybe I overlooked it. Can/should I just carry my script over from my current machine and run that via cron on the zimbra server?

Older versions of Zimbra didn't come with sa-update, although you could apparently install it yourself. As of 6.0.5 sa-update is included but it doesn't get run automatically. I'd either run it manually occasionally or stick it into cron.

- I see where individual white/black lists can be created in user preferences, but what is the preferred way to add a global whitelist/blacklist? I found something that said modify the salocal.cf.in, I think it was, but my customizations there were lost at some point. After a server or system restart or something, not entirely sure when.

Modifications to salocal.cf.in can be erased during an upgrade, so you should back them up. I don't know how to do a global whitelist/blacklist other than via salocal. You can find more on this in Improving Anti-spam system - Zimbra :: Wiki

- Is there a way to turn off the auto kill feature and allow all email to be passed on?

Possibly...see [SOLVED] How to turn off "Kill Percent" for spam filtering?. Note the bug at the end. Personally I would just raise the kill percent to 100 (or higher if allowed) and/or test to see if setting it to 0 will disable. You should be able to use GTUBE to test.

- The filters in the user preferences only work on mail that hits the Inbox right? Is there any way to have them apply to the Junk box as well?

See Bug 23886 - Run spam test before user filters during message delivery and Bug 12701 - Define a default filter rule for spam that users can apply before custom filters.