Zimbra spam system

[rajahd]

Ok first off I have done a search and read through many threads talking about how the AV/AS systems work as well as the wiki materials. Still, I'm confused as to why over the last several months I'll get blasted with dozens of the same e-mail over a period of hours/days and even after clicking junk on them, near identical e-mails will make it through. This particular server is FOSS 4.5.11 on Ubuntu. The server/global settings are default (kill=50,tag=33). Stuff does get into the junk folder so obviously stuff is working in some context.

For an example. Here is an e-mail that that I received 20ish copies of over the last 3-4 hours:

Received: from localhost (localhost.localdomain [127.0.0.1])
by email.domain.com (Postfix) with ESMTP id 81FFA98370
for <me@domain.com>; Mon, 14 Apr 2008 18:18:11 -0500 (CDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: 3.456
X-Spam-Level: ***
X-Spam-Status: No, score=3.456 tagged_above=-10 required=6.6
tests=[BAYES_50=0.001, URIBL_BLACK=1.955, URIBL_OB_SURBL=1.5]
Received: from email.domain.com ([127.0.0.1])
by localhost (email.domain.com [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id UpZW9rETr67Z for <me@domain.com>;
Mon, 14 Apr 2008 18:18:05 -0500 (CDT)
Received: from 47.srv.static.dedicatednetaxxes.com (47.srv.static.dedicatednetaxxes.com [63.217.246.47])
by email.domain.com (Postfix) with SMTP id C6C789836E
for <me@domain.com>; Mon, 14 Apr 2008 18:18:04 -0500 (CDT)
From: "Casting Dept" <DoyleSeal@tesho.rhinencephalic.com>
To: me@domain.com
Subject: Movie Extras, Actors, Models Wanted
Date: Mon, 14 Apr 2008 15:03:43 -0800
MIME-Version: 1.0
Content-type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit
Message-Id: <06797A6B7C6B6846747A68753469757356344207@tesho.rh inencephalic.com>


Hiring for TV / Movie Extras, Actors & Models in your area!
Make over $200/day as a movie extra.
Are you available http://www.menologies.com/hollywood/
































http://www.menologies.com/hollywood/...7f7dsm&pu=witx Get off List Here
To stop recieving these newsletters send all requests to:
Hollywood Extras
195 Hwy 50, #104
PMB 290, PO Box 7172
Stateline, NV 89449-7172


Do not envy the one you admire most, instead, try to make yourself one that can be envied
-"Love starts with a hug, grows with a kiss, and ends with a tear!"
-"The lesson is in the struggle, not in the victory!"
-"Only those who can see the invisible can do the impossible!"
-"Its ok to kiss a fool, its ok to let a fool kiss you, but never let a kiss fool you!"
"Friends are gods ways of apologizing for our families"
Life's a garden dig it
"You only live once...but if you live it right, once is enough"
Only after the last tree has been cut down,
only after the last river has been poisoned,
only after the last fish has been caught,
only then will you realize that money cannot be eaten
"If your ship doesn’t come in, swim out to it"-Jonathan winters.
"The best man for a job is a woman"-Ares (from Xena Warrior Princess)
"Knowledge talks, wisdom listens"
"To control others is to have power, to control yourself is to know the way" - Lao Ma
A woman has to work twice as hard as a man to be thought of as half as good. Luckily this is not difficult"
II can resist anything but temptation I believe in angels, the Kind that heaven sends ...I'm surrounded By angels, but I call Them my best friends

I clicked Junk on this e-mail. Later this e-mail came in:

Received: from localhost (localhost.localdomain [127.0.0.1])
by email.domain.com (Postfix) with ESMTP id 8062E98370
for <me@domain.com>; Mon, 14 Apr 2008 18:30:14 -0500 (CDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: 3.456
X-Spam-Level: ***
X-Spam-Status: No, score=3.456 tagged_above=-10 required=6.6
tests=[BAYES_50=0.001, URIBL_BLACK=1.955, URIBL_OB_SURBL=1.5]
Received: from email.domain.com ([127.0.0.1])
by localhost (email.domain.com [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id vbX2+wgCG9mA for <me@domain.com>;
Mon, 14 Apr 2008 18:30:08 -0500 (CDT)
Received: from 95.srv.static.region1node3.com (95.srv.static.region1node3.com [208.111.186.95])
by email.domain.com (Postfix) with SMTP id C58939836E
for <me@domain.com>; Mon, 14 Apr 2008 18:30:07 -0500 (CDT)
From: "Casting Dept" <WilmerMccarthy@mesox.ropeband.com>
To: me@domain.com
Subject: Movie Extras, Actors, Models Wanted
Date: Mon, 14 Apr 2008 15:15:46 -0800
MIME-Version: 1.0
Content-type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit
Message-Id: <06797A6B7C6B6846747A68753469757310233788@mesox.ro peband.com>


Hiring for TV / Movie Extras, Actors & Models in your area!
Make over $200/day as a movie extra.
Are you available http://www.boundenling.com/hollywood/
































http://www.boundenling.com/hollywood...7dxnw&lk=nxwxs Get off List Here
To stop recieving these newsletters send all requests to:
Hollywood Extras
195 Hwy 50, #104
PMB 290, PO Box 7172
Stateline, NV 89449-7172


Finish each day and be done with it. You have done what you could; some blunders and absurdities have crept in; forget them as soon as you can. Tomorrow is a new day; you shall begin it serenely and with too high a spirit to be encumbered with your old nonsense. -- Ralph Waldo Emerson

The ONLY things changed in these quotes are the user/domain of my e-mail and server to protect the guilty (me!).

It seems like the effectiveness of the spam system has dropped considerably over the last 2 months. Maybe this is just a byproduct of the spammers getting smarter...

Any help is appreciated.

[phoenix]

The first thing to point out is that your Kill/Tag percentages are not set to the default, that's 75/33 respectively. I have mine set to 66/25 and see very little spam. It looks like your messages are not being checked for many rules, a typical message in my Junk folder would have the following tests:

X-DSPAM-Result: Spam
X-DSPAM-Confidence: 0.6480
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 4801f0c4273521156014651
X-DSPAM-Factors: 15,
X-Virus-Scanned: amavisd-new at 
X-Spam-Flag: YES
X-Spam-Score: 13.053
X-Spam-Level: *************
X-Spam-Status: Yes, score=13.053 tagged_above=-10 required=5
	tests=[BAYES_99=3.5, DSPAM_SPAM=1.5, HTML_MESSAGE=0.001,
	RAZOR2_CF_RANGE_51_100=0.5, RAZOR2_CF_RANGE_E8_51_100=1.5,
	RAZOR2_CHECK=0.5, SPF_SOFTFAIL=0.596, URIBL_BLACK=1.955,
	URIBL_JP_SURBL=1.501, URIBL_OB_SURBL=1.5]

Have a look in the /opt/zimbra/conf/spamassassin folder and see if you've got all the .cf rules files in there, you should have about 110(ish) files. Do you update those files at all?

Have you made any modifications to the ant-spam/virus system by adding any RBLs, whitelists, SPF, Razor, PYZOR etc? If you have, what are they? I assume your spam/ham mailboxes are still there and are set, check them with:

zmprov gacf | grep SpamAccount

[phoenix]

I forgot to add that you can run spamassassin debug and see what's going wrong, have a look at this thread and specifically at post 8 onwards for some details on what to do.

[rajahd]

I thought the 50/33 was the default - this box has been thru several upgrades so either that "was" the default at some time or I changed it without realizing it somewhere along the line (certainly possible!). I have done nothing beyond whatever Zimbra does on it's own for spam. That is, I have not initiated any pre-training (though I've run thousands of spam over the last year through it from my e-mail box alone). I have not adjusted any rules at the command line. The spamassain directory does not contain 110ish files, only about 50ish...

/opt/zimbra/conf/spamassassin:

drwxr-xr-x 2 zimbra zimbra 4096 2007-12-24 20:41 .
drwxr-xr-x 7 zimbra zimbra 4096 2008-04-15 01:23 ..
-r--r--r-- 1 zimbra zimbra 5671 2007-11-17 14:47 10_default_prefs.cf
-r--r--r-- 1 zimbra zimbra 7509 2007-11-17 14:47 20_advance_fee.cf
-r--r--r-- 1 zimbra zimbra 6779 2007-11-17 14:47 20_body_tests.cf
-r--r--r-- 1 zimbra zimbra 1894 2007-11-17 14:47 20_compensate.cf
-r--r--r-- 1 zimbra zimbra 14836 2007-11-17 14:47 20_dnsbl_tests.cf
-r--r--r-- 1 zimbra zimbra 14998 2007-11-17 14:47 20_drugs.cf
-r--r--r-- 1 zimbra zimbra 10908 2007-11-17 14:47 20_dynrdns.cf
-r--r--r-- 1 zimbra zimbra 8386 2007-11-17 14:47 20_fake_helo_tests.cf
-r--r--r-- 1 zimbra zimbra 24693 2007-11-17 14:47 20_head_tests.cf
-r--r--r-- 1 zimbra zimbra 10480 2007-11-17 14:47 20_html_tests.cf
-r--r--r-- 1 zimbra zimbra 5287 2007-11-17 14:47 20_imageinfo.cf
-r--r--r-- 1 zimbra zimbra 3330 2007-11-17 14:47 20_meta_tests.cf
-r--r--r-- 1 zimbra zimbra 2524 2007-11-17 14:47 20_net_tests.cf
-r--r--r-- 1 zimbra zimbra 8645 2007-11-17 14:47 20_phrases.cf
-r--r--r-- 1 zimbra zimbra 2062 2007-11-17 14:47 20_****.cf
-r--r--r-- 1 zimbra zimbra 15877 2007-11-17 14:47 20_ratware.cf
-r--r--r-- 1 zimbra zimbra 5480 2007-11-17 14:47 20_uri_tests.cf
-r--r--r-- 1 zimbra zimbra 18015 2007-11-17 14:47 20_vbounce.cf
-r--r--r-- 1 zimbra zimbra 2576 2007-11-17 14:47 23_bayes.cf
-r--r--r-- 1 zimbra zimbra 1544 2007-11-17 14:47 25_accessdb.cf
-r--r--r-- 1 zimbra zimbra 1536 2007-11-17 14:47 25_antivirus.cf
-r--r--r-- 1 zimbra zimbra 1544 2007-11-17 14:47 25_asn.cf
-r--r--r-- 1 zimbra zimbra 1309 2007-11-17 14:47 25_dcc.cf
-r--r--r-- 1 zimbra zimbra 2184 2007-11-17 14:47 25_dkim.cf
-r--r--r-- 1 zimbra zimbra 2126 2007-11-17 14:47 25_domainkeys.cf
-r--r--r-- 1 zimbra zimbra 2929 2007-11-17 14:47 25_hashcash.cf
-r--r--r-- 1 zimbra zimbra 1310 2007-11-17 14:47 25_pyzor.cf
-r--r--r-- 1 zimbra zimbra 3389 2007-11-17 14:47 25_razor2.cf
-r--r--r-- 1 zimbra zimbra 7639 2007-11-17 14:47 25_replace.cf
-r--r--r-- 1 zimbra zimbra 2901 2007-11-17 14:47 25_spf.cf
-r--r--r-- 1 zimbra zimbra 1768 2007-11-17 14:47 25_textcat.cf
-r--r--r-- 1 zimbra zimbra 7615 2007-11-17 14:47 25_uribl.cf
-r--r--r-- 1 zimbra zimbra 29745 2007-11-17 14:47 30_text_de.cf
-r--r--r-- 1 zimbra zimbra 22164 2007-11-17 14:47 30_text_fr.cf
-r--r--r-- 1 zimbra zimbra 1858 2007-11-17 14:47 30_text_it.cf
-r--r--r-- 1 zimbra zimbra 23592 2007-11-17 14:47 30_text_nl.cf
-r--r--r-- 1 zimbra zimbra 19402 2007-11-17 14:47 30_text_pl.cf
-r--r--r-- 1 zimbra zimbra 3331 2007-11-17 14:47 30_text_pt_br.cf
-r--r--r-- 1 zimbra zimbra 49534 2007-11-17 14:47 50_scores.cf
-r--r--r-- 1 zimbra zimbra 1304 2007-11-17 14:47 60_awl.cf
-r--r--r-- 1 zimbra zimbra 2772 2007-11-17 14:47 60_shortcircuit.cf
-r--r--r-- 1 zimbra zimbra 5147 2007-11-17 14:47 60_whitelist.cf
-r--r--r-- 1 zimbra zimbra 2534 2007-11-17 14:47 60_whitelist_dk.cf
-r--r--r-- 1 zimbra zimbra 2558 2007-11-17 14:47 60_whitelist_dkim.cf
-r--r--r-- 1 zimbra zimbra 3584 2007-11-17 14:47 60_whitelist_spf.cf
-r--r--r-- 1 zimbra zimbra 1914 2007-11-17 14:47 60_whitelist_subject.cf
-r--r--r-- 1 zimbra zimbra 119702 2007-11-17 14:47 72_active.cf
-r--r--r-- 1 zimbra zimbra 14771 2007-11-17 14:47 active.list
-r--r--r-- 1 zimbra zimbra 948 2007-11-17 14:47 init.pre
-r--r--r-- 1 zimbra zimbra 101479 2007-11-17 14:47 languages
-r--r--r-- 1 zimbra zimbra 1206 2007-11-17 14:47 local.cf
-r--r--r-- 1 zimbra zimbra 2762 2007-11-17 14:47 regression_tests.cf
-r--r--r-- 1 zimbra zimbra 3304 2007-11-17 14:47 sa-update-pubkey.txt
-r--r--r-- 1 zimbra zimbra 60029 2007-11-17 14:47 STATISTICS-set0.txt
-r--r--r-- 1 zimbra zimbra 69854 2007-11-17 14:47 STATISTICS-set1.txt
-r--r--r-- 1 zimbra zimbra 60035 2007-11-17 14:47 STATISTICS-set2.txt
-r--r--r-- 1 zimbra zimbra 69863 2007-11-17 14:47 STATISTICS-set3.txt
-r--r--r-- 1 zimbra zimbra 1869 2007-11-17 14:47 user_prefs.template
-r--r--r-- 1 zimbra zimbra 2254 2007-11-17 14:47 v310.pre
-r--r--r-- 1 zimbra zimbra 922 2007-11-17 14:47 v312.pre
-r--r--r-- 1 zimbra zimbra 2067 2007-11-17 14:47 v320.pre

Having printed that out. The fact that the file dates are all from Nov of last year point out some obvious issues. I guess I thought the updates of the spam system were automatic but after going back into the Zimbra admin system I see that it's just the "Antivirus Settings" that update every 2hrs and does not include antispam.

I'm going to look through the threads you linked to and see if I can determine what I need to do to keep the AS portion of Zimbra up to date. Is there no automated way for this to happen? (I may find existing answers to this in my search which begins after I click Submit Reply!)

As always thanks for your help.

[phoenix]

I think my comment about the rules update might have been slightly misleading, we only supply the rules with each upgrade and there's no automatic update withing Zimbra.

There used to be a script called rules_du_jour but that seems to have died, you can run saupdate. Have a look at this thread for some details: Rules_Du_Jour

Did you check the spam/ham accounts were set correctly?

[uxbod]

Definitely look at implementing the SARE rules plus Justin Mason's auto-generated rules are extremely good from my testing :- taint.org: Justin Mason’s Weblog » Test my auto-generated ruleset

[rajahd]

Ok I did verify the spam/ham accounts exist so I think that's not an issue.

zimbra@email:~$ zmprov gacf | grep SpamAccount
zimbraSpamIsNotSpamAccount: ham.y0r2okm9es@domain.com
zimbraSpamIsSpamAccount: spam.1tphulj3@comain.com

SpamAssassin seems to be extremely flexible and powerful but from the novice Linux/e-mail sysadmin point of view it's far from intuitive. I'm wading through the links you both made now to see about getting the spam rules more up to date and ways to keep them up to date.

I haven't searched bugzilla yet but surely an automated spam rule updating system is on the drawing board for Zimbra. I guess an alternate option would be to incorporate a 3rd party spam pre-filtering system (Abaca, etc).

Thanks for the continued help!

Edit: Also, just wanted to make sure I understand what you said on the current AS rule set updates. The server that I upgraded to 5.0.4 we get a *lot* less spam on and this is because is has much newer rules, right? Meanwhile the 4.5.x servers (which is where we're seeing the large increases in spam) are lanquishing under older AS rulesets, hence the need to either upgrade Zimbra or follow one of the threads to get SpamAssassin rules updated.

[uxbod]

Originally Posted by rajahd

I guess an alternate option would be to incorporate a 3rd party spam pre-filtering system (Abaca, etc)

There is no reason why you could not install another system in-front of you ZCS server for that purpose (as I already do that) but Amavis & SA are more than capable.

Originally Posted by rajahd

Edit: Also, just wanted to make sure I understand what you said on the current AS rule set updates. The server that I upgraded to 5.0.4 we get a *lot* less spam on and this is because is has much newer rules, right? Meanwhile the 4.5.x servers (which is where we're seeing the large increases in spam) are lanquishing under older AS rulesets, hence the need to either upgrade Zimbra or follow one of the threads to get SpamAssassin rules updated.

You are indeed correct.

[rajahd]

We'll eventually get the 4.5.x boxes updated, just waiting for some outstanding issues in 5.x to get resolved. I fully agree that Amavis & SA are more than sophisticated enough to do the job it's just my available "tinker time" (which is required to handle the learning curve I need since I still struggle in the Linux world) has dwindled over the last year. I'll continue down the various paths and something will pan out I'm sure.

Thanks uxbod!

[LMStone]

Originally Posted by phoenix

I think my comment about the rules update might have been slightly misleading, we only supply the rules with each upgrade and there's no automatic update withing Zimbra.

There used to be a script called rules_du_jour but that seems to have died, you can run saupdate. Have a look at this thread for some details: Rules_Du_Jour

Did you check the spam/ham accounts were set correctly?

FWIW, we find RulesDuJour to be very helpful.

Here's our /etc/rulesdujour/config file:

TRUSTED_RULESETS="TRIPWIRE SARE_BML SARE_FRAUD SARE_OEM SARE_STOCKS SARE_BAYES_POISON_NXM SARE_EVILNUMBERS0 SARE_EVILNUMBERS1 SARE_REDIRECT_POST300 SARE_HTML0 SARE_HTML1 SARE_HTML_ENG SARE_HEADER0 SARE_HEADER1 SARE_SPECIFIC SARE_ADULT SARE_GENLSUBJ0 SARE_GENLSUBJ1 SARE_UNSUB SARE_URI0 SARE_URI1 SARE_OBFU0 SARE_OBFU1 SARE_SPOOF SARE_RANDOM"
SA_DIR="/opt/zimbra/conf/spamassassin"
RULES_DU_JOUR_SCRIPT="/usr/sbin/rules_du_jour"
MAIL_ADDRESS="thishasbeenchanged@reliablenetworks. com"
SA_RESTART="/opt/zimbra/bin/zmamavisdctl restart"
SA_LINT=" "

Hope that helps,
Mark