Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Spam questions 3.11

  1. #1
    cdyer is offline Junior Member
    Join Date
    May 2006
    Posts
    8
    Rep Power
    8

    Default Spam questions 3.11

    I'm confused on the best practices wrt spam blocking in v 3.11 - it seems like many of the suggestions on the forums are incorporated into the Zimbra feature set now and do not need to be done manually, but I'm not sure. Correction/clarification/answers on these assumptions/questions is appreciated.

    1. DSPAM and SA are two separate tools that work together to detect SPAM. both are incorporated into Zimbra without additional steps.
    2. Should DSPAM alone should catch some spam without any SA training?
    3. Will messages flagged by DSPAM into the junk mail or will they just be blackholed?
    4. In this latest Zimbra version what is the best way to train SA? I see references to various methods but it is unclear which are relevant to the latest release.
    5. How can I identify HAM if the messages are not identified as SPAM in the first place (i.e. if they don't make it into the Junk folder I can't click "Not Junk" to get them out which I believe is what tells Zimbra the message is ham).
    6. What log file will show me DSPAM and SA activity?

    Any insights are appreciated...

    Chad

    Example header that did not get marked as SPAM but was (I think) obviously SPAM
    Code:
    Received: from localhost (localhost [127.0.0.1])
    	by zimbra.dyers.net (Postfix) with ESMTP id ACABC10A1E8;
    	Thu, 18 May 2006 10:19:33 -0700 (PDT)
    Received: from zimbra.dyers.net ([127.0.0.1])
     by localhost (zimbra.dyers.net [127.0.0.1]) (amavisd-new, port 10024)
     with ESMTP id 07036-06; Thu, 18 May 2006 10:19:16 -0700 (PDT)
    Received: from 1841BB60 (p3211-ipad31hodogaya.kanagawa.ocn.ne.jp [220.107.234.211])
    	by zimbra.dyers.net (Postfix) with SMTP id 69C6010A1DB
    	for <chad@dyers.net>; Thu, 18 May 2006 10:19:15 -0700 (PDT)
    Received: from localhost ([199.181.132.21] helo=lwaxana.til.UFPE.BR)
    	by smtp6.cistron.nl with esmtp (Exim 2.70 #1 (til))
    	id 9AHg8h-8214bE-00; Thu, 18 May 2006 10:19:31 -0800
    Date: Thu, 18 May 2006 10:19:31 -0800
    From: "AnalysisMort" <Home.Analysis@ling.gu.se>
    To: chad@dyers.net
    Subject: NewBank
    Message-Id: <200410031437.i93NczTw006093@www5.gmail.com>
    X-DSPAM-Result: Innocent
    X-DSPAM-Processed: Thu May 18 10:19:16 2006
    X-DSPAM-Confidence: 0.6176
    X-DSPAM-Probability: 0.0000
    X-DSPAM-Signature: 446cac94178341986810238
    X-DSPAM-Factors: 27,
    X-Virus-Scanned: amavisd-new at 
    X-Spam-Status: No, score=-0.84 tagged_above=-10 required=5 autolearn=ham
     tests=[BAYES_20=-0.74, DSPAM_HAM=-0.1]
    X-Spam-Score: -0.84
    X-Spam-Level:

  2. #2
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,201
    Rep Power
    56

    Default

    Hi

    Welcome to the forums.

    The only modification that needs to be made in Zimbra for spam is the kill/tag percentages.

    Ok, your specific questions:

    1. Yes
    2. Yes
    3. They should end up in the Junk folder until DSPAM has been trained.
    4. You don't need to do anything manually to train SA, there's a cron job that runs daily that runs zmtrainsa (you can also run that manually to train for spam/ham.
    5. I'm not quite sure what you want in these circumstances. Anything in the Inbox is treated as Ham, if it's not you should only have the option of calling it Spam.
    6. /var/log/zimbra.log will show messages that have been detected as spam.

    There are additional Bayes rules that you could add into SA, there also a 'rules_du_jour' tutorial that will update your rules daily - that will add to the effectiveness of SA. The only problem you have to be aware of is that running lots of additional Bayes tests will add to the mail processing overhead. In a busy system that could be prohibitive.

    The headers you've posted do appear to be from a Spam mail but, as I mentioned at the beginning, you probably need to adjust your kill/tag percentages. The numbers need to be moved 'down' to catch/mark more spam. I've said before that catching spam is a juggling act between getting as much spam as you can (you'll never get it all) and getting no false positives - you don't want important mail deleted as spam. I have my kill/tag set at 66/26 respectively and I see about 1 spam in my inbox and a couple in the junk folder per month.

    Marking your example mail as junk when it's in your inbox will get that message trained as spam for future rejection and it should be moved automatically be moved into the junk folder in fuuture, depending on any changes you've made to kill/tag percentages.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  3. #3
    PNE
    PNE is offline Loyal Member
    Join Date
    Mar 2006
    Location
    Czech Republic
    Posts
    81
    Rep Power
    9

    Default

    Well, this is my personal opinion. I'm still running 3.1.0 but there will not be difference I suppose.

    1. Yes
    2. DSPAM is statistical analysis. It will not catch any SPAM until it get trained. Moreover, I do not think that DSPAM alone will move spam to junk folder. It just adds some points (just 0.5 by Zimbra default) to SA total score.
    3. See 2.
    4. Good question. Sure you can use built-in web based training, but it is quite exhausting for me. Especially if your zimbra is production one, you will depend on how your users will do the training and this is for sure not good. Zimbra alone does some kind of auto training - messages with sufficient total score end in junk and junk is used in nightly training job. This is good but not quite sufficient, if your users will not do their training part with spam that got to the inbox. BTW, how will you force users to train spam/ham if they use POP3? It would be great if zimbra had shared folders and users could easily move spam/ham to special shared folders used for training, or if training could be based on message IDs send to special location. What do you think of it, Zimbra folks?
    5. My another question to zimbra folks - does zimbra use SA auto training? Is the bayes_auto_learn switch working? And you are right cdyer, bayes needs to be trained both spam and ham to be effective. See http://wiki.apache.org/spamassassin/BasicConfiguration, bayes_auto_learn.
    6. Phoenix is right.

    I made myself some windows/samba app that uses my mail archive (see postfix always_bcc). From the archive it reads headers info from all messages and stores it into database. (also SA and DSPAM scores among others). Then in tab I can clearly see messages and their properties, I can open them in notepad to check them and I can copy messages to special disk folders and then use them for automatic SA/DSPAM training. This training is based on cron, zmtrainsa and sa-learn. I can sort and filter messages by SA score, so I can (for example) every day easily check messages with score from 2 to 5.6, which are kind of suspicious and appropriately train on them. I see that there is some space for zimbra team to make similar feature, I'm not able do it in linux.

    Personally, I use 5.6 SA threshold (28 in zimbra). After some training, I set both BAYES_99 and DSPAM spam scores to 4.4 - this is really better than to set total threshold too low because of false positives. Also it seems to me, that DSPAM almost never makes false positives so it is really good to increase its weight. Of course SA networks test are crucial too just as additional SA rulesets from www.rulesemporium.com. Now I see almost no spam. I'm still quite new to linux, but there is lot of documentation about SA, anyone can tailor it to his needs I think.

    And to the Zimbra team, I really appreciate your product, keep up the good work, waiting eagerly for new features!

  4. #4
    cdyer is offline Junior Member
    Join Date
    May 2006
    Posts
    8
    Rep Power
    8

    Default

    Thanks for the replies. I already have my kill levels at 66/25, and we are geting a fair amount of spam in our inboxes. But we only have 3 users right now and have been running for less than a week.

    It sounds like I should keep the kill levels where they are and continue to have everyone mark their spam for a while so the system can get trained. If after a couple weeks of training there is still a fair amount of spam in the inboxes I should experiment with the kill levels.

    Clarification on the #5 ham training question. My confusion is that I thought zmtrainsa trained the system what is spam and what is ham by checking the spam/ham *mailboxes*. I believe these mailboxes get populated when users correct false positives in the junk folder and and missed spam in the inbox using the junk/not junk buttons. Hence if I never get a false positive, never click "not junk," the ham *mailbox* would be empty and not useful for training.

    Phoenix, are you saying that for the purposes of training SA items in the inbox are used as examples of ham?

  5. #5
    KevinH's Avatar
    KevinH is offline Expert Member
    Join Date
    Aug 2005
    Location
    San Mateo, CA
    Posts
    4,789
    Rep Power
    18

    Default

    Quote Originally Posted by cdyer
    Phoenix, are you saying that for the purposes of training SA items in the inbox are used as examples of ham?
    No we just use the 'not junk' buttons to train for ham. If you'd like post some of you SPAM headers here for us to take a look. May just be a mis-config somewhere.
    Looking for new beta users -> Co-Founder of Acompli. Previously worked at Zimbra (and Yahoo! & VMware) since 2005.

  6. #6
    cdyer is offline Junior Member
    Join Date
    May 2006
    Posts
    8
    Rep Power
    8

    Default Examples

    Thanks for taking a look: all these were in inboxes....

    Example 1:
    Code:
    Received: from localhost (localhost [127.0.0.1])
    	by zimbra.dyers.net (Postfix) with ESMTP id D2D7C167CE6;
    	Sun, 21 May 2006 13:16:13 -0700 (PDT)
    Received: from zimbra.dyers.net ([127.0.0.1])
     by localhost (zimbra.dyers.net [127.0.0.1]) (amavisd-new, port 10024)
     with ESMTP id 17988-10; Sun, 21 May 2006 13:15:56 -0700 (PDT)
    Received: from 1749C858 (unknown [85.129.240.60])
    	by zimbra.dyers.net (Postfix) with SMTP id B30DB167C9D;
    	Sun, 21 May 2006 13:14:44 -0700 (PDT)
    Received: from smtp.x719.net (helo=smtp.lapidary.net)
    	by smtp2lapidary.nl with lsmtp (Exim 3.35 #8 (Debian))
    	id 1AGZO1-0000d1-00
    Date: Sun, 21 May 2006 13:14:52 -0800
    From: "Rolling.Casino" <888_Vegas@kriss.re.kr>
    
    Message-Id: <7801232355.ZM374728@sgaxaf.harvard.edu>
    To: chad@dyers.net
    Subject: Vegas.Money
    In-Reply-To: "888_Vegas@kriss.re.kr" <888_Vegas@kriss.re.kr>
    X-DSPAM-Result: Spam
    X-DSPAM-Processed: Sun May 21 13:15:56 2006
    X-DSPAM-Confidence: 0.5769
    X-DSPAM-Probability: 1.0000
    X-DSPAM-Signature: 4470ca7c160179080858887
    X-DSPAM-Factors: 15,
    X-Virus-Scanned: amavisd-new at 
    X-Spam-Status: No, score=1.501 tagged_above=-10 required=5 autolearn=no
     tests=[BAYES_60=1, DSPAM_SPAM=0.5, UNPARSEABLE_RELAY=0.001]
    X-Spam-Score: 1.501
    X-Spam-Level: *
    Example 2:
    Code:
    Received: from localhost (localhost [127.0.0.1])
    	by zimbra.dyers.net (Postfix) with ESMTP id F0B3016370E;
    	Sun, 21 May 2006 10:07:08 -0700 (PDT)
    Received: from zimbra.dyers.net ([127.0.0.1])
     by localhost (zimbra.dyers.net [127.0.0.1]) (amavisd-new, port 10024)
     with ESMTP id 03287-10; Sun, 21 May 2006 10:06:51 -0700 (PDT)
    Received: from 044.mx02.net (044.mx02.net [69.6.10.44])
    	by zimbra.dyers.net (Postfix) with ESMTP id 0B0E1163701
    	for <duane@dyers.net>; Sun, 21 May 2006 10:06:50 -0700 (PDT)
    Received: (from daemon@localhost)
    	by 044.mx02.net (8.8.8/8.8.8) id JAA67736;
    	Sun, 21 May 2006 09:06:33 -0700 (PDT)
    Date: Sun, 21 May 2006 10:01:41 -0700 (PDT)
    Message-Id: <200605211606.JAA67736@044.mx02.net>
    From: AuthenticDesignerBags <CustomerResponse@044.mx02.net>
    To: duane@dyers.net
    Subject: Get your complimentary* Gucci or Burberry designer handbag straight off the runway
    MIME-Version: 1.0
    Content-Type: text/plain; charset="iso-8859-1"
    Content-Transfer-Encoding: quoted-printable
    X-DSPAM-Result: Innocent
    X-DSPAM-Processed: Sun May 21 10:06:51 2006
    X-DSPAM-Confidence: 0.9996
    X-DSPAM-Probability: 0.0000
    X-DSPAM-Signature: 44709e2b242845209328925
    X-DSPAM-Factors: 27,
    X-Virus-Scanned: amavisd-new at 
    X-Spam-Status: No, score=-0.099 tagged_above=-10 required=5 autolearn=ham
     tests=[BAYES_50=0.001, DSPAM_HAM=-0.1]
    X-Spam-Score: -0.099
    X-Spam-Level:
    Example 3:
    Code:
    Received: from localhost (localhost [127.0.0.1])
    	by zimbra.dyers.net (Postfix) with ESMTP id 1CB1B16030B;
    	Sun, 21 May 2006 07:38:48 -0700 (PDT)
    Received: from zimbra.dyers.net ([127.0.0.1])
     by localhost (zimbra.dyers.net [127.0.0.1]) (amavisd-new, port 10024)
     with ESMTP id 24655-09; Sun, 21 May 2006 07:38:30 -0700 (PDT)
    Received: from yhaz.mylink2thenet.com (yhaz.mylink2thenet.com [209.9.156.134])
    	by zimbra.dyers.net (Postfix) with ESMTP id ED02F1602FB
    	for <duane@dyers.net>; Sun, 21 May 2006 07:38:29 -0700 (PDT)
    Message-ID: <96876.176227736.1148210583@mylink2thenet.com>
    Date: 21 May 2006 10:38:41 -0400
    From: "Pink Laptop  " <n.rolle@mylink2thenet.com>
    Subject: Get your Pink Ergo® Ensis S Laptop now, no cords attached.
    To: duane dyer <duane@dyers.net>
    MIME-Version: 1.0
    Content-Type: multipart/alternative;
    	boundary="176227736.1148210583"
    X-DSPAM-Result: Innocent
    X-DSPAM-Processed: Sun May 21 07:38:30 2006
    X-DSPAM-Confidence: 0.9997
    X-DSPAM-Probability: 0.0000
    X-DSPAM-Signature: 44707b6685971644115261
    X-DSPAM-Factors: 27,
    X-Virus-Scanned: amavisd-new at 
    X-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char AE hex) in message header 'Subject': Subject: Get your Pink Ergo\256 Ensis S Laptop...
    X-Spam-Status: No, score=4.577 tagged_above=-10 required=5 autolearn=no
     tests=[AWL=0.600, BAYES_99=3.5, DSPAM_HAM=-0.1, HTML_90_100=0.113,
     HTML_IMAGE_RATIO_02=0.463, HTML_MESSAGE=0.001]
    X-Spam-Score: 4.577
    X-Spam-Level: ****

  7. #7
    KevinH's Avatar
    KevinH is offline Expert Member
    Join Date
    Aug 2005
    Location
    San Mateo, CA
    Posts
    4,789
    Rep Power
    18

    Default

    Looks like in all cases more training will help catch these. Do you have the RBL's enabled? Not sure but those usually catch the popular spammers. Any gateway/proxy MTA in the network? Or does Zimbra's postfix get mail directly from the internet?
    Looking for new beta users -> Co-Founder of Acompli. Previously worked at Zimbra (and Yahoo! & VMware) since 2005.

  8. #8
    cdyer is offline Junior Member
    Join Date
    May 2006
    Posts
    8
    Rep Power
    8

    Default

    So wrt training my original confusion was: if I never get false positives, so I can thereby never click "not junk", is my training sub-optimal since I am only training the system what is spam and not what is ham?

    I just added a couple RBLs and also noticed one of the default ones was not enabled, so we'll see how it goes.

  9. #9
    KevinH's Avatar
    KevinH is offline Expert Member
    Join Date
    Aug 2005
    Location
    San Mateo, CA
    Posts
    4,789
    Rep Power
    18

    Default

    In general training with spam is good enough. You only need the 'not junk'/ham to correct if/when you get a false positive.
    Looking for new beta users -> Co-Founder of Acompli. Previously worked at Zimbra (and Yahoo! & VMware) since 2005.

  10. #10
    PNE
    PNE is offline Loyal Member
    Join Date
    Mar 2006
    Location
    Czech Republic
    Posts
    81
    Rep Power
    9

    Default

    Kevinh, sorry to not agree, see http://wiki.apache.org/spamassassin/BayesInSpamAssassin. Bayes needs to be trained on both spam and ham. So it is quite surprise for me that ham isn't trained some kind automatically, as I asked in point 5 in my post above. Can you really confirm it?

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Trying to understand Zimbra's anti-spam system
    By TaskMaster in forum Users
    Replies: 11
    Last Post: 01-25-2008, 09:59 AM
  2. Spam Filter - a few questions
    By sternfan in forum Administrators
    Replies: 12
    Last Post: 08-08-2007, 12:12 PM
  3. Spam being scored with BAYES_00
    By flyerguybham in forum Administrators
    Replies: 6
    Last Post: 04-24-2007, 12:07 PM
  4. clam and spam questions
    By reckless2k2 in forum Installation
    Replies: 2
    Last Post: 03-12-2007, 08:47 AM
  5. Training spam and ham
    By Justin in forum Developers
    Replies: 2
    Last Post: 10-31-2006, 03:39 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •