Results 1 to 2 of 2

Thread: Initial SPAM training ramp up

  1. #1
    pgienger is offline Active Member
    Join Date
    Apr 2007
    Posts
    32
    Rep Power
    8

    Default Initial SPAM training ramp up

    Please bear with the wall of text, I'm merely being complete as I've tried a few things here...

    We're about 4 weeks into our demo license while we're waiting for accounting to purchase the real thing and for the most part very happy with Zimbra with one exception - the SPAM filter really is performing quite horridly, at least compared to our old system (DSPAM on a OSS stack).

    I'm wondering what other people do to improve the built in performance, short of tacking on various other packages as stated in the wiki. I would like to keep the system as close to the packaged system as possible, since we're going to be paying for it, I'd like to think that 'whats in the box' should be all we'd need, and I don't really enjoy the thought of having all the parts to tweak. That's one reason we're going to a package after all and not tossing an edge filter in... yet.

    We've had about 15% of our user pool on Zimbra while the other 85% sit on the old system as we iron out the bugs. Among these users (about 25) there are a few that get between 30-75 messages a day that wind up in their inbox. They are attempting to train the filters but just not seeing any improvement. I see about 10-15 a night myself and maybe another 5 during the day. Compare that with the legacy system that was running around 5 false negatives a week for most people.

    Here are some things I have tried so far with not a lot of success.
    - I initially thought that to train I should probably toss the users' Inbox and Junk at the filter as ham/spam, so I ran
    Code:
    /opt/zimbra/bin/zmtrainsa  fqhn user pass spam Junk
    /opt/zimbra/bin/zmtrainsa  fqhn user pass ham Inbox
    on most of the users. I ran into a couple of issues with this. First, after a while the system just stopped filtering for people. I would basically deadlock and then I had to work some magic to get the process to even complete. As far as I can tell it wasn't on a large message or anything, something just got hosed. Something in this process went horribly wrong also since overnight the MTA basically died. Besides the issues, this didn't seem to improve the accuracy for the people that the process had run through on. I concluded that this wasn't something we could use as a step in migration.

    One hitch here is that I can see if it's getting confused because the hostname is pat.fqdn while some things are referring to it as a cname of webmail.

    I've since tried to turn on DSPAM. From what I can gather our system shouldn't get killed by the load as it's pretty large for our size environment. I've done the following:
    - enabled the 3 lines in crontab, ignoring the 'dont edit me' warnings:
    Code:
    0 1 * * * find /opt/zimbra/dspam/var/dspam/data/z/i/zimbra/zimbra.sig/ -type f -name \*sig -mtime +7 -exec rm {} \; > /dev/null 2>&1
    8 4 * * * /opt/zimbra/dspam/bin/dspam_logrotate -a 60 /opt/zimbra/dspam/var/dspam/system.log
    8 8 * * * /opt/zimbra/dspam/bin/dspam_logrotate -a 60 /opt/zimbra/dspam/var/dspam/data/z/i/zimbra/zimbra.log
    uncommented anything in zmtrainsa relating to dspam
    Code:
        for f in ${spamdir}/*; do
            test -f ${f} && ${zimbra_home}/dspam/bin/dspam_corpus --addspam zimbra ${f}
        done
        for f in ${hamdir}/*; do
            test -f ${f} && ${zimbra_home}/dspam/bin/dspam_corpus zimbra ${f}
        done
    
        /opt/zimbra/dspam/bin/dspam_clean -p0 $USER
    and
    Code:
    echo "Training DSPAM"
    
    /usr/bin/fetchmail -f ${zimbra_home}/conf/.fetchmailrc -n -a -u $USER -r "$FOLDER" -m "/opt/zimbra/dspam/bin/dspam --user zimbra --class=${MODE} --source=corpus --mode=teft --feature=chained,noise --stdout" ${SERVER}
    
    /opt/zimbra/dspam/bin/dspam_clean -p0 $USER
    and I believe this was blank, which I edited:
    amavisd.conf.in:
    $dspam = '/opt/zimbra/dspam';

    That line was initially pointing to the dspam binary, but then I noticed that things were getting stuck in deferred so I re-edited it to point to the dspam root and now junk is passing through, well at least not visibly hanging.

    I guess at this point it's pretty open ended as to what I'm looking for. I'd like to get a lot closer to where we were before I go moving executive types over but with the current performance I don't see it happening soon. Can I expect things to improve quickly now that DSPAM seems to be processing better? Our previous admin was pretty negative on SA and our DSPAM worked well which is why I'm pushing in that direction. Should I think about tossing people's current Junk and Inboxes through the train filter when I migrate them as a one-time catch up?

    One last question is as to how these filters actually learn. In our previous setups things learned quite obviously on a per-user basis. Now it seems to be that dspam at least is learning as zimbra. Will they be able to pick up per-user info? A perfect example is somebody who gets RHN notices junks them, while the other 3 admins who get that account would value them greatly. I don't want her screwing up the rest of our universe.
    Last edited by pgienger; 05-08-2007 at 02:32 PM. Reason: fields bracketed with greater-than, etc. disappeared

  2. #2
    auanton is offline Intermediate Member
    Join Date
    Mar 2007
    Location
    bz italy
    Posts
    23
    Rep Power
    8

    Default

    hi, newbie here:

    trying to follow your steps.

    but i noted this:

    "and I believe this was blank, which I edited:
    amavisd.conf.in:
    $dspam = '/opt/zimbra/dspam';"


    here the /var/log/zimbra.log continued to say:

    "May 20 21:25:12 test amavis[31950]: No $dspam, not using it"


    until i changed the line to the original. now it says:

    "May 20 22:01:00 test amavis[6747]: Found $dspam at /opt/zimbra/dspam/bin/dspam"

    anton
    ( still waiting for dspam-headers to appear in the junk-folder )
    edit: well, it appeared... looks like working!

    would some expert out there give an opinion on the following ?

    1) headers now, that dspam is there:

    X-DSPAM-Result: Spam
    X-DSPAM-Confidence: 0.9996
    X-DSPAM-Probability: 1.0000
    X-DSPAM-Signature: 4650abf7164831287611476
    X-DSPAM-Factors: 15,
    X-Virus-Scanned: amavisd-new at
    X-Spam-Flag: YES
    X-Spam-Score: 5.854
    X-Spam-Level: *****
    X-Spam-Status: Yes, score=5.854 tagged_above=-10 required=4 tests=[AWL=-7.018,
    BAYES_99=4.3, DSPAM_SPAM=1.5, HTML_50_60=0.134,
    HTML_FONT_LOW_CONTRAST=0.194, HTML_MESSAGE=0.001,
    MIME_HTML_ONLY=0.001, SPOOF_OURI=0.104, URIBL_SC_SURBL=4.498,
    URIBL_WS_SURBL=2.14]


    2) headers with dspam not active: (dofferent msg. )

    X-Virus-Scanned: amavisd-new at
    X-Spam-Flag: YES
    X-Spam-Score: 10.675
    X-Spam-Level: **********
    X-Spam-Status: Yes, score=10.675 tagged_above=-10 required=4
    tests=[AWL=-8.678, BAYES_50=0.001, HTML_50_60=0.134,
    HTML_MESSAGE=0.001, RAZOR2_CF_RANGE_51_100=0.5,
    RAZOR2_CF_RANGE_E8_51_100=1.5, RAZOR2_CHECK=2.5,
    RCVD_IN_NJABL_DUL=1.946, RCVD_IN_SORBS_DUL=2.046,
    URIBL_JP_SURBL=4.087, URIBL_SC_SURBL=4.498, URIBL_WS_SURBL=2.14]


    why in the first on there is no mention of RAZOR2 ???

    thanks in advance

    anton
    Last edited by auanton; 05-20-2007 at 01:24 PM.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Trying to understand Zimbra's anti-spam system
    By TaskMaster in forum Users
    Replies: 11
    Last Post: 01-25-2008, 09:59 AM
  2. Deleted spam training accounts by fault
    By karmek in forum Administrators
    Replies: 6
    Last Post: 07-13-2007, 05:05 AM
  3. How to check if spam training is working?
    By tbovingdon in forum Administrators
    Replies: 1
    Last Post: 03-13-2007, 05:57 AM
  4. Training spam and ham
    By Justin in forum Developers
    Replies: 2
    Last Post: 10-31-2006, 03:39 PM
  5. Spam training has no cron job
    By richard-hdd in forum Administrators
    Replies: 3
    Last Post: 09-13-2006, 11:50 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •