Page 1 of 3 123 LastLast
Results 1 to 10 of 24

Thread: Spam/Ham training under Outlook/Thunderbird/etc.

  1. #1
    chuckm's Avatar
    chuckm is offline Junior Member
    Join Date
    Aug 2006
    Location
    Vancouver, BC, Canada
    Posts
    7
    Rep Power
    8

    Default Spam/Ham training under Outlook/Thunderbird/etc.

    Having just joined the Zimbra forums, I would like to express my sincere appreciation for the information available here. It got me through the intial difficulties in understanding some of the tricky/obscure issues related to getting Zimbra running smoothly and cleanly. It's now up and running well on CentOS under VMWare. My thanks to the members, and to the Zimbra employees who so generously offer their advice.

    I have a question or two (or three...) related to spam training. Searching the forums has not produced anything I can use so far, but I may have overlooked something ... flames are ok, I'm still on the steep part of the learning curve

    The folks I'm supporting are Outlook and Thunderbird users (me included) who wish to continue using those tools rather than switching to the Zimbra web interface. We all monitor multiple IMAP servers which the web client is not designed to do.

    I'd like to make it as easy as possible for users to train the spam filter, so my current work-around is to create special folders in each user's mailbox clearly labeled for spam and ham training. The users simply drag messages into either of these folders, and a script runs each night to invoke 'zmtrainsa' against the folders. This seems like the most intuitive way to do it, involving the least effort on the part of the users and offering a relatively MUA-independent method.

    However, there are problems with this. Whenever a user changes his/her password in Zimbra, I have to ask them for it and edit the training script. This exposes passwords not only in the script but also in the Zimbra log file, something I would really rather not do. Some trusted users have root privileges and are free to review log files.

    The LDAP server stores these passwords in SSHA format, but of course fetchmail can't deal with them when passed in via the zmtrainsa script. Is there any possibility that future versions of zmtrainsa can be modified to query the LDAP database (if, and only if, run as userid 'zimbra') in order to avoid admin hassles with passwords? Can the password string sent to fetchmail be suppressed/obscured in the log files?

    Failing this, other suggestions for making it trivially easy for Outlook/Thunderbird users to invoke spam/ham training would be most welcome. Forwarding spam or ham messages to the special training account mailboxes is not so intuitive, and most users would quickly tire of it. A button-click or drag-and-drop method is much preferred.

    Many thanks!
    Chuck
    {Lead me not into temptation ... I can find it myself, thank you}

  2. #2
    bobby is offline Zimbra Employee
    Join Date
    Nov 2005
    Posts
    518
    Rep Power
    10

    Default

    have them forward as attachment messages as necessary to the spam/ham accounts. from there they get trained automatically

  3. #3
    chuckm's Avatar
    chuckm is offline Junior Member
    Join Date
    Aug 2006
    Location
    Vancouver, BC, Canada
    Posts
    7
    Rep Power
    8

    Default

    Yes, that was my original thought. I assumed that is one of the primary functions of those two email accounts.

    My aim is to make the process even easier, so that the user can simply drag spam or ham to special folders which are examined by a cron job each night. I have scripted this training process, and it seems to work well although adding a new user account requires manual addition of the two folders. The script simply runs thru all users looking for stuff they have dragged into the special folders. The only problem is that I have no idea how to extract account passwords (or have zmtrainsa do it for me). The current method requires that I maintain a small file with usernames and passwords for my script to parse. When users change their passwords, I have to ask them for it. That's ok for 10 or 20 users, but it gets problematic beyond that.

    Chuck
    {Lead me not into temptation ... I can find it myself, thank you}

  4. #4
    kogo is offline Active Member
    Join Date
    Nov 2006
    Location
    Chicago
    Posts
    35
    Rep Power
    8

    Default

    Quote Originally Posted by bobby View Post
    have them forward as attachment messages as necessary to the spam/ham accounts. from there they get trained automatically
    Outlook does not seem to have a 'Forward as Attachment' option. Should my users create a new message to the spam account and drag the message in question to the attachments before sending?

    If so, can you attach many messages as attachments to one email to the spam account, or do you have to do it one message attachment at a time?

    TIA

  5. #5
    phoenix is offline Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,201
    Rep Power
    56

    Default

    Quote Originally Posted by kogo View Post
    Outlook does not seem to have a 'Forward as Attachment' option.
    I don't quite think that's correct, have a look here.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  6. #6
    kogo is offline Active Member
    Join Date
    Nov 2006
    Location
    Chicago
    Posts
    35
    Rep Power
    8

    Default

    Ah... a non-default setting. Very good.

    How about my second questions - can i attach all my spam to one message, or does it have to be one at a time?

  7. #7
    phoenix is offline Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,201
    Rep Power
    56

    Default

    As far as I know, you can send multiple attachments to spam folders on a single email.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  8. #8
    chuckm's Avatar
    chuckm is offline Junior Member
    Join Date
    Aug 2006
    Location
    Vancouver, BC, Canada
    Posts
    7
    Rep Power
    8

    Thumbs up Latest release of Zimbra makes this easier!

    It's been a while since I visited this forum, but I though I'd post an update for the solution to my problem of last year ...

    I decided it was time to update my Zimbra installations, and the latest release of Zimbra presented me with a nice surprise ... 'zmtrainsa' no longer requires the user password when asking it to look at messages. Now my shell script simply requires a list of user names to examine, so if the user changes his/her password it's not a problem.

    The solution I still use is:

    - Create two folders in each user's account when you add the user, one named "Is_SPAM" and one named "IsNot_SPAM". These are simply names chosen for their obvious function, so use any names you see fit.

    - When a user encounters a message which should be sent to the trainer, he/she simply drags it into the appropriate folder.

    - Each night a shell script runs through the list of usernames stored in a file, and runs zmtrainsa in spam mode and then ham mode, looking at the associated folder created above.

    This method seems to work quite well, even though it requires a bit of effort up front. Users like it because it's a simple drag-and-drop operation for Outlook or Thunderbird or equivalent.

    One small problem remains ... zmtrainsa no longer causes fetchmail to mark a file as 'read' after it examines it. This behavior seems to have crept into the latest release. Is there a parameter I can stuff in somewhere to cause messages to be marked as read?

    Beyond this small glitch, I continue to be an extremely satisfied user of Zimbra.
    {Lead me not into temptation ... I can find it myself, thank you}

  9. #9
    kilrathi is offline Member
    Join Date
    Sep 2007
    Location
    Arkansas
    Posts
    13
    Rep Power
    7

    Default

    We have just migrated from Scalix to Zimbra and so far I'm pretty happy with the results. One piece of the puzzle that seems to be missing is our previous anti-spam functionality. On our old server I had scripts setup to search specified folders in all my users mailboxes, add the mail to our bayes db, then remove that email message from the folder. I used cron jobs to process the users mailboxes and ran the process after hours. I would like to find a way to get the same results with Zimbra.

    On our older server I had 3 folders each of my users used for spam training. Spam, Not Spam, and Junk E-mail. The Spam folder was similar to the Junk folder on Zimbra. All of the tagged email spam was automatically moved to that folder via client rules. The 'Not Spam' folder was for messages that were tagged as spam that my users wanted to see in their inbox. The third folder 'Junk E-mail' was the folder where all my users put spam email that showed up in their inbox. The last function of the script was the grab the last 100 messages from the inbox of a select list of users and add it to my bayes db as ham. Originally the script did this for all users. I modified it to only pick from a hand full of people whom i knew kept their inbox clear of spam. The biggest thing I haven't been able to find a way to do with Zimbra is make it remove the message from the users folder once it has been processed.

    I accomplished most of this from a perl script the Scalix community posted in their forums. I was able to modify it to fit our needs. I will add my old script below this post for reference, and in hopes that someone with far better perl scripting skills can modify it to fit Zimbra.


    I used two different versions of the script. One I ran daily, the other I ran weekly. The daily one processed the Junk E-mail folder. The weekly one processed the Spam folder.

    This is the Daily Script
    Code:
    #!/usr/bin/perl
    use strict;
    use warnings;
    use Mail::IMAPClient;
    my $host="<edit>";
    my $username="<edit>";
    my $password="<edit>";
    my @real_users=`/opt/scalix/bin/omshowu -m all -i`;     # get all real user names.
    foreach my $punter (@real_users)                        # Loop over them all.
    {
            chomp $punter;                                  # Remove trailing carriage return.
            if ("$punter" ne "postmaster")
            {
            print "$punter\n";                              # Some output. Feel free to remove.
            #$punter =~s/@/\\@/;
            my $user="mboxadmin:$username:$punter";         # Set up superuser login.
            my $imap  = new Mail::IMAPClient( 'Server' => $host , 'User' => $user , 'Password' => $password  ) or next;  # connect to server.
            my @folders=$imap->folders;                     # list folders.
            foreach my $i ( @folders ) { print $i; }
            foreach  my $folder (@folders)                  # Look through each of them.
            {
                    print lc($folder),"\n";
                    if (lc($folder) eq "junk e-mail")                                                                    # "junk email" folder.
                    {
                            print "Found a spam folder: $folder\n";
                           $imap->select($folder) or next;                                                                  # Select the folder.
                            print "Folder $folder selected.\n";
                            my @list=$imap->messages or next;                                                              # List all messages in folder.
                            print scalar(@list)." messages in folder.\n";
                            foreach my $msg (reverse(@list))                                                                # Loop over them all.
                            {
                                    my @email=$imap->fetch($msg,'RFC822');                                                  # Fetch message.
                                    open (SALEARN,"|/usr/bin/spamassassin -d | /usr/bin/sa-learn --spam") or print "$!\n";  # Feed to sa-learn.
                                    print SALEARN "$email[1]";
                                    close SALEARN;
                                    open (REPORT,"|/usr/bin/spamassassin -d | /usr/bin/spamassassin -r") or print "$!\n";   # Report it. (SpamCop and Pyzor).
                                    print REPORT "$email[1]";
                                    close REPORT;
                                    $imap->delete_message($msg) or next;                                                    # Delete it.
                            }
                            $imap->expunge($folder) or next;                                                                #Expunge folder.
                    }
                    elsif(lc($folder) eq "not spam")
                    {
                           $imap->select($folder) or next;                                                                  # Select the folder.
                            print "Folder $folder selected.\n";
                            my @list=$imap->messages or next;                                                              # List all messages in folder.
                            print scalar(@list)." messages in folder.\n";
                            foreach my $msg (reverse(@list))                                                                # Loop over them all.
                            {
                                    my @email=$imap->fetch($msg,'RFC822');                                                  # Fetch message.
                                    open (SALEARN,"|/usr/bin/spamassassin -d | /usr/bin/sa-learn --forget") or print "$!\n";# Sa-learn forget this message if already seen.
                                    print SALEARN "$email[1]";
                                    close SALEARN or print "$!\n";
                                    open (SALEARN,"|/usr/bin/spamassassin -d | /usr/bin/sa-learn --ham") or next;          # Feed to sa-learn as ham.
                                    print SALEARN "$email[1]";
                                    close SALEARN;
                                    $imap->delete_message($msg) or next;
                            }
                            $imap->expunge($folder) or next;
    
                    }
            }
            }
    }

    This is the weekly script
    Code:
    #!/usr/bin/perl
    use strict;
    use warnings;
    use Mail::IMAPClient;
    my $host="<edit>";
    my $username="<edit>";
    my $password="<edit>";
    my @real_users=`/opt/scalix/bin/omshowu -m all -i`;     # get all real user names.
    foreach my $punter (@real_users)                        # Loop over them all.
    {
            chomp $punter;                                  # Remove trailing carriage return.
            if ("$punter" ne "postmaster" )
            {
            print "$punter\n";                              # Some output. Feel free to remove.
            my $user="mboxadmin:$username:$punter";         # Set up superuser login.
            my $imap  = new Mail::IMAPClient( 'Server' => $host , 'User' => $user , 'Password' => $password  ) or next;  # connect to server.
            my @folders=$imap->folders;                     # list folders.
            foreach  my $folder (@folders)                  # Look through each of them.
            {
                    if (lc($folder) eq "inbox")             # "Inbox" is guaranteed to only have ham in it.
                    {
                            if ("$punter" eq "Trusted User 1" ||
                                "$punter" eq "Trusted User 2" ||
                                "$punter" eq "Trusted User 3" ||
                                "$punter" eq "Trusted User 4" ||
                                "$punter" eq "Trusted User 5")
                            {
                                    print "Inbox found.\n";         # Some debug output.
                                    $imap->select($folder) or next; # Select folder.
                                    print "Folder $folder selected.\n";
                                    my @list=$imap->seen or next;   # Get only messages which have been read.
                                                                    # Saves the possibility of reading in false positives.
                                                                    # Also stops us interfering with people's mail.
                                    print scalar(@list)." messages in folder.\n";
                                    my $counter=0;                  # Initialise counter. - we don't want the entire inbox.
                                    foreach my $msg (@list)         # Loop over each message.
                                    {
                                            my @email=$imap->fetch($msg,'RFC822');  # Fetch it.
                                            open (SALEARN,"|/usr/bin/spamassassin -d | /usr/bin/sa-learn --ham") or next;                # Feed it to sa-learn.
                                            print SALEARN "$email[1]\n";
                                            close SALEARN;
                                            $counter +=1;           # Increment counter.
                                            last if ($counter>100); # We only want 100 messages.
                                    }
                            }
                    }
                    elsif (lc($folder) eq "spam suspect")                                               # "Possible Spam" folder.
                    {
                            print "Found a spam folder: $folder\n";
                           $imap->select($folder) or next;                                              # Select the folder.
                            print "Folder $folder selected.\n";
                      # 1 Week
                      #     my $lastweek=time()-604800;                                                 # Get timestamp for this time last week.
                      # 3 Days
                            my $lastweek=time()-259200;
                            my @list = $imap->before($lastweek) or next;                                # List all messages older than that.
                            print scalar(@list)." messages in folder.\n";
                            foreach my $msg (reverse(@list))                                            # Loop over them all.
                            {
                                    my @email=$imap->fetch($msg,'RFC822');                              # Fetch message.
                                    open (SALEARN,"|/usr/bin/spamassassin -d | /usr/bin/sa-learn --spam") or print "$!\n";       # Feed to sa-learn.
                                    print SALEARN "$email[1]";
                                    close SALEARN;
                                    open (REPORT,"|/usr/bin/spamassassin -d | /usr/bin/spamassassin -r") or print "$!\n";        # Report it. (SpamCop and Pyzor).
                                    print REPORT "$email[1]";
                                    close REPORT;
                                    $imap->delete_message($msg) or next;                                # Delete it.
                            }
                            $imap->expunge($folder) or next;                                            #Expunge folder.
                    }
                    elsif (lc($folder) eq "spam")                                                                         # "spam"  folder.
                    {
                            print "Found a spam folder: $folder\n";
                           $imap->select($folder) or next;                                                                  # Select the folder.
                            print "Folder $folder selected.\n";
                            my $lastweek=time()-604800;                                                                     # Get timestamp for this time last week.
                            my @list = $imap->before($lastweek) or next;                                                    # List all messages older than that.
                            print scalar(@list)." messages in folder.\n";
                            foreach my $msg (reverse(@list))                                                                # Loop over them all.
                            {
                                    my $subject=$imap->subject($msg);                                                       # Fetch subject for message.
                                    my @email=$imap->fetch($msg,'RFC822');                                                  # Fetch message.
                                    unless ($subject=~m/\[SPAM\]/)
                                    {
                                            print "Learning message with subject: $subject\n";
                                            open (SALEARN,"|/usr/bin/spamassassin -d | /usr/bin/sa-learn --spam") or print "$!\n";  # Feed to sa-learn.
                                            print SALEARN "$email[1]";
                                            close SALEARN;
                                    }
                                    open (REPORT,"|/usr/bin/spamassassin -d | /usr/bin/spamassassin -r") or print "$!\n";   # Report it. (SpamCop and Pyzor).
                                    print REPORT "$email[1]";
                                    close REPORT;
                                    $imap->delete_message($msg) or next;                                                    # Delete it.
                            }
                            $imap->expunge($folder) or next;                                                                #Expunge folder.
                    }
    
            }
            }
    }
    Again please note that this script is the work of the people in the Scalix forums. I only modified it to fit my needs. Not taking any credit here just posting for reference and hopefully a solution to add to my new Zimbra server.

  10. #10
    charlesr is offline Starter Member
    Join Date
    Oct 2007
    Posts
    2
    Rep Power
    7

    Default

    We current have a system where a user drags their spam messages into 'Junk' and a cron job every hour reads the changed mail folders and updates the spam databases for each user. This means that I user could just drop their mail into any folder other than Trash or Junk and it would be marked as spam, and anything in the Junk folder would be marked as Spam, and everything in the Trash folder would be ignored.

    The problem that I see with the current setup is that I have a lot of users that love clients like thunderbird and outlook, along with a bunch of pine, mutt, and command-line mail users. They don't like using web interfaces and it would be easier if a simple action of moving mail into the Junk folder automatically marks the message as spam, and moving things out of the Junk folder marks them as ham.

    Has any one come up with a solution that would make this work?

Page 1 of 3 123 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Spam Training
    By Krishopper in forum Administrators
    Replies: 4
    Last Post: 07-17-2007, 09:07 PM
  2. Spam training and Junk folder behavior
    By flyerguybham in forum Administrators
    Replies: 9
    Last Post: 02-14-2007, 12:09 PM
  3. Training spam and ham
    By Justin in forum Developers
    Replies: 2
    Last Post: 10-31-2006, 03:39 PM
  4. Zimbra Training!
    By czaveri in forum Migration
    Replies: 5
    Last Post: 09-29-2006, 12:30 AM
  5. Spam training has no cron job
    By richard-hdd in forum Administrators
    Replies: 3
    Last Post: 09-13-2006, 11:50 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •