Results 1 to 4 of 4

Thread: [SOLVED] Script for migration from Openwebmail to Zimbra 4.5.6

  1. #1
    boris007 is offline Junior Member
    Join Date
    Aug 2007
    Posts
    6
    Rep Power
    8

    Default [SOLVED] Script for migration from Openwebmail 2.5.2 (mbox) to Zimbra 4.5.6

    EDIT: My problem is now SOLVED. See the second post for my working script

    Hi guys,

    First up- Hi, this is my first post here
    Second- Sorry this is such a long post- please bear with me!
    Third- Any help you can offer will be much appreciated

    Scenario:

    I'm currently trying to finish a perl script that I'm writing to migrate my users from openwebmail to zimbra.

    The script, when working properly, will do the following:

    -Read through a /mail folder, which contains a folder for each user (these folders are named in the format first_initial.surname)
    -Once in a user folder, decend another level to the user's mail folder
    -Once in the user's mail folder, process the mbox files (processing involves confirming that a file IS a mbox file, creating a zimbra folder with the same name as the mbox file, outputting the messages contained in a mbox file to a local location, and importing the messages from the mbox file into the newly created zimbra folder)

    In other words, on my zimbra server there is a folder:

    /tmp/mail

    within which there are all my user's folders from openwebmail:

    /tmp/mail/t.brown
    /tmp/mail/a.user
    ...

    within each user folder, there is a mail folder:

    /tmp/mail/a.user/mail

    within each user's mail folder are the mbox files (along with some other junk).

    Within the script is basic checking to confirm that a folder is a user folder (looks for a '.' as the second character in the folder name) and checking to confirm that a file is an mbox file (read the first line of the file, and look for the 'F' in 'From', which is always the first line of an mbox file).

    And yes, I am very new to perl, and yes I used the mbox migration example as a starting point.

    My Problem:

    Anyhow, the script in its current form works in every way EXCEPT that for some reason it seems to 'skip' some messages when processing mbox files. For example, I have an mbox file that I know has 10 messages contained in it, and it is processed without error by my script. When I look in zimbra the appropriate folder has been created in the right user account, and emails are there, just not ALL of them. From what I can see, sometimes it reaches a particular email and then proceeds to 'lump' all remaining emails into a single message.

    This doesn't happen with all mbox files though- there are some mbox files where ALL of the emails are properly processed and appear in the new zimbra folder. I have noticed that the mbox files that are processed correctly are the ones where all of the emails contained within are of the same MIME type- so my issue my be related to that.

    My script

    Code:
    #!/usr/bin/perl
    
    # mbox to Zimbra mailfile conversion
    # last updated 24.8.07
    
    # Variable declaration
    
        use strict;
        use Email::Folder;
        use Mail::Mailer;
        use MIME::Parser;
        use Net::SMTP;
        use Fcntl;
        use Getopt::Std;
        use File::Basename;
        use Email::FolderType qw(folder_type);
    
        my $email;
        my $user;
    
        my $next = "";
        my $where = "";
        my $next2 = "";
        my $where2 = "";
        my $dirname;
        my $dirname2;
        my $tempword;
        my $flagset = 0;
        my $message_hold;
        my @messages;
        my $count = 0;
        my $text2;
        my $test_pos2;
    
        system ("mkdir /tmp/mail_output");
    
        # Begin reading through the user email subfolders
    
        my @maildir2 = glob("/tmp/mail/$user/mail/*.*");
    
        if (-d "/tmp/mail") {
          $where2 = "/tmp/mail/";
        } else {
          $where2 = "/tmp/mail/";
        }
    
       while (defined($next2 = <$where2/*>)) {
    
       $user = basename($next2);
    
       $text2 = $user;
       print "$text2\n";
       $test_pos2 = index $text2,".";
       print "$test_pos2\n";
    
       if ($test_pos2 == "1") {
         
         $user = basename($next2);
         $email = $user."\@my.domain.goes.here";
    
        print "Username: $user\n";
        print "Email: $email\n";
    
        #---------------------------------------------------------------------
    
        # Process files in current directory, replace spaces with underscores
    
        opendir(DIR, "/tmp/mail/$user/mail") or die $!;
        my @files = readdir(DIR);
        close(DIR);
    
        for (@files) {
        next if -d;
        next if /^\./;
    
        my $new_name = $_; 
        $new_name =~ s/ /_/g;
        rename("/tmp/mail/$user/mail/$_" , "/tmp/mail/$user/mail/$new_name") or die $!;
        } 
    
        #--------------------------------------------------------------------
       
        # Setup directory to store converted mail, begin loop to read through mail
        # in user's folder
    
        system ("mkdir /tmp/$user");
    
        my @maildir = glob("/tmp/mail/$user/mail/*.*");
    
        if (-d "/tmp/mail/$user/mail/") {
          $where = "/tmp/mail/$user/mail/";
        } else {
          $where = "/tmp/mail/$user/mail/";
        }
    
        while (defined($next = <$where/*>)) {
    
         $dirname = basename ($next);
    
          my $folder = Email::Folder->new($next ||
          die "Usage: $0 mbox dest_address [smtp server]
          Forward all mail found in mail file mbox to address."
        );
    
       print folder_type "$next";
       print "\n";
    
         #-------------------------------------------------------------------------------
    
       #Tests to see if the file is a valid mail file, sets a flag if it is
    
       my $i=0;
       my $text;
       my $test_pos;
       $flagset = 0;
    
       print "$dirname\n";
    
       open (mailfile, "$next")|| die ("Could not open file <br> $!");
       $text = <mailfile>;
       print "Text on first line: $text\n";
       $test_pos = index $text,"F";
    
       if ($test_pos == "0") {
        @messages=$folder->messages;
        print "message: @messages\n";
        my $total=@messages;
        $message_hold = @messages;
        $flagset = 1;
        close mailfile;
       }
    
       if ($test_pos != "0"){
        print("Not a mail file\n");
        close mailfile;
       }
        
       #---------------------------------------------------------------------------------
    
       # If the file has been flagged a mail file, process it
     
      if ($flagset == "1") {
         
        system ("mkdir /tmp/$user/$dirname");
        system ("/opt/zimbra/bin/zmprov -z selectMailbox $user cf /$dirname");
        $count = 0;
      
      foreach (@messages){
    
            $count++;
            my $parser = new MIME::Parser;
    
       # Parser options-----------------------------------------------------------------
    
       #     $parser->output_under("/tmp/mail_output/");
       #     $parser->decode_headers(0);
       #     $parser->ignore_errors(0);
       #     $parser->extract_uuencode(0);
       #     $parser->extract_nested_messages(0);
    
       #-------------------------------------------------------------------------------
           
            my $entity = $parser->parse_data($_->as_string);
            print "entity: $entity\n";
            my $header = $entity->head;
            print "header: $header\n";
            my $sender = $entity->head->get('From');
            next if $header->get("subject") =~ m/FOLDER INTERNAL/;
            $header->replace('To', $email);
            $header->delete('Received');
            $header->delete('MIME-Version');
            $header->delete('Return-Path');
            $header->delete('User-Agent');
            $header->delete('Message-ID');
            $header->delete('X-Mailer');
            $header->delete('X-Security');
            $header->delete('X-Spam-Checker-Version');
            $entity->head($header);
            $entity->sync_headers;
            my $temp = $entity->as_string();
    
        open MAILDUMP, ">/tmp/$user/$dirname/mail_$dirname$count.txt";
        print MAILDUMP $temp;
        close MAILDUMP;
    
        system ("/opt/zimbra/bin/zmmailbox -z -m $user addmessage  /$dirname /tmp/$user/$dirname/mail_$dirname$count.txt");
    
            print "Done\n\n";
    
       }
     
     }
     
     print "Done processing emails for user $user\n";
    
     }
    
       if ($test_pos2 != "1"){
        print("Not a mail folder\n");
       }
     }
    }
    Many thanks for any help you can give
    Last edited by boris007; 10-04-2007 at 09:57 PM.

  2. #2
    boris007 is offline Junior Member
    Join Date
    Aug 2007
    Posts
    6
    Rep Power
    8

    Default

    Update: I've fixed the problem I was having

    Basically, I narrowed it down to the way that Email::Folder parses mbox files. A snippet from the code provided on the mbox migration entry on the Zimbra wiki reads as follows:

    Code:
        my $folder = Email::Folder->new($mbox ||
            die "Usage: $0 mbox dest_address [smtp server]
            Forward all mail found in mail file mbox to address.
        ");
    
        my $count=0;
        my @messages=$folder->messages;
        my $total=@messages;
        }
    With my original code (which used the provided code from the wiki including the above), I was finding that
    Code:
    my @messages=$folder->messages;
    wasn't able to properly distinguish all of the messages in the mbox file. For example, working with an mbox file that I know to have 26 messages in, the value of $total was always 2 (only two emails were being properly processed).

    After much mucking around, the eventual solution was to drop using Email::Folder and MIME::Parser altogether, and simply use Mail::MboxParser. I'm not saying that this is the best solution for everyone, but it is working perfectly for me where my previous script didn't, not to mention its a little faster and requires less code

    See the following if you are interested:
    Mail::MboxParser - read-only access to UNIX-mailboxes - search.cpan.org

    Anyhow, for those interested here is my script. Again, I stress that I've never done perl before, so its not going to be perfect!

    Code:
    #!/usr/bin/perl
    
    # mbox to Zimbra mailfile conversion
    # last updated 7.9.07
    # converts mail files of the mbox format to the Zimbra format
    
        #-------------------------------------------------------------------------------
    
        use Email::Folder;
        use Email::FolderType;
        use Email::FolderType::Mbox;
        use Mail::Mailer;
        use Mail::Message;
        use MIME::Parser;
        use Net::SMTP;
        use Fcntl;
        use Getopt::Std;
        use File::Basename;
        use Mail::Box::Mbox;
        use Mail::MboxParser;
    
        my $email;
        my $user;
        my $temp;
        my $next = "";
        my $where = "";
        my $next2 = "";
        my $where2 = "";
        my $dirname;
        my $dirname2;
        my $tempword;
        my $flagset = 0;
        my $count = 0;
        my $text2;
        my $test_pos2;
        my $subject;
        my @messages2;
    
        my $parseropts = {
            enable_cache    => 1,
            enable_grep     => 1,
            cache_file_name => 'mail/cache-file',
        };
    
        #-------------------------------------------------------------------------------
    
        @months = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);
        @weekDays = qw(Sun Mon Tue Wed Thu Fri Sat Sun);
        ($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek, $dayOfYear, $daylightSavings) = localtime();
        $year = 1900 + $yearOffset;
        $theTime = "$hour:$minute:$second, $weekDays[$dayOfWeek] $months[$month] $dayOfMonth, $year";
    
        system ("mkdir /tmp/mail_output");
        system ("mkdir /tmp/attachments");
    
        print "Mbox to Zimbra message migration script. Started at $theTime\n";
        print "-------------------------------------------------------------------------------\n"; 
    
        # Begin reading through the user email subfolders
    
        if (-d "/tmp/mail") {
          $where2 = "/tmp/mail/";
        } else {
          $where2 = "/tmp/mail/";
        }
    
       while (defined($next2 = <$where2/*>)) {
    
       $user = basename($next2);
    
       $text2 = $user;
       $test_pos2 = index $text2,".";
    
       if ($test_pos2 == "1") {
         
         $user = basename($next2);
         $email = $user."\<my domain goes here- for cosmetic purposes>";
    
        print "\nProcessing messages for user: $email\n";
    
        #-------------------------------------------------------------------------------
    
        # Process files in current directory, replace spaces with underscores for processing
    
        opendir(DIR, "/tmp/mail/$user/mail") or die $!;
        my @files = readdir(DIR);
        close(DIR);
    
        for (@files) {
        next if -d;
        next if /^\./;
    
        my $new_name = $_; 
        $new_name =~ s/ /_/g;
        rename("/tmp/mail/$user/mail/$_" , "/tmp/mail/$user/mail/$new_name") or die $!;
        } 
    
        #-------------------------------------------------------------------------------
       
        # Setup directory to store converted mail, begin loop to read through mail
        # in user's folder
    
        system ("mkdir /tmp/$user");
    
        if (-d "/tmp/mail/$user/mail/") {
          $where = "/tmp/mail/$user/mail/";
        } else {
          $where = "/tmp/mail/$user/mail/";
        }
    
        while (defined($next = <$where/*>)) {
    
        $dirname = basename ($next);
      
            my $folder2 = Mail::MboxParser->new($next, 
                                        decode     => 'ALL',
                                        parseropts => $parseropts);
    
       #-------------------------------------------------------------------------------
    
       # Tests to see if the file is a valid mail file, sets a flag if it is
    
       my $i=0;
       my $text;
       my $test_pos;
       $flagset = 0;
    
       print "Processing mail file: $dirname\n";
    
       open (mailfile, "$next")|| die ("Could not open file <br> $!");
       $text = <mailfile>;
       $test_pos = index $text,"F";
    
        if ($test_pos == "0") {
         $flagset = 1;
         close mailfile;
       }
    
        if ($test_pos != "0"){
         print("Not a mail file, skipping...\n");
         close mailfile;
       }
        
       #---------------------------------------------------------------------------------
    
       # If the file has been flagged a mail file, process it
     
       if ($flagset == "1") {
         
        system ("mkdir /tmp/$user/$dirname");
        system ("/opt/zimbra/bin/zmprov -z selectMailbox $user cf /$dirname");
        $count = 0;
    
      # For every message in the mail file, print the subject, output a text file and add to Zimbra
     
        for my $msg ($folder2->get_messages)
        {
        $count ++;
        $subject = $msg->header->{subject};
        print "Processing message: $subject\n";
        $msg->store_all_attachments(path => '/tmp/attachments');
        $temp = $msg;
        
        open MAILDUMP, ">/tmp/$user/$dirname/mail_$dirname$count.txt";
        print MAILDUMP $temp;
        print MAILDUMP "****END OF MESSAGE****";
        close MAILDUMP;
    
        system ("/opt/zimbra/bin/zmmailbox -z -m $user addmessage  /$dirname /tmp/$user/$dirname/mail_$dirname$count.txt");
    
        print "----Done processing message, stored in mail_$dirname$count.txt----\n\n";
    
        }   
    
       }
     
     }
     
     print "----Done processing messages for user $email----\n\n";
     print "-------------------------------------------------------------------------------\n";
    
    
     # If a folder is not a mail folder, skip it
    
     }
    
       if ($test_pos2 != "1"){
        print("$text2 is not a mail folder, skipping...\n");
       }
    
    }
    
        ($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek, $dayOfYear, $daylightSavings) = localtime();
        $year = 1900 + $yearOffset;
        $theTime = "$hour:$minute:$second, $weekDays[$dayOfWeek] $months[$month] $dayOfMonth, $year";
    
         print "----Done processing all messages, finished at $theTime----\n";
    
    
    # EOF
    Hope this is useful to someone.
    Last edited by boris007; 09-06-2007 at 11:15 PM. Reason: Update to issue

  3. #3
    boris007 is offline Junior Member
    Join Date
    Aug 2007
    Posts
    6
    Rep Power
    8

    Default

    Hi all,

    Just a bump to let you all know that the problem is resolved and my script now works. I tried to post my now-working script as a new post in this thread but it wouldn't work for some reason, so I edited my second post.

    I'm not sure if the problem I was having with the email processing is somehow unique to my situation, maybe if others are able to test both my script and the mbox user migration script from the Zimbra wiki and compare results we'll have a better idea.

    Cheers

  4. #4
    boris007 is offline Junior Member
    Join Date
    Aug 2007
    Posts
    6
    Rep Power
    8

    Default

    Mods, I've noticed that User Migration - ZimbraWiki still contains the original mbox processing script. If thought appropriate, I'm happy to add my script to the same entry (especially since this has dropped off the first page).

    EDIT: Tested and working with OSS 4.5.7
    Last edited by boris007; 10-01-2007 at 11:28 PM.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 12
    Last Post: 02-24-2008, 12:16 AM
  2. dspam logrotate errors
    By michaeln in forum Users
    Replies: 7
    Last Post: 02-19-2007, 12:45 PM
  3. Post instsallation problems
    By Assaf in forum Installation
    Replies: 14
    Last Post: 01-29-2007, 11:38 AM
  4. Zimbra server crashed
    By goetzi in forum Administrators
    Replies: 6
    Last Post: 03-25-2006, 01:00 PM
  5. FC3 Install and no zimbra ?
    By aws in forum Installation
    Replies: 10
    Last Post: 10-09-2005, 04:19 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •