[SOLVED] Script for migration from Openwebmail to Zimbra 4.5.6

[boris007]

EDIT: My problem is now SOLVED. See the second post for my working script

Hi guys,

First up- Hi, this is my first post here
Second- Sorry this is such a long post- please bear with me!
Third- Any help you can offer will be much appreciated

Scenario:

I'm currently trying to finish a perl script that I'm writing to migrate my users from openwebmail to zimbra.

The script, when working properly, will do the following:

-Read through a /mail folder, which contains a folder for each user (these folders are named in the format first_initial.surname)
-Once in a user folder, decend another level to the user's mail folder
-Once in the user's mail folder, process the mbox files (processing involves confirming that a file IS a mbox file, creating a zimbra folder with the same name as the mbox file, outputting the messages contained in a mbox file to a local location, and importing the messages from the mbox file into the newly created zimbra folder)

In other words, on my zimbra server there is a folder:

/tmp/mail

within which there are all my user's folders from openwebmail:

/tmp/mail/t.brown
/tmp/mail/a.user
...

within each user folder, there is a mail folder:

/tmp/mail/a.user/mail

within each user's mail folder are the mbox files (along with some other junk).

Within the script is basic checking to confirm that a folder is a user folder (looks for a '.' as the second character in the folder name) and checking to confirm that a file is an mbox file (read the first line of the file, and look for the 'F' in 'From', which is always the first line of an mbox file).

And yes, I am very new to perl, and yes I used the mbox migration example as a starting point.

My Problem:

Anyhow, the script in its current form works in every way EXCEPT that for some reason it seems to 'skip' some messages when processing mbox files. For example, I have an mbox file that I know has 10 messages contained in it, and it is processed without error by my script. When I look in zimbra the appropriate folder has been created in the right user account, and emails are there, just not ALL of them. From what I can see, sometimes it reaches a particular email and then proceeds to 'lump' all remaining emails into a single message.

This doesn't happen with all mbox files though- there are some mbox files where ALL of the emails are properly processed and appear in the new zimbra folder. I have noticed that the mbox files that are processed correctly are the ones where all of the emails contained within are of the same MIME type- so my issue my be related to that.

My script

#!/usr/bin/perl

# mbox to Zimbra mailfile conversion
# last updated 24.8.07

# Variable declaration

    use strict;
    use Email::Folder;
    use Mail::Mailer;
    use MIME::Parser;
    use Net::SMTP;
    use Fcntl;
    use Getopt::Std;
    use File::Basename;
    use Email::FolderType qw(folder_type);

    my $email;
    my $user;

    my $next = "";
    my $where = "";
    my $next2 = "";
    my $where2 = "";
    my $dirname;
    my $dirname2;
    my $tempword;
    my $flagset = 0;
    my $message_hold;
    my @messages;
    my $count = 0;
    my $text2;
    my $test_pos2;

    system ("mkdir /tmp/mail_output");

    # Begin reading through the user email subfolders

    my @maildir2 = glob("/tmp/mail/$user/mail/*.*");

    if (-d "/tmp/mail") {
      $where2 = "/tmp/mail/";
    } else {
      $where2 = "/tmp/mail/";
    }

   while (defined($next2 = <$where2/*>)) {

   $user = basename($next2);

   $text2 = $user;
   print "$text2\n";
   $test_pos2 = index $text2,".";
   print "$test_pos2\n";

   if ($test_pos2 == "1") {
     
     $user = basename($next2);
     $email = $user."\@my.domain.goes.here";

    print "Username: $user\n";
    print "Email: $email\n";

    #---------------------------------------------------------------------

    # Process files in current directory, replace spaces with underscores

    opendir(DIR, "/tmp/mail/$user/mail") or die $!;
    my @files = readdir(DIR);
    close(DIR);

    for (@files) {
    next if -d;
    next if /^\./;

    my $new_name = $_; 
    $new_name =~ s/ /_/g;
    rename("/tmp/mail/$user/mail/$_" , "/tmp/mail/$user/mail/$new_name") or die $!;
    } 

    #--------------------------------------------------------------------
   
    # Setup directory to store converted mail, begin loop to read through mail
    # in user's folder

    system ("mkdir /tmp/$user");

    my @maildir = glob("/tmp/mail/$user/mail/*.*");

    if (-d "/tmp/mail/$user/mail/") {
      $where = "/tmp/mail/$user/mail/";
    } else {
      $where = "/tmp/mail/$user/mail/";
    }

    while (defined($next = <$where/*>)) {

     $dirname = basename ($next);

      my $folder = Email::Folder->new($next ||
      die "Usage: $0 mbox dest_address [smtp server]
      Forward all mail found in mail file mbox to address."
    );

   print folder_type "$next";
   print "\n";

     #-------------------------------------------------------------------------------

   #Tests to see if the file is a valid mail file, sets a flag if it is

   my $i=0;
   my $text;
   my $test_pos;
   $flagset = 0;

   print "$dirname\n";

   open (mailfile, "$next")|| die ("Could not open file <br> $!");
   $text = <mailfile>;
   print "Text on first line: $text\n";
   $test_pos = index $text,"F";

   if ($test_pos == "0") {
    @messages=$folder->messages;
    print "message: @messages\n";
    my $total=@messages;
    $message_hold = @messages;
    $flagset = 1;
    close mailfile;
   }

   if ($test_pos != "0"){
    print("Not a mail file\n");
    close mailfile;
   }
    
   #---------------------------------------------------------------------------------

   # If the file has been flagged a mail file, process it
 
  if ($flagset == "1") {
     
    system ("mkdir /tmp/$user/$dirname");
    system ("/opt/zimbra/bin/zmprov -z selectMailbox $user cf /$dirname");
    $count = 0;
  
  foreach (@messages){

        $count++;
        my $parser = new MIME::Parser;

   # Parser options-----------------------------------------------------------------

   #     $parser->output_under("/tmp/mail_output/");
   #     $parser->decode_headers(0);
   #     $parser->ignore_errors(0);
   #     $parser->extract_uuencode(0);
   #     $parser->extract_nested_messages(0);

   #-------------------------------------------------------------------------------
       
        my $entity = $parser->parse_data($_->as_string);
        print "entity: $entity\n";
        my $header = $entity->head;
        print "header: $header\n";
        my $sender = $entity->head->get('From');
        next if $header->get("subject") =~ m/FOLDER INTERNAL/;
        $header->replace('To', $email);
        $header->delete('Received');
        $header->delete('MIME-Version');
        $header->delete('Return-Path');
        $header->delete('User-Agent');
        $header->delete('Message-ID');
        $header->delete('X-Mailer');
        $header->delete('X-Security');
        $header->delete('X-Spam-Checker-Version');
        $entity->head($header);
        $entity->sync_headers;
        my $temp = $entity->as_string();

    open MAILDUMP, ">/tmp/$user/$dirname/mail_$dirname$count.txt";
    print MAILDUMP $temp;
    close MAILDUMP;

    system ("/opt/zimbra/bin/zmmailbox -z -m $user addmessage  /$dirname /tmp/$user/$dirname/mail_$dirname$count.txt");

        print "Done\n\n";

   }
 
 }
 
 print "Done processing emails for user $user\n";

 }

   if ($test_pos2 != "1"){
    print("Not a mail folder\n");
   }
 }
}

Many thanks for any help you can give

[boris007]

Update: I've fixed the problem I was having

Basically, I narrowed it down to the way that Email::Folder parses mbox files. A snippet from the code provided on the mbox migration entry on the Zimbra wiki reads as follows:

    my $folder = Email::Folder->new($mbox ||
        die "Usage: $0 mbox dest_address [smtp server]
        Forward all mail found in mail file mbox to address.
    ");

    my $count=0;
    my @messages=$folder->messages;
    my $total=@messages;
    }

With my original code (which used the provided code from the wiki including the above), I was finding that

my @messages=$folder->messages;

wasn't able to properly distinguish all of the messages in the mbox file. For example, working with an mbox file that I know to have 26 messages in, the value of $total was always 2 (only two emails were being properly processed).

After much mucking around, the eventual solution was to drop using Email::Folder and MIME::Parser altogether, and simply use Mail::MboxParser. I'm not saying that this is the best solution for everyone, but it is working perfectly for me where my previous script didn't, not to mention its a little faster and requires less code

See the following if you are interested:
Mail::MboxParser - read-only access to UNIX-mailboxes - search.cpan.org

Anyhow, for those interested here is my script. Again, I stress that I've never done perl before, so its not going to be perfect!

#!/usr/bin/perl

# mbox to Zimbra mailfile conversion
# last updated 7.9.07
# converts mail files of the mbox format to the Zimbra format

    #-------------------------------------------------------------------------------

    use Email::Folder;
    use Email::FolderType;
    use Email::FolderType::Mbox;
    use Mail::Mailer;
    use Mail::Message;
    use MIME::Parser;
    use Net::SMTP;
    use Fcntl;
    use Getopt::Std;
    use File::Basename;
    use Mail::Box::Mbox;
    use Mail::MboxParser;

    my $email;
    my $user;
    my $temp;
    my $next = "";
    my $where = "";
    my $next2 = "";
    my $where2 = "";
    my $dirname;
    my $dirname2;
    my $tempword;
    my $flagset = 0;
    my $count = 0;
    my $text2;
    my $test_pos2;
    my $subject;
    my @messages2;

    my $parseropts = {
        enable_cache    => 1,
        enable_grep     => 1,
        cache_file_name => 'mail/cache-file',
    };

    #-------------------------------------------------------------------------------

    @months = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);
    @weekDays = qw(Sun Mon Tue Wed Thu Fri Sat Sun);
    ($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek, $dayOfYear, $daylightSavings) = localtime();
    $year = 1900 + $yearOffset;
    $theTime = "$hour:$minute:$second, $weekDays[$dayOfWeek] $months[$month] $dayOfMonth, $year";

    system ("mkdir /tmp/mail_output");
    system ("mkdir /tmp/attachments");

    print "Mbox to Zimbra message migration script. Started at $theTime\n";
    print "-------------------------------------------------------------------------------\n"; 

    # Begin reading through the user email subfolders

    if (-d "/tmp/mail") {
      $where2 = "/tmp/mail/";
    } else {
      $where2 = "/tmp/mail/";
    }

   while (defined($next2 = <$where2/*>)) {

   $user = basename($next2);

   $text2 = $user;
   $test_pos2 = index $text2,".";

   if ($test_pos2 == "1") {
     
     $user = basename($next2);
     $email = $user."\<my domain goes here- for cosmetic purposes>";

    print "\nProcessing messages for user: $email\n";

    #-------------------------------------------------------------------------------

    # Process files in current directory, replace spaces with underscores for processing

    opendir(DIR, "/tmp/mail/$user/mail") or die $!;
    my @files = readdir(DIR);
    close(DIR);

    for (@files) {
    next if -d;
    next if /^\./;

    my $new_name = $_; 
    $new_name =~ s/ /_/g;
    rename("/tmp/mail/$user/mail/$_" , "/tmp/mail/$user/mail/$new_name") or die $!;
    } 

    #-------------------------------------------------------------------------------
   
    # Setup directory to store converted mail, begin loop to read through mail
    # in user's folder

    system ("mkdir /tmp/$user");

    if (-d "/tmp/mail/$user/mail/") {
      $where = "/tmp/mail/$user/mail/";
    } else {
      $where = "/tmp/mail/$user/mail/";
    }

    while (defined($next = <$where/*>)) {

    $dirname = basename ($next);
  
        my $folder2 = Mail::MboxParser->new($next, 
                                    decode     => 'ALL',
                                    parseropts => $parseropts);

   #-------------------------------------------------------------------------------

   # Tests to see if the file is a valid mail file, sets a flag if it is

   my $i=0;
   my $text;
   my $test_pos;
   $flagset = 0;

   print "Processing mail file: $dirname\n";

   open (mailfile, "$next")|| die ("Could not open file <br> $!");
   $text = <mailfile>;
   $test_pos = index $text,"F";

    if ($test_pos == "0") {
     $flagset = 1;
     close mailfile;
   }

    if ($test_pos != "0"){
     print("Not a mail file, skipping...\n");
     close mailfile;
   }
    
   #---------------------------------------------------------------------------------

   # If the file has been flagged a mail file, process it
 
   if ($flagset == "1") {
     
    system ("mkdir /tmp/$user/$dirname");
    system ("/opt/zimbra/bin/zmprov -z selectMailbox $user cf /$dirname");
    $count = 0;

  # For every message in the mail file, print the subject, output a text file and add to Zimbra
 
    for my $msg ($folder2->get_messages)
    {
    $count ++;
    $subject = $msg->header->{subject};
    print "Processing message: $subject\n";
    $msg->store_all_attachments(path => '/tmp/attachments');
    $temp = $msg;
    
    open MAILDUMP, ">/tmp/$user/$dirname/mail_$dirname$count.txt";
    print MAILDUMP $temp;
    print MAILDUMP "****END OF MESSAGE****";
    close MAILDUMP;

    system ("/opt/zimbra/bin/zmmailbox -z -m $user addmessage  /$dirname /tmp/$user/$dirname/mail_$dirname$count.txt");

    print "----Done processing message, stored in mail_$dirname$count.txt----\n\n";

    }   

   }
 
 }
 
 print "----Done processing messages for user $email----\n\n";
 print "-------------------------------------------------------------------------------\n";


 # If a folder is not a mail folder, skip it

 }

   if ($test_pos2 != "1"){
    print("$text2 is not a mail folder, skipping...\n");
   }

}

    ($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek, $dayOfYear, $daylightSavings) = localtime();
    $year = 1900 + $yearOffset;
    $theTime = "$hour:$minute:$second, $weekDays[$dayOfWeek] $months[$month] $dayOfMonth, $year";

     print "----Done processing all messages, finished at $theTime----\n";


# EOF

Hope this is useful to someone.

[boris007]

Hi all,

Just a bump to let you all know that the problem is resolved and my script now works. I tried to post my now-working script as a new post in this thread but it wouldn't work for some reason, so I edited my second post.

I'm not sure if the problem I was having with the email processing is somehow unique to my situation, maybe if others are able to test both my script and the mbox user migration script from the Zimbra wiki and compare results we'll have a better idea.

Cheers

[boris007]

Mods, I've noticed that User Migration - ZimbraWiki still contains the original mbox processing script. If thought appropriate, I'm happy to add my script to the same entry (especially since this has dropped off the first page).

EDIT: Tested and working with OSS 4.5.7