Error importing via zmmailbox

[The Fold]

So not having had much luck with the perl or python scripts on the wiki I found a script here that converted my MBOX files to maildir, I have then been able to upload to different people's mailboxes with the command:

zmmailbox -z -m <user>
createFolder /Imported
addMessage Imported /tmp/oldmail/.maildir/cur

However, when using this method I found that I received a message saying

ERROR: service.FAILURE (system failure: Failure Indexing: com.zimbra.cs.mailbox.Appointment@1e29df9)

on one occasion. This was about 700 messages in out of 1655 messages. As it imports them in a seemingly random order I have no idea which message this would have been as they are all labelled in numerical order and the import looks like:

...
694 (/tmp/migration/test2/.old_MBX/cur/1208526616.000833.mbox:2,)
695 (/tmp/migration/test2/.old_MBX/cur/1208526616.000143.mbox:2,)
696 (/tmp/migration/test2/.old_MBX/cur/1208526616.000879.mbox:2,)
697 (/tmp/migration/test2/.old_MBX/cur/1208526616.000783.mbox:2,)
698 (/tmp/migration/test2/.old_MBX/cur/1208526616.000600.mbox:2,)
699 (/tmp/migration/test2/.old_MBX/cur/1208526616.000790.mbox:2,)
700 (/tmp/migration/test2/.old_MBX/cur/1208526616.000748.mbox:2,)
701 (/tmp/migration/test2/.old_MBX/cur/1208526616.000882.mbox:2,)
702 (/tmp/migration/test2/.old_MBX/cur/1208526616.000551.mbox:2,)
703 (/tmp/migration/test2/.old_MBX/cur/1208526616.000660.mbox:2,)
704 (/tmp/migration/test2/.old_MBX/cur/1208526616.001155.mbox:2,)
705 (/tmp/migration/test2/.old_MBX/cur/1208526616.000559.mbox:2,)
706 (/tmp/migration/test2/.old_MBX/cur/1208526616.000348.mbox:2,)
707 (/tmp/migration/test2/.old_MBX/cur/1208526616.000347.mbox:2,)
708 (/tmp/migration/test2/.old_MBX/cur/1208526616.000850.mbox:2,)
709 (/tmp/migration/test2/.old_MBX/cur/1208526616.000925.mbox:2,)
710 (/tmp/migration/test2/.old_MBX/cur/1208526616.000028.mbox:2,)
711 (/tmp/migration/test2/.old_MBX/cur/1208526616.000663.mbox:2,)
712 (/tmp/migration/test2/.old_MBX/cur/1208526616.000060.mbox:2,)
713 (/tmp/migration/test2/.old_MBX/cur/1208526616.000159.mbox:2,)
...

I thought it might be date order but those last 2 messages are both in August but one is 2006, the other is 2007.

Now I can get the messages in to the persons Inbox via the aforementioned Python script but obviously it makes it easier to automate if I can do it all in one run this way.

[webman]

I'm not sure why that script isn't working, but like you, I also had no luck with the MBOX scripts on the wiki.

However, I found one on the forum here, and made a few modifications which allowed me to run it on an individual mailbox - this also had the advantage I could do several at a time using virtual terminals/screen sessions to the linux box. I have migrated over 100 accounts using this script and only had a handful of missing messages.

MBOX to Zimbra.

Directory structure is accepted to be like this:

/tmp/migrate/craig.rodway/mail/Drafts
/tmp/migrate/craig.rodway/mail/Sent
/tmp/migrate/craig.rodway/mail/Trash
/tmp/migrate/craig.rodway/mail/Inbox

Usage:

mailimport.pl username

As that script replaces spaces with underscores in mail directory names I created another script to run after import to change them back to spaces.

Zimbra mailbox - underscores to spaces

Usage:

fixfolders.pl username

Hope this helps.

[The Fold]

Thanks for that script, I'm importing mail with zero fuss as we speak!

The only things I would like to mention is that your script has your e-mail domain listed in it, and for some reason having my mail in /tmp/migrate/user/mail didn't seem to work, it was fine though if I just moved the files to /tmp/mail/user/mail, so maybe a section to add in a parameter on the command line for it?

Also the speed seems slow but I guess if it re-opens zmmailbox for each message then that would explain it.

Other than that it works great! Shame this isn't on the wiki.

[phoenix]

Originally Posted by The Fold

Other than that it works great! Shame this isn't on the wiki.

The wiki is a Community resource, feel free to add the script to the wiki if you want.

[webman]

Hi The Fold

No problem at all.. as I said, it's a script I found on the forum which I had to modify a bit to get to work with the way I was migrating; but I'm pleased it's been of some help to you

Yes unfortunately it is fairly slow due to it's nature - it took me about a week (working about 12 hour days) to migrate 130 accounts of varying sizes - a lot were around the 500MB mark. It puzzled me how the PST Import Wizard could process dozens of messages per second where this script could average 1 per second but I guess it's just the code that zmmailbox uses

If nobody beats me to it I'll do some tidying up and then add it to the wiki.

[The Fold]

This is just a thought, but would it not be possible to modify the script so that it converts all the e-mail first for a folder then performs a zmmailbox addMessage against that whole folder?

I'm no perl expert by any means but if the line...

system ("/opt/zimbra/bin/zmmailbox -z -m $user addmessage  /$dirname /tmp/$user/$dirname/mail_$dirname$count.txt");

were moved just outside of the $msg loop and adjusted to read

system ("/opt/zimbra/bin/zmmailbox -z -m $user addmessage  /$dirname /tmp/$user/$dirname/;

would that not mean it would upload the whole folder?

[webman]

I'm not sure if that will work but it's worth a try. I would recommend creating a test account first, then run the command manually as the Zimbra user against the test account with one of the directories as input.

The help suggests it is possible, however:

addMessage(am)               [opts] {dest-folder-path} {filename-or-dir} [{filename-or-dir} ...

[The Fold]

It is possible, and altering the script so like so

                            print "----Done processing message, stored in mail_$dirname$count.txt----\n\n";
                        }       # end if subject not internal data

                }       # end of for $msg

        system ("/opt/zimbra/bin/zmmailbox -z -m $user addmessage  /$dirname /tmp/$user/$dirname/");

        }       # end of flagset=1

does work, I tested it with a couple of mailboxes without issue. However, if you were to encounter an error like I listed in my first post it stops importing for that folder altogether and moves on to the next folder.

I've re-adjusted my script back so that it's uploading on a per-message basis and logging the output so I can see which message is causing the error.

[webman]

Cool, thanks - that's good to know

Another thing - sometimes you may find it will appear to freeze on a message. This isn't anything to worry about - it seems that the Mailbox Purge background process begins to run on the account once you start importing messages, but the script will soon carry on.

[The Fold]

I've now found the message that was causing the error, it was a message that was forwarded, contained some appointment details and also an attachment. I don't know if this combination of different types is why it won't import?