[SOLVED] zmlmptinject challenge

[zwvpadmin]

I've got quite a complicated migration on my hands and need some insight.

Running Zimbra 5.0.2 open source on an ubuntu 6.06 LTS server distro (GNOME+minimum deps)

Everything seems to be working well. The previous server was a temporary solution and it crashed and burned. It was running FC8 sendmail + dovecot imap with the mbox format. It's predecessor was running a similar setup on RHEL 3.

When the RHEL box went down we hard copied all the files to the newer temp box and set it all using the old dirs to populate the new server. It mostly worked, but some of the users imap folders never got properly put in the new mbox format.

What this means is some users have a standard mbox config as /home/user/mail/ and some have an empty mbox and their old imap folders are sitting in /home/user/ instead of in the /home/user/mail/ where they belong.

imapsync is not an option as the temp server is now as toast as its predecessor. But I did manage to rescue the files again and have them on the new zimbra server sitting in /oldhome.

Is there a way to use zmlmtpinject to restore all the original messages from both sets of folders so that they reappear in the new zimbra inbox with the original to/from headers?

The users have stated they don't care about preserving the folders themselves and are happy to recreate the imap folder structures they want, but need to be able to read the messages in their original state in order to organize them properly.

I would like to write a script that uses zmlmptinject to take whatever mbox folder i can throw at it, parse the messages and just reprocess them through the queue to the new zimbra inbox, is this possible??

(keep in mind the mbox dirs are just files of cat'd emails)

Dying to figure this out (got 200 email users looking for old emails asap)

[mmorse]

Welcome to the forums,

Here's a sendmail method & a few scripts: Migrating_from_MBOX_files - User Migration - Zimbra :: Wiki

"The following perl script will forward from an MBOX file to the designated address, while preserving attachments"

-------------------------------
If you can convert them to single files using a parser to unconcatinate into a maildir format somehow:
Because as far as I know zmlmtpinject uses single files, not one big mbox format...the below also ignores folder structure, tags, read status...

So the /opt/zimbra/store has a subfolder called incoming that you can ignore & at least one numbered folder starting with 0.
Look at the subfolders of these numbered folders, which are also numbered. In other words, store/0/1, store/0/2, etc, each belongs to an individual.
(send someone a mail and see where it ends up)

In short each of these sub-subfolders contains all the mail for a single user. These leaf directories will be named something like store/0/1/msg/0
You can add the messages in the leaf folders in that directory to the appropriate account on your new zimbra server as you wish or choose to leave the whole thing at /recovery location for a while.
(files owned by zimbra:zimbra)

zmmailbox addMessage :
su - zimbra
attach to the new server via
zmmailbox -zadmin
or
zmmailbox adminAuthenticate -u https://server.domain.com:7071 admin@domain.com password
mbox> selectMailbox user@domain.com
mbox> createfolder /Recovery
mbox> zmmailbox addMessage /Recovery store/0/1/msg/1 (or whatever the path to the leaf folder is) -- all the mail in the subfolder will be added. Repeat for the remainder of the leaf folders in that mailbox's folder.
(Think it's -d if your having date trouble with the dates and the header.)
You can go zmmailbox help commands for more info.
Another method to fix dates is to rewrite X-Zimbra-Received header see this post: http://www.zimbra.com/forums/adminis....html#post4017

Repeat for next user etc etc

zmltmpinject:
zmlmtpinject -r user@domain.com -s sender@domain.com /location/#/#/msg/123-4.msg

Injecting a whole folder of files would be:
zmlmtpinject -r user@example.com -s sender@domain.com -d /location/#/#/

To get the dates correct you need to rewrite X-Zimbra-Received header see: http://www.zimbra.com/forums/adminis....html#post4017

Rinse and repeat

Or
REST/CURL:
You can POST an RFC822 formatted message to your inbox REST url to append messages to the inbox folder. Using the popular curl program, this would look like the following:

curl -u usernameassword --data-binary @/tmp/rfc822.txt https://server/service/home/username/inbox

Thick-Client:
Last but not least (if you manage to break them into single files) you can also use thick clients that can take .msg (if you want to drag /drop them all...)
Import EML and MSG email message files into Outlook

Good luck!

[zwvpadmin]

Wow that was a lot of great information. I will be trying a few of those methods however I have just a couple clarification questions.

First, if I use the add message method, I assume the original mbox folder files need to be uncat'd first, than processed one at a time. Is this correct?

Second, if I just use the lmtpinject method, when using the -r / -s switches, will the "sender" replace the original sender of the message when it appears in the inbox? If so this method would not work as I need to preserve the original senders information. (Please clarify?)

trying to narrow down my options to choose what will give me the most desirable results.

Thanks again for such a hasty response!

[mmorse]

Yes, anything beneath the line in my first post assumes you've gotten them into separate files. (uncat'd)

zmlmtpinject [options] [files]
[options]:
-a,--address lmtp server (default localhost)
-d,--directory message file directory
-D,--domain default per-connection recipient domain (default example.zimbra.com)
-N,--every report progress after every N messages (default 100)
-p,--port lmtp server port (default 7025)
-q,--quiet don't print per-message status
-r,--recipient envelope recipients (rcpt to). This option accepts multiple arguments, so it can't be last if a list of input files is used.
-S,--stopAfter stop after sending this many messages after warm-up
-s,--sender envelope sender (mail from)
-t,--threads number of worker threads (default 1)
-T,--trace trace server/client traffic
-u,--username username prefix (default "user")
-w,--warmUpThreshold warm-up server with first N messages, then start measuring (default no warm-up)
-z,--repeat repeatedly inject these messages NUM times
(won't actually make additional physical copies unless you turn off the zimbraMessageIdDedupeCacheSize)
[files]:
Direct link to file in rfc822 format, or specify a directory.

[zwvpadmin]

The issue has been resolved, however not by any of the above methods as they all seemed to A) not work B) not work as I needed them to or C) not work as intended.

However I did discover one way to get exactly what I needed through trial and error that I will share in case anyone runs into this again. It's by far the least efficient method there can possibly be. However it is "easy" and achieved exactly what I needed with out compromise, just several steps and very time consuming for each mbox.

A friend clued me into the fact that Thunderbird mail client is capable of multiple imap sessions as well as drag and drop copy via Imap protocols.

So the method is:

-First using thunderbird client on the zimbra server itself, create a mail account for the "end recipient account" and subscribe to appropriate folders.
-Then make sure IMAP is up and running on the old server.
*specific to my situation copying all of the old mbox dirs from /home/user/ into /home/user/mail respectively + "chown user:user *" (since I had to do so as root) allowed thunderbirds forgiving nature to allow those older formatted folders to be subscribed to properly.
-Next Create a second mail account in thunderbird (on the zimbra server also) connecting to the old imap server for the same user and again subscribe to all available folders.

At this point each folder or contents of a given folder are able to be GUI Drag and Dropped from one account to the other, and thunderbird imap protocols facilitate the transfer.

The Pro's:

It's simple, using two different types of authentication does not interfere, each message is able to be placed in any location in the new account as desired, original folders and contents are preserved, and original subject headers as well as original senders are also preserved, no parsing/chunking/scripting needed.

The Con's:

Takes FOREVER per user - therefore impractical for large migrations, alters the original delivery date of each message to time of migration, uses up lots of CPU/Bandwidth on the mail server itself (not recommended during peak usage times).

Otherwise its working great, just taking a LLOOOOOOONNNNNGGGGGG time; if I had more than 200 users this would be absolute torture, but considering the client is going from shaky at best usage for the last week, then absolutely nothing less than 48 hours ago to several hours later back at 100% this was acceptable.

If only Thunderbird could accept script/batch files, then I would just write one to automate the process - who knows maybe Mozilla will eventually work that in as an extension.

Thanks again for your help!