[SOLVED] Using zmlmtpinject

[tron]

I have the zimbra store directory containing the email messages as a backup. What is the best way to reinject the messages back into zimbra using zmlmtpinject? Is there documentation other than the command line help?

[KevinH]

See Dan's post below.

[tron]

So there is no way to re-inject the messages from the zimbra store directory back into zimbra?

Originally Posted by KevinH

EDITED... See Dan's response below.

[dkarp]

Check out this thread for zmlmtpinject syntax.

Your message store should be structured like this:

/opt/zimbra/store/{hash}/{mailbox-id}/msg/{hash}/{item}.msg

In other words, all messages in an on-disk directory belong to the same user. So, once you've determined whose mailbox the messages correspond to, just use zmlmtpinject to inject them into the new Zimbra M2 server. You'll lose tags, contacts, folder structure, received dates, and read/unread status, but it should work.

If you want to be extra-tricky, before you run zmlmtpinject you can write a perl script to add an extra "X-Zimbra-Received" header to each message, with the value being the same as the "Date:" header's value. Then zmlmtpinject will use that date as the "received date" when re-injecting the messages.

[tron]

Thank you. I followed your direction and wrote a little perl script to insert the X-Zimbra-Received header and zmlmtpinject worked.

Can you share with us the hashing algorithm that zimbra uses so that it would make locating a user's mail directory much easier?

Originally Posted by dkarp

Check out this thread for zmlmtpinject syntax.

Your message store should be structured like this:

/opt/zimbra/store/{hash}/{mailbox-id}/msg/{hash}/{item}.msg

In other words, all messages in an on-disk directory belong to the same user. So, once you've determined whose mailbox the messages correspond to, just use zmlmtpinject to inject them into the new Zimbra M2 server. You'll lose tags, contacts, folder structure, received dates, and read/unread status, but it should work.

If you want to be extra-tricky, before you run zmlmtpinject you can write a perl script to add an extra "X-Zimbra-Received" header to each message, with the value being the same as the "Date:" header's value. Then zmlmtpinject will use that date as the "received date" when re-injecting the messages.

[dkarp]

Originally Posted by tron

Can you share with us the hashing algorithm that zimbra uses so that it would make locating a user's mail directory much easier?

Mailboxes are assigned increasing numeric IDs when they're created. The account ID -> mailbox ID mapping is in the ZIMBRA.MAILBOX table in the old install's database, which you may or may not any longer have access to. (There's a COMMENT field in that table that by default holds the email address of the mailbox owner.)

The "hash" is just what you get by right-shifting the mailbox ID by N bits, so for your system all mailboxes will hash to 0.

[gitzc]

I have successfully used zmlmtpinject to import mail into Zimbra. The X-Zimbra-Received header was exactly what I needed. One more question on this topic:

Messages sent by the user aren't being imported quite right. They go to the Inbox, which is not a problem, but when I move them to the Sent Items folder, it shows the sender in the "To" field, not the recipient. Are there any other "X-Zimbra-" headers that may be useful in importing mail?

Below is the procedure I used to successfully import mail using zmlmtpinject. I wrote a bash script called zimdates to add the "X-Zimbra-Received" header. I will include the script at the end of this post. I put the script in my home directory (/home/user1) and ran the following commands as root:

cd /home/user1
./zimdates /path/to/user1/maildir/
cd /path/to/user1/maildir/
/opt/zimbra/bin/zmlmtpinject -d ./ -r user1@example.com \
-s root@example.com

zimdates:

#!/bin/bash
#
# zimdates
# Chris Gitzlaff 2005-11-16
#
# This script inserts an X-Zimbra-Received header into each message
# immediately after the Date header.
#

SCRIPTDIR=`pwd`
TMPFILE="$SCRIPTDIR/zimdates.tmp"

show_usage() {
   echo "Usage: zimdates DIRECTORY"
   echo "Inserts the X-Zimbra-Received header into a directory of messages"
   echo
   echo "Example: zimdates ./mail/"
}

# check for correct usage: 1 argument (directory)
if [ $# -eq 1 ]; then
   MSGDIR=$1
   if [ ! -d $MSGDIR ]; then
      show_usage
      exit 1
   fi
else
   show_usage
   exit 1
fi

# if the temporary file exists, delete it
if [ -f $TMPFILE ]; then
   rm -f $TMPFILE
fi

cd $MSGDIR
for file in *
do
   grep "^Date\:\ " $file > $TMPFILE
   # use the first Date occurrence
   datestring=`sed -n '1p' $TMPFILE`
   # remove the 'Date: ' prefix
   datestring=${datestring#*\ }

   sed -n '1,/^Date\:\ /p' $file > $TMPFILE
   echo "X-Zimbra-Received: $datestring" >> $TMPFILE
   sed '1,/^Date\:\ /d' $file >> $TMPFILE

   mv $TMPFILE $file
done

[dkarp]

Unfortunately, at present there are no headers that will do what you want. We mark a message as "sent by me" -- the criterion used to decide whether to display the sender or the recipients in the Sent folder -- only at send time, and only if it's automatically saved to sent.

[mutinda]

This post REALLY saved my bacon. Thanks for the script.

[RageLink]

I usually back up my mysql when I do zimbra backups and I took the liberty of wrapping your script with this other snippet to auto parse all folders simultanously. Usually this works if I keep a full backup and only do incremental message store backups to sort out new messages. I full backup every week (turn off zimbra and tar the whole /opt/zimbra) and then rsync the message store periodically.

asuming you are able to dump the mysql table with something like

mysql $> use zimbra;
mysql $> select id,description from mailbox;

and save that to list.txt (or something). I wrapped it up with this.

#!/bin/bash


for x in `awk '{print "/opt/zimbra/store/0/"$1"/msg"}' /opt/list.txt`; do
        for i in `ls $x`; do
                echo Zimdates::: sh /opt/zimdates.sh $x"/"$i;
                sh /opt/zimdates.sh $x"/"$i;
        done;
done;

for k in `awk '{print $1":::"$2}' /opt/list.txt`; do

        echo "TEST::: /opt/zimbra/bin/zmlmtpinject `echo $k | awk '{ split($0, a, ":::"); print "-d /opt/zimbra/store/0/"a[1]"/msg -r "a[2]" -s "a[2]; exit}'`";
        /opt/zimbra/bin/zmlmtpinject `echo $k | awk '{ split($0, a, ":::"); print "-d /opt/zimbra/store/0/"a[1]"/msg/* -r "a[2]" -s "a[2]; exit}'`;
done;

just a thought.