I'm working on migrating 224 users with about 74G of mail.
I really would prefer to do a mass cutover instead of a drawn out split domain migration. I'll do a full sync once then do a final sync with -delete2 after shutting off the email flow during the cutover.
So, I'm trying to find the quickest way to run imapsync. Here are some stats from an acount sync with imapsync running directly on the zimbra server.
- Messages: 33015
- Bytes: 1292550810
- Time: 02:21:00
- Msg/Sec: 3.9
That adds up to a lot of hours. I could turn off the content indexing in Zimbra, but it looks to me like imapsync is the biggest load on the system and zimbra could take much more. This is a pretty good sized Sun v40z; quad 2.4G opteron 16G.
Code:
top - 10:29:47 up 44 days, 18:21, 3 users, load average: 2.72, 2.70, 2.58
Tasks: 153 total, 2 running, 151 sleeping, 0 stopped, 0 zombie
Cpu(s): 33.3% us, 1.8% sy, 0.0% ni, 57.7% id, 7.0% wa, 0.1% hi, 0.1% si
Mem: 16359500k total, 16335916k used, 23584k free, 477672k buffers
Swap: 5261240k total, 274304k used, 4986936k free, 9218064k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8486 root 25 0 244m 168m 3084 R 100 1.1 40:28.97 imapsync.1.213
3432 zimbra 17 0 5418m 3.4g 19m S 35 22.0 509:57.15 java
30026 zimbra 16 0 5478m 1.4g 4548 S 2 8.7 77:14.97 mysqld
9667 zimbra 19 0 472m 42m 12m S 1 0.3 3:04.42 java
406 root 16 0 6276 1080 772 R 1 0.0 0:00.28 top
467 root 15 0 0 0 0 S 0 0.0 57:22.75 md0_raid5
2270 root 15 0 0 0 0 S 0 0.0 32:26.25 kjournald
So, I'm going to try running multiple imapsync processes on terciary servers and see if I can speed up the whole process by parallelizing it. I'm thinking 4 at a time. Anyone have any experience with that?
p.s. Here is how I'm running the sync. Feel free to copy.
Code:
#!/bin/bash
################################################################################
# $Id: sync.sh,v 1.2 2007/03/07 23:15:27 bewley Exp $
#-------------------------------------------------------------------------------
# Description:
# Syncs IMAP data from old Dovecot IMAP server to new Zimbra server.
#
# Set DOVECOT_MASTER to use "<username>*<masteruser>" logins. Include the "*".
# See http://wiki.dovecot.org/MasterPassword
#
# Usage:
# ./sync.sh <username>
#
################################################################################
USER_DIR=users
USER_DEFAULT=myself
PWORD_DEFAULT="$USER_DIR/.pwd"
HOST_FROM=mail
HOST_TO=zimbra
# Appended to username @ HOST_FROM see http://wiki.dovecot.org/MasterPassword
DOVECOT_MASTER="*zimbra"
# user on command line
if [ ! -z "$1" ]; then
USER="$1"
else
USER="$USER_DEFAULT"
fi
if [ ! -d "$USER_DIR" ]; then
echo "making dir $USER_DIR"
mkdir $USER_DIR
chmod 700 $USER_DIR
fi
if [ -r "$USER_DIR/$USER.pwd" ]; then
PWORD="$USER_DIR/$USER.pwd"
else
PWORD=$PWORD_DEFAULT
fi
# setup log file and begin
LOG="$USER_DIR/$USER.log"
date >> $LOG
echo "Syncing ${USER}${DOVECOT_MASTER}@$HOST_FROM to $USER@$HOST_TO with $PWORD"
time ./imapsync.1.213 \
--ssl1 --authmech1 PLAIN --host1 $HOST_FROM --user1 "${USER}${DOVECOT_MASTER}" \
--passfile1 $PWORD \
--exclude '^My-spam-\d\d' \
--ssl2 --authmech2 PLAIN --host2 $HOST_TO --user2 $USER --passfile2 $PWORD \
--regextrans2 's/My-spam/Junk/' \
--regextrans2 's/:/./g' \
--delete2 \
--syncinternaldates \
>> $LOG
date >> $LOG