Results 1 to 7 of 7

Thread: [Migration] Imapsync TOO slow.

  1. #1
    ElMoRsA is offline Intermediate Member
    Join Date
    Mar 2009
    Posts
    15
    Rep Power
    6

    Default [Migration] Imapsync TOO slow.

    Well, i'm currently in the process of migrating many mboxes (1TB total) into zimbra's mail format.

    After reading and investigating I got to know imapsync, which is supposed to be the best way for migrating user's mail. So I decided to give it a try with my own mailbox, whose size is 1.4GB.

    The first imapsync I've launched throwed these statistics (the last 3 lines are the output of time):

    ++++ Statistics ++++
    Time : 8000 sec
    Messages transfered : 74163
    Messages skipped : 0
    Total bytes transfered : 1446676706
    Total bytes skipped : 0
    Total bytes error : 56010292
    Detected 9 errors
    Please, rate imapsync at imapsync | freshmeat.net

    real 133m20.374s
    user 8m20.417s
    sys 0m27.247s


    133 minutes for the first imapsync looked like veeery slow. But I was shocked when, 4 hours after and without any load on Servers I resynced again both imap's (remember they were synced 4 hours ago and they only have 40 mails for transfer).

    I then got these horrible results:

    ++++ Statistics ++++
    Time : 17063 sec
    Messages transfered : 404
    Messages skipped : 73774
    Total bytes transfered : 4771170
    Total bytes skipped : 1441129069
    Total bytes error : 56010292
    Detected 9 errors
    Please, rate imapsync at imapsync | freshmeat.net

    real 284m22.895s
    user 11m18.068s
    sys 0m22.968s


    Yep, 284 MINUTES for just 404 messages (4mb)!

    I must say my mailbox is stored at a NAS and our current IMAP server is mounting it via NFS. It's important to say that while doing these tests NAS CPU Usage and IMAP Server load average never reached high levels. Also, Zimbra store remains locally on a PowerEdge 2950 with 10k RPM SAS disks.

    In order to speed it up a little bit (ha, ha, ha) I didn't even use SSL for IMAP.

    All these tests were done under RHEL 5 x86_64 and imapsync 1.99.

    Does anyone have any clues on how could I speed up this process ? Or any alternate way of migrating from mailbox to zimbra ?!

    Regards,

  2. #2
    Rich Graves is offline Outstanding Member
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    719
    Rep Power
    9

    Default

    2 hours for a gig is quite good.

    All configurations are different, but the second run shouldn't have taken more than a half hour.

    Where is imapsync itself running? It needs a lot of RAM.

    Have you given imapsync a --header option? I don't remember what the default is. I'd imagine that fetching an unindexed header from Zimbra's imap would require reading the actual message files and be slow.

    Look at zmstat-chart. It's something like zmstat-chart -d /opt/zimbra/jetty/webapps/zimbra/downloads -s /opt/zimbra/zmstat and then you can see various info at http://server/downloads/index.html. There are some wiki articles about zmstat.

  3. #3
    ElMoRsA is offline Intermediate Member
    Join Date
    Mar 2009
    Posts
    15
    Rep Power
    6

    Default

    Thanks Rich for your reply.

    The server is a PE2950 with 8GB of RAM with two quad-core.

    I don't see a --header option at imapsync, but i DO notice an option named "--skipheader":
    --skipheader : Don't take into account header keyword
    matching ex: --skipheader 'X.*'


    I don't know if that's the option you were talking about.

    zmstat gave nice info, good one =). As far as I can see, processors get a max of 20% load each while doing imapsync. Also, during sync Disk I/O throughput reached 800kb/s avg, which seems to be very low for such a server. I don't think hardware is being a limitation over here, at least that's my first impression!

    You say that maybe imapsync is fetching an unindexed header from Zimbra's imap. Is there any way I can force that indexation ?

    Thanks again and again =)

    Quote Originally Posted by Rich Graves View Post
    2 hours for a gig is quite good.

    All configurations are different, but the second run shouldn't have taken more than a half hour.

    Where is imapsync itself running? It needs a lot of RAM.

    Have you given imapsync a --header option? I don't remember what the default is. I'd imagine that fetching an unindexed header from Zimbra's imap would require reading the actual message files and be slow.

    Look at zmstat-chart. It's something like zmstat-chart -d /opt/zimbra/jetty/webapps/zimbra/downloads -s /opt/zimbra/zmstat and then you can see various info at http://server/downloads/index.html. There are some wiki articles about zmstat.

  4. #4
    UMDjwain is offline Trained Alumni
    Join Date
    Nov 2008
    Location
    Ann Arbor, MI
    Posts
    46
    Rep Power
    6

    Default

    Two imapsync options that help a lot with speed are --nosyncacls and --nofoldersizes. I'm usually seeing 1 GB of mail transferred in under an hour, but imapsync definitely depends on how many threads you're running in parallel and what kind of box you're running it on (consider doing the imapsync on a 3rd box between your two mail servers instead of on them directly).

  5. #5
    Rich Graves is offline Outstanding Member
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    719
    Rep Power
    9

    Default

    What I was thinking of before was --useheader.

  6. #6
    ElMoRsA is offline Intermediate Member
    Join Date
    Mar 2009
    Posts
    15
    Rep Power
    6

    Default

    I don't have those options available on imapsync 1.99:

    Unknown option: nosyncacls
    Unknown option: nofoldersizes
    Unknown option: useheader


    I have uploaded to a public folder zmstat results:

    zmstat Aug 12, 2009

    I will try to use different versions of imapsync in order to check if there are those options available.

    Regards!

  7. #7
    thorng is offline Active Member
    Join Date
    Apr 2009
    Posts
    46
    Rep Power
    6

    Default

    1. Here are you running the imapsync from? on Zimbra / on old server / on a client system?
    2. What is the iowait on the old server when you run imapsync?

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 5
    Last Post: 07-20-2009, 07:37 AM
  2. Imapsync and Outlook and nested folders
    By computerfixitguy in forum Migration
    Replies: 0
    Last Post: 06-19-2008, 07:00 AM
  3. imapsync and nginx
    By Rambo in forum Migration
    Replies: 3
    Last Post: 06-04-2008, 07:54 AM
  4. imapsync "out of memory"
    By EFSNetworks in forum Administrators
    Replies: 9
    Last Post: 10-13-2006, 01:51 PM
  5. imapsync and mail filters
    By Coilcore in forum Administrators
    Replies: 5
    Last Post: 08-10-2006, 08:21 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •