Results 1 to 10 of 10

Thread: imapsync "out of memory"

  1. #1
    EFSNetworks is offline Junior Member
    Join Date
    Jan 2006
    Posts
    9
    Rep Power
    9

    Default imapsync "out of memory"

    Any one know how to deal with "out of memory" issues when using imapsync? Is my only choice a faster server?

    Here's the complete story:

    In production I would like to run a backup Zimbra server in parallel with the main server in order to mirror data in semi-real time.

    I know Zimbra is working on these mirroring features to complement clustering redundancy. In the mean time I have am testing the following:

    I managed to setup a virtual server running Teknic on FC4 as an "open source" backup box. I also have a demo network edition of 3.0 running locally. I setup cron jobs to run imapsync to backup email messages from the network ed. to the open source ed. nightly. The sync's keep cutting out with "out of memory" errors.

    I only have 128mb ram on the virtual server with the following details: I have the 128 plan http://www.unixshell.com/plans.html

    Anyone know if I can increase the amount of ram/virtual memory that imapsync can use?

    I tried running imapsync from the virutal server as well as the local box -- both cut out because of these errors. The local box can use imapsync to sync with other servers fine (tried it with my previous shared hosting web co).

    Any ideas on imapsync or alternative methods for backup across servers?

  2. #2
    bobby is offline Zimbra Employee
    Join Date
    Nov 2005
    Posts
    518
    Rep Power
    10

    Default

    you have zimbra running in a virtual server with only 128mb ram? and you want imapsync to cram a bunch of messages at it all at once?

    actually, does imapsync have an option to pause a little between messages? at any rate, you're probably going to need a lot more ram to do this; imapsync is high-power.

  3. #3
    EFSNetworks is offline Junior Member
    Join Date
    Jan 2006
    Posts
    9
    Rep Power
    9

    Default

    I upgraded the server to 256mb plan and its working sweet imapsync and zimbra response time...now I gotta try the nightly "maintenance mode" backups between my open source and network edition boxes -- does anyone know if I just rsync the /opt/zimbra directory will this work as a backup?

    Also since I am backing up from network to open source edition I probably dont need to change users to maintenance mode since network ed can do hot backup?? Any ideas?

  4. #4
    tyco is offline Senior Member
    Join Date
    May 2006
    Posts
    52
    Rep Power
    8

    Default same problem here with fast server

    i just migraded a couple accounts form stalker communigate to zimbra with imapsync.

    i have 1.5GB ram and a dual processor atlon box.

    still getting that Out of Memory Message.

    so i started to sync folder by folder with the option: --folder 'paht/to/imap/folder' and still had that error.

    someone any idea

    christof

  5. #5
    macros is offline Starter Member
    Join Date
    Sep 2006
    Posts
    1
    Rep Power
    8

    Default

    The real problem here is that imapsync is not well written at all. I've had processes eating easily over 1GB of ram. Mailboxes over 3k items just tend to go bad. I've had it get stuck in a select loop chewing through all the processor.

    The biggest area that it leaks is with the header parsing and comparison code. If you are just doing a dump and aren't worried about syncing two mailboxes passing --useheader Message-Id can help a lot. Don't forget --syncinternaldates either.

    If I ever planned on using this tool again I might have a go at fixing it, but for now it will live in the realm of unnecessary pain and wasted hours.

    Jason

  6. #6
    kirme3 is offline Trained Alumni
    Join Date
    Apr 2006
    Location
    Illinois
    Posts
    194
    Rep Power
    9

    Default imapsync was successful for me

    I actually migrated 23000 accounts in the matter of a weekend using imapsync. Not too sure about the memory issues, our servers have 4 GB in them. Out of 23000 accounts, I've only had a few(maybe a dozen) phone calls of email missing. Maybe it has to do with the way we sync'd mail:

    Round 1, all mail in inbox newer than 30 days and also create subfolder structure(but no mail)
    Round 2, all mail in subfolders
    Round 3, all mail in inbox older than 30 days, but less than 180 and only read mail
    Round 4, all mail that has been read and older than 180 days

    We are now migrating any other mail on a request basis(and there haven't been many)

    We also split our imapsyncs into ~40 processes running simultaneously on multiple machines separate from source and destination servers. Each machine had 2 - 3 imapsyncs running.

  7. #7
    scalper is offline Partner (VAR/HSP)
    Join Date
    Jul 2006
    Location
    KL, Malaysia
    Posts
    123
    Rep Power
    8

    Default

    Out of memory also occurs during my migration. After checking, I found that the email have 300 MB attachment file..

    Is there any workaround for this?

  8. #8
    tyco is offline Senior Member
    Join Date
    May 2006
    Posts
    52
    Rep Power
    8

    Default big mails

    yeah same prob here.
    mails with more than arround 60mbs imap sync just exits (

    are there any alternatives to immap sync?

    thanx

  9. #9
    graffiti is offline Loyal Member
    Join Date
    Sep 2005
    Posts
    95
    Rep Power
    9

    Default

    Quote Originally Posted by kirme3
    I actually migrated 23000 accounts in the matter of a weekend using imapsync. Not too sure about the memory issues, our servers have 4 GB in them. Out of 23000 accounts, I've only had a few(maybe a dozen) phone calls of email missing. Maybe it has to do with the way we sync'd mail:

    Round 1, all mail in inbox newer than 30 days and also create subfolder structure(but no mail)
    Round 2, all mail in subfolders
    Round 3, all mail in inbox older than 30 days, but less than 180 and only read mail
    Round 4, all mail that has been read and older than 180 days

    We are now migrating any other mail on a request basis(and there haven't been many)

    We also split our imapsyncs into ~40 processes running simultaneously on multiple machines separate from source and destination servers. Each machine had 2 - 3 imapsyncs running.
    Smart plan. Would you mind sharing the arguments of imapsync of each round?

    TIA,

    -g
    Last edited by graffiti; 10-13-2006 at 01:30 PM.

  10. #10
    czaveri's Avatar
    czaveri is offline Former Zimbran
    Join Date
    Feb 2006
    Location
    Pune, India
    Posts
    294
    Rep Power
    9

    Default

    Quote Originally Posted by tyco
    are there any alternatives to immap sync?

    ImapSync looks like the best one. But you can evaluate the others, too. These forums name, I think, 3 other products similar to Imap.

    Grep others from here: http://www.imap.org/products/showall.php
    Regards,

    Chintan Zaveri
    (Yet another ZIMBRAN!)

    "Dhundhne par Bhagwan bhi ..."

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Migrating from Cyrus to Zimbra - imapsync
    By claud1e in forum Migration
    Replies: 4
    Last Post: 08-26-2009, 12:59 AM
  2. maximizing imapsync speed - parallel runs
    By bewley in forum Migration
    Replies: 4
    Last Post: 09-30-2008, 03:14 PM
  3. problems with imapsync from cyus
    By g8se in forum Migration
    Replies: 3
    Last Post: 02-22-2007, 07:52 PM
  4. imapsync & dovecot
    By PenguinWrangler in forum Migration
    Replies: 3
    Last Post: 01-18-2007, 03:36 PM
  5. imapsync and mail filters
    By Coilcore in forum Administrators
    Replies: 5
    Last Post: 08-10-2006, 08:21 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •