imapsync "out of memory"

[EFSNetworks]

Any one know how to deal with "out of memory" issues when using imapsync? Is my only choice a faster server?

Here's the complete story:

In production I would like to run a backup Zimbra server in parallel with the main server in order to mirror data in semi-real time.

I know Zimbra is working on these mirroring features to complement clustering redundancy. In the mean time I have am testing the following:

I managed to setup a virtual server running Teknic on FC4 as an "open source" backup box. I also have a demo network edition of 3.0 running locally. I setup cron jobs to run imapsync to backup email messages from the network ed. to the open source ed. nightly. The sync's keep cutting out with "out of memory" errors.

I only have 128mb ram on the virtual server with the following details: I have the 128 plan http://www.unixshell.com/plans.html

Anyone know if I can increase the amount of ram/virtual memory that imapsync can use?

I tried running imapsync from the virutal server as well as the local box -- both cut out because of these errors. The local box can use imapsync to sync with other servers fine (tried it with my previous shared hosting web co).

Any ideas on imapsync or alternative methods for backup across servers?

[bobby]

you have zimbra running in a virtual server with only 128mb ram? and you want imapsync to cram a bunch of messages at it all at once?

actually, does imapsync have an option to pause a little between messages? at any rate, you're probably going to need a lot more ram to do this; imapsync is high-power.

[EFSNetworks]

I upgraded the server to 256mb plan and its working sweet imapsync and zimbra response time...now I gotta try the nightly "maintenance mode" backups between my open source and network edition boxes -- does anyone know if I just rsync the /opt/zimbra directory will this work as a backup?

Also since I am backing up from network to open source edition I probably dont need to change users to maintenance mode since network ed can do hot backup?? Any ideas?

[tyco]

i just migraded a couple accounts form stalker communigate to zimbra with imapsync.

i have 1.5GB ram and a dual processor atlon box.

still getting that Out of Memory Message.

so i started to sync folder by folder with the option: --folder 'paht/to/imap/folder' and still had that error.

someone any idea

christof

[macros]

The real problem here is that imapsync is not well written at all. I've had processes eating easily over 1GB of ram. Mailboxes over 3k items just tend to go bad. I've had it get stuck in a select loop chewing through all the processor.

The biggest area that it leaks is with the header parsing and comparison code. If you are just doing a dump and aren't worried about syncing two mailboxes passing --useheader Message-Id can help a lot. Don't forget --syncinternaldates either.

If I ever planned on using this tool again I might have a go at fixing it, but for now it will live in the realm of unnecessary pain and wasted hours.

Jason

[kirme3]

I actually migrated 23000 accounts in the matter of a weekend using imapsync. Not too sure about the memory issues, our servers have 4 GB in them. Out of 23000 accounts, I've only had a few(maybe a dozen) phone calls of email missing. Maybe it has to do with the way we sync'd mail:

Round 1, all mail in inbox newer than 30 days and also create subfolder structure(but no mail)
Round 2, all mail in subfolders
Round 3, all mail in inbox older than 30 days, but less than 180 and only read mail
Round 4, all mail that has been read and older than 180 days

We are now migrating any other mail on a request basis(and there haven't been many)

We also split our imapsyncs into ~40 processes running simultaneously on multiple machines separate from source and destination servers. Each machine had 2 - 3 imapsyncs running.

[scalper]

Out of memory also occurs during my migration. After checking, I found that the email have 300 MB attachment file..

Is there any workaround for this?

[tyco]

yeah same prob here.
mails with more than arround 60mbs imap sync just exits (

are there any alternatives to immap sync?

thanx

[graffiti]

Originally Posted by kirme3

I actually migrated 23000 accounts in the matter of a weekend using imapsync. Not too sure about the memory issues, our servers have 4 GB in them. Out of 23000 accounts, I've only had a few(maybe a dozen) phone calls of email missing. Maybe it has to do with the way we sync'd mail:

Round 1, all mail in inbox newer than 30 days and also create subfolder structure(but no mail)
Round 2, all mail in subfolders
Round 3, all mail in inbox older than 30 days, but less than 180 and only read mail
Round 4, all mail that has been read and older than 180 days

We are now migrating any other mail on a request basis(and there haven't been many)

We also split our imapsyncs into ~40 processes running simultaneously on multiple machines separate from source and destination servers. Each machine had 2 - 3 imapsyncs running.

Smart plan. Would you mind sharing the arguments of imapsync of each round?

TIA,

-g

[czaveri]

Originally Posted by tyco

are there any alternatives to immap sync?

ImapSync looks like the best one. But you can evaluate the others, too. These forums name, I think, 3 other products similar to Imap.

Grep others from here: http://www.imap.org/products/showall.php