Single-Copy Message Store and imapsync

[mlippold]

Will transferring messages to Zimbra using imapsync create a single copy of messages sent to multiple users on the same server?

[dkarp]

No, imapsync does not preserve the single-copy store.

[mlippold]

I figured the MTA was responsible for this function. Is it a feature of Postfix?

Is there a way to rebuild the single-message store?

I am thinking about whipping up a Perl script to find messages in the message store that are the same and rebuilding the hard links. Do you see any issue with doing this?

[Rich Graves]

It's a feature of the tomcat/mailboxd backend.

Rather than writing your own, look at freedups file listing

Unless the other end of the imapsync was also a single-instance store, you're unlikely to get much. The messages truly need to be *identical*, down to the final Received: header and delivery time.

[mikelcu]

If imapsync essentially breaks the single message store, what is the recommendation on migrating a reasonably large set of mailboxes with a large number of shared mails? Does the exchange migration utility preserve the single store?

In my case, I've got ~100 mailboxes totaling more than 60GB that are chock-full of distribution-list and other mails stored on disk only once. If 25% of the mails are multi-instance-single-stored in the source mail server (ours could be more), that could throw off the storage calculations pretty badly...

-Mike

[Rich Graves]

Import wizards won't change the dedupe story on the Zimbra server end.

If you've really got 25% shared mail to a large number of recipients and the distribution list messages contain attachments as big or bigger than personal messages, then indeed you do have a (very unusual) problem. I'm skeptical that it's as big a problem as you think, but I can think of a few workarounds.

If the RFC822 message content that you can get out of Exchange is really *identical*, then you can probably de-duplicate retroactively with a program like hardlink or freedups. Migrate (portions of) two accounts and compare the "View Source" of each. I'd worry that Exchange or the import process would alter something in the headers. Try both imapsync and the import wizard to see if their behavior is different. If you can manage to end up with perfect duplicate blobs in different accounts, then pointing freedups at /opt/zimbra/store will recover disk space.

If freedups won't work for you and you have a particular sender or subject that has filled everyone's mailbox with crap, you can tell imapsync to ignore it, make a shared mailbox including all of the distribution list's mail, and subscribe everyone to that shared mailbox.

Storage technologies exist that virtualize and de-duplicate based on file-level or block-level content. They are mostly targeted at archive and backup, but NetApp advertises this for primary storage. Buzzphrases include "content-based addressing" and "global compression." These solutions tend to be far more expensive than would be justified for only 60GB, but if you're in the market for storage for other purposes, take a look.

Last and least attractively, you could ask people to move bulk list mail to local PST folders, and only migrate what remains on the server.

[stegbth]

hello everybody,

just to be sure.

zimbra support's
Single-Copy Message Store (or singleinstancestore as called on Cyrus imapd), if delivered eMails via smtp to zimbra
does this work out of the box or must it get activated?
especially if always_bcc is activated on Postfix?

after moving a cyrus with imapsync to zimbra it is save to run freedups on /opt/zimbra/store

greetings
thomas