Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: De-duplicate mailstore

  1. #1
    bill.carlson is offline Intermediate Member
    Join Date
    Nov 2008
    Location
    US
    Posts
    21
    Rep Power
    6

    Default De-duplicate mailstore

    Hey all,

    One of the useful features of Zimbra is that duplicate emails only get stored in the mailstore once. Tastes great, less filling.

    Now, suppose a bunch of accounts are moved to a new server/mailstore via zmmailboxmove and those accounts received much the same emails. Oops, mailstore size skyrockets!

    Is there a utility or process to de-dup a given mailstore?

  2. #2
    veronica is offline Outstanding Member
    Join Date
    Jun 2008
    Posts
    594
    Rep Power
    7

    Default

    De-dupe is per mail store as the users moved to separate mail store have new sql database.

  3. #3
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,366
    Rep Power
    10

    Default

    Bill,

    Are you saying that if user A and B (whose mailboxes are on server C at first) have been emailing each other big PowerPoint files, then have their mailboxes moved to server D that the hard links on server D for those PowerPoint files no longer exist (and so the store size goes up)?

    We've never tested that on our end.

    All the best,
    Mark

  4. #4
    bill.carlson is offline Intermediate Member
    Join Date
    Nov 2008
    Location
    US
    Posts
    21
    Rep Power
    6

    Default

    My scenario is the following:

    Users A,B,C,D on server S1 are on a mailing list and keep all their messages, so all the messages they receive are the same for each account.

    Say User D is moved to server S2 via zmmailboxmove, where S2 uses a different mailstore. Then User C is moved to S2. The mailstore on S2 will be twice the size on S1.

    I'm seeing this in practice, I've moved roughly half (in terms of number AND size) my users to a new server yet the mailstore size on the new server is much larger than the old server, by a factor of 2. I have users that get CC'd on a large number of emails.

  5. #5
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,366
    Rep Power
    10

    Default

    I guess I would say I am not surprised at that.

    Preserving the single instance store during a mailbox move would require the move script to compare the blobs in the mailbox being moved to every blob in the store on the target server in order to decide whether to create a new hard link or a new blob.

    That sounds non-trivial in terms of programming complexity and very, very demanding of compute resources.

    Veronica has already pointed out that the single-instance store is a creature of each mailbox server, not of a Zimbra multi-server farm, so this again seems "WAD" to me. ("Working As Designed" in old IBM mainframe-speak).

    Wouldn't hurt to fill out an RFE though; I'd vote for it.

    But the takeaway for me here is to be careful about correctly sizing a Zimbra mailbox server up front for the expected life of the server, so as to avoid the need to move mailboxes unless absolutely necessary. Or alternatively, to use 64-bit Xen deployments to move the Zimbra virtual server to new hardware when needed to avoid having to move mailboxes.

    Hope that helps,
    Mark

  6. #6
    bill.carlson is offline Intermediate Member
    Join Date
    Nov 2008
    Location
    US
    Posts
    21
    Rep Power
    6

    Default

    Quote Originally Posted by LMStone View Post
    Preserving the single instance store during a mailbox move would require the move script to compare the blobs in the mailbox being moved to every blob in the store on the target server in order to decide whether to create a new hard link or a new blob.
    I agree, overly complex. Makes more sense to do a batch utility that combs through the mail store and de-dupes. Which is what I was hoping had already been written.

  7. #7
    cayaraa's Avatar
    cayaraa is offline Special Member
    Join Date
    Jul 2009
    Posts
    100
    Rep Power
    5

    Default

    In that same vein of thought: Do emails get duped when doing an imap migration to zimbra? A batch script that de-duped would be very helpful for reducing the message store size in that scenario too.

  8. #8
    Klug's Avatar
    Klug is offline Moderator
    Join Date
    Mar 2006
    Location
    Beaucaire, France
    Posts
    2,292
    Rep Power
    13

    Default

    Quote Originally Posted by cayaraa View Post
    In that same vein of thought: Do emails get duped when doing an imap migration to zimbra?
    Yes and in PST migration too.

    Quote Originally Posted by cayaraa View Post
    A batch script that de-duped would be very helpful for reducing the message store size in that scenario too.
    Single-Copy Message Store and imapsync

    However, if you use "manual hardlinks" instead of integrated SIS, what happens if one user deletes the mail the hardlink points to?

  9. #9
    y@w's Avatar
    y@w
    y@w is offline Moderator
    Join Date
    Jan 2008
    Posts
    658
    Rep Power
    8

    Default

    Did anyone create an RFE for this? I couldn't find one and would gladly fill it out as this is a feature that would be incredibly useful for us.

  10. #10
    brian is offline Project Contributor
    Join Date
    Jul 2006
    Posts
    623
    Rep Power
    9

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. How to eliminate Duplicate Emails
    By dchristiaan in forum Administrators
    Replies: 3
    Last Post: 07-10-2009, 08:25 AM
  2. Need to move duplicate messages
    By ashish_clarion in forum Zimlets
    Replies: 2
    Last Post: 06-26-2009, 06:37 AM
  3. Replies: 0
    Last Post: 10-10-2008, 01:47 PM
  4. Prevent duplicate delivery
    By stace in forum Administrators
    Replies: 6
    Last Post: 07-13-2007, 06:34 AM
  5. A user on more then one mailstore.
    By The_W in forum Administrators
    Replies: 3
    Last Post: 05-12-2006, 08:23 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •