Message duplicates in message store

[stopak]

Hi,

lately I was worried about message duplicates. After checking message store, I found about 170567 duplicates in 836065 messages. Is it possible to somehow merge them together ?

[phoenix]

Originally Posted by stopak

lately I was worried about message duplicates. After checking message store, I found about 170567 duplicates in 836065 messages. Is it possible to somehow merge them together ?

What do you mean by "...merge them together"?

[stopak]

Well, i would like to delete all duplicates. I presume it's not as simple as deleting messages from disk. I would also have to correct MySql data. I have no idea how to do it :/

[Krishopper]

Lets back up for a moment.

How did you determine that there are duplicates?

Have you ever done any kind of copy/backup/restore/migration where you copied the /opt/zimbra directory path using cp, tar, or anything like that?

[stopak]

Yes, lately I was expanding disk space so I had to temporarly move whole zimbra folder. I run fdupes on /opt/zimbra/store and store results in file. Then I count all the non empty lines in file.

[phoenix]

Originally Posted by stopak

Yes, lately I was expanding disk space so I had to temporarly move whole zimbra folder. I run fdupes on /opt/zimbra/store and store results in file.

Copying the Zimbra store doesn't cause duplicate messages.

Originally Posted by stopak

Then I count all the non empty lines in file.

How does that determine that they are duplicate messages with exactly the same content?

[Klug]

It's very easy to have duplicates.

Importing PST files (differents users, same domain) causes duplicates, lots of them 8(

There's currently no "de-dup" tool in ZCS.
De-duplicate mailstore
Vote: Bug 17057 – tool to consolidate duplicate e-mails

[stopak]

Well fdupes prints all paths to duplicate messages (It counts md5 checksum and also compers file size). If I count them I get all messages that are duplicates. Will it work if I change duplicate files to hard links ?

[Klug]

We're waiting for an "official" answer from Zimbra's devs on this.

[soba@ukw.edu.pl]

I don't understand this - why you have a duplicate messages?