conversations a bit broken?

[johnmcc]

i installed zimbra on FC4 and pumped about a GB of mail into it with imapsync.

everything seems to be working, except that conversations often seem to include completely unrelated messages.

is this a known problem? any fixes?

regards, John

[anand]

can you give us some examples?

[johnmcc]

Originally Posted by anand

can you give us some examples?

sure - mail headers below (with obfuscated email addresses)

the first header is from the final in an exchange between myself and zzzzzzz@connect.com.fj. you can see the References: header - all the message in the conversation show up.

the conversation also includes an unrelated message - see the second header.

any ideas?



Return-Path: <zzzzzzz@connect.com.fj>
Delivered-To: xxxxxx@espresso.integrated-mapping.com
Received: from meme.inmap.co.nz (unknown [202.27.110.12])
by espresso.integrated-mapping.com (Postfix) with ESMTP id EC31A1DC048
for <xxxxxx@espresso>; Wed, 16 Oct 2002 23:19:38 -0400 (EDT)
Received: from mx.inmap.co.nz (mx.inmap.co.nz [202.27.110.240])
by meme.inmap.co.nz (8.9.3/8.9.3) with ESMTP id DAA27763
for <xxxxxx@inmap.co.nz>; Thu, 17 Oct 2002 03:19:40 GMT
Received: from homer.is.com.fj (homer.is.com.fj [202.62.124.238])
by mx.inmap.co.nz (Postfix) with ESMTP id DF3A5C73C
for <xxxxxx@integrated-mapping.com>; Thu, 17 Oct 2002 03:19:36 +0000 (UTC)
Received: from NEWNOTEBOOK (luts5300-dialup1.is.com.fj [202.62.121.17])
by homer.is.com.fj (8.12.2/8.12.2) with SMTP id g9H3JL5a033056
for <xxxxxx@integrated-mapping.com>; Thu, 17 Oct 2002 15:19:28 +1200 (FJT)
Message-ID: <003f01c2758c$01745e70$057a3eca@NEWNOTEBOOK>
From: "zzzzzzz" <zzzzzzz@connect.com.fj>
To: "xxxxxx McCombs" <xxxxxx@integrated-mapping.com>
References: <f02101018074785DF@local> <3DA74950.8000006@integrated-mapping.com>
<000f01c2725f$b9f95520$3f7a3eca@NEWNOTEBOOK> <3DA8F107.1040808@integrated-mapping.com>
<002f01c2728b$6eb1c580$207a3eca@NEWNOTEBOOK> <3DAA31FA.6000203@integrated-mapping.com>
<021901c27342$d43c69c0$1b793eca@NEWNOTEBOOK> <3DADE2D3.2090004@integrated-mapping.com>
<002601c2756d$9bd77930$0f7a3eca@NEWNOTEBOOK> <3DAE06B7.1000400@integrated-mapping.com>
Subject: Re: network
Date: Thu, 17 Oct 2002 16:16:01 +1300
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000


------------------------------
Return-Path: <yyyyyyy@yyy.com>
Delivered-To: xxxxxx@integrated-mapping.com
Received: from incognito.integrated-mapping.com (unknown [202.27.110.240])
by espresso.integrated-mapping.com (Postfix) with ESMTP id E00151DC048
for <xxxxxx@espresso.integrated-mapping.com>; Wed, 12 Oct 2005 08:30:37 +1300 (NZDT)
Received: from grunt6.ihug.co.nz (grunt6.ihug.co.nz [203.109.254.46])
by incognito.integrated-mapping.com (Postfix) with ESMTP id 4DA622D341D
for <xxxxxx@inmap.co.nz>; Wed, 12 Oct 2005 08:30:36 +1300 (NZDT)
Received: from 166-179-19-48.jamamobile.co.nz (eowyn.yyy.com) [166.179.19.48]
by grunt6.ihug.co.nz with asmtp (Exim 3.35 #1 (Debian))
id 1EPPpZ-0004KT-00; Wed, 12 Oct 2005 08:30:33 +1300
Message-Id: <6.1.2.0.0.20051012082902.01bd4568@pop.ihug.co.n z>
X-Sender: kkkkkkk@pop.ihug.co.nz
X-Mailer: QUALCOMM Windows Eudora Version 6.1.2.0
Date: Wed, 12 Oct 2005 08:30:33 +1300
To: "xxxxxx" <xxxxx@inmap.co.nz>,
xxxxx@incognito.integrated-mapping.com
From: "yyyyyyy" <yyyyyyy@yyy.com>
Subject: network
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed

[sam]

Zimbra conversations are based on subject. The two messages above have the same normalized subject (the normalized subject is the name of the conversation).

Conversations work as follows:

1. A message is received and the subject is normalized
2. if no conversation exists for the normalized subject, a new conversation is created and this is the first msg in the conversation.
3. if a conversation exists for the normalized subject
3a. if the subject of the current msg has no subject prefix, the existing conversation is closed, a new conversation is created and this is the first msg in the new conversation.
3b. if the subject of the current msg has a subject prefix, the new msg is appended to the existing conversation.

You can see this algorithm is based on the order messages are received. When migrating messages, it is important to migrate the messages in time order, from oldest to newest, to allow the server to build conversations correctly.

Given this description, is the behavior you are experiencing correct?

[johnmcc]

thanks for the response

Originally Posted by sam

Zimbra conversations are based on subject. The two messages above have the same normalized subject (the normalized subject is the name of the conversation).

Conversations work as follows:
....
Given this description, is the behavior you are experiencing correct?

I guess it is. You're saying that to do the migration you need a tool that will build a message list across all of a users folders and then upload them in that order? Are there any tools that will do this?

Why doesn't zimbra use the References and In-reply-to headers? The current algorithm seems to have 3 weaknesses

1. if you use imapsync, it gets about 20% of the conversations wrong
2. it's going to get it wrong anyway, e.g. I send a message to a mailing list then engage in a series of separate conversations, wont that result is some fairly muddle conversation threads?
3. if someone changes the subject line, the conversation gets split. Not really very helpful as you can never be sure if you're looking at the complete conversation - silent failure

[sam]

Why doesn't zimbra use the References and In-reply-to headers?

If I remember correctly (someone will chime in if I don't), we chose the subject based approach because it is a lowest-common-denomator solution. For example, we couldn't base the conversation-grouping algorithm based on References or In-Reply-To headers because not all MUA's supply them. It is possible that we could enhance the current algorithm to support In-Reply-To and References, but this is pretty low priority on our current roadmap.

You're saying that to do the migration you need a tool that will build a message list across all of a users folders and then upload them in that order? Are there any tools that will do this?

Exactly. The only tool that I am aware of that does this is our Migration Wizard (which currently only supports migrating from Exchange).

[andreychek]

Originally Posted by sam

If I remember correctly (someone will chime in if I don't), we chose the subject based approach because it is a lowest-common-denomator solution. For example, we couldn't base the conversation-grouping algorithm based on References or In-Reply-To headers because not all MUA's supply them. It is possible that we could enhance the current algorithm to support In-Reply-To and References, but this is pretty low priority on our current roadmap.

Sure, it makes a lot of sense to look at the subject. I believe even Outlook struggles to send a proper In-Reply-To, or at least did at one point.

Also, I think the Conversation View is a neat feature.

However, as currently implemented, I think it's of limited use, and perhaps even confusing. A lot of users who don't know any better don't do so well at choosing email subjects.... for our weekly staff meeting, it'll often just be "Meeting Today". Also, most of my family tends to send emails with a subject of "Hi". The current algorithm also makes it much more likely that two concurrent but unrelated conversations could get mixed together.

All the above could make the conversation view somewhat confusing for users.

It wouldn't be so bad, except that conversation mode is the default view. I think I would change that for our users until In-Reply-To is supported.

I have my current email client setup to first look at In-Reply-To, if it exists. If not, fall back to Subject. I've found that sorting to work really well.

I know you guys have plenty to do. That change is simple enough that I might even dig into the code and see if I can figure out how to alter the algorithm... I'll let you know how it goes :-)

Have a good one,
-Eric

[easytouch]

Two unrelated emails, one with an empty subject, and te other only having Re: as subject get mixed together into a conversation.

The user's have stress because they fear that the email to the boss and the email from the girlfriend got mixed up at the recipients side ;-)

As this is the only related post I found, and it's quite old, can someone give me a pointer what to do, or if a bug exists?

[phoenix]

The threaded view does look slightly wrong at times but there is absolutely no chance of them being 'mixed up' - the thread only shows in the users mailbox in the web UI and not anywhere else.

Have a look in bugzilla and see if there are any related topics or RFEs, you could file an entry for better threading (copies of the mail that ends up in the same thread might help - use the show original option).