Character Support

[ltabora]

Hello,

I'm testing the ZCS Open Source Edition in a RedHat ES 4 x86 Server. Everything seems to be working perfectly, except some character converstions taking places. Accented characters such as: á é í ó ú, ñ, Ñ, etc.. get converted to: �.

It doesn't always happen, when testing emails sent to my server from gmail arrive perfectly, but if I send them from hotmail.com, this strange conversion takes place.

Please help.

LT.

[marcmac]

If the problem occurs or not based on the source of the email - could the problem be the sending account, not the receiving? Try sending from hotmail -> gmail, and see if the same problem occurs.

[ltabora]

If sending email from gmail, I have no problems. If sending email from hotmail, I get the character set conversion.

After testing we have determined that it has more to do with the origin of the messages, although an interesting fact is that if sending from hotmail the email can be correctly viewed from the web interface, but once downloaded to a pop3 account characters get mixed up.

If sending from gmail, the email can be correctly viewed from the web, and correctly viewed from a pop3 account also.

We have tryied with several pop3 clients, such as Outlook, Outlook Express, Mozilla Thunderbird with the same results.

Any ideas?

[dkarp]

Can you attach a sample message for which this problem occurs?

And do I understand correctly that the message is displayed properly in the Zimbra Web UI, but not in Outlook/etc. after downloading it via POP3?

[ltabora]

Yes, that is right, from the Web GUI the characters don't get scrambled. To test this we've been sending emails with the subject: á é í ó ú ñ Ñ, both from hotmail and from gmail.

When sending from gmail, everything works just fine, when sending from hotmail, is when we sense the problem.

LT.

[dkarp]

Can you attach a sample message for which this behavior is occurring?

From the web UI, right-click on the message and "Show Original" will get you the original RFC822 message. Save the message as a .txt file. Then, in the forums, write a posting and "Additional Options > Attach Files" to add the attachment.

[goetzi]

I don't have this problem when receiving mails, but if I send mails to my hotmail account, the special characters are broken (ä, ö, ü, ß, Ä, Ö, Ü). I'm sorry that I can't send you the headers, but I didn't find a possibility to display the headers in the hotmail web GUI.

[chinook]

I'm experiencing the same problem as the one originally reported by LT with all messages containing extended characters that were sent through Hotmail or Yahoo! Mail as well as a few other mailers. As suggested by dkarp, I've attached the output of Show Original from a test message sent through Hotmail. Indeed, there seems to be a problem with the header as reported by Amavis:

X-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char E9 hex) in message header 'Subject': Subject: Testing extended characters in subject line: \351\350\340\357\347\n

This error does not occur when messages sent through Hotmail do not contain extended characters.

So far, I can say that extended characters are being altered when messages are fetched using POP3 in Outlook 2003, Outlook Express and TBird on Windows. Extended characters display correctly in all three clients (as well as Apple Mail) when mailboxes are accessed using IMAP4 or when using the Zimbra Web UI.

I'm running a clean install of M2 OSE on FC4. Thanks...

[ltabora]

Here are the headers for my message, which I'm copy/pasting because it's not that big, and as you may see, it agrees with Chinook's output.

X-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char E1 hex) in message header 'Subject': Subject: Prueba acentos \341 \351 \355 \363 \372 \361 \321\n



Received: from localhost (localhost.localdomain [127.0.0.1])
by zeus.bancopromerica.com (Postfix) with ESMTP id 75AB93C64E6
for <ltabora@bancopromerica.com>; Fri, 2 Dec 2005 12:55:26 -0600 (CST)
Received: from zeus.bancopromerica.com ([127.0.0.1])
by localhost (zeus.bancopromerica.com [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id 09962-09 for <ltabora@bancopromerica.com>;
Fri, 2 Dec 2005 12:55:25 -0600 (CST)
Received: from SMAM05CFK351K.bancopromerica.com (escm.bancopromerica.com [10.6.0.25])
by zeus.bancopromerica.com (Postfix) with ESMTP id D50063C64E5
for <ltabora@bancopromerica.com>; Fri, 2 Dec 2005 12:55:25 -0600 (CST)
X-eSCM-MailFrom: <ltabora@hotmail.com>
Received: from bay108-f14.bay108.hotmail.com (65.54.162.24)
by SMAM05CFK351K.bancopromerica.com (10.6.0.25);
Fri, 02 December 2005 12:53:40 -0600
Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
Fri, 2 Dec 2005 10:55:01 -0800
Message-ID: <BAY108-F14AD46657E356940FF6691A84C0@phx.gbl>
Received: from 65.54.162.200 by by108fd.bay108.hotmail.msn.com with HTTP;
Fri, 02 Dec 2005 18:55:01 GMT
X-Originating-IP: [63.245.22.74]
X-Originating-Email: [ltabora@hotmail.com]
X-Sender: ltabora@hotmail.com
From: "Lennin Alexander Tabora Portillo" <ltabora@hotmail.com>
To: ltabora@bancopromerica.com
Subject: Prueba acentos � � � � � � �
Date: Fri, 02 Dec 2005 18:55:01 +0000
Mime-Version: 1.0
Content-Type: text/plain; format=flowed
X-OriginalArrivalTime: 02 Dec 2005 18:55:01.0367 (UTC) FILETIME=[E618B070:01C5F771]
X-Virus-Scanned: amavisd-new at
X-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char E1 hex) in message header 'Subject': Subject: Prueba acentos \341 \351 \355 \363 \372 \361 \321\n
X-Spam-Status: No, hits=3.827 tagged_above=-10 required=6.6 autolearn=no
tests=[AWL=-0.597, BAYES_50=0.001, BLANK_LINES_70_80=1.515,
MSGID_FROM_MTA_HEADER=0.054, SUBJ_ILLEGAL_CHARS=2.854]
X-Spam-Level: ***

[dkarp]

Hotmail uses bare high-byte latin-1 characters in its Subject headers, which is a no-no by RFC 2047. The POP3 server is now much warier about header content and returns it verbatim, rather than attempting to parse it.

Fixed in next release, or in the CVS tree right now!