Pegasus Mail & Mercury

I find when I receive messages from a Korean correspondent I encounter problems using Pegaus 4.41:

1. Sometimes the sender and subject fields are not displayed properly in the folder view. Sometimes it is blank while other times a series of black squares is displayed. Sometimes I will get black squares in the new mail folder and the proper text will get displayed after I move it into a another folder. But sometimes I can't see the text there either.

2. When I hit reply and Pegasus copies the original text into the reply quoted with a "> " either there is nothing copied or I get a bunch of Chinese looking characters (not Korean) copied into the message. I have to manually copy the text from the original message and use the Paste Special function to get it copied into my reply as quoted text.

The messages are in English, but this seems to be a common problem that I have when corresponding  with Koreans. One of my regular correspondents uses webmail from empal.com but I also have these issues with other Korean correspondents. 

I am wondering if this is a bug or perhaps a configuration issue. I don't have this problem with Chinese correspondents.

Here is a partial header from a recent email with "x" replacing the personally identifiable information:

X-Auth-No:
Return-Path: <xxxxxxxxxxxx@empal.com>
Received: from tx9.empal.com not authenticated [61.97.70.219]
by xxxxxxxxx.com with NetMail SMTP Agent $Revision: 6820 $ on Novell NetWare;
Wed, 12 Sep 2007 00:39:31 -0400
Received: from external ([10.151.12.133])
by xptx9.empal.com (1.0) id l8C4dIw0076D;
Wed, 12 Sep 2007 13:39:18 +0900
X-SMF-UNIQ9B3F5470AEACB4007E467A39F5241948: l8C4dIw0076D
X-SMF-MAILFROMl8C4dIw0076D: xxxxxxxxxxxx@empal.com
Message-ID: <xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@empal.com>
From: =?ks_c_5601-1987?B?xxxxxxxxxxxxxxxxxxxxxxxxx
To: =?ks_c_5601-1987?B?xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Subject: =?ks_c_5601-1987?B?xxxxxxxxxxxxxxxxxxxxxxxxxxx
Date: Wed, 12 Sep 2007 13:39:18 +0900
MIME-Version: 1.0
Content-Type: text/html; charset="ks_c_5601-1987"
Content-Transfer-Encoding: base64
X-Priority: 3
X-Mailer: empal MIME interface v2.1
X-PMFLAGS: 34099904 0 1 P692C1LR.CNM

Make sure you have Korean fonts installed on the machine - Pegasus Mail actually recognizes the odd Korean character set in the headers (most mail programs do not) and tries to render it accordingly: as a result, if you don't have the necessary fonts installed, you'll get boxes or blanks displayed.

Another thing is to make sure that the Korean character strings are actually valid and map correctly onto the proper unicode characters.

It's very difficult for me to test this type of thing during development, so I'm not going to say hard and fast that there's no problem in the program handling this type of data, but certainly most of the reports I've seen so far suggest that on a properly-configured system, the program should handle Korean characters correctly.

Cheers!

-- David --

Just another point... In the sample you sent, the declaration for the message indicates that the character encoding for the message should be the Korean character set. Now, an error I have seen quite often is for the declaration to say "ks_c_5601-1987" but for the message body actually to be in US-ASCII, which is an error in the sending mailer. Ironically, other mail programs that do not recognize the Korean character set will display the message "correctly", because the defined fallback action for a MIME message where the character set is not recognized is to use US-ASCII. As I remarked above though, Pegasus Mail actually DOES know how to handle this character set, so if the body uses the wrong character mapping, you'll get a garbage display, even though Pegasus Mail is following the rules precisely.

If this is the problem, then it can only be fixed by getting the sending mail program repaired - Pegasus Mail is actually behaving correctly based on the information it is being given. You can check this out by manually editing the .CNM file and removing the "Charset" parameter from the Content-type header: if the message displays correctly after this change, then the sender's mail program is broken.

Cheers!

-- David --

To further support David's response, just check the first line of the message body, using the Raw View you used to supply the headers. If the body contains Western Ascii text it will be viewable. We had a problem recorded on this forum a couple of months ago, where the sender forgot to change their default email charset.  In Pegasus Mail the default charset is selectable in Tools/General Settings/Advanced/Default Mime charset drop down list

 
Martin.

1. I have the Korean character set installed on my machine and the default character set that was configured is ISO-8859-1. I have tried it set to US-ASCII but it didn't seem to change this.

2. Pegasus is actually displaying the text in the original message body properly. But the header From and Subject fields in the folder view are typically blank (sometimes a series of black boxes) although I can also typically see the subject line when the message is opened, although there the Korean name in the From field is not displayed. I can also generally see Korean text when it is embedded in the body of the message interspersed with English.

3. The message body itself looks to the unaided eye to be properly formatted unicode. At least it isn't US-ASCII.

Just to review: the problems are related to displaying the headers in the folder view and that the reply is not formatted properly.

I will send you the raw text and a screen shots in a private mail. It would not surprise me if the Korean mailer has a problem except that I have seen this problem in the past with other Korean emails not from this same person and that the messages do seem to display properly on other peoples machines here using Mozilla and Microsoft clients.

Thanks, 

Ralph

PLease send me a sample message, and I will confirm the character set problem and possible solution

Martin 

Just to confirm I have received your sample message.

Martin 

I loaded your message into MS Word, and it responded that the Dotumche font was needed to be installed. I got that from Microsoft site as file IE-KR.EXE which is part of Far East Support.

I then produced the following line in MS Word.  I trust it means something to you.  Meanwhile I have been unable so far to find a near equivalent that we can use in Pegasus Mail. 

 

 Regards,

        Martin
 

If I use Charset=euc-kr I get a different set of characters. Which one is right?

http://www3.telus.net/public/irelam/KoreanMsg2.jpg

 

Martin 

Thanks for trying guys.

 When I bring up the text I posted to you in Word it looks pretty much just like it looks in Pegasus. The body of the email is displayed properly by Pegagus. It is just the From and Subject fields in the folder view that are not displayed properly, and when Pegasus copies the text into a reply it gets garbled into Chinese looking characgters (not Korean). My computer has the Dotum Che font installed already.

I don't really know where the text above comes from. Neither look right. That text doesn't appear in the message that either Pegasus or Word displays on my Sony Vaio.  I also posted a JPG file of the screen shot that I got when looking at the headers for this message in the folder view in Pegaus that illustrates the problem.

 Ralph

Scenario 1: We have a good deal of correspondence with Korean families. Often times, their messages are not displayed properly in Pegasus Mail but if I edit the [Content-Type:] section of the headers to [charset=UTF-8]; all is well.

Scenario 2: In addition to the Korean messages, I recently received a message from http://www.dhs.gov that had used blat v 2.4 (http://www.blat.net) as the mailer and it had incorrectly encoded the MIME message body section with ISO-8859-1. This caused the character ô to be scattered throughout the message where either a CR or double quotes (") should have been. Again, changing the Content-Type: to charset=UTF-8 in the MIME message body section resolved the problem.

 
My question then - can a global filter setting in Mercury/32 that inserts the header "Content-Type: text/plain; charset=UTF-8" or "Content-Type: charset=UTF-8" be harmful or helpful in the first scenario?

I fully understand the multipart message problem (scenario 2) is not something that can be resolved on my end - not in an automated fashion, anyway. I also unterstand that both scenarios are due to a misconfigured sender (be it client or mass mailer) but I am trying to "fix" the problem so that users aren't assuming that there is some problem on my end.

---

Harvesters come get this address!

mapspots@tallulahfalls.org

For those of you encountering problems with ks_c_5601-xxx charsets I can provide a workaround: I've modified and re-compiled the ICONV library used by Pegasus Mail for converting charsets, if any of you want to give it a try I can make it available as a "community download", please let me know!

---

Michael

--
PGP Key ID (RSA 2048): 0xC45D831B
PGP Plugin for Pegasus Mail: <http://www.pmpgp.de/pmpgp/>
S/MIME Certificate Fingerprint: 26 5c a3 60 02 c6 e3 8a 75 70 d5 6a 67 ff d3 8d b0 b5 5e 5b

Michael, does this library exist?

I have problems with messages from someone who ocasionally uses charset="ks_c_5601-1987" for latin characters. I can read them via other mailers, or after manually replacing the charset info.  Would this new library help?

 

PaulW:

Michael, does this library exist?

It's part of Pegasus Mail 4.5, but you can also get it from the download section. If it doesn't help please forward a problematic message (idw.doc [at] t-online.de; preferably as zipped CNM-file from your new mail folder - assuming you know how to get these; forwarded without modifications might get deleted by my spam filters).

---

Michael

--
PGP Key ID (RSA 2048): 0xC45D831B
PGP Plugin for Pegasus Mail: <http://www.pmpgp.de/pmpgp/>
S/MIME Certificate Fingerprint: 26 5c a3 60 02 c6 e3 8a 75 70 d5 6a 67 ff d3 8d b0 b5 5e 5b

Thanks Michael, it didn't help so message now forwarded.