[Bug 47893] Unicode characters aren't being rendered correctly

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=47893

--- Comment #4 from huupoke12 at gmail.com ---
It seems like the server saves the file as UTF-8, but reads the file as ISO
8859-1 encoded and non-printable characters are replaced with something else
which something else printable or "?".

For example with the first one: Original comment is "日本語". Garbled is
"日本語".
When saving "日本語" in a text file with UTF-8, its hex value is "e697 a5e6 9cac
e8aa 9e0a". But if reading the file as ISO 8859-1, "e6" is "æ". "97" is "END OF
GUARDED AREA" (unprintable)
(https://www.fileformat.info/info/unicode/char/0097/index.htm) and being
replace with "—" (printable). "a5" is "¥"...

By the way, I have tested with changing my name on the AppDB. It's showing fine
on the changing name page (Preferences) even with Japanese. But when see it on
the comment, the characters are replaced with "?". And after more testing, it
seems that only the Basic Latin range (0x0000 => 0x007F) AND the Latin-1
Supplement range (0x0080 => 0x00FF) is being displayed correctly, the other
ranges are replaced with "?". This proves the above, since the ASCII only uses
7-bit and can only present the Basic Latin range (128 characters), but the
ISO8859-1 uses 8-bit (UTF-8 uses 8-bit too) so it can present both the Basic
Latin range AND the Latin-1 Supplement range (256 characters).
Both the UTF-8 and ISO 8859 is backward compatible with ASCII, but they are not
compatible with each other. In UTF-8, the first significant bit is used to mark
if it is not a ASCII character and tell the decoder to continue reading the
other bytes to decode it to a single character. But in ISO 8859, it used that
byte to make room for more character. Thus, every byte is a valid character
when reading as ISO 8859 (0x00 => 0xFF), while UTF-8 is not.

-- 
Do not reply to this email, post in Bugzilla using the
above URL to reply.
You are receiving this mail because:
You are watching all bug changes.