I'm writing a programming algorithm which converts code points of Kanji characters to their respective UTF-8 octets.

My problem is, if I don't include 4-octet characters, and only deal with 3-octet ones, I will store each code point in 2 bytes for every character in the text. If I also include 4-octet Kanji, I need to store each code point in 4 bytes (actually they theoretically consume 3 bytes, but it is not practical to store 3 byte variables for optimization concerns); that makes two times more memory consumption.

Normal Kanji ==> UTF-8 octets: 3 bytes ==> UTF-8 code point: up to 0xFFFF (2 bytes)
Rare Kanji   ==> UTF-8 octets: 4 bytes ==> UTF-8 code point: above 0xFFFF (4 bytes)

I read another question about this. It is said that only some very rarely-used characters in the "CJK Unified Ideographs Extension B" and "CJK Compatibility Ideographs Supplement" blocks are encoded with 3 octets.

What does rarely used mean in this case? How often are they used? Can I completely ignore them? What happens if my algorithm converts these rarely used characters into some kind of "unidentified character" code?

(Please don't move this question to Stack Overflow. I'm more concerned with linguistic usage of these characters. My question is not related to programming.)

  • 2
    “Can I completely ignore them?” How can anyone answer “yes” to this question without knowing what your program is for? After all, Unicode decided to include those “rare” characters for an obvious reason: they are used. – Tsuyoshi Ito Sep 21 '12 at 12:24

[I doubt that this is on topic, but per your request I will not move it. Others may decide otherwise.]

In theory, the often repeated answer is "not very often". However, in practice, actual usage is surprisingly higher than expected. With the continued adoption and usage of Unicode, more and more characters which were technically difficult or impossible to input are increasingly being used. Just the other day I needed to fix a bug in our software which did not handle this correctly. (The character in question was 𩸽, hokke, U+29E3D.)

Also note that there are some regular JIS (X 0213:2004) characters that are mapped to Unicode characters > U+FFFF. So these characters are expected to be available and will be used.

Personally speaking, I heavily rely on many characters above U+FFFF. At the moment, I am waiting for the next extension set to be passed because the current sets do not include several characters that I need. (Until then, I map them to the PUA, but this is not a long-term solution.)

What happens if my algorithm converts these rarely used characters into some kind of "unidentified character" code?

Then part of your text stream will be unreadable. This will naturally hurt human readability as well.

There is no escape: you must support 4-byte characters in UTF-8. Although I would strongly recommend UTF-16 for the best balance. Most of your text will be a single character in length, and those "few, rare" ones above U+FFFF will consist of two surrogates, which are easy enough to process.

  • 1
    Thank you. You answered my question. You those characters are actually used in Japanese. I thought they might be very old and all forgotten today, but it didn't came out to be so. I also thank you for your suggestion of UTF-16 for Japanese, I will consider it. – hkBattousai Sep 21 '12 at 12:30
  • 3
    If you read the question carefully, this question is definitely not off topic. It's a question on how often the rarer kanji are used (i.e. just how "rare" are they). Just because it pertains to programming doesn't mean it doesn't pertain to Japanese. – Ataraxia Sep 21 '12 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.