Are the 4-byte UTF-8 Kanji rare enough that I can ignore them?

Question

I'm writing a programming algorithm which converts code points of Kanji characters to their respective UTF-8 octets.

My problem is, if I don't include 4-octet characters, and only deal with 3-octet ones, I will store each code point in 2 bytes for every character in the text. If I also include 4-octet Kanji, I need to store each code point in 4 bytes (actually they theoretically consume 3 bytes, but it is not practical to store 3 byte variables for optimization concerns); that makes two times more memory consumption.

Normal Kanji ==> UTF-8 octets: 3 bytes ==> UTF-8 code point: up to 0xFFFF (2 bytes)
Rare Kanji   ==> UTF-8 octets: 4 bytes ==> UTF-8 code point: above 0xFFFF (4 bytes)

I read another question about this. It is said that only some very rarely-used characters in the "CJK Unified Ideographs Extension B" and "CJK Compatibility Ideographs Supplement" blocks are encoded with 3 octets.

What does rarely used mean in this case? How often are they used? Can I completely ignore them? What happens if my algorithm converts these rarely used characters into some kind of "unidentified character" code?

(Please don't move this question to Stack Overflow. I'm more concerned with linguistic usage of these characters. My question is not related to programming.)

Answer 1

[I doubt that this is on topic, but per your request I will not move it. Others may decide otherwise.]

In theory, the often repeated answer is "not very often". However, in practice, actual usage is surprisingly higher than expected. With the continued adoption and usage of Unicode, more and more characters which were technically difficult or impossible to input are increasingly being used. Just the other day I needed to fix a bug in our software which did not handle this correctly. (The character in question was 𩸽, hokke, U+29E3D.)

Also note that there are some regular JIS (X 0213:2004) characters that are mapped to Unicode characters > U+FFFF. So these characters are expected to be available and will be used.

Personally speaking, I heavily rely on many characters above U+FFFF. At the moment, I am waiting for the next extension set to be passed because the current sets do not include several characters that I need. (Until then, I map them to the PUA, but this is not a long-term solution.)

What happens if my algorithm converts these rarely used characters into some kind of "unidentified character" code?

Then part of your text stream will be unreadable. This will naturally hurt human readability as well.

There is no escape: you must support 4-byte characters in UTF-8. Although I would strongly recommend UTF-16 for the best balance. Most of your text will be a single character in length, and those "few, rare" ones above U+FFFF will consist of two surrogates, which are easy enough to process.