My problem is, if I don't include 4-octet characters, and only deal with 3-octet ones, I will store each code point in 2 bytes for every character in the text. If I also include 4-octet Kanji, I need to store each code point in 4 bytes (actually they theoretically consume 3 bytes, but it is not practical to store 3 byte variables for optimization concerns); that makes two times more memory consumption.
Normal Kanji ==> UTF-8 octets: 3 bytes ==> UTF-8 code point: up to 0xFFFF (2 bytes) Rare Kanji ==> UTF-8 octets: 4 bytes ==> UTF-8 code point: above 0xFFFF (4 bytes)
I read another question about this. It is said that only some very rarely-used characters in the "CJK Unified Ideographs Extension B" and "CJK Compatibility Ideographs Supplement" blocks are encoded with 3 octets.
What does rarely used mean in this case? How often are they used? Can I completely ignore them? What happens if my algorithm converts these rarely used characters into some kind of "unidentified character" code?
(Please don't move this question to Stack Overflow. I'm more concerned with linguistic usage of these characters. My question is not related to programming.)