5

I am a programmer and I recently wrote two browser extensions, one will translate English text to "pronunciation" replacing all words founf on a web page, and another will replace all Chinese characters with their Pinyin-with-tone-marks counterparts. Now I am thinking if something similar is possible to Japanese? If yes, what would be the approach? Anything can be done algoritmically or you can only use dictionary? Or even dictionary is not keeping in mind that the same characters can sound differently based on context? In other words can you do this conversion without actually attempting to do a machine semantic translation?

  • 1
    One approach is just to rely on the previous efforts by experts. Google Translate can do this. But there seems to be no public API for "romanization" for now. – naruto Feb 14 '18 at 6:34
  • 2
    You are looking for a "morphological analyzer", like mecab/unidic. Be aware that readings are stochastical and will have some errors. – melboiko Feb 15 '18 at 10:15
14

Basically this is very difficult.

Real Japanese sentences on the net are mixture of kanji, hiragana, katakana and English alphabet. See Japanese writing system on Wikipedia.

Among these, hiragana and katakana are almost "pronunciation symbols" themselves. You can replace them into romaji using this table and you're 80% done. The remaining 20% is a bit tricky but they can be handled algorithmically. Still, there are various romanization systems, so you have to make a wise decision. There are also some "extended katakana" which may not be transliterated straightforwardly.

Kanji is the difficult part. Character-based replacement makes no sense because one kanji can be read differently in different words, and there are many jukujikun's. So you absolutely need a dictionary of some sort, but even with a dictionary, they are difficult for some reasons.

  1. Japanese sentences are written without any spaces, so you cannot determine word boundaries with simple regular expressions. You need a dedicated morphological analyzer for this purpose, for example this and this (I have not tested them). Note that analyzers are not perfect.
  2. Sometimes the exact same word or phrase can be read differently depending on the context, although English has a similar problem, e.g., "minute", "read", "wind". See: Difference between こんにち and きょう
  3. Some uncommon words (especially proper nouns) are not on any dictionaries, but you still have to make a "reasonable guess" on them. Of course English has the same problem in this regard, but the algorithm for doing this in Japanese might be more complicated.
  • 2
    I think the biggest hurdle might be names with irregular readings. It would be tough to distinguish, for example 九十九 as つくも from 九十九 as きゅうじゅうきゅう if there isn't a title suffix (the distinction could be made if it was 九十九さん versus just 九十九 as in 九十九年間 or something because of the presence of さん) – psosuna Feb 15 '18 at 20:04
3

Naruto has addressed why this is difficult in great detail, so I am not going to talk about that. Attempting to build your own solution from scratch is probably not the right way.

However, Yahoo Japan has an API for producing the readings of words, which can be seen here (though the documentation is entirely in Japanese). Once you have the kana for a word, moving to romaji should be trivial, as all you are doing is mapping phonetic characters.

  • 2
    Sort of. It depends on whether you're transliterating the Japanese writing system or transcribing the Japanese language; "romanization" is a catch-all term which can refer to either of these. Kana do a great job of representing Japanese pronunciation, but they're not 100%. For example, 問う and are both とう in kana, but in transcriptions of Japanese they would usually be distinguished (e.g. as tou and ) since they aren't pronounced the same way. – snailboat Feb 15 '18 at 1:16
  • Very fair point. I admit I assumed transliteration; comprehensive phonetic transcription would probably require a dictionary. – Mindful Feb 15 '18 at 1:57
1

If you insist on creating something on your own, you might try doing a parser by character that considers the "next" few characters in the char array, and attempts to match the given character and its "nextChars" to words in a dictionary.

In light pseudocode fashion:

array kanjiArray = readChars();
if (kanjiArray.hasNext()){
  // [i+1]
  if (kanjiArray.hasNext().hasNext()){
    // [i+2]
    string dictSearch = kanjiArray[i]+kanjiArray[i+1]+kanjiArray[i+2];
  } else {

  }
}

if (dictSearch in dictionary){
  string translit = getTranslit();
} else { 

  // shave off chars from the end here?

  if (dictSearch == ""){
    // this shouldn't happen because you should always be able to return at least 1 romanization per char, even if the char is already romanized, so return error if you get here
  } else if (dictSearch in dictionary){
    string translit = getTranslit();
  } else {
    // recursion? until you have something readable
  }

}

// shave the defined chars from the start and continue along the sentence/text

What this should do is hopefully this, for example:

漢字すら読めない

  1. Reads the whole sentence as 漢字すら読めない
  2. Attempts to recognize the whole phrase as a dictionary term
  3. Fails, shaves off the い
  4. Continues to fail and shave from the end until it matches the word 漢字
  5. Transliterates 漢字 from dictionary into Kanji
  6. Shaves off 漢字 from start and repeats process with すら読めない
  7. (optional?) Reads すら as hiragana and transliterates directly
  8. Shaves off すら from start and repeats process with 読めない
  9. Recognizes 読めない as a word from dictionary and transliterates as yomenai

Provided that this is pseudocode, the logic is not perfect. There's of course more things to consider, but hopefully this is a starting mental template upon which you'd build. ;)

0

When considering where you might find such functionality (full text to hiragana with disambiguation) it's worthwhile considering where the demand is. Simply rendering hiragana for display is probably not that much in demand, except possibly for automatic addition of hurigana above text. Text to speech functionality however is very much in demand - a quick search turned up this site: http://voicetext.jp which appears to be run by Sharp. You can enter text with kanji and even romaji and get disambiguated voice. It would seems that arbitrary text to hiragana text functionality would be a subset of the total text to voice functionality. But it doesn't seem sufficiently in demand to be listed on that home page as one of the functions they supply.

They do offer an API for their text to voice.

The voicing is really good.

Maybe they be convinced to offer a arbitrary-text to hiragana API, and/or an arbitrary text to original-text-plus-hurigana API. Or at least offer some advice.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.