Take the 2-minute tour ×
Japanese Language Stack Exchange is a question and answer site for students, teachers, and linguists wanting to discuss the finer points of the Japanese language. It's 100% free, no registration required.

Unlike some languages (English, French ...) , written Japanese sentences don't have spaces between words.

I know that it is the same in Chinese for example, but the fact that the Chinese language only use one alphabet (hanzi) makes it easier to split words, most of the time it is 1 or 2 characters for each word.

I also know that a lot of it comes from knowing vocabulary/grammar to be able to know where each word begins and ends, but I was wondering if there were any tips that can be used to be able to differentiate them ?

Thanks for the answers :)

share|improve this question
4  
I think it's easier to delineate words with 3 scripts than 1. –  Flaw Apr 11 '13 at 9:27
    
This raises the question: are there any good examples (preferably in modern Japanese) of sentences with ambiguous word boundaries? I.e. sentences that can be read in two different ways, depending on how you separate the words. –  Pitarou Apr 12 '13 at 10:35
1  
@Pitarou "私はあくまで執事ですから。" Sebastian's catch phrase from Black Butler is a good example. –  Ataraxia Apr 13 '13 at 4:18
    
Not an answer, but an advice: there is a great browser plugin, rikaichan (for Firefox) / rikaikun (for Chrome), that shows vocabulary entries as you just hover your mouse along the text. It allows to check possible word boundaries really fast, even if different segmentations are possible. –  firtree May 9 '13 at 17:46
    
I've heard of this one and used it for a long time, I'm now using Perapera, it does the same things but I find it better : anyway, thanks for your advice :) –  tama May 10 '13 at 0:56
add comment

3 Answers

up vote 7 down vote accepted

Certainly vocabulary helps, but you can get quite far by considering the structure of a sentence.

  • Nouns are usually written in kanji (or katakana) and are practically always followed by a particle (を, が, は, から, etc.) (unless they are followed by a copula で, だ, です, etc.)
  • The stem of verbs (including adjectival verbs, or "i-adjectives") is usually written in 漢字, with the ending (the conjugated part) written in kana (called 送り仮名).
  • Adverbs are often written in kana
  • Adjectival nouns (AN) ("na-adjectives") are usually written in kanji and usually followed by な.

With that information, you can usually parse a sentence without knowing any vocabulary (but you need to know a list of particles and knowing the possible conjugation for verbs also helps).

For example,

住宅地域における本機の使用は有害な電波妨害を引き起こすことがあり、その場合ユーザーは自己負担で電波妨害の問題を解決しなければなりません。

becomes

(N) 住宅地域
(P) に
(V) おける
(N) 本機
(P) の
(N) 使用
(P) は
(AN) 有害な
(N) 電波妨害
(P) を
(V) 引き起こす
(N) こと
(P) が
(V) あり、
(D) その
(N) 場合
(N) ユーザー
(P) は
(N) 自己負担
(P) で
(N) 電波妨害
(P) の
(N) 問題
(P) を
(N) 解決
(V) しなければなりません。

where for the bold lines, I needed to know that

  • こと is a nominalizer, i.e. some sort of auxiliary noun, and thus is not written in 漢字
  • その is a determiner and not an adverb
  • 場合 is also an auxiliary noun and doesn't take a particle here.

But as you can see, you get quite far with the handful of rules above.

share|improve this answer
    
Great answer, and thanks for the example, it really helps :) –  tama Apr 11 '13 at 6:46
1  
"Nouns are usually written in kanji (or katakana) and are always followed by a particle (を, が, は, から, etc.)" Except when those particles are dropped in colloquial writing, of course :-). –  Darius Jahandarie Apr 11 '13 at 21:48
1  
Nouns can also be followed by a copula, of course. –  snailboat Apr 22 '13 at 11:55
    
Now is 住宅地域 one word or two? –  ssb Apr 22 '13 at 23:34
1  
@ssb I guess you can argue about words, but it certainly is a noun phrase, or, even stronger, a compound noun. After following the simple steps above, there is still more work you have to do before you can look up everything you need in a dictionary. –  Earthliŋ Apr 23 '13 at 4:18
add comment

Great question! This is something that I have struggled with (especially early on... when trying to read any large section of text...) but believe me, it does get easier with time and practice.

It may already be something you are using, but there is at least one thing that has helped me, and that is to look for particles in a given sentence. This doesn't always help... but it can provide boundaries for where many words (or phrases) begin and end.

Other than that, with time and practice it will become more evident where words begin and end as your vocabulary is further enriched!

share|improve this answer
1  
Thanks for the answer, I knew that the particles could help, except when they are sometimes part of the word before, but you also mentioned that ;) –  tama Apr 11 '13 at 6:47
add comment

Separating words in a sentence, at least when done by computer, is called segementation (分かち書き) or tokenization (トーケン化). When using an IME to input Japanese, when you hit the spacebar to convert the kana to kanji, the IME has to segment whatever it is that you typed, then use a dictionary to replace the kana with it's kanji. As you've probably learned by now, the IME is not 100% accurate. As for general tokenization rules, it depends on what you're tokenizing and what you define as a word or token. For example, Kuromoji, an opensource morphological parser, will segment user1205935's sentence above this way:

住宅 | 地域 | における | 本 | 機 | の | 使用 | は | 有害 | な | 電波 | 妨害 | を | 引き起こす | こと | が | あり | 、 | その | 場合 | ユーザー | は | 自己 | 負担 | で | 電波 | 妨害 | の | 問題 | を | 解決 |し | なけれ | ば |なり | ませ | ん | 。
Even though user1205935 parsed the last word しなければなりません as a single token, Kuromoji parsed it as 6 tokens. How you parse depends on what information you want to extract. Kuromoji uses a dictionary to aid the parsing, but another parser, TinySegmenter, is pattern based and also does a very good job:
住宅 | 地域 | に | おける | 本機 | の | 使用 | は | 有害 | な | 電波 | 妨害 | を | 引き起こす | こと | が | あり | 、 | その | 場合 | ユーザー | は | 自己 | 負担 | で | 電波 | 妨害 | の | 問題 | を | 解決 | し | なけれ | ば | なり | ませ | ん | 。
As you can see, there are a few differences between Kuromoji and TinySegmenter (both of these you can use right in the browser). Although no person can tokenize the sentence in the manner these small programs do, it should come naturally and unconsciously as you learn Japanese. If short on time however, a very crude way to tokenize is to just group characters contiguously by script (hiragana, katakana, kanji):
住宅地域 | における | 本機 | の | 使用 | は | 有害 | な | 電波妨害 | を | 引 | き | 起 | こすことがあり | 、 | その | 場合 | ユーザー | は | 自己負担 | で | 電波妨害 | の | 問題 | を | 解決 | しなければなりません | 。 | 
The way user1205935 parsed the sentence above was in fact still vocabulary intensive. For example, in order to segment 引き起こすことが into 引き起こす | こと | が (and not, say, 引き起 | こすこ | とが) he probably first started with the fact that こと is a nominalizer and so is a token. Next, delimiting the token こと then yields all 3 tokens 引き起こす | こと | が. If he didn't know beforehand that こと is a nominalizer he would not have known that it was a token and so he would not have been able to choose between the parsings 引き起こす | こと | が (correct) and 引き起 | こすこ | とが (incorrect). Well, I'm just guessing. Obviously I don't actually know what went on inside the head of user1205935, but I wanted to draw attention to the fact that when it comes to segmenting, having a vocabulary (or a dictionary if you're a computer) results in different segmentation strategies.

share|improve this answer
    
Thanks for the answer :) I happen to do coding too, so I can understand "tokenizer" notion, and that was an interesting reading, with good links too ! EDIT : and yes, of course vocabulary plays a major role in understanding a sentence, nothing new here :) –  tama May 7 '13 at 3:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.