I am learning hiragana and I've encountered a minor problem. The thing is that my book did mention how to deal with a long "O": just by adding another "O" or "U". However it doesn't explain in which cases I should use which.
As you can see I've made a mistake trying to put "O" in 7th case. In which cases of double "O" should I use "O" or "U"?
Really, all I can say is 'it depends on the word'. Generally on'yomi (Chinese-derived) readings use おう, while kun'yomi (native Japanese) readings use おお, but there may be exceptions. A note: if う is a verb ending, おう will not be pronounced おお but as お and う separately, as in 追う and 思う. A lot of what I've said also applies to えい and ええ.
As @Nothing at all notes, this depends on the word.
On'yomi always use -OU for long O sound and kun'yomi almost always -OO.1
However, the real problem here is that you are being asked to reconstruct hiragana from Hepburn romanization. In general this is impossible, because Hepburn romanization conflates certain hiragana spellings. (There are romanization systems that don't.)
The fact that there are many homophones is probably one of the reasons that kanji still exist in Japanese. Without kanji and only hiragana, we would have that 糖衣、東夷、当為、等位、… would all be とうい.
Hepburn romanization goes even further, and also conflates とうい (e.g. 糖衣、東夷、当為、等位、…) and とおい (e.g. 遠い) to tōi.
So, you would need a lot of information to know that there is no word しょおゆ, so that shōyu would have to be しょうゆ. However, there would be no way for you to know (without context) whether tōi should be とうい or とおい.
The only way to solve this type of question is to know the hiragana spelling of the words that come up. (Well, it suffices to know the words in a romanization system that does not conflate hiragana spellings.)
All of that said, a long O (in Hepburn -ō) is more likely to be -OU, so the most economical approach would be to default to -OU for -ō and learn words with -OO such as とおい、とおる、おおい、おおきい、… as exceptions.
1 For kun'yomi there are rare exceptions that arise from a sound shift as in 妹【いもうと】 from いも+ひと which also occurs as a long O sound in 弟【おとうと】, 素人【しろうと】, 客人【まろうど】, 若人【わこうど】, 蔵人【くろうど】. Across "word boundaries" Hepburn romanization also uses -ou-; the above exceptions used to be word boundaries, but now they are not anymore, so they have a long O sound, romanized as -ō- in Hepburn.)
Most of the time it is おう In rare cases, it is おお examples 遠く、通る おお also used for the kanji 大 or 多 , so 大きい、多い
