Am I correct assuming that both おお and おう are homophones in Japanese - both being pronounced as long o? I suspect I am, since that's what I'm indirectly reading in various tutorials. If so, are there any (non necessarily exhaustive) rules to guess the correct spelling of the words with long o? I speak several languages where there are no strict spelling rules, but still there are some hints that help guess the right spelling most of the time. Is there such a rule for おお vs おう in Japanese?
Etymology. おお comes from an earlier おほ or おを, while おう can come from any of おう、あう、おふ、or あふ (and potentially えう、えふ、ゑう、ゑふ if it's now よう). This is due to sound change - originally all of these were distinct pronunciations, but they have since been reduced to a single sound ([o:]).
Typically you can guess that [o:] in Chinese loanwords will be spelled with おう (since neither おほ nor おを ever occurred in any Chinese loanwords), and that in native Japanese words it will be spelled with おお (since the combinations that led to おう were fairly rare compared to おほ and おを, though they do occur).
Both おお and おう spell the same sound in modern Japanese, namely the long vowel /oː/.
The choice of spelling is etymological, but there is one rule of thumb: /oː/ spelled as おお can only occur inside a native Japanese word or be split between two morphemes. Unfortunately, the converse is not true: /oː/ inside a native Japanese word can be spelled おう, e.g. 扇【おうぎ】, and it is possible for /oː/ split between two morphemes to be spelled おう as well, e.g. 小道【こうじ】. It should go without saying that two /o/ in a row, i.e. /o.o/, is always spelled おお.
In terms of history, modern おお can only come from old おお, おほ, or おを; all others (e.g. あう, あふ, おふ) became おう. There are some irregular derivations as well, e.g. 妹【いもうと】 ← いもひと, 小道【こうじ】 ← こみち, 申【もう】す ← まをす, and even こうぶり ← かがふり!
It is worth noting that Christian sources show that Late Middle Japanese had a phonemic distinction between おお・おほ・おを /owo/ and おう・おふ /oː/ (including えう・えふ /joː/) and あう・あふ /ɔː/ (including やう・やふ /jɔː/); for instance, 十【とを】 is recorded as touo. All three have merged in modern Japanese.
It reflects old kana usage. Basically, where there's おお, the second お used to be either ほ or を。 There's some discussion here, with the example of 通り (used to be とほり、now とおり) vs 党利 (used to be たうり, now とうり).
Reading that site also suggests that they're not strict homophones, but the modern pronunciation is close enough that writing そのとうり instead of そのとおり is a common error (it seems to be the classic example used whenever I've seen this issue discussed).
Basically it is something to learn, be aware of and not dwell on too much.
The pronunciation can be different. I have one example from my notes:
You might be able to look these up on a modern work tank, the NHK dictionary for pronunciation or, the only place I have found this tested (using the above expressions), Nihongo Somatome N1 Choukai, p13, practice question 2
Unfortunately, there is no particular rule. The only way to know it is to learn it...
Regarding pronounciation, I would say that they are slightly different, even if the difference is barely audible... I would not say they are strict homophones...