I noticed that in the word 子牛 both "o" and "u" are pronounced while in the word 格子 there is a long "ō" and they are, respectively, written in rōmaji (Hepburn romanization) koushi and kōshi. What I'd like to ask you is if: 1) in Middle Japanese the diphthong "ou" was reduced to "ō" only when the two vowels were part of the same syllable (like in 格子) 2) the reduction happened only in kango and waseigo or it happened also in some yamato kotoba
Is coalescence blocked between syllables?
No, coalescence also occurs between syllables, and even involving vowels from different morphemes. In fact it's not clear whether /ou/ was even a diphthong to being with; it's quite possible that it only ever existed as a hiatus. In modern Tokyo Japanese probably only /ai/, /oi/ and /ui/ are actual diphthongs (every vowel sequence ending in -u needs another syllable for it; they coalesce nonetheless).
Was historical coalescence restricted to kango words?
No, it happened throughout the entire lexicon. Primitive Japanese had no sequential vowels, so yamato-kotoba examples aren't as plentiful as kango; but some vowel sequences did emerge later, and among them, /ou/ did coalesce into /o:/. For example, *saso-pu → saso-u → sasō.
Notice that /au/ also coalesced into /o:/, so that /au/ and /ou/ merged. Diphthongs in /-i/, by contrast, were resistant and survived to this day. So hayau → hayō but hayai → hayai.
Regarding questions raised in the comment thread:
Is coalescence blocked between morphemes? What about words?
- Conclusive (終止形) suffix -u: saso-u /saso:/
- Adjectival suffix -u: samuk-aro-u /samukaro:/
- 御宇 gyo-u /gyo:/
- 如雨露 jo-u-ro /Zo:ro/
- 子牛 ko-usi /kousi/
- 小唄 ko-uta /kouta/
- 壇ノ浦 dan-no-ura /daNnoura/
- 選挙運動 senkyo-undō /seNkjoundo:/
- 左顧右眄 sako-uben /sakouben/
For the purposes of coalescence, compound words (words made of two free words) count as "different words". So a word like usi or uta won't lose its u-, but a bound suffix might.
Notice that this is consistent with other forms of coalescence in Japanese. So au > ō regardless of whether it was found in native Japanese or Sinitic words, and across morpheme boundaries (haya-u → hayō), but not across word boundaries (matu-ga-ura → matu-ga-ura). Even other sound changes behave similarly; 買ひ → ka-wi → ka-i (cross-morpheme lenition) vs. 朝日 → asa-hi (no cross-word lenition).
What about words like 降雨 or 堂宇?
The final -u doesn't coalesce: /ko:u/, /do:u/. However, the previous /o:/ is already long, so they wouldn't coalesce anyway (/o::/ isn't a legal coda).
Does coalescence behavior differ between yamato-kotoba and kango words?
From the above, it seems like not (except indirectly, insofar as diphthongs are rare in yamato-kotoba and free-word compounds are plentiful).
- Kubozono, Diphthongs and vowel coalescence (in: The Handbook of Japanese Phonetics and Phonology)
- Kenkyūsha's Japanese-English dictionary, for pronunciation guides.
1) in Middle Japanese the diphthong "ou" was reduced to "ō" only when the two vowels were part of the same syllable (like in 格子)
I think what you mean is "the same morpheme", because that's the difference tells 子牛 from 格子. This is a part of four sound shifts have taken place in parallel.
- //au// → //ɔː// → //oː//
- //iu// → //yuː//
- //eu// → //yoː//
- //ou// → //oː//
This sound shift has had effect mostly inside a morpheme with a few exceptions, but most of those exceptions are eventually reverted in today's language. Some examples of surviving exceptional outcomes are:
- the volitional suffix: 行かむ //yuka-mu// → 行こう //ikoː//, 出でむ //ide-mu// → //(i)dyoː// (→ leveled to 出よう //deyoː//) etc.
- -うと/-うど pseudo-suffix series: 素人 //sira-bito// → //siraudo// → //siroːto//, 玄人 //kuro-bito// → //kuroudo// → //kuroːto// etc.
- 酔ふ //weɸ-u// → //eu// → //yoː// (→ leveled to 酔う //yo-u//)
- 言ふ //iɸ-u// → //iu// → //yuː// (deemed non-standard now)
In fact, 格子 has never been pronounced like //kousi// despite its modern spelling, because its actual historical development is //kakusi// → //kausi// → //kɔːsi// → //koːsi//. The reason it's spelled こうし is because the Modern Kana Orthography defines that all historical おう, おふ, あう, あふ that pronounced as //oː// should be replaced with おう.
2) the reduction happened only in kango and waseigo or it happened also in some yamato kotoba
This sound change affected all qualified diphthongs regardless of kango or wago (yamato-kotoba), but since such combinations are rare in native Japanese words, the great portion of the effect was seemingly exerted on kango.
- 塔 (kango) //taɸu// → //tau// → //toː//
- 尊き (wago) //taɸuto-ki// → //tauto-i// → 尊い //toːto-i//
But some wago apparently showed some degree of resistance.
- 倒るる //taɸur-uru// → //taur-uru// → 倒れる //taore-ru//
cf. 放る //ɸaɸur-u// → //ɸaur-u// → //hoːr-u//