I imagine this is something that Japanese speakers naturally build up as they are exposed and communicate in the language for many years. However, take the following scene:
「港町だけあって魚がやっぱり自慢みたい」- From context this is clearly not referring to the speaker or listener.
「お酒か……普段はあんまり飲まないけど、それならたまには……」 - Speaker
(they all get drunk and have a hang over and get woken up the next morning)
「少し昨日は、飲み過ぎたな……」- Ambiguous? The speaker (at the bare minimum) is saying that they drank too much yesterday, however they could also be saying "We" drank too much. How does one differentiate between these? Would the wording be slightly different or could the same wording be used for I/You/We?
「普段あまり飲まないからなぁ……加減が分からなくなってしまった」I guess that this is referring to the speaker as it is the same person as before who said they don't drink very much