姉 is pronounced ane, but お姉さん is pronounced oneesan. What happened to the initial a?
Likewise, 兄 is pronounced ani, but お兄さん is pronounced oniisan.
What is the rule governing the change in the pronunciation for the same kanji in these compound words? Or are these just exceptions one simply needs to remember?
First of all, let us take a look whether variant pronunciations exist.
There are some instances of 兄様【あにさま】, 兄【あに】さん, 姉様【あねさま】, 姉【あね】さん found in several writings accessible on the free digital library 青空文庫, all from authors around 1900.
おみの （あたりを窺ひて。）子之介【ねのすけ】は厩にゐると御門で教へられたが、はて何處へ行つたことであらう。
（奧より子之介出づ。）
おみの おゝ、弟……。
子之介 姉樣【あねさま】か。（なつかしげに寄る。）ようたづねて來てくだされた。
――岡本綺堂【おかもときどう】(1914) 『佐々木高綱【さゝきたかつな】』
Also, in the movie もののけ姫, Kaya apparently calls Ashitaka, who is not her brother, あにさま, as an expression of her affection or reverence.
In fact, 姉様 is even listed in some dictionaries:
あね‐さま【姉様】
① 姉を敬っていう語。
② 若い女性を親しんでいう語。
③「姉様人形」の略。
――大辞林
The same dictionaries lists 姉【あね】さん and 兄さんas well, including sense ①, ie. （姉・兄） 姉を親しみ敬っていう語.
Note that there are also some compound words where あに and あね do not get shortened:
姉上【あねうえ】, 兄上, 姉貴【あねき】, 兄貴
Today the versions without the initial mora are more common. Considering that the longer versions did exist, it is simply a shortening of the compound words, perhaps because おねえさん sounds smoother or is more easily pronounced, or perhaps to parallel お母【かあ】さん, お父【とう】さん, おば(あ)さん, おじ(い)さん.
There do exists some regular phonetic changes such as あう→おう, but they don't seem to apply here.
To put it in a perspective, shortening words in compounds is not a new phenomenon:
And it's still being done:
