Shogakukan's 大国語辞典 shows that 姪【めい】 has a historical hiragana spelling of めひ, not めい, showing that the modern mei reading is not on'yomi but rather kun'yomi. This different derivation is probably also why the pronunciation is different: [mei]
with a more distinct [i]
, and not [meː]
. The term for "nephew", 甥【おい】, has a historical hiragana spelling of をひ. As you note, め is an old morpheme indicating "female", and を is an old morpheme indicating "male". We now have a clear follow-on morpheme ひ.
This ひ is of uncertain derivation. The closest vaguely relevant match that I can currently find is prefix 曾【ひ】, often glossed as "great" in terms like 曾【ひ】お祖母【ばあ】さん "great-grandmother", 曾【ひ】孫【まご】 "great-grandchild", etc. Shogakukan's given definition for 曾【ひ】 is:
血縁関係を表す語について、それよりさらに一代離れた関係にあることを示す。
For terms expressing a blood relation, this indicates a relation separated by one more generation.
I wonder if there might have been an older meaning, still referring to relative distance from oneself, but not necessarily of a different generation; and an older usage, not as a prefix but as a noun. I'll keep looking.