Shogakukan's 大国語辞典 shows that 姪【めい】 has a historical hiragana spelling of めひ, not めい, showing that the modern mei reading is not on'yomi but rather kun'yomi. This different derivation is probably also why the pronunciation is different:
[mei] with a more distinct
[i], and not
[meː]. The term for "nephew", 甥【おい】, has a historical hiragana spelling of をひ. As you note, め is an old morpheme indicating "female", and を is an old morpheme indicating "male". We now have a clear follow-on morpheme ひ.
This ひ is of uncertain derivation. The closest vaguely relevant match that I can currently find is prefix 曾【ひ】, often glossed as "great" in terms like 曾【ひ】お祖母【ばあ】さん "great-grandmother", 曾【ひ】孫【まご】 "great-grandchild", etc. Shogakukan's given definition for 曾【ひ】 is:
For terms expressing a blood relation, this indicates a relation separated by one more generation.
I wonder if there might have been an older meaning, still referring to relative distance from oneself, but not necessarily of a different generation; and an older usage, not as a prefix but as a noun. I'll keep looking.