What is the connection between eyes and boundaries (the etymology of those phrases)?
So I couldn't find an authoritative source, but given there haven't been an answer, I'll add my guess.
My guess is that it's related to the use in 木目(wood pattern)、籠目(lattice pattern) etc. The commonality is here that 目 refers to how things look like (to the eye). 分け目 probably refers to the virtual "pattern" that emerges because of the distinction.
Yamato-go had simple words with broad semantics, of which め is one. It is now distinguished in written form as 目 or 芽 (don't know about 女 or 雌), but we should remember that these characters are in many cases a kind of ate-ji hiding the commonality of the underlying original lexical item.
My sense is that the intrinsic meaning of め is something along the lines of "point", or "defining point". The eye may be considered the defining point of a human. A bud can be considered the defining point of a plant. The め in 5つ目 (which can also be written 5つめ) also has this delineative meaning ("the point of the fifth one", "the one defined by being at the fifth point").
In this perspective, the use of め in 境目 or 分かれ目 is quite natural: it refers to the point or threshold defining the boundary between two areas, or the point defining where two roads diverge.
We should also remember the expression ひどい目にあう. What is the 目 (interestingly, also sometimes written め) here? Well, it's just a point--in other words, "encountering a horrible point (or situation)."
Finally, we have the め used with adjectives such as 軽め. Again, "lightness to a certain point", "a point of [some] lightness".
It is also possible that め had three distinct meanings: (1) eye (2) bud and (3) point, and that the character 目 used for (1) ended up by analogy sometimes being used for (3) as well.
The above has no scholarly background whatsoever, just my thoughts.