Etymologically, various usages of れる/られる derived from one base meaning, "without someone's will". In modern Japanese, れる/られる is still sometimes used in this sense (known as 自発 or "spontaneous"). See: Why is the passive form used in this sentence?
故郷が思い出される。 I (spontaneously) remember my hometown. (I didn't intentionally tried to recall that, but it occurred to me.)
Then the "passive" sense came into use. I think the reason is straightforward; the passive voice is basically used to describe something that happened to you without your will. This also explains why Japanese passive voice often implies you were negatively affected (aka 迷惑の受け身).
雨に降られた。 Rain fell (against my will, and I was bothered).
The "potential" sense of れる/られる originally derived from the negation of 自発. Something like "it never (naturally) happens that ～" or "not in the situation where ～" eventually became "cannot". The non-negative potential sense (i.e., "can") followed.
(archaic Japanese) 弓矢して射られじ。
It never happens that you shot them with an arrow.
→ You cannot shot them with an arrow.
≒ (modern Japanese) 弓矢で射られない。
れる/られる also has an "honorific" meaning. This is also an extension of "not my own will" (i.e., it's the will of "your highness", etc).
(Note that those shifts in meaning actually happened long ago when archaic Japanese was used. る/らる were the precursors of modern れる/られる.)