Re: organic evolution, I note that Shogakukan's comprehensive JA-JA Kokugo Dai Jiten Dictionary lists most (maybe all?) of these verbs as having originally been both transitive and intransitive. I suspect the different conjugations (ichidan vs. nidan) evolved as a way of being more specific.
By way of comparison, there are verbs in English that are both transitive or intransitive depending on context: consider begin or hang or stop. English and other Germanic languages also have vaguely similar pairs that differ by the core vowel, which changes depending on whether it's transitive or intransitive: consider lie, lay or sit, set. English grammar has certain requirements of subject-object word order and the explicit presence of subject and object that clarify the situation even for the verbs that don't have a vowel change to indicate transitivity (the test began, I began the test, etc.), whereas Japanese does not, making transitivity potentially very ambiguous without some other means of specifying a difference in sense. The ichidan to nidan shift may have evolved as such a way of being specific, perhaps in a way similar to the vowel shift in certain verb pairs in Germanic languages.
Given the scattershot nature of the original verbs (where two-mora verb forms could be either transitive or intransitive, depending on the individual verbs themselves, such as transitive 焼{や}く vs. intransitive 開{あ}く as in your example pair), and given also the presence of verbs like つく that still exhibit ambiguity (modern つく could still be either transitive or intransitive, as in 突く to poke something, to stab something vs. 着く to arrive), learning whether -u or -eru is transitive or intransitive appears to be one of those areas of the language where there isn't any systemic set of rules, and instead you'll just have to learn the verbs over time.
