I'm not sure if there's a real answer to that. At least not something that will help you learn which is which. Some 形容動詞 take な, some take の, and some take both. How did that happen? That's quite simple.
All 形容動詞 are in fact a special class of nouns. In academic English material, they are often called "adjectival nouns" or even "descriptive nouns", to emphasize the fact that they're nouns. So how do you use a noun to describe another noun? Since Classical Japanese we had two main methods of doing that:
- Using the the copula (ADJであるNOUN)
- Using the genitive relation particle (ADJのNOUN)
In Classical Japanese, the copula was なり, and for genitive relation we also had が besides の, but it probably worked more or less the same. In the modern language, the copula なり went out of use and was replaced by だ in all forms except for the positive present form, where な (which comes from なり) remains in use.
The inconsistency is mainly there, in the positive present form, since in other forms (past or negative), the copula is used always. That's because you just cannot use の in other forms, since it's not copula and therefore does not conjugate, so you have to replace it with forms of the modern copula だ. な, on the other hand, can be said to be a copula, but it conjugates like だ in all other forms, so in these forms all adjectival nouns behave the same.
But still, why do is the choice of の, な or both in the present-positive form is so inconsistent? That's just how languages tend to work, chaotically. When you have two possible ways of forming an adjective (with a copula and with a genitive particle), people use both, and both forms come into what linguists call 'a competition'. There are several possible resolutions to a competition, such as one of the forms dying in favor of the other, or each form grabbing a different meaning. In this case, we have a complementary distribution, where some adjectival nouns settled take な, some settled for の, and for a good measure of irregularity some settled for both.