I looked up the dictionary, and found "よう" and "わけ" are both grammatical nouns (形式体言). But why is the 接続 different?
Moreover, I found many sentence patterns have their own rules. Are there any universal rules which determine which form you should conjugate into?
This の is a particle joining two nouns and な is the describing form of the copula. As such, it is joining a sentence (ending in a noun or keiyōdōshi) with a noun. So in theory, の takes only the noun, but な the entire sentence.
This difference between わけ and よう can be explained with the different meanings: わけ means something like "conclusion", as such, it only makes sense with a sentence modifying it. 魚のわけ "the conclusion of the fish" makes no sense, or at least not the one we want. It only makes sense with a sentence: それは魚なわけだ "the conclusion that it is a fish" or "that means it's a fish". When one has to translate よう with a noun, maybe "lookalike" would match. 魚のよう "lookalike of a fish" or "like a fish" with just the noun already makes sense. When used in a sentence like それは魚のようだ "that is like a fish" can also be parsed as それは[魚のよう]だ, instead of "(それは魚)のようだ".
But なよう is also possible, just not so frequent.
It's because の emphasizes possession
AがBなわけだ → AがBであるわけだ
AがBのようだ → AがBの様子をしている
But の can also function as the copula, and ～であるよう can work, so both are viable. The meaning is just not as strong or obvious so the usage tends toward this.
