This の is not "of" but a subject marker usable in a relative clause. 色の白い女 is the same as 色が白い女, and they literally mean "a woman whose color is white". If you know how to parse 心の美しい人, 背の高い男, 尻尾が長い猫 and so on, it's exactly the same structure. (Please see this if you are unsure.)
So something like 色の青い車 is grammatical, but we usually don't say this because it's obviously redundant and 青い車 is enough. However, 色が白い is an exception. This is almost a set phrase that automatically refers to skin colors without explicitly saying 肌. We also have a no-/na-adjective version of this, 色白. Likewise, 色が黒い and 色黒 mean "dark-skinned" without explicitly saying 肌. 肌の白い人 and 色の白い人 roughly mean the same thing, but the former sounds a little too direct, and the latter is usually preferred.
Also note that 白い女 usually makes little sense even outside the story of 雪女; if you mean "white woman" in the sense of Caucasian, you should say 白人の女 or 白人女性.