Let's talk about い-adjectives:
In pre-kanji Japanese language, adjectives had an い ending. They always end with い、あ、お、or う before the い suffix. They never end with え, as in せ, れ, て, or け. The reason there are many い-adjectives is because they were inherent to Japanese language, even before Japanese people began using Chinese characters in writing. Arguably, there was no written language before the Golden Scrolls came to Japan, though. We have to base our assumptions on word of mouth, decryption of artifacts, and early writing in Japan, so it's difficult to be sure. Anyway, い-adjectives are used for abstract and concrete concepts because the words were originally い-adjectives and not derived from nouns, like の and な adjectives both were.
Let's talk about な-adjectives:
な-adjectives are words that were added to the Japanese language as nouns. な was added to the end of nouns which seemed to have a use as an adjective. Na-adjectives can have あ、い、う、え、or お before the な suffix. People never use な with colors, as in 紫なペン, except when they're using their own vocabulary. For example, if you have a desk full of different kinds of purple pens, you could ask someone if they liked 紫な; however, nobody would say that. People always say こんな紫のペン. The reason you don't see any な-adjective colors is because colors are extremely simple to understand--objective/concrete--unlike い-adjectives which embody both abstract and concrete concepts, and の-adjectives, which are for concrete concepts, only. な-adjectives are used to describe abstract/subjective things, only.
And about の-adjectives:
They're a lot like な-adjectives, but they're for concrete/objective concepts, only. It's possible to think of な- and の-adjectives with colors:
こんな色->colors like this and
この色->this color. I think that some linguists call na-adjectives 'adjectival nouns' because they are "derived from nouns", but な-adjectives are actually derived from の-adjectives, which are derived from "nouns". Perhaps it's confusion with the possessive form:
あなたのペン->your pen? If you say
紫のペン->a purple pen, purple is definitely an adjective in translation. 'Purple's pen' wouldn't make sense, in translation. Many color words are の-adjectives. の-adjectives are thought of in the spirit of the possessive tense, though, in that the adjective's qualities are contained in a specific object or set of objects.
Combining our knowledge of い-, な-, and の-adjectives, we can say: only い- and の-adjectives exist for colors. So, we know that colors are thought of as being concrete concept.
茶色 means tea-colored. Since the birth of kanji, Japanese people have been combining kanji to form meaning. In English, we use a hyphen to indicate an adverb, and an adverb can take an adjective's place, becoming an adverbial adjective. In Japanese, there are so many compound words, and we have suffixes indicating part of speech, so it would be over-productive to hyphenate.
Besides, kanji is more complicated than the English parts of speech accommodate. For example,
馬鹿 means horse-deer (originally an idiom). Because it's an idiom, we have to think of the relationship between the words to understand the meaning. It's from ancient Chinese history. One of the emperors of China never wanted his subjects to undermine him, so he had all of his advisers come to him for a test. The emperor called a horse a deer in front of his advisers. He killed each adviser who corrected him. People thought that this strategy of divining an adviser's competence was stupid, and an idiom was developed in China: 'horse-deer'--"as stupid as calling a horse a deer" or "as stupid as correcting the emperor who calls a horse a deer", perhaps. It's very complicated.
Words like 茶色 (chairo) and 馬鹿な (bakana) adapted into the Japanese language, despite their complexity. Because Japanese language was influenced by the native あいぬ, immigrant Koreans, and immigrant Chinese, there were many options and dialects; however, 茶色 and 馬鹿な are a standard because, for some reason or another, people accepted the dialect.
色 history, from the birth of kanji in Japan:
Why do I say "since the birth of kanji"? Because 色 is one of the kanji included in the Golden Scrolls that were initially sent to Japan, from China. The first record of Chinese characters being used in any significant way, in Japan, was during the time when the Golden Scrolls were sent to Japan. The Golden Scrolls were some of the first documents sent to Japan. It's not the first document, but it's very close to the top of the list. When the Golden Scrolls came to Japan, the Chinese basically gave Japan their first definitive writing system.
I think that い-adjectives and な-adjectives likely existed prior to の-adjectives because I have been told that many times. History teachers and text books have told me that な-adjectives were used in conjunction with nouns to form adjectives, and with foreign words that were abstract and fit better as adjectives than as nouns. の-adjectives were a later distinction.... If you actually look at Japanese writing, though, they pretty much all come at the same time (right when the initial phonetic alphabet, man'yōgana, came into place). At this point, Japanese people were using all three kinds of adjectives. Perhaps the people who wrote the books did some kind of comparison of man'yōgana to older Japanese texts, in an attempt to get more in touch with their history, and this is why people are taught that adjectives came about in this order?
In terms of history, who can be sure, though? The pronunciation of Japanese words, even beyond 500 AD, isn't clear to anybody. This is because there was no commonly used phonetic alphabet, and because so many dialects were borrowed. There was a lot of immigration to Japan in the early days of Japanese writing. Japan had many local dialects, as well. It wasn't until much later that a unified Japanese language was formed. The creators of today's official Japanese language appreciated language, much as we do today, for an art form. That, combined with the simplicity of seeing 色 and knowing it's a color, is probably why they chose to use words like 茶色 rather than some pre-existing い-adjective. Now, if there ever was an い-adjective for 茶色, it's deprecated.
Anyway, the kanji 色 first came to Japan in 57 AD. 色 was originally used like 表情 with 顔色, and it means: 'the expression on the face'. Other than that, it's used with color. That's the purpose of the kanji..., to identify color, as counters indicate something you're counting. The kanji is supposed to be two lovers sitting next to each other, looking at each other, and reading each others emotions. So, 色 is like: 'you can tell if you look closely'.
Great Seal Scroll characters came to Japan as Han-period Chinese writing (206 BCE–220 CE). It looks different today, but 色 has basically been there since the beginning of written language, and therefore historical records of language, in Japan. You can basically assume that it's always been used with colors. I know for sure that it was used prior to 622 AD in association with colors, in authentic Japanese, because the librarian at our school showed us the kanji being used in a texts. 色 just happened to be used, which was exciting for us. The book explained how Chinese writing was hard to discern from authentic Japanese writing, but this was one of the ways....
The writer of the poem, which we couldn't actually understand, appears to be either Japanese, or some eccentric Chinese person who wasn't using Chinese properly, due to the syntax. There's some hentai kanbun. Maybe it's still a Chinese person, trying to describe something, but it seems like a bunch of Chinese characters that don't belong together. For example, if I were to say something in English like: "It's super-cala-fragil-ist-ic." There are too many word parts, and it looks very odd.