Imagine if you were an English speaker, asked by a Japanese that: "Should I pronounce 'header' as ヘダー or ヘッダー?"
From a native speaker's standpoint, it is actually a non-problem: ちゃいろ is always in three units (morae) cha-i-ro. The fact is not affected by word composition either, as バイト "part-time job" is always ba-i-to too. (And はい ha-i.)
So what leads to your question is, I think, in fact the mismatch between the Japanese tongue and [your native language]'s ear. In every language, fast (I mean, most of the daily) speech compromises on phonological clearness for articulatory smoothness; which means in this case, the hiatus a-i becomes so seamless that many incidental factors such as relative length of a to i, emphasis, or surrounding intonation etc. could make you sometimes hear the diphthong //ai// and other times not. It is hard to make a generalized advice on this because it also depends on your language's threshold when to hear //ai// and //a.i// as syllables.
Either way, no matter whether you believe that "chai-ro" and "cha-iro" are distinct sounds uttered in separate manners, it is just an irrelevant matter of where you accidentally put more muscle for Japanese speakers. The correct way I think is to train yourself to utter cha / i / ro each in same length by default (i.e. when no stress or intonation is in consideration).