I've heard this used (also as 寝ぼけんじゃねぇよ！) in informal situations with nothing but smiles all around. But when I tried to use it in an informal situation with a colleague, I got the distinct feeling I'd just insulted him. I know it's tough (for me, at least) always to accurately gauge just how "friendly/informal" to be, and I try to err on the side of politeness always, but this one just stumped me. How friendly do you have to be with someone before you can use this expression?
Trust yourself. If you have to apologize for being too casual, that's fine. That's just how you learn what's appropriate. Everyone here is just guessing at what they would do, etc. You're the only one who was there and really felt the atmosphere.
Also, it may be hard to tell if he was insulted or if he was just surprised that you would use that phrase. Maybe he's insulted that you're getting better and more confident in Japanese and don't fit into his impression of you anymore. (totally speculating)
This phrase is definitely too informal for using with a colleague at work, for three reasons:
It makes the assumption that the listener's mind is fuzzy from drowsiness, which (unless this detail is offered by the listener) is kind of a rude thing to assume.
It uses a strong negative command form (～んじゃない), further emphasized by the sentence-ending よ.
The ない is pronounced as ねぇ, which drops the formality level another notch.
You should reserve constructions like this for situations where your position relative to the listener justifies your use of this form (i.e. when you are well above the listener), or among very good friends who can take a verbal jab in good humor (since this sentence has a sharp bite to it).
the 「じゃねぇよ」 part is very very impolite as it is like yelling at them for even suggesting that you were 寝ぼけ in the first place. It's how you would respond if you were falsely accused of something. 「じゃないよ」 is much more more appropriate