Here is the context including the word which a boy spoke to himself.
おとなしく規則を守っておねんねしなさい...か
じょうだんいうないっ いまのおれはめしを食うひまさえおしいくらいに思ってるんだ
I think it should be 言わない, so 「じょうだんいわない」 would be translated as 'this is not a funny story'.
Is it like that because of a dialectal, colloquial or some other reason?
For the word 「おねんねしなさい」, why does he use the honorific prefix 「お」 even though he's speaking to himself?
「いうないっ」 is a form of negative imperative, the dictionary form of which would be 「いうな」. It sounds masculine and very informal.
You could call 「いうないっ」 dialectal because it is not used all over Japan. You will hear it around Tokyo for sure, but not really in Western Japan to my knowledge. We certainly do not say it around Nagoya, which is only about 300 kilometers west of Tokyo.
「じょうだんいうないっ」 = "Don't kid me!"
"For the word 「おねんねしなさい」, why he is use the honorific prefix 「お」when he spoke to himself?"
He is not saying it to himself. Someone said it to him and he is just quoting it. That other person said to him 「おとなしく[規則]{きそく}を[守]{まも}っておねんねしなさい。」.
「（お）ねんね」 is baby talk for "sleeping" or "going to bed". I am sure that you can almost see the word 「[寝]{ね}る」 in it. Many words in baby talk just naturally come with the polite 「お/ご」 in front.
「おねんねしなさい」 = "Please go to bed." or "Please get some sleep."
EDIT: Just found an entry for the sentence-ending particle 「い」 in 大辞林
（ 終助 ）
〔終助詞「や」の転。近世以降の語。主として男性に用いられる〕 文の末尾に接続する。
①念を押したり，語気を強めたりする気持ちを添える。多く「だい」「わい」「ない」の形をとる。 「これ，ぼくのだ－」 「いっぱい食わされたわ－」 「むちゃをするな－」 「早くしろ－」
Summarized TL:
Mainly used by males. Used in reminders and emphatic speech. Often takes the forms 「だい」「わい」「ない」.
(I am surer than ever that this is definitely a Kanto, if not just a Tokyo, phenomenon.)
