Literally, 冗談じゃない means "it is not a joke" but actually it seems that the meaning of this is "something unthinkable even for a joke"( japanese definition:冗談にもそんなことを言ったりするな。).
「道楽に働いているんだろうだって? 冗談じゃない.」
"I'm working for the fun of it? Nothing of the sort!"
「冗談じゃない、この暑い中に誰かと結婚するなんて！」
"Imagine marrying anybody in this heat!"
In the examples you provided, your interpretation is correct:
but actually it seems that the meaning is "something unthinkable even for a joke"
In these cases, here are some additional expressions to capture the nuance
For this example:
「道楽に働いているんだろうだって? 冗談じゃない.」
I interpret this as "You think I'm working just for fun? Please!" (not as in "pretty please")
For this example:
「冗談じゃない、この暑い中に誰かと結婚するなんて！」
I interpret this as "I'm not even halfway joking, who would seriously get married to anyone in this heat!?"
