It is almost same and both can be commands, but やめるんだ is a little bit softer than やめろ.
In the first place, we don’t use やめるんだ as "stop it!" in verbal communication.
It is almost only used in Anime, Manga or some fiction. it is also used in literary language.
By the way, やめるんだ has other meanings: it also means "Actually, I decided to stop it.." or "Wow, You decided to stop it..."
"Actually, I decided to leave school... (maybe drop out)"
or "(I am sorry.) You decided to leave school.. (I will miss you.)"
- ピアノやめるんだ・・・。 means
"Actually, I decided to stop playing piano (maybe forever)"
or "You decided to stop playing piano... (Unfortunately...)"
It may be difficult to tell someone a decision or truth, but if you should/need to tell them, you may be able to use やめるんだ・・・.
This word implies your sadness, frustration or bitterness.
For examples, the following case is in the past tense.
やめたんだ（やめた-んだ）has three meanings.
- One is "Actually, I stopped/left/quitted it (though I feel frustration/bitterness) "
- One is "I did stop it! Please do not mention it any more!"
- The other is... If the subject is another person (not you), it is used to sympathize with them.
We may often say "そっか・・・、やめたんだ・・・。", "You mean…, You stop/leave it...". Before that, You might say "I am sorry," though it depends on the situation.
However, Please note んだ is used in a lot of situations.
I think it is very difficult...