I've searched them online, but still couldn't distinguish these two. I know: ①The former is to remark or to express a strong feeling. ②The later is more about personal emotion, obligation and social common sense.
There are two sentences below. ③理由はどうであれ、飲酒運転でまったのだから、罰金を払わ「ないではすまない」。⭕ ④理由はどうであれ、飲酒運転でまったのだから、罰金を払わ「ないではおかない」。 ❌ I know why the ③ is correct, but I don't understand why the ④ is wrong. Why can't I view the ④ as a kind of remark? So it will be correct.
I think the key difference is one is passive while the other is active. おかない implies the speaker's active choice (罰金を払わないでおく、無視しておけない）. すまない implies the lack of it.
事故にならないで済む <- Natural
事故にならないでおく <- Grammatical but strange
事故にしないでおく <- Only natural if it means e.g. "don't process as 事故"
止まらないで済む <- I didn't have to stop
止まらないでおく <- I chose not to stop
Coming back to your examples:
The above means roughly "the situation won't come to a conclusion if the fine is not paid" or "the issue doesn't go away while the fine is not paid".
The above sounds odd because it implies you are very eager to pay the fine because you were caught. The example below, on the other hand, sounds natural:
Now let's take 卑劣な手段で報いないでは済まさない. The basic meaning of 済まない is still "the situation won't pass". However, it's changed to 済まさせない. This is an application of the causative verb させる as in 食べさせる、開けさせる. 済まさせる implies somebody is acting to make the situation go away, or is at least not acting hoping that the situation will go away. 済まさない means the speaker will not allow this.
So in this case, the sentence means "They will not allow the situation to pass, without exerting revenge through cowardry means".