from the context I believe it roughly translates to:
"Don't let what anyone says bother you!"
I am afraid that that is not what it means. No sure where you get the negative imperative form "Don't let ~~~~"
The original sentence is a regular negative statement, not an imperative.
"I have no intention of letting anyone/people/others say anything!"
It is indeed causative as it talks about not allowing others to say something.
why not use 誰にでも?
Because that just will not fit the context. 「誰にでも」 means "anyone" only in the context of saying "anyone can do ~~", "anyone could understand ~~", etc.
You could, however, use 「誰にも」 without a 「で」 if you changed 「何」 to 「何も」 and 「気も」 to 「気は」. The double-も is quite common and grammatical, but not triple-も.