X に失敗する means "fail to X" or "fail to achieve success in X", in other words, your objective is met with failure trying to do X. Otherwise, it just means intransitive fail, or "make a failure". で is just there to indicate the circumstance.
I failed the exam, but I was able to pass it. (contradictory)