My understand of the sentence is
SWAMが悪い(...)のに he's essentially saying "Despite the fact the SWAM are the ones that failed"
話をすり替えやがって they had the nerve of changing the subject
I took it "They said they'll raid us over the suspect of us committing "虐待の疑い".
Which put together the jailer is saying "The SWAM group is the one that failed, but they change the topic and said they'll house search us over the fact we may have committed abuse".
What is the actual grammar of 疑いアリだって?