に, like most Japanese particles, can have a number of possible uses.
In a passive sentence, に can mark the actor (similar to "by" in English), but in a passive sentence, に can also be used in any of the same ways it can be used in other sentences, too, so just because it can be used that way doesn't mean it necessarily is.
In this case, as you noted, interpreting it as indicating the person/thing doing the action does not make a lot of sense, so that's probably not the right meaning. What is the right meaning? Well, 乱暴 can be an adjective ("violent"), に is commonly used to form adverbs from adjectives (adjective + に --> adverb), so without other context this seems like the straightforward interpretation:
乱暴 ("violent") + に (adverb) --> 乱暴に ("violently")
乱暴に ("violently") + されている ("is being done") --> 乱暴にされている ("is being done violently")