The character enters into a room without permission and woman tells him "You shouldn't touch without permission" and "the owner of that room treasures those stuff". It's not her room so she's simply admonishing him.
To which he apologizes and the woman tells him
私に謝られても
I'm a bit stumbled here by the use of the passive... gooling around it seems akin to a way of say? Why is the passive being used? From the dialog I can gather it's something like "NO point apologizing to me"?