both the speaker and XXX are confident mysterious types, at the moment the reader isn't that informed about either character
XXX provokes MC to do something shocking,
At this point who is 自分? is it obvious? At this point i thought it was referring to XXX noticing that she lost her composure for a moment. (this was indeed the first time the reader has seen this happen).
reading a bit further I lean toward the MC constantly trying to put on a "『平時』" face to live out a normal life. And not putting on that sort of face made XXX scared.
in general, I feel that i have to work back to confirm if 自分 (and 自身/自ら as well) in a prior sentence was referring to who i initially thought it was more often that it should happen.
Is there a general rule of thumb to determine who 自分 is referring to without purely relying on context?