「別に」is being used here with「無関係な」to indicate that something is "not particularly related".
While you are correct that「別に」is often used in this way with the negative form of verbs and adjectives, here the 形容動詞 「無関係（な）」already has a negative meaning. In fact, if we were to further negate 「別に無関係」 to 「別に無関係じゃない」it would change meaning from 「not particularly related」to something like「not necessarily unrelated」.
For me, the subject of my older sister isn't really one I want touched upon. It's not particularly because I get worked up about it or anything, like that it's painful for me or something to remember her death, it's because it brings back memories I don't really need brought back and gets me a little depressed.