私に言われても is Suffering Passive (迷惑の受身), a kind of Indirect Passive Structure (間接受身構文), and 私が言われても is Direct Passive Structure (直接受身構文).
「（あなたが）私に言う」 (Active/能動) "You tell me."
→ Direct Passive: 「私が（あなたに）言われる」 "I am told (by you)."
→ Indirect Passive: 「（私が）（あなたに）私に*言われる」 "You tell me (and it affects me in some way)." This means "You do the action 私に言う (you tell me) and it affects me or I am suffered/annoyed, etc." *The に marks the indirect object of 言う.
So... yes, that is a correct interpretation, and 「私に言われても（困る）...」 is actually far more natural and more common than 「私が言われても...」 for saying "Even if I'm told / Even if you tell me (I can't do anything)..."
For more about Indirect Passive, you could refer to: