私に言われても is Suffering Passive (迷惑の受身), a kind of Indirect Passive Structure (間接受身構文), and 私が言われても is Direct Passive Structure (直接受身構文).
「(あなたが)私に言う」 (Active/能動) "You tell me."
→ Direct Passive: 「私が(あなたに)言われる」 "I am told (by you)."
→ Indirect Passive: 「(私が)(あなたに)私に*言われる」 "You tell me (and it affects me in some way)." This means "You do the action 私に言う (you tell me) and it affects me or I am suffered/annoyed, etc." *The に marks the indirect object of 言う.
So... yes, that is a correct interpretation, and 「私に言われても(困る)...」 is actually far more natural and more common than 「私が言われても...」 for saying "Even if I'm told / Even if you tell me (I can't do anything)..."
For more about Indirect Passive, you could refer to: