(0) My friend who studies psychology.
(2) A frined of mine. I study psychology.
(4) My friend who studies my psychology.
(1) is perfectly correct as the interpretation of (0).
As you know, the reason why modifiers exist not only in Japanese but also in English is because it is easier for the modified word to be understood when a modifier exists than when it does not.
(2) tells us two facts: he/she is my friend and I study psychology. Two facts are irrelavant with each other. Therefore, if (0) and (2) exist grammatically as the interpretations of (1), (2) which does not have the significance of existence as a modifier is eliminated and only (0) remains.
Your understanding that (3) or (4) is an incorrect interpretation for (0) is correct.
In order to avoid the confusion you feel, we sometimes say like (5) as the interpretation of (0). I think it's a good solution for this kind of expressions.