If I want to talk about a friend of mine who studies psychology, can I say following?
心理学を専攻している私の友達。
It sounds like I'm the one studying shinrigaku. Then I could say
私の心理学を専攻している友達。
But it sounds that I'm talking about "my psychology", even if I now it doesn't make sense. So how could I translate the following nominal group ?
My friend who studies psychology.
Although technically ambiguous, you can safely say 心理学を専攻している私の友達 without being misunderstood. Introducing a friend of yours after describing your own major doesn't make much sense, so people normally take this as "my friend who studies psychology."
私の心理学を専攻している友達 is acceptable but a little puzzling.
Related: Are Japanese modifiers "greedy", "anti-greedy", or do they mean whatever people choose them to mean?
I think the best way to go about this is to simply exclude 私の from the sentence entirely. In Japanese, personal pronouns just aren't used as often as they are in English and aren't necessary in many(most) cases that they are in English. It's a little bit of a nuance, but simply by talking about a 友達 in this case implies that it is your friend that you're speaking about imo.
(0) My friend who studies psychology.
(1) 心理学を専攻している私の友達。
(2) A frined of mine. I study psychology.
(3) 私の心理学を専攻している友達。
(4) My friend who studies my psychology.
(5) 私の友達で、心理学を専攻しているやつ/人。
(1) is perfectly correct as the interpretation of (0).
As you know, the reason why modifiers exist not only in Japanese but also in English is because it is easier for the modified word to be understood when a modifier exists than when it does not.
(2) tells us two facts: he/she is my friend and I study psychology. Two facts are irrelavant with each other. Therefore, if (0) and (2) exist grammatically as the interpretations of (1), (2) which does not have the significance of existence as a modifier is eliminated and only (0) remains.
Your understanding that (3) or (4) is an incorrect interpretation for (0) is correct.
In order to avoid the confusion you feel, we sometimes say like (5) as the interpretation of (0). I think it's a good solution for this kind of expressions.
