Hmm, this may be a tough answer to argue against the two previous which are answered by native speakers. But here goes...
To me it just seems like a matter of perspective. With
日本人の相手, it's saying "Out of (the group of) Japanese people, the one who is an 相手." With
相手の日本人, it's "Out of all the 相手's, the one who is Japanese". So it seems like a choice of with what you want to associate the person more; are you placing more emphasis on them being an 相手 or a 日本人. However, for reasons I can't explain, it does sound like
日本人の相手 could be representing more than one person, whereas
相手の日本人 sounds like only one person.
I also stumbled upon something similar once. A friend of mine was saying "Enjoy the rest of your summer". She said it as 「残りの夏休みを過ごしてください」and it didn't sit well with me either (and still doesn't) -- I thought it should be
夏休みの残り. But it seems like the same perspective issue. "Of all the things that have a remainder, the one that is summer" vs. "Of all the parts of summer, the remaining part".
I think as English speakers,
日本人の相手 sounds more natural to us because in Japanese 101, we're taught to "adjectivize" nouns this way, like
ドイツのビール, and so on. So we project this same pattern onto people. The confusion is that with the previous examples, it makes no sense to switch the order. Take
日本の車: the two groups are Japan and cars, and there is no larger "semantic group" that contains both of them. The groups are so separate that only "a car from Japan" makes sense; "a Japan from cars" makes no sense. However, in the case of 相手 and 日本人, the two groups are part of the same larger "semantic group": people. So it confuses us about which one should modify the other one. But going back to our knowledge of "adjectivizing" nouns, we tend to put 日本人の as the adjective.