I heard from my Japanese friends that I should avoid using the word 「来日」 when I'm outside of Japan.
Is it true, and, if it is, has this something to do with the kanji 「来」?
Yes, it is true. 「来」 means to come.
If you are outside of Japan, you can use 「[訪日]{ほうにち}」 instead.
「訪」 means to visit.
The word 「来日{らいにち}」 means visiting Japan. The kanji 「日」 in that word is an abbreviation of 「日本{にほん}」 (Japan). The kanji 「来」 can be used to express coming somewhere, but I don't think there is a generally understood abbreviation method for every country.
One way to express arriving in a country:
[country]に到着{とうちゃく}する (ni tōchaku suru)