I think a lot of the political debate stems greatly from how one recognizes a person, in which 外人 and 外国人 have subtly different implications.
'outside' + 'person'
stranger, foreigner, outsider
In the context of culture, someone who is not Japanese and is among Japanese people, or in Japanese land, is a 'stranger,' a 'foreigner,' or an 'outsider.' That this word can be used as a pejorative though not always carries the connotation that the 'outside person' may not be welcome, may be rude, may be disrespectful, as a virtue of being from outside.
'outside' + 'country' + 'person'
someone from another country
This variation can be used to describe the same person, in the context of prominence. This person does not have to be a cultural outsider, but is from another country. This word is usually not used as a pejorative, although it can be used to mark a possibility for lack of understanding, though most often, in a negative comment, 外人 is more commonly heard, thus lending its negative connotation.
The end result is that there's a fine line that separates both words, because both have similar, but not exactly the same, meanings. I would say that 外人 does not always but can border on xenophobia and come up in xenophobic remarks, whereas 外国人 doesn't necessarily carry the same connotation.
This is all, of course, up for debate, and up to interpretation. I generally go by the rule that tone and context provide the true meaning of the word when it is said, so if someone refers to you as a 外人 in a non-negative way, I doubt that they are being xenophobic towards you, for example.