We have a question on this site, What is the proper differentiation between 来る and 行く?, which does seem to correctly explain the basic general differentiation between 行く and 来る, which is that it is chosen based on the speaker's perspective (regardless of sentence subject, etc.).
However, this is what still confuses me: Physical location of speaker at speech time is not what determines the verb.
If you are at school and you invite your friend over to your house, you say 「うち来る？」, not 「うち行く？」, right? If you live in Japan but you're visiting America and invite your American friend to Japan, you say 「日本来る？」 right? (Although, if you're just asking if they will go to Japan at all, not to visit you or something like that, it's 「日本行くの？」, yeah?)
I can imagine two ways to account for this: 1. it's the speaker's location at the time of movement that's relevant, not their current location, or 2. it's the speaker's "ownership" of the location that's relevant (which I think accounts for the 日本来る・日本行く difference in nuance pretty well).
I think it's two, but physically being somewhere gives you a sense of ownership over the location?
「パーティー来る？」 is okay to say even if it's not your party, but you just happen to already be there, right? 「彼は一週間前ここに来たみたい。」 is fine, even if where you are is totally not yours, yeah?
How about being physically somewhere at movement time? 「俺は１時から５時まで太郎の家にいるから、その間に来るよね？」 seems like it could be okay, but I think 行く is at least also is fine here...
But if course, it's not just physical location at speech time, or physical location at movement time, because 「私がいない間うちに来たのよ…」 works even if you're not at home.
Yet, if you use 一緒に, then you must use 行く 「一緒にうちまで行く？」 I think, even if it's yours...
Is there some simple explanation which accounts for all this (assuming my judgments are correct, and I'm not too sure about them to say the least)?