This is a variation on an idiom, あさっての方を向く, which means you're focusing on the wrong thing or failing to notice what you should. The key to the idiom is あさって. Literally, it comes from the idea that you're focusing on the day after tomorrow when you should be paying attention to tomorrow!
The idiom, though, is now more general than that, and it can be used without a verb like 向く. In fact, the only strictly necessary part of the idiom is あさって itself!
Phrases like あさっての方向 mean something like "the wrong way (without realizing)", and that "way" can be either literal or figurative. If it's literal, it could be used in phrases meaning going the wrong way, looking the wrong way, or aiming something the wrong way. If it's figurative, it could mean missing the point, talking about the wrong subject, or just failing to notice something. It can even mean feigning ignorance if you're doing it on purpose―close to the English phrase "looking the other way".
To figure out the exact meaning, you'll have to pay attention to context. In your example, it looks like the basic figurative use:
In this case, I think your explanation was pretty good―they were saying something "off-topic", right? They were focusing on the wrong thing (figuratively on あさって, not on あした, the thing they should have been focusing on). The speaker was pointing that out. Depending on context, a phrase like "That's besides the point" or "You're barking up the wrong tree" could be appropriate as a translation.
For reference, here is the definition from the 明鏡ことわざ成句使い方事典: