As I have learned, the に particle is marking the target of the verb. And it make sense for most cases I have seen except for time. How exactly can time be the target of any verb? Wouldn't it make more sense if it's marked by で which gives us context on the action?
Target of the verb is one of many usages of particle に. It is also used to mark a place, a point in time, originator of action in passive sentences and more
While dictionary is not a primary source for grammar, for starters you can check out https://jisho.org/word/%E3%81%AB
So marking a time is simply within a scope of what に does.
That said: で is also used with time words. As far as I know this gives the nuance of "by (the marked time)" and that に has a nuance of a more exact time in comparison
Time can also be a target of a verb. The target is for the verb to be performed at the time marked by に. It is not just part of the context around the verb (the place where the verb happen, the means by which the verb happen...) therefore で is not suitable.
The target of 「行く」is to go to Japan and to be there in 2020. That's also why omitting the に from the time means this is no longer the target of the verb to be performed at that exact time, so it turns to be just part of the context.