First I think it helps to start by thinking of the verb involved. It is important to know if the verb is capable of licensing a に argument.
I like to think of に as essential to understanding what happens with the verb. Take away whatever that is marked with に and your understanding of the situation changes dramatically. While whatever marked with で is additional to your understanding; removing it changes your understanding, but it doesn't augment the situation.
Consider:
- 道に絵を描く, it means "to draw on(to) the street" where the drawing's surface is the street.
- Then consider 道で絵を描く, it means "to draw in(or on) the street" where the artist is in the street drawing (on some unknown surface).
- Then consider 道で紙に絵を描く, it means "to draw on a piece of paper, in(or on) the street".
I specifically chose this example because in English you can be both on the street and in the street and I think it is relevant to your question since it asks about in/on.
Notice that if you remove 紙に from example 3, the interpretation of draw on changes drastically and reduces to the case of example 2 - "to draw on the street" (but not on the street in the sense of example 1).
Keep in mind that 描くis capable of licensing a に argument, if you try to select a different verb that cannot take on a に, you will realise that this confusion ceases to exist since if you include a で argument it becomes immediately clear that the verb does not interact with it.
It might look a little confusing, but here's how I try to clear up the confusion - by intentionally creating a nonsense sentence.
Consider:
- 本を読む - "to read a book"
- 図書館で本を読む - "to read a book at(or in) a library"
here the verb's situation has not changed compared to the previous case, you've only gained information about where the reading happens.
- 図書館に本を読む - "to read a book to the library"
here the verb's situation has changed, you're not just reading a book now, you're standing outside reading the book to the building as though the building was a intended recipient of storytelling.