They are both used on non-exhaustive lists but how do I know what particle to use? Is there a nuance?
There is a difference. も is used to indicate that the noun it follows is connected to something previously mentioned. や is used to indicate that the list of items mentioned is only a partial list, and that other things could be added. But perhaps you meant も...も when listing things? See below.
私も行きたい。 I want to go too.
デパートで服や本を買った。 I bought clothes and books (among other things) at the department store.
You can use も...も to list more than one thing, but now you are emphasizing each of those items, rather than implying that there are more things in the list. For example,
私はビールもワインも好きです。 I like both beer and wine.
Compare that with the use of や...や (which needs など to complete it), where the items are not specifically emphasised and it is implied that other items could be added:
宴会の料理は、てんぷらや、すしや、さしみなどだった。The food at the party was tempura, sushi, sashimi, and so forth.