This is perhaps a question best left answered by a native speaker. While I will nevertheless give it a try, I won't try to answer the question more generally, but rather just specifically to the two sentences you provided.
Consider the first question,
What sort of things do you do on Sundays?
I do things like listen to music and read books.
This is definitely a general question about your habits, not a one time incident. Though I suspect that here using the たり..たり..する construct might be permissible:
However, situation B is rather different. It's asking about a particular trip when you went to New York.
What sort of things did you do in New York?
I did things like watching musicals and going to museums.
If you answered this way, I have a feeling that it's so not much ungrammatical as it is perplexing. The person asking the question would be right to wonder: "How is it they don't know what they did on their trip?"
To answer as:
This sounds much more definite.
To my ears, the ...とか...とかする construct is listing possibilities, not concrete occurrences whereas ...たり...たりする has a more definite and concrete feel. If you're enumerating examples of things you know you did or (and here's where I'm hazarding a guess) things you're definitely planning to do, then the ...たり...たりする is more appropriate.
So for example, I suspect that both of the following situations below are acceptable with only a subtle difference in meaning.
Person A: 来週ニューヨークに行くんですよね。どういうことするんですか。
Person B: ミュージカルを見るとか美術館に行くとかするかなと思ってます。
Person B: ミュージカルを見たり美術館に行ったりするつもりです。
The difference between these being, that in the first case, you're merely listing possibilities. If you don't do one of the things mentioned, no big deal, it wasn't a definite thing. But I think in the second instance, if you didn't go to any museums, then your friend might be right to ask you on your return from New York, "what happened? Why weren't you able to visit museums?" (That's of course assuming that they remember perfectly word for word what you said and how you said it.)
So, that's my gut feel about it. I'd love to hear from the native speakers what they think. Of course, feel free to downvote if I've got this completely wrong, but at least point out where I'm getting it wrong.