Are there rules for when pronouns in Japanese can be omitted or you learn that by experience only?
Pronouns are usually dropped IF the subject is OBVIOUSLY known. Examples in English:
1) "Do you want to go get lunch?"
2) "Want to go to lunch?" (When the person knows you are speaking directly to them)
3) "Lunch?" VERY informal.
Also... dropping the "Anata wa" after asking multiple people, you can add "Anata GA" to emphasize to ONE person that you mean them specifically, and avoid ambiguity. Similar to in english: "Lunch?" "Want lunch?" "You?" "Hey you, Do YOU want some lunch?" (Ga fits here)
I would not call it a "topic change", but it draws the attention of the one person after the repitition might have made them numb to the fact that you were speaking to them specifically, instead of just asking EVERYONE.
Also, this is all CASUAL conversation. In formal conversation you DO NOT drop the subject, in fact "anata" might not even be appropriate. You should try to use the persons NAME-san (sama) or their TITLE to show propper respect.