I am not a native speaker of Japanese but I know 年寄り as a neutral word, which gets used not just in conversation but in notices. I took the following from SpaceALC:
ここはお年寄りや体の不自由な人のための優先席です。
PRIORITY SEATING FOR SENIOR CITIZENS AND DISABLED PERSONS
I think you just need to be sensitive when referring to people's age, as you would in English eg use お年寄り if you feel awkward and want to express some reserve.
FWIW, I don't really see a difference between aged-person and elderly but these things vary from place to place and change with time. (My aged mother would not thank you to refer to her using either word though neither word's roots have a -ve connotation.)