I think you might be misled by the non perfectly literal translation.
Ok, so let's try to break down the sentence.
Since the ~なさい indicates an imperative (some sort of order or exhortation to o something), we can safely assume that the subject here is the listener so there would be an implied "あなたは" before 弟の...
This might help to clarify the の a little bit. In fact, it's just the usual possessive particle: (you) become your brother**'s** better example = a better example for your brother.
に here is just the particle that is used when a change, expressed through the verb なる, (in this case conjugated as なりなさい) happens. In particular, "change into something" (in this particular case there is に because 手本 is a noun)
You can look up about に + なる here for example.
So, to sum it up, there is really nothing fancy about either of the particles in this sentence. This should be clear if you try to translate the sentence literally and you consider as I mentioned before that there is an implied subject in the beginning:
(あなたは)弟のもっといいお手本になりなさい
To break it down further ask yourself the following:
- Who's the subject? あなたは
- What is the action? An order to become a something (...になりなさい)
- What's that something? もっといいお手本.
- To whom is that related? To the subject's younger brother
- How? As possession: 弟の
All together: (you) become your brother's better example.
That in English renders much better as: become a better example for your brother.
I think the confusion here is in that this sentence is asking you to become actually "a better example" as that's a thing. What I'm saying as that probably you are focusing on that "for" rather than a possessive "'s".
PS: also more on なる here.