So, it's not very clear to me, but I think the ……苦労が絶えないな is basically conveying that he's being put in a difficult position, and that he's frustrated by that.
From what you said and the passage, I understand that the parents are (too) laissez-faire in their parenting approach, so they won't force their children to study; and the sister, and I guess the brother by proxy, remain worried about the friends' lack of studying, so they feel like they should say something about it. But, he doesn't want to tell them to study as he understands the friends' desire to have fun. So either they say something and are the partypoopers, or they say nothing and the friends potentially fail their exams. They are damned if they do and damned if they don't.
So, I think 苦労が絶えないな in this case is kind of like saying "ugh, the problems never end" or "it's just one thing after another", perhaps when you are exasperated, and have some complex/awkward/tricky problem that is left to you to sort out, and you are trying to work through it, but every way you turn it seems like there's no easy solution.
I believe that 言われ方 is just using the passive form, so その言われ方ヤダ means (rendered awkwardly but structurally in English) "that way of being spoken to is unpleasant". Perhaps very loosely, depending on context it could be more naturally translated as "don't speak to (me/them) in that unpleasant way", though this is possibly a bit too direct/cutting.
Does that make sense within the context of what happens next? Hope so, and hope that helps in any case!