The context is : some people are acting stupidly, and someone else watching them says this :
My first thought was to translate the sentence in the following way :
That kind of guys are annoying.
With the idea that the speaker is annoyed BY (に) the group and that it's a personal feeling (ものだ).
But the translation I have roughly says :
"That kind of guys should be ashamed."
I know that ものだ can have a meaning close to べきだ, like if we were expecting the action to take place, which would fit with the translation I have. But then I would expect the sentence to end with 連中は and not 連中には for the group to be the subject... Unless there can be a construction of 困る with に, other than "be embarrased by something..." that I'm not aware of...
So, which に/ものだ is it?