I don't get much of the context, nor how the character's attitudes relate to each other, but I were to say something,「気にされる」is used to infer that "...it's better for ヒューバート to not take notice of Person 1 and Person 2 (because it could be 面倒くさい, awkward, clunky, faux pas...etc for them).
気にされる action comes from ヒューバード, and it's better for him to not come and take notice of Person 1 and 2 because it's meddlesome, fearful (whatever the emotion is here). If ヒューバードー doesn't come, we can maintain our 気楽-like state.
To better understand 気にされる、think of a student being looked down up by a teacher. There is a power difference here, a hierarchical difference, and although you may not be able to disagree with your teacher directly, you have alternate means of talking behind the teacher's back to avoid confrontations. Although Western culture values individualism and directness, Japanese value harmony and group cohesion, hence a tendency to be in a state of 気にする when the hierarchical nature of the Japanese culture makes it difficult to express your own opinion.
So they are 「気にする」（気にしている）, but they expressed it in a way that, simply stated, "that it would be bothersome/troublesome/meddlesome/気にする" to be 気にされる by ヒューバート (assuming he is of higher status). This is supported by the first sentence: 「あんまり偉い人が来ないなら俺たちも気楽で良いな」
Apologies for redundancies, but I hoped it served to help you understand!