The sentence in question is not really well-organized, but at least understandable without ambiguity. First, the と after 揺らぎ is connecting two reasons. The sentence can be rephrased as:
(僕がすっきりしない原因は、)ひとつには偶像としての彼女の姿が揺らいでしまったからであり、もうひとつには僕自身が生々しい二人の間に入ってしまったからだ。
As for the first reason, 揺らぐ has a negative connotation (lose stability, stumble. See my previous answermy previous answer). The good image of 綺羅めくる which the main character held has been negatively affected during the conversation.
As for the second reason, obviously he's referring to the complicated relationship between めくる and her manager (都). 落木部さん doesn't play an important role in this chapter. ~の間に入ってしまった in this context means to end up being involved. 生々しい here has a bit of negative nuance like "(too) realistic", "overly uncovered". For example one can say 「夢を語っているときに、お金の話は生々しいからやめて!」「生々しい芸能界の闇闇」「生々しい話をすると、ディズニーランドのミッキーマウスの中には人間が入っている」.