I have not read the story, but apparently the character pictures emotions as squares, triangles and circles. I got that from a comment found here, which says
よかった(語彙力)。たしかに、Amaz○nのレビューにあるように、なぜ沙弥香が女の子にしか恋ができないかを描いているわけではない。内容紹介にもあるように、沙弥香自身の納得のためにこの物語は存在する。ならば描かれるのは沙弥香の“好き”のきっかけだろう。そもそも三角や四角、丸で表される感情に理由をつけるなんて無粋なのである。
So as long as the person who wrote this comment is not making things up or just completely wrong, it is an expression unique to the story that only people who have read it could possibly understand.
Applying it to the second excerpt:
女の子の隣にいることで、ざわめくものがある。拾いあげることのできないそれは、砂浜に広がる波の音に少し似ている。不明瞭な音は三角や四角の文字以外の形で、わたしにそれを伝えようとする。
When the character is sitting next to a girl she gets a feeling she can't quite pin down, she compares it to the sound of the waves at the seashore and that unclear sound is trying to convey a feeling to her that is something other than (what she understands as) a triangle or square.
I realize a single comment is pretty feeble evidence, but it seemed plausible enough to post an answer.