In this context, 惚れた女 clearly means "the woman whom he fell in love with" rather than "the woman who fell in love with someone". The subject of 惚れる is 男. Grammatically, this is an innate ambiguity of Japanese relative clauses, which work by moving a modified noun and removing the accompanying particle such as が/を/に/へ. You have to determine the correct meaning from the context.
The man fell in love with the woman.
the woman whom the man fell in love with (note that に is gone)
the woman whom the man fell in love with (subject is omitted)
The woman fell in love with someone.
the woman who fell in love with someone (note that は is gone)
the woman who fell in love with someone (target is omitted)
Likewise, 紹介した人 can mean both "the person who introduced someone" and "the person whom someone introduced" depending on the context. あげた人 can mean both the giver and the receiver. Also see: Relative Clause Ambiguous
As an aside, please recheck the meaning of 無粋.