Going by the context you describe in the question, the meaning of that response does constitute a negative answer to the effect of something like
Well, I admire/like her. That's it. The feeling is not reciprocated.
So my rendering is a bit wordy because I am trying to render it in natural English. Although it is a negative response to the question asked, I don't think the いや here is a hard, clipped "No". Rather, it is closer to a filler and better translated as "well" or maybe "well, not really." For more on いや, see this answer:
The usage of いや in response to questions
一方的に, as you say, means "one-sided", "unreciprocated".
憧れる means something closer to "like". The aspect in the original Japanese line is continuous (憧れてる, contracted from 憧れている), telling us the action is ongoing, but I'm reluctant to use the present continuous or the present perfect continuous. In such situations usually the simple present does the job just fine, and the other two tense-aspect combinations would sound weird.
だけ is a noun meaning "only", "just", "merely", or "simply"
で is the conjunctive form of copula だ