Yes, in general, something like 彼女に渡された is an ambiguous expression; this 彼女 can be either the giver or the receiver depending on the context and the other particles in the sentence. Similar ambiguity happens when a verb related to giving or passing is used in the passive form.
However, there is no ambiguity in your sentence. There is already あいつから which clearly specifies the giver, so 彼女に marks the receiver of the book. In addition, if 彼女 is the giver, you have to mark 本 using を, not が:
本を彼女に渡された。
[I] was given a book by/from her.
本が彼女に渡された。
The book was passed to her.
For more discussion regarding this type of ambiguity in passive sentences, see: