Particle に indicates point where effect of action/deed reaches. It is not for indicating direction. Particle へ is the one that indicates direction.
For example, 道路の向こう側に渡る is telling that 道路の向こう側 is the point where action/deed of 渡る reaches at last. Action/deed of 渡る is intended to reach that point instead of that direction.
Particle に is also used to indicate indirect object for transitive verb which takes direct and indirect objects. It is because of the above said character of に, i.e. indicating point where effect of action/deed reaches.
In the example sentence of 私は兄にボールを投げた, 兄 is indirect object and ボール is direct object for verb 投げた. Particle に is telling that action of 投げた will bring ボール to reach 兄.
When this active voice sentence is converted to passive voice, 兄は私にボールを投げられた will come out. Verb 投げられた takes direct and indirect objects which are still supported by same particles as before.
According to my opinion, 私 and 兄 are at terminal position for movement of ボール. In case of active voice, ボール as effect of action 投げた moved to the position of 兄, and if we see that movement from opposite side, action 投げられた will take 私 as reaching point.
I know wording "reaching point" sounds funny but concept seems something like this, if we watch movement of ボール between two terminal positions from opposite view point.
Now, we try to apply above concept.
A. 友達にお菓子をもらった。
B. 友達にお菓子をあげた。
I believe that you understand action あげる and action もらう shows opposite direction for transfer of お菓子.
私はあなたにお菓子をあげる。
あなたは私にお菓子をもらう。
They are in the relation of active voice and passive voice essentially, even though not in standard format. It is because あげる cannot be changed to passive voice by adding れる/られる.
I hope you can accept 友達にお菓子をもらった finally by now.
Certainly you can use から which indicates starting point.
Using から is for emphasizing starting point purpose instead of using に.
あなたは私からお菓子をもらう。
私からあなたにお菓子をあげる。