The particle に can mark the agent performing the action in a passive sentence if that agent is specified, but 1) the particle に can also do lots of other things and 2) in some passive sentences (including this one) the agent is not specified.
In this case, に isn't marking the agent or “action doer” (the person or people by whom the work was assigned), but the indirect object – that is, the people to whom the work was assigned or (to use a translation that I think works a little better here) to whom the different tasks were allocated. This に would still be there even if you rephrased the sentence using the active voice, as in 社長が全員に作業を割り当てました (“The president of the company allocated tasks to everyone”).
As for why “Work was assigned (to me) by all members” isn’t a very likely translation for this sentence, as I’ve indicated above, the verb 割り当てる doesn’t simply mean “assign,” but is more like “allocate,” “parcel out,” or “apportion” – the idea is not just that something is being assigned, but that it is being divided (割る) into parts which are assigned (当てる) to different people. Thus, for this verb it’s generally much more natural for the agent performing the action to be singular and the indirect object receiving the action to be plural, rather than the reverse. (Although as in English it's not impossible for the entity performing the allocation to be plural, and in some contexts you could say something like 私に割り当てられた作業 to mean “the tasks that were allocated to me.”)