With verbs of motion,
- に and へ indicate destination
- を means 'along (road, river)' or 'across' (going from A to B)
- で is where you do something (not going anywhere).
The latter case has no real change of location, like swimming in a pool. To me this is simply the で for verbs of action (swimming in a pool is an action but does not go anywhere, so in this context it is not a verb of motion).
を is foremost the particle for direct objects, but that does not mean that it always is. Likewise, に is the particle for indirect objects, but one does not generally consider that '[place]に' is an indirect object.