I would say the distinction between 渡す, 送る and 届ける maps quite neatly to the distinction between the English "hand over", "send" and "deliver". Of course two words are never exactly equivalent, but I'd say the correspondences are close enough to be very helpful in this case.
渡す is "hand over"
The focus is on the specific act of passing the item from one person to another. For that reason it's generally only used for face-to-face encounters, since other methods of transferring an item are more indirect and there isn't one particular moment where it passes from one person's hands to the other's.
I don't think there's an actual restriction on using it in a particular direction, but it may be more common with the speaker as the "giver" simply because it's a very neutral, matter-of-fact verb, and you'd usually use a more grateful-sounding verb like くれる when talking about receiving something. Nonetheless, I think the passive 渡される at least is relatively common.
送る is "send"
In contrast to 渡す, this one is inherently indirect. You're almost certainly not physically handing over the item yourself when you 送る something. You could be sending it through the mail, or passing it on through an acquaintance, or sending it over the internet, but not handing it over face-to-face.
Note that 贈る, pronounced the same but with different kanji, actually functions quite differently and doesn't have any inherent indirectness - it can be used regardless of whether you're sending something via mail or handing it over in person; the only thing that matters is that you're providing it as a gift. I wouldn't equate this おくる to "send", and indeed I don't think we have a close English equivalent.
届ける is "deliver"
The focus here is on the act of transporting an item from one place to an intended destination. It doesn't make much difference whether the item is something you prepared yourself, something you received from someone else or something you just happened to find on the street, what's important is that you're physically taking it to its destination yourself. That destination can be the intended recipient, or, if you don't know the recipient, somewhere that can help it on its way to its recipient (such as a police station, lost and found collection, etc.)
Your sentence 警察官が僕に忘れちゃった荷物を届けた wouldn't usually be used, but only because you don't usually use plain verbs in general when you yourself are the object - most verbs feel like they're by default from the perspective of the subject, so they sound a little awkward on their own when talking from the complete opposite perspective. You would usually say 届けてくれた for someone delivering something to you, or perhaps something like 届けに来た if you want to make a more neutral-sounding statement.
よこす is ???
Unlike the other three verbs (but similar to 贈る), this one doesn't really have a neat English equivalent. Thankfully, it's also the least common of the lot, and I can't think of many situations where you would need to use it over one of the other alternatives.
As you say, its main quirks are that it is always from the perspective of the recipient, and sounds significantly less respectful than any of the other options. It can certainly come in handy in rough, casual banter, but should be used only with great care, if at all, in polite company.
