Are you taking those examples from "A Dictionary of Basic Japanese Grammar"?
The book itself says the reason:
"In indirect passive sentences the agent of the event is usually animate and the action is volitional".
This kind of passive is usually used when you suffered the actions of someone else.
When you say 僕は雨に降られた it is as if you are putting the blame on the rain, as if the rain wanted to hurt you on purpose.
Of course that is not true, so usually you cannot use that kind of passive for inanimate objects.