くだけた日本語 (casual Japanese), by definition, is used when people are trying to speak friendly, not rudely. If you want to keep a distance from someone, the normal way to do so is to keep using the polite language.
Many native speakers use くだけた日本語 even when they meet you for the first time. Generally speaking, elderly people or people living in rural areas tend to speak to you using くだけた日本語. And of course they don't want to be rude.
Of course くだけた日本語 would sound over-friendly or even rude depending on the situation. In very limited circumstances, it may be possible that someone uses it because they want to be rude on purpose. For example, one may use くだけた日本語 against their boss and show that they're upset.