Exactly. See how different they are. (Sorry for oversized image.)
We imagine 手袋 as a kind of clothes for everyday life and work, and グローブ as a sports gear. Of course, more accurate transcription of glove is グラブ, which is preferred by baseball players.
Some Japanese traditional sports have gloves that are never called グローブ. For example, in 剣道【けんどう】 they use 小手【こて】（籠手） (but technically because it's rather like cloth gauntlet).