The potential for rudeness in this sentence will depend upon a combined number of factors: setting, status, enunciation, race, and gender. (More are possible, but I will only cover these.)
Setting: if you are in a classroom setting, host family, or other 'protected learning' environment, this will not likely be perceived as rude. Other settings could be more problematic.
Status: this will be a big issue. Japanese society still has lingering traces of the social stratification which used to be common a few hundred years ago. In modern times, these traces have generally degraded to be a function of simply age, societal and/or business position, and relationship. The lingual form presented is a low form, generally used among intimate/friendly people of equal status, or used from people of higher status towards people of lower status. Using this sort of a phrase "upwards" is a big no-no. If someone uses this towards you, it is a clue as to what they think your status is in relation to them. This usage could also used by those attempting to be deliberately short or rude towards someone else.
Enunciation: despite official Japanese being a fairly mono-tonal language (with the exception of a lilting accent being placed on double consonants), the reality is that emotional context is conveyed by intonation and enunciation, pretty much just like every other culture out there. Lump body-language and facial expressions in here as well. Also, smiling does not always convey friendliness in Asia, but has many other possible meanings.
Race: if you are not Asian appearing, most Japanese will give you a pass on this sort of usage, assuming that you are either ignorant, or have simply not been taught proper polite usage ("proper" usage being difficult even for Japanese natives). They will probably praise your Japanese efforts as well.
Gender: if you are female, this sort of usage might even come across as being cute, especially depending upon non-verbal cues. Sorry guys.