There are two different definitions of auxiliary verb.
Auxiliary verb as the translation of 助動詞
Words like ます, (ら)れる are generally taught as "part of conjugations/forms" in most Japanese-as-a-second-language textbooks, but they are categorized as 助動詞 in monolingual Japanese dictionaries and grammar books. This word class is usually translated as auxiliary verb (or just auxiliary). The list of 助動詞 is relatively small (see below), and you may already be familiar with most of them (as part of "forms"). Generally, 助動詞 is a mishmash of functional words that work by modifying the meaning of the preceding word. Most auxiliary verbs can conjugate, and multiple auxiliary verbs can be "stacked" to say complicated things like "negative-causative-passive". For example, 飲ませられない ("is not forced to drink") is a verb 飲む followed by three auxiliary verbs (causative させる, passive られる and negative ない). Despite its name, a 助動詞 does not necessarily conjugate like a verb; for example, たい conjugates like an i-adjective.
List of Japanese 助動詞
- (さ)せる for causation
- (ら)れる for passive voice, ability, etc
- ない・ぬ・ん for negation
- (よ)う for volition
- まい for negative inference
- たい・たがる for desire
- た/だ for past tense
- ようだ for similarity
- そうだ for hearsay
- らしい for hearsay, behavior
- だ・です for predication/politeness (aka copula)
- ます for politeness
- やがる for disdain
- べし for requirement/certainty
- ...Plus some rare archaic auxiliary verbs and dialectal forms. See this Wikipedia article (in Japanese) for the full list.
Auxiliary verb as the translation of 補助動詞
Words like (～て)いる, (～て)みる, (～て)おく are usually called subsidiary verbs (補助動詞), but some people like to call them auxiliary verbs. For details, please see: What is a subsidiary verb?
Regarding です/ます, yes, they are categorized as 助動詞 in Japanese monolingual dictionaries. But I think most beginner Japanese-as-a-second-language textbooks usually regard です as a (polite) copula and ます as a politeness marker.
If you are a beginner, you may be overwhelmed by the number of "forms" in Japanese verbs, but once you reach an intermediate level and start using learning materials written in Japanese, you may want to familiarize yourself with the concept of 助動詞. See also: Why Japanese verb has so many forms?