A quick search on jisho.org reveals that many of the first page results of words that end with a ず are adverbs. So this type of adverb is common.
In fact, it's true (at least for these first page results) that a word is an adverb if and only if the final ず is written in kana instead of being part of the reading of a kanji.
This doesn't seem like a coincidence.
I know that ず was historically a negative conjugation (per this question), but I can't really why that would make it a common adverb ending.
Does anyone know why this might be the case?