According to thisthe website source国語文法 (under the heading 連体修飾語になる), 【副詞】 (adverbs) can modify 【体言】 (non-inflected words, i.e nouns) in some cases. This is called 【連体修飾語】(modifier of an uninflected word), and can happen when the adverbs are in isolation or before 【の】.
Adverbs don't only modify verbs in English either. However, they modify noun phrases (i.e "I saw only the mailman") and similar grammatical structures in these cases, and not nouns by themselves ("I saw only mailman", which is not correct).
To follow this, I think it's important to remember that there isn't always a one-to-one conversion between languages and therefore we assume our adverb is the same as 【副詞】, which isn't really the case. This is one example of that, but another one is the concept of の-adjectives (縦じまのシャツ ➡ "vertical-striped shirt"), where some nouns are erratically listed as "の-adjectives" under some dictionaries and not others, when there really is no such thing. It's just one of the functions of の and works with any nouns as long as it, in essence, makes sense for the noun to describe the other noun in some way.