First, I don't think "grotesque" is a particularly accurate translation of 気味の悪[い]. "Creepy" is much closer and would be fine in some contexts; here, a less colloquial alternative like "uncanny," "disturbing," or "sinister" would be better.
Second, nothing in this passage compares the flapping of the owl's wings to the smell of fallen leaves or says that 落葉の匂 itself is creepy. Rather, as you seem to have recognized, the phrase 気味の悪さと云ったらございません refers to the violent flapping of the owl's wings each time it attacks. This flapping produces an indefinable but unsettling sensation that changes the room's atmosphere; it might be the scent of fallen leaves, or it might be the feeling of spray from a waterfall, or of hot, damp air infused with the sour odor given off by a monkey's stash of fermenting fruit. Whatever the sensation is, it's profoundly out of place, because what all these things have in common is that one would not normally encounter them in an artist's studio, but out in the wilderness – which is to say, in the owl's natural habitat. And sure enough, the paragraph's final sentence (which you haven't included in your quotation) tells us that the apprentice later said that at the time of the attack "it seemed to him that even the dim glow of the oil lamp was the faint light of the moon shining through the haze, and he felt as frightened as if the artist's studio had been some forbidding hollow in the depths of the mountains" (さう云へばその弟子も、うす暗い油火の光さへ朧げな月明りかと思はれて、師匠の部屋がその儘遠い山奥の、妖気に閉された谷のやうな、心細い気がしたとか申したさうでございます).
So there's nothing inherently "creepy" about the smell of fallen leaves (or the spray of a waterfall, etc.), either "in the culture" or in Akutagawa's own sensibility. Rather, what is disturbing is the way the owl's attack evokes a desolate wilderness, effectively transforming the artist's studio – which ought to be a place of culture and safety – into a place of savagery and danger.