The correct sentence in standard Japanese is:
弟は部屋を掃除しました。 He cleaned his room.
This is because 掃除する is a verb, and a verb must be modified by an adverbial phrase. Since 掃除する is a transitive verb, its object should be marked with を.
The following sentence is also correct:
弟は部屋の掃除をしました。 He did the cleaning of his room.
In this case, しました is the verb that just means "to do", and 掃除 is a simple noun, and 部屋の is an adjectival phrase that modifies 掃除. 部屋の掃除 as a whole serves as the object of the verb する.
The following sentence is acceptable only in casual conversations:
[?] 弟は部屋の掃除しました。
掃除しました is a verb, but apparently it's modified by 部屋の, which is an adjectival modifier. This is clearly wrong in written formal Japanese. It is heard in hasty conversation where を can be omitted. Just as you can say 本読む instead of 本を読む in conversations, you can say 部屋の掃除する as the short version of the second example above.
Anyway, since this is an N5 question, I think you should choose を following the basic formal rule.
See: Difference Between べんきょう する and べんきょうを する