It is probably because in this case the word 掃除{そうじ} is a not a verb but a noun. In fact, you can see it is followed by the particle を.
It's a small but very important thing to notice. If that particle wasn't there it would be indeed a (suru-) verb and your answer would be correct.
To elaborate as you asked, the point is the following:
The word 掃除{そうじ} is by itself a noun, but it can become a verb if used in combination with する (so called suru-verbs).
If you use it as a verb you could say: 弟はへやをそうじしました and you could translate this as "My little brother cleaned the room" (the verb is cleaning).
The particle を instead marks the object of the sentence (replying to the question the younger brother (subject) did what? Cleaning (object)). Hence そうじをしました means that the cleaning is the object of the sentence. Object of what? Of the verb "to do" (しました). Referred to what? What is that is being cleaned? The room. So it makes sense to use の to specify that we are talking of the cleaning "of the room".
So in this case you would translate 弟はへやのそうじをしました as My little brother did the cleaning of the room (here the verb is to do and cleaning is a noun).
So finally why is wrong to put two を? Because you would have two objects and it would not be clear to what the verb refers to. It would be like "My brother did..." what? The cleaning or the room (as if he created a room)? In other words, there would be no clear connection between subject object and verb and the grammatical structure of the sentence falls. In short, you can't have two を in a single clause, as you said.