掃除する
The phrase 掃除する acts like a single verb. It's technically made of two words:
- the verbal noun 掃除
- the verb する
But together they act like a single verb. In this case, that verb is transitive, which means it takes a direct object marked by を:
部屋をobject 掃除するverb 'clean the room'
The verb is 掃除する, and its direct object is 部屋.
掃除をする
When you put を between 掃除 and する, they no longer act together like a single verb:
掃除をobject するverb 'do cleaning'
Now the verb is する by itself, and its direct object is the regular noun 掃除.
How do we add 部屋 to this clause? Well, the verb する already has one direct object, and we can't add another:
*部屋をobject 掃除をobject するverb ← ungrammatical
But we don't really want to relate 部屋 to する anyway. We want to relate it to 掃除, and that's a noun. How do we show a relationship between two nouns in Japanese? With the particle の:
[ 部屋の掃除 ]-を する 'do [the cleaning of the room]'
Now the verb is する 'do' and the direct object is the entire phrase 部屋の掃除 'the cleaning of the room'.
Notes
Sometimes people explain 掃除する as ellipsis of 掃除をする. But it's important to realize that this is not the case:
部屋を掃除 する ← OK
*部屋を掃除をする ← ungrammatical
Adding を in like this makes it ungrammatical, so it can't be ellipsis.
In this answer, I used English translations like 'do the cleaning'. In fact, you'd usually translate both 掃除する and 掃除をする to 'clean'. That's because when you translate, you look for the most natural way to express something in the target language. But in this case, my goal was to try to show the grammar of the original Japanese, so I deliberately used less natural sounding translations.
In linguistics, phrases like 掃除する are called 'incorporated', while phrases like 掃除をする are called 'unincorporated'.
The giant asterisk * is used in this answer to mark a sentence as ungrammatical.