(This question had to show up eventually… :) For my answer, I'll be borrowing most example sentences and categorizations from pages 176-179 of 初級【しょきゅう】を教【おし】える人【ひと】のための日本語【にほんご】文法【ぶんぽう】ハンドブック and from this PDF.
Cases where only の is allowed
When the following verb deals with one of the senses: 聞【き】く, 聞【き】こえる, 見【み】る, 見【み】える, 感【かん】じる, and so on.
隣【となり】の家【いえ】でだれかが叫【さけ】ぶのが聞【き】こえた。 I could hear someone shouting in the house next door.
船【ふね】の中【なか】から、魚【さかな】が泳【およ】いでいるのが見【み】えます。 From inside the boat, I can see fish swimming.
When the following clause occurs in concert with the preceding clause: 待【ま】つ, 手伝【てつだ】う, じゃまする, and so on.
テニスコートが乾【かわ】くのを待【ま】っています。 I'm waiting for the tennis court to dry.
このパソコンを運【はこ】ぶのを手伝【てつだ】ってください。 Please help me carry this computer.
When the following verb is one of 止【と】める, やめる, and so on.
彼【かれ】が出【で】て行【い】こうとするのを止【と】めました。 I stopped him trying to leave.
タバコを吸【す】うのをやめましょう。 Stop smoking.
As you can see, the common thread running through these cases is that there is an immediacy of time and/or location. That is, the outer clause necessarily occurs at the same time and/or same location as the inner clause.
Cases where only こと is allowed
When the following verb deals with communication or internal thoughts: 話【はな】す, 伝【つた】える, 約束【やくそく】する, 祈【いの】る, 希望【きぼう】する, and so on.
ゼミに出【で】られないことを先生【せんせい】に伝【つた】えてください。 Please tell the teacher I can't make it to the seminar.
復興【ふっこう】が速【はや】く進【すす】むことを祈【いの】っています。 I'm praying that the recovery proceeds quickly.
When the following clause is one of だ, です, or である.
私【わたし】の趣味【しゅみ】は映画【えいが】を見【み】ることです。 My hobby is watching movies.
(This is because if の were used, it would be confused with the ~のだ pattern.)
When the こと is part of a set pattern such as ことができる, ことがある, ことにする, ことになる, and so on.
私【わたし】は外国【がいこく】で暮【く】らしたことがあります。 I've lived in a foreign country before.
あれを見【み】なかったことにする。 I'm going to pretend I didn't see that.
With こと, the immediacy expressed by の is lost, and matters are considered from a more abstract, removed standpoint.
Cases where both are allowed
In general, for any cases not covered in the above lists, you can use either こと or の, but there are some times when you might choose one over the other. For example, consider this pair of sentences from a page in the 日本語Q&A at ALC:
僕【ぼく】はこうしてのんびり映画【えいが】を観【み】ることが好【す】きだ。 I like relaxing with a movie like this.
僕【ぼく】はこうしてのんびり映画【えいが】を観【み】るのが好【す】きだ。 I like relaxing with a movie like this.
The key here is the こうして ("like this"), which indicates the speaker is making a statement about something happening right now. Thus the statement has the immediacy of time and place that の is best for. Using こと here isn't technically incorrect, but it sounds a little unnatural, so の is the better option.
Non-nominalizing uses of の
As a side note, there was one example sentence in the PDF I linked which doesn't belong, in my opinion:
毎月【まいつき】おこづかいを貯金【ちょきん】しているのは、新【あたら】しいスケートボードを買【か】いたいからです。 The reason I'm saving my allowance every month is because I want to buy a new skateboard.
This use of の is the "unspecific noun" pattern (I cannot seem to find the proper grammatical term at the moment). In this sentence, の could be replaced with the more specific 理由【りゆう】. Another example:
日本【にほん】にはじめて行【い】ったのは5年【ねん】前【まえ】です。 The first time I went to Japan was five years ago.
の could be replaced with 時【とき】 here.