What function does は provide in statements such as 本ではない or 本だとは思わない? I notice the は after と is often left out, at least if a Google search for 思わない is any indication, but there still seems to be a general trend of sticking は into negatives where I presume they wouldn't be used in positive sentences (one hears です and である, but not ではある). What's going on here?
To answer that, I think we first have to look at one of the more important roles of topic markers (in any language that has them): marking contrast.
The topic marker as a marker of contrast
Look at this conversation for instance:
A: 一番好きな中華料理か？ たぶん麻婆豆腐だと思う。
My favorite Chinese dish? Probably Mapo Doufu, I guess.
B: そうか。俺はちょっと辛いのが苦手なんだ。 I see. I'm not good with spicy stuff.
(I really hope it's natural Japanese, but it's really not that important here)
A is asked what is his favorite Chinese dish. He picks Mapo Doufu, which is (or at least should be if the cook is not squeamish) very spicy. B says he doesn't like spicy, and 俺 (which is the topic here) stands in contrasts to A. He doesn't just say out of nowhere that he doesn't like spicy stuff - look how unnatural the English translation reads here. A more natural translation would have to use something extra to mark the contrast, e.g. "I see. Well, I'm not so good with spicy stuff."
B can also add later:
(This is actually a quote from Google, so at least I know it's natural :))
The は after 中華料理 here (especially with 大好き, which usually takes が) is a strong hint that the 中華料理 stands in contrast to something. So B likes Chinese food, but in contrast, he's doesn't like the numbing-spicy cooking of Sichuan.
The contrast marker and negation
When you say that sentence is negative, you can actually mean that several different things are negated in the sentence:
I didn't see him. (but maybe someone else did)
One of the other complements, such as the direct object:
I didn't see him (but maybe I saw someone else)
The predicate itself (which can be a verb or a noun):
I didn't see him (but maybe I heard him)
That's not a book (but maybe it's an umbrella)
The entire predication (the link of the predicate with its complements - i.e. the whole sentence):
I didn't see him. (this whole thing of "me seeing him" just didn't happen)
In Japanese verbal sentences, this stuff is quite easy - you use a negative form and if you want to emphasize that you are negating some specific complement (and not the entire predication) you can just mark it with は and change the intonation a little:
You could negate the verb itself in several ways, but I think the most obvious would be this one:
Negation of copulative sentences in Japanese
When you come to copulative sentences (i.e. sentences with a copula, in the general pattern of
XがYだ), things don't work quite as smoothly, and my impression is that people mark the negated part using mostly intonation, just as in English.
What's interesting in copulative sentences, however, is that は is used almost in all cases after で (which is the adverbial form of the copula だ), so it looks like we put the entire predication (which is held by the copula) in contrast to something. Now, this is not the case, since
～ではない is also used in sentences where the subject or the predicate alone are negated. In fact, the only place where we commonly see
～でない (without the は) is in relative clauses (I'm not sure about that, but I think even this use of
～でない is not common anymore in colloquial language).
So how did the は come to be used anywhere? This is probably part of an historical process of trivialization and loss of meaning. It seems to me that originally, the pattern
～ではない was used to only mark a strong contrast of the entire predication to something else (e.g. something that was said or implied before by someone else), but later became more and more popularized until what originally was a contrast marker became an almost necessary feature of the negative form. And this is one very important rule of structural linguistics: when an optional marker becomes necessary, it doesn't mark anything anymore.