Think about it this way.
When you have a verb, it has certain 'slots' that have to be filled with nouns. 食べる, being a transitive verb, has two slots - subject and object. である also has two slots (though it's not a transitive verb, it's a special kind of thing) - subject and predicate. We've got three nouns here (これ, 本 and 犬), and we need to figure out how they fill those four slots.
One is quite clear - 本 is the object of 食べる. But what's its subject, これ or 犬? Notice that 食べる is in a relative clause, meaning that one noun has been pulled out of this clause and placed in the main clause, filling two slots simultaneously. In Japanese, this moved noun always appears immediately after the relative clause's verb, meaning that 犬 has to be the subject of 食べる. Looking at the main clause, you can see that 犬 also must be the predicate of である, filling that slot also.
Now we've filled three of four slots: object of 食べる, subject of 食べる and predicate of である. This leaves only one place for これ to go: subject of である. The whole thing then translates as 'this is (a/the) dog that eats big books'.
(It helps to notice that if これ is somehow attached to 食べる, you end up trying to fill two slots with three nouns, resulting in something like *the dog that this eats big books.)
Edit: Noticed a thing - 大きい also takes one noun, but again because of relative clauses, that noun has to be 本. So five slots total, but still the same idea.