Oh my, a question by sawa!
I suspect I have less of an idea than you do, even -- particularly since I've never even been to a Japanese school! -- but I'll write my thoughts anyway. These are just my feelings and hunches. I hope I'm not wasting everyone's time by writing stuff that's completely wrong. :(
Firstly, I don't think this is just examinations; it's textbooks as well. Examinations just follow the pattern. In making this assertion I'm going by my knowledge of the Japanese used in this book series and other Japanese mathematical writings I've seen.
In the very early stages of learning, it seems more constructive/productive to ask young children to "join in" with a (fun) activity with ましょう, rather than ordering them to do a task with なさい (which is never much fun - なさい is what you hear when you're told to clean your room or go to bed early). It's likely to change their perception of the task, if only on a subconscious level. It's friendly.
ください works in a slightly different way, being a request rather than a command, but again shies away from coldly ordering the student to do anything.
As you move up the schooling system, and the students grow up, it's natural to treat them less like kids, so switching to command forms like なさい is sensible to avoid mollycoddling. The switch to a command form also acts as a subconscious signal impressing upon the student that the task requires more of a sense of responsibility and seriousness to solve than the more casual tasks done before.
The switch from なさい to せよ is even more interesting, as it represents a switch in the relationship between the textbook and the student. Before, the textbook was a parent or teacher, nagging you and telling you strictly to solve problems. But by this point, you chose your subject. Now, you study on your own, because you want to. You signed up. So the textbook directs you with the same type of language that might be used in the army or similar organizations. There is more respect this way; the textbook no longer questions whether you will follow the orders, and it doesn't speak down to you; it simply hands out its orders in the most efficient way possible, knowing you were waiting for them and that you feel honored to follow them.
I think most of that would be lost in using しろ rather than せよ, but even so しろ is yet preferable to なさい, as once again it cuts out mollycoddling (the book isn't pretending to be friendly any more and neither are the questions).
せい just sounds old or angry? I don't know.
である is taken in preference to だ because it keeps some distance between the author and the reader. だ is casual and can be used between friends. である only appears in written works. It makes it clearer that the author isn't chatting to you, he's dictating, lecturing, reading and stating facts. This tone is more appropriate than a conversational one. It also might sound more authoritative.