Your guess is mostly correct.
This sentence sounds like you are reporting direct speech, because 書かないで is a rather casual way to say don’t write. To convey what your friend should be told without actually quoting anyone, you may use the negative imperative Vる+な form (which is not rude in this construction) or いけない.
Please tell your friend not to write in my book.
Please tell your friend he is not allowed to write in my book.
Your second sentence is also acceptable. An extreme literal translation of this sentence could be ‘tell your friend what you should tell him so that he does not write in my book’.
By the way, あなたの友達に doesn’t seem very natural to me (but it is gramatically correct for sure).