Here are the two sentences:
あなたの友達に私の本で書かないでと言ってください
あなたの友達に私の本で書かないように言ってください
My guess is that the first is
Please say to your friend "Do not write in my book"
and the second is
Please tell your friend to not write in my book.
The second is more asking the person to explain it in their own words. The first is asking them to just say that quote.
Is this guess correct?
Your guess is mostly correct.
あなたの友達に私の本で書かないでと言ってください
This sentence sounds like you are reporting direct speech, because 書かないで is a rather casual way to say don’t write. To convey what your friend should be told without actually quoting anyone, you may use the negative imperative Vる+な form (which is not rude in this construction) or いけない.
あなたの友達に私の本で書くなと言ってください
Please tell your friend not to write in my book.
あなたの友達に私の本で書いてはいけないと言ってください
Please tell your friend he is not allowed to write in my book.
Your second sentence is also acceptable. An extreme literal translation of this sentence could be ‘tell your friend what you should tell him so that he does not write in my book’.
By the way, あなたの友達に doesn’t seem very natural to me (but it is gramatically correct for sure).
