Here are the two sentences:
あなたの友達に私の本で書かないでと言ってください
あなたの友達に私の本で書かないように言ってください
My guess is that the first is
Please say to your friend "Do not write in my book"
and the second is
Please tell your friend to not write in my book.
The second is more asking the person to explain it in their own words. The first is asking them to just say that quote.
Is this guess correct?