(I borrowed a book from Yamada-san.)
Is it ok to use から or do I have to use に ?
Without context, I can only say either is fine.
These postpositions are not exactly interchangeable even in a sentence with the same grammatical structure though. For example, 図書館に本を借りた sounds a bit strange; it sounds like you're personifying the library and saying you owe him a favor or something. In a normal context, it should be either 図書館で (marking the "place" by で), 図書館から (marking "from who/where/which" by から) or something along those lines. But since X is a person in your question, and because there's not much context to take into account here, they both make perfect sense as isolated sentences and are as grammatical as it can get.