もう一度やってみるようにと彼を励ましなさい。 I'm confused by the usage of と here. もう一度やってみるように彼を励ましなさい。would be translated as "in order to make him try one more time, please encourage him" but with と I'm not sure how it'd be translated.
The 「と」 is the quotative particle. Whenever you encounter a 「と」 that seems difficult to understand, try putting the preceding phrase in quotation marks. If that helped you comprehend the sentence, that 「と」 would be quotative 99% of the time.
Thus, the sentence means:
Please encourage him to try to do it one more time.
or, more literally,
Please encourage him by saying "Try to do it one more time."
「～～ように」 as a request/order is explained here: