Both に and を are used, but:
に is "formally correct" and far more common.
を seems 'fishy' to some native speakers, but some usages are correct to some native speakers.
に can mark any of the following (all similar): the receiver of the revenge, the "reason" for an act of revenge, the originator of the cause of desire for revenge. (.. and possibly other cases as well).
[Note: following have been discussed/confirmed with a native speaker and are consistent with the examples I've seen, but are 'shaky' ~ if you see counter-examples or have specific doubts please let me know so I can continue to improve this answer]
を is more limited! It can only mark a physical entity that is receiving the act of revenge/retaliation. It can't mark a "reason" for retaliation like "terrorism", but it can mark a target like "the country the terrorist came from". That is 「テロに報復する」 is grammatical, 「テロを報復する」 is not, but both「その国に報復する」 and「その国を報復する」 are grammatical.
Although を's usage is low overall, its relative usage appears to be greater in the context of news articles and broadcasts, especially when marking countries targeted by retaliatory attacks. The intent might be to give a greater air of neutrality, matter-of-factness, etc. It is more likely to be used by a third-party and/or to emphasize that the marked noun was "just the actual/physical target" of a retaliatory action.
In a comment, chocolate correctly points out that some dictionaries mark 報復する as an intransitive verb, and that intransitive verbs generally don't take a direct object via を. However, it is also worth mentioning that exceptions exist. In particular, consider 走る, which is intransitive. However, it can be used with を, e.g., 「道を走る」 is grammatical. To me, を報復する seems like a very similar usage.
A wild and unsubstantiated theory about the を usage that is nonetheless consistent with all examples I've seen so far is as follows. Is it possible that in this case を and に collapse to に when both are applicable to the same noun? According to some approaches to Japanese grammar, は and が behave the same way (in 私はアメリカ人です, 私 is semantically both は and が, but は "eats" the が by a forced convention). If so it would partially explain the prevalence of に usage as を is eaten in all but special cases. If so, に might be more about "reason" for revenge and を specifically for target ~ except that in the vast majority of cases where the same noun is both, に wins out.
EXAMPLE: In any case, the following article actually uses both を報復する and に報復する so it is an example of how a single native author uses them differently:
Roughly: "Even if terrorism were to occur in Japan, (Japan) must not retaliate on the other (country)"
[note that "the terrorist/act of terrorism" and "country from which that terrorist came" are subtly different, and here the later is marked with を]
Roughly: "(Not just for murder, but) even for terror, vengeance must not be pursued."
[Here, "terror" is the action for which vengeance is being sought, thus marked with に.]
[Note: this answer has gone through multiple heavy edits]