僕にしかできない事
Potential ambiguity here is caused by particle に having multiple functions:
Indirect object marker ("to ..., for ...")
Agent marker in clauses with passive/potential verbs ("by ...")
In English and many other languages, passive and potential are syntactically or morphologically distinct constructions, but in Japanese, at least historically, they were different functions of the same constructions.
Bjarke Frellesvig (2010, "A History of the Japanese Language", page 63) in chapter about Old Japanese (about 700 - 800):
The OJ passive functioned as a pure passive, a medium voice, and as a potential. OJ had two competing passive markers of which -(a)ye- was used rather more frequently than -(a)re- (their roles are reversed in EMJ into which -(a)ye- only survived in a number of lexicalized forms (kikoye-, miye-, omopoye-) and in reading glosses to Chinese texts, see 9.1.6). -(a)ye- was used with QD, n-irr, r-irr and UM verbs; -(a)re- with QD, n-irr, and r-irr verbs. Passives were not formed on verbs from other conjugation classes. -(a)ye- thus had a wider use than -(a)re-, both in terms of distribution and in terms of frequency. The straightforward interpretation of these facts is that -(a)ye- was the earlier passive marker and was replaced by innovative -(a)re-. There are a few lexicalized passive forms with a slightly irregular formation: kikoye- <= kik- 'hear'; omopoye- <= omop- 'think' (found alongside omopaye-, but was much more frequent).
The passive auxiliary variant -rare-, attaching to vowel base verbs, does not appear in the language until EMJ (see 8.4.1) and forms no part of OJ, although it is included in some grammars.
Bjarke Frellesvig (2010, "A History of the Japanese Language", page 338) in chapter about Late Middle Japanese (about 1200 - 1600):
The ancestor of the cNJ potential auxiliary -re- makes its appearance in the second half of LMJ. It is found with consonant base verbs in the shape -e-, e.g. yom-e-nu 'cannot read' in the Shiki-shō from 1477, and initially had the same functions as the passive: passive, potential, and respect. -E- is usually said to have developed as a reduced variant of the passive -rare- and for example in the Shiki-shō we find both yom-e-nu and yoma-re-nu 'read-PASS-NEG; cannot read'. Alternatively, -e- may have developed from the auxiliary verb -e- 'be able to' (< e- 'get'). During NJ this auxiliary became specialized as an independent potential auxiliary, but only recently in cNJ was it analogically extended to be used with vowel base verbs, e.g. cNJ tabe-re- 'be able to eat'.
Verb できる also has meanings "be able to do" (potential meaning) and "be made" (passive meaning), and other meanings.
will taking away に as 僕しかできない事 make any difference?
しか replaces が and を, so without しか this sentence could be:
what should I say if I wanted to express "(1) something (you) can do only to me"?
To avoid aforementioned ambiguity of particle に, another construction can be used, e.g. 〜の為{ため}に:
僕の為{ため}にしかできない事