I believe the most proper wording is the one in the other answers [独学]{dokugaku}. This word means (at least to me as a non-native speaker/learner of Japanese) self-study as in to learn something substantial by oneself.
I think another phrase that one can use is [自分]{じぶん}で[勉強]{べんきょう}しました. This phrase also literally means "I studied it by myself." I would take this one to be less formal in use and meaning. But I could be wrong on this point (and I'm sure I'll get downvoted if that's the case).
While [自己]{じこ} does mean self (actually each half of it means self in a different sense in both Japanese and Chinese), it doesn't mean self as in "by oneself" so it cannot produce a compound that would mean study by oneself. Cf. [自己紹介]{じこしょうかい} = to introduce oneself. So [自己勉強]{jikobenkyou} would mean the study of the self. The word [自我]{じが} also means self but refers to something like the ego in psychology -- not the self of self-study. (Here's the Japanese wikiepdia on it)
Consequently, the newly added answer [自己]{じこ}で[勉強]{べんきょう}する is one that sounds extremely unnatural.