Is this a grammatically sound sentence?
今年、私が勉強したことがない科目を習うことにします。
It's a grammatically sound sentence. That said...
1) The mixing of 勉強 and 習う is kind of weird, I think.
2) Unless it's there for special emphasis, I think you can dispose of the 私が.
3) This might just be a question of style, but ~ことにする should be in the past tense when making a decision (since you've already made it). If you're making a selection, then nonpast sounds right (てんぷらにする).
今年は勉強したことがない科目を勉強することにした。
(In fact, 今年は勉強したことがない科目にした might be the least redundant. And you can also say 今年は勉強したことがない科目にする, too, since it's a selection.)
In this case, if you're going to use nonpast, then it sounds kind of redundant to say 勉強したことがない科目を勉強することにする - just say 勉強したことがない科目を勉強する。
4) I have a slight nagging doubt about using 勉強する with 科目. Something in me wants to say that とる or some other verb might be more natural. That's beyond the limits of my 勘 as a nonnative speaker, though. :) Maybe a native speaker will chime in.
Yes, it is, if only that's what you're trying to say.
It means: This year, I'm(we're) decided to learn (i.e. take classes of) subject(s) that I've never studied.