The two sentences are rather different.
When you say
You probably meant "I study Japanese when I go home from work." But, this is not what the Japanese means. The Japanese says, "I will study Japanese before I return home."
Captures your most likely intended meaning.
First とき really doesn't mean when quite as we use it in English. It really has more of a meaning of before/after depending on the tense of the verb preceding it.
帰る時 > before I return home.
帰った時 > after I got home.
When using <S1>時<S2>, event <S1> describes the state of affairs that define the time when event <S2> occurs.
Verbs of movement, like 帰る, complicate this since they are instantaneous verbs in Japanese (ie., 帰る=>"I will return home", 帰った=>"I've returned home"). It is possible to construe
as meaning, "On my way home from work, I studied Japanese."
Since 帰る does not describe a completed state of affairs, it cannot be construed to mean you studied after getting home since then the act of 帰る'ing (to create an linguistic abomination) hasn't been completed: ie., you're not home yet.
You might ask, then how is 帰るとき and 帰る前 different?
The verb 帰る generally means "to return home". 帰る時 means "before I got home" (yes, I used past tense in English). But 帰る前 would mean something more akin to "before I left to go home".
So, if you say 仕事から家に帰るとき、日本語を勉強する。 you could mean that on your way home you study Japanese (on the bus or train) or you go to a language school or something like that. You've already left work, but you haven't made it home yet. All because 帰る hasn't been completed yet.
So, now let's revisit:
I believe this would sound a bit off to a native Japanese speaker. In the construction <S1>と<S2>, event <S2> is not generally something that you have control over. と shows a correlation between two events (in which case we usually translate it as whenever). Or, it expresses a consequence between two different events (event S1 having happened, S2 happens). But, in either case, S2 isn't something you really have much control over.
So, if you want to say something like "When I get home from work, I study Japanese" then try something like