I was browsing through the Japanese Dictionary for the definition of "歯医者”
One of the definition is as follow "歯の治療をする医者".
I believe that the の here is replaceable with が here. But I am not sure how do I identify relative clauses so as to know when I can replace the の with が.
To my understanding, if it is が, the general meaning would be "the doctor who cures teeth", with the が targeting "歯" as the thing to be cured.
Is my interpretation correct? If not, how does の work in this case?
I believe that the の here is replaceable with が here
No, it isn't. Just because の appears in a relative clause does not necessarily mean it's a subject marker. You cannot replace this の to が. 歯が治療をする医者 is still "grammatical", but it means something funny like "a physician whose teeth treat patients"!
This の in 歯の治療をする医者 is simply a noun-linking particle similar to "of" in English (of course the word order is different; AのB is B of A). Read it like "a physician who does treatment of teeth". You can rephrase it like 歯を治療する医者 ("a physician who treats teeth").
At the end of the day, your question is essentially not about relative clauses but about the usage of suru-verbs. You can read similar discussion in the following questions:
