I have read in several sources that it is grammatically incorrect to use は (topic particle) in relative clauses. However, I am almost certain that I have seen constructions like the following:
学生ではない男子は今自分で遊んでいる。
Is this technically incorrect? Should it be 学生でない? Or is this は not considered the same particle that is forbidden in relative clauses?
Firstly, the rule that the “non emphatic/themataic-は” cannot be used in subordinate clauses also has many exceptions, in particular with fixed constructs, for instance:
あの人はやはり参加しないということですか？ -> “So you mean to say he won't be attending after all?”
Technically speaking the outer matrix sentence seems to have no particular subject and “あの人はやはり参加しないと” is itself a subordinate clause of “の人はやはり来るという”, which is a relative clause of “こと”, which is the complement of the sentence in theory. This is fine because “ということですか？” is simply a fixed expression that can be placed after any sentence roughly meaning “so you mean to say that...?”. This works the same with things such as “〜たことがある” or “ことができる” and many other forms that technically make what would be the main clause a subordinate clause of something else. It also applies to other things such as “実は” at the start of a sentence.
Secondly however, even if we remove all the exceptions that typically have to do with fixed patterns, “emphatic” or “contrastive” topics can still be used within relative clauses fine, which is the origin of “〜ではない”, though many argue it has lost all its original contrastive function in modern Japanese and that “〜でない” is archaic and almost never used, but I find that “〜でない" is indeed more common in relative clauses.
“emphatic topics” are more stressed, and they typically but not always contrast with something else and have a “not this, but that” kind of nuance to them. The thing they contrast with need not be explicitly mentioned however. With a clearly emphatic/contrastive topic, this is still fine:
あの人は好きな映画は私は好きじゃない -> “I don't like the films he likes.”
This sentence has three topics, but only one, the middle one, is non-emphatic/thematic, the other two are emphatic and the first one is inside of a relative clause, which is fine for an emphatic topic. So the first and third topic are contrasted with each other here. It is fine for one of them to be inside of a relative clause and the other in the main clause.
But again, “〜ではない” doesn't really seem to obey the normal rules of an emphatic topic in Japanese. People simply “say this” because the nuance weakened over time.
I think that using ではない in relative clauses is correct because ではない as a whole is the negative conjugation of です, and not a set of particles.
That said, I'm aware that the construction Aではない (meaning not to be A) comes from Aでは ない (the quality of A does not exist) in a more literal sense, but I'd look to the verb as a unit rather to parsing it down to particles. At the end of the day, when we say 学生ではない男子 we mean the boys who are not students.
