Is the の here used as nominalizer meaning "one" or is it used as explanatory/emphasizing のです?
I assume the sentences translate to "Amongst these (courses) which is the most difficult one? Math is the most difficult one."
Therefore I think the の here is a nominalizer, but I rememberer reading that nominalizing の cannot be used before です and we should use こと instead. For example: