The i-adjective はやい can refer to being fast or being early, but each of the meanings has affinity towards separate kanji: 速い (fast) and 早い (early). Yet, while it's not surprising that the antonyms of both 速い and 早い have the same pronunciation おそい, both おそい that means "slow" and "late" seems to share the same kanji: 遅い. Although WWWJDIC and dictionary@goo both include 鈍い as an alternative kanji for おそい, it does not seem to have affinity towards any of the two meanings either.
Is this really the case that the same kanji form can refer to both meanings, or were there separate kanji for each (maybe abolished during the reform or something)?