I don't understand the difference between these two words. In my textbook they appear like "miss the stop" (norisugosu) and "pass my station" (norikosu), but they both sound like you don't go out the train when you should.
乗り過ごす (nori-sugosu): "to miss one's stop (by mistake)". This happens typically when a passenger was sleeping. After you do this, you usually get on another train going in the opposite direction.
乗り越す (nori-kosu): This can refer to the same thing as 乗り過ごす, but usually refers to something different, "to go beyond one's stop designated by your ticket (intentionally)". This typically happens when you change your destination after getting on a train/bus. After you do this, you usually get off the train near your new destination, and pay for the additional distance at a fare adjustment machine/office (aka 乗り越し精算).