Does it make a difference whether they teach at my school, even if they don't teach my class?
As long as you are interacting with the said person in its role of teacher, you should use 先生. Meaning that you won't if the person is your friend, family member, partner or any other relationship that doesn't involve its teacher job, even though it can still be used in this case as a joke, irony or to emphasis the teacher position.
Additional Info: As long as you're learning something from an entity (doesn't necessarily have to be a human being), it can be called "先生" (at least in spoken Japanese). For example: It is very common for people to say "Google 先生に教えてもらった" (Google sensei ni oshiete moratta) when they learn something new via Google.