What's the difference between 差 and 違い? When would I use each?
Which (if either) would I use for describing the difference between something like sample data and the best fitting equation?
違い simply means "difference".
差 means difference in the sense of a margin or gap, and seems to relate to things that are "calculate-able".
誤差【ごさ】 → Margin of error
時差【じさ】 → Difference in time : 時差ぼけ → Jet lag
It seems like you could replace most instances of
差, but not the other way around.
差 is definitely more appropriate when talking about sample data, etc.
However, there might be times when the two have distinct meanings. For example, say you have a boy who is 10 years old, and another who is 20.
年齢の差 would be "10 years", but
年齢の違い might describe how the characteristics of their ages differs; like "10-year-olds have more energy, but 20-year-olds are smarter." Not sure if this is correct, but seems different to me.
I agree in general with istrasci's answer. Just to add,
差 is used particularly for things whose difference is evaluated by the difference in the mathematical sense (that is, by subtracting the value of one from that of another). Times and ages are usually compared in this way, so it it appropriate to use
違い is a general term for describing differences. Difference among things that do not have a clear quantity or things that are not evaluated by subtraction is expressed as
違い. Things like the difference between Picasso and Monet's paintings, which cannot be easily quantitized, or the volume of a sound, which is usually evaluated as magnitude rather than difference, can be expressed as
You can use both to denote a "difference" in a casual conversation. E.g. in many contexts, 違いが分からない ＝ 差が分からない. If a person is really nice to your friend and really mean to you, you could say「この（扱いの）差はなんだ……？」. Similarly you could use 差 to describe change, e.g.「一年前との差」. In both cases, you could replace 差 with 違い if you wanted to.
The differences are as follows: