The entry for WWWJDIC lists 飯場 as a possibly sensitive word (sens). However, the definition is just
construction camp; bunkhouse; workers' living quarters
and it appears in 燃え尽きた地図, a novel from 1967, without any particular prejudice attached (in the description of a construction site). What's going on with this word?