Though I don’t have any knowledge about the context of the sentence from which the phrase was picked up, I don’t find any particular problem with the expression in the phrase “僕だけがいない街” on its alone. “僕だけがいない” is used an adjective clause to depict ”街” - the town.
It means “the town lacking of only me,” i.e, “the town where everybody is there, but me.” Nothing is strange, but it sounds creative and poetical to me. I love it.