I think you can take ～といえばそれまでだ as a specialized idiom which derives from ～それまでだ. It carries a nuance that would be like:
If you say —, you're not wrong and that's a way to end the discussion.
If you say that it's nothing more than —, I can't say anything.
It's true in a sense that it's nothing but —.
This is a common phrase as such.