There are a few other questions related to this, but they do not explain the why of the matter.
As I understand it です is present tense while でした is past tense. As such, I would (wrongly) intuit to say おいしかったでした, i.e., where both おいしい and です are changed to the past tense.
I tried to ask Japanese friends and they did not really know the answer. The argued that "if おいしい is in paste tense, です doesn't need to be because おいしい already is". But by that logic why is it not おいしいでした for example.
Clearly past tense works a little bit differently then I am used to, but I have not found an explanation of this anywhere. If anybody has a detailed answer or can link to a clear source, I would be very grateful.