According to スーパー大辞林, the 呆気 in あっけない is just 当て字 (kanji used purely for the pronunciation, rather than the meaning), so it contributes nothing to the definition. Here is the entry for 呆気ない:
あっけ-な・い [4] 【呆気ない】 (形)[文]ク あつけな・し
〔物足りないの意の「飽く気(ケ)なし」の転。「呆気」は当て字〕
予期や期待に反して簡単・貧弱で物足りない。「―・い幕切れ」「―・く敗れる」
I've bolded the part of the entry which described the word's origin. The latter half of that sentence notes that あっけない evolved from the phrase 「飽く気なし・飽気無」which originally means 「もの足りない。あきたりない。満足しない。」or "not enough/ unsatisfactory".
As for why it can also be translated as "too quickly", if something enjoyable or pleasurable happens too quickly, I imagine it would also be unsatisfying. Just like your example, if your "sweet mornings" pass too quickly, you do not have the time to enjoy it.