覚える means to memorize something, and 覚えている means that someone remembers something.
Your first sentence means I will memorize his ex-girlfriend's name.
If you want to say ''I will memorize and will not forget her name'', you can also say 覚えておく／覚えておきます.
覚えます is also used when you say the feature or the nature of someone (e.g. 彼は人の名前をすぐ覚えます。), however, your first sentence is too limited, so nobody thinks you're speak about your own nature.
Your second, the last sentence is natural as a translation for I remember his ex-girlfriend's name. It's not the progressive tense. ''Verb + ている'' is not necessarily progressive tense, can be resultant state. Some verb are used for both progressive and resultant, others are used for only progressive, the others are used for only resultant.
We think 勉強している is to be studying at a glance, but how is it in a sentence そんな簡単なことは、ずっと前に勉強している? Its 勉強している is not present progressive tense but rather present perfect tence in English.
By the way, people who are living...live in western Japan (more west than Okayama, Tottori, and Shikoku area) distinguish these two verb + ている by using different form like 書きよる (be writing) and 書いとる (have written or be written).
The form of the verb can be different between Japanese and English.
For example, 住んでいる is not to be living but just to live, 着ている is to wear cloths, or to be wearing cloths on in some context.