はじめて(hajimete) is the te-form of the verb はじめる(hajimeru) - to begin. How come it means "for the first time" and is used appropriately to mean the whole phrase? Is it an idiom?
There are two kanji with the same kun-reading "haji-". The kanji for hajimete meaning "for the first time" is 初, which means initial, first, or beginning but does not mean to begin. Don't mix it with 始, which means to begin.
And yes, some te-forms have lexicalized and idiomatic meanings, and I think 初めて is one of them. You can see similar examples in this question: What is the role of あるいて?