When it comes to any question about "why is word XX spelled with kanji YY instead of kanji ZZ?", or "what is the difference in nuance between kanji spelling AA and kanji spelling BB of the same word CC?", you're ultimately asking about the kanji themselves -- and this often means we want to look at those words in Chinese to uncover additional information.
For instance, if we examine the senses of the Chinese term 初, we see the following definitions:
- initial; inceptive
- original; first; primary
- elementary; basic
- beginning; start
- initially; originally
Meanwhile, the Chinese term 始 has these definitions:
- to begin; to start
- beginning; start
- initial; first
- then; only then; only after
Turning our attention back to the Japanese terms that include these kanji, we see some of this Chinese background reflected in the modern Japanese meanings. For instance, 初級 isn't just the first level, but more specifically the primary, elementary, or basic level, senses that we find with the 初 kanji but not with the 始 kanji. Meanwhile, for 開始 is literally 開 ("to open; to start") + 始 ("to begin; first; then, only after"). One way of parsing this kanji spelling is "things open up or start up, and then [other things happen]", a sense found with the 始 kanji but not the 初 one. (Note that I'm stretching the point here -- in truth, 開始 was coined in Chinese, and could also be parsed as just "to open; to start" + "to start" as a kind of duplication of near-synonyms to reinforce the basic meaning.)
Looking at the alternative spellings of the wago or native-Japanese terms はじめ or はじめて, the Chinese-derived meanings of the Chinese-derived kanji still seem to have some relevance, where はじめ or はじめて spelled with 初 tends to have overtones like those above listed for 初, and where はじめ or はじめて spelled with 始 tends to have overtones like those listed above for 始.
Please comment if the above doesn't answer your question.
