Googling with the word "transcription," I was able to compile the following answer:
The apostrophe is used before the romaji letters a, i, u, e, o, y when they follow n/ん in order to distinguish, for example, kani from kan'i. The apostrophe makes it clear that ren'yōkei means …んよ… and not …にょ… as in renyōkei.
The Japanese government’s guidelines for romanization suggest inserting apostrophes in these situations:
The apostrophe is also part of the romanization system used by the U.S. Library of Congress, a system based on the revised Hepburn system (first laid out in Kenkyusha's New Japanese-English Dictionary, 3rd ed.):
Transcribe the apostrophe ( ' ) between syllables when the first syllable ends with the letter n and the following syllable begins with the letter a, i, u, e, o, or y and when it is necessary to separate romanization.
In the revised Hepburn system the apostrophe is used to mark “the
separation of easily confused phonemes (usually, syllabic n/ん from a
following naked vowel or semivowel). For example, the name じゅんいちろう, is
written with the kana characters ju-n-i-chi-ro-u, and romanized as
Jun'ichirō in Revised Hepburn. Without the apostrophe, it would not be
possible to distinguish this correct reading from the incorrect
ju-ni-chi-ro-u. This system is widely used in Japan and among foreign
students and academics.”