To tell the truth, this question was so unexpected for me who am not familiar with colloquial English that I couldn't figure out what it means if it weren't for an English speaker's guidance. Maybe I still don't grasp what you're asking, but there are so many reasons it couldn't be with ン.
- "Roman" in Japanese
In English, Roman is an adjective derives from Rome. Japanese has a word ロマン too, but it's from French, where it means English romance. Moreover, now we use it especially for "an emotional attraction or aura belonging to an especially heroic era, adventure, or activity" (Merriam-Webster) sense. This word is associated with "romanticism" and has little to do with the city Rome, so it can be said that English Roman and Japanese ロマン are false friends you cannot use them interchangeably for Rome Roman.
Strictly speaking, there's another word ローマン from English Roman, but usage of this one is confined to a few technical jargon like ローマン体 "roman type" (↔ 𝕱𝖗𝖆𝖐𝖙𝖚𝖗 or italic) in typography or グレコローマンスタイル in wrestling.
- How did Japanese know Rome and its alphabet
It appears to be Christian missionaries that introduced the knowledge about the city. 日本国語大辞典 cites its first appearance in 1605.
*妙貞問答〔1605〕下「イタリヤの内、朗磨と云都に本寺を立て」
"they built the cathedral in a city in Italy, called Rome, ..."
The author of this tract is educated with Latin in a official Jesuit seminary in Japan, so perhaps it could be direct transliteration from medieval Latin Rōma. Anyway, the Latin, Italian, Spanish and Portuguese pronunciations of this city had little difference at that time.
The name of letters they used was, however, not known besides in a very vague term throughout the Edo period: "the horizontal writing".
*俚言集覧〔1797頃〕「横文字。西洋の国四十二国みなよこ文字のかななりといへり。左より書はじめ右におはる。草のつるの如く書てよみがたし」
"Sideways letters: All 42 Western countries are said to write in sideways phonetic letters. It starts in left and ends in right. Written like plant vines that hard to read."
The solid term ローマ字 wasn't attested until Meiji. Again the first appearance is:
*羅馬字早学び〔1885〕〈矢田部良吉〉一「羅馬字即ち横文字は、二十六字にて」
"Roman alphabet i.e. the sideways letters consists of 26 letters..."
Originally ローマ字 stood for general notion of "Roman alphabet", parallel to 漢字 ("Chinese character") or 梵字 ("Siddham script"). The etymology is quite transparent, from ローマ "Rome" + 字 "writing".
Later, its usage was gradually limited within the discussion on Japanese orthography reform, eventually became a word means "romanization" or more specifically "Japanese romanization", but that wasn't the original meaning. Today we usually call the writing system itself as ラテン文字 ("Latin script") or just アルファベット.
- Why not adjective?
If someone's still not convinced, then it's grammar time. It is true that European languages, including English, use adjective forms when describing an idea "something from somewhere". But in Japanese, you have to use bare place name in this situation, especially when the entire train of words represent a monolithic idea.
In fact, many English words are created in this way too, like "stun gun", "ice cream" or "summer time". But when it comes to locations or personal names, we revive adjectives and say "Central European Summer Time". What Japanese does is just to remove "-al" and "-an"s completely, and we get 中央ヨーロッパ夏時間 (lit. "center Europe summer time"). Similarly, we only say ローマ帝国 ("Roman Empire"), ローマ市民 ("Roman citizen"), ローマ文明 ("Roman civilization") or ローマ教会 ("Roman church"), but never *ローマン帝国 or *ローマン教会 (of course not *ローマ的帝国 etc.) otherwise it'll obscure the relationship between its base word ローマ.
It's not just because we've traditionally called them so, but works as well in new words: an imaginary extraterrestrial intelligent development in a dwarf planet, "Cererian civilization", would likewise be translated as ケレス文明. Thus, if we call the city ローマ, then the Roman alphabet would be automatically called ローマ字.
Using Japanese adjectives (though they don't work exactly as European adjectives) or genitives to qualify nouns is not prohibited, but it breaks the integrity of idea all-noun compounds have. ローマの帝国 would only indicate "a Roman empire" (cf. ローマの休日 Roman Holiday) but not "the Roman Empire" (which existed in 27 BC – 476 AD).