Generally when you see romanization of Japanese it is in the Hepburn system; however, I recently came across the Nihon-shiki system which seems to be preferable. Why is it that the Hepburn system caught on as opposed to the Nihon-shiki system?
In answer to this question, I'll give my personal understanding, although I can't fully substantiate it.
The superiority of kunrei-siki
Kunrei-siki is often felt to be a "better" system than Hepburn in representing Japanese. That is because kunrei-shiki mirrors kana usage more closely than Hepburn. This is noticeable in the さ row, the た row, the は row, and those syllables using ゃ, ゅ, and ょ.
さ し す せ そ
sa si su se so (kunrei-siki)
sa shi su se so (Hepburn)
た ち つ て と
ta ti tu te to (kunrei-siki)
ta chi tsu te to (Hepburn)
は ひ ふ へ ほ
ha hi hu he ho (kunrei-siki)
ha hi fu he ho (Hepburn)
じゃ じゅ じょ
zya zyu zyo (kunrei-siki)
ja ju jo (Hepburn)
Kunrei-siki preserves the regularity of the syllabic system, whereas Hepburn obscures it. By following the kana syllabaries, kunrei-siki arguably comes closer to the intuitive Japanese perception of their own language. This becomes particularly apparent in verb conjugations like:
まつ まちます まてば
matu matimasu mateba (kunrei-siki)
ma*tsu* ma*chi*masu mateba (Hepburn)
Whereas Hepburn obscures the regularity of the conjugation of まつ, kunrei-siki shows it quite clearly. For people who prize the insight into the underlying regularity of the phonetic system, kunrei-siki is felt to be a more elegant system than Hepburn. From that point of view, Hepburn may be felt to be a bastard system that doesn't respect the phonology of Japanese.
Why Hepburn has largely supplanted kunrei-siki in most situations
Despite kunrei-siki's apparent superiority, Hepburn is definitely the more popular system of romanisation (although you might be surprised at how many Japanese mix systems in using romanisations). I would suggest several reasons for the popularity of Hepburn.
First, the two systems are rather different in their target audience. Kunrei-siki is a system that is easily understood by the Japanese, and could be perceived as a plausible writing system for the Japanese language. The days when people seriously suggested abandoning characters and kana are probably long past, but if it came to writing Japanese in romanisation, this would probably be the logical choice.
Hepburn has no such pretensions. It is a system of spelling Japanese for the benefit of non-speakers and nothing more. It is meant to be convenient for foreigners to read, and it does that job well.
Given that Japanese does fine with characters and kana for most purposes, it is probably quite natural that romanisation now largely serves as a script for foreigners. There is little need for the Japanese to resort to romaji in their own language.
My understanding is that this also represents the historical situation. Before World War II, use of kunrei-siki appears to have been more widespread. I believe that this was because it was an era of Japanese nationalism (thus, romanisation is for the Japanese, not for foreigners). (It was also a time when ideas about script reform (reform of characters and kana) were still around. People who proposed kunrei-siki did so because they felt it was a good system for writing Japanese. I should point out that nationalism and ideas of script reform didn't necessarily go together!)
After the war the situation changed completely. With the United States in control of Japan, romanisation took on the role of showing foreigners how Japanese was pronounced. The kunrei-siki system did not come off very well for this purpose. Jack Seward in his popular book Japanese in Action described the ludicrous results of using kunrei-siki in post-war Japan, when the 第一ホテル, written 'Dai-iti Hoteru', was inevitably mangled by American servicemen into 'Dai Itty Hotel' and eventually 'Dai Itty-Bitty Hotel'. (I'm quoting this example from memory; I don't have the actual book with me).
That is probably a very practical reason why Hepburn and not kunrei-siki became the most popular romanisation in Japan.
A third reason, I would suggest, is the fact that English has come to be widely studied in Japan and is regarded as a kind of international standard. A system of writing, like Hepburn, that follows English is thus probably regarded as more 'international' and more prestigious by Japanese speakers. I would even suspect that kunrei-siki is regarded as a little ださい by the Japanese.
The superiority of kunrei-siki (revisited)
During the early 20th century, kunrei-siki could probably be described as a better fit for the Japanese language. Those, after all, were the days when words like 'team' and 'diesel' were imported into Japanese as チーム and ヂーゼル.
But the language has moved on since then, and the phonology of Japanese has been penetrated by foreign sounds that didn't exist before. For instance, ヂーゼル is now usually written and pronounced ディーゼル. In other words, the sound 'di' has now entered Japanese where once there was only 'ji'. ティ (as in パーティー) and トゥ (as in トゥトゥ) are now familiar to ordinary Japanese, alongside the old native sounds ち and つ. Katakana struggles to accommodate these pronunciations, coming up with devices like ティ, トォ, フィ, ヴィ etc. to represent them.
With these gradual changes in the sound system, it's now not quite so self-evident that kunrei-siki is the superior system for representing Japanese. Unlike Hepburn, which can easily distinguish 'chi' and 'ti', kunrei-siki has no obvious way of distinguishing these two sounds.
You are looking at this from the perspective of someone with a reasonable knowledge of Japanese, but romaji is wider in application than that. It is often read by people who have no knowledge of the language, perhaps not even a desire to learn it. In fact, those people may be the main readers of romaji.
The advantage of Hepburn over Nihon-shiki is largely that Hepburn is more consistent and intuitive in how it maps letters to pronunciations, particularly for English speakers. Someone with absolutely no knowledge of Japanese will be able to produce a closer approximation to the Japanese original when presented with Hepburn over Nihon-shiki, and it will take them less time to become familiar with the system. There are fewer pronunciation quirks.
For instance, take ち and つ.
In Nihon-shiki, this would be rendered "ti" and "tu". A person with no knowledge of Nihon-shiki or kana may well think that these are pronounced in the same way as "ta", "te", and "to". On the other hand, Hepburn is very explicit in pointing out that ち and つ use a different consonant sound, with its romanizations "chi" and "tsu".
Similarly, not many English speakers would think to produce anything like "sh" or "ch" from the spelling "syatyô".
Neither is superior than the other. They represent different stages of the phonological derivation. Any natural spoken language has a set of phonological rules. In case of Japanese, the rules include those that change /s/ into [sh], /t/ into [ch] or [ts], /p/ into [h], /h/ into [f], /h/ into [w], or delete /w/ under respective conditions. If you are interested in describing the inflection of a Japanese word, it is more convenient to describe the underlying representation:
This is close to the Japanese method (nihon-shiki), but has some differences: for example, wu, wi, we would be described u, i, e in Japanese method. In this respect, the Japanese method is actually less consistent/precise than the Hepburn system (described below), contrary to what you wrote.
Within the derivation, these forms undergo phonological rules, which change them to their surface representation:
This is the form that is pronounced, and if one wants to transcribe the pronunciation, describing as above would be convenient. This is close to the Hepburn system.
Depending on your needs, you should chose the appropriate one. In the majority of cases when romanization is used, the pronunciation matters, and Hepburn system would be convenient.
It's not. Kunrei-siki (the updated version of Nihon-siki) is still widely used in Japan, as are non-standard romanisations and mixtures of different systems. They're taught in Japanese education systems and used by companies and government institutions, you'll see them on signs a lot. It was designed in Japan to make logical sense for people already familiar with hiragana.
(Revised) Hepburn romanisation is more widely used for international media, being almost completely the dominant system for learning Japanese and official transliterations intended for foreign readers. They reason for this is that it leads to more faithful pronunciations by English speakers and other western readers (learning hiragana and Japanese phonetics is still recommended). It was designed by an American for this purpose and is based on how latin characters are read in English or Italian.
"si", "tu" and "ti" are the same as "shi", "tsu", and "chi" for Japanese readers (some even prefer other systems) but English speakers will pronounce them differently. For example, if you tell a tourist to meet you at "Shimbashi", it will be more difficult for them to find it written on a map as "Shinbasi". It's to aid newcomers to Japan who cannot read Japanese. For this reason, Hepburn has become the dominant romanisation.
Hepburn is preferable for foreign readers and is more intuitive for them. Other systems were designed with Japanese readers in mind (although studying latin alphabet and English is common so this is no longer as important).
I'm going to give a very simple answer: convention.
Korean and Chinese had systems like Hepburn, based on foreign phonology, but South Korea and China more or less decreed that a system based on native phonology rather than English approximation was to be used, and it worked in that case.
The Japanese government is softly pushing Kunrei, but not putting it's foot down in the same way the Chinese -, and Korean governments did and demanded it, so it never happened — most likely, if they had been just as aggressive with demanding it diplomatically as the Chinese government was, we would be using Kunrei now everywhere; I don't really believe it has much to do with what is "best". Pīnyīn has x, q, j and a variety of consonants that in no way indicate the pronunciation to English speakers, and it must be learned, but it's still used everywhere, because it doesn't really matter: the truth of the matter is that English speakers will, with no prior knowledge of Japanese, probably when reading out Hepburn still sound largely unintelligible, put the stress wrong, pronounce the vowels wrongly, and might read the "y" in things like "kyo" as a vowel, which is how "Tokyo" is generally pronounced in English. What matters is that it creates something that can be recognized. "毛泽东" is not something easily recognizable for those unaccustomed to Han-characters, but "Máo Zédōng" is, even if the tones be completely ignored, even if the z be mispronounced like an English /z/, even if it would be completely unintelligible to a speaker of Mandarin when read out loud.