Why do all Japanese verbs in dictionary form end with -u? What is the reason, linguistically?
Japanese dictionaries lemmatize (choose the headword form of) conjugating words using the so-called 終止形【しゅうしけい】 or terminal / terminative / predicative form (the form used to end a sentence in a plain indicative mood). For the entirety of the history of the Japanese language (where we have text written in Japanese), the shūshikei of almost all verbs has ended in -u.
(The one notable exception in older texts is the copula ("to be" verb) 有【あ】り and its derivatives. That said, at least some linguists theorize that this is an exceptional form that evolved from an older shūshikei of aru. It is generally the case that the most exceptional terms in any language are the ones that are used the most: consider the oddities of English "to be" or "to go", and the stranger conjugation forms for these very-common verbs. Within modern Japanese, consider the oddities of how だ conjugates.)
Did they start off with some not ending with -u, then evolving to end with -u by analogy?
As for why the terminal forms for Japanese verbs all end in -u, the reasons for this are not clear. I recall reading some commentary somewhere that certain linguists theorize that ''-u'' may have been an ancient verb-forming suffix. In modern Japanese, we do see that ''-ru'' can be used in a similar fashion to form some slang-y neologisms like ググる (from グーグル) or スタバる (from スターバックス). But why was -u ultimately used for the shūshikei verb ending instead of something else? Who knows? I've never read any theories about this. It may have to do with unknowables, such as cultural perceptions of sound textures over time, or the ease of pronunciation of specific sounds in specific sequences, etc. etc.
Looking at European languages, we could ask similar "why" questions -- why do all Spanish infinitives (the dictionary form) end in -r? Why do all German infinitives end in -en? Why do all Hungarian infinitives end in -ni? Why do all English infinitives start with to?
Exploring "why" questions about Japanese
⇒ When it comes to languages, certain kinds of "why" questions are very difficult to answer. The younger the records of the language, the more difficult these questions become. Consider that Hungarian is only reliably recorded since the 900s, while texts in Japanese only go back to the 700s. English is a bit of a mongrel, but even so, we have textual evidence since the mid-600s. Spanish, meanwhile, is attested in some form or another all the way back into Latin, and Latin itself has literature dating from the 200s BC.
Digging into "why" questions becomes a bit easier if the given language has known relatives, and we can explore comparisons and connections between the languages and reconstruct more of the pre-historical (not-yet-recorded) proto-language, based on what we know and can derive about how sounds and meanings change over time. For English, Spanish (and Latin before that), German, and Hungarian, there are numerous known relatives, and we can compare these to derive more information about earlier stages, stretching back before our textual horizons. For Japanese, however, there isn't quite so much -- we've got a good number of Japanese dialects, and Okinawan and the other Ryūkyūan languages, but no confirmed relatives outside of the Japanese archipelago.
There is a good bit of recent and current ongoing research comparing the various Japonic languages (mainstream Japanese, Japanese dialects, and the Ryūkyūan languages). I've noticed that some volunteers are creating and editing various Proto-Japonic entries at Wiktionary; that said, I myself haven't read enough of the underlying academic research to tell how accurate these are.
If you're interested in the historical development of Japanese, and current theories about the likely pre-history of Japanese, here are a few authors you might find informative:
- Alexander Vovin
- Bjarke Frellesvig
- J. Marshall Unger
- Samuel Martin
- Marc Miyake
I'm sure there are others; this list is just intended as a starting point.