/h/ is from original *p
The Japanese fricative /h/ is reconstructed as coming from earlier *p (a voiceless labial stop; "labial" is a phonetic term for consonants pronounced with the lips). You can read more about this in books or articles about the phonology and phonetics of Old Japanese; Wikipedia mentions it in the "Old Japanese" article.
The development is as follows.
First, original *p developed to a voiceless labial fricative [ɸ].
Between vowels, [ɸ] eventually was voiced, which caused it to merge with the voiced labial approximant /w/. And /w/ in turn was eventually lost before any vowel except for /a/. There is an isolated remnant of those sound changes in the irregular modern spellings of the particles は /wa/ and へ /e/, which use the hiragana for "ha" and "he". Pre-reform spellings have many more examples of "h"-series kana being used in the middle of words that aren't pronounced with /h/ in modern Japanese. For more details on this, see Boaz Yaniv's answer to the following question: Why are the particles “は” (ha⇒wa), “へ” (he⇒e), and “を” (wo⇒o) not spelled phonetically?
In word-initial position, the voiceless fricative [ɸ] remained, but it ended up losing its labial quality before any vowel except for /ɯ/, causing it to develop further to [h] (some modern Japanese speakers have [h] even before /ɯ/). It's not too uncommon for fricatives to lose their original place articulation and become [h]; this sound change is called "debuccalization" and applied in a similar manner in Spanish, turning Latin f in most positions to h (which then ended up being lost completely, so Latin factum corresponds to Spanish hecho [ˈet͡ʃo]). We have evidence for the pronunciation [ɸ] originally being used before all vowels in the form of the Nippo Jisho, a Japanese–Portuguese dictionary from 1603 that uses the spellings "fa", "fi", "fu", "fe", and "fo" to represent the Japanese pronunciation.
Words with /p/ in modern Japanese
The plosive /p/ came to be a separate phoneme from /h/ through certain processes that I don't understand very well. My understanding is that modern Japanese words with /p/ usually fall into one of the following three categories:
onomatopoeic or mimetic words, where the plosive sound [p] seems to have been used for some reason (maybe it somehow was felt to represent the concept better). Example: ぴかぴか pikapika, related to the verb ひかる hikaru (光る).
words where [p] is part of a geminate cluster [p.p] /Qp/. Old Japanese did not have geminate clusters; they only came to exist in Middle Japanese. Syllables ending in consonants were introduced in words from Chinese, and in certain positions, the syllable-final consonant combined with a following consonant to become a geminate. For example, 一匹 is pronounced いっぴき ippiki because the first part, 一, comes from Chinese and was originally pronounced as a single syllable ending in the consonant *t. The combination *tp turned into the geminate [p.p].
In the Middle Japanese period, geminates also developed in certain native Japanese words from phonological processes like contraction or lengthening. I don't know enough to give a summary, but one example is 河童 (かっぱ kappa) which Wiktionary says is contracted from an original form kapawarapa (→ /kawawappa/ → /kappa/).
words where [p] is part of a moraic nasal + voiceless stop cluster [m.p] /Np/. Voiceless stops don't occur in this position in native Japanese words (Rice 126). But they can occur in words that come from Chinese, such as 散歩, which is pronounced さんぽ sanpo (Rice 128). Kumagai lists many more examples in (1c) that show that [p] in this context corresponds to [h] elsewhere.
As you can see, words pronounced with /p/ in any of these categories often are related to words pronounced with /h/, as with ぴかぴか pikapika and ひかる hikaru, いっぴき ippiki and ひき hiki, さんぽ sampo (散歩) and ほ ho (歩). So there is some benefit to writing syllables starting with /p/ with modified versions of the kana for syllables starting with /h/.
How /b/ is related
Because of the historical development of /h/ from *p, the sound /b/ acts as the voiced counterpart to /h/ for the purposes of certain sound changes.
For example, it is possible for /h/ to correspond to /b/ after a moraic nasal. Apparently, this is usual for native Japanese words: Kumagai 2017 gives the example hun 'to step on' + haru 'stretch' -> hunbaru 'stand firm' (1e, page 4). Martin 2004 indicates that hun here is a contraction of humi (p. 401), which seems to be a form of the verb ふむ (踏む).
The sound /h/ also turns to /b/ in the context of rendaku (voicing of the initial consonant of a word when it is the final element of a compound). The historical development of rendaku seems to be a bit obscure, but one hypothesis is that it originally involved the absorption of a preceding nasal consonant originally derived from the particle no, or possibly in some cases ni (Vance 338). However, in modern standard Japanese, rendaku can be interpreted as just a morpho-phonological rule that turns voiceless consonants into the corresponding voiced ones in compounds (subject to certain complicated restrictions).
Kumagai, Gakuji. 2017. "Testing the OCP-labial effect on Japanese rendaku"
Martin, Samuel E. 2004. A Reference Grammar of Japanese.
Rice, Keren. 1996. "Japanese NC clusters revisited: is postnasal voicing redundant?"
Vance, Timothy J. "On the Origin of Voicing Alteration in Japanese Consonants". Journal of the American Oriental Society
Vol. 102, No. 2 (Apr. - Jun., 1982), pp. 333-341.