Are you familiar with hito, futa, mi, yo, itsu, mu, nana, ya, kokono? They appear in the counter words for objects (eg hitotsu, futatsu), and in some counter words for people (hitori, futari, ..., yonin), and in some of the days of the month (the second day of the month being futsuka).
Edit: My understanding is that ichi, ni, san, shi, ... are Chinese words for the numbers, and that hito, futa, mi, yo, itsu, are the Japanese words for the numbers. Sometimes the Japanese use Chinese words for numbers in the counters, and sometimes they use the Japanese words. They tend to use the Japanese words more often when it comes to numbers containing 4 or 7, because of superstition. I think (but have no evidence for this) that hito and futa in hitori and futari because, well, one person and two people are special, and they don't want to replace their Japanese words for it with Chinese words.
And then there's the more or less constant part at the end of a counter. Sometimes it's constant, such as "ji" in hours of the day, and sometimes it varies slightly because of pronunciation, such as "fun" and "pun" in the counter words for minutes.
Once I realized that there was a number part and a more or less constant part in counter words, I found them easier to memorize, and a bit more logical.