Part of the reason why this is a little tough to explain is that there are actually several historical "alphabetical" orders, so to speak. The two that are probably the most recognized today are the 五十音｛ごじゅうおん｝and the いろは systems of organizing the alphabet.
In most cases, you'll encounter the 五十音 system, such as when browsing a dictionary, or shopping for titles at a CD, movie, or bookstore. As you mentioned before, the consonant categories are grouped together, so for example, you might just see inserts that are あ か さ た な は ま and so on. Going from here, you should know that the 五十音 system typically orders words based on the first character's sound, and so, for example, items that begin with the sounds あいうえお are all under the あ tab, and in that order. Then, あい will be before うた, and うた is before えき in a dictionary.
Aside from this, oftentimes, you must consider 濁点{だくてん} ( ゛) and 半濁点{はんだくてん} ( ゜) usage in words. This means that words like へた and べんきょう and ペン might all be in the same general section of the dictionary (because they are all starting with the character へ). Most dictionaries will have all the non-濁点 words first, followed by the 濁点-starting words, followed by the 半濁点-starting words, before moving to the next kana. So, は then ば then ぱ then ひ, for example.
In words with「っ」in them, these might be placed after wherever the つ is located (i.e. さつえい before さっき) or might be all the way at the end of the kana section (i.e. さんぽ before さっき), but not before the 濁点 and 半濁点 variations (i.e. さっき before ざっし). This varies to some degree depending on your source material.
Many beginner/learner dictionaries will have a word lookup based on the learner's base alphabet (in English this would be ABC order), and is useful until you become used to the standard 五十音 grouping. In addition, once you start your journey into reading 漢字｛かんじ｝ you'll find it much easier to locate the word you're looking for in a Japanese-language dictionary.
