I think that the sequence beginning
一番・二番・三番 corresponds roughly to number one, number two, number three. Each word in this sequence is formed from the combination of a numeral (
一・二・三) with a counter meaning number (
You can take this list and add the ordinal suffix
目 to each word, producing the list
一番目・二番目・三番目, corresponding roughly to first, second, third.
What may be confusing here is that an adverb has been derived from
一番, but not from the other words formed with
番. As a result,
一番 can represent either number one or first/most, but
三番 mean number two and number three. You can see this if you compare dictionary entries; the entries for 一番 in 大辞林 and 大辞泉 list it as both a noun and an adverb, while the corresponding entries for 二番 don't list a derived adverbial sense.
So, if you're talking about the tallest, second tallest, and third tallest buildings, I think the list naturally looks like this:
And your example
二番速い人 is, I think, ungrammatical.
As for whether
〜目 is a counter, I don't think so. It doesn't combine directly with numbers like counters do. Instead, it's a suffix that attaches to words that consist of a number and a counter. If we check 大辞林, we find the following:
二 （ 接尾 ）
① 数を表す語に付いて，順序を表す。 「一つ目」 「三番目」
You'll notice that it says this is a 接尾語 (suffix). I've bolded the actual definition, which I'll translate here as "attaches to words that express number, and expresses ordinality."
You'll note the definition includes an example where
目 attaches to a word with
つ, as well as another example where it attaches to a word with
目 attached, both
一番目・二番目・三番目 express first, second, third, but of course the
つ list can only count to nine, while the
番 list can be used to express arbitrary positions in a list.