In modern あいうえお, the column of わ is わいうえを, but it used to be わゐうゑを. The pronunciation of these were more like "we" and "way" as opposed to the normal い and え sound. I didn't know this myself, but apparently these old forms have disappeared as recent as 1946.
ゝ is a short-hand that means "repeating the previous letter", such as 学問のすゝめ (学問のすすめ). As you probably know, dakuten is composable decoration of a letter, and it can also decorate ゝ.
ヶ is an interesting one. I think its origin is at least in part katakana ケ, but I can't think of other katakanas that do this. Aside from the usage in 藤ヶ谷, which means 藤の丘, you also use ヶ to count objects, like 10ヶ.