If one were to look at the 万葉仮名 article on Wikipedia, they would find that no specific marks were added to represent the bunch, that is, the ザダガバ 行, with there having been different representations given to the sounds. However, excepting those kanji, there does not seem to have been anything that could represent those sounds per se.
Why would none of them ever be reduced to kana? Was there some factor that led to this preference in using diacritic marks to distinguish them?