First of all, I appreciate that this is a difficult subject, not least because when trying to research it, in English, the terms "honorifics" "respectfulness" "formality" "politeness" etc. often get used differently by different people. I'll try to stick to your terminology, but apologies if I slip up!
Your understanding is correct: you can't combine "respectful" and "humble" forms grammatically, as they make reference to another's or one's own (/one's in-group's) actions, respectively. They are thus at opposite ends of the same "honorific" spectrum of {respectful 尊敬語 OR neutral OR humble 謙譲語}, in that in being either respectful or humble, you are creating a difference in 'level' between you and the other party. In being respectful, you directly exalt the other party; in being humble, you indirectly exalt the other party, through deprecating yourself.
The important thing to remember is the {polite です・ます OR non-polite だ・る} choice primarily depends on your relationship to the listener (ignoring the idea of switching between polite and non-polite within a conversation... see Question about switching formality in a conversation for more detail...).
On the other hand, the {respectful 尊敬語 OR neutral OR humble 謙譲語} choice is about the relationship between actors in the sentence. Of course, they are often the same, but not always (e.g. speaking to your friend about a teacher, you might choose to index your 'respect' for the teacher by using 尊敬語 or 謙譲語, but you presumably would speak in non-polite speech to your friend).
Grammatically speaking, you therefore have two spectra:
The {formal OR informal} choice, as broccoli forest notes, is the "odd one out", in the sense it is not determined by grammar, but stylistic decisions appropriate for the medium/context in which you are communicating, including word choice. I see it as the difference between すごく and 大変, このあいだ and 先日, どうですか and いかがですか, いいです and 結構です (when refusing something), etc.
The interesting case, I think, is choosing to use an irregular keigo verb, which would impart e.g. formality as well as respect, say 食べる vs. 召し上がる. So that begs the question: is 召し上がる more formal than other acceptable regular keigo forms 食べられる or お食べになる? I would say the answer is: yes.
Incidentally, I also understand that the usage of お食べになる is increasing amongst younger people, which is possibly a reflection of those younger people be(com)ing "less formal" than their older counterparts; or, perhaps they want make a statement that they aren't inclined to guard or protect formal language.
In any case, formality is a much more fuzzy concept than politeness and respectfulness, given formality is not strictly determined by grammar.
