In case I am talking with someone using the informal form but we are not close friends yet (perhaps a close acquaintance or someone I'm chatting with online) and I want to ask him a personal question (what is his occupation, something about his family ,etc.), should I proceed with the informal form or should I switch to polite (or even honorific) language to ask him the question?
This heavily depends on your character, your age, the relationship between you and him, and how personal the information you're trying to get is. In general, using honorific forms is probably an overkill, but temporarily switching to the です/ます style will not harm.
Personally, I am relatively slow to start dropping です/ます, and I am unlikely to talk with someone using the informal language if I don't even know his occupation.
It's not that uncommon to be talking simultaneously to two or more people using different levels of familiarity/politeness, but for the same person, using the same level throughout a conversation is the norm.
Switching between levels of formality/humility in the same conversation with the same person is just not necessary. Even if the kinds of things you wish to talk about change from "general" to "personal", whatever form of speech the two of you began with, you can continue with. That said, if you choose to switch to using the polite "masu" form of verbs rather than the root form, I don't think that's bad. Your conversation partner might misunderstand and think that you suddenly feel less comfortable with them, but I doubt any real conflicts will occur.
I have always had a problem with "keigo". I get easily confused by humble vs polite vs honorific speech... though I have been told over and over that as a foreigner, it is fine to simply use the polite form with everyone all the time, until you consider them to be close friends (and as long as you are outside of work... at work always use polite speech... a rule I break far too often, but get away with because my schools are very kind....)
There have been times when I switch back and forth between levels of politeness, and as a foreigner these are either ignored as "he just doesn't know" situations, or it gets a small laugh or a humorous comment.
doing a google search for "when to use keigo" might lead you to useful websites.
How formal/polite you are (or should be) depends on the nature of your relationship. If they’re older, in a more senior position, or you don’t know them very well, you use some degree of formality. This should not change during the course of a conversation.
Mixing formal and plain forms is clumsy, although it will be forgiven for non-native speakers unless you’re in a position where you should really know better (such as working in a Japanese company). It’s best to be consistent and use the same forms throughout a conversation (or letter or email) if you can. If you are a non-native speaker, it is unlikely that you'll be a situation where you are expected to use honorifics or 敬語. Using the polite ます forms should be acceptable with friends or acquaintances.
If this is someone you meet regularly, you should use the same level of politeness for an entire meeting. If you are unsure if it’s appropriate, it’s probably best to keep with polite forms and let them initiate if they want to speak as peers or friends. In Japanese culture, it’s not uncommon to maintain formalities with friends, rather than switching to a “first name basis”. This does not mean they don’t regard you as a friend, they may reserve using plain language for people they’re really close with such as family and childhood friends.
If you are learning Japanese and they’re a friend of yours they should be willing to help you if you ask them nicely. It would be better to ask what they’re comfortable with rather than assuming as it depends a lot on how they view your relationship (and how familiar they are with foreign customs). Some will feel obliged to use formal speech when speaking Japanese which is why they may prefer to speak English since they can speak “more freely” without worrying about politeness or hierarchy.
If it’s a friend, you can reciprocate if they drop formalities but I’d wait for them to do so first (or ask them). If it’s a colleague, your boss, or someone older than you, it’s more complicated: they may also use plain forms as you’re their “subordinate” and they’ll still expect you to use polite forms with them. Since many Japanese people meet their friends through the workplace, it can be confusing since they’ll speak differently when talking as friends or coworkers.