As far as I know, おはいりください might be used to invite someone in when they are visiting your home. I was wondering if this would be used by friends who have visited each other's houses a few times before. Also, would the guest reply しつれいします, おじゃまします, or ありがとう, or is there some other more casual way to say this?
いらっしゃい and おじゃまします are the most common and important phrases for this scene. I myself always say おじゃまします when I enter friends' houses and いらっしゃい when I invite friends. While as a host you might have more freedom than always saying いらっしゃい, the level of politeness that おじゃまします, as a guest, conveys is almost always expected even between friends in my opinion.
Here is a typical conversation:
(short conversation at Genkan).
Host: さあ，あがって (or どうぞどうぞ)
(put off shoes)
Guest: おじゃまします (with the first step to the main part of the house)
おはいりください may perhaps be used after いらっしゃい but どうぞおあがりください, あがってください or あがって sound more natural than おはいりください. This is probably because the main part of a traditional Japanese house is higher than genkan and so the first step is stepping up.
おじゃまします is like asking for permission: "I'm just about being in your private area!"
Another good phrase to remember is おじゃましてます, which is like "I'm already in your private area!" Imagine a situation in which you are already at a friend's home, enjoying the lunch with him/her and his/her father has just come home. You might want to say おじゃましてます instead of おじゃまします