Phone etiquette is one of those things I still struggle with at times, often fumbling because I'm not sure what expression I'm supposed to use at a given time. 失礼します at the end of a phone call is one of these. Technically if the other person is calling me, they're the ones "intruding," but if the conversation ends in such a way that I'm the one who is ending the conversation, it seems like I should say it even though the other person called me. I've had some weird moments where the other person said it and I just said it in return because I'm so used to saying "bye" "bye" in English.

Does the person calling always say it, or is it wiggly based on the feel of the conversation, or am I wrong and in fact both people say it? I don't usually get to hear both ends of the conversation when I hear other people on the phone.