I think 「さすがに」 generally means something like "Generally acknowledging a fact, but still has more to say regarding that fact in this specific situation". But I recently encountered a usage of this phrase, to which I don't quite think the above explanation is applicable.
To provide the context first, there was a male and a girl making a conversation. The male stood on a stone which was much higher than the place where the girl stood. So in the middle of their conversation, the girl said the following to the male:
Then the reaction of the male following this remark is:
I think I get the rough meaning of these paragraphs, that is the male (whose name is 「静馬」) accepted the girl's protest and climbed down from the stone so that the two were on the same footing. But at the same time the male came up with a retort, which he decided was rather indecent. At last he refrained from spitting it out. But I don't really understand how「さすがに」 could fit into this paragraph…