The very literal meaning of しわ寄せ is "gathered wrinkles", although only a few people use this term in this literal sense (Shirring is sometimes called しわ寄せ(加工)).
To understand しわ寄せ, suppose you are ironing a dress. It's difficult to iron out the wrinkles perfectly; you iron somewhere, and a new wrinkle appears somewhere else. That's the idea of しわ寄せ; you strain yourself and do good things somewhere, but as a result, something bad happens somewhere else.
Now let's take a look at the lyrics (whole lyrics here). I would say the lyrics of this song are vague and abstract overall, but the first part is relatively easy to understand:
The person ("ぼく") has behaved like a gentle person everywhere else, but that must have been stressful to him. In front of "きみ", he couldn't behave kindly any more (雑に here means "not courteously", "crudely"). So this しわよせ is used in the most basic and common sense of the word. Can you see, throughout the lyrics, that ぼく has been taking a bit harsh attitude toward きみ?
EDIT: That said, しわよせ is not commonly used for something happening only in the internals of one person. This しわよせで may be replaced with 反動で (as a counter-reaction/rebound), although I'm not sure if these are suitable for lyrics.