し損なう is a compound verb, consisting of the continuative form (連用形) of the verb する + the verb 損なう, meaning "fail to do (correctly)" "to miss (the chance to do~~)", e.g.
計算をし損なう "make an error in calculation"
留学をし損なう "missed the chance to go study abroad"
(You can also say 計算しそこなう / 留学しそこなう, leaving out を. In this case, you could see them as compound verbs consisting of "する-verb 「計算する/留学する」+ verb 「損なう」", rather than "noun「計算/留学」+ compound verb「し損なう」".)
損なう is a verb meaning "to ruin/spoil/hurt", e.g.
健康を損なう "ruin/lose one's health"
友情を損なう "lose friendship"
And 損なう can be attached to another verb to make a compound verb, such as し損なう, and:
乗りそこなう "fail to catch → miss (a train, bus, etc.)".
撃ち損なう "miss the mark/shot"
I don't think we say 損ないする.
支払う is a verb (not a compound verb する + 払う), meaning "to pay (money/bill)".
[Edit: As in @broccoliforest's comment, 支払う is derived from し (continuative form of verb する) + 払う (verb "pay").]
払う can be used for ほこりを払う "to brush/dust", 注意を払う "to pay (attention)", as well as for "to pay (money/bill)". I think 払う is more casual and less formal, and 支払う is more used in a formal/business situation.
I don't think we say 払いする, but you can say 支払いをする: noun 「支払い」(payment) + object particle 「を」 + verb 「する」.
[Edit: You don't use just 「損ない + verb する」「払い + verb する」(turning 損ない/払い into a する-verb), but if you're referring to phrases like 「損ないはする」「払いはする」, you can say like 「損ない+は+しない」「払い+は+しない」「損ない+は+する(が～)」「払い+は+する(が～)」etc. and it's a different usage than simply turning a noun into a する-verb.)
So, my answer to your final question
can I do this with any verb? is, no, you can not always attach し (continuative form of する) to the top of a verb, and not all nouns can be used for a する-verb.