I know that the wave dash
〜 can be used in place of the long vowel marker
ー sometimes, but I'm not exactly sure what the differences are.
I think the difference between the is really captured by their appearance alone. As you mentioned, ～ sometimes has a sort of wavy 'tremolo' type feel to it, or at least that's the image evoked by looking at it. I'm not sure how many times you would actually fluctuate the pitch like that in an actual reading, though. I usually associate it with a kind of lightheartedness/plyfulness/nonseriousness. This kind of ties into the second one you mentioned, with it being "cute." If something is said in a (supposedly) cute way then you can imagine it being potentially tremolo'd, maybe more childish or stereotypically feminine. For example, can you see the difference between 行きたくなーい and 行きたくな～い？ The first one sounds more bored and disinterested while the second feels a little less めんどくさい. Or at least that's the feeling I get! Similarly you get the difference between あ～～～ and あー. One is like "あ～～いや～～～～" and one is like "あー、そっか."
Another example is the vending machines here. For cold drinks the machine says つめた～い and the hot have あったか～い. It just makes it a little warmer and more inviting.
Also to get technical about it you wouldn't use ~ in katakana words, so no ラ～メン or ウ～ロン茶
I think that the long vowel marker is used for long vowels, like in バーモント, アーモンド, etc. As though usually used with カタカナ, sometimes it is used with ひらがな as well. For example
- in ひらがな mimeses like びゅーびゅー, where the うs in びゅうびゅう are evidently just "long vowel markers"
- in children's books, which write everything in ひらがな, and therefore カタカナ words (with the long vowel dash) in ひらがな as well
- in ads or headlines, where a word should be Japanified; e.g. らーめん is a very popular ひらがな + long vowel dash combination
The tilde (～ にょろ) in principle goes everywhere a long vowel marker should not go, because the word doesn't have a long vowel. E.g. 行ってきま～す, あ～疲れた～, etc.
That said, of course there are people who confuse these uses and use ー where there is no long vowel in the word. (But I have never seen ～ being used where a long vowel marker should go...)
I generally agree with what ssb and Earthlin say, and would add the following thought: consider the potential consequences of using the long vowel marker indiscriminately. It can look an awful lot like the kanji for "one", i.e. 一 (いち). That could potentially be confusing. In many cases, readers could figure it out by context, but by using ~, you very clearly delineate in Japanese that what you want is a "hyphen-style" effect and not the kanji いち. That's why I think in Japanese, outside of cute situations or very informal text, you see ~ being used.