I have seen this script in several games—and possibly elsewhere—but the only one I can think of now is [Samurai Shodown]{サムライ・スピリッツ}. The script characters are typically very rounded and/or squiggly.
I don't know Samurai Shodown lore very deeply, but this script is often found in relation to the character [天草四郎時貞]{あまくさ・しろう・ときさだ}. I know the video game character is loosely based around the historical figure of the same name, but the video game character is depicted as a kind of sorcerer with different kinds of powers and magic. Here are some images of this script in relation to the character:
(Click to see expanded version)
So what is this script?
- Is it Japanese? It kind of looks like it could be some type of cursive kanji 字体.
- Is it something related to East Asian religious/supernatural themes? I thought maybe something related to Shinto due to the 四手 and しめ縄 in the pictures. I looked a bit into [神代]{かみよ, じんだい}文字 and アヒル草文字, but all the examples I could find didn't look much like this.
- Is it something invented purely for the game?
If the answer is only tangentially related to the Japanese language, feel free to close as necessary. But any insight is appreciated because this script has been haunting me for a long time.