Let's break this sentence down.
虫に変ってしまっているのに気づいた
At a basic level this sentence breaks up into two fundamental parts:
(A) 虫に変った -- Someone/thing changed into a bug
and
(B) 気づいた -- Someone noticed something.
What is noticed is being changed into a bug. That would make sentence A the object of 気づいた. Two things need to happen here to complete this joining of sentence A and B.
When you say "I noticed A" in Japanese, the particle that marks what you noticed is に. So, "I noticed A" becomes
Aに気づいた
But in Japanese, A must be a noun, it can't be a sentence or a verb or anything else. To make sentence A a noun we nominalize it. There are various ways of doing this, but often in a situation like this the nominalizer used is の. This works as follows:
A + の --> 虫に変ったの
(There's no nice way to translate this grammatical construction into English.)
So, now we can put A and B together.
A + の + に + B --> 虫に変った + の + に気づいた --> 虫に変わったのに気づいた。
So notice: the のに here is not the のに you're looking for; this is a different grammatical construct. There are two distinctly different grammatical constructs here: nominalizing a sentence with の and singling out that which was noticed with に.
To get a bit of a better feel for this, let's change the sentence a bit to
Mr B saw Mr A change into a bug.
Again there are two basic underlying sentences:
Mr A changed into a bug --> Aさんが虫に変わった。
and
Mr B saw [it]. --> Bさんは見た。
In this case, what is seen is marked by the particle を, but still Aさんが虫に変わった needs to be nominalized (again we'll use の). Put together we get
Aさんが虫に変わったのをBさんは見た。
On to てしまう
In English we like short to-the-point objective-sounding sentences: "He noticed he'd changed into a bug." Perfect! But, in Japanese these sorts of expressions come across as blunt, emotionless, and rather inhuman. In other words, 虫に変わったのに気づいた sounds a bit like a droid; it feels like it completely lacks any human feeling or emotion.
The てしまう construct has a wide range of meanings. On one level, it gives the sentence a sense of completeness.
虫に変わってしまった --> He'd completely changed into a bug.
And, I think for a native speaker, there is at least a minimal residue of this meaning in the sentence. But, てしまう can also express shock, regret, disappointment. Just try to imagine your reaction upon waking up noticing that you were no longer human but a bug. You would be in a bit of shock.
[皿]{さら}が割れてしまった。
The dish was broken.
Here てしまう expresses a sense of finality: the dish is broken and there's nothing that can be done about it; it's been destroyed. And if the dish were something that you were given by, let's say, your gramma, you might be very sad about this. This てしまう is a very powerful and useful construct for expressing such emotions.
But, back to Kafka. We could write:
虫に変わってしまったのに気づいた。
This would however leave the reader wondering, "is he a bug or not?" Perhaps he only noticed somehow that during the night he'd been transformed into a bug, but currently he's not a bug.
Imagine a werewolf or vampire sort of story. You might read
起きたとき夜[狼]{オオカミ}に変わってしまったのに気づいた。
When he woke up, he noticed that last night he'd changed into a werewolf.
He's not a werewolf right now, but there's some kind of clear evidence that that's what happened.
In the case with Kafka, it's not just that he changed into a bug, but he's still a bug. So, to get this across, we need to say
虫に変わってしまっている
Perfect!
Why not instead just say
虫に変わってしまう
Tense in Japanese doesn't work like in English. There's not really a present tense in Japanese, just a non-past form and a perfective form. I recommend you read up these different forms else where. But, in short, the non-past form (which we often call the present tense) indicates an incomplete state. So,
虫に変わってしまう
means something more like "I will become a bug."
