First of all, verbs in Japanese have only two tenses, past and non-past. The latter is also known as the “present” tense, although it is also used to describe a future event.
Secondly, as you may have already learned, [V て-form]-いる can mean many different things depending on the verb and/or the context. [V て-form]-いた is simply its past version and it also can mean many different things.
Replacing きた with きていた makes a grammatical sentence.
大きな犬がこっちに走ってきていたので、びっくりした。
However, it probably won’t be understood the way you want it to be. Because of the verb 走る, which describes a visible action, 走ってきていた will more likely be understood as describing a progressive action in the past.
I was startled because a large dog was running up toward me.
きていた without 走る can be understood as referring to a past state that resulted from an earlier action, in this case that of 来る, and this might be translated using the past perfect tense in English.
大きな犬がこっちにきていたので、びっくりした。
I was startled because a large dog had come where I was.
The simple past tense is fine in the original sentence (and in its English translation) because it is meant to describe an action that happened at one point of time in the past.
大きな犬がこっちに走ってきたので、びっくりした。
I was startled because a large dog ran up toward me.