The way I've understood it, although ように means something like "so that", the reason it overlaps with "try" is that often you are making an attempt, and do not have full control over the situation.
The grammar dictionary lists that ように is common with potential and negative verbs. For instance, consider:
I will try as much as possible / make sure that I don't meet anyone
なるべく should make this even more clear. Yes, you have made sure that you won't meet anyone. But you can't be sure of that. I'm not a native speaker or even that fluent, but I think:
Sounds pretty strange.
First I have to make the letters readable.
And again, you don't seem to have full control over the situation. To contrast with ようとする:
I also tried to talk to him!
Again, just my opinion, but this sounds better than 声かけるようにしてた. It seems to me that ようとする is better for cases especially after you've tried. For instance, before trying to talk to him, you probably wouldn't expect it to be hard. But let's say you were trying to get opportunities to talk to him. Then I think:
Is ok, and then after the fact:
When ようとする is in the past tense it sounds like it was a failure. If you use ようとする before the event, it kind of sounds like... although you have control over the situation, you expect it to be hard. Like for instance, if you go to lift a weight but you expect to be too weak to lift it. Note however that in this case てみる might be more appropriate.
- ように meaning "so that" sounds like although you are going to do something to accomplish something else, you don't have full control over the situation.
- ようとする is especially common after the fact to say that you tried to do something but failed.
- ようとする doesn't seem that common before the event, since てみる seems to be preferred, but it just sounds like you expect to try but maybe you lack the ability or skills necessary (which is a direct consequence of you: it's not as though someone else makes the situation uncontrollable).