I always thought that the word ても means 'even if'. For example, 一緒に行かなくてもいい would mean 'Even if you don't go with me, it is ok'(speaker really wants to go together but is ok even if they dont go together) . I heard that removing the も gives the same meaning, but if I look at it logically, wouldn't "一緒に行かなくていい" mean "Don't come with me" (speaker does not want to go together). So is there a difference between using も and not using it in the sentence above?
Maybe this type of も is close to "also" rather than "even". For the (subtle) difference in meaning, please see: Difference between てもいい and ていい? For example:
I can die (for it, although I don't want to).
I am allowed to die (because I want to die).
行かなくて is just a te-form of 行かない ("not to go"), and it does not have an imperative meaning by itself. A word-by-word translation of "行かなくていい?" is something like "Not going, then, fine?". Although 行かないで works as a request "(Please) don't go", 行かなくて does not have this meaning.
も adds a level of uncertainty, so it's like saying "even if this happens".
心配しなくてもいい。 You don't need to worry.
The も also has a softening effect. Part of the softening comes from the level of uncertainty, as though saying, "Even if you don't worry, it's okay." It's as though the speaker is allowing for the listener's potential worrying. On the contrary, for example:
This sounds like the speaker is annoyed with the listener for worrying, and that they really want them not to worry, adding less wiggle room for personal idiosyncrasy. In Japanese culture and language, the levels of meaning can often be subtle. The も here creates a subtle understanding that oh, I'm fine to be myself, whereas when you leave out the も, it sounds like more of a demand.