- の is being used as a nominalizer here. By placing の after an adjective or verb, you are able to create a dummy noun. Essentially, you are creating a noun phrase. For example:
the new one.
I like the new one.
の here acts as a "dummy noun" that is being described by 新しい. In your case, this dummy noun is being modified by the verb 旅する.
一人で旅する = to travel alone (verb)
一人で旅するの = travelling alone (noun phrase / gerund)
Likewise, 友達と旅するの is a noun phrase meaning "travelling with friends".
Your second and third questions both fall into one answer. The first reason why you cannot remove の from this pattern is because と can only list nouns (の is what allows 旅する to become a noun). The second reason is because this is a set pattern.
As for this grammar pattern, it is one of the main ways to create a list from which 「どれ、どちら、どっち」 can choose from. では here essentially means "out of" or "between". If this does not sound clear, here is an example:
Do you like cats, dogs, or rabbits the most?
More literal translation: Out of cats, dogs, and rabbits, which one do you like most?
Here is another example I stole from IMABI:
As for jogging, which is best, morning or night?
So for your example:
Literal Translation: Between travelling alone and travelling with friends, which one is better?
Natural translation: Do you prefer travelling alone or travelling with friends?
The addition of the second と makes it sound a bit more old fashioned. It is omitted often. In addition, と can be replaced with か when implying that you can choose from something outside of the list. You can find more information about this pattern here.