Can someone help clarify. When should I use "baggu" and when "kaban"? Is there any difference at all?
If you use google image it can give you a general idea:
kaban is generally for men and baggu is generally for women (as shown in the image results). Also, kaban seems more practical and business use while baggu is more for fashion/design.
Another difference is already fixed names. For example, エコバッグ (eco baggu) and キャリーバッグ (kyari baggu) are specific types of bags.
According to the dictionary バッグ is pretty much the same as "bag", anything that you can put things inside and carry with you.
かばん means "briefcase", a special type of bag, usually made of leather, designed to carry things like papers and documents.