I don't think you can differentiate them without looking at the context.
With the passive form, you'll usually see the doer/"culprit", indicated by ～に／～によって (there are some rules about which to use, but that's beyond the scope of this topic):
- 父にハンバーガーを食べられることが多い → My hamburgers are often eaten by my father ("My father often eats my hamburgers")
- 「モナ・リザ」はダ・ヴィンチによって描かれた → The "Mona Lisa" was painted by DaVinci.
Don't forget that there is a third conjugation of this form that is a type of keigo (somewhat between normal politeness and super politeness)
- 課長は明日会議に行かれる → The section chief is going to the meeting tomorrow.