I think I can speak to this because I used to have (maybe still do) trouble getting vowel pronunciation right, and have struggled a lot with it.
Japanese may have situations when they drop vowels, such as the ones you name.
This is not an indication that it's simply generally applicable, though. There is a feel to when it is right that is the kind of thing a native speaker can do without knowing any particular rule.
As a non-native speaker, though, it's harder to plug into what that rhythm is, and when it's right and when it's wrong to do so.
For me personally, one of my main issues is that I have some trouble with two vowel sounds in a row, such as 青{あお}い (blue)
. In the English accent that I speak natively, the tendency is to compress vowel sounds. So, before much deliberation and practice, I would pronounce 青{あお}い
as a two syllable "ow-ii" (something like that, not sure how to spell phonetics). Japanese speakers would not understand me.
In summary, to answer your question, yes, it does matter a lot whether or not you drop a vowel sound, and you will in fact very likely be not understood if you drop, or change, a vowel sound. Native speakers definitely do notice.
My recommendation is to try and emulate pronunciation of all vowel sounds as clearly accurately as you can for now, and then slowly introduce dropped sounds and other variations by copying them as you hear them on a case by case basis.