It may be a dumb question, both mean circle/round thing but 玉 is more of a jewel/sphere/ball. Could I say "5円丸" instead of "5円玉"? Or should I just stick with "5円硬貨"
When I was a kid, some older people did actually called coins as:
Now that those people are gone, I no longer encounter that usage anywhere. It might have been a regional (Nagoya, in my case) and/or old way of calling coins. You would look quite weird if you used it today.
In informal daily speech, we almost exclusively use:
Only in very formal situations, we use:
Most people just do not actively use the last one much even though everyone knows what it means.