I think I can see how you are confused about these two words, but I feel like the confusion might have stemmed from the English Japanese dictionary definitions, or rather the confusing nature of the English word "surgery". I don't think the two Japanese words overlap as much as the related English concepts.
"Surgery" is a medical discipline, or specialty as they call it, whereas "operation" is a medical procedure that's also called "surgical procedure". Colloquially, "surgery" usually stands in for "operation". "Jane is having a surgery tomorrow." and "Jane is having an operation tomorrow." mean the same thing, (although not without ambiguity as to whether Jane is performing the operation or receiving it.) But if someone says a medical student studies surgery, it is more likely they mean the student is majoring in the discipline. If someone
is studying surgical operation, it means they are learning how to operate on people. "Surgical students" but not "operation students". The confusion at its root comes from the multiple meanings of "surgery".
外科 means the science and discipline of surgery. Wikipedia:
外科学は、手術によって創傷および疾患の治癒を目指す臨床医学の一分野である。
That's why such words are in use: 外科医, 整形外科, 心臓外科, 口腔外科, 胸部外科, 外科的な技術, 外科的な処置, 外科的な治療, 外科的な手法.
手術 is the operation performed by people specialized in 外科, namely surgeons. 手術 is also called オペ. Wikipedia:
手術とは、外科的機器やメスなどを用いて患部を切開し、治療的処置を施すこと。
Especially noteworthy are some interesting words and phrases like 外科手術 and 外科的な手術. They essentially translate as "surgical operation" and are used to distinguish from other medical procedures such as injection.
Another example is 心臓外科. Note the difference between the Japanese page and English page of the same Wikipedia article:
心臓血管外科学とは、心臓や血管などを中心に扱う外科学の一分野である。
Cardiac surgery, or cardiovascular surgery, is surgery on the heart or great vessels performed by cardiac surgeons.
The Japanese concept refers to the subject/medical subdiscipline. You can't perform or receive a 心臓外科. You perform a 心臓手術.
In comparison, the English term denotes the medical procedure. And in Anglophone context it is implied or understood that people who do or study said medical procedure are experts in or students of that medical specialty.