Let us suppose the family name is [山田]{やまだ}.
Most commonly, we would say:
「山田[家]{け}」 ← 「家」 is read 「け」 for this usage.
「山田[一家]{いっか}」
Formally, we could say:
「山田[一族]{いちぞく}」
「一族」is not an everyday word. It could sound too dramatic or theatrical if used in a casual convo.
"And would you use the same language when talking to a member of the family you're referring to and when talking about that family to a third party?"
The same phrase can be used on both occasions.
http://image.eiga.k-img.com/images/movie/83447/poster2/200.jpg?1457577317 http://silvercopen2.c.blog.so-net.ne.jp/_images/blog/_e73/silvercopen2/E382B9E382AFE383AAE383BCE383B3E382B7E383A7E38383E38388EFBC882011-07-142019.01.47EFBC89.png?c=a0
(source: k-img.com)
(source: so-net.ne.jp)