Goo辞書 entry #4 under うち【内】 had this to say:
4 (Also rendered 「裡」; Used in the form 「...のうちに」) Denotes the conditions in which something is done (*I'll come back to the examples later.)
Although the dictionary for some reason only mentions the adverbial form 「...のうちに」, the adnominal form 「...のうちの」 is also in use, though perhaps with less occurrence. (e.g.「暗黙のうちの了解」,「知らず知らずのうちのマナー違反」 )
With this in mind, let's look at your sentence 「無意識のうちの行動なのだろう」.
The adnominal 「のうちの」 relates 「行動(an action)」 to 「無意識(unawareness, unintentional-ness)」 .
「無意識」 is the state, or condition, within whose notional "boundaries" the 行動 was taken. The action was done, so to speak, in the territory of unawareness.
Notably, the English preposition in has a very similar meaning to this use of うちに.
- With reference to circumstances or conditions
- With reference to physical surrounding, personal states, etc., abstractly denoted
(Both cited from here.)
To illustrate using the examples from goo辞書:
暗黙のうちに理解しあう ⇔ understand each other in silence
会は成功のうちに終わる ⇔ the meeting/gathering ends in success
In the two examples above, both ~のうちに and in describe in what way or under what circumstances/conditions something happens. However, this does not mean they are always the best translation for each other. For instance, "tacitly" would be a better word for「暗黙のうちに」 most of the time.