What is the function of the particle に in this sentence?
That means that everyone had to register their names in temples? But why isn't there で particle then (place of action function)?
Both are grammatically correct, but they have different meanings.
お寺で名前を登録する means "to register their names at the temple". The temple happens to be where they registered their names. Their names will be recorded in the list which may or may not be managed by the temple itself.
お寺に名前を登録する means "to register their names with the temple". It's clear that your names will be managed by the temple.