Originally, Watarai clan served in both the Inner Shrine and Outer Shrine of Ise-jingu Shrine, but Arakida clan of the Nakatomi lineage of Fujiwara clan became new shake for the Inner Shrine, and as the result, Watarai clan became the shake for the Outer Shrine only.
元は、内宮、外宮ともに度会氏が担っていたのだが、内宮は藤原氏、中臣氏系の荒木田氏が新たな社家となり、その結果、渡会氏は外宮のみの社家となった。