He lived his childhood along with his younger brother Naoharu in the house where he was born located in an isolated location during the end of the Edo period, and because of its isolation, the place was often used by loyalists to the Emperor for their clandestine meetings and as a hiding place.
生家は近隣に家のない寂しい一軒家であったが、直輝や弟・直温が幼少時代を過ごした時期は幕末であり、その立地条件から勤王の志士に秘密の集会所として、また潜伏場所としてたびたび利用された。