Between May and June, 1194, his grandfather Tomotsuna UTSUNOMIYA was sued by Yukifusa NORO, kokushi (a provincial governor) of the Shimotsuke Province, for looting and possessing the land administered directly by a ruler (over 99 hectare), so the Imperial Court punished the Tomotsuna family and put them under custody of the provincial office of the Bungo Province.
建久5年（1194年）5月、祖父宇都宮朝綱が下野国国司の野呂行房より公田掠領（百余町）を訴えられ、朝廷は朝綱一族を処分し豊後国国府預かりとする。