Two years later, in May, 1194, his grandfather Tomotsuna UTSUNOMIYA was sued by Yukifusa NORO who was the governor of Shimotsuke Province about the embezzlement of the public rice field, and in July of the same year, Tomonari was also accused of complicity and was exiled to the provincial capital of Suo Province.
2年後の建久5年（1194年）5月には祖父・宇都宮朝綱が下野守野呂行房より公田横領を訴えられ、同年7月には朝業も連座の咎で周防国国府へと配流される。