However, when the Taira family was driven from the capital by MINAMOTO no Yoshinaka in July 1183, Narimori accompanied them with his father and older brothers.
しかし寿永2年（1183年）7月、源義仲に追われ平家一門が都落ちすると、業盛も父や兄らともにこれに従った。