However, they are said to have taken a no cooperative stance toward Nobunaga ODA because they were attendants to the Shogun, so at the Honnoji Incident in 1582 Yoritsugu NOSE supported Mitsuhide AKECHI and after Mitsuhide was destroyed by Hideyoshi TOYOTOMI Yoritsugu was forced from Nose and hid in Bizen Province, which was associated with his ancestor Yorisada TADA.
しかし、将軍家の近習であった経緯から織田信長には非協力的であったとされ、天正10年（1582年）の本能寺の変では能勢頼次が明智光秀方に加勢し、豊臣秀吉によって光秀が滅ぼされると頼次は能勢を追われ先祖多田頼貞に縁のある備前に潜伏したという。