However, in 1602 he fell out with Naomasa HORI, who was from the same family, resulting in conflict, and therefore he pretended to be ill and retired to the residence of his deceased father located in Fushimi, Kyoto.
しかし慶長7年（1602年）同族の堀直政と不和になって対立し、病と称し京都伏見にあった亡父の屋敷に隠遁した。