the new bodyguards were constantly watched and rigorously tested by samurai who were weary about allowing dishonest people into the heart of their province , but a few ninja eventually grew to deserve the respect of the daimyo and those entrusted to help guard his family were given the responsibility to defend against all threats , even other ninja .