it was risky to stay in town when we did n't have to , and making the adjustment to having me in his life after all those years alone with no one to question him had to be hard , but he had yet to really complain .