Later in life, Izac chose to downplay the accomplishments of the Medal of Honor again. After leaving office, he moved his family to land inherited from his father-in-law in Gordonsville, Virginia, where Izac became a farmer and led a simple life raising cattle and growing different kinds of fruit and vegetables. The family then grew most of its own food and survived on pensions from the Navy and Congress. At one point, he damed a stream to make a small lake, which has since been known as Lake Izac. In retirement the family also took to traveling, visiting Jerusalem almost every year and using these visits as the basis for a 1965 book, "The Holy Land: Then and Now." Of the book, he later said, "No one who has not visited the most momentous events in the history of the human race will ever be able to visualize just how it all happened. You simply have to go there."