In 1845, after a failed run for governor of Florida, Call retired from active politics. Afterwards, he devoted himself more to the life of a gentleman planter and took up residence at Orchard Pond plantation. By the late 1840s, his daughter, Ellen, married a promising lawyer and future state legislator, Medicus A. Long, and started a family of her own. Call eventually deeded her The Grove along with seven slaves in 1851. He spent the rest of his life at Orchard Pond, but was a frequent visitor of The Grove. In one of the notable events of The Grove's history, Call, who was an ardent Southern Unionist, came to Tallahassee to protest the state's secession convention taking place at the Capitol in January 1861. He decided at the last minute not to go, however, and stayed instead at The Grove. It was here that a group of secessionists decided to accost him after they voted to secede. Call stood out on the front porch, and raising his cane above his head, told them "You have opened the gates of hell, from which shall flow the curses of the damned which shall sink you to perdition." He returned to The Grove permanently as his health deteriorated, and died in 1862.