On April 14, 2016, Swisher signed a minor league contract to return to the Yankees. They assigned him to the Scranton/Wilkes-Barre RailRiders of the Class AAA International League. Playing first base, Swisher batted .255 with seven home runs and 25 RBIs in 55 games. The Yankees opted not to promote Swisher to the major leagues, in spite of injuries suffered by Mark Teixeira, Dustin Ackley, and Chris Parmelee. On July 2, Swisher opted out of his contract to be with his family following the birth of his second daughter. He announced his retirement on February 17, 2017, and became an analyst for Fox.