Swisher is the son of former MLB catcher Steve Swisher, who played for various National League baseball clubs in the 1970s and 1980s. Swisher was born in Columbus, Ohio, but grew up in Parkersburg, West Virginia. Before his professional career, Swisher played college baseball for the Ohio State Buckeyes. Drafted by the A's in the 2002 MLB draft, Swisher made his MLB debut with the Athletics in 2004, and played for the team through 2007. After he spent one year with the White Sox in 2008, the Yankees acquired him prior to the start of the 2009 campaign. He played in New York for four years before signing with the Cleveland Indians prior to the 2013 season.