Born in Lynn, Massachusetts to Greek immigrants, Georgios Agganis and Georgia Papalimperis, Agganis first gained notice as a college football player at Boston University, becoming the first person in school history to be named All-American. He passed up a professional career with the Cleveland Browns in order to play his favorite sport, baseball, close to his hometown. Agganis was signed to a bonus baby contract, and after one season playing minor league baseball, Agganis became the starter at first base for the Red Sox.