Annenberg was born in Milwaukee and raised in New York. He attended the Wharton School, the business school at the University of Pennsylvania, though he dropped out to pursue stock investing. His father, Moses Annenberg, was convicted of tax evasion and incarcerated for two years. During that time, and following his father's death in 1942, Annenberg took control of the "Inquirer," expanding its influence and his own. He built up his family's magazine business with great success, extending it into parts of the media industry such as radio and television.