Bremer was born in Hartford, Connecticut. He grew up in New Canaan, Connecticut. He went to school at Phillips Academy and graduated in 1959. Then he went to Yale University. He received a Bachelor of Arts degree in 1963. In 1996, he received a master's degree from Harvard University. His degree was in business. He also received a degree in Paris.