Baxter was born into a wealthy family in Portland where his father James Phinney Baxter served six terms as mayor and had made his fortune in the canning industry. He graduated from Portland High School in 1894, Bowdoin College in 1898. While at Bowdoin, Baxter founded the school's literary magazine, The Quill. He later earned a law degree from Harvard University in 1901. He went into the family real estate business in Portland. He had seven siblings. However, he was to inherit the bulk of the family fortune.