Brown is of English ancestry, from a family that has been in New Hampshire since the colonial era. His earliest American ancestor was 17th century immigrant Francis Matthews, who sailed from Devonshire, England. Brown is part of a 9th generation New Hampshire family and was born on September 12, 1959 at the Portsmouth Naval Shipyard located on Seavey's Island in Kittery, Maine. Brown's father, Claude Bruce Brown, and mother, Judith Ann "Judi" (née Rugg), divorced when he was about a year old. When he was a young child, his mother moved with him to Wakefield, Massachusetts. He often spent his summers in Newburyport, Massachusetts, where his father served as a city councilor for 18 years. He also spent summers in Portsmouth, New Hampshire during his youth. His father and his grandfather were Republicans. His father has said that young Scott became interested in running for political office in the mid-1960s while accompanying him on a campaign for state office.