The town was named for Samuel Spencer, first president of the Southern Railway, who is credited with establishment of the railroad's mechanical shops at the site in 1896. The site was approximately the midpoint of the railroad's mainline between Atlanta, GA and Washington, DC. As the shops were being built the Southern Railway developed a town, also named Spencer, alongside the shops for worker housing. Initially, the Southern partitioned 85 acres into 500 lots. Instead of creating a traditional "company" town in which the workers rented houses Southern sold the lots to workers or businesses for $100 apiece. The deeds did contain restrictive covenants which maintained that a dwelling costing in excess of $400 and approved by a Southern appointed architect be built within a year. The Southern donated lots for religious institutions. Southern also helped establish a YMCA in the town. The community grew quickly and by 1901 had 625 residents.