Lamar was born at the family home of "Fairfield," near Eatonton, Putnam County, Georgia, the son of Lucius Quintus Cincinnatus Lamar and Sarah Williamson Bird. He was a cousin of future associate justice Joseph Lamar, and nephew of Mirabeau Buonaparte Lamar, second president of the Republic of Texas. In 1845 he graduated from Emory College (now Emory University), then located in Oxford, Georgia. He was a member of Sigma Alpha Epsilon fraternity and was among the first initiates in that fraternity's chapter at the University of Mississippi.