After Madison and his younger brother Eston were freed, they each worked and married free women of color; they lived with their families and mother Sally in Charlottesville until her death in 1835. Both brothers moved with their young families to Chillicothe, Ohio to live in a free state. Madison and his wife Mary lived there the remainder of their lives; he worked as a farmer and highly skilled carpenter. Among their ten children were two sons who served the Union in the Civil War: one in the United States Colored Troops and one who enlisted as a white man in the regular army.