Robert Seldon Duncanson was born in Fayette, New York, in 1821. Duncanson was one of the five sons of John Dean Duncanson (c. 1777 – 1851), a free black tradesman, and Lucy Nickles (c. 1782 – 1854). Often, it is cited that Robert's father was Scot-Canadian; however, there is no evidence to support this claim, and it is unclear when or where the original source of the claim began. All evidence points to Robert Seldon being the descendant of freed slaves from Virginia. John Dean's father, Charles Duncanson, was a former slave from Virginia who was freed from bondage by his owner. Charles received special privileges, including his emancipation and the opportunity to learn a skilled trade, because he was likely the illegitimate son of his owner. After becoming emancipated, Charles and his son John Dean lived as freemen in Virginia. However, at the end of the eighteenth century, white opposition toward free black men grew in the Upper South. In response, Charles, his son John Dean, and his wife Lucy Nickles, like many free African Americans, moved north. The Duncanson family settled in Fayette, New York, where Robert Seldon was born. Charles' knowledge of carpentry and house painting was passed down to his son, John Dean, and his grandchildren. This knowledge would later allow Robert Seldon Duncanson to develop as an artisan and later as an artist.