After she had already given birth to two children by him, James Wedderburn sold Rosanna, then five months pregnant with his third child, back to her previous owner. (James Wedderburn later returned to live in Britain. His legitimate son and heir, Andrew Colville, defending his father after these details were made public in the British press, denied the paternity and further claimed Rosanna was both promiscuous and unable to control her temper.) James Wedderburn stipulated, however, that Rosanna's child (Robert) when born should be legally free, and he officially registered both Robert and an elder brother, James, as free. Although born free, Wedderburn was raised in a harsh environment, as his mother was often flogged due to her "violent and rebellious temper". She was eventually re-sold away from her son, who was then raised by his maternal grandmother, a woman known as "Talkee Amy".