Like his father and other family members, Rives operated his plantations using enslaved labor. In the 1830 federal census, he owned one enslaved man, an enslaved woman also between 20 and 30 years old, and a girl in Albemarle County. A decade later, Rives owned 17 enslaved men and boys and 13 enslaved women and girls. In the 1850 federal census, Rives owned 69 slaves in Albemarle County. In the final prewar census, Rives owned 66 slaves in Albemarle County. His brother or nephew Robert Rives Jr. owned 43 slaves in Albemarle County in 1850. and 70 slaves a decade later. His other brother William C. Rives owned slightly more enslaved people in the county's Frederickville section. Following the war, Judge Rives continued to operate the farms using paid labor until entering the federal judicial service in 1871, as described below.