James was tutored and trained to be a planter and slaveholder, and member of the landed gentry. His widowed mother never remarried, which was unusual at that time for a woman her age; she had extensive Taylor family in the county. As the eldest son, James Madison Sr. inherited Mount Pleasant when he came of age in 1744. He called the plantation Home House. Acquiring more land, he eventually owned 5,000 acres, making him the largest landowner in Orange County. By the time of his death, he owned 108 slaves.