He and his wife had no children together, but he was known to have three illegitimate children, including two daughters and a son, each by different women. One was Dido Elizabeth Belle, a mixed-race daughter born into slavery in 1761 in the West Indies. He entrusted the girl to his maternal uncle William Murray, 1st Earl of Mansfield and Mansfield to raise free in England. Murray served as Lord Chief Justice, ruling on cases important to the abolition of slavery. Belle was educated, married, and inherited money from Murray.