James Douglas was the second son of Sir George Douglas of Pittendreich, Master of Angus, and Elizabeth Douglas, daughter David Douglas of Pittendreich. He wrote that he was over 61 years old in March 1578, so was probably born around 1516. Before 1543 he married Elizabeth, daughter of James Douglas, 3rd Earl of Morton. In 1553 James Douglas succeeded to the title and estates of his father-in-law, including Dalkeith House in Midlothian, and Aberdour Castle in Fife. Elizabeth Douglas and her two elder sisters, who were married to Regent Arran and Lord Maxwell, suffered from mental ill-health. Their children either did not survive to adulthood, or in the case of three daughters were declared legally incompetent in 1581. James also had five illegitimate children