James' mother was Mary Van Cortlandt, who had married Peter Jay in 1728, in the Dutch Church. They had ten children together, seven of whom survived into adulthood. Mary's father, Jacobus Van Cortlandt, had been born in New Amsterdam in 1658. Cortlandt served on the New York General Assembly, was twice mayor of New York City, and also held a variety of judicial and military offices. Two of his children (the other one being his son Frederick) married into the Jay family.