Born in Easton, Pennsylvania in 1860, Cattell grew up the eldest child of a wealthy and prominent family. His father, William Cassady Cattell, a Presbyterian minister, became president of Lafayette College in Easton shortly after James' birth. William Cattell could easily provide for his children, because he had married Elizabeth "Lizzie" McKeen in 1859; together they shared Lizzie's substantial inheritance. To this picture of the family's success one could add political power as well: James's uncle Alexander Gilmore Cattell represented New Jersey in the United States Senate.