In 1619 his father perished on his way from Ireland to England in a shipwreck near the Skerries off the coast of Anglesey. James inherited his father's courtesy title Viscount Thurles. The year following that disaster, his mother brought young Thurles, as he now was, back to England, and placed him, then nine years old, at school with a Catholic gentleman at Finchley — this doubtless through the influence of his grandfather, the 11th Earl. His mother remarried to George Mathew of Thurles.