He was the grandson of James Clavering (1565–1630), a merchant adventurer, Mayor of Newcastle upon Tyne in 1607, who bought an estate at Axwell House, near Blaydon on Tyne in 1629.
Educated at Magdalene College, Cambridge and Gray's Inn, Sir James inherited the Axwell estate on the death of his father in 1648.