General Howe evacuated Boston in March 1776, taking all British troops and Loyalists. James was transported with the British fleet as a prisoner and taken to Halifax (former city), Nova Scotia, where he spent nine months in the jail on Hollis Street. His father also sailed with the British fleet to Halifax in March 1776, as part of the general exodus of Loyalists when British forces abandoned the city. In November 1776, James was exchanged for Colonel Philip Skene. When he got back to Boston in December, he was elected to be a delegate to the Continental Congress. He would serve in Congress until 1782.