James was born in Boston and had his preparatory education at the Boston Latin School. His father, John Lovell (1710–1778) was the school's headmaster from 1738 until 1775. James attended Harvard and graduated in 1756. He then joined his father and taught at the Latin School, while continuing his own studies. He received an MA (Master of Arts) degree from Harvard in 1759. Father and son continued their work in the Latin School until it was closed in April 1775, during the Siege of Boston in the American Revolutionary War.