James was the fourth child and second son of property developer James Burton (formerly James Haliburton) and Elizabeth Westley (12 December 1761 – 14 January 1837), of Loughton, Essex, daughter of John and Mary Westley. The son was christened 'James Haliburton' but his father changed the family surname to Burton in 1794. The son James then changed his surname to Burton also, although he was the only member of the family to subsequently change his surname back to Haliburton (in 1838). He was an older brother of the architect Decimus Burton, the physician Henry Burton, and the gunpowder manufacturer William Ford Burton.