Smirke was born in London on 1 October 1780, the second son of the portrait painter Robert Smirke; he was one of twelve children. He attended Aspley School, Aspley Guise, Bedfordshire, where he studied Latin, Greek, French and drawing, and was made head boy at the age of 15. In May 1796 he began his study of architecture as a pupil of John Soane but left after only a few months in early 1797 due to a personality clash with his teacher. He wrote to his father: