Throughout his childhood, Cary spent many summers at his grandmother's house in Ireland and at Cromwell House in England, home of a great-uncle, which served as a base for all the Cary clan. Some of this upbringing is described in the fictionalised memoir "A House of Children" (1941) and the novel "Castle Corner" (1938) – i.e., Cary Castle, one of his family's lost properties in Inishowen in Ulster. Although Cary remembered his West Ulster childhood with affection and wrote about it with great feeling, he was based in England for the rest of his life. The feeling of displacement and the idea that life's tranquility may be disturbed at any moment marked Cary and informs much of his writing. His health was poor as a child. He was subject to asthma, which recurred throughout his life, and was nearly blind in one eye, which caused him to wear a monocle when he was in his twenties. Cary was educated at Clifton College in Bristol, England, where he was a member of Dakyns House. His mother died during this period, leaving him a small legacy which served as his financial base until the 1930s.