The son of a tanner, Avery began working at a local factory at the age of 16 and supported himself for decades with a succession of blue-collar jobs. The death of his brother-in-law in 1915 left Avery, as the sole remaining adult male in his household, responsible for the support of nine female relatives. His interest in art led him to attend classes at the Connecticut League of Art Students in Hartford, and over a period of years, he painted in obscurity while receiving a conservative art education. In 1917, he began working night jobs in order to paint in the daytime.