Larrabee was the seventh of nine children, and grew up on his father's Connecticut farm. He was educated in local schools until the age of 19. At around age 15 Larrabee lost the eyesight in his right eye after a gun he was holding accidentally discharged. As a result, he was unfit for many careers available to young men of his class in New England. Larrabee chose to become a teacher. In 1853, at age 21, he moved to Iowa following an older sister in search of opportunity.