Alger was born in Massachusetts, and attended Harvard College. He became a Unitarian minister, but his career as a clergyman was brief. It ended when his congregation charged him with child molestation. Criminal charges were not placed against him, but his career in the church was finished.