Hartley was born in Lewiston, Maine, where his English parents had settled. He was the youngest of nine children. His mother died when he was eight, and his father remarried four years later to Martha Marsden. His birth name was Edmund Hartley; he later assumed Marsden as his first name when he was in his early twenties. A few years after his mother's death when Hartley was 14, his family moved to Ohio, leaving him behind in Maine to work in a shoe factory for a year. These bleak occurrences led Hartley to recall his New England childhood as a time of painful loneliness, so much so that in a letter to Alfred Stieglitz, he once described the New England accent as "a sad recollection rushed into my very flesh like sharpened knives".