As a young man, Harvey served in World War II. On the night before he left to go to Europe, he had a one-night stand with an African-American "lady of the evening" who later gave birth to a son named Lester who, at the age of sixty, was able to track down Harvey and introduce him to his family, including Lester's children and grandchildren. After the war, Harvey came home and married a woman named Helen, and remained faithful to her until she died, although she did have an affair. He frequently reflected back on Helen, who died shortly before the series began. In Season 3, a stock dip erased Harvey's life savings.