Although Southworth immediately resumed his successful minor league managerial career, the firing and personal tragedy — the death of his wife Lida at age 42 — began a downward spiral. Beset by struggles with alcoholism, he quit a coaching job with the 1933 Giants during spring training and left baseball for two seasons. After a recovery, he rejoined the Cardinals' minor league system in 1935 and by 1939 he was again enjoying success as Rochester's manager. He remarried in 1935, wedding the former Mabel Stemen, with whom he had a daughter.