Ross knew what his reputation was, and with the outbreak of the Civil War in early 1861, and especially after one of his sons joined the Confederate States Army, Ross left for England for a few months. He tried returning a year later, but by 1863 had left for the duration of the war. While he was in exile he wrote, "Not that I am guilty of any act against the government of the U.S., but I am considered to entertain opinions which are pronounced by some people as disloyal. For that reason I remain out of the country, hoping that the American people may some day return to their reason, I may return in safety to spend the remainder of my days in a country ruined by the madness and fanaticism of its own people." Ross returned, and lost many of his investments, but by his act of avoidance, probably prevented further personal ruin.