Accused by naval officer Lt. Clark Henry Wells of disloyal conduct and sentiments after the Battle of Fredericksburg, Haller was dismissed from the Army in July 1863. Wells alleged that Haller had toasted him "Here's to a Northern Confederation and a Southern one while Lincoln is President", blamed the President for casualties at Fredericksburg, and in a later dispute offered that a "Black Republican" would be a more suitable quarters-mate for him; Haller denied giving such a toast entirely, and protested that he did not blame the government for "disasters" at Fredericksburg but only the change in strategy caused by Lincoln's removal of General George B. McClellan. He and his wife eventually returned to the American West, settling on Whidbey Island in Coupeville, Washington Territory in 1866 where he built a home on Front Street and started a business that extended credit to pioneer families. In 1879, Haller's case of dismissal was re-examined by Congress. Following a six-day long court of inquiry in Washington, D.C., he was exonerated and his commission reinstated with a promotion to colonel.