By 1877, Clifford's mental faculties had declined and impaired his ability to be an effective Justice. Justice Samuel Miller wrote that Clifford's mental deterioration was "obvious to all of the Court" and "in the work we do, no man ought to be there after 70." (Clifford was 74.) In 1880, Clifford experienced a stroke that, according to Miller, "rendered him a babbling idiot." He did not participate in any cases during that year, but still refused to step down, hoping that a Democratic president would be elected in 1880 and appoint a successor. He died on July 25, 1881. His successor on the bench, Horace Gray, instead being appointed by Republican president Chester Arthur.