James urged President Grant to allow him to succeed Alonzo B. Cornell as Surveyor of the Port. Grant instead did one better and made him the Postmaster of New York City on March 17, 1873, and four years later reappointed by President Hayes. James instituted several reforms during his terms, including increasing the efficiency of the postal workforce and improving postal facilities wherever possible. Once General Arthur stepped down, James was offered the position but turned it down as he did not feel it right to supersede his superior. However, David M. Key resigned as Postmaster General in 1880, and James was offered that position by Hayes instead. After consulting with his friends, he declined it as well. He was asked to be the Republican nominee for mayor of New York the same year but turned this down also.