In 1906, Fish was removed from his position at the Illinois Central by E. H. Harriman, possibly because of Fish's cooperation and participation with the state government in investigating the Mutual Life Insurance Company. However, it is also possible that the reason was because his wife, Mamie, had snubbed Harriman's wife, Mary, from a society tea party. Mamie found Mary too "dull" and so snubbed her from a tea party at their home in Newport, Rhode Island, where they spent the summer season.