By the census of 1860, the population of the village was 599. A few years later, in 1863, Irvington was touched by the New York Draft Riots. Fearing that the violence in the city, which had to be put down by Federal troops, would spread to Westchester, special police were brought in and quartered in a schoolhouse on Sunnyside Lane. They were commanded by James Hamilton – the third son of Alexander Hamilton – whose estate, Nevis, was on South Broadway. The presence of this special force deterred any violence a group of draft protestors which passed through Greenburgh on their way to Tarrytown may have intended. This was the only instance in which Civil War-related activity directly affected Irvington.