No new permanent buildings were built specifically for the Civil War, though a temporary hospital was built. The hospital treated victims of cholera and yellow fever in epidemics during the 1850s and 1860s. During the war, Governors Island was used mostly as a support facility to muster soldiers, though the fortifications were still operational. Castle Williams held Confederate prisoners of war and Fort Columbus held captured Confederate officers. The austere accommodations frequently held over a thousand prisoners, and they frequently escaped and swam across to "mainland" Manhattan. In 1863, in the midst of the New York City draft riots, protesters unsuccessfully attempted to take over the island when Army troops were deployed to Manhattan.