By 1864, 2,200 freedmen lived on Roanoke Island, settled on individual household plots, and working for pay for the Army. James recruited teachers and founded several schools. A sawmill was built and a fisheries started. More than 150 freedmen from the island joined the nearly 4,000 from North Carolina who joined the United States Colored Troops. After the war, the Army dismantled the forts. Due to President Andrew Johnson's returning lands to Confederate landowners, the colony was abandoned. Most of the freedmen chose to be transported to the mainland, where they looked for work. By 1870, only 300 remained on the island.