Booth sold his railway to the Grand Trunk Railway in 1904. In 1923, the railway became part of the government-owned Canadian National Railways. The reconstruction of the Welland Canal in 1932, along with 1933 abandonment of a portion of the line in Algonquin Provincial Park (as a consequence of the Cache Lake trestle being damaged by ice), and a drop in grain prices during the Great Depression, contributed to a loss of importance for Depot Harbour, and the CNR closed the facilities in favour of its own at South Parry. The town fell into disrepair and as its population gradually declined, Depot Harbour was abandoned.