In 1923, the line became part of the Southern Railway and like many other lines around the country it suffered from competition from bus services. Passenger services were withdrawn after 1 January 1931, due to falling numbers of passengers - from 51,00 in 1925 to 31,000 in 1927 and 23,000 in 1929. There were up to four freight trains daily at this time. The signal box at Whitstable closed on 11 February 1931 and the branch was worked as a siding from then on. It continued to carry coal, grain and roadstone, with munitions to the harbour during World War II. By 1948, when it became part of British Railways, Whitstable Harbour had fallen into disuse and what was left of the line's trade had disappeared.