The beginnings of the Victor Harbor line was a horse-drawn broad () gauge tramway built from the port of Goolwa on the Murray River to an ocean harbour at Port Elliot in 1854. This line was used to move freight between the shallow-draft vessels navigating the Murray, and coastal and ocean-going vessels, without either having to traverse the narrow and shallow mouth of the river with unpredictable currents. It was later extended from Port Elliot to Victor Harbor in 1864 and from Goolwa to Strathalbyn in 1869. It was extended to Mount Barker Junction on the South Line in 1884 and strengthened to carry steam trains.