Chadwick began his career as a junior with Southampton, and became the youngest player to appear for their reserves on 31 October 1959, aged 16 years 73 days. He turned professional in October 1960 and made his first-team debut on 4 November 1961, at home to Bristol Rovers in place of the injured Terry Paine. His six years at the Dell were spent in the shadow of Paine and John Sydenham and as a result he only made 25 league appearances for The Saints, in which he scored once. In July 1966 he moved to Middlesbrough, costing the Ayresome Park side £10,000, and slotted straight into the side, making his Boro debut on 20 August against Colchester United.