After initially struggling to displace Eric Day and Eddy Brown in the side, he established himself as a consistent scorer with Southampton, despite the club suffering relegation from the Second Division during the 1952–53 season. Although he suffered from a case of appendicitis during the year, he scored 14 times in 23 matches, including hat-tricks against Doncaster Rovers and Fulham. In October 1953, having been plagued by knee problems, Dudley moved to the First Division for the first time in his career, joining Cardiff City in exchange for Bobby McLaughlin. He made his debut for the "Bluebirds" on 24 October 1953 during a 3–0 defeat to Burnley but made just four further appearances for the club, scoring once during a 5–0 victory over Charlton Athletic, as a persistent knee injury and the signing of Welsh international Trevor Ford saw him unable to break into the first-team. He left the club after just four months to sign for Brentford.