James was born in Gorseinon and began his career with local side Swansea City. He made his debut at the end of the 1972–73 season which ended with Swansea being relegated to the Fourth Division. They slowly recovered and James 16 goals in 1976–77 and 17 in 1977–78 helping the Swans gain promotion back to the Football League Third Division. He then scored a career best of 21 in 1978–79 as Swansea gained back to back promotions. After two seasons in the Second Division they completed a remarkable rise gaining promotion to the First Division for the first time in the club's history. James took to the top flight well scoring 14 goals in 46 appearances as Swansea finished in sixth position. However the following season saw Swansea relegated back to the Second Division and James joined Stoke City.