In September 1936, newly promoted, First Division Brentford sent Brown to Tottenham Hotspur for a transfer fee of 1,200 £. In his one season there, he played only four first team games, but scored twenty-one goals in thirty games for the reserves. In July 1937, Brown moved to the semi-professional Guildford City of the Southern Football League for £750. Over his two seasons with Guildford City, Brown scored 148 goals in 150 games and helped the club win the Southern League title during the 1937–38 season. During the 1938/39 season, Brown recorded five Hat-Tricks and scored seven goals (six of them headers) in one game versus Exeter City. Guildford finished in second place, losing out on a second consecutive title by just one point. James had been sick and out of most of the play during the last month of the season. He was loaned out to Brentford's First Division team, to play center-half in the FA Cup match against Fulham, where he scored a goal. He left Guildford City at the end of the 1939/40 season, because of increasing cartilage problems. Guildford City suspended all League play in September 1940. He then moved north to finish his career with Scottish First Division club Clyde, signing in November 1940 and played two matches with his brother John against Queen's Park in February and March 1941. In the February match with Clyde, just like his debut with Manchester United, Brown scored directly from a corner kick. He retired due to injuries soon after. Clyde finished in second place, only three points behind eventual champions Rangers during the 1940/41 season.