They dropped to 10th in 1979–80, before shooting back up fourth in 1980–81, eight points behind champions Aston Villa. Brown played for Portland Timbers on loan in 1981, scoring nine goals and bagging six assists in 24 NASL appearances for Vic Crowe's side.
In 1979-80, the team dropped to tenth place, but rebounded to fourth in 1980-81, trailing champions Aston Villa by eight points. During his loan spell with Portland Timbers in 1981, Brown scored nine goals and contributed six assists in 24 NASL matches for Vic Crowe's squad.