After 186 league appearances, he left St Andrew's in March 1983 and joined Division 3 side Portsmouth. He was involved in Portsmouth's Division 3 title run-in that season and he also played his part, as Portsmouth went on to finish fourth in Division 2 in both the 1984–85 and 1985–86 seasons. Portsmouth were finally promoted to Division 1 in the 1986–87 season, when they finished as runners-up to Derby County. Pompey only stayed in the top flight for one season, as they finished 19th and were relegated. Dillon had made 206 league appearances for Portsmouth before leaving Fratton Park in July 1989.