The Timed Up and Go Test (TUG) is a simple fall risk screening test that covers basic functional movement; thus, quantifying the subtask movement ability may provide a clinical utility. The video-based system allows individual’s movement characteristics assessment. This study aimed to investigate the concurrent validity and test-retest reliability of the video-based system for assessing the movement speed of TUG subtasks among older adults. Twenty older adults participated in the validity study, whilst ten older adults participated in the reliability study. Participant’s movement speed in each subtask of the TUG under comfortable and fast speed conditions over two sessions was measured. Pearson correlation coefficient was used to identify the validity of the video-based system compared to the motion analysis system. Intraclass correlation coefficient (ICC3,2) was used to determine the reliability of the video-based system. The Bland-Altman plots were used to quantify the agreement between the two measurement systems and two repeatable sessions. The validity analysis demonstrated a moderate to very high relationship in all TUG subtask movement speeds between the two systems under the comfortable speed (r = 0.672–0.906, p < 0.05) and a moderate to high relationship under the fast speed (r = 0.681–0.876, p < 0.05). The reliability of the video-based system was good to excellent for all subtask movement speeds in both the comfortable speed (ICCs = 0.851–0.967, p < 0.05) and fast speed (ICCs = 0.720–0.979, p < 0.05). The Bland-Altman analyses showed that almost all mean differences of the subtask speed of the TUG were close to zero, within 95% limits of agreement, and symmetrical distribution of scatter plots. The video-based system was a valid and reliable tool that may be useful in measuring the subtask movement speed of TUG among healthy older adults.