In January 2004, now racing as a three-year-old, Smarty Jones was given his first major test in the Count Fleet Stakes at Aqueduct Racetrack in New York City. In the home stretch, the colt pulled away from the field to win by 5 lengths. In February, Smarty Jones was shipped to Oaklawn Park racetrack in Hot Springs, Arkansas in preparation for the Kentucky Derby. There, he won the Southwest Stakes, the Rebel Stakes (earning the fastest Thorograph number that had ever been given to a 3 year old), and the important Arkansas Derby.