In 1964, the Giants sold Larsen to the Houston Colt .45s, who pressed Larsen back into a starting role. He responded with a 4–8 record and a 2.27 ERA. In 1965, he was traded to the Orioles for Bob Saverine and cash. The Orioles released him prior to the 1966 season, and he pitched the year for the Phoenix Giants of the Class-AAA Pacific Coast League (PCL) in the San Francisco minor league organization. Larsen was on the Chicago Cubs roster for two weeks of the 1967 season, pitching only four innings in what would be his final stint in the major leagues. He spent the rest of the season with the Dallas-Fort Worth Spurs of the Class-AA Texas League. Larsen started the 1968 season in the Cubs' minor league system, pitching for the San Antonio Missions of the Texas League and Tacoma Cubs of the PCL, before retiring in the summer.