She broke Serena, 32, in the sixth game of the final set to beat her for the first time in five years.
Venus, ranked 26 in the world, will play Agnieszka Radwanska after the third seed overcame Ekaterina Makarova 7-6 (7-1) 7-6 (7-3) in Montreal.
Defending champion Serena later tweeted of the result: "Big sister taught little sister a lesson."
Venus said: "I think for both of us, what's so unique about the situation is that we're both very good players.
"We both know when we walk out there, it's not like you're guaranteed a win. I think that's what makes it challenging for both of us."
Serena still leads 14-11 in the head-to-head with her sister, who she played for the first time on the professional circuit 16 years ago,
Ahead of the final, Radwanska, who has won her last two matches against Venus, said: "She's definitely on fire. Definitely she's playing amazing tennis this week.
"She beat a lot of good players on the way to the final, especially Serena. It's not going to be the easy one."
The win in Canada will be a boost to Venus, 34, in her build-up to the final Grand Slam of the year, the US Open, which starts at Flushing Meadows on 25 August.
Venus Williams has beaten world number one and sister Serena 6-7 (2-7) 6-2 6-3 to reach the final of the Rogers Cup.