McKay, who played under Mark Warburton while the Forest boss was manager of the Scottish side, has signed a four-year deal with the Championship club.
The 22-year-old has one full cap and scored five goals in 35 Scottish Premiership appearances last season.
McKay is Forest's second summer arrival after Hibernian striker Jason Cummings.
He had one year left on his Rangers contract and was left out of their squad for the Europa League game with Progres Niederkorn on Tuesday.
A part of the Rangers side that climbed up the divisions and back into the Scottish Premier League, following their demotion to League Two in 2012, McKay made 126 appearances for the club, scoring 13 goals.
He had spells on loan at Morton and Raith Rovers, before featuring more regularly for the Gers under Warburton.
He was named in the Scottish Championship team of the year for 2015-16 and made his international debut at the end of that campaign as an 84th-minute substitute against France.
Andy Burke, BBC Scotland
McKay is a talented player who is comfortable playing on the right or the left and is strong with both feet. He likes to go at defenders, is a really creative player and can pitch in with a few goals too. He has scored some spectacular goals.
Warburton is probably the man best placed to get the best out of him. Warburton placed a lot of faith in Barrie McKay when he was Rangers manager and was rewarded with a lot of great performances.
Warburton is the kind of manager - and I think this will also benefit Cummings - that makes players feel special and encourages young players to express themselves.
He can blow hot and cold, and this is the case with a lot of wingers, but when he is on it he can be a devastating attacker.
Find all the latest football transfers on our dedicated page.
Nottingham Forest have signed Scotland international winger Barrie McKay from Rangers for an undisclosed fee, thought to be about £500,000.