Montanier, 51, left his job as boss of French side Rennes in January having previously had stints at Real Sociedad, Boulogne and Valenciennes.
A former goalkeeper, the Frenchman's playing career included spells with Toulouse, Nantes and St Etienne.
Forest sacked Dougie Freedman in March and ended last season with Paul Williams as temporary manager.
The East Midlands club finished 16th in the Championship table and Williams left soon afterwards.
Montanier had been linked with the coach's job at Lens, but has instead taken the opportunity to manage in England.
He took over at Real Sociedad in 2011 and guided them to a fourth-placed finish and a spot in the Champions League in 2013.
However, his contract was not renewed and he then spent three years at Rennes, who reached the Coupe de France final in 2014.
BBC Radio Nottingham's Robin Chipperfield:
"It's a time of great intrigue at the City Ground. Montanier becomes the first non-British manager appointed by the Reds - an appointment that has taken a long time to announce. Remember, Dougie Freedman was sacked on 13 March.
"One of the most interesting aspects of Montanier's tenure could be how well he works with Pedro Perreira, who was recently appointed as the club's new director of football.
"If they can forge a strong partnership, and their views on the style of football, and potential new signings dovetail, then a prosperous future is more likely.
"The success achieved by Watford and Sheffield Wednesday under Slavisa Jokanovic and Carlos Carvalhal respectively have to be the models for Forest to follow.
"There is still plenty of work to be done at Forest, but this feels like a very significant step."
Nottingham Forest have appointed Philippe Montanier as head coach on a two-year contract.