The 20-year-old has made only one substitute appearance in four years with the Blues.
Brown has had loan spells with Dutch side Vitesse Arnhem, Rotherham and Huddersfield.
He scored five goals for Huddersfield last season as they won promotion to the Premier League via the play-offs.
"The natural progression for Izzy, having enjoyed such a successful season in the Championship last season, was to step up to the Premier League - and we are delighted he will be doing that with us," said Brighton manager Chris Hughton.
"He is a player with great potential and will bring a different dimension to our attacking options for the season ahead, alongside our existing strikers."
Find all the latest football transfers on our dedicated page.
Chelsea forward Izzy Brown has joined Brighton on a season-long loan - on the day he signed a four-year contract with the Premier League champions.