Rossi, who raced in F1 for Marussia last year, came through from 11th on the grid to claim an unlikely win.
The 24-year-old was running 33rd and last at one stage but took a gamble on strategy before running out of fuel just moments after the chequered flag.
"I've no idea how we pulled that off," the Andretti Autosport driver said.
Rossi made his final pit stop for fuel 36 laps from the end of the race, even though the expected mileage for a single fuel stint around the 2.5-mile track is around 32 laps.
"I'll cherish the fact that at one point we were 33rd," Rossi added. "We rolled the dice and came through and made it happen.
"This is unbelievable. I have no doubt it's going to change my life.''
IndyCar veteran Helio Castroneves of Brazil was the last rookie to win the Indy 500 15 years ago.
Rossi's team-mate Carlos Munoz had to settle for second place after being forced to pit for fuel from the lead with four laps remaining, with American Josef Newgarden third.
James Hinchcliffe, the polesitter who missed this race last year after a near-fatal accident in a practice session, finished seventh.
Max Chilton, like Rossi a former F1 driver with Marussia, came 15th on his Indy 500 debut, while fellow Briton Stefan Wilson - brother of Justin Wilson, who lost his life in an IndyCar race nine months ago - retired after 119 of the 200 laps.
Britain's Pippa Mann - the only woman in the race - finished 18th, two places behind Yorkshireman Jack Hawksworth.
American Alexander Rossi won the 100th Indianapolis 500 on Sunday to become the first driver to win the race on his debut since 2001.