The Spanish world number four had won all 17 of his matches on clay this year, including against Thiem in the Barcelona Open and Madrid Open finals.
But Thiem, ranked seventh, dominated on Friday, frequently forcing Nadal behind the baseline to set up a 6-4 6-3 win.
He will face world number two Novak Djokovic of Serbia in the semi-finals.
Djokovic beat Argentina's Juan Martin del Potro 6-1 6-4 after their rain-affected match was completed on Saturday.
American John Isner beat Croatia's Marin Cilic 7-6 (7-3) 2-6 7-6 (7-2) and will meet Alexander Zverev in the last four, after the 20-year-old German saw off Canada's Milos Raonic 7-6 (7-4) 6-1.
After triumphs in Monte Carlo, Barcelona and Madrid, 30-year-old Nadal was looking to win his fourth straight title on clay this year before the French Open, which starts on 28 May.
However, Thiem could now prove to be a major obstacle as Nadal aims to win a 10th title at Roland Garros.
The 23-year-old Austrian raced into a 5-1 lead with a double break and held his nerve to serve out the first set after Nadal had broken back to make it 5-4.
He continued to thrive by playing closer to the baseline than in recent defeats by Nadal, breaking in the seventh game of the second set.
Thiem then saved break points to hold, before breaking Nadal again to claim only his second win in six meetings with the 14-time Grand Slam champion.
Defending French Open champion Garbine Muguruza came through to claim a 6-2 3-6 6-2 victory over Venus Williams, who beat Britain's Johanna Konta in the last 16.
The 23-year-old Spaniard took the first set with a double break before American Williams, 36, hit back to level, but Muguruza responded in impressive fashion to win the decider with another double break.
In the semi-finals, third seed Muguruza faces eighth seed Elina Svitolina after the Ukrainian upset second seed Karolina Pliskova of the Czech Republic with a 6-2 7-6 (11-9) victory.
Romania's Simona Halep reached her third straight clay semi-final as she won 6-2 6-4 against Estonia qualifier Anett Kontaveit, who knocked out world number one Angelique Kerber in round two.
Halep, 25, will play Kiki Bertens in the last four after the Dutchwoman beat unseeded Australian Daria Gavrilova 6-3 6-3.
Seven-time champion Rafael Nadal is out of the Italian Open after a straight-set defeat by Dominic Thiem of Austria in the quarter-finals in Rome.