Saturday's 46-25 home defeat by Ulster consigned Ospreys to eighth place in the Pro12 table.
Osprey went into the game needing a win to clinch sixth spot and a place in the European Champions Cup.
"From an Ospreys point of view they'll be devastated," said Bristol head coach Holley.
Media playback is not supported on this device
"They finished as the third Welsh region and had a poor season by their standards."
Blues' 21-17 win over Edinburgh was the only Welsh success on a final weekend of matches which saw Scarlets beaten 31-15 in Munster and Dragons thrashed 47-22 by Zebre.
Scarlets will be Wales' only representatives in the top-flight Champions Cup, with Ospreys missing out for the first time since the region was created in 2003.
They will now play in the second-tier Challenge Cup along with Dragons and Blues.
Holley, who was head coach at the Swansea-based region between 2008 and 2012, says high-profile players will be particularly disappointed.
"It's the first time out of the upper echelons of Europe, how are they going to bounce back?" he asked.
"If you're Alun Wyn Jones or Dan Biggar you don't want to play in the Challenge Cup - you want to be in the top competition.
"They are used to dining at the top table and attracting signings for the following year - for recruitment it is huge."
Former Ospreys coach Sean Holley says the region will be "devastated" after missing out on top-flight European rugby next season.