Notts began the day on 42-2, chasing an improbable 468 and looked set to secure a draw, led Steven Mullaney (137).
His dismissal saw Notts slump from 245-3 to 262-8 either side of tea, with two wickets in two balls for Brad Wheal (6-51) including Samit Patel first ball.
The hosts were bowled out for 291 as Mason Crane chipped in with 3-95.
Notts are now 30 points adrift of seventh-placed Durham with only four games left in their season, while Hampshire reduce their gap to safety to 16 points.
Hampshire's hero was 19-year-old Wheal, who was born in South Africa but has represented Scotland internationally.
The teenager, playing in only his ninth first-class game, started Nottinghamshire's demise and ended it in emphatic fashion.
Despite losing Michael Lumb for 17 in the first session, Mullaney and Brendan Taylor (58) then shared a fourth-wicket partnership of 162.
Just when a draw looked possible and an outside chance of a push for victory, the dismissal of Mullaney, who had scored his second Championship century at Trent Bridge in as many games, saw Notts crumble.
Harry Gurney's stump went flying as he was last man out to Wheal who secured career-best bowling figures.
Hampshire's victory was their second of the season - both coming against Notts - and they take 22 points.
Notts, however, have not won since beating Surrey in the opening round of fixtures and their nine-year stay in the top flight in serious doubt.
Hampshire beat Nottinghamshire by 176 runs at Trent Bridge to boost their survival hopes as the hosts slipped to the bottom of the Division One table.