Part of it was that many of the actors were moving on, part of it was cost as Xantec mentioned.
Another factor was that the man driving the Star Trek franchise after Gene died was Rick Berman.
By the time Nemesis came out, many fans and critics had grown restless and were saying that Star Trek had become stale and lifeless under Berman's guidance, that it wasn't going boldly anywhere.
Berman was pushing forward for an eleventh film with a group of writers, but they weren't getting very far coming up with an acceptable script.
Then, as often happens, the powers that be changed and Paramount got a new president.
Their script in progress was shelved indefinitely by the new president.
A few months later Rick announced that he was walking away from Star Trek, stating that if Star Trek were ever re-energized, it would be by fresh faces that hadn't been involved before.
Over the years several people proposed reboots of the series that were all rejected by Paramount, including Bryan Singer (who failed to reboot Superman successfully), J. Michael Straczynski (Babylon 5), Jonathan Frakes (Riker) and William Shatner.
They finally got an acceptable proposal for a reboot from JJ Abrams and that brought us to where we are now.
scifi.stackexchange.com/users/1551/BBlake
Honestly, there's never going to be a live-action film resolution of B4 - the whole reason Brent Spiner wanted to leave the show is that he felt he was growing too visibly aged to continue playing a character that wasn't supposed to age
(who looks like he did in the mid-80s).
The entire series has been suffering for a while from a lack of new ideas, and it will likely never live up to the expectations of fans.
DS9 and Voyager aren't really suited to feature-length movies (Voyager especially, since it has canonically finished it's journey), and Enterprise was a disappointment to the studio (not to mention many fans), so it's low on the list for a live-action movie.
In short, the studios won't do it because there's no money in it.
And the direction they seem to be heading is for the sure-thing: retreading TOS with the reboot series.
scifi.stackexchange.com/users/656/Jeff
After fifteen years of working on Star Trek (1987 - 2002)
it is likely that (A) the actors were ready to move on and (B)
it was becoming very expensive to hire the actors.
Not to mention that Star Trek Nemesis came out about mid-way through Star Trek Enterprise, arguably during the waning years of the second Star Trek Era (Voyager had just ended, Enterprise wasn't doing so hot and people were pretty much Trekked out).
scifi.stackexchange.com/users/1234/Xantec
Mainly because the opinion of most Star Trek fans I know (including myself), Insurrection and Nemesis were bad movies and ended up being financial flops.
Look to The Numbers or Box Office Mojo for specific numbers and poor reviews of both films.
scifi.stackexchange.com/users/11322/Tony Koonts
I blame the budget that Paramount gave Next Generation to work with.
If Paramount would have given them the same as the re-boot from JJ Abrams, they could have made better movies.
scifi.stackexchange.com/users/13074/Bob
I believe that cost and creative issues ended the TNG series of movies.
The budget for Star Trek: Nemesis was $67 million.
(It earned $67.3 million.)
The budget for StarTrek (2009) was $150 million.
I suspect a $70 - $75 million budget in the hands of someone like Jonathan Frakes would have done quite well.
scifi.stackexchange.com/users/16316/Dennis Donlon
The Next Generation crew filmed Generations, First Contact, Insurrection and Nemesis. Then nothing. Why was there no follow up film?  I was expecting some kind of resolution to the Data / B4 story.
Why didn't they make a new TNG movie after Nemesis?
scifi.stackexchange.com
<star-trek><star-trek-tng>
scifi.stackexchange.com/questions/6071
scifi.stackexchange.com/users/143/Reinstate Monica - Goodbye SE
It is believed that some of the actors wanted to move on to other things and/or were getting older. Cost and creative issues may also have been factors, along with the death of Gene and the performance of the previous films. Most of the actors were ready to move on with the series. Aside from that the original director had died, which was one of the reasons the franchise started to become dull and boring. In the end the franchise wasn making too much of a loss in the box office so another film was never made.
Most of the actors were ready to move on from the series. An example being Brent Spiner wanted to leave the franchise is that he felt he was growing too visibly aged to continue playing a character that wasn't supposed to age The original director died leaving Rick Berman to take his place. Aside from Star Trek being rediculed by fans for becoming dull and boring it also started to flop in the box office turning up very little to no profit whatsoever.