Although the show could have been "squeezed out" for another few series, the star said he thought creator Julian Fellowes "would have run out of ideas".
He was speaking at a Bafta event, where the ITV drama was being given a special honour for its global success.
The series will return this autumn and also air a final Christmas special.
"We've been blessed by having a writer who's got a fund of stories to play with, and these characters that have been fully-formed over the last six years," Bonneville said.
"To have extended it would have been, I think, a bit of a misstep."
Lord Fellowes told the BBC he made the right decision to end the drama now.
"I think you want to finish when people are still a little bit sorry that its over," he told entertainment correspondent Colin Paterson.
"You don't want to wait until it's a huge relief all round, so we decided at [series] six people would be sorry there wasn't a [series] seven and that seemed right."
However, the writer said he would not be against the prospect of a Downton Abbey film.
"I'd like the idea of a film because with a film budget you could do things you can't do on television," he said.
"You could open the whole thing up in a way that rather appeals to me. But we'll have to see."
Bonneville said he had filmed his final scenes on Monday and joked that he was now looking forward to growing a beard.
"I filmed my last scene with Michelle Dockery - Lady Mary - and we were recalling the very first scene we'd done in 2010. Who'd have thought six years later we'd still be mates?" he said.
Jim Carter, who plays butler Charles Carson in the drama, revealed co-star Laura Carmichael "gets to do the last shot" of the series as Lady Edith Crawley.
Lily James, who stars as Lady Rose Aldridge, said she was "100%" sure fans would not be disappointed with how the show ends.
"Julian Fellowes has really done well," she said. "Reading the final episode I cried a lot - in a good way - happy tears."
The special Bafta tribute, hosted by Jonathan Ross, will air on ITV later this year.
Downton Abbey actor Hugh Bonneville has said it would have been "a misstep" for the period drama to have gone on beyond its forthcoming sixth series.