British riders won six gold medals, four silvers and one bronze in Brazil, having taken 12 medals at London 2012.
World championships in between have yielded fewer medals and several rivals - including Australia's Anna Meares - have questioned the team's improvement.
"It is a little bit frustrating because there's a lot of hard work gone into that performance," Trott, 24, said.
Meares, who came 10th in the individual sprint in Rio having won the title at London, had said: "We're all just scratching our heads going 'how do they lift so much when in so many events they have not even been in contention in the World Championships?'"
German rider Kristina Vogel called Team GB's performance "questionable", while France's sprint coach Laurent Gane said he "could not understand" their improvement.
Great Britain won three silver medals at the 2015 World Championships in France and nine medals at this year's event in London in March, including five golds.
Trott, who won two golds in Rio to become Britain's most successful female Olympian, told BBC Radio 5 live: "British Cycling has always been very much an Olympic-based programme, so for us it wasn't about clearing up at the World Championships.
"It's always been around the Olympics and that's what our funding is pushed towards."
Media playback is not supported on this device
Four-time Olympic champion Laura Trott is frustrated rival riders have questioned Britain's Rio track success.