The 20-year-old, from Guernsey, came third at the British trials on Sunday to boost his World Championship hopes.
It has been a breakthrough season for Chalmers, who initially planned to run at the Island Games before pulling out.
"I actually got a phone call yesterday offering me a lane which I turned down, as crazy as that sounds," he said.
"If you're running at the Olympic Stadium, there's potentially a bit of money there as well and it would definitely have been the highest level race I've ever done in my life.
"But it's just the wrong timing, to put it plainly. I've run half-a-second quicker than I thought I would [this season] and that's suddenly opened a lot of doors, so I'm a victim of my own success a little bit."
Chalmers, who only started in the sport four years ago, ran 46.11 seconds at the 400m trials in Birmingham, just short of his personal best time of 45.64 seconds, set in June.
He missed out on automatic qualification for next month's World Championships in London, but hopes to figure as part of Great Britain's 4x400m squad.
"If they take five, six or seven athletes then, based off my performance, I'm definitely in the mix to be considered for that," he added to BBC Sport.
"But it's really out of my control now. I'll just have to wait and see whether I'm picked for that."
Cameron Chalmers says he turned down a 400m spot at this weekend's London Anniversary Games to rest before next week's European Under-23 Championships.