Her Majesty wore blue at the State Opening of Parliament but by the time she arrived at the races, she had changed into yellow.
William Hill spokesman Rupert Adams said his betting firm had taken just 13 bets on yellow.
The Queen wore green on Monday, when she attended the opening of the event.
Mr Adams said: "After seeing the Queen wear a blue hat for her speech in Parliament, many punters expected her to remain in her clothing for Royal Ascot.
"However, they were left disappointed when she was seen in her procession in a yellow hat, the second favourite in our betting."
The Queen had to attend the second day of the racing event on Wednesday without the Duke of Edinburgh who was admitted to hospital on Tuesday evening.
The Queen's quick outfit change before heading to Royal Ascot for a second day of racing has left many punters stumped and bookmakers smiling.