Batsmen Alex Hales, Gary Ballance and James Vince made a combined 498 runs in the Test series against Pakistan, which ended 2-2 after defeat at The Oval.
Bayliss said players had been given a "decent run" to prove themselves.
"We've got around a month before we get together and discuss the next series so there will be tough decisions to make."
Ballance, recalled by England against Pakistan after Nick Compton took a break from cricket, had the highest series average of the trio at 27.85 with the 26-year-old's high score a 70 at Edgbaston.
Opener Hales, 27, averaged just 18 and, to add to an indifferent series, was fined 15% of his match fee for going into the match referee's office after being dismissed to a catch he felt had not been taken cleanly by Pakistan's Yasir Shah.
Vince, 25, was caught driving on day three of the 10-wicket defeat at The Oval to just his third delivery, as the hosts fought to set a target.
It prompted former England batsman and BBC pundit Geoffrey Boycott to call it the "stupidest shot you can imagine", and add: "For me it's the end of his Test career."
Former England captain Michael Vaughan said he had been "frustrated" by England's performance, which came just a few days after a 141-run win at Edgbaston had given them a 2-1 series lead and the possibility of topping the Test rankings.
"Vince I don't think will make it, which is sad because I do think he's a good player, but it's a mental thing," Vaughan said on Test Match Special.
"Hales goes on the plane, but what I want to see from him is more aggression on the pitches in Bangladesh and India."
England's Australian coach Bayliss did not offer much encouragement, however, to rivals from the County Championship.
"There doesn't seem to be anyone crying out to be selected above anyone else," he said.
"I think I heard [captain Alastair Cook] say the two or three guys we've got in the team have been very good players over the last two or three years, experienced players at county level, and they have found it tough to jump up to this level.
"They have shown bits and pieces, but I think it just proves that the jump between first-class cricket and Test cricket is a big one."
England next play five one-day internationals and a Twenty20 against Pakistan, starting at Southampton on 24 August.
England face some "tough" selection decisions for their winter tours to Bangladesh and India, coach Trevor Bayliss says.