Washington (CNN)The day after 25 House Republicans attempted to prevent Speaker John Boehner from getting a third term, Boehner says he's starting "a family conversation" with his critics. The Speaker and his allies were furious with GOP members who publicly rebuked him on the opening day of the new session of Congress. In front of live television cameras they voted for someone else or declined to back him for the job. Hours after the vote the message was clear that those defectors would be punished. Two Florida Republicans who opposed Boehner - Rep. Daniel Webster, who got 12 votes for his own bid to become Speaker, and Rep. Richard Nugent, who voted for him - were removed from serving on the House Rules committee. That panel is essentially an extension of the Speaker's office since it decides which bills go to the floor and sets parameters for debate.
@highlight
More than two dozen Republicans voted against Boehner gaining another term as Speaker
@highlight
Boehner plans conversations with GOP critics
@highlight
Some critics are being removed from key committee seats
@placeholder sources told CNN that prior to the vote for Speaker, they had a good idea of the group that would vote for other candidates.