but the unforgivable bit had been learning that they all , including his little brother , had used their abilities to control him to keep him from realizing what they were before he was old enough to decide if he wished to join them .