White Supremacists, White Nationalists, US Nazi party(ies), Alt-right, etc.  Just as members of ISIS, or Al Quaeda, or the PLO can and should be presumptively considered "Terrorists" for national security purposes, SO SHOULD the US White supremacist organizations.
For the Middle Eastern groups, their "Terrorist" labels are earned by the acts of their individual, bomb detonating, CAR RAMMING members.
So to must US national security forces and administration attribute and view the acts of individual members of White Supremacists organizations.
White Supremacists have, for the 8 years prior to Trump, been an open and direct threat to not only American civility, but indeed the sitting American President.
That threat is, perhaps, something that is at the forefront of Trump's mind.  
If he dares call White Supremacists "Terrorists", how can he go to his last bastion of political support - his rallies - without fearing violent reprisal from a member of the audience?
TERRORISTS own POTUS.