The problem is that these groups support Trump and campaigned for him because they view him as sharing their beliefs and Trump did not do enough to disavow them. Recently in his speech in Belgium, Trump mentioned the importance of protecting and preserving "Western Civilization"-- if you look at the literature of the Alt Right, the Nazis and various hate groups, they use the same words about preserving "Western Civilization" which are meant to mean white European "civilization".  Many of those groups cited this language as a sign of support from Trump and their white supremacy agenda. David Duke gave a statement about the Charlottesville rally in which he said the groups at the rally would carry out the policies of Donald Trump. When Trump gave his condemnation of all sides, that angered Duke who took that as a criticism of the White supremacists-- he said Trump should not forget that his people-- whites--- elected him.