Overall you are right that restrictions on immigration/travel are not unconstitutional.  However this executive order is problematic not just because its roll out was flawed.  The order includes exceptions for religious minorities coming from muslim countries (e.g. christians) and many in the Trump administration have referred to it as a ban on muslims.  Its this apparent religious test that is the basis of the constitutional challenge.