but this design was slow to ripen , and was never wholly achieved ; for men ( it is said ) were at first very few in number , whereas morgoth grew afraid of the growing power and union of the eldar and came back to angband , leaving behind at that time but few servants , and those of less might and cunning .