they gathered in mournful crowds upon the western shores , shivering in the cold wind , and their first complaints and anger were against the master , who had left the town so soon , while some were still willing to defend it .