If permissions are being inherited from the parent, you cannot remove users at this securable object. If you want to delete users and SharePoint groups from the parent securable object (which this securable object inherits those permissions from), you must manage the permissions of the parent.