regarding a predefined image quality metric. Semi-automatic methods are allowed for submission if the manual interaction with the test data is strictly limited to pre-processing and (single seed point) initialization of an otherwise fully automated process. may be included into the contests’ final ranking, but are regarded as non-competing, since the structure of the challenges is solely designed for automated approaches. The challenge [@litjens2014evaluation] had a separate category for proposed interactive approaches, where the user (in this case, the person also describing the algorithm) may add an unlimited number of hints during segmentation, without observing the experts’ ground truth for the test set. No group of experts was provided to operate the interactive method for comparative results. The submitted interactive methods’ scores in the challenge’s ranking are therefore highly dependent on the domain knowledge of single operating users and can not be regarded as an objective measure.

### Comparisons for Novel Segmentation Approaches

In principle, with every new proposal of an interactive segmentation algorithm or interface, the authors have to demonstrate the new method’s capabilities in an objective comparison with already established techniques. The effort spent for these comparisons by the original authors varies substantially. According to [@kohli2012user], many evaluation 