context information for crucial aspects of the data acquisition procedure creates a challenging task objectifying the evaluation results. Specialized fraud detection methods are commonly used in an attempt to pre-filter the recorded corpus and extract a usable subset of data. McGuinness and O’Connor [@mcguinness2010comparative] proposed an evaluation of via extensive user experiments. In these experiments, users are shown images with descriptions of the objects they are required to extract. Then, users mark foreground and background pixels utilizing a platform designed for this purpose. These acquisitions are more time-consuming and cost intensive than , since they require a constant involvement of users. However, the study’s creators are able to control many aspects of the data recording process, which enables detailed observations of user reactions. The data samples recorded are a representative subset of the focus group of the finalized system. A user study aims at maximizing repeatability of its results. In order to increase the objectivity of the evaluation in this work, a user study is chosen to be conducted. The study is described in **Sec.**\[sec:usability\_test\_setup\].

State-of-the-art Evaluation of Interactive Segmentation Systems
---------------------------------------------------------------

### Segmentation Challenges

In segmentation challenges like  [@van20073d] (mainly) fully automated approaches are competing for the highest score 