Recent visual analytics systems make use of multiple machine learning models to better fit the data as opposed to traditional single, pre-defined model systems. However, while multi-model visual analytic systems can be effective, their added complexity poses usability concerns, as users are required to interact with the parameters of multiple models. Further, the advent of various model algorithms and associated hyperparameters creates an exhaustive model space to sample models from. This poses complexity to navigate this model space to find the right model for the data and the task. In this paper, we present Gaggle, a multi-model visual analytic system that enables users to interactively navigate the model space. Further translating user interactions into inferences, Gaggle simplifies working with multiple models by automatically finding the best model from the high-dimensional model space to support various user tasks. Through a qualitative user study, we show how our approach helps users to find a best model for a classification and ranking task. The study results confirm that Gaggle is intuitive and easy to use, supporting interactive model space navigation and automated model selection without requiring any technical expertise from users. Visual analytic (VA) techniques continue to leverage machine learning (ML) to provide people effective systems for gaining insights into data [27] . Systems such as Interaxis [36] help domain experts combine their knowledge and reasoning skills about a dataset or domain with the computational prowess of machine learning. These systems are traditionally designed with a pre-defined single ML model that has a carefully chosen learning algorithm and hyperparameter settings. Various combination of learning algorithms and hyperparameters give rise to a vast number of different model types (see Table 1 ). These different models constitute an exhaustive model space from which unique models can be sampled using a distinct combination of a learning algorithm and associated hyperparameters. For example, support vector machine (SVM) models have many options for kernel functions (i.e., linear, poly or radial) and hyperparameters (i.e., C (regularization parameter), γ (kernel coefficient), etc. ). When a model is correctly chosen for the phenomena, task, data distribution, or question users try to answer, existing VA techniques can effectively support users in exploration and analysis. However, in cases where the right model to use for a problem is not known a priori, one needs to navigate this model space to find a fitting model for the task or the problem. To combat this, recent VA systems use multiple ML models to support a diverse set of user tasks (e.g., Regression, Clustering , etc. [15, 21, 17, 63] ). For example, the VA system Clustervision [39] allows users to inspect multiple clustering models and select one based on quality and preference. Similarly, Snowcat [16] allows inspecting multiple ML models across a diverse set of tasks, such as classification, regression, time-series forecasting, etc. However, these multimodel systems are often more complex to use compared to single-model alternatives (e.g, in Clustervision users need to be well-versed with cluster model metrics and shown models.) We refer to this complex combination of parameter and hyperparameter settings as model space, as there are a large number of models that can be instantiated in this hyperdimensional space. Further, the interactive exploration of different parameter and hyperparameter combinations can be referred to as model space navigation. Our definition of model space is related to the work by Brown et al. [14] where they presented a tool called ModelSpace to analyze how the model parameters have changed over time during data exploration. In this paper we present Gaggle, a visual analytic tool that provides the user experience of a single-model system, yet leverages multiple models to support data exploration. Gaggle constructs multiple classification and ranking models, and then using a bayesian optimization hyperparameter selection technique, automatically finds a classification and ranking model for users to inspect, thus simplifying the search for an optimal model. Furthermore, our technique utilises simple user interactions for model space navigation to find the right model for the task. For example, users can drag data items into specific classes to record classification task's user preferences. Similarly, users can demonstrate that specific data items should be higher or lower in rank within a class by dragging them on top of each other. Gaggle uses ML to help users in data exploration or data structuring tasks, e.g, grouping data in self-defined categories, and ranking the members of the group based on their representativeness to the class. For example, a professor may want to use a tool to help categorize new student applications in different sets, and then rank the students in each set. Similarly, a salesman may want to cluster and rank potential clients in various groups. These problems fall under classification tasks in ML; however, unlike a conventional classification problem, our use case specifically supports interactive data exploration or data structuring, the models constructed are not meant to predict labels for unseen data items in future. Using this workflow, we expect our technique guards against possible model overfitting incurred due to adjusting the models confirm to specified user preferences. Furthermore, Gaggle addresses a common problem of datasets that either lack adequate ground truth, or do not have it [61, 72, 49] . To resolve this problem, Gaggle allows users to iteratively define classes and add labels. On each iteration, users add labels to data items and then build a classifier. We conducted a qualitative user study of Gaggle to collect user feedback on the system design and usability. The results of our study indicate that users found the workflow in Gaggle intuitive, and they were able to perform classification and ranking tasks effectively. Further, users confirmed that Gaggle incorporated their feedback into the interactive model space navigation technique to find the right model for the task. Overall, the contributions of this paper include: • A model space navigation technique facilitated by a Bayesian optimization hyperparameter tuning and automated model selection approach. • A VA tool Gaggle, that allows interactive model space navigation supporting classification and ranking tasks using simple demonstration-based user interactions. • The results of a user study testing Gaggle's effectiveness. Large Model Search Space: Searching models by combining different learning algorithms and hyperparameters leads to an extremely large search space. As a result, a small set of constraints on the search process would not sufficiently reduce the space, leading to a large number of sub-constrained and ill-defined solutions. Thus, how many interactions are considered optimal for a given model space? In this work, we approached this challenge by using Bayesian optimization for ranking models. However, larger search spaces may pose scalability issues while too many user constraints may "over-constrain" models leading to poor results. In this paper, we present an interactive model space navigation approach for helping people perform classification and ranking tasks. Current VA techniques rely on a pre-selected model for a designated task or problem. However, these systems may fail if the selected model does not suit the task or the user's goals. As a solution, our technique helps users find a model suited to their goals by interactively navigating the high-dimensional model space. Using this approach, we prototyped Gaggle, a VA system to facilitate classification and ranking of data items. Further, with a qualitative user study, we collected and analyzed user feedback to understand the usability and effectiveness of Gaggle. The study results show that users agree that Gaggle is easy to use, intuitive, and helps them interactively navigate the model space to find an optimal classification and ranking model.
Gaggle, an interactive visual analytic system to help users interactively navigate model space for classification and ranking tasks. A new visual analytic system which aims to enable non-expert users to interactively navigate a model space by using a demonstration-based approach. A visual analytics system that helps novice analysts navigate model space in performing classification and ranking tasks.