 that the traditional statistical aggregation cannot offer. In particular, we considered three visual designs, stacked line graph and ThemeRiver [27,28] for aggregated opinions and pixel-based visualization for constituent predictions of individual models. During our effort for using such visual designs to support our ensemble ML workflows, we discovered a variant of ThemeRiver, which is more effective than the original ThemeRiver and stacked line graph. Fig. 1 shows an example of such a variant, and a section of the music is also shown in a stacked line graph and an original ThemeRiver plot. This variant allows viewers to observe the change in the opinions of ML models more easily, and perceive whether the majority opinion has passed a critical threshold more precisely. We refer to this new variant as dual-flux ThemeRiver.
While we use the dual-flux ThemeRiver variant to address the need for temporal and collective mood visualization in challenges (a) and (b), we use pixel-based visualization to address the need for visualizing the performance of many individual models in the above challenge (c). The combination of dual-flux ThemeRiver and pixel-based visualization also enables the juxtaposition of the overall mood flow and the contribution of individual models. The overall contribution of this work is the introduction of appropriate visualization techniques into ensemble ML workflows. In particular, we firstly introduce dual-flux ThemeRiver (Section 4.1) as a novel visual representation to enable effective visual communication of multiple mood predictions by many ML models. This is crucial to address the above challenges (a) and (b) and to support pixel-based visualization in terms of challenge (c). Secondly, we provide ML model developers with investigative visualization for evaluating the performance of many models in a context sensitive to individual parts of the music (Section 4.2). This allows ML developers to observe a huge amount of information that statistical indicators cannot convey, facilitating the transformation from actionable information to ML-developmental decisions (e.g., weighting schemes in ensemble ML). Hence, this addresses challenge (c) that is routinely encountered in developing an ensemble ML technique involving many models.
We present two case studies (Sections 5.1 and 5.2) to demonstrate the usability of the proposed techniques. We conduct an analytical evaluation to reason the shortcomings of the existing solutions and the advantages and potential limitations of the proposed techniques (Section 5.3). We report the evaluation by music and ML experts, confirming the merits of this work (Section 5.4).

RELATED WORK
This paper relates to visualization and visual analytics (VIS), music mood classification and machine learning (ML) in several aspects. In this section, we give a brief overview of each aspect.  [20]. There are many versions of line graphs for visualizing time series. Harris [25] proposed a visual design using area graph stacking lines on top of each other as a variant of the original layer area graph. Among variants of line graphs, ThemeRiver is a popular visual design and is highly relevant to this work. Harve et al. [27,28] proposed the ThemeRiver design for visualizing textual data or large document collections. There are many variants of ThemeRiver, including the spiral variant by Jiang et al. [39], a multi-dimensional variant by Kalouli et al. [40], a variant with trend line and change indicators by Shi et al. [72], variants with labelling and wiggling deformation by Byron and Wattenberg [10], a variant with heatmap by Hashimoto and Matsushita [26]. With all these designs, viewers have to determine the ordering of different layers by visually estimating the thickness of each layer at each time step.
To depict ordering change explicitly, Javed et al. [38] introduced an overlapped design, where multiple layers are superimposed and ordered from lower values (at front) to higher values (at back). However, while the change of ordering can be perceived easily, the visual estimation of the thickness of each layer at each step requires cognitive reasoning about the perceived color-filled shapes as well as the occluded shapes. Ensemble Data Visualization. Ensemble data visualization is a collection of visual designs and techniques for visualizing data derived by ensemble modelling and simulation. Wang et al. provided a survey on VIS techniques for ensemble data [80]. A variety of VIS techniques have been developed, including a collection of interactive visualization techniques by Potter et al. [61]; topology preserving mappings by Baruque et al. [2] an approach called Noodles for visualizing uncertainty of ensemble weather modelling by Sanyal et al. [70], a cluster-based ensemble visualization technique by Kumpf et al. [43], several visual representations under a Lagrangian framework by Hummel et al. [34], contour boxplots for uncertainty visualization by Whitaker et al. [81], 3D animation of integrated ensemble data by Phadke et al. [59], a multi-view tool with continuous parallel coordinate plots and ensemble bars by Chen et al. [11], a multi-view tool with geospatial visualization, spaghetti plots, cluster-based parallel coordinates by Jarema et al. [37], and a multi-view tool with plots for displaying time-hierarchical clustering, stacked spatiotemporal data, time-step animation, and space-time surface by Ferstl et al. [21].
Because almost all these works deal with complex outputs (e.g., 3D geometry and spatiotemporal data), much focus was placed on displaying and exploring the output data resulting from ensemble simulation. Our work differs from these works in two aspects. (i) While the predictions in each time series are simple nominal values, the temporal dimension contains much more semantically-meaningful information.
(ii) There is a stronger need to observe individual model's predictions at individual time steps and there are 210 such models. For these two reasons, we focused on the ThemeRiver and pixel-based visualization.
Visualization for Machine Learning. Sacha et al. [68] provided an ontology for showing various places in ML workflows which can benefit from visualization, followed by Tam et al. [75], who used information theory to estimate the benefit of visualization in ML workflows. We are going to apply a similar strategy for ML. Rauber et al. [64] proposed an overview visual design for comparing model structures, in which we find the ensemble models for time-series data in music should also have similar functions and to compare within the model. In the VIS4ML ontology outlined by Sacha et al. [68], this work is part of the step of "Evaluate-Model" in ML workflows, including the process blocks "Model-Testing", "Quality-Analysis", and "Understanding-Model". Our contributions include a new general-purpose visual design as well as new application-specific solutions and experiences.
Ensemble Machine Learning Models. Ensemble ML techniques have been developed to address the problem that a single ML model may not be able to reach the required accuracy. Dietterich et al. [18], Polikar et al. [60], and Sagi et al. [69] provided surveys on ensemble ML techniques. These surveys showed that the majority of ensemble ML models were designed to improve model prediction by combining different opinions offered by models in the ensemble. The aggregated decision is commonly obtained using a weighting function or a machinelearned meta-model. Opitz and MaClin [55] studied how each classifier and dataset would affect the results of ensemble classification. Rokach et al. [66] discussed various ensemble classifiers techniques, including their weighting functions. Many weighting functions have been proposed, including distribution summation by Clark et al. [16] and performance weighting by Opitz [56]. For meta-learning, "stacking" is a popular approach developed by Wolpert et al. [83], and "mixture of experts" is another popular method proposed by Jacobs et al. [35]. Omari and Figueiras-Vidal [54] proposed a post-aggregation method with combined weighting and meta-learning.
In the ensemble ML literature, some researchers reported the applications of multi-label classification, e.g., semantic annotation [62] and music mood classification [82]. There are mainly two approaches for multi-label classification [77]: algorithm adaptation and problem transformation. With the algorithm adaptation approach, multi-label predictions are generated directly by multi-label classification models, e.g., kNN [88], adaboost [71] and neural network [87]. With the problem transformation approach, a multi-label problem is decomposed into multiple single-label problems [6]. The Binary Relevance (BR) method considers each label as an independent class and trains a single-label model independently [51]. The Label Powerset (LP) method considers different a new combination of labels as a new class [78]). The RAkEL method reduces the complexity of LP by using an ensemble of multiple LP classifiers [79]. The Classifier Chain (CC) method adapts BR by chaining a set of independent models in a particular order [65].
In this work, we use ensemble ML for multi-label classification and we adopted the problem transformation approach.

Machine Learning for Music and Music Mood Classification.
Basili et al. [3] proposed several ML methods for music classification and explored the impact of different music features on the model accuracy. In the literature, ML techniques used for music classification include: support vector machine by Xu et al. [49,85], adaboost by Bergstra et al. [4], and transformer-based methods by Zhuang et al. [90,92]. Choi et al. [15] proposed to use topic modeling tech-niques for categorizing users' interpretation of song lyrics for music classification.
Laurier et al. [46] used support vector machine for music mood classification. Hu