# An Explanatory Model Steering System for Collaboration between Domain
Experts and AI

Aditya Bhattacharya  0000-0003-2740-039X<EMAIL_ADDRESS>KU
LeuvenLeuvenBelgium ,  Simone Stumpf  0000-0001-6482-1973
<EMAIL_ADDRESS>University of GlasgowGlasgowScotland, UK and
Katrien Verbert  0000-0001-6699-7710<EMAIL_ADDRESS>KU
LeuvenLeuvenBelgium

(2024)

###### Abstract.

With the increasing adoption of Artificial Intelligence (AI) systems in high-
stake domains, such as healthcare, effective collaboration between domain
experts and AI is imperative. To facilitate effective collaboration between
domain experts and AI systems, we introduce an Explanatory Model Steering
system that allows domain experts to steer prediction models using their
domain knowledge. The system includes an explanation dashboard that combines
different types of data-centric and model-centric explanations and allows
prediction models to be steered through manual and automated data
configuration approaches. It allows domain experts to apply their prior
knowledge for configuring the underlying training data and refining prediction
models. Additionally, our model steering system has been evaluated for a
healthcare-focused scenario with 174 healthcare experts through three
extensive user studies. Our findings highlight the importance of involving
domain experts during model steering, ultimately leading to improved human-AI
collaboration.

Explainable AI, XAI, Interactive Machine Learning, IML, Explanatory Model
Steering

††journalyear: 2024††copyright: rightsretained††conference: Adjunct
Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and
Personalization; July 1–4, 2024; Cagliari, Italy††booktitle: Adjunct
Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and
Personalization (UMAP Adjunct ’24), July 1–4, 2024, Cagliari, Italy††doi:
10.1145/3631700.3664886††isbn: 979-8-4007-0466-6/24/07††ccs: Human-centered
computing Human computer interaction (HCI)††ccs: Human-centered computing
Interaction design††ccs: Computing methodologies Machine learning

Figure 1. Our Explanatory Model Steering system consists of four aspects: (1)
an explanation dashboard for providing explainability of prediction models,
(2) domain expert driven model steering, (3) manual and automated data
configuration approaches for fine-tuning the training data, and (4) updating
prediction models and explanations on the configured data.

[Explanatory Model Steering for healthcare]Our Explanatory Model Steering
system consists of four aspects: (1) an explanation dashboard for providing
explainability of prediction models, (2) domain expert driven model steering,
(3) manual and automated data configuration approaches for fine-tuning the
training data, and (4) updating prediction models and explanations on the
configured data.

##  1\. Introduction

The growing popularity of Artificial Intelligence (AI) systems in diverse
application domains has led to an increased demand for including domain
experts with little to no machine learning (ML) knowledge in the process of
debugging and improving prediction models (Rojo et al., 2021; Bhattacharya et
al., 2024; Bayer et al., 2021; Kulesza et al., 2010, 2015; Teso et al., 2022).
Moreover, prior researchers have noted the advantages of a thorough domain
knowledge possessed by domain experts for debugging and improving prediction
models (Feuerriegel et al., 2020; Schramowski et al., 2020). For instance, in
healthcare, healthcare experts generally have a better understanding of
patient’s medical data, which is crucial for identifying biases and
limitations in the training data used by prediction models (Bhattacharya et
al., 2024, 2023b). Additionally, an effective collaboration with prediction
models allow domain experts to better understand them and have higher
confidence in their predictions (Lakkaraju et al., 2022).

To facilitate effective collaboration between domain experts and AI, we
present an explanatory model steering system that relies on diverse
Explainable AI (XAI) methods for explaining various aspects of ML systems
(Adadi and Berrada, 2018; Bhattacharya, 2022; Wang et al., 2019; Pawar et al.,
2020; Guidotti et al., 2018; Anik and Bunt, 2021; Lakkaraju et al., 2022) and
allows user-in-the-loop model steering for fine-tuning prediction models (Wang
et al., 2022; Dudley and Kristensson, 2018; Teso et al., 2022; Teso and
Kersting, 2019; Schramowski et al., 2020; Bertrand et al., 2023). The approach
of explanatory steering provides a proficient human-centric solution to the
challenges of obtaining rich feedback from domain experts for improving
prediction models (Teso et al., 2022). More specifically, this approach
facilitates domain experts in a better contextual understanding of the
training data through interactive explanations. Unlike popularly adopted one-
off explanations such as feature importance explanations (Bhattacharya, 2022;
Adadi and Berrada, 2018) or saliency maps (Simonyan et al., 2014;
Bhattacharya, 2022; Adadi and Berrada, 2018), multifaceted explanations (i.e.,
combination of different types of data-centric and model-centric explanations)
(Bhattacharya et al., 2024) included in our system, allow domain experts in
identifying potentially misleading and biased predictors that can impact the
prediction model.

Additionally, the system allows domain experts to steer prediction models by
configuring the training data. It includes two distinct approaches for
configuring the data: (1) manual configuration and (2) automated
configuration. The manual configuration provides more control over the
predictor variables. It allows domain experts to remove corrupt, biased, or
unimportant predictor variables. With the manual configuration, domain experts
can modify the upper and lower limits of predictor variables to filter
anomalies present in the data. Meanwhile, the automated configuration requires
less effort to configure the training data. It can identify data issues and
offer potential corrections through automated algorithms. Furthermore, our
explanatory model steering system has been evaluated for a healthcare-focused
scenario with 174 healthcare experts across three distinct user studies, each
analysing different components of the system. The results of our user studies
underscore the importance of combining different types of global explanations
during model steering. Additionally, we found that domain experts with more
domain experience are more confident and effective during model steering with
manual configurations. This paper introduces our healthcare-focused model
steering system, detailing its components and the technical architecture
adopted in its development. A demonstration of this system can be accessed on
YouTube. 111Demo video is available on YouTube:
https://www.youtube.com/watch?v=-9itqTkyQ6s.

##  2\. Explanatory Model Steering System

This section first discusses the two main components of our explanatory model
steering system: (1) a multifaceted explanation dashboard that combines
different types of explanation methods and (2) data configuration mechanisms
that allow domain experts to steer prediction models by configuring the
training data. Then, we present the technical implementation of the system.
The source code of the system is open-sourced on GitHub: [Code
Repository]https://github.com/adib0073/EXMOS/.

###  2.1. Explanation Dashboard

Inspired by recent studies that demonstrated the effectiveness of multifaceted
explanations achieved by combining different types of explanations
(Bhattacharya et al., 2023a) for enhancing user understandability (Teso et
al., 2022; Teso and Kersting, 2019; Schramowski et al., 2020; Bertrand et al.,
2023; Kulesza et al., 2015, 2013; Wang et al., 2023), we designed and
developed a explanation dashboard (Figure 2), which combined the following two
explanation types:

Figure 2. Screenshot of the explanation dashboard which combines diverse data-
centric and model-centric global explanations. The different visual components
are designed based on the guidelines from Bhattacharya et al. (Bhattacharya et
al., 2024).

[Explanation Dashboard]Design of multifaceted explanation dashboard from
Bhattacharya et al. which combine data-centric and model-centric global
explanations. The different visual components included in this design are: Top
Decision Rules (TDR), Key Insights (KI), Important Risk Factors (IRF), Data
Quality (DQ) and Data Density Distribution (DDD).

  * •

Data-centric explanations: These explanations elucidate different aspects of
the training data to the user (Anik and Bunt, 2021; Bhattacharya, 2022). We
followed the design guidelines from Bhattacharya et al. (Bhattacharya et al.,
2024) to include diverse data-centric global explanations that presented
distinct patterns from the training data using descriptive statistics. It also
presented the frequency distribution for each predictor variable used in the
prediction model and the estimated quality of the data used in training the
prediction model.

  * •

Model-centric explanations: Unlike data-centric explanations, model-centric
explanations describe the various factors considered by the model for
generating the predictions, such as feature weights or model hyper-parameters
(Adadi and Berrada, 2018; Bhattacharya, 2022). We followed the design
guidelines from Bhattacharya et al. (Bhattacharya et al., 2024) for including
top decision rules generated by surrogate explainers (Bhattacharya, 2022) and
SHAP (Lundberg and Lee, 2017) feature importances for obtaining the important
actionable and non-actionable factors.

Furthermore, the explanation dashboard also presents the model’s overall
prediction accuracy, along with the training data size and the number of
predictor variables considered by the model. It also displays the change in
prediction accuracy after the model is retrained after the data
configurations. Domain experts are introduced to the explanation dashboard
before the data configuration mechanism for effective model steering.

(a) Manual configuration design, which includes (1) feature selection control
to include or exclude predictor variables and (2) feature filtering control to
set the upper and lower limits for the predictor variables.

[Manual configuration design]Manual configuration design, which included (1)
feature selection control to include or exclude predictor variables and (2)
feature filtering control to set the upper and lower limits for the predictor
variables.

(b) Automated configuration design, which includes data issue explanations
through (1) the quantified impact of these issues, (2) visualisations
displaying before and after correction changes to the data or predictor
variables and (3) description of the issue and how its correction can impact
the model performance.

[Automated configuration design]Automated configuration design, which included
explanations of the data issues through (1) displaying the quantified impact
of these issues, (2) visualisations displaying before and after correction
changes to the data or predictor variables and (3) description of the issue
and how its correction can impact the model performance.

Figure 3. Screenshots of the manual and automated configuration screens from
the system.

[Manual and automated data configuration designs]Data configuration designs
for domain expert AI collaboration.

###  2.2. Data Configuration Approaches

Our system provides domain experts with control over refining prediction
models by leveraging their prior domain knowledge of the training data. It
allows them to interact with the training data to identify and correct diverse
data quality issues. Training data configuration can be achieved by the
following distinct approaches:

  1. (1)

Manual configuration: Following the principles of data-centric AI (Zha et al.,
2023), the manual configuration approach allows domain experts to perform
feature selection, feature filtering and provide data guardrails to prevent
abrupt changes to the training data. Using this mechanism, domain experts gain
higher control over the training data for removing irrelevant features or
potentially corrupt predictor variables. This approach also allows domain
experts to identify the correct data ranges for the selected predictor
variables for more reliable predictions.

  2. (2)

Automated configuration: The automated configuration enables domain experts to
select data issues that correction algorithms can automatically rectify (Zha
et al., 2023). It includes providing an explanation of the potential data
issues and their impact on the prediction model. Then, to minimise the impact
of these potential data issues, domain experts could select the potential
issues for automated corrections with a single button click. However, despite
requiring less effort than the manual mechanism, the automated method provides
lesser control for configuring training data.

Figure 4. The model steering application adopts a three-layered architecture
to facilitate user interactions.

[Explanation Dashboard]High-level technical solution architecture for model
steering application.

###  2.3. Technical Implementation

Our model steering application adopts a three-layered architecture, as
illustrated in Figure 4, to facilitate user interactions:

  1. (1)

User Interaction Layer: This layer utilizes Meteor’s full-stack framework
(Meteor.js, 2024) in conjunction with React.js to provide interactive visual
explanations to users. It also captures user feedback for model steering,
enabling users to guide the explanation process.

  2. (2)

Middleware Layer: Developed using the FastAPI framework (FastAPI, 2022), this
layer acts as a bridge between the user interface layer and the back-end XAI
engine. It receives requests from the user interface, transmits them to the
XAI engine for processing, and returns back the responses to the user
interface layer.

  3. (3)

XAI Engine: The XAI engine is the core of the application where the ML and XAI
algorithms are applied. This layer is implemented in Python. It connects to a
MongoDB database to store training data and user configurations. This layer is
responsible for refining the prediction model based on user feedback and
regenerating the explanations based on the configured training data and the
fine-tuned ML model.

To ensure scalability and reproducibility, the entire application is
containerised using Docker and deployed on our KU Leuven managed servers.

##  3\. Use Case: Steering Diabetes Prediction System

We instantiated the generic design of our model steering system for a
healthcare-focused diabetes prediction system. In this scenario, healthcare
experts were provided with a model steering system that explained a diabetes
prediction model using the Pima Indians dataset (Smith et al., 1988). Through
our experiments, we analysed how healthcare experts could steer the prediction
model using this system. We conducted three user studies involving 174
healthcare experts in total to analyse the effectiveness of the explanation
dashboard and the data configuration approaches during model steering. The
findings from our extensive user studies highlight the importance of the
multifaceted explanations and the various data configuration approaches for
effective collaboration between domain experts and AI. Further details of the
first two user studies and their in-depth findings have been published in ACM
CHI 2024 (Bhattacharya et al., 2024). The detailed description of the third
user study and its key findings is currently undergoing review for publication
in another ACM conference paper. In general, the model steering system
facilitates the involvement of domain experts during model steering,
ultimately leading to improved human-AI collaboration.

##  4\. Conclusion

In this paper, we introduce an Explanatory Model Steering system that
facilitates effective collaboration between domain experts and AI. This system
allows domain experts to steer prediction models using manual and automated
data configuration approaches. It involves domain knowledge driven
configuration of the training data by domain experts for fine-tuning
prediction models. While evaluated for the healthcare domain, the
applicability of such a system can be extended to other application domains.

###### Acknowledgements.

We thank Maxwell Szymanski and Robin De Croon for their valuable feedback on
this research. This research was supported by Research Foundation–Flanders
(FWO grants G0A4923N and G067721N) and KU Leuven Internal Funds (grant
C14/21/072) (Bhattacharya, 2024). We also thank the participants of our user
studies for their contribution.

## References

  * (1)
  * Adadi and Berrada (2018) Amina Adadi and Mohammed Berrada. 2018.  Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI).  _IEEE Access_ 6 (2018), 52138–52160.
  * Anik and Bunt (2021) Ariful Islam Anik and Andrea Bunt. 2021.  Data-Centric Explanations: Explaining Training Data of Machine Learning Systems to Promote Transparency. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. ACM, Yokohama Japan, 1–13.  https://doi.org/10.1145/3411764.3445736
  * Bayer et al. (2021) Sarah Bayer, Henner Gimpel, and Moritz Markgraf. 2021.  The role of domain expertise in trusting and following explainable AI decision support systems.  _Journal of Decision System_ (08 2021).  https://doi.org/10.1080/12460125.2021.1958505
  * Bertrand et al. (2023) Astrid Bertrand, Tiphaine Viard, Rafik Belloum, James R. Eagan, and Winston Maxwell. 2023.  On Selective, Mutable and Dialogic XAI: A Review of What Users Say about Different Types of Interactive Explanations. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 411, 21 pages.  https://doi.org/10.1145/3544548.3581314
  * Bhattacharya (2022) Aditya Bhattacharya. 2022.  Applied Machine Learning Explainability Techniques.  In _Applied Machine Learning Explainability Techniques_. Packt Publishing, Birmingham, UK.  https://www.packtpub.com/product/applied-machine-learning-explainability-techniques/9781803246154
  * Bhattacharya (2024) Aditya Bhattacharya. 2024.  Towards Directive Explanations: Crafting Explainable AI Systems for Actionable Human-AI Interactions. In _Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24)_ (Honolulu, HI, USA) _(CHI EA ’24)_. ACM, New York, NY, USA, 6.  https://doi.org/10.1145/3613905.3638177
  * Bhattacharya et al. (2023a) Aditya Bhattacharya, Jeroen Ooge, Gregor Stiglic, and Katrien Verbert. 2023a.  Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorations. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_ (Sydney, NSW, Australia) _(IUI ’23)_. Association for Computing Machinery, New York, NY, USA, 204–219.  https://doi.org/10.1145/3581641.3584075
  * Bhattacharya et al. (2023b) Aditya Bhattacharya, Simone Stumpf, Lucija Gosak, Gregor Stiglic, and Katrien Verbert. 2023b.  Lessons Learned from EXMOS User Studies: A Technical Report Summarizing Key Takeaways from User Studies Conducted to Evaluate The EXMOS Platform.  arXiv:2310.02063 [cs.LG]
  * Bhattacharya et al. (2024) Aditya Bhattacharya, Simone Stumpf, Lucija Gosak, Gregor Stiglic, and Katrien Verbert. 2024.  EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurations. In _Proceedings of the CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’24)_. Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/3613904.3642106
  * Dudley and Kristensson (2018) John J. Dudley and Per Ola Kristensson. 2018.  A Review of User Interface Design for Interactive Machine Learning.  _ACM Trans. Interact. Intell. Syst._ 8, 2, Article 8 (jun 2018), 37 pages.  https://doi.org/10.1145/3185517
  * FastAPI (2022) FastAPI. 2022.  .  FastAPI.  https://fastapi.tiangolo.com/ Accessed: 2024-04-15.
  * Feuerriegel et al. (2020) Stefan Feuerriegel, Mateusz Dolata, and Gerhard Schwabe. 2020.  Fair AI.  _Business & information systems engineering_ 62, 4 (2020), 379–384.
  * Guidotti et al. (2018) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018.  A Survey of Methods for Explaining Black Box Models.  _ACM Comput. Surv._ 51, 5, Article 93 (aug 2018), 42 pages.  https://doi.org/10.1145/3236009
  * Kulesza et al. (2015) Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015.  Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In _Proceedings of the 20th International Conference on Intelligent User Interfaces_. ACM, Atlanta Georgia USA, 126–137.  https://doi.org/10.1145/2678025.2701399
  * Kulesza et al. (2010) Todd Kulesza, Simone Stumpf, Margaret Burnett, Weng-Keen Wong, Yann Riche, Travis Moore, Ian Oberst, Amber Shinsel, and Kevin McIntosh. 2010.  Explanatory Debugging: Supporting End-User Debugging of Machine-Learned Programs. In _2010 IEEE Symposium on Visual Languages and Human-Centric Computing_. IEEE, Leganes, Madrid, Spain, 41–48.  https://doi.org/10.1109/VLHCC.2010.15
  * Kulesza et al. (2013) Todd Kulesza, Simone Stumpf, Margaret Burnett, Sherry Yang, Irwin Kwan, and Weng-Keen Wong. 2013.  Too much, too little, or just right? Ways explanations impact end users’ mental models. In _2013 IEEE Symposium on Visual Languages and Human Centric Computing_. 3–10.  https://doi.org/10.1109/VLHCC.2013.6645235
  * Lakkaraju et al. (2022) Himabindu Lakkaraju, Dylan Slack, Yuxin Chen, Chenhao Tan, and Sameer Singh. 2022.  Rethinking Explainability as a Dialogue: A Practitioner’s Perspective.  arXiv:2202.01875 [cs.LG]
  * Lundberg and Lee (2017) Scott Lundberg and Su-In Lee. 2017.  A Unified Approach to Interpreting Model Predictions.  arXiv:1705.07874 [cs.AI]
  * Meteor.js (2024) Meteor.js. 2024.  .  Meteor.  https://www.meteor.com/ Accessed: 2024-04-15.
  * Pawar et al. (2020) Urja Pawar, Donna O’Shea, Susan Rea, and Ruairi O’Reilly. 2020.  Incorporating Explainable Artificial Intelligence (XAI) to aid the Understanding of Machine Learning in the Healthcare Domain.. In _AICS_. 169–180.
  * Rojo et al. (2021) Diego Rojo, Nyi Nyi Htun, Denis Parra, Robin De Croon, and Katrien Verbert. 2021.  AHMoSe: A knowledge-based visual support system for selecting regression machine learning models.  _Computers and Electronics in Agriculture_ 187 (Aug. 2021), 106183.  https://doi.org/10.1016/j.compag.2021.106183
  * Schramowski et al. (2020) Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. 2020.  Making deep neural networks right for the right scientific reasons by interacting with their explanations.  _Nature Machine Intelligence_ 2 (08 2020), 476–486.  https://doi.org/10.1038/s42256-020-0212-3
  * Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps.  arXiv:1312.6034 [cs.CV]
  * Smith et al. (1988) J W Smith, J E Everhart, W C Dickson, W C Knowler, and R S Johannes. 1988.  Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus.  In _Proceedings of the Annual Symposium on Computer Application in Medical Care_. 261–265.
  * Teso et al. (2022) Stefano Teso, Öznur Alkan, Wolfang Stammer, and Elizabeth Daly. 2022.  Leveraging Explanations in Interactive Machine Learning: An Overview.  http://arxiv.org/abs/2207.14526 arXiv:2207.14526 [cs].
  * Teso and Kersting (2019) Stefano Teso and Kristian Kersting. 2019.  Explanatory Interactive Machine Learning. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ (Honolulu, HI, USA) _(AIES ’19)_. Association for Computing Machinery, New York, NY, USA, 239–245.  https://doi.org/10.1145/3306618.3314293
  * Wang et al. (2019) Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y. Lim. 2019.  Designing Theory-Driven User-Centric Explainable AI. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. ACM, Glasgow Scotland Uk, 1–15.  https://doi.org/10.1145/3290605.3300831
  * Wang et al. (2023) Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, and Tianyi Zhang. 2023.  DeepSeer: Interactive RNN Explanation and Debugging via State Abstraction. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 740, 20 pages.  https://doi.org/10.1145/3544548.3580852
  * Wang et al. (2022) Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, and Rich Caruana. 2022.  Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ (Washington DC, USA) _(KDD ’22)_. Association for Computing Machinery, New York, NY, USA, 4132–4142.  https://doi.org/10.1145/3534678.3539074
  * Zha et al. (2023) Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. 2023.  Data-centric Artificial Intelligence: A Survey.  arXiv:2303.10158 [cs.LG]