# Multiple Instance Learning for Glioma Diagnosis using Hematoxylin and Eosin
stained Whole Slide Images: An Indian Cohort Study

Ekansh Chauhan  Amit Sharma  Megha S Uppin  C.V. Jawahar and Vinod P.K We
acknowledge IHub-Data, IIIT Hyderabad (H1-002) for financial assistance.Ekansh
Chauhan, Amit Sharma and C.V. Jawahar are with Centre for Visual Information
Technology, International Institute of Information Technology Hyderabad,
Hyderabad 500032 India (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>jawahar@iiit.ac.in).Megha S Uppin is with the
Department of Pathology, Nizam’s Institute Of Medical Sciences, Hyderabad
500082 India (e-mail: megha_harke@yahoo.co.in).Vinod P.K. is with Center for
Computational Natural Sciences and Bioinformatics, International Institute of
Information Technology Hyderabad, Hyderabad 500032 India (e-mail:
vinod.pk@iiit.ac.in).

###### Abstract

Brain tumors represent a severe and life-threatening condition, demanding
precise diagnosis and tailored treatment strategies. This study advances
patient care with findings from rigorous multiple-instance-learning
experimentations across various feature extractors, and aggregators in brain
tumor histopathology. It establishes new performance benchmarks in glioma
subtype classification across multiple datasets, including a novel dataset
focused on the Indian demographic (IPD-Brain), providing a valuable resource
for existing research. Using a ResNet-50, pretrained on histopathology
datasets, for feature extraction, combined with DTFD feature aggregator, our
approach achieves state-of-the-art AUCs of 88.08 ± 3.98 on IPD-Brain and 95.81
± 1.78 on TCGA-Brain dataset respectively for three-way glioma subtype
classification. Moreover, it establishes new benchmarks in grading and
detecting IHC molecular biomarkers (IDH1 (mutant R132H), TP53, ATRX, Ki-67)
through H&E stained whole slide images for the IPD-Brain dataset. The work
also highlights a significant correlation between the model’s decision-making
processes and the diagnostic reasoning of pathologists, underscoring its
capability to mimic professional diagnostic procedures.

Brain Tumor, Deep Learning, Digital Histopathology, Multi-Instance Learning
(MIL)

##  1 Introduction

Brain tumors, known for their lethality, represent a critical medical
condition characterized by the abnormal growth of cells within the central
nervous system, which can lead to severe neurological dysfunction. According
to the World Health Organisation (WHO), central nervous system (CNS) tumors
have extensive classification and subtypes. Due to genetic predisposition,
environmental factors, and lifestyle-related variables in India, the
occurrence of CNS tumor cases fluctuates between 5 and 10 cases per 100,000
individuals [1, 2]. Glial tumors are the most common type of brain tumor,
primarily Astrocytoma, Glioblastoma, and Oligodendroglioma [3, 4, 5].
Glioblastoma, the most aggressive among them, which accounts for approximately
57% of all gliomas, represents a highly malignant glioma, with a median
survival of just 12-15 months even with aggressive treatment approaches and
associated with a high rate of recurrence due to its infiltrative nature [6,
7]. Astrocytoma, on the other hand, originates from astrocytes and is graded
based on its degree of malignancy, low or high. Oligodendroglioma, a rarer
subtype, arises from oligodendrocytes and is often characterized by genetic
alterations, which can influence treatment strategies and predict better
outcomes [8].

Histopathology is a fundamental branch of medical science involving
microscopic examination of tissue specimens to diagnose diseases, specifically
cancers, and understand the underlying cellular and structural abnormalities.
Traditionally, it involves pathologists visually inspecting stained tissue
sections under a microscope. Hematoxylin and Eosin (H&E) staining and
Immunohistochemistry (IHC) staining are two standard techniques in
Histopathology that serve distinct purposes in medical research and
diagnostics. Hematoxylin stains cell nuclei blue-purple, whereas Eosin stains
cytoplasm and extracellular matrix various shades of pink. H&E staining is
essential for assessing tissue structure, identifying cellular components, and
diagnosing subtypes and corresponding WHO grade [9]. Glioma grading is
essential for determining patient prognosis and guiding treatment plans. The
differentiation between high-grade gliomas (HGG) and low-grade gliomas (LGG)
is necessary, as it indicates variations in tumor aggressiveness and
correlates with patient survival rates [10, 11, 12, 13]. Thus, along with
three-way subtype classification, we also focus on distinguishing between HGG
and LGG to assess tumor progression and inform therapy planning accurately.

Figure 1: Multi-Instance Learning Framework for Subtype Classification in
Brain Histopathology. Tissue from surgical samples is digitized and processed
into patches, and patch-level features are extracted using a pre-trained
feature extractor network. Subsequently, the feature aggregation method pools
the patch-level representation into slide-level representation, which is then
classified to determine tumor type, grade, and molecular markers. Then
attention mapping is used for interpretability, providing critical insights
into the neural network’s decision-making process.

Additionally, IHC staining is a specialized technique to detect specific
proteins or antigens within tissue samples. It involves using antibodies
tagged with chromogenic or fluorescent markers to visualize the presence and
distribution of specific proteins in tissues. IHC is valuable for
investigating biomarkers like isocitrate dehydrogenase (IDH / IDH1 (mutant
R132H)), alpha thalassemia/mental retardation syndrome X-linked (ATRX), TP53,
Ki-67 and identifying specific cellular components related to diseases. In
this study, we explored the mutation status of IDH, ATRX, TP53, and Ki-67
using H&E stain to significantly alleviate the workload in pathology labs and
reduce associated costs. By diminishing the need for multiple IHC tests, this
approach conserves resources, removes the necessity since it may not be
available in all healthcare systems, and streamlines the diagnostic process.
To mimic the diagnosis of pathologists, we treated Ki-67, the percentage of
positively stained cells among the total number of cancer cells assessed, as a
binary classification task by varying thresholds.

In recent years, the field of histopathology has undergone a significant
transformation, primarily due to the emergence of digital histopathology [14],
the available datasets like TCGA [15], and the application of deep learning
methods for their analysis [16, 17]. This evolution in the brain tumor
research domain underscores the importance of diverse datasets that extend
beyond the confines of American or European populations. The global landscape
of brain tumors demands a comprehensive understanding facilitated by including
datasets from various demographics. Deep learning methods have demonstrated
remarkable efficiency in processing extensive histopathological data with
unprecedented speed and accuracy.

While existing research is able to predict the subtype, grading, and
biomarkers directly from routine H&E stained histopathology slides for many
tumor types [18, 19, 16], including brain tumor [20, 21], showing the
potential to accelerate diagnostic workflows and reduce the costs associated
with molecular testing. However, a limited focus is still given to brain tumor
classification based on the updated WHO guidelines (the fifth edition, WHO
CNS5, 2021) [22]. Due to the large size of histopathology images and lack of
patch-level labels in a whole slide image (WSI), the Multiple Instance
Learning (MIL) framework has been widely adopted in the field of digital
pathology. MIL is a two-stage framework: (i) Feature extractor and (ii)
Feature Aggregator. The use of standard ImageNet pre-trained weights and
large-scale domain-aligned (pathology) pre-training in a self-supervised
manner [23, 24, 25] for extracting latent features of patches underscores the
effectiveness on various downstream pathology tasks. To aggregate or pool
these features, Campanella et al. [18] employed an RNN-based approach for
feature-level aggregation in the classification of clinical pathology whole
slide images. ABMIL, as introduced by Ilse et al. [19], leverages attention
mechanisms to determine instance weights through a trainable neural network.
This method enhances traditional MIL pooling operations, such as max or mean
pooling, by aggregating bag-level representations based on weighted averages
of instance-level features. Expanding on the concept of attention-based
instance scoring, CLAM, developed by Lu et al. [16], delves into multi-class
predictions using instance-level clustering, thereby facilitating the learning
of class-specific features. Dual-stream MIL (DS-MIL) [26] employs non-local
attention for aggregation to learn classifiers. A Transformer-based Multiple
Instance Learning approach (TransMIL) [27] explores the relationships between
instances in terms of their morphology and spatial characteristics using the
Nyström method for approximating all-to-all self-attention, though it requires
significant computation resources. DTFD-MIL, introduced by Zhang et al. [28],
introduces the concept of ”pseudo-bags” addressing issues associated with
small cohorts. Another contribution is a transformer-based hierarchical ViT
proposed by [25], focusing on using spatial relationships among instances.
However, despite these advancements, the downstream classification tasks
involving brain tumor subtypes and the identification of immunohistochemistry
(IHC) biomarkers remain underexplored.

Our research addresses a critical gap in brain tumor studies by introducing a
glioma subtype dataset within the India Pathology Dataset (IPD)
consortium.111https://hai.iiit.ac.in/ipd/ This dataset facilitates research in
India and encourages global collaboration to advance our understanding and
combat this lethal affliction. Fig. 1 shows the pipeline followed in this
work. Through rigorous experimentation, we aim to identify a specific
combination of feature extractor and aggregator that performs well for various
tasks based on the 2021 WHO guidelines. The outcomes of our study emphasize
the opportunity to advance the state of the art in deep learning applications
for brain tumor classification.

The main contributions of our work are as follows:

  * •

We created a dataset tailored to the Indian region and one of the largest
collections of brain whole slide images globally, providing a valuable
resource for research in this domain.

  * •

To our knowledge, this is the first study predicting Immunohistochemistry
(IHC) molecular biomarkers, utilizing cost-effective Haematoxylin and Eosin
(H&E) slides, eliminating the need for expensive IHC staining. This not only
enhances accessibility but also explores the potential of H&E stains in
accurately determining these biomarkers.

  * •

We achieve state-of-the-art (SOTA) results on The Cancer Genome Atlas (TCGA)
data for glioma subtype classification by using a specific self-supervised
pre-trained feature extractor and a feature aggregator.

  * •

We benchmark the IPD-Brain Dataset for six subtype classification tasks, i.e.,
glioma subtype classification, grading, IDH, ATRX, TP53, and Ki-67.

  * •

Through rigorous experimentation with all combinations of four feature
extractors and three aggregators, we provide insights into the most effective
approaches for analyzing brain histology data.

##  2 Dataset Description

As a part of the India Pathology Dataset (IPD) consortium, in this study, 530
digitized slides from 367 patients (retrospective and prospective cases) were
acquired at the Department of Pathology of Nizam’s Institute Of Medical
Sciences (NIMS), Hyderabad, from a diverse patient cohort presenting three
brain tumor types (Glioblastoma, Astrocytoma, Oligodendroglioma). Table I and
Fig. 2 summarize multiple statistics of the IPD-Brain (in-house) dataset. Each
specimen was collected following stringent ethical protocols and quality
control measures to ensure representativeness and integrity. The collection
process was closely monitored to maintain the tissue’s physiological state,
minimizing any alterations that might affect the histopathological outcomes. A
sample of this dataset at multiple magnification levels can be seen in Fig. 3.

Table 1: Statistics of IPD-Brain histopathology dataset | Patient level | Slide level
---|---|---
Scan Statistics:
Total Scanned WSIs | 367 | 530
| Scanned WSIs w/complete EHR record
---
328 | 484
Glioma Subtype Statistics:
Glioblastoma (GBM) | 168 (51.22%) | 246 (50.83%)
Astrocytoma (A) | 88 (26.83%) | 133 (27.48%)
Oligodendroglioma (O) | 72 (21.95%) | 105 (21.69%)
Phenotypic Statistics of Glioma Patients:
Sex: |
Female | 136 (41.46%)
Male | 192 (58.54%)
Age | 45.89 $\pm$ 12.93
Female | 45.37 $\pm$ 12.64
Male | 46.27 $\pm$ 13.15

(a) Age

(b) Ki-67

(c) IHC biomarkers

(d) Grade

Figure 2: Distribution of phenotypic information and labels in IPD-Brain.
Figure 3: IPD-Brain Dataset sample at various magnifications demonstrate the
enhanced detail from staining and digitization, crucial for deep learning
analysis.

The digitization of stained slides into high-quality digital images was
accomplished using the Morphle DigiPath 6T Scanner, capturing details at a 40x
magnification akin to conventional microscopy. The digital images were saved
in the TIFF format. Each tiff file was also enriched with metadata detailing
scan specifics and magnification, enhancing the data’s utility. Furthermore,
NIMS provided electronic health records (EHRs) for all slides, identifying
tumor subtypes, grades, and IHC biomarkers, complemented by extensive metadata
that includes patient demographics, radiology, and diagnostic reports, thereby
offering a nuanced understanding of each specimen within its clinical context.

This India region-specific dataset is developed to support global
collaboration in brain tumor research, enhancing diversity among existing
cohorts and providing insights into potential regional and ethnic variations
in the disease.

##  3 Methodology

###  3.1 MIL Problem formulation

In histopathology classification, Multiple Instance Learning (MIL) is a
compelling solution to the challenge of sparse and imprecise annotations often
encountered in medical image analysis. Unlike traditional supervised learning,
which relies on precisely labeled instances (patches), MIL operates on ’bags’
of instances, with the label determined by the collective characteristics of
these instances (slide-level). Here, the mathematical formulation of the MIL
framework for subtype classification is extended and defined as follows:

Consider a dataset $X$ comprising $M$ WSIs, denoted as
$X=\\{X_{1},X_{2},...,X_{M}\\}$. Each WSI is a ’bag’ in MIL terminology,
representing an entire histopathology slide. The label of the $m$-th WSI,
denoted as $Y_{m}$, in an n-way subtype classification task, is determined
based on the subtype presence within the slide, with each slide categorized
into one of $n$ possible subtypes or negative slide. The instances within a
bag denoted as $x_{i}$, are individual image patches or regions extracted from
the histopathology slide. For a given $m$-th WSI, the instances are
represented below in (1):

$X_{m}={x_{1},…,x_{i},…,x_{p-1},x_{p}}$ (1)

where $p$ is the number of patches in the $m$-th WSI, and each
$x_{i}\in\mathbb{R}^{256\times 256\times 3}$. Although the exact labels
$y_{i}$ for these instances are unknown in MIL, they are implicitly associated
with the overall label $Y_{m}$, (2), of the bag.

$Y_{m}=\begin{cases}k,&\text{if }\exists i\in p:y_{i}=k\\\ 0,&\forall i\in
p:y_{i}=0\end{cases}$ (2)

Here, $k\in n$ represents the subtype class labels. This implies that the bag
belongs to class $k$ if any instance within it is determined to belong to that
class. Otherwise, if all the instances in a bag are negative, the bag is
considered negative (i.e., belongs to the negative class - $0$). In this work,
we only focused on positive subtype classes.

The feature extractor function, denoted as $f$ (section: 3.3), is crucial in
transforming each instance $x_{i}$ into a deep meaningful representation. It
maps each instance to a fixed-dimensional feature vector $d$, expressed as
(3).

$f(x_{i})=v_{i}$ (3)

where $v_{i}\in\mathbb{R}^{d\times 1}$ is the feature vector of size $d$
extracted from the $i$-th instance $x_{i}$. The function $f$ could be
implemented through various methods, including handcrafted feature extractors,
deep CNNs, or ViTs.

A bag $B\in\mathbb{R}^{p\times d}$ represents the entire histopathology slide
and is defined as a collection of instance feature vectors. Formally, the bag
is constructed through the concatenation of the feature vectors of its
instances, expressed using the concatenation symbol $\oplus$ in (4).

$B=\bigoplus_{i=1}^{p}v_{i}$ (4)

The goal is to learn a hypothesis function $g$ (section: 3.4) that effectively
maps a bag to its corresponding label. This function can be viewed as a
combination of an Aggregator and a Classifier. The aggregator transforms the
representation of the bag by using any aggregation functions, such as the
average or maximum of instance features or more complex attention-based
aggregation methods. The output of the Classifier is the predicted class out
of $n$ classes.

###  3.2 WSI Preprocessing

The process begins by segmenting tissue regions within digitized slides. The
image is converted to the HSV color space, and a mask is created by
thresholding the saturation channel after smoothing. Morphological operations
fill gaps and holes, and contours are filtered based on area. We have included
optimal segmentation parameters for our dataset in the section 4. Following
segmentation, the algorithm crops 256x256 patches from within the segmented
regions based on user-specified magnification.

###  3.3 Feature Extractors (f)

While ImageNet pre-trained weights are commonly utilized in histopathology
tasks, we expanded our comparative analysis to encompass various SSL methods
recognized for their strong performance. The selection of multiple SSL
approaches was motivated by the absence of a definitive superior way across
all settings; apart from the baseline ResNet-50 pre-trained on ImageNet, our
evaluation considered high-performance networks with domain-specific pre-
training (Histopathology). These networks included a ResNet-50 model pre-
trained via the Barlow Twins method, a Vision Transformer model pre-trained
using the DINO method, and CTransPath, a hybrid model merging a convolutional
neural network (CNN) with a multiscale Swin Transformer architecture, pre-
trained through semantically-relevant contrastive learning. By incorporating
these diverse options, we aimed to comprehensively evaluate their
generalizability to our in-house histopathology dataset.

###  3.4 MIL Aggregation Models (g):

We also employed three feature aggregator models selected based on their
performance across various studies [16, 26, 28]. These aggregation models
leverage attention-based learning to automatically identify the subtype or
diagnostically significant sub-regions for precise whole-slide classification.

####  3.4.1 CLAM

It uses gated attention to assign scores to patches within a whole slide,
leveraging these scores for slide-level aggregation through a weighted average
for multi-class classification. It operates with n parallel attention branches
for n class-specific representations, though studies show the single-branch
configuration often performs better. It incorporates domain knowledge for
additional guidance in instance-level clustering.

####  3.4.2 DSMIL

It employs a dual-stream architecture with self-supervised contrastive
learning, for instance embeddings. One stream identifies the key instance,
whereas the other aggregates embeddings for a comprehensive bag
representation. It uses a unique approach to weight instance contributions
based on similarity to the key instance, enhancing the final bag score through
a sophisticated attention mechanism.

####  3.4.3 DTFD

It generates pseudo-bags from bag divisions, maintaining the original label
for each. It utilizes a two-tier attention-based MIL model, first estimating
pseudo-bag probabilities, then deriving instance probabilities and distilled
features for final parent bag likelihood prediction. Despite potential noise,
it leverages deep neural network resilience, focusing on Aggregated Feature
Selection (AFS) for feature distillation in this context.

###  3.5 Transfer Learning for IHC Biomarkers

Furthermore, our investigation aimed to assess ”Is H&E stain enough?” by
determining the capability of our weakly-supervised deep learning subtype
classification model in predicting common diagnostic molecular markers from
glioma H&E slides. We sought to evaluate its potential utility as a clinical
secondary check or a cost-effective alternative to actual molecular biology
experiments. Using transfer learning, a technique that adapts a model trained
on one task to a related new task, we systematically explored the
effectiveness of the best-performing model, initially trained for tumor
subtype classification, in inferring molecular biomarkers within a weakly
supervised context. Fine-tuning was conducted on the three-class subtyping
model for each of three binary classifications of IHC mutations (somatic
mutation versus wild type for IDH1, TP53, and ATRX mutations) and one
classification task at various cutoffs for assessing tumor cell proliferation
(Ki-67).

##  4 Experimentation

###  4.1 Experiment Setup

In our study, we conducted comprehensive experiments using two datasets: the
in-house IPD-Brain Dataset and the publically available TCGA-Brain. This
section delineates the experimental setup employed. For feature extraction at
the patch level from WSIs, we used four distinct networks: ResNet-50 pre-
trained on ImageNet in a supervised manner, ResNet-50 pre-trained using the
Barlow Twins self-supervised method for histopathology, CTransPath pre-trained
with semantically relevant contrastive learning for histopathology, and ViT-S
pre-trained via the DINO approach for histopathology.

Upon extracting features using these models, we applied three aggregation
methods - CLAM, DSMIL, and DTFD - for the downstream classification task. We
identified the most effective extractor-aggregator combination on the IPD-
Brain dataset for the three-way subtype classification task and subsequently
extended its application for further experimental analysis. This involved
first testing the chosen combination on the TCGA-Brain dataset for subtype
classification, followed by employing the same combination for classifying
grading (LGG vs HGG) and IHC biomarkers mutations (TP53, IDH, ATRX, Ki-67)
using H&E stained slides. This systematic approach allowed us to rigorously
assess the efficacy of our models across different datasets and tasks for
brain histopathology.

###  4.2 Evaluation Metrics

The primary performance metric used for all the experiments is the Area Under
Curve (AUC), which offers a comprehensive assessment and is less susceptible
to class imbalance. Additionally, we consider slide-level accuracy (Acc) and
the macro-averaged F1 score. These metrics collectively provide a holistic
view of model performance: the AUC score indicates the ability to
differentiate between classes, whereas the F1 score offers insight into the
actual performance in the context of imbalanced data.

###  4.3 Implementation Details

The preprocessing of WSIs was done at 40x magnification. We used CLAM
preprocessing steps with median blurring using a 9x9 kernel, followed by Otsu
thresholding. Morphological closing with a 5x5 kernel was applied for finer
segmentation. Tissue contours were identified with an area filter threshold of
80, while holes within these contours were managed with a threshold of 10 and
a limit of 20 holes per contour.

We adhered to official implementations of the methods wherever possible,
exerting diligent efforts to replicate the code accurately. A comprehensive
evaluation was conducted for each combination of feature extractors and
aggregators. The dataset was divided into training, validation, and testing
phases with proportions of 80%, 10%, and 10%, respectively, on a patient-wise
basis. This approach ensured data leakage prevention, meaning all slides from
a single patient were exclusively allocated to one of these phases.
Additionally, we implemented a 10-fold cross-validation for each experimental
setup.

Table 2: Tumor subtype classification on IPD-Brain | Feature
---
Extractor
| Dataset
---
Category
SSL Method |  | Feature
---
Aggregator
AUC | ACC | F1
Resnet-50 | Imagenet | Supervised | CLAM-sb | 78.85 $\pm$ 6.27 | 65.9 $\pm$ 6.72 | 56.86 $\pm$ 12.79
DSMIL | 79.72 $\pm$ 5.78 | 69.29 $\pm$ 6 | 61.37 $\pm$ 7.47
DTFD | 80.5 $\pm$ 3.88 | 75.62 $\pm$ 4.11 | 69.78 $\pm$ 5.08
Resnet-50 | Histopathology | Barlow Twin | CLAM-sb | 84.55 $\pm$ 4.09 | 73.27 $\pm$ 5.38 | 67.61 $\pm$ 7.03
DSMIL | 80.87 $\pm$ 2.56 | 67.12 $\pm$ 5.95 | 59.98 $\pm$ 6.80
DTFD | 88.08 $\pm$ 3.98 | 79.83 $\pm$ 3.31 | 75.87$\pm$ 4.79
CTransPath | Histopathology | SRCL | CLAM-sb | 83.33 $\pm$ 3.63 | 72.08 $\pm$ 5.70 | 68.11 $\pm$ 5.57
DSMIL | 83.32 $\pm$ 5.16 | 69.73 $\pm$ 7.11 | 65.08 $\pm$ 7.68
DTFD | 87.12 $\pm$ 3.96 | 78.75 $\pm$ 5.78 | 75.73 $\pm$ 6.43
ViT-S | Histopathology | DINO | CLAM-sb | 82.95 $\pm$ 5.04 | 69.88 $\pm$ 6.96 | 65.2 $\pm$ 7.17
DSMIL | 81.96 $\pm$ 6.06 | 68.84 $\pm$ 5.56 | 62.61 $\pm$ 7.64
DTFD | 87.67 $\pm$ 4.14 | 78.89 $\pm$ 3.98 | 75.41 $\pm$ 3.50

Training was conducted for a minimum of 50 epochs and a maximum of 200 epochs,
with early stopping implemented with a patience of 25 epochs on validation
loss. The Adam optimizer was used with a batch size of 1. Specific learning
rates and weight decay parameters were set for each aggregator: DSMIL at 2e-4
(learning rate) with a 5e-3 weight decay, CLAM-sb (CLAM) at 2e-4 (learning
rate) with a 1e-3 weight decay, and DTFD at 1e-4 (learning rate) with a 1e-4
weight decay. These parameters were broadly consistent with those defined in
the original implementations of these methods. All experiments are performed
on RTX 3080ti 12GB GPU.

##  5 Results

Our analysis across various datasets demonstrates the effectiveness of
combining feature extractors, aggregators, and pretraining techniques. On the
India Pathology Dataset Brain (IPD-Brain), Table 2, the combination of a
Resnet-50 backbone, pretrained via the Barlow twin self-supervised learning
(SSL) method, and the DTFD aggregator emerged as superior, achieving an
average AUC of 88.08%, an accuracy of 79.83%, and an F1 score of 75.87%.
Variation in AUC and F1 score across 10-folds for all the combinations can be
observed in Fig. 4. This configuration proved particularly adept at
classifying brain tumor subtypes, showcasing its suitability for the unique
challenges of histopathological data analysis.

(a) Area Under Curve (AUC) Scores

(b) Macro F1 Scores

Figure 4: Comparative Analysis of Feature Extractors and Aggregation Methods.
The box plots illustrate the 10-fold AUC and macro F1 scores distribution for
all feature extractor and aggregation combinations used. Mean values are
indicated by dots within each box, whereas lines represent medians.

Table 3 details the evaluation of The Cancer Genome Atlas (TCGA) Brain
dataset, comparing CLAM (baseline), TPMIL [29] (current state-of-the-art), and
DTFD (best on IPD-Brain). We established a new state-of-the-art, achieving a
95.81% AUC, 86.94% accuracy, and an 80.41% F1 score, highlighting DTFD’s
superior performance on TCGA-Brain. Interestingly, our reimplementation of
CLAM using default parameters surpassed TPMIL’s documented results but not
DTFD, emphasizing DTFD’s adaptability and performance generalization across
various histopathological datasets.

Table 3: Tumor subtype classification results on TCGA-Brain

Feature Aggregator | AUC | ACC | F1
---|---|---|---
CLAM | 95.62 $\pm$ 2.04 | 85.62 $\pm$ 4.25 | 79.69 $\pm$ 5.86
$\text{TPMIL}^{*}$ | 94.17 | 83.16 | –
DTFD | 95.81 $\pm$ 1.78 | 86.94 $\pm$ 3.90 | 80.41 $\pm$ 4.38

  * •

∗: Results are taken from the original paper.

Interestingly, applying stain normalization using StainNet [30] on both IPD-
Brain and TCGA-Brain datasets, which can be observed in Table 4, did not
improve performance. It resulted in a decrease in AUC and F1 scores for both
datasets. This outcome suggests that the employed feature extractor was
already capable of effectively handling stain variability inherent in
histopathological images. Consequently, the use of stain normalization
appeared to diminish model performance slightly, emphasizing the feature
extraction process’s intrinsic ability to accommodate stain variations without
requiring additional normalization procedures.

Table 4: Effect of Stain Normalization on both the datasets. Dataset | Stain Norm | AUC | ACC | F1
---|---|---|---|---
IPD | $\bm{\times}$ | 88.08 $\pm$ 3.98 | 79.83 $\pm$ 3.31 | 75.87 $\pm$ 4.79
$\checkmark$ | 86.34 $\pm$ 5.25 | 77.09 $\pm$ 5.47 | 70.81 $\pm$ 8.24
TCGA | $\bm{\times}$ | 95.81 $\pm$ 1.78 | 86.94 $\pm$ 3.90 | 80.41 $\pm$ 4.38
$\checkmark$ | 93.37 $\pm$ 3.06 | 86.67 $\pm$ 4.62 | 79.96 $\pm$ 5.38

Table 5 details the classification of Immunohistochemistry (IHC) biomarkers on
the IPD-Brain dataset, achieving high accuracy and AUC rates for crucial
biomarkers like IDH, ATRX, and TP53. This highlights the model’s potential as
an accessible IHC molecular biomarker identification tool directly from H&E
slides.

Table 5: IHC biomarkers classification results on IPD-Brain Task | AUC | ACC | F1
---|---|---|---
IDH | 90.66 $\pm$ 5.22 | 86.65 $\pm$ 4.58 | 86 $\pm$ 4.58
ATRX | 84.07 $\pm$ 6.10 | 83.61 $\pm$ 8.68 | 67.3 $\pm$ 6.49
TP53 | 73.32 $\pm$ 8.63 | 77.43 $\pm$ 9.53 | 65.83 $\pm$ 10.93

Furthermore, Table 6, the Ki-67 marker classification task showcased the
model’s capability to accurately predict tumor cell proliferation rates, with
significant proficiency observed at the cutoff value of 10, achieving an
89.55% AUC and a 91.17% F1 score. This proficiency in Ki-67 classification
underlines the model’s utility in providing vital prognostic information
necessary for effective treatment planning and patient management.

Table 6: Ki-67 as classification task results Task | Cutoff Value | AUC | ACC | F1
---|---|---|---|---
Ki-67 | 5 | 86.76 $\pm$ 6.92 | 91.73 $\pm$ 6.56 | 92.23 $\pm$ 2.19
10 | 89.55 $\pm$ 5.27 | 86.93 $\pm$ 4.79 | 91.17 $\pm$ 3.33
20 | 86.81 $\pm$ 6.82 | 83.4 $\pm$ 6.65 | 85.99 $\pm$ 6.38

Our findings demonstrate that a ResNet-50 model pre-trained with Barlow Twin
SSL and combined with DTFD aggregation enhances histopathological analysis,
especially in brain tumor classification. This approach underscores the
effectiveness of careful feature extraction method selection, showing that
proper pretraining and aggregation negate the need for stain normalization and
improve diagnostic accuracy.

##  6 Explainability

Fig. 5 enhances the interpretability of feature aggregators across glioma
subtypes by analyzing attention weights assigned to tissue patches. This
visualization highlights critical tissue patches that guide the glioma subtype
classification. By employing a gradient scale from blue to red, our model
signifies the importance of specific regions within the WSI, with blue
indicating regions of lesser importance and red highlighting the most critical
areas. In comparison, the traditional manual microscopic examination by
neuropathologists in clinical settings is an exhaustive process, requiring
extensive scrutiny of the entire slide to identify significant glioma
morphological patterns.

Our model’s practical applicability and accuracy were further substantiated
through a rigorous single-blind evaluation conducted with a seasoned
pathologist from the Department of Pathology at Nizam’s Institute of Medical
Sciences, a premier tertiary hospital in India. The pathologist was presented
with a WSI as our model. The pathologist scrutinized specific regions within
the WSIs to formulate their diagnoses, some of which are shown as ’Raw Patch’
in Fig. 5.

Figure 5: Attention heatmap visualization for brain tumor WSI classification,
with green and red dots indicating correct classifications and
misclassifications, respectively, across all feature aggregators.

By qualitatively analyzing, we can observe the DTFD’s superior ability to
focus on regions indicative of each glioma’s medical characteristics, aligning
with known pathological features. For Glioblastoma, it highlights areas
showing its aggressive nature, like microvascular proliferation and
pseudopalisading necrosis. In Astrocytoma, the emphasis is on cellular
structures absent of necrosis, showcasing gemistocytic features.
Oligodendroglioma’s distinctive ’fried egg’ appearance and ’chicken wire’
vascular pattern are accurately identified, distinguishing it from other
subtypes. This assessment underlines DTFD’s effectiveness in pinpointing
diagnostically relevant regions, enhancing model transparency and reliability
in histopathological evaluations.

The model demonstrated proficiency in identifying critical morphological
microstructures, showing a strong alignment between its focus on important
regions and the key areas within the raw whole slide images (WSIs) that are
essential for glioma subtype classification. This congruence closely reflects
the diagnostic strategies employed by pathologists, underscoring the model’s
ability to prioritize and analyze patch segments in a manner similar to how
experts identify glioma patterns.

##  7 Conclusion

Our study showcases the potential of AI in enhancing brain tumor
classification and IHC biomarker detection through advanced machine learning
models. Employing a ResNet-50 architecture pre-trained with the Barlow Twin
SSL approach and utilizing DTFD for feature aggregation, we achieved
significant improvements in diagnostic precision across various
histopathological datasets. This approach validates traditional diagnostic
criteria and unveils novel morphological insights, enriching clinical
understanding and patient care. Incorporating AI into pathology heralds a new
era of diagnostics, offering profound implications for personalized treatment
and patient outcomes, demonstrating AI’s critical role in revolutionizing
medical diagnostics and therapeutic strategies.

## References

##

  * [1] A. Dasgupta, T. Gupta, and R. Jalali.  Indian data on central nervous tumors: A summary of published work.  South Asian Journal of Cancer, 5(3):147–153, 2016.
  * [2] M. Nair, Cherian Varghese, and Swaminathan.  Cancer: Current scenario, intervention strategies and projections for 2015.  NCHM Background Papers-Burden of Disease in India, pages 219–225, 01 2005.
  * [3] Ajay Singh Thakur, Renuka Gahine, and Vishal Kulkarni.  A study on morphologic and histological pattern of the central nervous system tumors.  International Journal of Research in Medical Sciences, 6(12):3879–3882, Nov. 2018.
  * [4] Farina Hanif, Kanza Muzaffar, Kahkashan Perveen, Saima M Malhi, and Shabana U Simjee.  Glioblastoma multiforme: a review of its epidemiology and pathogenesis through clinical presentation and treatment.  Asian Pacific journal of cancer prevention: APJCP, 18(1):3, 2017.
  * [5] Ahmad Faleh Tamimi and Malik Juweid.  Epidemiology and outcome of glioblastoma.  Exon Publications, pages 143–153, 2017.
  * [6] Roger Stupp, Warren P Mason, Martin J Van Den Bent, Michael Weller, Barbara Fisher, Martin JB Taphoorn, Karl Belanger, Alba A Brandes, Christine Marosi, Ulrich Bogdahn, et al.  Radiotherapy plus concomitant and adjuvant temozolomide for glioblastoma.  New England journal of medicine, 352(10):987–996, 2005.
  * [7] Quinn T Ostrom, Gino Cioffi, Kristin Waite, Carol Kruchko, and Jill S Barnholtz-Sloan.  Cbtrus statistical report: primary brain and other central nervous system tumors diagnosed in the united states in 2014–2018.  Neuro-oncology, 23(Supplement_3):iii1–iii105, 2021.
  * [8] David N Louis, Arie Perry, Guido Reifenberger, Andreas Von Deimling, Dominique Figarella-Branger, Webster K Cavenee, Hiroko Ohgaki, Otmar D Wiestler, Paul Kleihues, and David W Ellison.  The 2016 world health organization classification of tumors of the central nervous system: a summary.  Acta neuropathologica, 131:803–820, 2016.
  * [9] David N Louis, Arie Perry, Pieter Wesseling, Daniel J Brat, Ian A Cree, Dominique Figarella-Branger, Cynthia Hawkins, HK Ng, Stefan M Pfister, Guido Reifenberger, et al.  The 2021 who classification of tumors of the central nervous system: a summary.  Neuro-oncology, 23(8):1231–1251, 2021.
  * [10] Hwan-Ho Cho, Seung-Hak Lee, Jonghoon Kim, and Hyunjin Park.  Classification of the glioma grading using radiomics analysis.  PeerJ, 6:e5982, November 2018.
  * [11] Lei Jin, Feng Shi, Qiuping Chun, Hong Chen, Yixin Ma, Shuai Wu, NU Farrukh Hameed, Chunming Mei, Junfeng Lu, Jun Zhang, et al.  Artificial intelligence neuropathologist for glioma classification using deep learning on hematoxylin and eosin stained slide images and molecular markers.  Neuro-oncology, 23(1):44–52, 2021.
  * [12] Justin Ker, Yeqi Bai, Hwei Yee Lee, Jai Rao, and Lipo Wang.  Automated brain histology classification using machine learning.  Journal of Clinical Neuroscience, 66:239–245, 2019.
  * [13] Jocelyn Barker, Assaf Hoogi, Adrien Depeursinge, and Daniel L. Rubin.  Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles.  Medical Image Analysis, 30:60–71, 2016.
  * [14] Humayun Irshad, Antoine Veillard, Ludovic Roux, and Daniel Racoceanu.  Methods for nuclei detection, segmentation, and classification in digital histopathology: a review—current status and future potential.  IEEE reviews in biomedical engineering, 7:97–114, 2013.
  * [15] Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, Lawrence Tarbox, and Fred Prior.  The cancer imaging archive (tcia): Maintaining and operating a public information repository.  Journal of Digital Imaging, 26(6):1045–1057, Dec 2013.
  * [16] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood.  Data-efficient and weakly supervised computational pathology on whole-slide images.  Nature biomedical engineering, 5(6):555–570, 2021.
  * [17] David Tellez, Geert Litjens, Jeroen van der Laak, and Francesco Ciompi.  Neural image compression for gigapixel histopathology image analysis.  IEEE transactions on pattern analysis and machine intelligence, 43(2):567–578, 2019.
  * [18] Gabriele Campanella, Matthew G. Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J. Busam, Edi Brogi, Victor E. Reuter, David S. Klimstra, and Thomas J. Fuchs.  Clinical-grade computational pathology using weakly supervised deep learning on whole slide images.  Nature Medicine, 25(8):1301–1309, Aug 2019.
  * [19] Maximilian Ilse, Jakub Tomczak, and Max Welling.  Attention-based deep multiple instance learning.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2127–2136. PMLR, 10–15 Jul 2018.
  * [20] Sidong Liu, Zubair Shah, Aydin Sav, Carlo Russo, Shlomo Berkovsky, Yi Qian, Enrico Coiera, and Antonio Di Ieva.  Isocitrate dehydrogenase (idh) status prediction in histopathology images of gliomas using deep learning.  Scientific Reports, 10(1):7733, May 2020.
  * [21] Linmin Pei, Karra A. Jones, Zeina A. Shboul, James Y. Chen, and Khan M. Iftekharuddin.  Deep neural network analysis of pathology images with integrated molecular data for enhanced glioma classification and grading.  Frontiers in Oncology, 11, 2021.
  * [22] David N Louis, Arie Perry, Pieter Wesseling, Daniel J Brat, Ian A Cree, Dominique Figarella-Branger, Cynthia Hawkins, H K Ng, Stefan M Pfister, Guido Reifenberger, Riccardo Soffietti, Andreas von Deimling, and David W Ellison.  The 2021 WHO classification of tumors of the central nervous system: a summary.  Neuro Oncol, 23(8):1231–1251, August 2021.
  * [23] Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo, and Sérgio Pereira.  Benchmarking self-supervised learning on diverse pathology datasets.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3344–3354, June 2023.
  * [24] Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han.  Transformer-based unsupervised contrastive learning for histopathological image classification.  Medical Image Analysis, 81:102559, 2022.
  * [25] Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, and Faisal Mahmood.  Scaling vision transformers to gigapixel images via hierarchical self-supervised learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16144–16155, June 2022.
  * [26] Bin Li, Yin Li, and Kevin W. Eliceiri.  Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14318–14328, June 2021.
  * [27] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al.  Transmil: Transformer based correlated multiple instance learning for whole slide image classification.  Advances in Neural Information Processing Systems, 34:2136–2147, 2021.
  * [28] Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E. Coupland, and Yalin Zheng.  Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18802–18812, June 2022.
  * [29] Litao Yang, Deval Mehta, Sidong Liu, Dwarikanath Mahapatra, Antonio Di Ieva, and Zongyuan Ge.  Tpmil: Trainable prototype enhanced multiple instance learning for whole slide image classification, 2023.
  * [30] Hongtao Kang, Die Luo, Weihua Feng, Shaoqun Zeng, Tingwei Quan, Junbo Hu, and Xiuli Liu.  Stainnet: A fast and robust stain normalization network.  Frontiers in Medicine, 8, 2021.


# The Hitchiker’s Guide to Successful Living Lab Operations

Alan Wang<EMAIL_ADDRESS>University of VirginiaOlsson Hall,
Charlottesville, VA 22903CharlottesvilleVirginiaUSA22903 ,  Feng Yi Change
<EMAIL_ADDRESS>University of VirginiaOlsson Hall, Charlottesville, VA
22903CharlottesvilleVirginiaUSA22903 ,  Siavash Yousefi<EMAIL_ADDRESS>University of VirginiaOlsson Hall, Charlottesville, VA
22903CharlottesvilleVirginiaUSA22903 ,  Beatrice Li<EMAIL_ADDRESS>University of VirginiaOlsson Hall, Charlottesville, VA
22903CharlottesvilleVirginiaUSA22903 ,  Brad Campbell<EMAIL_ADDRESS>University of VirginiaOlsson Hall, Charlottesville, VA
22903CharlottesvilleVirginiaUSA22903 and  Arsalan Heydarian
<EMAIL_ADDRESS>University of VirginiaOlsson Hall, Charlottesville, VA
22903CharlottesvilleVirginiaUSA22903

(2018)

###### Abstract.

Living labs have been established across different countries to evaluate how
the interaction between humans and buildings can be optimized to improve
comfort, health, and energy savings. However, existing living labs can be too
project-specific, not scalable, and inflexible for comparison against other
labs. Furthermore, the lack of transparency in its software infrastructure
inhibits opportunities for critique and reuse, reducing the platform’s overall
potential. In the face of climate change and global energy shortage, we
envision the future of living labs to be open source and scalable to support
the integration of different IoTs, subjective measures, human-building
interactions, security, and privacy contexts. In this work, we share our
living lab software stack and present our experience developing a platform
that supports qualitative and quantitative experiments from the ground up. We
propose the first open-source interoperable living lab platform for
multidisciplinary smart environment research.

living labs, cyber-physical systems

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: ; May 9–12, 2023; San Antonio, TX††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computer systems organization High-
level language architectures††ccs: Software and its engineering Software
design engineering

##  1\. Introduction and Related Works

The average American spends more than 90% of their lives indoors (Klepeis et
al., 2001; Awada et al., 2021), and buildings account for 40% of the total
energy consumption in America (EIA, 2017). Together, it is unsurprising to
find that if a building is properly designed and operated around the
occupants’ needs, preferences, and comfort levels, we can reduce consumption
significantly (Wagner et al., 2018). In addition to reducing energy
consumption, the study of the indoor environment has also been shown to have a
dramatic effect on the occupant’s health, and well-being (Wang et al., 2022;
Baniassadi et al., 2021; Cedenõ-Laurent et al., 2018). These studies indicate
that the improvement of the indoor environment is not only worthwhile
financially but also pressing healthfully. With so many benefits, why is
occupant data still so vastly under-exploited (Wagner et al., 2018)?

On the energy side, a lack of standardization in the production of buildings
compared to the automobile industry and poor information and communications
technology (ICT) infrastructure in pre-existing buildings prevents building
managers from achieving the 15%-50% energy-saving advanced control strategies
have demonstrated (Drgoňa et al., 2020). On the health side, studies have
shown that proper management of the environment can lead to better
physiological and psychological outcomes for occupants. However, relying on
employee self-reported surveys instead of quantitative measures through Health
Performance Indicators (HPI) limits the potential for buildings to support
occupant health and well-being (Allen and Macomber, 2020). Researchers have
created an approach called ”Living Labs” to tackle these issues together.
While many definitions for a living lab exist (Ballon et al., 2005; Niitamo et
al., 2006; Bergvall-Kareborn et al., 2009), a previous survey of existing
living labs proposed a general definition (Cureau et al., 2022):

> “A living lab… is a … typical indoor environment where everyday tasks are
> performed by occupants over a significant period of time to experimentally
> characterize their activities and responses, and with a permanent setup that
> allows hosting scientific experiments … by monitoring and controlling the
> indoor conditions…”

However, the definition assumes certain qualifiers that make it flexible for
interpretation. For instance, what is a typical indoor environment, what are
everyday tasks, and what is a significant period of time? We sampled a list of
living labs from the surveys and extended columns about survey distribution,
device deployment, and accessibility in Table 1. Notably, most existing living
labs do not have their software infrastructure code readily accessible. We
identify this lack of a generalizable ICT infrastructure as an opportunity to
promote international collaboration and to retrofit existing buildings with a
platform to design and test smart indoor environment applications. Research
demonstrating this bottom-up approach to smart environments can already be
seen in innovations for rooms (Zhao et al., 2017) and through smart furniture
applications (Aryal et al., 2019). These studies not only enable more user-
centered control schemes to target environments at a micro-climate level but
also give researchers the freedom to explore interactions that, if failed,
will not compromise essential building systems.

Name |  Country |  Source Code Accessible |  Surveys Distributed |  Sensors Deployed |  Longest Study Duration |  Reference
---|---|---|---|---|---|---
BNZEB |  Italy |  N |  N |  Y |  1+ years |  (Ascione et al., 2020)
|  |  |  N |  Y |  1+ years |  (Ascione et al., 2019)
Carleton University Buildings |  Canada |  N |  N |  Y |  15 months |  (Bursill et al., 2020)
|  |  |  N |  Y |  7-12 months |  (Gunay et al., 2018)
|  |  |  N |  Y |  1 month preliminary data collection + 9 weeks study |  (Gilani and O’Brien, 2018)
CIRIAF Offices |  Italy |  N |  Y |  Y |  1 year |  (Pisello et al., 2016)
|  |  N |  Y |  Y |  1 year and 2 years |  (Piselli and Pisello, 2019)
David Bower Center |  California, USA |  N |  Y |  Y |  2 months |  (Roa et al., 2020)
ENERGISE Living Labs |  Denmark, Finland, Germany, Hungary, Ireland, Switzerland, the Netherlands, and United Kingdom |  N |  Y |  N |  11 months |  (Sahakian et al., 2021)
House Living Labs Project |  Australia |  N |  N |  Y |  1 year |  (Eon et al., 2018)
Sutardja Dai Hall at University of California, Berkeley |  California, USA |  N |  N |  Y |  Two weekend days |  (Ma and Maasoumy, 2011)
Well Living Lab |  Minnesota, USA |  N |  Y |  Y |  18 weeks |  (Clements et al., 2019)
|  |  |  Y |  Y |  4 weeks |  (Zhang et al., 2020)
|  |  |  Y |  Y |  14 weeks |  (Jamrozik et al., 2019)
ZEB Living Lab |  Norway |  N |  N |  Y |  N/A (description of the test facility) |  (Goia et al., 2015)
Table 1. Sample of living labs constructed after 2000 (adopted from (Cureau et
al., 2022)). Studies can last anywhere from 4 weeks to 2 years, almost all the
labs have sensors deployed, and some use surveys as an additional input
stream. We could not readily access the underlying infrastructure for any of
these labs.

While applications often receive more direct attention from occupants, the
infrastructure and platform layers are also significant areas to address
(Dryjanski et al., 2020). In this work, we elect to focus our contributions in
the living lab ICT software infrastructure domain. Specifically, we share the
lessons we learned while attempting to create a living lab from the ground up.
We then propose a set of standardized components for living labs
infrastructure and open source our code to support future research teams in
streamlining their software development efforts.

We organize our paper as follows: we first describe the problems we’ve
encountered while setting up a living lab (Section 2). Then, we elaborate on
the solution by describing the overview of our framework and the relationship
between the problems and the components (Section 3). We then describe the
solutions and lessons we’ve learned when establishing a living lab (Section
4). Then, we point towards limitations and directions for future work (Section
5). Finally, we give our concluding remarks (Section 6), acknowledgements
(Section 8), and share the online resources (Section 7).

##  2\. Challenges

In this section, we share the problems we encountered as we set up our living
lab from the ground up, leading to the architecture and framework described in
Section 3.

###  2.1. Sensor, participants, and surveys can scale faster than the research
team

One of the first challenges we encountered when setting up our living lab was
the registration of newly arrived devices and their locations. At first, we
settled the problem by having a building information model (BIM) and an excel
sheet with some device metadata. However, as the number of participants and
sensors grew, we realized that it was ineffective to manually update the BIM
model or the excel sheet whenever researchers introduced a new device or an
occupant submitted a maintenance request. We identify a need to enable
granular control of access to smart environment metadata information,
including keeping track of an inventory of cyber and physical assets.

Keeping this inventory is helpful because, over time, the same issue for
managing the physical object of a living lab can manifest itself in tracking
the system’s data quality. For example, one of the more insidious problems we
encountered came from sensors that were still streaming data. However,
information became lost in transit due to the number of walls between the
sensor and the gateway receiving the data (Wang et al., 2020). If we had
assumed all sensors that were streaming data to be valid, we would have missed
an increasing amount of data loss over time. Another problem we encountered
was modern-day “smart-outlets”, which disabled themselves based on their local
occupancy sensor readings. In other words, data continued to stream until late
at night when no occupants were at the lab and when the researchers might be
sleeping. If we had a dashboard platform where we could visualize the system’s
activity at night, we would have been able to observe this issue. Finally, we
also encountered the challenge of human hazard, as previously reported (Hnat
et al., 2011; Barrenetxea et al., 2008). The daily activity of the occupants
(e.g., running into walls) dislodged sensors and removed gateways. Through all
these challenges, we identified a need to locate invalid data through
dashboards and automated routines. Visualizing plots and automatically
discovering outliers reduce the maintenance burden for the research team and
enable consistent data collection.

###  2.2. Proprietary, heterogeneous software, hardware, and skill sets can
limit the ability for a team to work together

During our experience collaborating with other labs, teams generally enter
with a different collection of physical and digital tools, bolstering the
number of interactions and avenues for research. However, many of these IoTs
come with companion algorithms and online platforms, leading to avoidable
subscription fees and repeated development efforts to integrate the tools.
Furthermore, keeping a chain of custody for these different software and
hardware components becomes burdensome and difficult to assign among the labs
and participants. These challenges exponentiate when a mixture of
collaborators with different skill sets, backgrounds, and interests come
together. We either needed to build additional infrastructure to improve
access or ran into bottlenecks for the operation that rose to the combined
level required to navigate the tools, often leading to projects with a low
truck factor (Avelino et al., 2016). Consequently, we identified a need fora
living lab platform that can reduce the technical barriers of entry and enable
people to maintain it with a variety of skill sets.

While lowering the barrier to entry enables different people to help with a
living lab operation, to do research involves exploring new areas and
implementing connections that might not yet exist. We realized that living lab
platforms mandate the ability to integrate new research areas into an existing
ecosystem of people and devices. For instance, using self-powered sensors can
require a gateway topology, requiring researchers to consolidate between
different time scales and radio protocols. We also were allowed to integrate
autonomous sensors (i.e., robots), which required the ability to fit the
robot-sampled environmental variables into a shared spatial coordinate with
other known devices (Jin et al., 2017). Finally, edge computing paradigms
challenge assumptions of deployment locations for living lab platforms. For
example, IoTs can sometimes better be deployed on the edge without utilizing
cloud platforms (Nasir and Campbell, 2021). Through this large variety of
topics, we identified a need for a living lab platform to enable the
interaction of heterogeneous devices, software, and changing paradigms of
computing.

##  3\. Framework Description

In this section, we describe the framework we have built to address the needs
identified in the previous section. We first describe an overall architecture
of the system (Section 3.1). Then, we elaborate on the individual modules and
considerations (Section 3.2).

###  3.1. Overview

Figure 1 showcases a conceptual diagram of the living lab system. We separate
digital and physical types. Physical members are researchers, developers,
organizers, and participants. A researcher analyzes and interprets the
collected information from the environment and the participants. Participants
are occupants who are enrolled in a living lab study. A developer focuses on
updating and maintaining the software infrastructure, and an organizer helps
with the operations from a limited-technical capacity. The digital
representation specifies an increasing amount of permissions, from users to
staff, to administrators. For example, a user has access to the web interface
but cannot access privileged views that a staff user can. Staff users can
additionally gain permission to modify the value of models stored in the
framework. An administrator privilege gives a user staff privileges but
additional the ability to create or delete models. In other words, a user is
permission granted to people in cyberspace with access to the system, which
stands distinct from an occupant, anyone who physically occupies the building.
People who dwell in the building but do not subscribe to the system are
considered occupants but not users (i.e., a non-participant). Researchers can
be a user and not an occupant. We make these conceptual distinctions because
we recognize during our deployment that blanket assumptions about the
technical skill set and categorical designation for people in the building can
limit the research teams’ ability to interact with the community. We expect
future users of living labs will face similar difficulties, so summarize these
distinctions in Table 2.

Figure 1. Physical overview of the Living Link Lab System. The bottom-up architecture assumes that people can start their research agnostic to existing building infrastructure systems. Physical Descriptor |  Technical |  Cyber Permissions |  Role
---|---|---|---
Researcher |  No |  Admin |  Oversees the operation of the living lab and analysis of data
Developer |  Yes |  Admin |  Maintains and develops the framework
Organizer |  No |  Staff |  Maintains and develops the relationship between participants and researchers, and also support with the logistics the research study
Participant |  No |  User |  A person who interacts with and has data collected to the living lab
Non-participant |  No |  Non-user |  A person who occupies the building that a living lab study is being conducted
Table 2. Conceptual organization of members. Limiting permissions for
different types protect the system from invalid use and also effectively
consolidates responsibilities. The technical column explains the necessity for
the research team member to know how to programmatically engage the system
(instead of engaging a Graphical User Interface (GUI)).

In addition to the conceptual categorization of use cases, the implemented
living link lab web framework uses a Model View Template (MVT) architecture
(Vincent, 2021). Specifically, a model provides an interface to data stored in
a database, a template handles all static components of a web page, and a view
renders a response to the user by combining information drawn from models and
templates. Generally, under each module is a testing sequence and a view that
takes the model and template and serves an HTTP response to the participant.
Figure 2 shows a high-level relationship graph between the different modules.
For example, a participant can be digitally represented and have a connection
with a set of devices, locations, and surveys. First, a researcher creates a
floor plan representing the location where occupants can inhabit. Then, a
participant is assigned a seat, relating the participant to the floor plan.
The system can then use the distance between participant seating and
registered device locations to assign specific sensors to the participant.
When a participant is created and assigned a sensor, the system creates a
participant-specific dashboard and a device-specific dashboard. Finally, a
digital representation is created in the survey model when a participant is
assigned surveys. For further details about each of the models and fields, we
include a detailed export of our existing system in Figure 10 of the appendix.

Figure 2. Cyber model diagram describing the dependency relationship for
occupants in living labs. The direction of the arrow describes the sequence
with which model objects are deleted when they are removed. For example,
deleting a user from the system triggers a cascade of deletions downstream,
removing the user’s dashboard, the user’s seating information, and all
associated survey metadata. However, removing a custom user will not delete
registered devices from the system, nor delete floor plans or surveys.

###  3.2. Design Decisions

In order to build in extensibility while minimizing complexity, we operated
under the assumption that each module should be only loosely-coupled with the
user’s digital representation (the user’s model). Specifically, the less code
required to sustain a system, the easier it can be to debug and maintain it.
For example, we anticipate that some use cases might not need a survey module
(as shown in Table 1), so the framework allows researchers to remove the
surveys module in settings.

We separated the components that would benefit from a living lab
infrastructure into the following modules: accounts, surveys, devices, floor
plans, and dashboards. The accounts module encapsulates user metadata and
access. The surveys module encapsulates the survey metadata. The devices
module encapsulates the device metadata. The floor plans module stores
different environmental contexts which allow tagging of user or device into
the time series database, and the dashboard module automates the generation of
panels based on created devices. Below, we elaborate on the roles and
functions of these modules.

#### Accounts

The accounts module represents the user in digital form, which contains the
addition of necessary information to the participant’s metadata, such as age
group or occupation. For example, an organizer with staff user privileges can
read compliance views and write to the surveys model to help handle the day-
to-day operation of surveys. For example, if a user needs more time to
complete a survey or if a user does not receive an email, an organizer would
be able to edit the system to redistribute or extend the deadline for surveys.
However, limiting the privileges disallow organizers to create or delete
accounts model. This lack of admin user privileges means that staff users
cannot inadvertently delete a researcher’s account when they edit the system.

The user authentication workflow generally works “out of the box” using modern
web frameworks such as Django, which also helps alleviate the burden on the
researchers to maintain the most up-to-date security features (something
commonly overlooked by living lab researchers and organizers). In our
framework, we utilize three classes of privileges as described in Table 2: 1)
user, who interacts with the system with no additional privileges, 2) staff,
who can gain read and write and gain access to restricted views such as a
survey compliance dashboard or specific admin data as assigned, and 3) admin,
who has the highest level of access to the web interface of the system, who
can not only alter values for existing models but can also create and delete
objects to improve the operation of the framework. For instance, admin users
have the authority to remove staff users or delete other users from the
system. These distinctions help facilitate the smooth scaling of the living
lab operations by allowing researchers to share responsibility.

#### Floor Plans

We include a module called Floor Plans because we found it essential for users
to note their seating arrangements during our deployment. Additionally,
modern-day occupancy habits no longer conform to a single permanent seating
location. Hybrid work environments and hoteling spaces make tracking multiple
spaces for users mandatory in certain situations. We couple the floor plans
model with the user and device models, allowing us to explore custom ways to
automate device visibility with users. For example, a custom routine can be
created to assign all known devices of a building to users within a fixed
geometric radius to their assigned seating. Floor plans are critical because
it enables the adding, removing, and relation of digital representations of
physical locations.

#### Surveys

We created a surveys module because we found that designing surveys correctly
often requires a dedicated service such as Qualtrics. The survey system must
be flexible to support the distribution of surveys based on the research or
study’s needs. Based on our experience and reviewing previous living lab
studies, surveys could be distributed at the occurrence of an event, daily,
weekly, monthly, or other set periods. Furthermore, tracking survey compliance
is another element from which the organizer and researcher can benefit. Such
compliance checks can be done internally to enable real-time adjustments and
delivery strategy testing. As a result, we implemented an ”anonymous link”
survey workflow, shown in Figure 3, where we keep track of the association
between user-and data by altering the ensuing hash of user information. For
example, if we hash the user password, researchers with only access to survey
data cannot identify the user who took the survey. We can also create an
identifier by including survey metadata and user information, resulting in
unique hashes for every row of survey data based on user-survey pairs. By
doing so, a researcher can only identify which participant answered the survey
by accessing the metadata and survey databases.

Figure 3. Example survey workflow: participants can be protected from
identification from researchers downstream who only have access to the survey
data if the survey identifier is a hash of user-survey specific information. A
researcher would require both the user and survey metadata information to
retrieve the hash to identify the survey owner.

#### Devices

The devices module contains critical pieces of information to help streamline
data upload and retrieval. Specifically, we store the “tag” information for
time-invariant data and open APIs to stream “field” information for time-
variant data. For example, we can include a URL for adding humidity as an
input stream to a device or a URL to check and only upload data whose field
matches known fields. We include an example data point in Figure 5 to help
illustrate the process and the concept between fields and tags.

⬇

{

’time’: ’2020-12-23T23:54:50.727Z’,

’device_id’: ’503eaa71b92a’, # tag

’location_general’: ’Link Lab’, #tag

’location_specific’: ’grid_5’, #tag

’fieldname’: ’heartbeat’ # tag

’system_version’: ’lll-1.0.0’, #tag

’value’: 1, # field

’counter’: 256, #field

}

Figure 4. Example data point written to the time series database. Generally,
tags are variables to query on that are more time-invariant, while fields are
variables that change more dynamically.

In the figure, time tracks when the data point is uploaded. The device
identifier is a unique value that identifies the devices, location general
allows for context-specific coordinates that are specified in location
specific. For example, location general can be the building where the device
is placed, and location specific the room the device is in. The field
variables enable visualization of different devices onto the same dashboard,
such as humidity. The value stores the sampling information. For instance,
humidity can be a field, and 40% can be a value. However, we only track the
device identifier, the location general, and the location specific to the
metadata database. For the actual data entry, we enable API access on the
website to pipe information directly into a back-end time series database. It
is important to note that there is a difference between tags and fields. For
our database, the difference is that one signifies which variables will be
queried versus which variables will just be tracked.

#### Dashboards

During our deployment, we found that having real-time dashboard users can
access was a greatly desired feature. Real-time dashboards enable users to
monitor their surroundings and potentially enable new avenues for users to
utilize their microclimate information. However, we found that dashboards also
have a public and private components. Some public information is irrelevant to
the user, such as the air quality of the building next door that they do not
care about sharing. However, some information is private, such as the
occupancy schedule detected at their seat, which they mind sharing with
others. To adjust to the needs and ethics of information distribution, we
approached the design of the dashboard module by separating a user-specific
dashboard and a device-specific dashboard. User dashboards allow users to
customize the information relevant to their day-to-day. In contrast, device
dashboards contain information about the individual device, which enables
device-specific insights and debugging. We make no stipulations about which
information should or should not be shared. We mainly provide opportunities
for developers and researchers the space to pick and choose how the assignment
policy should be.

##  4\. Lessons learned

In this section, we describe our lessons learned and the requirements we have
observed that map to one or a combination of the module described in the
previous section.

###  4.1. Quickly see if a sensor is working

When installing a sensor, reducing the number of steps to generate a
corresponding dashboard of the sensor’s collected values helps researchers
validate the sensor. In most cases, owners of devices only want to verify that
data is actually streaming in from the devices and does not need to be
replaced. For example, a light sensor can be observing data within range of
its data specification sheet, but showing zero lux when light is on and 1,000
lux when light is off. A casual inspection of the signals with software will
not be able to find this difference, but a researcher could catch this error
to mark and fix the faulty device. However, having a dashboard is also
tremendously rewarding when everything is functioning correctly as the
researcher can see a possible noteworthy trend or issues with the sensor. In
Figure 5, we showcase an example of a generated dashboard that organizers can
create without coding.

Figure 5. A generated dashboards showing real-time air quality data for the
last 6 hours.

In addition to manual inspection, automated fault detection can assist
researchers in discovering more insidious faults. In figure 6, for example, we
show that sensors can fail less noticeably by collecting partial data and ways
to detect it (Wang et al., 2020). Having the device model paired with the
floor plans module allows us to quickly identify which sensors are
malfunctioning and quickly fix or replace them if needed. However, as we have
described Section 2.1, having a system that does not need to sleep routinely
check in place of the user would free the researcher up to do other
developments that can not easily be automated, such as installing the sensor
or onboarding participants. Without knowing where the sensors are, we cannot
quickly locate them to replace them. We did realize that it might be
challenging to map precisely the coordinate of where the device is concerning
a pre-defined origin in space; therefore, we introduced the grid system so the
users can change the resolution to a level they can support. Consensus-based
methods can also be used to help flag outliers in collected data (Chen et al.,
2018).

Figure 6. A month of lighting data, showcasing partial and complete data loss
for more than 30 sensors. Figure 7. Tracking device locations in 3D software
enable researchers to record precise locations and spatial context. However,
the skill set required to model and manipulate the digital model makes
labeling and recording the location of devices prohibitive for organizers
without modeling skills. Furthermore, licensing or other operating systems
requirements make it challenging to access and maintain automated routines to
track changes across time.

###  4.2. Allow for flexible survey delivery

During our framework deployment, we realized that we often ran into issues
about how the surveys were deployed, such as needing to re-word and remove
questions that might have been redundant. Furthermore, we ran into situations
where we had to onboard participants at separate times of their convenience,
which meant that we could not do things like send out a mass email after
everyone had been registered for the study. These challenges come in addition
to needing to calculate and see which surveys have been completed to reimburse
the participants correctly.

The survey module is designed with open times and close times and pairs with
users. We marked open time, user, and survey URLs as unique together because,
individually, they can repeat with survey objects. For example, different
users can be subject to the same surveys simultaneously at the same time.
Having user-survey-time-specific objects also allows the organizers to
automate portions of survey emailing and track compliance in real time. For
example, we show a quick example in Listing 8 where developers can quickly
filter surveys by user and by completion status.

⬇

from surveys.models import Survey

total_user_surveys= Survey.objects.filter(user__username = <username>)

total_user_surveys_completed = total_surveys.filter(completed=True)

Figure 8. listing

Query number of completed surveys for any given user in real-time

###  4.3. Allow users and devices to move

One of our most rewarding experiences came from the need to deploy a Temi-
robot 111https://www.robotemi.com/ to sample environmental components. We
quickly realized a need to make a mental model for which aspects of our system
are dynamic and which parts of the system are static. For example, static
components can be stored as tags, which enables us to query the database and
find relevant data about that item. This manifests in things like device
identifiers or the floor where the sensor is installed. However, for a robot,
what could have been a tag can now become a field, or things we need to record
but do not make queries directly. For example, tracking the robot’s x, y, z
coordinates through space with floating points might not reoccur for long
periods, reducing the value gained with the query. In this example, though, we
demonstrate a fundamental tension between the need to track movements in space
and the immutable nature of the environment. Towards this end, we implemented
grids, making it easier for users to query and record sensor locations but
giving room for a more granular field such as “coordinates”.

###  4.4. Minimize the technical skill set required for researchers and
organizers to participate in the operation of the living lab

Depending on the team size and resources, it may be infeasible to expect
everyone in the research group to have the necessary software development
skills. However, checking for user compliance and reaching out to users are
essential tasks that can be accomplished without technical skills. By
incorporating user interfaces that allow for the query and modification of
data without needing to write code, developers can empower organizers to help
maintain and fix the operations of the living lab. Additionally, incorporating
traditional web forms as opposed to navigating computer-aided design software
such as Rhino in Figure 7 reduces the technical barrier for organizers and
researchers to help track sensor location information.

###  4.5. Allow participants to control and protect their data

Collecting long-term sensitive participant information can quickly become
privacy-intrusive. Creating schemes to enable users to have the pro-activity
to delete data and observe their information gives control and ownership of
information back to the participants. Exposing models into forms and APIs such
as through Django Rest Framework 222https://www.django-rest-framework.org/api-
guide/schemas/ enables participants to make changes to the model controlling
the system. Similarly, using data coupling instead of control coupling
(Jasinski, 2016) between the web framework and associated applications allows
the severing of the relationship without deleting the user.

Figure 9. Example screenshot from Qualtrics. Embedding a hash of the user
information alongside with the surveys identifiers allow participants to
remove themselves from prior identification by changing the portions of the
hashed information (e.g., changing the password changes the user-identifier on
the survey platform).

###  4.6. Connect to humans as soon as possible and beware the timescales

We observed interesting environmental trends across annual periods during our
living lab infrastructure setup. However, we could not make any claims about
the relationships between health aspects and environmental exposure without
connecting the data to the underlying population. During our more recent
works, we realized that merging datasets also brings to light issues of time
scales. For example, a light sensor can have a sampling frequency of once
every fifteen minutes, assuming that the data being tracked relates to the
sun’s movement. These types of sensors would be challenging to coalesce if the
behavior observed is at a smaller time scale, such as once every second.
Referencing the Nyquist sampling rate (Landau, 1967), we suggest aiming for at
least twice the sampling rate than the fastest observable instance of the
behavior and noting that combining sensor information brings the time scale
down to the largest common denominator. Furthermore, having insights that
directly relate to a human component (such as occupancy count) instead of a
proxy variable (such as CO2) can improve the quality of the observation.

###  4.7. Software development is not research

We found it non-trivial to organize and retain a changing collection of user,
device, technical backgrounds, and surveys, and also keep up with the
development work necessary to test out new ideas and connect between
commercial off the shelf devices and custom-build devices. We realized that we
spent a significant amount of time implementing the software stack, of itself
does not lead to tangible research outcomes. Furthermore, we realized that
there are large variety of team structure for research labs, some of which do
not have dedicated software development staff to support the research
questions they might have but have great ideas to contribute.

##  5\. Limitations and Future Works

We anticipate many interesting research areas to emerge from the use of this
system, and also many improvements that can be made with our initial
implementation. We list a couple of future directions we anticipate are
possible:

#### Autonomous Occupancy Polling Stations

Occupant polling stations have been investigated as a strategy to track
thermal comfort in a building (Lassen et al., 2020), but the integration of
polling stations with robots have yet to our knowledge be fully explored. By
combining the survey, device, accounts, and floor plans modules it could be
possible to conduct thermal comfort surveys with granular location and time
information.

#### Simulations and Digital Twins

The current floor plans module can be replaced with more sophisticated smart
building simulation frameworks to represent larger spatial contexts. For
example, Weber et al. demonstrates how caustic light patterns can be predicted
reliably using photon mapping for complex 3D-printed glass structures (Weber
et al., 2020). Connecting the system to simulation platforms enable more
research into more sophisticated interfaces or machine learning applications,
such as tracking user activity through lighting signals, but more importantly
this enables researchers to implement the new application into their own
living labs.

##  6\. Conclusion

This paper introduces a bottom-up living lab framework and demonstrates key
strategies to implement and maintain operations for a living lab
infrastructure. By using the accounts, surveys, devices, floor plans, and
dashboard modules, future researchers are freed to better explore
relationships and implementations for the living labs of tomorrow.

##  7\. Online Resources

We include at https://github.com/livinglinklab/lll.git the repository
containing the source code, and docker for the proposed living lab framework.

##  8\. Acknowledgments

We thank the University of Virginia (UVa) Link Lab and affiliated professors,
staff, and students for their contributions and discourse that without which
this work would not have been possible. We also want to thank the support of
the UVa Facility Management Services team, who supplied us with plans and 3d
models of the building.

## References

  * Klepeis et al. [2001] Neil E. Klepeis, William C. Nelson, Wayne R. Ott, John P. Robinson, Andy M. Tsang, Paul Switzer, Joseph V. Behar, Stephen C. Hern, and William H. Engelmann.  The national human activity pattern survey (nhaps): a resource for assessing exposure to environmental pollutants.  _Journal of Exposure Science & Environmental Epidemiology 2001 11:3_, 11:231–252, 7 2001.  ISSN 1559-064X.  doi: 10.1038/sj.jea.7500165.  URL https://www.nature.com/articles/7500165.
  * Awada et al. [2021] Mohamad Awada, Burcin Becerik-Gerber, Simi Hoque, Zheng O’Neill, Giulia Pedrielli, Jin Wen, and Teresa Wu.  Ten questions concerning occupant health in buildings during normal operations and extreme events including the covid-19 pandemic.  _Building and Environment_ , 188:107480, 1 2021.  ISSN 0360-1323.  doi: 10.1016/J.BUILDENV.2020.107480.
  * EIA [2017] US EIA.  How much energy is consumed in us residential and commercial buildings?  _United States Energy Information Administration, Washington, DC, accessed Dec_ , 11:2018, 2017.
  * Wagner et al. [2018] Andreas Wagner, William O’Brien, and Bing Dong.  Exploring occupant behavior in buildings.  _Wagner, A., O’Brien, W., Dong, B., Eds_ , 2018.
  * Wang et al. [2022] Mengmeng Wang, Lili Li, Caixia Hou, Xiaotong Guo, and Hanliang Fu.  Building and health: Mapping the knowledge development of sick building syndrome.  _Buildings_ , 12:287, 3 2022.  ISSN 2075-5309.  doi: 10.3390/BUILDINGS12030287.  URL https://www-mdpi-com.proxy01.its.virginia.edu/2075-5309/12/3/287/htmhttps://www-mdpi-com.proxy01.its.virginia.edu/2075-5309/12/3/287.
  * Baniassadi et al. [2021] Amir Baniassadi, Brad Manor, Wanting Yu, Lewis Lipsitz, and Alvaro Pascual-Leone.  A platform to study the effects of home environment on health and wellbeing of older adults.  _Innovation in Aging_ , 5:95, 12 2021.  doi: 10.1093/GERONI/IGAB046.3453.  URL https://academic.oup.com/innovateage/article/5/Supplement_1/957/6468226.
  * Cedenõ-Laurent et al. [2018] J. G. Cedenõ-Laurent, A. Williams, P. MacNaughton, X. Cao, E. Eitland, J. Spengler, and J. Allen.  Building evidence for health: Green buildings, current science, and future challenges.  _Annual Review of Public Health_ , 39:291–308, 4 2018.  ISSN 15452093.  doi: 10.1146/ANNUREV-PUBLHEALTH-031816-044420.  URL https://www.annualreviews.org/doi/abs/10.1146/annurev-publhealth-031816-044420.
  * Drgoňa et al. [2020] Ján Drgoňa, Javier Arroyo, Iago Cupeiro Figueroa, David Blum, Krzysztof Arendt, Donghun Kim, Enric Perarnau Ollé, Juraj Oravec, Michael Wetter, Draguna L Vrabie, et al.  All you need to know about model predictive control for buildings.  _Annual Reviews in Control_ , 50:190–232, 2020.
  * Allen and Macomber [2020] Joseph Allen and John D Macomber.  Healthy buildings.  2020\.
  * Ballon et al. [2005] Pieter Ballon, Jo Pierson, and Simon Delaere.  Test and experimentation platforms for broadband innovation: Examining european practice.  _Available at SSRN 1331557_ , 2005.
  * Niitamo et al. [2006] Veli-Pekka Niitamo, Seija Kulkki, Mats Eriksson, and Karl A Hribernik.  State-of-the-art and good practice in the field of living labs.  In _2006 IEEE international technology management conference (ICE)_ , pages 1–8. IEEE, 2006.
  * Bergvall-Kareborn et al. [2009] BHMSA Bergvall-Kareborn, M Hoist, and Anna Stahlbrost.  Concept design with a living lab approach.  In _2009 42nd Hawaii international conference on system sciences_ , pages 1–10. IEEE, 2009.
  * Cureau et al. [2022] RJ Cureau, Ilaria Pigliautile, Anna Laura Pisello, Mateus Bavaresco, Christiane Berger, Giorgia Chinazzo, Zs Deme Belafi, A Ghahramani, Arsalan Heydarian, D Kastner, et al.  Bridging the gap from test rooms to field-tests for human indoor comfort studies: A critical review of the sustainability potential of living laboratories.  _Energy Research & Social Science_, 92:102778, 2022.
  * Zhao et al. [2017] Nan Zhao, Asaph Azaria, and Joseph A Paradiso.  Mediated atmospheres: A multimodal mediated work environment.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 1(2):1–23, 2017.
  * Aryal et al. [2019] Ashrant Aryal, Burcin Becerik-Gerber, Francesco Anselmo, Shawn C Roll, and Gale M Lucas.  Smart desks to promote comfort, health, and productivity in offices: A vision for future workplaces.  _Frontiers in Built Environment_ , page 76, 2019.
  * Ascione et al. [2020] Fabrizio Ascione, Martina Borrelli, Rosa Francesca De Masi, and Giuseppe Peter Vanol.  Nearly zero energy target and indoor comfort in mediterranean climate: Discussion based on monitoring data for a real case study.  _Sustainable Cities and Society_ , 61:102349, 2020.
  * Ascione et al. [2019] Fabrizio Ascione, Martina Borrelli, Rosa Francesca De Masi, Filippo de Rossi, and Giuseppe Peter Vanoli.  A framework for nzeb design in mediterranean climate: Design, building and set-up monitoring of a lab-small villa.  _Solar Energy_ , 184:11–29, 2019.
  * Bursill et al. [2020] Jayson Bursill, Liam O’Brien, and Ian Beausoleil-Morrison.  Multi-zone field study of rule extraction control to simplify implementation of predictive control to reduce building energy use.  _Energy and Buildings_ , 222:110056, 2020.
  * Gunay et al. [2018] H Burak Gunay, William O’Brien, Ian Beausoleil-Morrison, and Jayson Bursill.  Development and implementation of a thermostat learning algorithm.  _Science and Technology for the Built Environment_ , 24(1):43–56, 2018.
  * Gilani and O’Brien [2018] Sara Gilani and William O’Brien.  A preliminary study of occupants’ use of manual lighting controls in private offices: A case study.  _Energy and Buildings_ , 159:572–586, 2018.
  * Pisello et al. [2016] Anna Laura Pisello, Veronica Lucia Castaldo, Cristina Piselli, Claudia Fabiani, and Franco Cotana.  How peers’ personal attitudes affect indoor microclimate and energy need in an institutional building: Results from a continuous monitoring campaign in summer and winter conditions.  _Energy and Buildings_ , 126:485–497, 2016.
  * Piselli and Pisello [2019] Cristina Piselli and Anna Laura Pisello.  Occupant behavior long-term continuous monitoring integrated to prediction models: Impact on office building energy performance.  _Energy_ , 176:667–681, 2019.
  * Roa et al. [2020] Carlos Duarte Roa, Stefano Schiavon, and Thomas Parkinson.  Targeted occupant surveys: A novel method to effectively relate occupant feedback with environmental conditions.  _Building and Environment_ , 184:107129, 2020.
  * Sahakian et al. [2021] Marlyne Sahakian, Henrike Rau, Eoin Grealis, Laurence Godin, Grégoire Wallenborn, Julia Backhaus, Freja Friis, Audley T Genus, Gary Goggins, Eimear Heaslip, et al.  Challenging social norms to recraft practices: A living lab approach to reducing household energy use in eight european countries.  _Energy Research & Social Science_, 72:101881, 2021.
  * Eon et al. [2018] Christine Eon, Gregory M Morrison, and Joshua Byrne.  The influence of design and everyday practices on individual heating and cooling behaviour in residential homes.  _Energy Efficiency_ , 11(2):273–293, 2018.
  * Ma and Maasoumy [2011] Yudong Ma and Mehdi Maasoumy.  Optimal control of the operation of building cooling systems with vav boxes.  Technical report, Citeseer, 2011.
  * Clements et al. [2019] Nicholas Clements, Rongpeng Zhang, Anja Jamrozik, Carolina Campanella, and Brent Bauer.  The spatial and temporal variability of the indoor environmental quality during three simulated office studies at a living lab.  _Buildings_ , 9(3):62, 2019.
  * Zhang et al. [2020] Rongpeng Zhang, Carolina Campanella, Sara Aristizabal, Anja Jamrozik, Jie Zhao, Paige Porter, Shaun Ly, and Brent A Bauer.  Impacts of dynamic led lighting on the well-being and experience of office occupants.  _International journal of environmental research and public health_ , 17(19):7217, 2020.
  * Jamrozik et al. [2019] Anja Jamrozik, Nicholas Clements, Syed Shabih Hasan, Jie Zhao, Rongpeng Zhang, Carolina Campanella, Vivian Loftness, Paige Porter, Shaun Ly, Selena Wang, et al.  Access to daylight and view in an office improves cognitive performance and satisfaction and reduces eyestrain: A controlled crossover study.  _Building and Environment_ , 165:106379, 2019.
  * Goia et al. [2015] Francesco Goia, Luca Finocchiaro, and Arild Gustavsen.  The zeb living laboratory at the norwegian university of science and technology: a zero emission house for engineering and social science experiments.  _Proceedings of 7PHN Sustainable Cities and Buildings_ , 2015.
  * Dryjanski et al. [2020] Marcin Dryjanski, Mateusz Buczkowski, Youssouf Ould-Cheikh-Mouhamedou, and Adrian Kliks.  Adoption of smart cities with a practical smart building implementation.  _IEEE Internet of Things Magazine_ , 3(1):58–63, 2020.
  * Wang et al. [2020] Alan Wang, Jianyu Su, Arsalan Heydarian, Bradford Campbell, and Peter Beling.  Is my sensor sleeping, hibernating, or broken? a data-driven monitoring system for indoor energy harvesting sensors.  In _Proceedings of the 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation_ , pages 210–219, 2020.
  * Hnat et al. [2011] Timothy W Hnat, Vijay Srinivasan, Jiakang Lu, Tamim I Sookoor, Raymond Dawson, John Stankovic, and Kamin Whitehouse.  The hitchhiker’s guide to successful residential sensing deployments.  In _Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems_ , pages 232–245, 2011.
  * Barrenetxea et al. [2008] Guillermo Barrenetxea, François Ingelrest, Gunnar Schaefer, and Martin Vetterli.  The hitchhiker’s guide to successful wireless sensor network deployments.  In _Proceedings of the 6th ACM conference on Embedded network sensor systems_ , pages 43–56, 2008.
  * Avelino et al. [2016] Guilherme Avelino, Leonardo Passos, Andre Hora, and Marco Tulio Valente.  A novel approach for estimating truck factors.  In _2016 IEEE 24th International Conference on Program Comprehension (ICPC)_ , pages 1–10. IEEE, 2016.
  * Jin et al. [2017] Ming Jin, Shichao Liu, Yulun Tian, Mingjian Lu, Stefano Schiavon, and Costas Spanos.  Indoor environmental quality monitoring by autonomous mobile sensing.  In _Proceedings of the 4th ACM International Conference on Systems for Energy-Efficient Built Environments_ , pages 1–4, 2017.
  * Nasir and Campbell [2021] Nabeel Nasir and Bradford Campbell.  Enabling elasticity on the edge using heterogeneous gateways.  In _Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems_ , pages 395–396, 2021.
  * Vincent [2021] William S Vincent.  _Django for Beginners: Build websites with Python and Django_.  WelcomeToCode, 2021.
  * Chen et al. [2018] Alexander B Chen, Madhur Behl, and Jonathan L Goodall.  Trust me, my neighbors say it’s raining outside: ensuring data trustworthiness for crowdsourced weather stations.  In _Proceedings of the 5th Conference on Systems for Built Environments_ , pages 25–28, 2018.
  * Jasinski [2016] Ricardo Jasinski.  _Effective coding with VHDL: principles and best practice_.  MIT Press, 2016.
  * Landau [1967] HJ Landau.  Sampling, data transmission, and the nyquist rate.  _Proceedings of the IEEE_ , 55(10):1701–1706, 1967.
  * Lassen et al. [2020] Niels Lassen, Francesco Goia, Stefano Schiavon, and Jovan Pantelic.  Field investigations of a smiley-face polling station for recording occupant satisfaction with indoor climate.  _Building and Environment_ , 185:107266, 2020.
  * Weber et al. [2020] Ramon E Weber, Christoph Reinhart, and Neri Oxman.  Photon mapping of geometrically complex glass structures: Methods and experimental evaluation.  _Building and Environment_ , 180:106957, 2020.

Figure 10. Example model and fields for the proposed living lab
infrastructure. By building off of an existing framework, researchers can
focus more time developing features directly related to their research
question, as opposed to debugging and testing infrastructural connections.


* [53] G. Grilli di Cortona, E. Hardy, J. Pardo Vega, and G. Villadoro, “The QCD axion, precisely,” JHEP 01 (2016) 034, arXiv:1511.02867 [hep-ph].
  * [54] T. Vonk, F.-K. Guo, and U.-G. Meißner, “Precision calculation of the axion-nucleon coupling in chiral perturbation theory,” JHEP 03 (2020) 138, arXiv:2001.05327 [hep-ph].
  * [55] T. Vonk, F.-K. Guo, and U.-G. Meißner, “The axion-baryon coupling in SU(3) heavy baryon chiral perturbation theory,” JHEP 08 (2021) 024, arXiv:2104.10413 [hep-ph].
  * [56] L. Di Luzio, G. Martinelli, and G. Piazza, “Breakdown of chiral perturbation theory for the axion hot dark matter bound,” Phys. Rev. Lett. 126 no. 24, (2021) 241801, arXiv:2101.10330 [hep-ph].
  * [57] L. Di Luzio and G. Piazza, “a → $\pi$$\pi$$\pi$ decay at next-to-leading order in chiral perturbation theory,” JHEP 12 (2022) 041, arXiv:2206.04061 [hep-ph]. [Erratum: JHEP 05, 018 (2023)].
  * [58] H. Leutwyler and M. A. Shifman, “GOLDSTONE BOSONS GENERATE PECULIAR CONFORMAL ANOMALIES,” Phys. Lett. B 221 (1989) 384–388.
  * [59] J. Gasser and H. Leutwyler, “Chiral Perturbation Theory: Expansions in the Mass of the Strange Quark,” Nucl. Phys. B 250 (1985) 465–516.
  * [60] A. Pich, “Chiral perturbation theory,” Rept. Prog. Phys. 58 (1995) 563–610, arXiv:hep-ph/9502366.
  * [61] R. Gao, Z.-H. Guo, J. A. Oller, and H.-Q. Zhou, “Axion-meson mixing in light of recent lattice $\eta$–$\eta$’ simulations and their two-photon couplings within U(3) chiral theory,” JHEP 04 (2023) 022, arXiv:2211.02867 [hep-ph].
  * [62] S. Scherer, “Introduction to chiral perturbation theory,” Adv. Nucl. Phys. 27 (2003) 277, arXiv:hep-ph/0210398.
  * [63] A. V. Manohar, “Effective field theories,” in Perturbative and Nonperturbative Aspects of Quantum Field Theory, H. Latal and W. Schweiger, eds., pp. 311–362.  Springer Berlin Heidelberg, Berlin, Heidelberg, 1997.
  * [64] E. Jenkins and A. V. Manohar, “Baryon chiral perturbation theory using a heavy fermion lagrangian,” Physics Letters B 255 no. 4, (1991) 558–562. https://www.sciencedirect.com/science/article/pii/037026939190266S.
  * [65] J. F. Donoghue, E. Golowich, and B. R. Holstein, Dynamics of the Standard Model : Second edition, vol. 2.  Oxford University Press, 2014.
  * [66] G. Ecker, “Chiral perturbation theory,” Prog. Part. Nucl. Phys. 35 (1995) 1–80, arXiv:hep-ph/9501357.
  * [67] L. Di Luzio, M. Giannotti, F. Mescia, E. Nardi, S. Okawa, and G. Piazza, “Running into QCD axion phenomenology,” arXiv:2305.11958 [hep-ph].
  * [68] H.-Y. Cheng, “Low-energy Interactions of Scalar and Pseudoscalar Higgs Bosons With Baryons,” Phys. Lett. B 219 (1989) 347–353.
  * [69] H.-Y. Cheng and C.-W. Chiang, “Revisiting Scalar and Pseudoscalar Couplings with Nucleons,” JHEP 07 (2012) 009, arXiv:1202.1292 [hep-ph].
  * [70] Flavour Lattice Averaging Group Collaboration, S. Aoki et al., “FLAG Review 2019: Flavour Lattice Averaging Group (FLAG),” Eur. Phys. J. C 80 no. 2, (2020) 113, arXiv:1902.08191 [hep-lat].
  * [71] J. Gasser and H. Leutwyler, “Quark Masses,” Phys. Rept. 87 (1982) 77–169.
  * [72] A. Alloul, N. D. Christensen, C. Degrande, C. Duhr, and B. Fuks, “FeynRules 2.0 - A complete toolbox for tree-level phenomenology,” Comput. Phys. Commun. 185 (2014) 2250–2300, arXiv:1310.1921 [hep-ph].
  * [73] N. Fettes, U.-G. Meissner, and S. Steininger, “Pion - nucleon scattering in chiral perturbation theory. 1. Isospin symmetric case,” Nucl. Phys. A 640 (1998) 199–234, arXiv:hep-ph/9803266.
  * [74] J. A. Oller, M. Verbeni, and J. Prades, “Meson-baryon effective chiral lagrangians to O(q**3),” JHEP 09 (2006) 079, arXiv:hep-ph/0608204.
  * [75] Particle Data Group Collaboration, R. L. Workman et al., “Review of Particle Physics,” PTEP 2022 (2022) 083C01.
  * [76] B. K. Sahoo, “Improved limits on the hadronic and semihadronic $CP$ violating parameters and role of a dark force carrier in the electric dipole moment of 199Hg,” Phys. Rev. D 95 no. 1, (2017) 013002, arXiv:1612.09371 [hep-ph].
  * [77] C. Abel et al., “Measurement of the Permanent Electric Dipole Moment of the Neutron,” Phys. Rev. Lett. 124 no. 8, (2020) 081803, arXiv:2001.11966 [hep-ex].
  * [78] J. M. Pendlebury et al., “Revised experimental upper limit on the electric dipole moment of the neutron,” Phys. Rev. D 92 no. 9, (2015) 092003, arXiv:1509.04411 [hep-ex].
  * [79] E949, E787 Collaboration, S. Adler et al., “Measurement of the K+ –$>$ pi+ nu nu branching ratio,” Phys. Rev. D 77 (2008) 052003, arXiv:0709.1000 [hep-ex].
  * [80] KOTO Collaboration, J. K. Ahn et al., “Search for the $K_{L}\\!\to\\!\pi^{0}\nu\overline{\nu}$ and $K_{L}\\!\to\\!\pi^{0}X^{0}$ decays at the J-PARC KOTO experiment,” Phys. Rev. Lett. 122 no. 2, (2019) 021802, arXiv:1810.09655 [hep-ex].
  * [81] E787 Collaboration, S. Adler et al., “Search for the decay K+ —$>$ pi+ pi0 neutrino anti-neutrino,” Phys. Rev. D 63 (2001) 032004, arXiv:hep-ex/0009055.
  * [82] E391a Collaboration, R. Ogata et al., “Study of the $K^{0}_{L}\to\pi^{0}\pi^{0}\nu\bar{\nu}$ decay,” Phys. Rev. D 84 (2011) 052009, arXiv:1106.3404 [hep-ex].