# Node-wise Localization of Graph Neural Networks

Zemin Liu1 Yuan Fang1 Chenghao Liu2&Steven C.H. Hoi1,2
1Singapore Management University, Singapore
2Salesforce Research Asia, Singapore
{zmliu<EMAIL_ADDRESS>{chenghao.liu<EMAIL_ADDRESS>
###### Abstract

Graph neural networks (GNNs) emerge as a powerful family of representation
learning models on graphs. To derive node representations, they utilize a
global model that recursively aggregates information from the neighboring
nodes. However, different nodes reside at different parts of the graph in
different local contexts, making their distributions vary across the graph.
Ideally, how a node receives its neighborhood information should be a function
of its local context, to diverge from the global GNN model shared by all
nodes. To utilize node locality without overfitting, we propose a node-wise
localization of GNNs by accounting for both global and local aspects of the
graph. Globally, all nodes on the graph depend on an underlying global GNN to
encode the general patterns across the graph; locally, each node is localized
into a unique model as a function of the global model and its local context.
Finally, we conduct extensive experiments on four benchmark graphs, and
consistently obtain promising performance surpassing the state-of-the-art
GNNs.

##  1 Introduction

Graphs are powerful data structures to model various entities (i.e., nodes)
and their interactions (i.e., edges) simultaneously. To learn the
representations of nodes on a graph, graph neural networks (GNNs) Wu et al.
(2020) have been proposed as a promising solution. Generally, state-of-the-art
GNNs Kipf and Welling (2017); Hamilton et al. (2017); Veličković et al. (2018)
learn the representation of each node by recursively transferring and
aggregating information from its receptive field, which is often defined as
its set of neighboring nodes. Consider a toy citation graph in Fig. 1(a),
consisting of papers in three areas: biology (_bio_), bioinformatics
(_bioinf_) and computer science (_cs_). To derive the representation of a
node, say $v_{1}$, GNNs aggregate keyword features from not only $v_{1}$
itself, but also its neighbors, namely $v_{2},v_{4},v_{5}$ and $v_{6}$, as
illustrated in Fig. 1(b). Such neighborhood aggregation can be performed
recursively for the neighbors as well in more layers, to fully exploit the
graph structures. To extract useful representations from the neighbors, GNNs
aim to learn the model parameters formulated as a sequence of weight matrices
$\mathbf{W}^{1},\mathbf{W}^{2},\dots,\mathbf{W}^{l}$ in each layer.

Figure 1: Comparison of conventional GNNs and the proposed localization (best
viewed in color).

However, the implicit assumption of a global weight matrix (in each layer) for
the entire graph is often too strict. Nodes do not distribute uniformly over
the graph, and are associated with different local contexts. For instance, in
Fig. 1(c), node $v_{1}$ is associated with a _bio_ context, $v_{2}$ with a
_bioinf_ context and $v_{3}$ with a _cs_ context. Different local contexts are
characterized by different keyword features, such as “gene” and “cell” in
_bio_ , “gene” and “svm” in _bioinf_ , as well as “svm” and “vc-dim” in _cs_.
Thus, a graph-level global weight matrix is inherently inadequate to express
the varying importance of features at different localities. More specifically,
a global weight matrix can be overly diffuse, for different nodes often have
distinct optimal weight matrices, which tend to pull the model in many
opposing directions. This may result in a biased model that centers its mass
around the most frequent patterns while leaving others not well covered. A
natural question follows: _Can we allow each node to be parameterized by its
own weight matrix?_ Unfortunately, this is likely to cause severe overfitting
to local noises and suffer from significantly higher training cost.

In this work, to adapt to the local context of each node without overfitting,
we localize the model for each node from a shared global model. As illustrated
in Fig. 1(c), a localized weight matrix $\mathbf{W}^{l}_{v_{i}}$ for each node
$v_{i}$ can be derived from both the global weight $\mathbf{W}^{l}$ and the
local context of $v_{i}$. That is, $\mathbf{W}^{l}_{v_{i}}$ is a function of
the global information and the local context. Globally, all nodes depend on a
common underlying model to encode the general patterns at the graph level.
Locally, each node leverages its local context to personalize the common model
into its unique localized weight at the node level. It is also useful to
incorporate a finer-grained localization at the edge level, where information
received through each edge of the target node can be further adjusted. The
proposed localization, which we call _node-wise localized GNN_ (LGNN), aims to
strike a balance between the global and local aspects of the graph. Moreover,
LGNN is agnostic of the underlying global model, meaning that it is able to
localize any GNN that follows the paradigm of recursive neighborhood
aggregation, and subsumes various conventional GNNs as its limiting cases.

In summary, our main contributions are three-fold. (1) We identify the need to
adapt to the local context of each node in GNNs. (2) We propose a node-wise
localization of GNNs to capture both global and local information on the
graph, and further discuss its connection to other works. (3) We conduct
extensive experiments on four benchmark datasets and show that LGNN
consistently outperforms prior art.

##  2 Proposed Approach

In this section, we introduce the proposed approach LGNN, and discuss its
connection to other works.

###  2.1 General Formulation of Localization

We start with an abstract definition of localizing a global model to fit local
contexts. Specifically, the localized model of an instance $v$ is a function
of both the global model and the local context. Let $\Theta$ denote the global
model, and $C_{v}$ the local context of $v$. Then the localized model for $v$
is

$\displaystyle\Theta_{v}=f(\Theta,C_{v}),$ (1)

where $f$ can be any function such as a neural network. While we focus on
graph data where $v$ is a node, the general formulation is also pertinent to
other kinds of data when there are varying, non-uniform contexts associated
with instances. In particular, on a graph $G=(V,E)$ with a set of nodes $V$
and edges $E$, the local context of a node $v$ can be materialized as the
neighbors of $v$, in addition to $v$ itself. That is,

$\displaystyle C_{v}=\\{v\\}\cup\\{u\in V:\langle v,u\rangle\in E\\}.$ (2)

###  2.2 Localization of GNNs

A typical GNN consists of multiple layers of recursive neighborhood
aggregation. In the $l$-th layer, each node $v$ receives and aggregates
information from its neighbors to derive its hidden representation
$\mathbf{h}_{v}^{l}\in\mathbb{R}^{d_{l}}$, with $d_{l}$ being the dimension of
the $l$-the layer representation. Note that the initial representation
$\mathbf{h}_{v}^{0}$ is simply the input feature vector of $v$. The
aggregation is performed on the local context of $v$ which also covers the
neighbors of $v$, as follows.

$\displaystyle\mathbf{h}^{l}_{v}=\sigma\left(\textsc{Aggr}\left(\left\\{\mathbf{W}^{l}\mathbf{h}^{l-1}_{u}:\forall
u\in C_{v}\right\\}\right)\right),$ (3)

where $\mathbf{W}^{l}\in\mathbb{R}^{d_{l}\times d_{l-1}}$ is a weight matrix
in the $l$-th layer, $\sigma(\cdot)$ is an activation function, and
Aggr$(\cdot)$ is an aggregation function. Different GNNs differ in the choice
of the aggregation function. For instance, GCN Kipf and Welling (2017) uses an
aggregator roughly equivalent to mean pooling Hamilton et al. (2017), GAT
Veličković et al. (2018) uses an attention-weighted mean aggregator, and GIN
Xu et al. (2019) uses a multi-layer perceptron (MLP).

#### Node-level localization.

In the above setup, we have a global model $\mathbf{W}^{l}$ for all nodes in
the $l$-th layer. At the node level, the global model is localized to adapt to
the local context of each node. Specifically, we modulate the global weight
$\mathbf{W}^{l}$ by a node-specific transformation through scaling and
shifting. The localized weight matrix of node $v$ is given by

$\displaystyle\mathbf{W}^{l}_{v}=\mathbf{W}^{l}\odot\left[\left(\mathbf{a}^{l}_{v}\right)_{\times
d_{l}}\right]^{\top}+\left[\left(\mathbf{b}^{l}_{v}\right)_{\times
d_{l}}\right]^{\top},$ (4)

where $\mathbf{a}^{l}_{v},\mathbf{b}^{l}_{v}\in\mathbb{R}^{d_{l-1}}$ are
$v$-specific vectors for scaling and shifting, respectively. Here the notation
$[(\mathbf{x})_{\times n}]$ represents a matrix of $n$ columns all of which
are identical to the vector $\mathbf{x}$, and $\odot$ denotes element-wise
multiplication. We essentially transform each row of the global matrix
$\mathbf{W}^{l}$ by $\mathbf{a}^{l}_{v}$ and $\mathbf{b}^{l}_{v}$ in an
element-wise manner, to generate the localized weight $\mathbf{W}^{l}_{v}$, so
that various importance levels are assigned to each feature dimension of the
node embedding $\mathbf{h}^{l-1}_{u}$.

It is important to recognize that the node-specific transformation should not
be directly learned, for two reasons. First, directly learning them
significantly increases the number of model parameters especially in the
presence of a large number of nodes, causing overfitting. Second, node-wise
localization is a function of both the global model and the local context as
formulated in Eq. (1), in order to better capture the local information
surrounding each node. Thus, for node $v$, we propose to generate the
$v$-specific transformation in the $l$-th layer from its local contextual
information $\mathbf{c}^{l}_{v}$. A straightforward recipe for
$\mathbf{c}^{l}_{v}$ is to pool the $(l-1)$-th layer representations of the
nodes in the local context $C_{v}$:

$\displaystyle\mathbf{c}^{l}_{v}=\textsc{Mean}\left(\left\\{\mathbf{h}^{l-1}_{u}:\forall
u\in C_{v}\right\\}\right),$ (5)

in which we adopt the mean pooling although other forms of pooling can be
substituted for it. Given the local contextual information
$\mathbf{c}^{l}_{v}\in\mathbb{R}^{d_{l-1}}$, we further utilize a dense layer,
shared by all nodes, to generate the $v$-specific transformation as follows.

$\displaystyle\mathbf{a}^{l}_{v}=\sigma\left(\mathbf{M}^{l}_{a}\mathbf{c}^{l}_{v}\right)+\mathbf{1},\quad\mathbf{b}^{l}_{v}=\sigma\left(\mathbf{M}^{l}_{b}\mathbf{c}^{l}_{v}\right),$
(6)

where $\mathbf{M}^{l}_{a},\mathbf{M}^{l}_{b}\in\mathbb{R}^{d_{l-1}\times
d_{l-1}}$ are learnable parameters shared by all nodes, and $\mathbf{1}$ is a
vector of ones to ensure that the scaling factors are centered around one.
Note that if the dimension $d_{l-1}$ is too large, such as that of the high-
dimensional raw features handled in the first layer of a GNN, we could employ
two dense layers and set the first one with a smaller number of neurons.

#### Edge-level localization.

The node-level localization of the global weight matrix is coarse-grained, as
the same localized weight is applied on all neighbors of the target node. That
is, the target node receives information through each of its edges uniformly.
To enable a finer-grained localization, we further modulate how information
propagates at the edge level.

Consider a node $v$ with edges $\\{\langle u,v\rangle:\forall u\in C_{v}\\}$.
In the $l$-th layer of the GNN, let $\mathbf{c}^{l}_{u,v}$ denote the local
contextual information of the edge $\langle u,v\rangle$, which is given by its
two ends. Specifically, we concatenate the $(l-1)$-th layer representations of
nodes $u$ and $v$, as follows.

$\displaystyle\mathbf{c}^{l}_{u,v}=\textsc{Concat}\left(\mathbf{h}^{l-1}_{v},\mathbf{h}^{l-1}_{u}\right).$
(7)

To implement the edge-level localization, we similarly adopt an edge-specific
transformation through scaling and shifting. Specifically, to derive the
$l$-th layer representation of $v$, we scale and shift the information from
each edge $\langle u,v\rangle$ during aggregation, by rewriting Eq. (3) as
follows.

$\displaystyle\mathbf{h}^{l}_{v}=\sigma\left(\textsc{Aggr}\left(\left\\{\mathbf{W}^{l}_{v}\mathbf{h}^{l-1}_{u}\odot\mathbf{a}^{l}_{u,v}+\mathbf{b}^{l}_{u,v}:\forall
u\in C_{v}\right\\}\right)\right),$ (8)

where $\mathbf{a}^{l}_{u,v},\mathbf{b}^{l}_{u,v}\in\mathbb{R}^{d_{l}}$ are
edge $\langle u,v\rangle$-specific vectors for scaling and shifting,
respectively. Like the node-specific transformation, we generate the edge-
specific transformation with a dense layer given by

$\displaystyle\mathbf{a}^{l}_{u,v}=\sigma\left(\mathbf{N}^{l}_{a}\mathbf{c}^{l}_{u,v}\right)+\mathbf{1},\quad\mathbf{b}^{l}_{u,v}=\sigma\left(\mathbf{N}^{l}_{b}\mathbf{c}^{l}_{u,v}\right),$
(9)

where $\mathbf{N}^{l}_{a},\mathbf{N}^{l}_{b}\in\mathbb{R}^{d_{l}\times
2d_{l-1}}$ are learnable parameters shared by all edges.

###  2.3 Semi-supervised Node Classification

In the benchmark task of semi-supervised node classification, each node
belongs to one of the pre-defined classes $\\{1,2,\ldots,K\\}$. However, we
only know the class labels of a subset of nodes $V_{Y}\subset V$, called the
labeled nodes. The goal is to predict the class labels of the remaining nodes.
Following standard practice Kipf and Welling (2017), for $K$-way
classification we set the dimension of the output layer to $K$, and apply a
softmax function to the representation of each node. That is, given a total of
$\ell$ layers,

$\displaystyle\textstyle\mathbf{z}_{v,k}=\textsc{Softmax}\left(\mathbf{h}_{v,k}^{\ell}\right)=\frac{\exp\left(\mathbf{h}_{v,k}^{\ell}\right)}{\sum_{k^{\prime}=1}^{K}\exp\left(\mathbf{h}_{v,k^{\prime}}^{\ell}\right)}.$
(10)

For a labeled node $v\in V_{Y}$, let $Y_{v,k}\in\\{0,1\\}$ be 1 if and only if
node $v$ belongs to class $k$. The overall loss is then formulated using a
cross-entropy loss with regularization as

$\displaystyle-$ $\displaystyle\textstyle\sum_{v\in
V_{Y}}\sum_{k=1}^{K}Y_{v,k}\ln\mathbf{z}_{v,k}+\lambda_{G}\|\Theta_{G}\|^{2}_{2}+\textstyle\lambda_{L}\|\Theta_{L}\|_{2}^{2}$
$\displaystyle+$
$\displaystyle\lambda\left({\|A-1\|_{2}^{2}}/{|A|}+{\|B\|_{2}^{2}}/{|B|}\right),$
(11)

where $\Theta_{G}=\\{\mathbf{W}^{l}:l\leq\ell\\}$ contains the parameters of
the global model,
$\Theta_{L}=\\{\mathbf{M}_{a}^{l},\mathbf{M}_{b}^{l},\mathbf{N}_{a}^{l},\mathbf{N}_{b}^{l}:l\leq\ell\\}$
contains the parameters of localization, and $A$, $B$ are sets respectively
containing the transformation vectors for scaling ($\mathbf{a}^{l}_{v}$’s and
$\mathbf{a}^{l}_{u,v}$’s) and shifting ($\mathbf{b}^{l}_{v}$’s and
$\mathbf{b}^{l}_{u,v}$’s). Note that the norms $\|\cdot\|^{2}_{2}$ are
computed over all tensor elements in the sets. While $A$ and $B$ are not
learnable, they are functions of the learnable $\Theta_{L}$. We explicitly
constrain them to favor small local changes, i.e., close-to-one scaling and
close-to-zero shifting. Moreover, as $A$ and $B$ grow with the size of the
graph, their norms are further normalized by the total number of tensor
elements in them, denoted by $|A|$ and $|B|$, respectively. $\lambda_{G}$,
$\lambda_{L}$ and $\lambda$ are non-negative hyperparameters.

###  2.4 Connections to Other Works

Next, we discuss how LGNN is related to other lines of work in GNNs, network
embedding and hypernetworks.

#### Relationship with existing GNNs.

The proposed LGNN is able to localize any GNN model that follows the scheme of
recursive neighborhood aggregation in Eq. (3). In particular, LGNN is a
generalized GNN that subsumes, as special limiting cases, several conventional
GNNs by adopting an appropriate aggregation function and regularization.

Specifically, as $\lambda\to\infty$ in Eq. (2.3), the scaling and shifting
would approach 1’s and 0’s in the limiting case, respectively. This is
equivalent to removing all node-wise localization at both node and edge
levels, where all nodes assume a global model. Thus, LGNN would be
asymptotically equivalent to GCN Kipf and Welling (2017) and GIN Xu et al.
(2019), when using a graph convolution aggregator (roughly equivalent to the
mean pooling) and an MLP aggregator on the sum-pooled representations,
respectively. Furthermore, with an appropriate regularization, LGNN can be
asymptotically reduced to GAT111Here we only discuss GAT with one attention
head. Veličković et al. (2018). Consider the vectors for scaling in $A$, which
can be split into the node-specific $A_{V}$ (containing
$\mathbf{a}^{l}_{v}$’s) and edge-specific $A_{E}$ (containing
$\mathbf{a}^{l}_{u,v}$’s). For $A_{V}$, we maintain the same regularization
$\|A_{V}-1\|^{2}_{2}$; for $A_{E}$, we adopt the regularization
$\|A_{E}-\alpha\|^{2}_{2}$ such that $\alpha$ contains the corresponding
attention coefficients on all edges. That is,

$\displaystyle\mathbf{a}^{l}_{u,v}\to\left[\left(\alpha^{l}_{u,v}\right)_{\times
d_{l}}\right]^{\top},$ (12)

where $\alpha^{l}_{u,v}\in\mathbb{R}$ is the attention coefficient on edge
$\langle u,v\rangle$. Coupled with a mean aggregator, we obtain GAT as the
limiting case when $\lambda\to\infty$. Alternatively, we can also maintain the
same regularization for both node- and edge-specific transformations, and
adopt a weighted mean aggregator in accordance to the attention coefficients.
Note that in GAT the attention coefficient on each edge is a scalar, whereas
in our edge-level localization, the edge-specific scaling is represented by a
vector, which is able to flexibly vary the contribution from different
dimensions of the information received through the edges.

#### Relationship with network embedding.

Network embedding Cai et al. (2018) is a popular alternative to GNNs to learn
node representations on a graph. In these methods, each node is directly
encoded with a learnable parameter vector, which is taken as the
representation of the node. In contrast, in GNNs, the representations of the
nodes are derived from a shared learnable model. Thus, network embedding can
be viewed as a set of individual local models that are loosely coupled by
local graph structures, whereas GNNs capture a more global view of the
structures. In contrast, the proposed LGNN achieves a balance between the
local and global views, to allow localized variations grounded on a global
model.

#### Relationship with hypernetworks.

Our localization strategy can be deemed a form of hypernetworks Ha et al.
(2017), which use a secondary neural network to generate weights for the
target network. In our case, we employ dense layers as a secondary network to
generate the vectors for scaling and shifting, which are leveraged to
ultimately generate the localized weights for the target GNN. Moreover, our
approach boils down to the feature-wise modulation of information received
from the conditioning local context (i.e., neighboring nodes) at both node and
edge levels, which is inspired by the feature-wise modulation of neural
activations in FiLMs Perez et al. (2018); Birnbaum et al. (2019).

On graphs, GNN-FiLM Brockschmidt (2020) also uses a form of FiLM on GNNs.
However, there are several fundamental differences between GNN-FiLM and our
LGNN. First, in terms of _problem and motivation_ , GNN-FiLM aims to model
edge labels on relational networks for message passing. In contrast, our LGNN
aims to model local contexts for node-wise localization, and works on general
graphs. Second, in terms of _model_ , GNN-FiLM models relation-specific
transformations conditioned only on a node’s self-information, which does not
sufficiently reflect the full local context of the node as LGNN is conditioned
on. Furthermore, when there is no edge labels, GNN-FiLM reduces to a “uniform
edge” model. In LGNN, we still have both node and edge level modulation to
achieve localization. Third, in terms of _empirical performance_ , as
discussed therein Brockschmidt (2020), GNN-FiLM only achieves at best
comparable performance to existing GNNs on citation networks (such as _Cora_
and _Citeseer_) where there is no edge label. Our own experiments also have
reproduced similar results in the next section.

##  3 Experiments

In this section, we evaluate and analyze the empirical performance of our
proposed approach LGNN.

###  3.1 Experimental Setup222Additional implementation details and
experimental settings are included in Sections A and B of the supplemental
material.

#### Datasets.

We utilize four benchmark datasets. They include two academic citation
networks, namely _Cora_ and _Citeseer_ Yang et al. (2016), in which nodes
correspond to papers and edges correspond to citations between papers. The
input node features are bag-of-word vectors, indicating the presence of each
keyword. A similar citation network for Wikipedia articles called _Chameleon_
Rozemberczki et al. (2021) is also used. Finally, we use an e-commerce co-
purchasing network called _Amazon_ Hou et al. (2020), in which the nodes
correspond to computer products and the edges correspond to co-purchasing
relationships between products. The input node feature vectors are constructed
from product images. The statistics of the datasets are summarized in Table 1.

Dataset | # Nodes | # Edges | # Classes | # Features
---|---|---|---|---
Cora | 2,708 | 5,429 | 7 | 1,433
Citeseer | 3,327 | 4,732 | 6 | 3,703
Amazon | 13,381 | 245,778 | 10 | 767
Chameleon | 2,277 | 36,101 | 5 | 2,325
Table 1: Summary of datasets. Methods | # Params | Cora | Citeseer | Amazon | Chameleon
---|---|---|---|---|---
(Cora) | Accuracy | Micro-F | Accuracy | Micro-F | Accuracy | Micro-F | Accuracy | Micro-F
DeepWalk | 693K |  73.8$\pm$0.3 |  74.9$\pm$0.1 |  61.6$\pm$0.2 |  60.5$\pm$1.0 |  80.1$\pm$1.6 |  77.3$\pm$1.3 |  41.2$\pm$1.3 |  40.1$\pm$1.1
Planetoid | 345K |  66.1$\pm$0.4 |  64.5$\pm$0.5 |  64.5$\pm$0.3 |  62.9$\pm$0.4 |  69.8$\pm$1.7 |  64.5$\pm$1.5 |  39.3$\pm$1.8 |  37.7$\pm$1.7
GCN | 11K |  81.5$\pm$0.7 |  80.8$\pm$0.5 |  70.4$\pm$0.5 |  68.3$\pm$0.7 |  81.9$\pm$0.5 |  81.0$\pm$0.8 |  46.7$\pm$4.3 |  46.4$\pm$2.4
GCN-64 | 92K |  82.0$\pm$0.3 |  80.9$\pm$0.3 |  71.1$\pm$0.3 |  69.2$\pm$0.4 |  82.1$\pm$0.5 |  81.2$\pm$0.8 |  48.3$\pm$3.3 |  46.3$\pm$1.8
GCN-96 | 138K |  81.9$\pm$0.2 |  80.8$\pm$0.3 |  71.3$\pm$0.4 |  69.4$\pm$0.5 |  82.2$\pm$0.4 |  81.5$\pm$0.7 |  45.5$\pm$2.4 |  43.8$\pm$2.5
GCN-FiLM | 35K |  78.1$\pm$0.6 |  76.9$\pm$0.5 |  69.8$\pm$1.1 |  67.9$\pm$1.0 |  79.2$\pm$1.0 |  77.1$\pm$1.5 |  42.8$\pm$1.1 |  39.9$\pm$1.3
LGCN | 104K |  83.5$\pm$0.3 |  82.1$\pm$0.4 |  72.2$\pm$0.4 |  70.2$\pm$0.4 |  83.7$\pm$1.5 |  82.3$\pm$2.0 |  50.9$\pm$1.1 |  49.7$\pm$0.7
(improv.) | - | (1.8%) | (1.5%) | (1.3%) | (1.2%) | (1.8%) | (1.0%) | (5.4%) | (7.1%)
GAT | 92K |  82.9$\pm$0.6 |  82.0$\pm$0.6 |  72.4$\pm$0.7 |  70.4$\pm$0.8 |  82.4$\pm$1.3 |  80.1$\pm$1.9 |  47.2$\pm$1.1 |  46.2$\pm$2.1
GAT-64 | 738K |  83.1$\pm$0.4 |  81.9$\pm$0.6 |  71.6$\pm$1.5 |  69.8$\pm$1.6 |  83.0$\pm$0.9 |  81.2$\pm$1.4 |  51.2$\pm$1.5 |  50.2$\pm$1.3
GAT-96 | 1108K |  83.2$\pm$0.6 |  81.9$\pm$0.6 |  71.4$\pm$0.9 |  69.6$\pm$0.9 |  83.1$\pm$1.0 |  81.5$\pm$1.4 |  51.9$\pm$1.2 |  50.2$\pm$1.8
GAT-FiLM | 277K |  82.0$\pm$0.5 |  80.6$\pm$0.6 |  71.2$\pm$1.0 |  69.2$\pm$1.1 |  83.3$\pm$0.6 |  81.9$\pm$0.8 |  46.8$\pm$5.7 |  45.1$\pm$5.2
LGAT | 836K |  83.6$\pm$0.4 |  82.3$\pm$0.4 |  72.8$\pm$0.4 |  70.8$\pm$0.5 |  83.7$\pm$0.7 |  82.3$\pm$0.8 |  52.6$\pm$1.0 |  51.1$\pm$0.9
(improv.) | - | (0.5%) | (0.4%) | (0.6%) | (0.6%) | (0.5%) | (0.5%) | (1.3%) | (1.8%)
GIN | 11K |  80.2$\pm$0.5 |  78.8$\pm$0.3 |  68.5$\pm$0.7 |  66.5$\pm$1.0 |  79.6$\pm$1.7 |  78.5$\pm$2.6 |  45.8$\pm$3.0 |  41.2$\pm$4.0
GIN-64 | 92K |  80.3$\pm$1.1 |  79.1$\pm$1.0 |  67.8$\pm$1.5 |  66.1$\pm$1.1 |  79.8$\pm$1.1 |  79.0$\pm$1.4 |  45.7$\pm$4.5 |  40.7$\pm$5.7
GIN-96 | 138K |  79.9$\pm$1.1 |  78.9$\pm$1.0 |  68.6$\pm$1.4 |  66.6$\pm$1.6 |  80.2$\pm$2.1 |  79.0$\pm$3.2 |  45.9$\pm$3.5 |  41.5$\pm$4.1
GIN-FiLM | 35K |  79.8$\pm$0.7 |  78.5$\pm$0.5 |  67.7$\pm$1.4 |  65.8$\pm$1.5 |  78.6$\pm$2.8 |  77.2$\pm$3.3 |  38.8$\pm$2.6 |  34.2$\pm$2.9
LGIN | 126K |  82.6$\pm$0.8 |  81.6$\pm$0.8 |  71.3$\pm$0.4 |  69.5$\pm$0.5 |  84.0$\pm$1.2 |  82.7$\pm$1.7 |  48.3$\pm$1.9 |  47.3$\pm$1.9
(improv.) | - | (2.9%) | (3.2%) | (3.9%) | (4.4%) | (4.7%) | (4.7%) | (5.2%) | (14.0%)
Table 2: Average classification performance with standard deviation (percent)
over 10 runs. Improvements of LGNN are relative to the best performing
baseline with the corresponding GNN architecture.

#### Baselines and our approach.

We compare against three categories of baselines. (1) Embedding models:
_DeepWalk_ Perozzi et al. (2014) and _Planetoid_ Yang et al. (2016). Both
employ a direct embedding lookup and adopt random walks to sample node pairs.
However, DeepWalk is unsupervised such that node classification is performed
as a downstream task on the learned representations, whereas Planetoid is an
end-to-end model that jointly learns the representations and the classifier.
(2) GNN models: _GCN_ Kipf and Welling (2017), _GAT_ Veličković et al. (2018)
and _GIN_ Xu et al. (2019). (3) GNN-FiLM: The original GNN-FiLM works with a
GCN-style model, which we call _GCN-FiLM_. We further extend it to two other
GNN architectures GAT and GIN, resulting in two more versions _GAT-FiLM_ and
_GIN-FiLM_.

On the other hand, our approach LGNN can also be implemented on different GNN
architectures. Specifically, we employ GCN, GAT and GIN as the global model,
and obtain localized versions _LGCN_ , _LGAT_ and _LGIN_ , respectively.

#### Main settings.

For DeepWalk, we sample 10 random walks per node with walk length 100 and
windows size 5, and set the embedding dimension to 128. We further train a
logistic regression as the downstream classifier. For Planetoid, we set the
path length to 10, window size to 3 and the embedding dimension to 50. We
select the best results from their transductive and inductive versions for
report. For all GNN-based approaches, we adopt two aggregation layers, noting
that deeper layers often bring in noises from distant nodes Pei et al. (2020)
and cause “over-smoothing” such that all nodes obtain very similar
representations Li et al. (2018); Xu et al. (2018). The dimension of the
hidden layer defaults to 8, while we also present results using larger
dimensions. The regularization of GNN parameters is set to
$\lambda_{G}=0.0005$. These settings are chosen via empirical validation, and
are largely consistent with the literature Perozzi et al. (2014); Kipf and
Welling (2017); Veličković et al. (2018). For our models, the additional
regularizations are set to $\lambda_{L}=\lambda=1$ (except for $\lambda=0.1$
in LGAT).

#### Training and testing.

For all datasets, we follow the standard split in the literature Yang et al.
(2016); Kipf and Welling (2017); Veličković et al. (2018), which uses 20
labeled nodes _per class_ for training, 500 nodes for validation and 1000
nodes for testing. We repeat 10 runs for each experiment, and report the
average accuracy and micro-F score.

Figure 2: Ablation study on the effect of each localization module.

###  3.2 Performance Comparison

Table 2 shows the comparison of the classification performance on the four
datasets. For each GNN architecture, we also include additional models by
increasing the dimension of its hidden layer. For instance, GCN-64 denotes
that the hidden layer of GCN has 64 neurons. The results reveal a few
important observations.

_Firstly_ , we observe that LGNN consistently achieves significant performance
boosts w.r.t. three state-of-the-art GNNs on four datasets. This suggests
that, our localization approach is agnostic of the particular GNN architecture
of the global model, and can be used universally on different GNNs.
Furthermore, LGNNs also outperform GNN-FiLMs, as GNN-FiLMs only achieve at
best comparable performance with the base GNNs. A potential reason is that
GNN-FiLM is only conditioned on the target node’s self-information to modulate
message passing for different edge labels, which may not be useful on graphs
without edge labels. In contrast, LGNN is conditioned on the full local
context of the target node to achieve node-wise localization that could be
useful to the classification of nodes residing across different localities on
the graph. _Secondly_ , GAT-based models generally attain better performance
than GCN- and GIN-based models, and the performance gain in LGAT w.r.t. GAT is
smaller. This is not surprising as the attention coefficients on the edges can
be understood as a limited form of localization to differentiate the weight of
different neighbors. _Thirdly_ , the results imply that increasing the number
of parameters alone cannot achieve the effect of localization. It may be
argued that more parameters lead to higher model capacity, which can
potentially assign some space to encode the local aspects. However, when the
parameters are still shared globally, there is no explicit node-wise
constraint and thus the model will still try to find a middle ground.
Nevertheless, for a fair comparison, we use more parameters on each GNN
baseline to match or exceed the number of parameters on the corresponding
LGNN, as listed in Table 2444Details of the calculation are included in
Section C of the supplemental material.. As expected, more parameters only
result in marginal improvements to the baselines. Thus, the efficacy of LGNN
is derived from our localization strategy rather than just more parameters.
_Fourthly_ , LGNN is robust and stable given their relatively small standard
deviations in many cases. We hypothesize that, in conventional GNNs, a bad
initialization would risk a suboptimal global model shared by all nodes
without recourse. In contrast, node-wise localization offers an opportunity to
adjust the suboptimal model for some nodes, and is thus less susceptible to
the initialization.

It has also been discussed that in GNNs overfitting to the validation set is
an issue Shchur et al. (2018), and thus models with more parameters tend to
perform better given a larger validation set. Therefore, we further compare
different methods using a smaller validation set of only 100 nodes555These
results are in Section D of the supplemental material.. Our proposed LGNN
still outperforms all the baselines across the four datasets, demonstrating
its power and stability.

###  3.3 Further Model Analysis

Due to space constraint, we only present an ablation study here666More
analyses on the complexity and effect of regularization, as well as model
visualization, can be found in Sections E, F and G of the supplemental
material.. In the ablation study, we investigate the effectiveness of the
node- and edge-level localization modules in LGNN. To validate the
contribution of each module, we compare the accuracy of four variants in Fig.
2: (1) global only without any localization; (2) node-level localization only;
(3) edge-level localization only; and (4) the full model with both
localization modules. We observe that utilizing only one module, whether at
the node or edge level, consistently outperforms the global model. Between the
two modules, the node-level localization tends to perform better.
Nevertheless, modeling both jointly results in the best performance, which
implies that both modules are effective and possess complementary localization
power.

##  4 Related Work

Graph representation learning has received significant attention in recent
years. Earlier network embedding approaches Perozzi et al. (2014); Tang et al.
(2015); Grover and Leskovec (2016) employ a direct embedding lookup for node
representations, which are often coupled via local structures such as skip-
grams. To better capture global graph structures, GNNs Kipf and Welling
(2017); Hamilton et al. (2017); Veličković et al. (2018); Xu et al. (2019)
open up promising opportunities. They generally follow a paradigm of recursive
neighborhood aggregation, in which each node receives information from its
neighbors on the graph. More complex structural information is also exploited
by recent variants Zhang et al. (2020); Pei et al. (2020); Chen et al. (2020),
but none of them deals with the node-wise localization.

While our approach can be deemed a form of hypernetworks Ha et al. (2017);
Perez et al. (2018) as discussed in Sect. 2.4, several different forms of
local models have been explored in related problems. In manifold learning Yu
et al. (2009); Ladicky and Torr (2011), local codings are used to approximate
any point on the manifold as a linear combination of its surrounding anchor
points. In low-rank matrix approximation Lee et al. (2013), multiple low-rank
approximations are constructed for different regions of the matrix before
aggregation. These methods are tightly coupled with their problems, and cannot
be easily extended to localize GNNs. Some meta-learning frameworks Vinyals et
al. (2016); Snell et al. (2017); Finn et al. (2017) can also be viewed as
adapting to local contexts. Given a set of training tasks, meta-learning aims
to learn a prior that can be adapted to new unseen tasks, often for addressing
few-shot learning problems. In particular, existing meta-learning models on
graphs are mostly designed for few-shot node classification Zhou et al.
(2019); Yao et al. (2020); Liu et al. (2021) or regression Liu et al. (2020).
Such few-shot settings are fundamentally different from our localization
objective, in which all nodes are seen during training and we leverage the
local contexts of the nodes in order to enhance their representations.

##  5 Conclusions

In this work, we identified the need to localize GNNs for different nodes that
reside in non-uniform local contexts across the graph. This motivated us to
propose a node-wise localization approach, named LGNN, in order to adapt to
the locality of each node. On one hand, we encode graph-level general patterns
using a global weight matrix. On the other hand, we modulate the global model
to generate localized weights specific to each node, and further perform an
edge-level modulation to enable finer-grained localization. Thus, the proposed
LGNN can capture both local and global aspects of the graph well. Finally, our
extensive experiments demonstrate that LGNN significantly outperforms state-
of-the-art GNNs.

## Acknowledgments

This research is supported by the Agency for Science, Technology and Research
(A*STAR) under its AME Programmatic Funds (Grant No. A20H6b0151).

## References

  * Birnbaum et al. (2019) Sawyer Birnbaum, Volodymyr Kuleshov, Zayd Enam, Pang Wei W Koh, and Stefano Ermon.  Temporal FiLM: Capturing long-range sequence dependencies with feature-wise modulations.  In NeurIPS, pages 10287–10298, 2019.
  * Brockschmidt (2020) Marc Brockschmidt.  GNN-FiLM: Graph neural networks with feature-wise linear modulation.  In ICML, pages 1144–1152, 2020.
  * Cai et al. (2018) Hongyun Cai, Vincent W Zheng, and Kevin Chang.  A comprehensive survey of graph embedding: problems, techniques and applications.  TKDE, 30(9):1616–1637, 2018.
  * Chen et al. (2020) Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li.  Simple and deep graph convolutional networks.  In ICML, pages 1725–1735, 2020.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In ICML, pages 1126–1135, 2017.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec.  node2vec: Scalable feature learning for networks.  In KDD, pages 855–864, 2016.
  * Ha et al. (2017) David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  In ICLR, 2017.
  * Hamilton et al. (2017) Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In NeurIPS, pages 1024–1034, 2017.
  * Hou et al. (2020) Yifan Hou, Jian Zhang, James Cheng, Kaili Ma, Richard TB Ma, Hongzhi Chen, and M Yang.  Measuring and improving the use of graph information in graph neural networks.  In ICLR, 2020.
  * Kipf and Welling (2017) Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In ICLR, 2017.
  * Ladicky and Torr (2011) Lubor Ladicky and Philip HS Torr.  Locally linear support vector machines.  In ICML, pages 985–992, 2011.
  * Lee et al. (2013) Joonseok Lee, Seungyeon Kim, Guy Lebanon, and Yoram Singer.  Local low-rank matrix approximation.  In ICML, pages 82–90, 2013.
  * Li et al. (2018) Qimai Li, Zhichao Han, and Xiao-Ming Wu.  Deeper insights into graph convolutional networks for semi-supervised learning.  In AAAI, pages 3538–3545, 2018.
  * Liu et al. (2020) Zemin Liu, Wentao Zhang, Yuan Fang, Xinming Zhang, and Steven C.H. Hoi.  Towards locality-aware meta-learning of tail node embeddings on networks.  In CIKM, pages 975–984, 2020.
  * Liu et al. (2021) Zemin Liu, Yuan Fang, Chenghao Liu, and Steven C.H. Hoi.  Relative and absolute location embedding for few-shot node classification on graph.  In AAAI, 2021.
  * Pei et al. (2020) Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang.  Geom-GCN: Geometric graph convolutional networks.  In ICLR, 2020.
  * Perez et al. (2018) Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  FiLM: Visual reasoning with a general conditioning layer.  In AAAI, pages 3942–3951, 2018.
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena.  DeepWalk: Online learning of social representations.  In KDD, pages 701–710, 2014.
  * Rozemberczki et al. (2021) Benedek Rozemberczki, Carl Allen, and Rik Sarkar.  Multi-Scale Attributed Node Embedding.  Journal of Complex Networks, 9(2), 2021.
  * Shchur et al. (2018) Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann.  Pitfalls of graph neural network evaluation.  arXiv preprint arXiv:1811.05868, 2018.
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In NeurIPS, pages 4077–4087, 2017.
  * Tang et al. (2015) Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei.  LINE: Large-scale information network embedding.  In TheWebConf, pages 1067–1077, 2015.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  In ICLR, 2018.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  In NeurIPS, pages 3630–3638, 2016.
  * Wu et al. (2020) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  TNNLS, 32(1):4–24, 2020.
  * Xu et al. (2018) Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka.  Representation learning on graphs with jumping knowledge networks.  In ICML, pages 5453–5462, 2018.
  * Xu et al. (2019) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  In ICLR, 2019.
  * Yang et al. (2016) Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov.  Revisiting semi-supervised learning with graph embeddings.  In ICML, pages 40–48, 2016.
  * Yao et al. (2020) Huaxiu Yao, Chuxu Zhang, Ying Wei, Meng Jiang, Suhang Wang, Junzhou Huang, Nitesh V Chawla, and Zhenhui Li.  Graph few-shot learning via knowledge transfer.  In AAAI, pages 6656–6663, 2020.
  * Yu et al. (2009) Kai Yu, Tong Zhang, and Yihong Gong.  Nonlinear learning using local coordinate coding.  In NeurIPS, pages 2223–2231, 2009.
  * Zhang et al. (2020) Kai Zhang, Yaokang Zhu, Jun Wang, and Jie Zhang.  Adaptive structural fingerprints for graph attention networks.  In ICLR, 2020.
  * Zhou et al. (2019) Fan Zhou, Chengtai Cao, Kunpeng Zhang, Goce Trajcevski, Ting Zhong, and Ji Geng.  Meta-GNN: On few-shot node classification in graph meta-learning.  In CIKM, pages 137–144, 2019.


# Leveraging Hybrid Intelligence Towards Sustainable and Energy-Efficient
Machine Learning

Daniel Geißler<EMAIL_ADDRESS>0000-0003-2643-4504
DFKIKaiserslauternGermany and  Paul Lukowicz<EMAIL_ADDRESS>0000-0003-0320-6656 DFKI University of Kaiserslautern-
LandauKaiserslauternGermany

(2024)

###### Abstract.

Hybrid intelligence aims to enhance decision-making, problem-solving, and
overall system performance by combining the strengths of both, human cognitive
abilities and artificial intelligence. With the rise of Large Language Models
(LLM), progressively participating as smart agents to accelerate machine
learning development, Hybrid Intelligence is becoming an increasingly
important topic for effective interaction between humans and machines. This
paper presents an approach to leverage Hybrid Intelligence towards sustainable
and energy-aware machine learning. When developing machine learning models,
final model performance commonly rules the optimization process while the
efficiency of the process itself is often neglected. Moreover, in recent
times, energy efficiency has become equally crucial due to the significant
environmental impact of complex and large-scale computational processes. The
contribution of this work covers the interactive inclusion of secondary
knowledge sources through Human-in-the-loop (HITL) and LLM agents to stress
out and further resolve inefficiencies in the machine learning development
process.

Hybrid Intelligence, Interactivity, Sustainability, Energy Awareness

††journalyear: 2024††copyright: rightsretained††conference: Companion of the
2024 ACM International Joint Conference on Pervasive and Ubiquitous Computing
Pervasive and Ubiquitous Computing; October 5–9, 2024; Melbourne, VIC,
Australia††booktitle: Companion of the 2024 ACM International Joint Conference
on Pervasive and Ubiquitous Computing Pervasive and Ubiquitous Computing
(UbiComp Companion ’24), October 5–9, 2024, Melbourne, VIC, Australia††doi:
XXXXXXX.XXXXXXX††isbn: ††ccs: Human-centered computing Visualization systems
and tools††ccs: Human-centered computing Interactive systems and tools††ccs:
Human-centered computing Systems and tools for interaction design

##  1\. Problem Statement

The field of sustainable machine learning has witnessed increasing relevance
in recent years, driven by the fast-paced expansion of machine learning
applications in pervasive systems (Yao et al., 2023). Even though the
efficiency, considering hardware from low-power embedded devices to high-
performance clusters, has improved over time, it cannot keep up with the
rapidly increasing amount of devices that are utilized in our daily lives
(Desislavov, 2023). When it comes to energy-efficient optimization, research
commonly focuses on optimizing for time reduction instead of energy (Mehlin et
al., 2023). However, shrinking time commonly requires even more computational
resources to counteract.

From the data-level perspective, the quality of training data significantly
impacts the final performance of machine learning models (Soni et al., 2023).
Small nuances in data quality, such as issues arising from sensor quality,
placement, and annotation, can drastically affect the overall model
performance, especially in the fields of Human Activity Recognition HAR (Zhang
et al., 2022). These imperfections lead to inefficient utilization of
resources, as weak data quality necessitates additional training cycles and
resource expenditure to achieve acceptable performance levels. Moreover,
common approaches to address these inefficiencies, like enlarging the model
architectural complexity to improve generalization, often result in an even
greater waste of resources due to unawareness of the causes (Li et al., 2020).

Another significant issue is the lack of a clear framework for measuring
energy consumption in machine learning training (Jay et al., 2023). Many
existing approaches only track parts of the system, such as GPU usage, without
considering the overall energy footprint of the entire infrastructure
(Henderson et al., 2020). Work in this direction needs to cover the custom
hardware landscape to offer more precise measurements and obtain guidelines on
how to efficiently utilize resources.

To improve the sustainability of machine learning practices, it is essential
to not only determine what aspects of the training process are inefficient but
also to understand why they are inefficient (Alicioglu and Sun, 2022). This
requires a holistic examination of the hardware, data quality, and model
architecture in conjunction to cover the potential abstraction levels for
optimization. Hybrid Intelligence offers means for the integration of
additional knowledge through visual and numeric analysis to incorporate
computational resource savings within the model optimization process (Akata et
al., 2020).

##  2\. Related Work

Several studies focus on the either enhancing energy awareness or improving
the interaction and visualization of machine learning development as presented
in the following sections. To the best of our knowledge, there is currently no
work combining these two areas into one coherent topic.

###  2.1. Energy Tracking

In the AI sector, energy consumption tools are still a niche, especially when
it comes to generating awareness about the energy expenditure for training
machine learning models (Möbius et al., 2013). They are usually designed to
capture energy information by building an additional layer between the
system’s hardware configuration and the user’s model training process
(Henderson et al., 2020). The power consumption of the GPU is considered the
largest part of the training process as it performs the core work with the
parallel processing of mathematical tasks (Mittal and Vetter, 2014).
Nevertheless, other hardware components and even secondary power consumers
like the cooling system or the power supply unit itself contribute to the
overall energy consumption.

Works like Carbontracker (Anthony et al., 2020), eco2Ai (Budennyy et al.,
2022) and Green Algorithms (Lannelongue et al., 2021) utilize this approach to
gather data from the hardware. To handle missing elements in the calculations,
software like Carbontracker multiplies its results with an efficiency constant
to incorporate untracked secondary power needs and efficiency losses. With an
extended focus on user experience, projects like Cloud Carbon (cloud-carbon
footprint, 2023) or CodeCarbon (Code-Carbon, 2023) extend the gathered
knowledge and present it in analytic-based dashboards. Based on the calculated
energy consumption and the user’s location, the average local energy mix from
fossil and renewable energy sources is utilized to quantify the carbon
emissions (Lacoste et al., 2019).

###  2.2. Interactive Machine Learning Visualizations

Interactive visualizations can open up the black-box nature of machine
learning models, making them more interpretable by allowing users to
interrogate, explain, and even validate the process (Hurley et al., 2021).
Various tools are available, essentially aiming to enhance trust in machine
learning models through increasing human perception of the model’s internals
(Chatzimparmpas et al., 2020). TensorFlow Graph Visualizer for instance helps
users understand complex machine learning architectures by visualizing their
underlying dataflow graphs, making it easier to debug and improve model
structures (Wongsuphasawat et al., 2018). However, it is still a challenge to
select the right visualization, especially when lossy dimension reduction
strategies are applied and thus information content is missing (Rauber et al.,
2017).

Further, the interactivity of visualizations can elevate the insights and feed
the user’s input back into the model for adaptation (Fuchs et al., 2009).
Works like (Sacha et al., 2017) have proven the advantages of Human-centered
and HITL systems to enhance machine learning development through human
knowledge integration and feedback. Similar to the idea of this work, (Wei et
al., 2022) proposes a human-in-the-loop approach to enhance deep neural
network classification accuracy by leveraging human knowledge. By projecting
high-dimensional latent spaces onto a two-dimensional workspace, users can
interactively modify coordinates, which are then fed back to fine-tune the
network, improving classification results.

##  3\. Methodology

Figure 1. The concept of embedding Hybrid Intelligence into the traditional
Machine Learning training to enhance energy efficiency from a data-driven
perspective.

The proposed concept leverages hybrid intelligence to elevate sustainable
machine learning development. By incorporating HITL systems and LLMs as smart
agent assistance, we aim to enhance resource efficiency and significantly
reduce energy consumption. This approach ensures that machine learning models
achieve performance goals without compromising on environmental
considerations. The integration of Hybrid Intelligence support into machine
learning workflows represents a promising solution to existing challenges,
leading to more effective and environmentally responsible practices. The
following sections introduce the ongoing research steps to connect the
relevant areas into a coherent framework.

###  3.1. Hybrid Intelligence

(a) Scatter plot to rejoin outliers with their desired clusters (black arrow)
or increase the distance between whole clusters (red arrow).

(b) Parallel Coordinates Plot to investigate the relevance of each dimension.

Figure 2. Potential use cases of latent space visualizations for human-in-the-
loop interaction and optimization of the machine learning training.

Within this work, Hybrid Intelligence forms the core to leverage extended
knowledge. The general idea is to combine human expertise with artificial
intelligence to integrate such knowledge directly into the system. The
training of machine learning models usually does not provide any deep
insights, acting mostly as a black-box system. Even though statistical
features can be extracted from the training, such as loss or accuracy, there
is still a lack of visualizations on how the training data itself is
propagated through the model.

As shown in Figure 2, such visualizations can be implemented through two
general latent space exploration strategies of the model layer outputs.
Firstly, this can involve the use of high-dimensional visualization techniques
such as radar plots or parallel coordinates plots as shown in Figure 2(b). The
advantage of such visualization correlates with the intention of this work to
inspect the latent space on the data level. Due to the plotting of all
available dimensions, the user can easily distinguish the importance of each
dimension and how the data spreads within one dimension. As an interactive
step, different classes can be dragged apart by moving them on the dimensions
axis. This information may then be fed back into the model to help convergence
of the training.

Secondly, dimension reduction methods (Reddy et al., 2020) like Principle
Components Analysis can provide useful insights in the form of scatter plots.
However, reducing dimensions from latent spaces leads to a loss of information
which may lead to insufficient interpretation. Additionally, dimension
reduction techniques like t-SNE or UMAP are unable to invert the
transformations, even though they are particularly effective in capturing the
structure of high-dimensional data for visualization purposes. Therefore, the
possibilities of those techniques are limited to visual analysis. As shown in
Figure 2(a), the scatter plots can still be utilized to organize outliers
through drag and drop commands or even move close clusters away from each
other. The distance information can be incorporated through the HITL system
for training optimization.

The data-oriented insight offers isolated analysis of data points to
investigate if the data or the model itself leads to decreased performance.
Especially for HAR, correlations between sensor data and classes can be
derived to enhance the understanding of training procedures.

Connecting within the training loop, such a HITL system enables experts to
intervene during the training by visualizing the data and highlighting
inefficiencies. Based on that, advanced adjustment strategies can be
implemented through modification of the latent space visualization. Classic
user interface inputs such as marking, selecting, or dragging data points can
be utilized to set focal points in the model training to generate a dynamic
training concept. In the sense of this work, such dynamic adjustments can
leverage human expertise to especially fine-tune and optimize the training
towards efficient resource utilization but are not limited to this use case.

LLM agents such as the popular Chat-GPT or Llama can further support this
process by providing smart suggestions during training, assisting the human in
the interactive process. Especially when utilizing multi-modal LLMs, such as
VisionLLaMA (Chu et al., 2024), the agent may have an enhanced understanding
of the problem since it is able to investigate the problem visually and
numerically, especially if visualizations exceed human interpretability and
complexity. The agent can therefore either assist the HITL system or even
operate on its own depending on the user’s choice.

###  3.2. Energy Awareness

Effective energy tracking is crucial for sustainable machine learning to
quantify the expenses of computational resources. As discussed, tools like
Carbontracker (Anthony et al., 2020) or eco2Ai (Budennyy et al., 2022) can be
utilized to monitor the energy consumption of models during training. However,
current issues include the lack of comprehensive energy tracking, as some
tools only monitor specific components like the GPU. Especially secondary
energy consumption, for instance from cooling, is commonly neglected in the
tracking even though it contributes to the overall energy investment. To
address this, we propose integrating energy awareness directly into the
training loop. This involves using sensors such as hall sensors to confirm
deviations in energy usage and provide accurate measurements of the full
system by gathering the energy consumption directly from the power cord. By
incorporating these energy metrics into the hyperparameter optimization
process, we ensure that models are trained with energy efficiency in mind.
Furthermore, utilizing it in combination with Hybrid Intelligence provides
explainability and insights into the energy consumption patterns of different
model and training components.

###  3.3. Framework Integration

The final outcome of this work targets to integrate the Hybrid Intelligence
and the energy awareness approaches into a robust framework that connects all
components seamlessly. A user-friendly frontend tool is essential for
interacting and monitoring the training process, helping to visualize the
training process, track resource usage, and allow for user interaction.

The aforementioned visualizations should be concise and interpretable to allow
proper HITL interaction. Therefore, a generalized and flexible architecture
will adapt to the machine learning problem, the model architecture, and the
user’s goals. The smart agent can be added as an assistant next to the HITL in
order to detect ambiguities and provide suggestions for sustainable
adaptations. Multiple scenarios are possible to either give the human or the
smart agent more room for interaction depending on the agent’s capabilities.

The energy-awareness approach will not only be added through simple monitoring
into the framework, moreover, the energy information can be utilized to
optimize the training process by integrating the energy information through
penalization of inefficient training states. Further, resources can be
artificially limited or extended during training to force the training process
to run more efficiently. For instance, Hybrid Intelligence can isolate the
bad-performing parts of the training data and set priorities to utilize the
available resources solely for resolving the current issues. From that, we can
estimate the energy savings and generate forecasts for the total energy
investment for training the model. In the long term, such a framework can be
extended beyond pure training adaptation in order to provide a lifecycle-
oriented toolkit for estimating energy investments from initial idea to final
deployment.

##  4\. Evaluation

We envision evaluating our approach through a comprehensive two-stage process
with an extended focus on machine learning applications from the HAR field.

First, a numerical analysis of the machine learning problem from data,
architecture, and resource investment perspectives. This involves a detailed
examination of data quality impacts, such as sensor placement variability,
noise, and labeling errors, on the performance of the machine learning models.
By identifying these issues from common HAR datasets, we aim to quantify their
effects on model accuracy and efficiency with respect to the utilized model
architecture. Further, we will measure and compare the energy consumption
across various stages of the machine learning pipeline. This includes data
collection, preprocessing, model training, and inference. We will evaluate
energy usage with and without the integration of hybrid intelligence
interventions to demonstrate potential energy savings and improved efficiency.

Secondly, we investigate the effectiveness of HITL from our Hybrid
Intelligence approach. We will gather feedback from human experts who
participate in the HITL process. Such feedback will help us assess the
effectiveness of real-time adjustments and optimizations made during the model
training and deployment phases. Additionally, the extension from HITL to LLM
agents can help collect user feedback to determine the perceived value and
satisfaction with the LLM interventions. This evaluation will focus on how
smart agents assist non-experts in generating an optimized development
environment.

By combining numerical analysis with user feedback, we aim to provide a
thorough evaluation, demonstrating the potential of the final framework to
enhance both the sustainability and performance of machine learning models
concurrently.

##  5\. Contribution

The main goal of this work is to connect smart and interactive visualizations
with sustainability approaches to reduce computational resource requirements.
Our contributions can be divided into two main research directions as
discussed before, the Hybrid Intelligence visualizations and the energy
awareness.

Building on previous work in latent space exploration as described in (Geißler
et al., 2023), this research initially conducted an asynchronous analysis of
model layer outputs. The next step is to make this process more interactive by
incorporating real-time visualizations and user feedback within the training
directly. After extending this work with an LLM-based agent, future research
aims to compare the effectiveness of human versus LLM agents in terms of user
experience and overall effectiveness in optimizing the training process.

Towards energy awareness, the impact of hyperparameter settings and different
hardware environments on energy consumption has already been investigated
through (Geißler et al., 2024). Ongoing research is focused on embedding
energy consumption metrics directly into the training process. One approach
being explored is energy-aware hyperparameter optimization using methods like
the successive halving algorithm. This integration aims to optimize model
training not just for performance but also for energy efficiency.

In the field of ubiquitous computing, particularly within HAR, the reliance on
time series sensor data from real-world applications presents several
challenges. Wearable sensing technologies often suffer due to their real-world
nature gathered from human activities, such as sensor quality, placement
variability, environmental noise, inconsistent experimental procedures, and
error-prone data labeling. These challenges can significantly degrade machine
learning model performance, leading to inefficient resource usage as attempts
are made to compensate, often through increased model complexity. Future
research will focus on state-of-the-art datasets from HAR to identify
ambiguities within the data itself. By adapting models and training processes
to be aware of these data issues, we aim to improve overall efficiency. The
Hybrid Intelligence approach is crucial here, as it allows us to bridge the
gap between energy-aware model optimization and HITL systems in the scenario
of HAR.

##  6\. Conclusion

By leveraging the strengths of human cognition and artificial intelligence,
Hybrid Intelligence can significantly improve the sustainability and
effectiveness of ubiquitous applications, addressing the various challenges
from data, models, and hardware. Most of the introduced approaches are still
in the conceptualization phase and require in-depth investigation through
future work.

###### Acknowledgements.

This work is supported by the European Union’s Horizon Europe research and
innovation program (HORIZON-CL4-2021-HUMAN-01) through the ”SustainML” project
(grant agreement No 101070408).

## References

  * (1)
  * Akata et al. (2020) Zeynep Akata, Dan Balliet, Maarten De Rijke, Frank Dignum, Virginia Dignum, Guszti Eiben, Antske Fokkens, Davide Grossi, Koen Hindriks, Holger Hoos, et al. 2020\.  A research agenda for hybrid intelligence: augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence.  _Computer_ 53, 8 (2020), 18–28.
  * Alicioglu and Sun (2022) Gulsum Alicioglu and Bo Sun. 2022.  A survey of visual analytics for explainable artificial intelligence methods.  _Computers & Graphics_ 102 (2022), 502–520.
  * Anthony et al. (2020) Lasse F Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. 2020.  Carbontracker: Tracking and predicting the carbon footprint of training deep learning models.  _arXiv preprint arXiv:2007.03051_ (2020).
  * Budennyy et al. (2022) Semen Andreevich Budennyy, Vladimir Dmitrievich Lazarev, Nikita Nikolaevich Zakharenko, Aleksei N Korovin, OA Plosskaya, Denis Valer’evich Dimitrov, VS Akhripkin, IV Pavlov, Ivan Valer’evich Oseledets, Ivan Segundovich Barsola, et al. 2022\.  Eco2ai: carbon emissions tracking of machine learning models as the first step towards sustainable ai. In _Doklady Mathematics_ , Vol. 106. Springer, S118–S128.
  * Chatzimparmpas et al. (2020) Angelos Chatzimparmpas, R. M. Martins, Ilir Jusufi, K. Kucher, Fabrice Rossi, and A. Kerren. 2020.  The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations.  _Computer Graphics Forum_ 39 (2020).  https://doi.org/10.1111/cgf.14034
  * Chu et al. (2024) Xiangxiang Chu, Jianlin Su, Bo Zhang, and Chunhua Shen. 2024.  VisionLLaMA: A Unified LLaMA Interface for Vision Tasks.  _arXiv preprint arXiv:2403.00522_ (2024).
  * cloud-carbon footprint (2023) cloud-carbon footprint. 2023.  Cloud Carbon Footprint.  https://github.com/cloud-carbon-footprint/cloud-carbon-footprint.  [Online; accessed 12-06-2024].
  * Code-Carbon (2023) Code-Carbon. 2023.  Code Carbon.  https://github.com/mlco2/codecarbon.  [Online; accessed 12-06-2024].
  * Desislavov (2023) et al. Desislavov. 2023.  Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning.  _Sustainable Computing: Informatics and Systems_ 38 (2023), 100857.
  * Fuchs et al. (2009) Raphael Fuchs, J. Waser, and M. E. Gröller. 2009.  Visual Human+Machine Learning.  _IEEE Transactions on Visualization and Computer Graphics_ 15 (2009).  https://doi.org/10.1109/TVCG.2009.199
  * Geißler et al. (2024) Daniel Geißler, Bo Zhou, Mengxi Liu, Sungho Suh, and Paul Lukowicz. 2024.  The Power of Training: How Different Neural Network Setups Influence the Energy Demand.  _arXiv preprint arXiv:2401.01851_ (2024).
  * Geißler et al. (2023) Daniel Geißler, Bo Zhou, Paul Lukowicz, and RPTU Kaiserslautern-Landau. 2023.  Latent inspector: an interactive tool for probing neural network behaviors through arbitrary latent activation. In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence_. 7127–7130.
  * Henderson et al. (2020) Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. 2020.  Towards the systematic reporting of the energy and carbon footprints of machine learning.  _The Journal of Machine Learning Research_ 21, 1 (2020), 10039–10081.
  * Hurley et al. (2021) C. Hurley, Mark O’Connell, and Katarina Domijan. 2021.  Interactive Slice Visualization for Exploring Machine Learning Models.  _Journal of Computational and Graphical Statistics_ 31 (2021), 1 – 13.  https://doi.org/10.1080/10618600.2021.1983439
  * Jay et al. (2023) Mathilde Jay, Vladimir Ostapenco, Laurent Lefevre, Denis Trystram, Anne-Cécile Orgerie, and Benjamin Fichel. 2023.  An experimental comparison of software-based power meters: focus on CPU and GPU. In _2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)_. 106–118.  https://doi.org/10.1109/CCGrid57682.2023.00020
  * Lacoste et al. (2019) Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019.  Quantifying the carbon emissions of machine learning.  _arXiv preprint arXiv:1910.09700_ (2019).
  * Lannelongue et al. (2021) Loïc Lannelongue, Jason Grealey, and Michael Inouye. 2021.  Green algorithms: quantifying the carbon footprint of computation.  _Advanced science_ 8, 12 (2021), 2100707.
  * Li et al. (2020) Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020.  Train big, then compress: Rethinking model size for efficient training and inference of transformers. In _International Conference on machine learning_. PMLR, 5958–5968.
  * Mehlin et al. (2023) Vanessa Mehlin, Sigurd Schacht, and Carsten Lanquillon. 2023.  Towards energy-efficient deep learning: An overview of energy-efficient approaches along the deep learning lifecycle.  _arXiv preprint arXiv:2303.01980_ (2023).
  * Mittal and Vetter (2014) Sparsh Mittal and Jeffrey S Vetter. 2014.  A survey of methods for analyzing and improving GPU energy efficiency.  _ACM Computing Surveys (CSUR)_ 47, 2 (2014), 1–23.
  * Möbius et al. (2013) Christoph Möbius, Waltenegus Dargie, and Alexander Schill. 2013.  Power consumption estimation models for processors, virtual machines, and servers.  _IEEE Transactions on Parallel and Distributed Systems_ 25, 6 (2013), 1600–1614.
  * Rauber et al. (2017) P. E. Rauber, Samuel G. Fadel, A. Falcão, and A. Telea. 2017.  Visualizing the Hidden Activity of Artificial Neural Networks.  _IEEE Transactions on Visualization and Computer Graphics_ 23 (2017), 101–110.  https://doi.org/10.1109/TVCG.2016.2598838
  * Reddy et al. (2020) G Thippa Reddy, M Praveen Kumar Reddy, Kuruva Lakshmanna, Rajesh Kaluri, Dharmendra Singh Rajput, Gautam Srivastava, and Thar Baker. 2020.  Analysis of dimensionality reduction techniques on big data.  _Ieee Access_ 8 (2020), 54776–54788.
  * Sacha et al. (2017) D. Sacha, M. Sedlmair, Leishi Zhang, J. Lee, J. Peltonen, D. Weiskopf, S. North, and D. Keim. 2017.  What you see is what you can change: Human-centered machine learning by interactive visualization.  _Neurocomputing_ 268 (2017), 164–175.  https://doi.org/10.1016/J.NEUCOM.2017.01.105
  * Soni et al. (2023) A. Soni, Chirag Arora, Rajkumar Kaushik, and Vaishali Upadhyay. 2023.  Evaluating the Impact of Data Quality on Machine Learning Model Performance.  _Journal of Nonlinear Analysis and Optimization_ (2023).  https://doi.org/10.36893/jnao.2023.v14i1.0013-0018
  * Wei et al. (2022) Jiafu Wei, Haoran Xie, Chia-Ming Chang, and Xi Yang. 2022.  Fine-tuning Deep Neural Networks by Interactively Refining the 2D Latent Space of Ambiguous Images.. In _IJCAI_. 5948–5951.
  * Wongsuphasawat et al. (2018) Kanit Wongsuphasawat, D. Smilkov, James Wexler, Jimbo Wilson, Dandelion Mané, Doug Fritz, Dilip Krishnan, F. Viégas, and M. Wattenberg. 2018.  Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow.  _IEEE Transactions on Visualization and Computer Graphics_ 24 (2018), 1–12.  https://doi.org/10.1109/TVCG.2017.2744878
  * Yao et al. (2023) Zhenpeng Yao, Yanwei Lum, Andrew Johnston, Luis Martin Mejia-Mendoza, Xin Zhou, Yonggang Wen, Alán Aspuru-Guzik, Edward H Sargent, and Zhi Wei Seh. 2023.  Machine learning for a sustainable energy future.  _Nature Reviews Materials_ 8, 3 (2023), 202–215.
  * Zhang et al. (2022) Shibo Zhang, Yaxuan Li, Shen Zhang, Farzad Shahabi, Stephen Xia, Yu Deng, and Nabil Alshurafa. 2022.  Deep learning in human activity recognition with wearable sensors: A review on advances.  _Sensors_ 22, 4 (2022), 1476.