original network. Each isolated node dynamics is close to a Hopf bifurcation.
The pairwise coupling function $h(z,w)=z\bar{w}$. Inset b) shows the
hypernetwork reconstructed from the phase dynamics of the original dynamics.
The state of a given node is influenced by the triple interaction of its own
state in combination of the incoming links. Our theory also predicts the
emergent of such hypernetwork. The hypernetwork emerges as a combination of
the original coupling function $h(z,w)=z\bar{w}.$, the network, and the
resonance relations of the isolated frequencies.

$\displaystyle\dot{z}_{1}$ $\displaystyle=\gamma_{1}z_{1}-\beta
z_{1}|z_{1}|^{2}+\alpha(z_{1}\overline{z}_{2}+z_{1}\overline{z}_{3}+z_{1}\overline{z}_{4})$
(S172) $\displaystyle\dot{z}_{2}$ $\displaystyle=\gamma_{2}z_{2}-\beta
z_{2}|z_{2}|^{2}+\alpha(z_{2}\overline{z}_{1}+z_{2}\overline{z}_{3}+z_{2}\overline{z}_{5})$
$\displaystyle\dot{z}_{3}$ $\displaystyle=\gamma_{3}z_{3}-\beta
z_{3}|z_{3}|^{2}+\alpha(z_{3}\overline{z}_{1}+z_{3}\overline{z}_{2}+z_{3}\overline{z}_{6})$
$\displaystyle\dot{z}_{4}$ $\displaystyle=\gamma_{4}z_{4}-\beta
z_{4}|z_{4}|^{2}+\alpha(z_{4}\overline{z}_{5}+z_{4}\overline{z}_{6}+z_{4}\overline{z}_{1})$
$\displaystyle\dot{z}_{5}$ $\displaystyle=\gamma_{5}z_{5}-\beta
z_{5}|z_{5}|^{2}+\alpha(z_{5}\overline{z}_{4}+z_{5}\overline{z}_{6}+z_{5}\overline{z}_{2})$
$\displaystyle\dot{z}_{6}$ $\displaystyle=\gamma_{6}z_{6}-\beta
z_{6}|z_{6}|^{2}+\alpha(z_{6}\overline{z}_{4}+z_{6}\overline{z}_{5}+z_{6}\overline{z}_{3})\,.$

We will assume either one of

  1. 1.

$\gamma_{2}\approx\gamma_{5}$, $\gamma_{1}\approx\gamma_{6}$ and
$\gamma_{3}\approx\gamma_{4}$, with $\gamma_{1}\not\approx\gamma_{2}$ (and
hence $\gamma_{1},\gamma_{6}\not\approx\gamma_{2},\gamma_{5}$),
$\gamma_{2}\not\approx\gamma_{3}$ and $\gamma_{1}\not\approx\gamma_{3}$.

  2. 2.

$\gamma_{2}\approx\gamma_{5}$ and $\gamma_{i}\not\approx\gamma_{j}$ for all
$i,j\in\\{1,\dots,6\\}$ with $i\not=j$ and $(i,j)\not=(2,5),(5,2)$.

In each case, we may transform ODE (S172) into

$\displaystyle\dot{v}_{1}$ $\displaystyle=\gamma_{1}v_{1}-\beta
v_{1}|v_{1}|^{2}-\epsilon\frac{v_{1}\overline{v}_{2}v_{5}}{\overline{\gamma}_{2}}+\mathcal{O}(|\epsilon,v|^{5})$
(S173) $\displaystyle\dot{v}_{2}$ $\displaystyle=\gamma_{2}v_{2}-\beta
v_{2}|v_{2}|^{2}-\epsilon\frac{v_{2}\overline{v}_{5}v_{2}}{\overline{\gamma}_{5}}+\mathcal{O}(|\epsilon,v|^{5})$
$\displaystyle\dot{v}_{3}$ $\displaystyle=\gamma_{3}v_{3}-\beta
v_{3}|v_{3}|^{2}-\epsilon\frac{v_{3}\overline{v}_{2}v_{5}}{\overline{\gamma}_{2}}+\mathcal{O}(|\epsilon,v|^{5})$
$\displaystyle\dot{v}_{4}$ $\displaystyle=\gamma_{4}v_{4}-\beta
v_{4}|v_{4}|^{2}-\epsilon\frac{v_{4}\overline{v}_{5}v_{2}}{\overline{\gamma}_{5}}+\mathcal{O}(|\epsilon,v|^{5})$
$\displaystyle\dot{v}_{5}$ $\displaystyle=\gamma_{5}v_{5}-\beta
v_{5}|v_{5}|^{2}-\epsilon\frac{v_{5}\overline{v}_{2}v_{5}}{\overline{\gamma}_{2}}+\mathcal{O}(|\epsilon,v|^{5})$
$\displaystyle\dot{v}_{6}$ $\displaystyle=\gamma_{6}v_{6}-\beta
v_{6}|v_{6}|^{2}-\epsilon\frac{v_{6}\overline{v}_{5}v_{2}}{\overline{\gamma}_{5}}+\mathcal{O}(|\epsilon,v|^{5})\,,$

where $\epsilon=\alpha^{2}$ leading to the emergent higher order network
displayed in Figure 12b).

##  8 Model recovery and normal form representation

Let $x\in\mathbb{R}^{m}$ and consider

$\displaystyle\dot{x}$ $\displaystyle=$ $\displaystyle F(x)$ (S174)

We assume for simplicity that $F:\mathbb{R}^{m}\rightarrow\mathbb{R}^{m}$ is a
polynomial map and

$\displaystyle\dot{x}_{i}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{k}a_{ij}p_{j}(x),$ (S175)

where $x_{i}$ is the $i$th coordinate of $x$ and $p_{j}$’s form a basis of
homogeneous polynomials. Notice that in a network context,
$x\in\mathbb{R}^{m}$ would represent the state vector of the network and $F$
would model isolated dynamics and interactions. Once a trajectory $x(t)$ and
$\dot{x}(t)$ are known, we perform a model recovery as follows. Fix a sampling
$h$ and introduce

$V=\left(\begin{array}[]{cccc}\dot{x}_{1}(0)&\dot{x}_{2}(0)&\dots&\dot{x}_{m}(0)\\\
\dot{x}_{1}(h)&\dot{x}_{2}(h)&\dots&\dot{x}_{m}(h)\\\
\vdots&\vdots&\ddots&\vdots\\\
\dot{x}_{1}(T)&\dot{x}_{2}(T)&\dots&\dot{x}_{m}(T)\\\
\end{array}\right)\mbox{\leavevmode\nobreak\ \leavevmode\nobreak\ and
\leavevmode\nobreak\ \leavevmode\nobreak\
}X=\left(\begin{array}[]{cccc}x_{1}(0)&x_{2}(0)&\dots&x_{m}(0)\\\
x_{1}(h)&x_{2}(h)&\dots&x_{m}(h)\\\ \vdots&\vdots&\ddots&\vdots\\\
x_{1}(T)&x_{2}(T)&\dots&x_{m}(T)\\\ \end{array}\right)$

along with

$\Phi(X)=\left(\begin{array}[]{cccc}p_{1}(x(0))&p_{2}(x(0))&\cdots&p_{k}(x(0))\\\
p_{1}(x(h))&p_{2}(x(h))&\cdots&p_{k}(x(h))\\\ \vdots&\vdots&\ddots&\vdots\\\
p_{1}(x(T))&p_{2}(x(T))&\cdots&p_{k}(x(T))\\\ \end{array}\right)$

Let $v_{i}$ be the $i$th column of $V$ and
$\xi_{i}=(a_{i1},a_{i2},\dots,a_{ik})^{*}$. Here, ∗ denotes the transpose.
Then by construction

$\Phi(X)\xi_{i}=v_{i}$ (S176)

and if for large $T$ the operator $\Phi$ is full rank the solution of Eq.
(S176) is unique. Solving this equation for all coordinates, we recover the
differential equation.

In data, however, due to numerical round-off errors or noise Eq. (S176) is
perturbed and one seeks for solutions allowing a small error
$\|\Phi(X)\xi_{i}-v_{i}\|<\varepsilon_{0}$ but under a model simplification
such as imposing that some coefficients of $\xi_{i}$ are zero, that is,
looking for sparse solutions.

The sparse model recovery of the coefficients $\xi_{i}$ is the problem

$\min_{q\in\mathbb{R}^{k}}\|q\|_{0}\mbox{\leavevmode\nobreak\
\leavevmode\nobreak\ subjected to \leavevmode\nobreak\ \leavevmode\nobreak\
}\|\Phi(X)q-v_{i}\|\leq\lambda$

for a suitably chosen $\lambda>0$.

Now we are ready to prove the following

######  Theorem 8.1.

Consider Eq. (S175) and the following assumptions

  * (H0) Eq. (S175) is generic (coefficients $a_{ij}$ are non vanishing)

  * (H1) Eq (S175) has a normal form

$\displaystyle\dot{y}$ $\displaystyle=$ $\displaystyle G(y)+R(y)$ (S177)

where $G$ contains no non-resonant terms and $\|R(y)\|=O(y^{d+1})$ for some
large $d$. Moreover the coordinates of $y$ have the expansion

$\dot{y}_{i}=\sum_{j=1}^{k}b_{ij}p_{j}(y)+R_{i}(y)$

  * (H2) The trajectories $\\{x(t)\\}_{t=0}^{T}$ as well as $\\{\dot{x}(t)\\}_{t=0}^{T}$ are given with $T$ sufficiently large and stay in a sufficiently small neighbourhood $V_{\varepsilon}$ of the origin such that

$\sup_{t\in[0,T]}\|x(t)\|_{C^{1}}\leq\varepsilon$

for initial conditions in an open neighbourhood of the origin.

  * (H3) The operator $\Phi$ is full rank.

Then there exist $\lambda=\lambda(\varepsilon,d)>0$ such the solution to the
sparse recovery problem

$\min_{q\in\mathbb{R}^{k}}\|q\|_{0}\mbox{\leavevmode\nobreak\
\leavevmode\nobreak\ subjected to \leavevmode\nobreak\ \leavevmode\nobreak\
}\|\Phi(X)q-v_{i}\|_{2}\leq\lambda$

is the vector of coefficients of $(b_{i1},b_{i2},\dots,b_{ik})^{*}$ of the
normal form of Eq. (S175)

###### Proof.

We break the arguments into three steps:

Step1: Approximations and Uniqueness solutions. By normal form theory there
are functions $Q_{1}$ and $Q_{2}$ such that

$x=y+Q_{1}(y)\mbox{\leavevmode\nobreak\ \leavevmode\nobreak\ and
\leavevmode\nobreak\ \leavevmode\nobreak\ }y=x+Q_{2}(x)$ (S178)

where $\|Q_{1}(y)\|=O(\|y\|^{2})$ and $\|Q_{2}(x)\|=O(\|x\|^{2})$. Given a
trajectory $y(t)$ we construct the matrix $Y$ in the same manner as $X$ and
consider

$u_{i}=\left(\begin{array}[]{c}\dot{y}_{i}(0)\\\ \dot{y}_{i}(h)\\\ \vdots\\\
\dot{y}_{i}(T)\end{array}\right)\mbox{\leavevmode\nobreak\ and
\leavevmode\nobreak\ }\rho_{i}(Y)=\left(\begin{array}[]{c}R_{i}(y(0))\\\
R_{i}(y(h))\\\ \vdots\\\ R_{i}(y(T))\end{array}\right)$

As the basis is formed by homogeneous polynomials, using Eq. (S178) we
conclude that there is $L$ such that

$\|\Phi(X)-\Phi(Y)\|_{2}\leq L\varepsilon^{2}$

By $(H3)$ $\Phi(X)$ is full rank and for $\varepsilon^{2}$ small enough, we
conclude that $\Phi(Y)$ is also full rank since the rank is lower
semicontinuous. Next notice that the equation

$\Phi(Y)\zeta_{i}+\rho_{i}(Y)=u_{i}$ (S179)

also has a solution $\zeta_{i}=(b_{i1},b_{i2},\dots,b_{ik})^{*}$ by
construction and it is unique since $\Phi(Y)$ is full rank. Furthermore, in
$V_{\varepsilon}$ there is a constant $M$ such that

$\|\rho_{i}(Y)\|_{2}\leq M\varepsilon^{2},\,\,\,\,\forall i\in\\{1,\dots,m\\}$

Using Eq. (S178) we obtain

$\displaystyle v_{i}=u_{i}+z_{i}$ (S180)

where $z_{i}$ corresponds to terms as $DP(y)\dot{y}$. By $(H2)$ trajectories
stay in the neighbourhood $V_{\varepsilon}$, thus, there is a constant $C$
such that

$\|z_{i}\|_{2}\leq C\varepsilon^{2},\,\,\,\,\forall i\in\\{1,\dots,m\\}$

Step 2: A sparse solution. Consider the unique solution $\zeta_{i}$ of Eq.
(S179) and let $\sigma_{i}=\|\zeta_{i}\|_{0}$. Consider the set

$B_{\lambda,\sigma_{i}}=\\{q\in\mathbb{R}^{k}:\|q\|_{0}\leq\sigma_{i}\mbox{\leavevmode\nobreak\
and \leavevmode\nobreak\ }\|\Phi(X)q-v_{i}\|_{2}\leq\lambda\\}$

Now we claim that if $\lambda:=(L\|\zeta_{i}\|_{2}+C+M)\varepsilon^{d}$ then
$\zeta_{i}\in B_{\lambda,\sigma_{i}}$. Indeed, consider

$\displaystyle\|\Phi(X)\zeta_{i}-v_{i}\|_{2}$ $\displaystyle=$
$\displaystyle\|\Phi(X)\zeta_{i}+\Phi(Y)\zeta_{i}-\Phi(Y)\zeta_{i}-v_{i}\|_{2}$
$\displaystyle=$
$\displaystyle\|[\Phi(X)-\Phi(Y)]\zeta_{i}+\Phi(Y)\zeta_{i}-u_{i}-z_{i}+\rho_{i}-\rho_{i}\|_{2}$
$\displaystyle=$
$\displaystyle\|[\Phi(X)-\Phi(Y)]\zeta_{i}-z_{i}-\rho_{i}\|_{2}$
$\displaystyle\leq$
$\displaystyle\|\Phi(X)-\Phi(Y)\|_{2}\|\zeta_{i}\|_{2}+\|z_{i}\|_{2}+\|\rho_{i}\|_{2}$
$\displaystyle\leq$ $\displaystyle(L\|\zeta_{i}\|_{2}+M+C)\varepsilon^{2}$

Step 3: Uniqueness. Assume that there is $\eta\in B_{\lambda,\sigma_{i}}$ with
$\|\eta\|_{0}<\sigma_{i}$. Since $\Phi(X)$ is full rank, this implies that
there is $\hat{R}$ such that $\|\hat{R}(x)\|_{2}\leq K\|x\|_{2}^{d+1}$ for
some $K$ and $\dot{x}=\hat{G}(x)+\hat{R}(x)$. Thus, $\hat{G}$ has fewer
coefficients than $G$, implying that either $G$ must have a non-resonant term
or $(H0)$ was violated. This contracts $(H1)$ and completes the proof.
∎

######  Remark 8.2.

Assumption $H2$ is natural in our context. Notice since the isolated system
has a limit cycle near the origin. Thus, an open set of initial conditions is
attracted to the cycles and stays for all times near the origin where we
control the norm of solutions [13]. When coupling such dynamics to a network
this behaviour persists.

######  Remark 8.3.

Assumption $H3$ is in general not restrictive. If solutions of $\dot{x}=F(x)$
are not degenerated such as all solutions converge to fixed points, then
typically $\Phi(X)$ is full rank. In fact, if solutions converge to an
attractor, we can adapt the basis to the dynamics such that in the adapted
basis $\Phi^{*}(X)\Phi(X)$ is close to identity for large $T$ [38]. This
implies that $\Phi(X)$ is close to orthogonal.

######  Remark 8.4.

Another interesting case is when

$\dot{x}=F(x)+U(t,x)$

and $U$ has fast oscillations. This happens typically in phase dynamics when
we subtract the trends of linear frequencies. For example, consider

$\displaystyle\dot{\theta}$ $\displaystyle=$ $\displaystyle
1+\epsilon\sin(\theta-\phi)+\epsilon\cos(\theta+\phi)$ (S181)
$\displaystyle\dot{\phi}$ $\displaystyle=$ $\displaystyle
1+\delta+\epsilon\sin(\phi-\theta)+\epsilon\cos(\theta+\phi)$ (S182)

where $\delta\ll 1$. Subtracting the trend $\vartheta=\theta-t$ and
$\varphi=\phi-t$ leads to

$\displaystyle\dot{\vartheta}$ $\displaystyle=$
$\displaystyle\epsilon\sin(\vartheta-\varphi)+\epsilon\cos(\vartheta+\varphi-2t)$
(S183) $\displaystyle\dot{\varphi}$ $\displaystyle=$
$\displaystyle\delta+\epsilon\sin(\varphi-\vartheta)+\epsilon\cos(\vartheta+\varphi-2t)$
(S184)

Since $\dot{\varphi}$ and $\dot{\vartheta}$ are $O(\delta)$ by the averaging
Theorem, fast oscillating terms containing $\cos$ are averaged out and can be
neglected in a time scale as $1/\delta$. Thus, also in this case when
performing a model recovery with finite amount of data the function $U$ cannot
be recovered. This also happens for our examples in the main text. Thus,
sparsity and fast oscillations can contribute to the impossibility of
recovering the original model.

##  9 Emergent hypernetworks in an integrate-and-fire model

Integrate and fire model. We used an autocatalytic integrate-and-fire model
[39] to simulate the behavior of four oscillators in a ring configuration with
state variable $v_{k}$, and a parameter for each oscillator $p_{k}$ that
determines whether the variable is increasing or decreasing. In the model, we
introduce nonlinear time-delayed coupling, and the oscillators are governed by
the equations

$\frac{dv_{k}}{dt}=\frac{p_{k}v_{k}-(1-p_{k})v_{k}B}{F_{k}}+p_{k}K\sum_{\l=1}^{4}A_{k,l}(\tilde{v}_{k}+\tilde{v}_{k}^{2})\tilde{v}_{l}(t-\tau)$
(S185)

where $F_{k}$ is a rescaling factor that affects the natural frequency of
$k$th oscillator, $K$ is the coupling strength, $\tilde{v}_{k}$ is the signal
corrected for offset ($\tilde{v}_{k}=v_{k}-0.626$), $A_{k,l}$ is the adjacency
matrix, and $\tau$ is the time delay.

When the variable $v_{k}$ reaches 1 from below, then $p_{k}$ is smoothly set
to 0, and $v_{k}$ decreases. Similarly, when the variable $v_{k}$ reaches from
above $A$, $p_{k}$ is set to 1, and the variable starts to increase. We
selected the threshold parameter $A=0.36$ and the timescale parameter
$B=3.333$ so that only the one-cluster is stable with positive coupling. Then
we adjusted the parameter $F_{k}$ ($F_{1}$=4.950, $F_{2}$=1.955,
$F_{3}$=3.177, $F_{4}$=1.970) of each oscillator to have a frequency ratio
with respect oscillator 1 as $\omega_{2}/\omega_{1}\approx 2.5$,
$\omega_{3}/\omega_{1}\approx 1.5$ and $\omega_{4}/\omega_{1}$$\approx 2.5$.
Note that $F_{k}$ only affects the local dynamics of oscillator $k$ and not
the coupling term. Figure 13 (a) shows the time series of the variable $v_{k}$
for $K$=0.234 and $\tau$=1.65 s.

Figure 13: Simulations and network reconstruction with the integrate-and-fire
model a) Time series of the $v_{k}$ variable. The blue, orange, yellow, and
green lines correspond to oscillators one to four. b) Time series of the slow
phases, $\phi_{1}$ (blue) and $\phi_{2}$ (green) with coupling and delay. c)
The instantaneous frequency and the phase model fitted values (red) for
oscillators 1 to 4 (corresponding from top to bottom). d) Coupling amplitudes
for the four oscillators ($k$) from hypernetworks one (blue) and 2 (red).

Fitting of phase dynamics. Similar to the experiments, we extract the phase of
each oscillator using the peak-finding approach [36] from the time series of
the variable $v_{k}$. When there is coupling and delay, the triplet phase
differences, $\phi_{j}$, $j$=1, 2, show a phase slip behavior Supplementary
Fig. 13 (b). As described in the main text we used LASSO to fit the
$\dot{\theta}_{k}$ values accoriding to equation (S28) with drifting in the
natural frequencies. The time series of the $v_{k}$ variable (see
Supplementary Fig. 13 (a)) showed a more nonlinear wave form, and we fitted
the amplitudes of sin and cos until the seond order harmonics ($C_{j,2}^{k}$
and $D_{j,2}^{k}$). The $\dot{\theta}_{k}$ was filtered by a first order
Savitzky-Golay filter for 125 s. Supplementary Fig. 13 (c) shows the
corresponding fits for oscillator 1 to 4. (In the LASSO fit, we used a
regularization parameter that represented an error 40% higher than the best
fit). The fitted parameters are shown in Supplementary Fig. 14.

Figure 14: Hypernetwork fitting coefficients for phase dynamics of the
integrate-and-fire model.

The strength of the triplet interactions on oscillator $k$ is given by the
amplitudes of the first and second harmonics ($H_{j}^{k}$); the amplitudes are
shown in Supplementary Fig. 13d. In agreement with the experiments, the
dynamics of oscillators 1 and 3 are impacted by both triplet interactions
$\phi_{1}$ and $\phi_{2}$. For oscillator 1, the amplitudes are 1.7 $\times
10^{-3}$ and 1.0 $\times 10^{-3}$, and for oscillators 3 the amplitudes are
5.6 $\times 10^{-4}$ and 7.4 $\times 10^{-4}$ respectively. However, the
dynamics of oscillator 2 and 4 are only impacted by $\phi_{1}$ (2.1 $\times
10^{-3}$) and $\phi_{2}$ (1.1 $\times 10^{-3}$).

We conclude that in an integrate-and-fire model, the phase dynamics of the
oscillators coupled in a ring can be described by an emergent hypernetwork.

##  10 Mean field interaction

We consider the system

$\displaystyle\dot{z}_{k}$
$\displaystyle=\gamma_{k}z_{k}-\beta_{k}z_{k}|z_{k}|^{2}+\alpha\sum_{\ell=1}^{n}A_{k\ell}(z_{\ell}+\bar{z}_{\ell}z_{k}^{2})\,,$
(S186)

where $A$ is the 4-ring network, with nodes labelled $1$ through $4$ along the
ring. These frequencies satisfy the resonance conditions
$\omega_{1}+\omega_{3}\approx 2\omega_{2}$ and $\omega_{2}+\omega_{4}\approx
2\omega_{1}$. A priori, it is unclear what the behavior of the system (S186)
will look like. To elucidate this, we conjugate the system by a transformation
designed to get rid of the third order coupling terms in $\alpha$. To this
end, we define new coordinates

$w_{k}=z_{k}-\alpha\sum_{\ell=1}^{n}\frac{A_{k\ell}}{\gamma_{k}+\bar{\gamma}_{\ell}}z_{k}^{2}\bar{z}_{\ell}\,.$
(S187)

This causes new terms in $\alpha$ to appear, related to the
$\beta_{k}z_{k}|z_{k}|^{2}$ terms in Equation (S186). We therefore perform
another coordinate transformation

$u_{k}=w_{k}-\alpha Q_{k}(w)\,,$ (S188)

for some suitably chosen polynomials $Q_{k}$. We get equations for
$\dot{u}_{k}$, which involve, among others, combinations of the linear and
non-linear terms in the coupling

$\alpha\sum_{\ell=1}^{n}A_{k\ell}(z_{\ell}+\bar{z}_{\ell}z_{k}^{2})\,.$ (S189)

We then discard non-resonant terms in $\alpha^{2}$, which leaves the equations

$\displaystyle\dot{u}_{1}$
$\displaystyle=\gamma_{1}u_{1}-\beta_{1}u_{1}|u_{1}|^{2}+\alpha(u_{2}+u_{4})+\alpha^{2}\frac{u_{2}^{2}\bar{u}_{3}}{\gamma_{2}+\bar{\gamma}_{3}}+\text{h.o.t.}$
(S190) $\displaystyle\dot{u}_{2}$
$\displaystyle=\gamma_{2}u_{2}-\beta_{2}u_{2}|u_{2}|^{2}+\alpha(u_{1}+u_{3})+\alpha^{2}\frac{u_{1}^{2}\bar{u}_{4}}{\gamma_{1}+\bar{\gamma}_{4}}+\text{h.o.t.}$
$\displaystyle\dot{u}_{3}$
$\displaystyle=\gamma_{3}u_{3}-\beta_{3}u_{3}|u_{3}|^{2}+\alpha(u_{2}+u_{4})+\alpha^{2}\frac{u_{2}^{2}\bar{u}_{1}}{\gamma_{2}+\bar{\gamma}_{1}}+\text{h.o.t.}$
$\displaystyle\dot{u}_{4}$
$\displaystyle=\gamma_{4}u_{4}-\beta_{4}u_{4}|u_{4}|^{2}+\alpha(u_{1}+u_{3})+\alpha^{2}\frac{u_{1}^{2}\bar{u}_{2}}{\gamma_{1}+\bar{\gamma}_{2}}+\text{h.o.t.}\,.$

See Section 10.3 for more details on these normal form calculations. If we
ignore the (non-resonant) terms $\alpha(u_{2}+u_{4})$ and
$\alpha(u_{1}+u_{3})$ in Equation (S190), then averaging yields the emergent
phase dynamics. To this end, we set $\varphi_{1}:=\phi_{1}-2\phi_{2}+\phi_{3}$
and $\varphi_{2}:=\phi_{2}-2\phi_{1}+\phi_{4}$ for the slow phases. The
different monomials in Equation (S190) then yield terms in the phase equations
according to:

  * •

for node 1, $\frac{u_{2}^{2}\bar{u}_{3}}{\gamma_{2}+\bar{\gamma}_{3}}$ gives
terms involving sin/cos of $\varphi_{1}$;

  * •

for node 2, $\frac{u_{1}^{2}\bar{u}_{4}}{\gamma_{1}+\bar{\gamma}_{4}}$ gives
terms involving sin/cos of $\varphi_{2}$;

  * •

for node 3, $\frac{u_{2}^{2}\bar{u}_{1}}{\gamma_{2}+\bar{\gamma}_{1}}$ gives
terms involving sin/cos of $\varphi_{1}$;

  * •

for node 4, $\frac{u_{1}^{2}\bar{u}_{2}}{\gamma_{1}+\bar{\gamma}_{2}}$ gives
terms involving sin/cos of $\varphi_{2}$.

###  10.1 Frequency shifts

The linear terms $\alpha(u_{2}+u_{4})$ and $\alpha(u_{1}+u_{3})$ nevertheless
have an effect on the emergent dynamics, in the following way. Whereas the
natural frequencies of the uncoupled system (i.e. for $\alpha=0$) are given by
$\omega_{1},\dots,\omega_{4}$, they are in general given by the imaginary part
of the eigenvalues of the perturbed matrix

$U=\lambda I+i\Omega+\alpha A$

where, $\Omega$ is the diagonal matrix with entries
$\omega_{1},\dots,\omega_{4}$, and $A$ is the adjacency matrix of the network.
Eigenvalue perturbation then gives augmented frequencies of the form
$\omega_{k}+O(\alpha^{2})$. Note that the frequency perturbation is again of
order $\alpha^{2}$. This can be explained by a linear transformation bringing
the perturbed system $\lambda I+i\Omega+\alpha A$ to that of the form $\lambda
I+i\Omega+\alpha^{2}B$, similar to our techniques for non-linear terms.

Therefore, whenever $\alpha>0$ the frequencies will shift providing a
frequency mismatch between the slow phases

$\displaystyle\varphi_{1}=\theta_{1}-2\theta_{2}+\theta_{3}$ (S191)
$\displaystyle\varphi_{2}=\theta_{2}-2\theta_{1}+\theta_{4}$ (S192)

namely, they will be modelled as

$\displaystyle\dot{\varphi}_{1,2}=\varepsilon_{1,2}+G_{1,2}(\varphi_{1})+H_{1,2}(\varphi_{2}).$
(S193)

where $\varepsilon_{1,2}=O(\alpha^{2})$.

###  10.2 Model Recovery

We integrate Eq. (S186) with $\Omega_{1}=2,\Omega_{2}=3,\Omega_{3}=4$ and
$\Omega_{4}=1$, by employing a wrapper of ODEPACK routine. Numerical
integration was performed for 25000s with 0.01 time step. We discard the first
5000s points as transient.

We apply sparse regression using PySINDy Python package [40] with the Lasso
optimizer on the phases $\theta_{i}$ considering the slow phases
$\varphi_{1,2}$ with a penalty term $\lambda=5\times 10^{-3}$, we obtain

$\displaystyle\dot{\theta}_{1}$ $\displaystyle=2.001+0.018\cos(\varphi_{1})$
(S194) $\displaystyle\dot{\theta}_{2}$
$\displaystyle=2.999-0.015\cos(\varphi_{2})$ (S195)
$\displaystyle\dot{\theta}_{3}$ $\displaystyle=3.992-0.011\cos(\varphi_{1})$
(S196) $\displaystyle\dot{\theta}_{4}$
$\displaystyle=1.008+0.011\cos(\varphi_{2}).$ (S197)

To recover the slow phase dynamics of $\varphi_{1}$ and $\varphi_{2}$ we apply
the Lasso method with a penalty term $\lambda=10^{-5}$ after applying a
rolling window averaging process using window size of 100s to smooth the fast
oscillations to have better fit on slow phases. The obtained equation after
the Lasso approach reads as

$\displaystyle\dot{\varphi_{1}}$
$\displaystyle=-0.008+0.002\sin(\varphi_{1})+0.001\cos(\varphi_{2})$ (S198)
$\displaystyle\dot{\varphi_{2}}$
$\displaystyle=\,\,\,\,\,0.008-0.001\cos(\varphi_{1})+0.002\sin(\varphi_{2}).$
(S199)

The theory and the fitting are also in a perfect agreement for this mean-field
case.

###  10.3 Normal Form Calculations

Here we consider the case where we have both (non-resonant) linear coupling
terms as well as higher order ones. More precisely, we consider the system

$\displaystyle\dot{z}_{k}$
$\displaystyle=\gamma_{k}z_{k}-\beta_{k}z_{k}|z_{k}|^{2}+\alpha\sum_{\ell=1}^{n}c_{k,\ell}z_{\ell}+\alpha
H_{k}(z)\,,$ (S200)

where $H_{k}$ has only terms of degree $3$ and higher. Later, we will set
$H_{k}(z)=\sum_{\ell=1}^{n}c_{k,\ell}z_{k}^{2}\bar{z_{\ell}}$. We assume that
corresponding functions $P_{k}$ exist that solve

$\displaystyle\Gamma P_{k}-\gamma_{k}P_{k}=H_{k}\,.$ (S201)

In particular, when
$H_{k}(z)=\sum_{\ell=1}^{n}c_{k,\ell}z_{k}^{2}\bar{z_{\ell}}$ we assume that
$\omega_{k}\not=\omega_{\ell}$ whenever $c_{k,\ell}\not=0$, so that we may
define

$\displaystyle
P_{k}(z)=\sum_{\ell=1}^{n}\frac{c_{k,\ell}}{\gamma_{k}+\bar{\gamma}_{\ell}}z_{k}^{2}\bar{z}_{\ell}\,.$
(S202)

With slight abuse of notation, the sum in Equation (S202) is taken over all
$\ell$ such that $c_{k,\ell}\not=0$. As before, we consider the coordinate
transformation $w_{k}=z_{k}-\alpha P_{k}(z)$, which gives

$z_{k}=w_{k}+\alpha
P_{k}(w)+\alpha^{2}[P_{k}||P](w)+\mathcal{O}(|\alpha|^{3}|w|^{7})\,,$ (S203)

by lemmas 2.18 and 2.21. A calculation as before reveals that

$\displaystyle\dot{w}_{k}$
$\displaystyle=\gamma_{k}w_{k}-\beta_{k}w_{k}|w_{k}|^{2}+\alpha\sum_{\ell=1}^{n}c_{k,\ell}w_{\ell}+\alpha(\gamma_{k}P_{k}(w)-\Gamma
P_{k}(w)+H_{k}(w))$ (S204)
$\displaystyle+\alpha(L^{1}_{k}(w)-L^{2}_{k}(w))+\alpha^{2}[\gamma_{k}P_{k}-\Gamma
P_{k}+H_{k}||P](w)$
$\displaystyle+\alpha^{2}\sum_{\ell=1}^{n}c_{k,\ell}P_{\ell}(w)-\alpha^{2}[P_{k}||(\dots,\sum_{\ell=1}^{n}c_{j,\ell}w_{\ell},\dots)](w)-\alpha^{2}[P_{k}||H](w)$
$\displaystyle+\mathcal{O}(|\alpha|^{2}|w|^{7}+|\alpha|^{3}|w|^{5})$
$\displaystyle=\gamma_{k}w_{k}-\beta_{k}w_{k}|w_{k}|^{2}+\alpha\sum_{\ell=1}^{n}c_{k,\ell}w_{\ell}+\alpha(L^{1}_{k}(w)-L^{2}_{k}(w))$
$\displaystyle+\alpha^{2}\sum_{\ell=1}^{n}c_{k,\ell}P_{\ell}(w)-\alpha^{2}[P_{k}||(\dots,\sum_{\ell=1}^{n}c_{j,\ell}w_{\ell},\dots)](w)-\alpha^{2}[P_{k}||H](w)$
$\displaystyle+\mathcal{O}(|\alpha|^{2}|w|^{7}+|\alpha|^{3}|w|^{5})$

where in the last step we have used Equation (S201), and where we again set

$\displaystyle L^{1}_{k}(w)$ $\displaystyle:=[P_{k}||(\dots,\beta
w_{j}|w_{j}|^{2},\dots)](w)\text{ and }$ (S205) $\displaystyle L^{2}_{k}(w)$
$\displaystyle:=[\beta_{k}w_{k}|w_{k}|^{2}||P](w)\,.$

Next, we perform a second transformation $u_{k}=w_{k}-\alpha Q_{k}(w)$, where
$Q_{k}$ solves

$\displaystyle\Gamma Q_{k}-\gamma_{k}Q_{k}=L^{1}_{k}-L^{2}_{k}\,,$ (S206)

and where

$w_{k}=u_{k}+\alpha Q_{k}(u)+\mathcal{O}(|\alpha|^{2}|u|^{9})\,.$ (S207)

This gives

$\displaystyle\dot{u}_{k}$
$\displaystyle=\gamma_{k}u_{k}-\beta_{k}u_{k}|u_{k}|^{2}+\alpha\sum_{\ell=1}^{n}c_{k,\ell}u_{\ell}+\alpha(\gamma_{k}Q_{k}(u)-\Gamma
Q_{k}(u)+L^{1}_{k}(u)-L^{2}_{k}(u))$ (S208)
$\displaystyle+\alpha^{2}\sum_{\ell=1}^{n}c_{k,\ell}Q_{\ell}(u)-\alpha^{2}[Q_{k}||(\dots,\sum_{\ell=1}^{n}c_{j,\ell}u_{\ell},\dots)](u)$
$\displaystyle+\alpha^{2}\sum_{\ell=1}^{n}c_{k,\ell}P_{\ell}(u)-\alpha^{2}[P_{k}||(\dots,\sum_{\ell=1}^{n}c_{j,\ell}w_{\ell},\dots)](u)-\alpha^{2}[P_{k}||H](u)$
$\displaystyle+\mathcal{O}(|\alpha||u|^{7}+|\alpha|^{3}|u|^{5})\,.$

By Equation (S206) we therefore get

$\displaystyle\dot{u}_{k}$
$\displaystyle=\gamma_{k}u_{k}-\beta_{k}u_{k}|u_{k}|^{2}+\alpha\sum_{\ell=1}^{n}c_{k,\ell}u_{\ell}$
(S209)
$\displaystyle+\alpha^{2}\sum_{\ell=1}^{n}c_{k,\ell}Q_{\ell}(u)-\alpha^{2}[Q_{k}||(\dots,\sum_{\ell=1}^{n}c_{j,\ell}u_{\ell},\dots)](u)$
$\displaystyle+\alpha^{2}\sum_{\ell=1}^{n}c_{k,\ell}P_{\ell}(u)-\alpha^{2}[P_{k}||(\dots,\sum_{\ell=1}^{n}c_{j,\ell}w_{\ell},\dots)](u)-\alpha^{2}[P_{k}||H](u)$
$\displaystyle+\mathcal{O}(|\alpha||u|^{7}+|\alpha|^{3}|u|^{5})\,.$

We now return to the special case of
$H_{k}(z)=\sum_{\ell=1}^{n}c_{k,\ell}z_{k}^{2}\bar{z_{\ell}}$. By assumption,
the terms in $\alpha$ are resonant. Of the terms in $\alpha^{2}$, only

$\displaystyle\alpha^{2}\sum_{\ell=1}^{n}c_{k,\ell}P_{\ell}(u)-\alpha^{2}[P_{k}||(\dots,\sum_{\ell=1}^{n}c_{j,\ell}w_{\ell},\dots)](u)$
(S210)

is third order in $u$ (as opposed to fifth order). A direct calculation shows
that

$\displaystyle\sum_{\ell=1}^{n}c_{k,\ell}P_{\ell}(u)-[P_{k}||(\dots,\sum_{\ell=1}^{n}c_{j,\ell}w_{\ell},\dots)](u)$
(S211)
$\displaystyle=\sum_{\ell=1}^{n}\sum_{p=1}^{n}\frac{c_{k,\ell}c_{\ell,p}}{\gamma_{\ell}+\bar{\gamma}_{p}}u_{\ell}^{2}\bar{u}_{p}-\sum_{\ell=1}^{n}\sum_{p=1}^{n}\frac{c_{k,\ell}c_{\ell,p}}{\gamma_{k}+\bar{\gamma}_{\ell}}u_{k}^{2}\bar{u}_{p}-2\sum_{\ell=1}^{n}\sum_{p=1}^{n}\frac{c_{k,\ell}c_{k,p}}{\gamma_{k}+\bar{\gamma}_{\ell}}u_{k}\bar{u}_{\ell}u_{p}\,.$

From these the resonant terms can be selected, which leads to a hypernetwork
description of the dynamics.


# Visual Abstraction

Ivan Viola
King Abdullah University
of Science and Technology (KAUST)
Saudi Arabia
<EMAIL_ADDRESS>
&Min Chen
University of Oxford
United Kingdom
<EMAIL_ADDRESS>
Tobias Isenberg
Inria Saclay
France
<EMAIL_ADDRESS>

###### Abstract

In this article we revisit the concept of abstraction as it is used in
visualization and put it on a solid formal footing. While the term
_abstraction_ is utilized in many scientific disciplines, arts, as well as
everyday life, visualization inherits the notion of data abstraction or class
abstraction from computer science, topological abstraction from mathematics,
and visual abstraction from arts. All these notions have a lot in common, yet
there is a major discrepancy in the terminology and basic understanding about
visual abstraction in the context of visualization. We thus root the notion of
abstraction in the philosophy of science, clarify the basic terminology, and
provide crisp definitions of visual abstraction as a process. Furthermore, we
clarify how it relates to similar terms often used interchangeably in the
field of visualization. Visual abstraction is characterized by a conceptual
space where this process exists, by the purpose it should serve, and by the
perceptual and cognitive qualities of the beholder. These characteristics can
be used to control the process of visual abstraction to produce effective and
informative visual representations.

_K_ eywords abstraction $\cdot$ visualization $\cdot$ visual abstraction

##  1 Definitions

The term abstraction often lacks a precise definition in many fields. While
several fields have defined the term for their own purposes, there is only a
vague understanding of its meaning that is shared by all fields. Some
scientific disciplines and scholarly fields have adjusted the vaguely
understood meaning to fit the needs of the respective discipline or field. In
this article we first present our key definitions related to visual
abstraction, and we then provide the justification for the definitions. In
giving these definitions, we revise our previous set of definitions relating
to the concept of abstraction [1], based on new discussions related to, and
insights from, our further literature study. Terminology related to
abstraction has been adopted from Leppänen [2] and is discussed in subsection
2.7.

###### Definition 1

An abstraction is a process that transforms a _source thing_ into a less
concrete _sign thing_ of the _source thing_. Abstraction uses a concept of
_point-of-view_ , which determines which aspects of source thing should be
preserved in its sign thing and which should be suppressed.

###### Definition 2

A data representation is a sign thing that stands in digital form for a
_referent thing_ from reality or another _sign thing_ , using data structures
or _concept things_. Similarly, a visual representation is a _sign thing_ that
stands for a _referent_ from reality or another _sign thing_ so that it can be
visually perceived and cognitively processed by a human observer.

###### Definition 3

Visual abstraction is a particular type of abstraction where the sign thing is
visual, while the source thing is either non-visual or visual. A visual
representation results from a process of visual abstraction if such
transformation intentionally disregards certain aspects of data
representations.

###### Definition 4

The abstraction process also involves a point-of-view component defined
through the task, which the visualization process aids to accomplish. This
task is represented as a combination of _targets_ on which particular
_actions_ are performed.

###### Proposition 1

The amount or significance of abstraction of a thing can be, in computer or
signal representations, quantified by means of information theory.

###### Definition 5

A meaningful visual abstraction is a visual abstraction such that, for a given
point-of-view and for a given _purpose_ or _goal_ , key aspects of the
underlying _referent thing_ are preserved in the visual representation so that
the cognitive load when perceiving it as a stimulus is significantly reduced.

###### Definition 6

A visualization is a process that transforms data representations of a thing
from reality into visual representations. Visualization is a process that is
intended to be a meaningful visual abstraction process. The designers of
visualization processes must understand the point-of-view component and tasks.
Otherwise, they would not reach the full meaningfulness intended.

###### Definition 7

An abstraction axis is the _perceived_ sequence of visual representations that
is assembled by the designer of a visualization system to illustrate a given
point or series of points about reality. Each of the building blocks of an
abstraction axis is the result of an individual abstraction process to a
visual representation. Each transition between two successive abstraction axis
building blocks can but does not have to remove information, some can also
both remove and add information based on chosen blocks specific abstraction.
If two or more abstraction axes are constructed such that they affect
independent aspects of the visual representations, they can be combined into
an abstraction space that observers can explore.

##  2 Flavors of Abstraction

The notion of what is abstract and what is concrete is a fundamental
discussion in philosophy, without a clear consensus. In its simplest terms, an
abstract object has no physical referent, while concrete objects have physical
referents. Reiterating Frege’s writings, “The Thought” [3] is even stronger in
restricting what an abstract thought is: “An object is abstract if and only if
it is both non-physical and non-mental.” An object is acknowledged as mental
when “it exists at a time if and only if it is the object or content of some
mental state or process at that time.” This statement implies that an abstract
object is an object if and only if it cannot be found in nature, cannot be
constructed, and one cannot even form a mental image of it.

Another definition of abstract objects is that they lack causal powers [4].
This means that abstract objects cannot affect other objects in any way. An
empty set is such a case of an abstract entity as it does not have any causal
powers. The definition of abstract entity is often so strict that some
philosophers deny the existence of an abstract entity as such. However, there
seems to be better agreement on what an abstraction is: “It is a distinctive
mental process in which new ideas or conceptions are formed by considering
several objects or ideas and omitting the features that distinguish them” [4].
Lewis [5] proposed that “abstract entities are abstractions from concrete
entities. They result from somehow subtracting specificity, so that an
incomplete description of the original concrete entity would be a complete
description of the abstraction.” In the rest of the article we use the term
abstraction aligned with these definitions to only describe a process, as we
have also done in our own definitions at the beginning. The entity after
abstraction is, in our case, denoted as a representation. We do not enter the
dispute of whether it is an abstract entity or not. In such a way we build on
the part that philosophers agreed upon, while we avoid the terminological
controversy. Before we look at the use of abstraction in visualization, let us
first consider its occurrence in related arts and sciences.

###  2.1 Abstraction in the arts

In the arts, the term _abstract art_ refers to non-figurative artwork, where
the intent is to develop art beyond depiction of natural or man-made objects.
The composition may exist with a degree of independence from visual references
in the world [6]. This art movement started during early 20th century and
emerged from figurative art. Artists such as Picasso, Mondrian, Kandinski, and
many others originally depicted natural objects. The beginning of non-
figurative art started with a deep analysis and observation of the creative
process, where the graphical elements that composed the rendering became
themselves the subject of study. The natural objects were gradually
represented through collection of simpler geometric primitives. The artists
searched for an expression of minimal set of visual elements that is still
able to carry the figurative meaning. But they did not stop there. Artists
further experimented with the graphical elements beyond recognizability of any
corresponding figure from the rendering itself. Interestingly, one can
sometimes discover a correspondence to their earlier works where a particular
figure is still recognizable, thus transitively the figure can be imagined in
the fully abstract art with such aids as well. It indicates that the artists
still had a particular figure in mind, when rendering a particular art, while,
without the prior work context, this figure would not be discovered by another
human observer. This gradual process, which transformed figurative art into
what is now called abstract art, is abstraction.

###  2.2 Abstraction and generalization in cartography

In cartography, depending on a chosen scale for a map and its type/target
audience, a subset of information is selected, the elements to be depicted are
simplified and their depiction is adjusted. For example, streets can be shown
with a much larger width than in reality, yet fine details of their path are
removed. When zooming out, important elements and landmarks in the map are
depicted, while generally less relevant elements are suppressed. At a
particular level of scale, for example, the post office, a religious place, a
building of historical significance, a bridge over the river, or the main
streets are clearly depicted in the map, while similar objects in terms of
spatial dimensions are abstracted into very simplified representations, if
they are shown at all. The field has created a solid vocabulary and guidelines
on how certain elements should be depicted and when should they be visible. In
cartographic visual language, the umbrella term for guidelines of how
different scales should depict certain information is  _map generalization_
[7]. We discuss the specific meaning of the term _generalization_ below, but
other principles such as _grouping_ or _classification_ are applied here as
well. In prior work, these concepts are considered as distinct abstraction
principles and we discuss their specifics below.

###  2.3 Abstraction in shape analysis

In shape analysis, the term _abstraction_ typically refers to a
skeletonization or extraction of topological features that represent essential
characteristics of the underlying shape [8, 9]. Here, abstraction preserves
the key properties of the geometric components such as their connectivity. The
levels of detail of these abstracted representations are controlled through
measures like persistence: this measure determines which structures are too
small for particular scale to justify their validity and which are grouped
into other larger-scale structures. Such abstracted representations facilitate
the extraction of hierarchies in shapes to facilitate geometric linkage,
multi-scale representations, and—importantly—the topological representation is
much sparser and facilitates an unobstructed clear view on the key geometric
properties. The same holds for the topology of flow data, where a flow field
is _classified_ into points and regions of certain uniform properties such as
sinks, sources, and separatrices (curves or surfaces) that partition the flow
according to its long-term behavior.

###  2.4 Mathematical abstraction

The term mathematical abstraction refers to a process of transforming a
specific real-world situation into generalized form using mathematical
formalism.111https://en.wikipedia.org/wiki/Abstraction_(mathematics) The
specifics which do not affect the solution to a given problem are removed so
that, in the end, only a set of key elements with properties and relations to
each other remains, which can be expressed formally. Problems to solve in
mathematics class are frequently expressed as real-world situations. The tasks
are to abstract from the real-world specifics and apply a mathematical
formalism that provides the answer to the given problem. The development of
mathematics and physical sciences has advanced through mathematical
abstraction into Euclidean geometry, algebra, and analysis. These developments
have been possible due to humans being capable of thinking in an abstract way.

###  2.5 Abstract thinking

School students are trained in abstract thinking by being challenged to solve
a specific real-world problem. To be able to do so, they are trained to
abstract from the case specifics by extracting only the essential components
so that a formal solution can be calculated and, finally, interpreted back for
the specific real-world scenario. Abstract thinking is, according to cognitive
psychology [10], the most complex stage in the development of cognitive
thinking, where generalizations and concepts are used in the thought process.
From a set of observations, hypotheses can be formed and logical reasoning can
lead to conclusive statements [10].

###  2.6 Abstraction in object-oriented design

In computer science, the term abstraction achieves yet another flavor of its
meaning. In object-oriented design, the most frequently used programming
methodology, it primarily relates to the definition of classes and methods
that cannot be instantiated. Typically, classes and methods are hierarchically
grouped into increasingly abstract constructs such that implementations of
particular functionality can be shared among many different elements. While
for most of these classes it is possible to create instances, an abstract
class is a construct that itself cannot be instantiated but which organizes
the functionality into a comprehensive representation. The class hierarchy as
the outcome of such abstraction gives a clear understanding of differences in
functionality among various classes as well as what they have in common. It
also facilitates further extensibility of existing code to support new cases
that were not considered in the initial software design.

###  2.7 Abstraction ontology

In the area of information and knowledge modeling, a particularly interesting
past work closely relates to our own investigation. Leppänen [2] distinguishes
between first-order and second-order abstraction. First-order abstraction is
associated with primary things, while second-order abstraction acts upon a
predicate that defines the primary things. An example of a primary thing is
_sedan_ with several predicates, among others a _color_. The result of the
abstraction of a sedan would be a _car_ or a _vehicle_ , which corresponds to
first-order abstraction. Let us assume that an instance of a sedan is painted
with a particular blue, for example _Maya Blue_. This predicate can also be
abstracted to _light blue_ or _blue_ , a process which is of the second-order
abstraction type and is also termed as predicate abstraction.

Importantly, Leppänen defines four elementary abstraction principles:
classification, generalization, composition, and grouping. First,
classification is defined through the term _isInstanceOf_ or that instances
are _typeOf_. The opposite to classification is instantiation. Second,
generalization is a principle of abstraction where the differences of subtypes
are suppressed to fit a supertype. This refers to an _isA_ relationship and
the antonym to generalization is specialization. Third, composition is a
principle of abstraction in which a whole concept is composed of part
concepts. These parts are abstracted to form a whole object. This refers to a
_partOf_ relationship and its opposite is the decomposition. Finally, the last
principle of abstraction is grouping which relies on a _isMemberOf_
relationship and whose opposite is individualization. For example, a
particular person can be a member of a political party. This abstraction
includes aggregation, set membership, and association. Both, first-order and
second-order abstractions can benefit all four elementary abstraction
principles. In all cases, an important property to highlight is that
abstraction is associated with an intentional and controlled loss of
information.

Leppänen’s work stresses the importance of the concept _point-of-view_ that
plays crucial role during the abstraction process. When using classification
on a _thing_ termed, for example, _Margaret Thatcher_ , the abstraction along
classification would lead to entity _female_ or _UK Prime Minister_. If we
would be using grouping, the abstraction would lead to _Conservative Party_.
In case the composition principle is used for abstraction of _UK Prime
Minister_ , the outcome would be _UK Government_. Therefore, things might
generally have many different kinds of abstractions as things from reality are
typically embedded in a complex and intertwined abstraction hierarchies.

In his work, Leppänen combines philosophical and semiotic standpoints. In the
context of semiotic frameworks [11], they refer to three kinds of _things_ : a
_concept thing_ , a _referent thing_ , and a _sign thing_. Concepts are mental
constructs, words of mind, and form basic components of human knowledge. A
referent is an element of reality that relates to the concept. A reality
describes a set of anything that exists or can possibly exist, physically or
virtually. A sign is anything that can stand for something else, including
symbols, text, or images. As such it is a representation of a concept. These
concepts are used below in the discussion of abstraction in visualization. We
applied the same terminology in our definitions from the start of the article,
but we added the concept of a _source thing_ (Def. 1).

###  2.8 Summary of abstraction in the world outside visualization

The intuitive understanding of abstraction has been reinforced by this brief
excursion into various fields and that stand and argue for abstraction. We can
observe that the term is not used uniformly and that it is frequently
exchanged with other terms. The recurrent pattern is that abstraction relates
to formation of some higher-order constructs or representations that are
result of a transformation of lower-level entities. The lowest entities are
more tangible, while the higher levels of the abstraction hierarchy are
further removed from tangibility and become more mental constructs and
concepts (defined as _the constituents of thoughts_ [4]) that, in one way or
another, allow humans to recognize certain characteristics clearer than the
lower-level representations. The ability to abstract seems to be one of the
core properties of humans, present while shaping the entire body of analytical
knowledge humankind has formed throughout our history.

##  3 Abstraction for Visualization

Let us now investigate how abstraction manifests itself in visualization. We
propose that abstraction is equally central to visualization as it is to other
areas in which analytic reasoning is the core part of a processing workflow.
Visualization is the process of transforming the digital representation of
data into visual representations that are exposed to a human viewer (Def. 6).
It takes advantage of the fact that most humans are extremely efficient in
comprehending information presented as a visual stimulus. Naturally, this
stimulus has to be well designed to convey the intended information (Def. 5).
This aspect is the main concern of the visualization mapping stage of the
visualization pipeline. Visualization is omnipresent in studying various real-
world phenomena, conveying structures, methods, or concepts. In visualization,
the abstraction process guides the transformation into visual representations
(Def. 2, 3), similar to the process of abstract thinking. In some sense it
serves as an extension of the working memory, where needed information can be
instantaneously accessed. We thus first clarify the meaning of abstraction in
visualization and then discuss its core properties.

To bring visualization into the context of semiotic frameworks, the _sign_ is
termed as _representation_ , both digital and visual, and the _referent_ is
the studied phenomenon from _reality_ (Def. 1, 2). The _concept_ is what
relates to the _referent_ and can be conveyed through the representation. In
visualization, abstraction is performed at least in three stages: first, the
abstraction of the reality into data representations and, second, the data
representation is, through abstraction, transformed into visual
representations. Third, a visual representation is transformed to a mental
model or a memory representation through the perceptual and cognitive
processes of the human observer.

Abstraction has occurred if the quantum of information before the abstraction
is higher than in the representation after abstraction, while some aspects of
the original representation are preserved and become more prominent (Def. 1).
In case there is no intended information loss, we refer to a more general term
_transformation_ or _mapping_. For example, several simultaneous abstraction
processes that individually work on different aspects of the _things_ could be
combined, some work in a positive direction (removal of information) and
others in a negative direction. This could lead to composite _transformation_
or _mapping_ that transfer one representation into another, with information
loss and information gain at the same time.

###  3.1 Task abstraction

Visualization is driven by a particular _intent_. There is a reason behind a
visualization, even in the casual scenarios. This intent defines the _point-
of-view_ (Def. 4), which, as a controlling mechanism, can steer how
abstraction changes the representations. In the visualization literature,
Munzner [12] describes a hierarchical framework into which specific individual
visualization usage scenarios can be abstracted. On the highest level, Munzner
classifies the tasks as a combination of an _action_ upon a _target_. The
_action_ class is instantiated into _analyze_ , _search_ , and _query_ , which
can be further instantiated into lower-level classes of _actions_. The
_target_ is instantiated into _data_ in general, _attributes_ , _networks_ ,
and _spatial data_ which are further instantiated into more detailed targets.
It is the combination of the _action_ and _target_ that would define the
_point of view_ to guide the abstraction process.

###  3.2 Data abstraction

Munzner [12] also defines various types of _data_ and _data sets_ for
visualization. Data types are _items_ , _attributes_ , _links_ , _positions_ ,
and _grids_. Data set types are _tables_ , _networks_ , _trees_ , _fields_ ,
_geometry_ , _clusters_ , _sets_ , and _lists_. All these types are _concept
things_ (Def. 2). The data abstraction here refers to the transformation from
the real-world phenomenon, the _referent thing_ , into data structures
(_concept thing_) and digital representations (_sign thing_), to facilitate an
efficient and automatized computational processing. This task of data
abstraction is somewhat similar to the mathematical abstraction process. In
both cases we end up with a formal representation on which standardized
mathematical or computational machinery can be applied.

The initial data abstraction is typically performed during the acquisition
process. Either real-world observations are made and digitally stored in a
particular data representation or even a mathematical model is formulated
based upon these observations. Both forms are data representations abstracted
from the thing that exists in reality, and these representations have been
achieved through a classification process.

The result of the initial data abstraction is frequently further abstracted
into another data representation to promote a particular _point-of-view_ ,
neglecting unimportant aspects of the original data representation. As such,
the filtering operation is typically applied, which might be considered to
relate to map generalization and as such corresponds to the generalization
abstraction principle. Once the data representation contains the relevant data
prominently, a conversion into data representation is performed that can
efficiently be visually represented.

##  4 Visual Abstraction

Figure 1: Abstraction space in which a _thing_ from reality is gradually
transformed into visual representations: a) initial abstraction into a digital
form, b) data abstraction into new data representation, c) different data
abstractions can lead to identical data representation, d) visual abstraction
transforms the data representation into a visual representation, e) visual
abstraction transforms one visual representation into another visual
representation, f) the abstraction space encodes less and less information
from the original _thing_ from reality. The further from center the more
sparse the representation is. g) the dotted line conveys a visualization
pipeline that can be seen as a composite visual abstraction.

After series of data abstractions and transformations (the latter when no
information loss happens), in visualization, the data is
transformed/abstracted into visual representations. A visual representation is
then shown on a display, perceived, and further cognitively processed by a
human observer. The visual abstraction process that generates this visual
representation can be performed in many ways: In the case of kernel density
estimation plots or clustering techniques, for instance, data can be visually
abstracted using a composition principle such that smaller elements become a
part of higher-order representations. In case of volumetric scalar fields, the
voxel values can be classified into color and opacity ranges. By this, some
voxels become abstracted into types such as air, soft tissue, or hard tissue.
Level of detail techniques would typically relate to composition or grouping;
in atomistic visualization, individual atoms become member of particular
molecules, which in turn become members of certain compartments, up to cells.
In many cases of particular visual abstraction it can be simultaneously argued
for different abstraction principles, and there might be more principles than
those proposed by Leppänen [2].

Munzner [12] provides a conceptual framework according to which visual
representations or encoding can be categorized. This framework is rather
extensive, however on the low level the visual encoding can abstract data
representations through two key aspects. The first aspect is the graphical
_mark_ that positions each data element: _points_ , _lines_ , and _areas_.
These marks further encode quantitative information or their mutual
relationship is conveyed through various perceptual channels: _position_ ,
_shape_ , _color_ , _size_ , and _angle_. These basic low-level perception-
driven visual elements can be combined to create rich spectrum of possible
visual representations. These visual representations can be used to _encode_ ,
_manipulate_ , _compare_ , or _reduce_ the data in the visual representation
space.

Data representations and visual representations can be ordered according to
how much they abstract a particular phenomenon. The abstraction process is
depicted in Figure 1. We can see the abstraction space related to one _thing_
, one entity from reality. There are several ways how the _thing_ can be
abstracted into a digital form. After this first stage of the process, the
data representation has been abstracted from the _thing_. There could be
several data abstractions applied, under which the data becomes sparser and
sparser so that the information sought by the user or intended by the
visualization creator becomes gradually clearer. Sometimes even the series of
abstractions can take different paths yet still result into the same data
representation. In practice, however, such data abstractions would only apply
to a given path to a particular visual representation, as most visualization
systems will maintain their original datasets to allow users to also observe
different visual representations, which would be the result of a different
sequence of data abstractions.

After the sequence of data abstractions, the data representation is still non-
visual. If we apply a visual mapping to such data (whether with intentional
loss of information or not), we achieve a visual representation that can be
viewed on a display. But even visual representations can be further
transformed into sparser visual representations by means of visual
abstraction. The more far away in the abstraction space, the less information
from the reality is preserved. If we concatenate a path from the reality to
the final visual representation, we can see a visualization pipeline. In case
we perform a transformation so that the distance between the original
representation and the reality and the target representation and the reality
are the same, we do not perform an abstraction. If the target representation
is closer to the reality than the original representation, we perform an
inverse operation to abstraction. Yet this inverse abstraction only happens in
the eyes of the beholder, as we always remove information along the path from
reality to visual representation.

###  4.1 Meaningful Abstraction

It is not clear whether an abstraction has to be meaningful or whether its
only condition is a loss of information. What if, for example, a high-
dimensional data set is projected onto fewer dimensions? Projection is, in
principle, a valid abstraction. But what if we project only every second data
element and create a confusing data representation in which only half of the
data set is projected onto lower-dimensional space. Is such a meaningless
projection also an abstraction? From the information-theoretic point of view
we have lost a certain amount of information, so it can be considered as an
abstraction. To differentiate us from this view, we should define the term
_meaningful abstraction_ for those abstractions that are useful in some
application contexts (Def. 5).

Visual mapping may result into a representation with an equal amount of
information, however, more visually confusing than the previous
representation. For example, it is known that humans have difficulties with
identifying portraits of known faces if they are rotated by 180 degrees from
the natural portrait orientation [13]. From the information-theoretic point of
view, the rotation does not remove information from the image, but there is a
significant difference in cognitive load between these two representations.
Such a rotation is consequently not a meaningful visual mapping. The same
holds for two visual representations of a graph, a node-link diagram and an
adjacency matrix. When the one visual representation is transformed into
another, no information is lost. Yet the cognitive load for viewers differs
between these two representations. Building on the term of _meaningfulness_ ,
a visual representation can be more meaningful (or effective) for a particular
intent than another visual representation. Visual abstractions that lead to
these representations might be ordered or perhaps even quantified in how
meaningful they are.

The concept of _meaningfulness_ in terms of visual abstraction processes is
tightly related to visual perception processes. In principle, a meaningful
visual abstraction makes the job of visual processing simpler so that less of
a cognitive processing needs to be invested, for a given purpose or goal, in
comprehending the abstracted visual representation to understand the intended
aspects of the reality. Therefore, visual abstractions relevant for
visualization will need to result into lower cognitive load when comprehending
the abstracted representation, for the chosen intention. Therefore the
_meaningful visual abstraction_ has to pass two conditions: the target visual
representation has to formally contain less information and the cognitive load
has to be lower. The perceived information, if not increase, should decrease
at most linearly with the cognitive load.

At this point we solidify the previous discussion and define some key terms in
visualization. Abstraction is a process, it is a transformation along which
some information is intentionally lost to give prominence to the higher-level
information within. The abstraction process results into a representation. For
pure data abstraction, it results in a data representation, while, when visual
abstraction is involved, it results in a visual representation. These
abstractions can be considered as meaningful as long as they are benefiting
particular application example or purpose. The meaningfulness property is
scoped by the set of meaningful applications. A visual representation is the
result of a visual transformation. When information is intentionally lost and
the cognitive load is lower, while the perceived information loss is, at most,
linear with the cognitive load difference, then we consider the visual
abstraction as meaningful. Visual mapping and visual encoding, while both
having their distinct meaning, can be used interchangeably with visual
transformation. Visual metaphor operates on the concept of _analogy_. It
presents a sign thing of a different referent thing from reality than the one
originally regarded. This way visual mapping associates properties of one
referent thing to another referent thing. An example of a visual metaphor are
Chernoff faces, where different facial properties encode multivariate data
[14].

###  4.2 Abstraction Axes and Abstraction Spaces

So far we mainly discussed the process of abstraction from reality via data
representations to visual representations. Yet we also showed that positive or
negative abstraction can be _perceived_ by a viewer as he or she is
manipulating this abstraction chain or visualization pipeline. For better
describing the latter aspect, Viola and Isenberg [1], inspired by earlier
examples in visualization [15, 16, 17, 18] as well as in the arts world,
proposed the notion of _axes of abstraction_ which could form an _abstraction
space_. With these two concepts we can describe the abstraction that is
perceived and controlled by the beholder, in contrast to the abstraction that
is applied as a particular visual representation is generated (Def. 7).

An _abstraction axis_ in this concept is the previously mentioned virtual,
perceived connection between different end points of the previously discussed
abstraction process. This connection arises for observers as they adjust the
settings of the visualization pipeline. This notion, however, assumes that,
for each abstraction axis, there is a clearly identifiable succession of
changes to the visual representation that (a) decreases the amount of
information in each step and (b) provides a meaningful generalization of the
depicted content to the viewer. In fact, Viola and Isenberg [1] even state
that abstraction axes do not need to be unique: for their chosen example from
structural biology [18] they show that a molecular van-der-Waals surface-based
molecular representation can be subjected to two alternative forms of
structural abstraction (a phenomenon they call “forking” of axes)—one leading
to a surface-based abstraction via different probe sizes and one leading to a
second-order representation via balls-and-sticks, licorice (sticks-only), and
backbone representations.

In particular for this latter form of abstraction, one could argue that
condition (a) is not necessarily met: while the transition from van-der-Waals
surfaces to the licorice representation in van der Zwan’s [18] model certainly
removes the detail of the graphical atom representations, it simultaneously
also adds representations of the bonds between atoms that did not exist in the
starting configuration: the representations of atoms with implicitly
represented bonds are continuously replaced with representations of bonds with
implicitly represented atoms. One could thus argue that in this transition no
abstraction happens, only one representation is smoothly transitioned into
another. This transition, however, only happens in the eyes of the beholder;
at any given point along the transition still abstraction happens from reality
(source things) via data representations to visual representations.

A more recent example is the work by Miao et al. [16] who similarly
constructed a progression of transitions from an atom-based representation of
DNA nanostructures and the mechanical building blocks of the nanostructures to
be built. Interesting in their progression of ten abstraction stages is that,
while the first and last are fairly clear, the specific order of the sequence
in-between is not and was created based on the discussions with and needs of
their collaborating domain scientists.

Based on these two examples, we thus suggest that pure and continuous
abstraction axes are rather rare. Instead, abstraction axes are typically
composed of smaller building blocks where one representation is (typically)
seamlessly transformed into another and as such forms a constructed sequence.
In practice we often find abstraction axes that progress from a representation
with more information to a representation with less detail, thus the name
abstraction axis. We can also find transitions, however, that remove one type
of visual detail and replace it with another type of visual information.
Abstraction axes are always constructed with a given purpose and application
case in mind and are not unique. If two abstraction axes work on independent
attributes of the visual representation and can thus be independently
controlled, then they form an _abstraction space_ (Def. 7).

##  5 An Information-Theoretic Analysis of Abstraction as a Process

As described above, the notion of “abstraction” encompasses a wide range of
definitions in different contexts. It can be quite difficult for a single
mathematical formulation to encapsulate the essences of these definitions. In
this section, we examine the characteristics of the process of abstraction
using information-theoretic measures. We show that the definitions given at
the beginning of this article can be explained using an information-theoretic
metric, which therefore offers a potential means for modelling and measuring
visual abstraction.

|
---|---
(a) London underground map [19] | (b) pen-and-ink volume rendering [20]
|
(c) 3D streamline flow visualization [21] | (d) glyph-based video visualization [22]
|
(e) network visualization [15] | (f) parallel coordinates visualization [23]
|
(g) 3D flow visualization [24] | (h) volume visualization and deformation [25]
Figure 2: Examples of visualization images that may attract different views as
to whether they are the results of visual abstraction processes. Most would
agree that (a)-(d) are visually abstract, many would content that (e) and (f)
are considered so, and some might be hesitant about (g) and (h).

Figure 2 shows several visualization images generated using some typical
visualization techniques. Most visualization researchers would unreservedly
refer to the first four images, (a)–(d), as results of visual abstraction, and
many would contentedly accept a suggestion that (e) and (f) are also results
of visual abstraction, but some would be hesitant to consider (g) and (h) as
such abstracted representations. Nevertheless, one can also argue that the
latter four images, (e)–(h), are also results of visual abstraction because,
in comparison with the source data, some information has been abstracted away
and, in comparison with statistical abstraction of the source data, the
information presented is visual.

First, the level of willingness for people to consider a visualization image
as an abstracted visual representation does not appear to be related to the
quality of the image or the usefulness of the technique that generates the
image. Second, we can observe that both Figure 2(a) and Figure 2(h) feature
some deformation, and deformation does not seem to be a critical factor that
influences the perception of visual abstraction results. Similarly, from a
comparison of (b) vs. (h) and (c) vs. (g) we can observe that the types of
data to be visualized do not have decisive influence upon the perception of
the term “visual abstraction”. Third, we can also observe that an impression
of photorealism or just a perceived intention seems to bring about the
hesitation in characterizing a visualization image as the result of visual
abstraction. Meanwhile, having no or less photo-realistic effect in an image
(e.g., (e) or (f)) does not immediately imply visual abstraction either, at
least to some people. Here the adjective “photo-realistic” indicates that the
rendering algorithm used was designed to achieve a photo-realistic effect,
without implying that the image resulting from the rendering process actually
resembles a photograph.

One hypothesis is that our willingness or hesitation to consider a
visualization image as resulting from visual abstraction relates to an
unconsciously-integrated reasoning about two conditions of visual abstraction.

  1. A.

A visual abstraction is a transformation from data to its visual
representation with some information loss—Here data can be of any data types
including visual data (e.g., image corpora and videos). This can be considered
as a broad definition of visual abstraction, and encapsulates the
aforementioned definitions in cartography and shape analysis. While
introducing a constraint of visual output, it exhibits a parallel with the
definitions in relation to mathematical abstraction, abstract thinking, and
grouping in object-oriented design. All eight images in Figure 2 satisfy this
condition in general. We will discuss information loss in detail later.

  2. B.

A visual abstraction is a transformation from a more photo-realistic visual
representation to a less photo-realistic one—This can be considered as a
narrow definition of visual abstraction, and encapsulates the aforementioned
definitions in art, cartography, and shape analysis. It applies to
transformations with visual input as well as visual output. Considering the
examples in Figure 2, for images (a), (b), (c), and (d), it is relatively easy
for one to imagine their photo-realistic counterparts. Although some of these
images can be generated directly from source data that may not be visual, a
subjective impression of a transformation that decreases photorealism is
sufficient for viewers to associate these images with abstraction. Meanwhile,
it is harder to imagine a photo-realistic version of (e) or (f), and therefore
this condition does not appear to be applicable to them. For images (g) and
(h), it is intuitive to consider them more photo-realistic than less. They not
only fail to satisfy, but also negate, this condition.

We can easily see that reading data using a spreadsheet or reading their
statistical summary do not meet either condition. Images (a), (b), (c), and
(d) in Figure 2 satisfy both conditions. Images (e) and (f) satisfy condition
A but not B. Images (g) and (h) satisfy condition A but negate B. Suppose that
we had a numerical score 2 for condition A, score 1 for condition B, score 0
for not applicable, and score $-1$ for negation. Spreadsheet or statistical
summary would score 0; (g) and (h) would score 1; (e) and (f) would score 2;
(a), (b), (c), and (d) would score 3. Such a scoring system would reflect the
level of willingness for one to characterize a visualization image as the
result of visual abstraction.

We can also infer that condition A is more essential than condition B. Without
A, images (e) and (f) would not be considered as results of visual abstraction
at all. Without B, there would not be any hesitation about whether images (g)
and (h) are results of visual abstractions.

However, condition A does not in itself meet the expectation for the minimal
quality that the process of meaningful visual abstraction should possess,
since arbitrarily throwing away information should not be referred to as
meaningful abstraction. Below we use several information-theoretic measures to
clarify Condition A.

Let $P_{d\rightarrow v}$ be a process for transforming a dataset $d$ to a
visualization image $v$. Let $\mathbb{D}$ be the data space containing all
possible datasets that $P_{d\rightarrow v}$ can take as its input, and
$\mathbb{V}$ be the data space containing all possible visualization images
that $P_{d\rightarrow v}$ can generate. In information theory, $\mathbb{D}$
and $\mathbb{V}$ are referred to as _alphabets_. The dataset $d$ is thus a
letter in the input alphabet $\mathbb{D}$, and the visualization image $v$ is
a letter of the output alphabet $\mathbb{V}$. The process $P_{d\rightarrow v}$
can thus be written as $P_{d\rightarrow
v}:\mathbb{D}\longrightarrow\mathbb{V}$.

The Shannon entropy measures the amount of uncertainty or variation of an
alphabet. Let $p(d)$ be the probability of a dataset $d$ in the context of an
application. The Shannon entropy of $\mathbb{D}$ is thus defined as:

$\mathcal{H}(\mathbb{D})=-\sum_{d\in\mathbb{D}}p(d)\log_{2}p(d)$

When all letters in $\mathbb{D}$ have the same probability, we have
$\mathcal{H}(\mathbb{D})=\log_{2}\|\mathbb{D}\|$, where $\|\mathbb{D}\|$ is
the number of different letters in $\mathbb{D}$. Similarly, we can measure the
Shannon entropy $\mathcal{H}(\mathbb{V})$ as the amount of uncertainty or
variation of $\mathbb{V}$.

_Alphabet Compression_ [26] is the difference
$\mathcal{H}(\mathbb{D})-\mathcal{H}(\mathbb{V})$, which is a coarse
indication of the amount of information loss of the visualization process
$P_{d\rightarrow v}$. Consider a simple example. $\mathbb{D}$ is defined by a
real variable, $X$, which may take valid values between 0.00 and 10,000.00 at
two decimal point precision. There are thus 1,000,001 possible values. Let all
values have the same probability. We have $\mathcal{H}(\mathbb{D})\approx 20$
bits.

Meanwhile, we consider a process $P_{d\rightarrow v}$ that plots a value
$d\in\mathbb{D}$ as a bar in a single-variable bar chart using a canvas with
1000H $\times$ 100W pixels. The maximum resolution available for the mapping
function $P_{d\rightarrow v}:\mathbb{D}\longrightarrow\mathbb{V}$ is 1000
pixels, thus 1001 bar charts with different bar heights. We have
$\mathcal{H}(\mathbb{V})\approx 10$ bits. The alphabet compression is
therefore about 10 bits. In terms of Condition A, there is about 10 bits of
information loss. Therefore, any visualization process, which features many-
to-one mapping from data to visual objects, typically exhibits positive
alphabet compression. Only when the variation of $\mathbb{D}$ is very small,
e.g., using the above canvas to plot an integer variable in the range of [0,
100], the amount of alphabet compression can be zero. In the worst scenario,
the plotting function randomly depicts a bar with a height between 0 and 1000
pixels, the amount of alphabet compression would be negative.

All images in Figure 2 feature many-to-one mappings. For example, the
distortion in Figure 2(a) is a kind of many-to-one mapping, since many
potential track layouts would lead to the same metro map. In the image
rendered with a pen-and-ink effect in Figure 2(b), each white pixel could be a
placeholder for many differently colored pixels that have been abstracted
away. Each glyph in Figure 2(d) is a very low resolution visual representation
of some 20 values, most of which are real numbers. In the volume-rendered
image in Figure 2(h), each pixel results from a rendering integral that
transforms a few hundred voxel values to an RGB trio. Many different
combinations of these voxel values could result in pixels with the same color.

Hence, a process for generating visualization images from relatively complex
datasets features many-to-one mappings, which means information loss or
positive alphabet compression. According Condition A, such a process is thus a
process of visual abstraction.

However, what quantifies a visualization or a meaningful visual abstraction
must be a process that is intended to generate “meaningful” visualization
images from input datasets. The word “meaningful” implies three factors: (i)
the viewer can interpret what is being depicted; (ii) the viewer’s
interpretation of what is depicted is reasonably correct in relation to the
original data; and (iii) the viewer’s interpretation errors due to information
loss do not have serious impact on the viewer’s task.

Figure 3: The effectiveness of a visual abstraction process depends on the
succeeding task process as well as the viewer’s knowledge, biases, and
cognitive capability.

Consider that a viewer’s interpretation is a process $Q_{d\leftarrow
v}=P^{-1}_{d\rightarrow v}$ that attempts to reconstruct a dataset from a
given visualization image. This process can be written as $Q_{d\leftarrow
v}:\mathbb{V}\longrightarrow\mathbb{D}^{\prime}$. We use $\mathbb{D}^{\prime}$
to denote an alphabet that has the same set of letters as $\mathbb{D}$ but a
different probability mass function from that of $\mathbb{D}$. For example,
given a bar that is 499 pixels tall, a viewer may interpret it as one of these
values in the original $\mathbb{D}$, $\\{498.00,498.01,...,499.99,500.00\\}$.
Imagine that the interpretation is biased towards 500.00 due to the
corresponding mark on the vertical axis. The probability $q(500)$ would be
undesirably higher than the original probability $p(500)$.

In information theory, such errors in the interpretation can be collectively
measured by the Kullback-Leibler divergence, which is defined as:

$\mathcal{D}_{KL}(\mathbb{D}^{\prime}\|\mathbb{D})=\sum_{d\in\mathbb{D}}q(d)\log_{2}\frac{q(d)}{p(d)}$

where $p()$ and $q()$ are the probability mass functions of $\mathbb{D}$ and
$\mathbb{D}^{\prime}$ respectively, and $q(d)/p(d)$ is a discrete
representation of the Radon-Nikodym derivative of $q$ with respect to the
original $p$.

In the context of visual abstraction, this measurement offers a counterbalance
to the measurement alphabet compression. It is referred to as _Potential
Distortion_ [26]. While it is desirable to have the results of visual mapping
$P_{d\rightarrow v}$ as abstract as possible, i.e., for $P_{d\rightarrow v}$
to have a high amount of alphabet compression, it is also necessary to keep
the inaccuracy of the interpretation function $Q_{d\leftarrow v}$ as low as
possible, i.e., for $Q_{d\leftarrow v}$ to have a low amount of potential
distortion.

Since $Q_{d\leftarrow v}$ is a human-centric process, $Q_{d\leftarrow v}$ may
feature inaccuracy due to perceptual errors and cognitive biases. However,
$Q_{d\leftarrow v}$ can also make use of human knowledge that is not encoded
in the data to help more accurate reconstruction. For example, imagine that a
viewer is asked to guess what would be the original colors on the patch of
white pixels between two black lines in the pen-and-ink visualization image in
Figure 2(b). A naïve guess would be either white (as what is seen) or an
arbitrary selection from various grey colors. Most viewers, especially those
familiar with the depicted object or volume visualization methods, can do much
better than the naïve guess. Hence the process of “knowledge-assisted
guessing”—a heuristic process—has a lower amount of potential distortion than
the naïve guessing. In general, it is this human knowledge that enables visual
abstraction to be deployed effectively in many situations, such as those
illustrated in Figure 2. Whether users have the adequate ability to interpret
the results of visual abstraction is thus one of the key criteria for judging
if a visual abstraction process is appropriate or its results are meaningful,
which reflecting the two factors (i) and (ii) described above.

Nevertheless, since $P_{d\rightarrow v}$ is usually a many-to-one mapping, and
$Q_{d\leftarrow v}$ is usually a one-to-many mapping, one may wonder why we
should go through such “unnecessary fuss” to apply the process
$P_{d\rightarrow v}$ first to $\mathbb{D}$ and another process $Q_{d\leftarrow
v}$ to reconstruct $\mathbb{D}^{\prime}$. One important rationale is about the
_task_ succeeding $P_{d\rightarrow v}$ and $Q_{d\leftarrow v}$. The judgment
about whether a visual abstraction process is appropriate or its results are
meaningful thus depends on another process $P_{v\rightarrow t}$. As
illustrated in Figure 3, $P_{v\rightarrow t}$ takes $\mathbb{V}$ as the input,
and generates another output alphabet $\mathbb{T}$ that may consist of a
collection of letters, e.g., different options of a decision, different levels
of an assessment, different categories of a situation, etc.

The process of visual abstraction $P_{d\rightarrow v}$ and the reconstructive
interpretation $Q_{d\leftarrow v}$ can collectively affect the task process
$P_{v\rightarrow t}$, especially its _Cost_ $\mathbf{Ct}(P_{v\rightarrow
t},Q_{v\leftarrow t})$. Similar to $Q_{d\leftarrow v}$, here $Q_{v\leftarrow
t}$ is an interpretation process for reconstructing $\mathbb{V}$ from
$\mathbb{T}$. For a univariate value (e.g., 499.38), there is little merit to
visualize it using a bar chart. The difference of the cost for reading the
number and that of viewing a bar is negligible for most tasks. The potential
distortion caused by visual abstraction can only affect the process
$P_{v\rightarrow t}$ negatively. However, if the number of variables
increases, e.g., 10 variables, the cognitive load for viewing and comparing 10
numbers is likely to be higher than viewing and comparing 10 bars using a bar
chart. It is not difficult to imagine the merits of visualization when the
number of variables increases. For the volume datasets featured in Figure
2(b,h), the number of variables in a dataset is typically at the scale of
$256\times 256\times 256$ or more. It is inconceivable to perform a decision
task by reading the numerical values of such a volume dataset. Hence, visual
abstraction can be used to transform a volume dataset with a huge number of
variables to visualization images as shown in (b) and (h), which reduces the
cost $\mathbf{Ct}(P_{v\rightarrow t},Q_{v\leftarrow t})$ significantly.

The above information-theoretic discourse on visual abstraction is based on
the cost-benefit metric for data intelligence proposed by Chen and Golan [26].
For any data intelligence process $P_{i}$ with an input alphabet
$\mathbb{Z}_{i}$ and an output alphabet $\mathbb{Z}_{i+1}$, its cost-benefit
ratio is defined as:

$\displaystyle\frac{\textit{Benefit}}{\textit{Cost}}=$
$\displaystyle\frac{\textit{Alphabet Compression}-\textit{Potential
Distortion}}{\textit{Cost}}$ $\displaystyle=$
$\displaystyle\frac{\mathbf{AC}(P_{i})-\mathbf{PD}(Q_{i})}{\mathbf{Ct}(P_{i},Q_{i})}=\frac{\mathcal{H}(\mathbb{Z}_{i})-\mathcal{H}(\mathbb{Z}_{i+1})-\mathcal{D}_{KL}(\mathbb{Z}^{\prime}_{i}\|\mathbb{Z}_{i})}{\mathbf{Ct}(P_{i},Q_{i})}$

When this metric is applied to the two processes in Figure 3, we have the
combined cost-benefit ratio as:

$\displaystyle\frac{\textit{Benefit}}{\textit{Cost}}$
$\displaystyle\bigl{(}d\rightarrow v\rightarrow
t\bigr{)}=\frac{\mathbf{AC}(P_{d\rightarrow v})-\mathbf{PD}(Q_{d\leftarrow
v})+\mathbf{AC}(P_{v\rightarrow t})-\mathbf{PD}(Q_{v\leftarrow
t})}{\mathbf{Ct}(P_{d\rightarrow v},Q_{d\leftarrow
v})+\mathbf{Ct}(P_{v\rightarrow t},Q_{v\leftarrow t})}$ (1) $\displaystyle=$
$\displaystyle\frac{\mathcal{H}(\mathbb{D})-\mathcal{H}(\mathbb{V})-\mathcal{D}_{KL}(\mathbb{D}^{\prime}\|\mathbb{D})+\mathcal{H}(\mathbb{V})-\mathcal{H}(\mathbb{T})-\mathcal{D}_{KL}(\mathbb{V}^{\prime}\|\mathbb{V})}{\mathbf{Ct}(P_{d\rightarrow
v},Q_{d\leftarrow v})+\mathbf{Ct}(P_{v\rightarrow t},Q_{v\leftarrow t})}$
$\displaystyle=$
$\displaystyle\frac{\mathcal{H}(\mathbb{D})-\mathcal{H}(\mathbb{T})-\mathcal{D}_{KL}(\mathbb{D}^{\prime}\|\mathbb{D})-\mathcal{D}_{KL}(\mathbb{V}^{\prime}\|\mathbb{V})}{\mathbf{Ct}(P_{d\rightarrow
v},Q_{d\leftarrow v})+\mathbf{Ct}(P_{v\rightarrow t},Q_{v\leftarrow t})}$

In comparison, if one has to perform the task by reading the data without
visualization, the cost-beneficial ratio would be:

$\displaystyle\frac{\textit{Benefit}}{\textit{Cost}}\bigl{(}d\rightarrow
t\bigr{)}=$ $\displaystyle\frac{\mathbf{AC}(P_{d\rightarrow
t})-\mathbf{PD}(Q_{d\leftarrow t})}{\mathbf{Ct}(P_{d\rightarrow
t},Q_{d\leftarrow t})}$ (2) $\displaystyle=$
$\displaystyle\frac{\mathcal{H}(\mathbb{D})-\mathcal{H}(\mathbb{T})-\mathcal{D}_{KL}(\mathbb{D}^{\prime\prime}\|\mathbb{D})}{\mathbf{Ct}(P_{d\rightarrow
t},Q_{d\leftarrow t})}$

Note that the term $\mathcal{D}_{KL}(\mathbb{D}^{\prime}\|\mathbb{D})$ in Eq.
(1) and the term $\mathcal{D}_{KL}(\mathbb{D}^{\prime\prime}\|\mathbb{D})$ in
Eq. (2) are of different quantities as they relate to $Q_{d\leftarrow v}$ and
$Q_{d\leftarrow t}$ respectively.

When the dataset $d$ is large and complex, we can see that the cost
$\mathbf{Ct}(P_{d\rightarrow t},Q_{d\leftarrow t})$ in Eq. (2) would be much
higher than the combined costs in Eq. (1) in terms of time and cognitive load
in performing the task. In other words, we have:

$\mathbf{Ct}(P_{d\rightarrow t},Q_{d\leftarrow t})>\mathbf{Ct}(P_{d\rightarrow
v},Q_{d\leftarrow v})+\mathbf{Ct}(P_{v\rightarrow t},Q_{v\leftarrow t})$

Although reading data might appear to be more accurate, the reconstruction
process $Q_{d\leftarrow t}$ from the task alphabet $\mathbb{T}$ (e.g., the
patient has a tumor or not) to the data alphabet $\mathbb{D}$ (e.g., a volume
dataset) is much more error-prone than the reconstruction process via
visualization. In other words, we have:

$\mathbf{PD}(Q_{d\leftarrow t})>\mathbf{PD}(Q_{d\leftarrow
v})+\mathbf{PD}(Q_{v\leftarrow t})$

With $\mathbf{AC}(P_{d\rightarrow v})+\mathbf{AC}(P_{v\rightarrow
t})=\mathbf{AC}(P_{d\rightarrow t})$, it is not difficult to conclude:

$\frac{\textit{Benefit}}{\textit{Cost}}\bigl{(}d\rightarrow v\rightarrow
t\bigr{)}>\frac{\textit{Benefit}}{\textit{Cost}}\bigl{(}d\rightarrow
t\bigr{)}$

Under Condition A, we can thus mathematically reason that, for any slightly
large or complex dataset, the process from data alphabet $\mathbb{D}$ to task
alphabet $\mathbb{T}$ with visual abstraction is usually more cost-beneficial
than the process without.

For some very simple datasets, such as a univariate value, visual abstraction
may not have an information-theoretic merit. However, this is not to say that
it could not have cognitive merit in disseminative visualization. More likely,
the results of visual abstraction could attract more attention from the
viewers who unconsciously devote more cognitive load to the task. Although the
viewers’ cost-beneficial ratio increases, the presenter of the disseminative
visualization benefits from the contribution of extra cognitive load from the
viewers. In many ways, this is similar to scenarios of disseminative
visualization, where the amount of visual abstraction is purposely reduced in
order to attract viewers’ attention, and hence their cognitive load. Such
scenarios may include, for instance, showing an animated chart, whilst a
static chart could adequately convey the information, or showing visualization
in theatre-based virtual environments [27].

Similarly, we can also use the cost-benefit metric to analyze the scenarios
under condition B by comparing the cost-benefit ratio of a more photo-
realistic technique with a less photo-realistic technique. Similar to
Condition A, the potential distortion is affected by viewers’ knowledge as
well as their biases. The cost is affected by the viewer’s task as well as
cognitive capability.

Furthermore, this metric can be applied to human-centric processes (e.g.,
visualization and interaction) as well as machine-centric processes (e.g.,
statistics and algorithms). In general, statistical abstraction and
algorithmic abstraction usually result in more alphabet compression as well as
more potential distortion but less cost than visual abstraction. In designing
a visual analytics workflow, the metric can be used to compare the cost-
benefit of a human-centric process with that of a machine-centric process by
analyzing the trade-off among alphabet compression, potential distortion, and
cost. The metric can also be used to guide a visualization designer in
choosing different forms of visual abstraction, e.g., in reasoning about the
trade-off among the amount of abstraction, the potential perceptual errors,
and the cost of task performance.

In summary, as defined at the beginning of this article, meaningful visual
abstraction depends on some points of view and some tasks. From the
perspective of information theory, the points of view may be in either or both
of the following forms:

  * •

The factors that influence the _alphabet compression_ and _cost_ of the
process $P_{d\rightarrow v}$ for transforming data to visualization. These
factors may include the designers’ wish to keep or highlight some information
while removing or deemphasizing other information, their understanding of the
task requirements, their appreciation of the resources available for
visualization, and their awareness of the viewers’ knowledge of visual
representations and skills of visual analysis.

  * •

The factors that influence the _potential distortion_ and the _cost_ of the
process $Q_{d\leftarrow v}$ for reconstructing data from visualization. These
factors may include the viewers’ knowledge related to the data being depicted
and the visual representations used, their understanding about the information
required for performing their tasks, and their cognitive load and time
constraint in executing the process $Q_{d\leftarrow v}$.

Meanwhile, tasks can be defined as processes that succeed the processes
$P_{d\rightarrow v}$ and $Q_{d\leftarrow v}$. As long as the tasks fall
broadly in the category of data intelligence tasks, the cost-benefit metric
proposed by Chen and Golan [26] can also be applied to these succeeding
processes. Therefore, from the information-theoretic perspective, the most
meaningful visual abstraction, or the most effective visualization in general,
is the process with the optimal cost-benefit measure.

##  6 Summary

In this article we thus formally defined the concepts of abstraction and
visual abstraction as they relate to the field of visualization and based on
existing notions of the terms in related fields such as the arts and in
philosophy. We argued that any visual representation is the result of multiple
abstraction steps from reality, and we called the step from data
representation to visual representation visual abstraction. We also showed
that as users of a visualization system we do not observe this abstraction
process but instead adjust settings to transition from one visual
representation to another—each being an independent result of the abstraction
process from source thing to sign thing. Yet as designers of visualization
systems we can provide guided interaction such that several results of
meaningful abstractions can be assembled into sequences that we call
abstraction axes to better illustrate how different aspects of reality relate
to each other, and several of these abstraction axes can be assembled into
abstraction spaces to illustrate the interrelation of several independent
aspects. So while we argue that any visual representation is the result of an
abstraction process, it is still important to discuss abstraction and visual
abstraction as it teaches us about visualization as a process in general.

## Acknowledgments

The authors would like to thank Jos Roerdink, Helwig Hauser, Stefan Bruckner,
Hans-Christian Hege, and Torsten Möller for fruitful discussion that helped
shaping the article. Thanks to Peter Mindek for illustrating the abstraction
space in Figure 1. This work was funded through the ILLUSTRARE grant by both
the Austrian Science Fund (FWF; I 2953-N31) and the French National Research
Agency (ANR; ANR-16-CE91-0011-01).

## References

  * [1] Ivan Viola and Tobias Isenberg.  Pondering the concept of abstraction in (illustrative) visualization.  IEEE Transactions on Visualization and Computer Graphics, 24(9):2573–2588, September 2018.
  * [2] Mauri Leppänen.  Towards an abstraction ontology.  In Marie Duží, Hannu Jaakkola, Yasushi Kiyoki, and Hannu Kangassalo, editors, Information Modelling and Knowledge Bases XVIII, volume 154 of Frontiers in Artificial Intelligence and Applications, pages 166–185. IOS Press, Amsterdam, 2007.
  * [3] Gottlob Frege.  Der Gedanke [The thought].  Beiträge zur Philosophie des deutschen Idealismus, 1(2):58–77, 1918.  English translation in Mind: A quarterly Review of Psychology and Philosophy 65(259): 289–311, July 1956.
  * [4] Stanford encyclopedia of philosophy.  Web site.
  * [5] David K. Lewis.  On the Plurality of Worlds.  Blackwell, Oxford, 1986.
  * [6] Rudolph Arnheim.  Visual Thinking.  University of California Press, Berkeley, Los Angeles, London, 1969.
  * [7] Barbara P. Buttenfield and Robert B. McMaster, editors.  Map Generalization: Making Rules for Knowledge Representation.  Longman Scientific & Technical, Harlow, Essex, UK, 1991.
  * [8] Nicu D. Cornea, Deborah Silver, Xiaosong Yuan, and Raman Balasubramanian.  Computing hierarchical curve-skeletons of 3D objects.  The Visual Computer, 21(11):945–955, 2005.
  * [9] Tobias Isenberg.  Capturing the Essence of Shape of Polygonal Meshes.  PhD thesis, Otto-von-Guericke University of Magdeburg, Germany, 2004.
  * [10] Mosby.  Mosby’s Medical Dictionary.  Elsevier, 9th edition, 2012.
  * [11] Charles K. Ogden and Ivor A. Richards.  The Meaning of Meaning: A Study of the Influence of Language upon Thought and of the Science of Symbolism.  Kegan Paul, London, 1923.
  * [12] Tamara Munzner.  Visualization Analysis and Design.  A K Peters Visualization Series. CRC Press, Boca Raton, FL, USA, 2014\.
  * [13] Peter Thompson.  Margaret Thatcher: A new illusion.  Perception, 9(4):483–484, August 1980.
  * [14] Herman Chernoff.  The use of faces to represent points in k-dimensional space graphically.  Journal of the American Statistical Association, 68(342):361–368, 1973.
  * [15] Haichao Miao, Elisa De Llano, Tobias Isenberg, M. Eduard Gröller, Ivan Barišić, and Ivan Viola.  DimSUM: Dimension and scale unifying maps for visual abstraction of DNA origami structures.  Computer Graphics Forum, 37(3):403–413, June 2018.
  * [16] Haichao Miao, Elisa De Llano, Johannes Sorger, Yasaman Ahmadi, Tadija Kekic, Tobias Isenberg, M. Eduard Gröller, Ivan Barišić, and Ivan Viola.  Multiscale visualization and scale-adaptive modification of DNA nanostructures.  IEEE Transactions on Visualization and Computer Graphics, 24(1):1014–1024, January 2018.
  * [17] Haneen Mohammed, Ali K. Al-Awami, Johanna Beyer, Corrado Cali, Pierre Magistretti, Hanspeter Pfister, and Markus Hadwiger.  Abstractocyte: A visual tool for exploring nanoscale astroglial cells.  IEEE Transactions on Visualization and Computer Graphics, 24(1):853–861, January 2018.
  * [18] Matthew van der Zwan, Wouter Lueks, Henk Bekker, and Tobias Isenberg.  Illustrative molecular visualization with continuous abstraction.  Computer Graphics Forum, 30(3):683–690, May 2011.
  * [19] Transport for London.  Tube map, 2018.
  * [20] Stefan Bruckner and M. Eduard Gröller.  Style transfer functions for illustrative volume rendering.  Computer Graphics Forum, 26(3):715–724, September 2007.
  * [21] T. Oster, C. Rössl, and H. Theisel.  Core lines in 3D second-order tensor fields.  Computer Graphics Forum, 37(3):327–337, 2018.
  * [22] B. Duffy, J. Thiyagalingam, S. Walton, D. J. Smith, A. Trefethen, J. C. Kirkman-Brown, E. A. Gaffney, and M. Chen.  Glyph-based video visualization for semen analysis.  IEEE Transactions on Visualization and Computer Graphics, 21(8):980–993, 2015.
  * [23] Y. Wang, C. Shi H. Haleem, T. Wu, X. Zhao, S. Fu, and H. Qu.  Towards easy comparison of local businesses using online reviews.  Computer Graphics Forum, 37(3):63–74, 2018.
  * [24] X. Zhai, F. Hou, H. Qin, and A. Hao.  Inverse modelling of incompressible gas flow in subspace.  Computer Graphics Forum, 36(6):100–111, 2017.
  * [25] Åsmund Birkeland and Ivan Viola.  View-dependent peel-away visualization for volumetric data.  In Proc. SCCG, pages 121–128, New York, 2009. ACM.
  * [26] M. Chen and A. Golan.  What may visualization processes optimize?  IEEE Transactions on Visualization and Computer Graphics, 22(12):2619–2632, December 2016.
  * [27] M. Chen, K. Gaither, N. W. John, and B. McCann.  An information-theoretic approach to the cost-benefit analysis of visualization in virtual environments.  IEEE Transactions on Visualization and Computer Graphics, 25(1):32–42, January 2019.