# Multi-times Monte Carlo Rendering for Inter-reflection Reconstruction

Tengjie Zhu
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
&Zhuo Chen
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
&Jingnan Gao
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
&Yichao Yan
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
&Xiaokang Yang
Shanghai Jiao Tong University
<EMAIL_ADDRESS>

###### Abstract

Inverse rendering methods have achieved remarkable performance in
reconstructing high-fidelity 3D objects with disentangled geometries,
materials, and environmental light. However, they still face huge challenges
in reflective surface reconstruction. Although recent methods model the light
trace to learn specularity, the ignorance of indirect illumination makes it
hard to handle inter-reflections among multiple smooth objects. In this work,
we propose Ref-MC2 that introduces the multi-times Monte Carlo sampling which
comprehensively computes the environmental illumination and meanwhile
considers the reflective light from object surfaces. To address the
computation challenge as the times of Monte Carlo sampling grow, we propose a
specularity-adaptive sampling strategy, significantly reducing the
computational complexity. Besides the computational resource, higher geometry
accuracy is also required because geometric errors accumulate multiple times.
Therefore, we further introduce a reflection-aware surface model to initialize
the geometry and refine it during inverse rendering. We construct a
challenging dataset containing scenes with multiple objects and inter-
reflections. Experiments show that our method outperforms other inverse
rendering methods on various object groups. We also show downstream
applications, e.g., relighting and material editing, to illustrate the
disentanglement ability of our method. Our project page:
https://zhutengjie.github.io/Ref-MC2/

##  1 Introduction

Neural Radiance Fields (NeRF) [25] and 3DGS [16] have demonstrated their
excellent performance on novel view synthesis. However, it is difficult to
directly apply their reconstructed 3D model to the current industrial
pipeline, leading to the lack of flexibility in many downstream applications,
e.g., relighting and material editing. To better cooperate with mature
techniques, inverse rendering bases the physical rendering [23] and utilizes
the neural network to learn disentangled materials that can be seamlessly
plunged into the industrial pipeline for further manipulation.

Previous methods [27, 9, 43, 54] have explored how to disentangle geometry,
diffuse, roughness, metalness, and environmental light from multi-view images,
but they still face challenges in shading and reflective objects.
Specifically, Nvdiffrec [27] proposes a differentiable pipeline that enables
gradient-based optimization on both meshes and volumetric textures. These
methods ignore the shadow when modeling the illumination, resulting in failed
disentanglement for diffuse and shading appearances. To model more realistic
shadings, Nvdiffrecmc [9] further incorporates ray tracing and Monte Carlo
sampling [26] into inverse rendering, significantly improving the
decomposition of shape, materials, and lighting. However, it ignores the
indirect illuminations during path tracing and treats rays attacking object
surfaces as sheltered illumination. As a consequence, methods like Nvdiffrecmc
increase the ambiguity of geometry reconstruction and undermine material
learning. Unfortunately, the scene with multiple inter-reflections is common
in the real world. The failure in these scenes hinders the wider applications
of these methods. Recent methods [17, 38] take inter-reflections into account
and can reconstruct the reflective objects well. However, the implicit
representations of these methods for materials and renders sacrifice their
scalability to downstream tasks. Recently, Nefii [43] further incorporated the
implicit neural radiance to estimate ray tracing, alleviating the ambiguity
between materials and indirect illuminations. However, this leads to a huge
computational consumption for path tracing.

In this paper, we proposed a full inverse rendering method, Ref-MC2, which
considers inter-reflections during ray tracing to improve the decomposition of
explicit materials and environmental lighting. The core of our method is to
use Multi-times Monte Carlo integration [26] and BRDF [23] rendering to
approximate the indirect illumination at multiple reflection points along the
light propagation path. Although our method takes advantage of hardware-
accelerated ray tracing to model the indirect illumination, we still face two
major challenges brought by the multi-times Monte Carlo sampling. 1)
Efficiency: the explosive computational growth from multiple times is too
heavy for the hardware algorithm only. 2) Geometry: the geometry quality
greatly affects the calculation of indirect light and the decomposition of the
materials, because the error will accumulate over and over again as the times
of Monte Carlo sampling increase.

To address these challenges, we correspondingly propose two strategies. 1) For
efficiency, we flatten the multi-times sampling into sequential single-time
sampling. In a Lambert model [30], the diffuse light is independent of the
direction of reflection and thus can be pre-computed as a map to query at any
time. When we trace the reflected light from an object, instead of
recalculating the diffuse component, we can take the value directly from the
diffuse map. Besides, we also limit the specular component to a small lobe
along with the reflective direction. 2) For geometry, we refer to the SDF-
based methods [39, 17, 6] and replace the common positional encoding with the
Sphere Gaussian encoding to get an accurate initial geometry for reflective
objects. We use this geometry to initialize Flxicubes [33] that optimizes
surface meshes based on the gradient, and further fine-tunes the Flexicubes in
a differentiable pipeline. To evaluate our framework, we construct a dataset
containing difficult scenes which contain strong inter-reflections between
multiple surfaces. Extensive experiments demonstrate that our framework can
successfully decompose indirect illumination and materials. In summary, our
contributions are:

  * •

We propose a full inverse rendering method that employs multi-times Monte
Carlo sampling to correctly decompose indirect illumination and materials.

  * •

We reduce the computational consumption when tracing indirect illumination by
pre-computing the diffuse map based on the Lambert model.

  * •

We refine the SDF-based architecture with Spherical Gaussian encoding to
obtain a high-quality initial geometry which further releases the accumulated
error during multi-times sampling.

  * •

We construct a dataset to evaluate the performance on indirect illumination.

##  2 Related Work

###  2.1 Implicit Neural Representations

Neural implicit representations [22, 25, 31, 52, 42, 32, 10, 3, 4, 55, 34, 7]
have achieved impressive success in many computer vision and computer graphics
tasks. These methods use neural radiance fields to capture color and volume
density, generating photo-realistic novel views through volume rendering [15].
However, the unconstrained volumetric representation of the original NeRF
method leads to low-quality geometry. Recent 3D Gaussian Splatting (3DGS) [16,
11, 12, 20, 50] has gained popularity in novel view synthesis. Different from
NeRF, it is an explicit representation that involves the optimization of
multiple Gaussians to reconstruct 3D objects. 3DGS learns color and density in
a volumetric point cloud, but it also fails to produce accurate geometry due
to its discrete representation. Besides, these methods are all entangled
learning, integrating all inherent materials and the environment map into the
appearance. This limits the downstream applications in the current industrial
pipeline. To produce a high-quality geometry, follow-up works [46, 41, 48, 29]
use a function to associate signed distance field (SDF) and volume density.
The surface mesh can be extracted from neural implicit surfaces by Marching
Cubes [19], and this 3D asset can be further applied in other applications.
However, they usually perform badly in reconstructing the reflective objects
due to the ambiguity of reflective appearance.

###  2.2 Neural Inverse Rendering

Although neural implicit surfaces have achieved impressive performance in
geometry reconstructing and novel views synthesis, they do not obtain the
fundamental materials of PBR which limits their flexibility in downstream
tasks. Neural inverse rendering methods [17, 44, 8, 2, 51, 54, 13, 36, 21]
introduce the physical rendering equation to estimate the disentangled diffuse
and specular component from RGB images. They approximate the rendering
equation based on neural networks or basis functions, e.g., Spherical
Gaussians [40, 45, 47, 51] and Spherical Harmonics [5, 1, 35, 49]. Nvdiffrec
[27] introduces the full inverse rendering that estimates shape, materials,
and environmental light into gradient-based optimization. However, it does not
consider the shadows, leading to the entanglement of materials. Recent works
[6, 39] extend SDF-based architectures with an additional appearance branch to
model the reflections on the object. RefNeuS [6] introduces a
reparametrization method to distinguish the reflective appearance from the
diffuse appearance by a direction-relative process, but it fails to faithfully
reconstruct non-reflective objects. UniSDF [39] proposes to use a weight-MLP
to balance the reflective and non-reflective branches for different objects.
However, these methods still face challenges in scenes with complex inter-
reflections. Nvdiffrecmc [9] extends Nvdiffrec with Monte Carlo sampling to
trace the light path but still ignores indirect illumination between objects.
Further methods [17, 18, 43] consider indirect illumination in their design.
ENVIDR [17] employs a neural renderer to learn the physical light interaction,
without explicitly formulating the rendering equation. NeRO [18] applies the
split-sum approximation to approximate the shading effects of both direct and
indirect lights. Nefii [43] introduces ray tracing to the radiance field to
model indirect illuminations. However, these methods do not fully disentangle
the materials from RGB images, but decompose the appearance into reflective
and diffuse color. It is hard to apply to the current industrial pipeline
directly. In contrast, our Ref-MC2 is a full inverse rendering method that
also considers indirect illumination.

##  3 Method

###  3.1 Preliminaries

The rendering equation [14] is commonly used to compute the outgoing radiance
$L_{o}$($\mathbf{p}$, $\bm{\omega}_{o}$) from the point $\mathbf{p}$ in
outgoing direction $\bm{\omega}_{o}$:

$L_{o}\left(\mathbf{p},\bm{\omega}_{o}\right)=\int_{\Omega}L_{i}\left(\mathbf{p},\bm{\omega}_{i}\right)f\left(\mathbf{p},\bm{\omega}_{i},\bm{\omega}_{o}\right)\left(\mathbf{n}\cdot\bm{\omega}_{i}\right)d\bm{\omega}_{i},$
(1)

where $L_{i}$($\mathbf{p}$, $\bm{\omega}_{i}$) is the incoming radiance into
$\mathbf{p}$ from the direction $\bm{\omega}_{i}$, $\mathbf{n}$ is the normal
of the point $\mathbf{p}$, $\Omega$ is the hemisphere of directions above
$\mathbf{p}$, $f(\mathbf{p},\bm{\omega}_{i},\bm{\omega}_{o})$ is the BSDF
evaluated for $\bm{\omega}_{i}$ and the current incoming direction
$\bm{\omega}_{i}$. A physics-based BSDF function, GGX [23] is proposed to
decompose the function into several physical terms. The GGX is described as:

$f\left(\mathbf{p},\bm{\omega}_{i},\bm{\omega}_{o}\right)=f_{d}+f_{s}=f_{d}+\frac{DFG}{4\left(\mathbf{n}\cdot\bm{\omega}_{i}\right)\left(\mathbf{n}\cdot\bm{\omega}_{o}\right)},$
(2)

where $f_{d}$ is the diffuse term and $f_{s}$ is the specular term. $D$, $F$,
and $G$ are the microfacet distribution function, the Fresnel reflection
coefficient, and the geometric attenuation, respectively.

Previous works [27] use the split sum function [24] to approximate the
rendering equation, but it inevitably omits the shadow and indirect
illumination. In contrast, Monte Carlo integration [26] is a simple but
unbiased estimation method that comprehensively considers all the physical
terms for the outgoing radiance. The Monte Carlo integration rendering
equation is:

$L_{o}\left(\mathbf{p},\bm{\omega}_{o}\right)\approx\frac{1}{N}\sum_{i=1}^{N}\frac{L_{i}\left(\mathbf{p},\bm{\omega}_{i}\right)f\left(\mathbf{p},\bm{\omega}_{i},\bm{\omega}_{o}\right)\left(\bm{\omega}_{i}\cdot\mathbf{n}\right)}{p\left(\bm{\omega}_{i}\right)},$
(3)

where $\bm{\omega}_{i}$ is the $i$th sample drawn from density $p$. As the
number of samples grows, the estimation variance reduces, but the computation
increases. Multiple importance sampling [37] (MIS) is proposed to inhibit the
computational consumption. When a function can be expressed as a
multiplication of $n$ functions, it draws $n_{i}$ samples $\bm{\omega}_{i,j}$
from $n$ sampling distributions $p_{i}$ in turn. The MIS Monte Carlo estimator
for the rendering equation is:

$L_{o}\left(\mathbf{p},\bm{\omega}_{o}\right)=\sum_{i=1}^{n}\frac{1}{n_{i}}\sum_{j=1}^{n_{i}}W_{i}\frac{F_{i}\left(\mathbf{p},\bm{\omega}_{o},\bm{\omega}_{i,j}\right)}{p_{i}},$
(4)

where $p_{i}$ $\propto$ the $i$-th multiplication function of the integrated
function of the rendering equation and $W_{i}$ is the the balance heuristic
weighting function.

Figure 1: We perform Monte Carlo sampling at the viewpoint. When the sampling
ray from the point is not blocked, it is the direct illumination from
environmental lighting. When the sampling ray hits an object, we divide this
indirect illumination from the object into diffuse light and specular light.
For diffuse light, we pre-compute the coarse diffuse color under one-time
sampling and store it as a diffuse map to query during the multi-times
sampling. For specular light, we only need to partially trace the rays in a
small specular lobe along the reflective direction. The gradients are backward
along the tracing path, and are passed to optimize $\bm{k}_{d},\bm{k}_{orm}$,
normals, and environment maps.

###  3.2 Multi-times Monte Carlo Sampling

As discussed in Sec. 3.1, the split sum approximation ignores the shadows and
the indirect illumination. Nvdiffrecmc notes this and accounts for shadows
using the Monte Carlo integration method. However, they give up continuously
tracing the light rays attacking the surface of objects and treat them as zero
illumination. It leads to bad performance in scenes with inter-reflections.
Therefore, we propose the Ref-MC2 method to trace the light rays continuously.
When taking into account indirect illumination, based on [14], the rendering
equation can be expressed in this version:

$L_{o}\left(\mathbf{p},\bm{\omega}_{o}\right)=\int_{\Omega}L_{i}\left(r\left(\mathbf{p},\bm{\omega}_{o}\right),-\bm{\omega}_{i}\right)f\left(\mathbf{p},\bm{\omega}_{i},\bm{\omega}_{o}\right)\left(\mathbf{n}\cdot\bm{\omega}_{i}\right)d\bm{\omega}_{i}.$
(5)

In this equation, $r\left(\mathbf{p},\bm{\omega}_{o}\right)$ represents the
location of a surface point on the object surface hit by a ray cast from
$\mathbf{p}$ in direction $\bm{\omega}_{i}$ for the first time. The
corresponding Monte Carlo integration that considers the indirect illumination
can then be expressed as:

$L_{o}\left(\mathbf{p},\bm{\omega}_{o}\right)\approx\frac{1}{N}\sum_{i=1}^{N}\frac{L_{i}\left(r\left(\mathbf{p},\bm{\omega}_{o}\right),-\bm{\omega}_{i}\right)f\left(\mathbf{p},\bm{\omega}_{i},\bm{\omega}_{o}\right)\left(\bm{\omega}_{i}\cdot\mathbf{n}\right)}{p\left(\bm{\omega}_{i}\right)}.$
(6)

It can be interpreted as that when the sampling ray is not blocked, it can be
seen as direct illumination from environmental lighting. When the surface
point of an object blocks the sampling ray, it needs to be continuously traced
from this surface point. This continuous tracing is an iterated process of
Monte Carlo integration. It is noteworthy that when continuous sampling at the
blocking surface point it is unnecessary to consider the indirect illumination
like calculating the outgoing radiance at initial point $\mathbf{p}$. This is
because the energy of light gradually degrades and the impact of light after
twice reflections is negligible compared to the explosively increased
computational load. However, the additional computational load of ray tracing
at a depth of two is still huge. To reduce our computational load, we further
propose to approximate the diffuse light and transfer the computations.

In GGX, the diffuse term $f_{d}$ is:

$f_{d}=\frac{c_{\text{diff}}}{\pi}\left(1+\left(F_{D90}-1\right)\left(1-\left(\mathbf{n}\cdot\bm{\omega}_{i}\right)\right)^{5}\right)\left(1+\left(F_{D90}-1\right)\left(1-\left(\mathbf{n}\cdot\bm{\omega}_{o}\right)\right)^{5}\right),$
(7)

$F_{D90}=0.5+2r\cos^{2}\theta_{d},$ (8)

where $c_{\text{diff}}$ is the diffuse albedo of the material. In this
equation, $f$ is related to the direction of the incoming radiance, which can
yield more realistic results. However, it brings significant computations and
limits the flexibility during Monte Carlo integration. Because the term
$f_{d}$ is related to the normal $n$ and the direction of rays ${\omega}_{o}$,
we cannot pre-compute the diffuse map before deep ray tracing. The following
term is to introduce the roughness for diffuse light to avoid too dark edges
in extremely low grazing angles. However, in our task, the common sampling
hardly faces such a problem, while this term limits the Flexible computation
in inverse rendering. To reduce the burden, according to [24] we can use the
Lambert diffuse to approximate it as:

$f_{d}=\frac{c_{\text{diff}}}{\pi}.$ (9)

In this approximation, $f_{d}$ is independent on the incoming direction
$\omega_{i}$ and the outgoing direction $\omega_{o}$. Based on this, the
diffuse part of outgoing light can be simplified as:

$L_{\text{diff}}\left(\mathbf{p}\right)=\frac{c_{\text{diff}}}{\pi}\int_{\Omega}L_{i}\left(\mathbf{p},\bm{\omega}_{i}\right)\left(\mathbf{n}\cdot\bm{\omega}_{i}\right)d\bm{\omega}_{i}.$
(10)

It no longer relates to the direction of the sampling ray, so we can pre-
compute it via an MLP $M$. Compared to the diffuse part, the specular part is
strongly related to the ray direction. It is a disadvantage that we cannot
approximate the specular part like the diffuse part, but it is also an
advantage that we only need to sample minor rays in a small specular lobe
along the reflective directions. Therefore, we significantly reduce the
computations of both the diffuse part and the specular part during the multi-
times Monte Carlo Sampling.

Figure 2: Differences between previous SDF architectures and the architecture
for inter-reflections.

###  3.3 Geometry Initialization

As seen in the equation 5, our indirect lighting highly depends on the
geometry. However, the strong ambiguity of reflections makes it hard to
directly learn high-quality disentangled shapes. Therefore, we flatten the
joint learning process of both geometry and materials into separate learning
of them and thus reduce the number of physical terms to be disentangled in
each learning stage.

We first utilize SDF-based architectures [46, 6, 18] to learn an initial
geometry. The two branches of the diffuse and reflective networks well
disambiguate the appearance with reflections and empower the SDF network to
produce a high-quality geometry, as shown in Fig. 2 (a). However, this
architecture encounters challenges of indirect illumination due to the
expressive capacity of Integrated Positional Encoding (IPE). It performs well
in general scenes but shows limitations in representing inter-reflections. Due
to the capacity of Sphere Gaussians (SG) to represent radiance directions, we
introduce SG encoding instead of IPE to enhance the expressive capacity of
reflective MLP, as shown in Fig. 2 (b). Besides, we directly parameterize
diffuse appearance as SG coefficients which is suitable for objects with
multiple reflective surfaces. The geometry comparison is also shown in Fig. 2.

The learned SDF can be then converted into a surface mesh using the Marching
Cubes [19], which supports further tuning in the following inverse rendering
pipeline. Different from Nvdiffrecmc which adopts DMTET for differentiable
mesh optimization, we introduce Flexicubes [33] into our inverse rendering
pipeline. Flexicubes use the SDF, weight, and the deformation for vertexes of
the grid cells to extract the surface mesh using the DMC [28]. The mesh is
converted into a differentiable representation that can be optimized based on
gradients.

###  3.4 Training Objectives

The main objective is to minimize the photometric loss between rendered images
and ground truth:

$\mathcal{L}_{rgb}=||C-C_{gt}||^{2},$ (11)

where $C$ is the rendering result, and $C_{gt}$ is the corresponding ground
truth. Following previous work [53, 27, 9], we adopt a smoothness loss for
diffuse and material as regularization:

$\mathcal{L}_{d}=\sum_{\mathbf{x}_{\text{surf
}}}\left|\bm{k}_{d}\left(\mathbf{x}_{\text{surf
}}\right)-\bm{k}_{d}\left(\mathbf{x}_{\text{surf }}+\epsilon\right)\right|,$
(12) $\mathcal{L}_{orm}=\sum_{\mathbf{x}_{\text{surf
}}}\left|\bm{k}_{orm}\left(\mathbf{x}_{\text{surf
}}\right)-\bm{k}_{orm}\left(\mathbf{x}_{\text{surf }}+\epsilon\right)\right|,$
(13)

where $\mathbf{x}_{\text{surf}}$ represents the world coordinates of points on
the object’s surface. $\bm{k}_{d}(\mathbf{x}_{\text{surf }})$ and
$\bm{k}_{orm}(\mathbf{x}_{\text{surf }})$ is the material of this point.
$\epsilon$ is a randomly distributed vector of tiny deformations. A self-
supervised loss is used to regulate the learned diffuse color:

$\mathcal{L}_{\text{diff}}=\mathcal{L}_{rgb}\left(\mathbf{C}_{\text{diff}},\bm{k}_{\text{diff}}\left(\bm{x_{\text{surf}}}\right)\right),$
(14)

where $\mathbf{C}_{\text{diff}}$ is the diffuse light from the object surface
obtained during rendering. $\bm{k}_{\text{diff}}(\mathbf{x}_{\text{surf}})$ is
the diffuse light obtained by the MLP. Overall, the full loss function is:

$\mathcal{L}=\mathcal{L}_{rgb}+\omega_{1}\mathcal{L}_{d}+\omega_{2}\mathcal{L}_{orm}+\omega_{3}\mathcal{L}_{\text{diff}},$
(15)

where $\omega_{1}$, $\omega_{2}$ and $\omega_{3}$ are three predefined
scalars.

Figure 3: Qualitative comparison. The results of renderings, materials, and
environment maps are presented. Note that, the material of Nefii contains only
roughness without metalness. Our method achieves the best renderings with
clear reflections, compared to other inverse rendering methods. Our method is
also superior to others in the disentanglement of materials and environment
maps.

##  4 Experiments

###  4.1 Implementation Details

Dataset. We construct a dataset of multiple reflective objects based on the
existing single objects to evaluate the performance. Our dataset consists of
16 groups of object compositions, most of which contain indirect illumination
between reflective objects. We render the composed objects with various
environmental lighting in the Blender engine. Each group contains 300 images,
with 200 for the training set and 100 for the test set.

Experiment setup. We optimize the 3D model on 1 RTX 3090 GPU with 24G memory.
We use the Adam optimizer for the material and the environment map with an
initial learning rate of 0.03. The coefficients of loss function $\omega_{1}$,
$\omega_{2}$, and $\omega_{3}$ are set to 0.1, 0.05, and 1, respectively. The
rate of Monte Carlo sampling is commonly set to 128 consistent with the
setting in Nvdiffrecmc [9].

Table 1: Quantitative Comparisons. Ours ($x$) means the $x$ times of sampling in our method. Ours (1) is about 1.5h longer than Nvdiffrecmc [9] as we need additional time to learn the initial geometry. Method |  Nvdiffrec [27] |  Nvdiffrecmc [9] |  Nefii [43] | Ours |  Ours (w/o Acc.) |  Ours (w/o Geo.) |  Ours (3) |  Ours (1)
---|---|---|---|---|---|---|---|---
PSNR$\uparrow$ | 26.9 | 25.7 | 22.3 | 28.1 | 28.0 | 24.6 | 28.4 | 27.3
Training time$\downarrow$ | 30min | 45min | 20h | 6.5h | 11.5h | 5h | 26.5h | 2.25h

###  4.2 Comparison with Baseline

In this section, we compare our method with Nvdiffrec [27], Nvdiffrecmc [9],
and Nefii [43] on our constructed reflective dataset, and the results are
shown in Fig. 3. Nvdiffrec achieves photo-realistic results in the metal balls
but fails on the smooth and glossy table + horse. Besides, Nvdiffrec cannot
well disentangle the materials because it does not consider the shading. In
contrast, Nvdiffrecmc performs well in material learning, while its rendering
results are bad. These two methods also suffer from indirect illumination from
the reflection of inner objects, leading to low-quality environment maps.
Nefii is the recent work that considers the indirect illumination in the
radiance field, but it tends to produce low-reflective results and performs
badly in these high specular objects. Compared to these methods, our method
can achieve both photo-realistic rendering results and well-disentangled
material learning. We can handle highly specular objects, e.g., the metal
table. The environment maps learned by our methods are also superior to
others.

Figure 4: Ablation study on the depth of ray tracing, i.e., the times of Monte
Carlo sampling. The results with depth$=1$ show fewer and darker reflections
compared to the ground truth. $k_{d}$ maps also illustrate the limited
capacity to disentangle the material from environmental light, for example,
mistaking the diffuse color of the table as the color of the sky. In contrast,
with depth=2 or 3, results show more realistic renderings and disentangled
materials. Figure 5: Ablation study on geometric initialization. As shown, a
better-quality geometry can significantly improve the material learning and
also refine the rendering results.

###  4.3 Multi-times Monte Carlo Sampling

In this section, we conduct an ablation study on the times of Monte Carlo
Sampling, i.e.the depth of ray tracing. Multi-times sampling considers the
indirect illumination, and thus successfully disentangles the environment
light and inner reflective light. As shown in Fig. 4, the environment maps
learned by single-time sampling contain noise and shades that are actually the
inner reflection. In contrast, multi-times sampling significantly releases the
problem, producing a clearer environment map. Single-time sampling also
undermines the rendering results, with darker inverted reflections. When the
sampling ray is sheltered by objects during path tracing, the ray returns no
light, resulting in a dark point. Multi-times sampling returns the reflective
light of the sheltering points containing both diffuse and specular light. As
a consequence, the rendering results of multi-times sampling show realistic
appearance and reflections. Besides, it can be seen that results with 2-time
sampling are comparable to results with 3-time sampling. Tab. 1 also
quantitatively shows minimal improvement by deeper tracing. However, each
additional sampling time induces a 4-times computation increment. Therefore, a
2-times sampling is a more cost-effective setting. Additionally, we also
compare the efficiency between the sampling with and without acceleration. As
shown in Tab. 1, our method can make the training speed twice as fast while
keeping a comparable PSNR. The degree of acceleration is related to the
complexity of geometry, and here we adopt the median in our dataset.

###  4.4 Initial Geometry

In this section, we further study the necessity of the initial geometry
discussed in Sec. 3.3. As shown in Fig. 5, without initial geometry, the
rendering results show bumpy surfaces with badly learned materials in the area
of hollow shape. Because multi-times sampling amplifies the errors introduced
by geometry, the learned materials are even worse than the single-time
sampling. When applying a good-quality initialization, the network bypasses
the ambiguity brought by the geometry. It helps our method learn well-
disentangled materials and finally produces high-quality rendering results.
Quantitative comparison in Tab. 1 also demonstrate the necessity of geometry
initialization.

Figure 6: Relighting and editing. We compose the reconstructed objects in a
unified scene and change the environmental light. The relighting results show
our strong ability to disentangle the light and shading. We also perform
material edits which shows the flexibility in wide applications.

###  4.5 Relighting and Material Editing

The disentangled environment map and material empower our methods to relight
and edit reconstructed objects in the downstream application. Our method can
also be seamlessly plugged into the industrial pipeline. As shown in Fig. 6,
our method enables the arbitrary combination of reconstructed objects. We can
easily edit their metalness, roughness, and albedo color by manipulating the
learned materials. Due to the well-disentangled shading, our rendering results
are natural and realistic in all five environment maps, even the point light
in a dark environment. They also perform well after editing materials thanks
to the well-learned material map. For example, after we increase the metalness
and reduce the roughness of the teapot, the teapot clearly reflects the
neighboring objects on its surface. In another case, when we extremely
increase the reflectance and change the base color of the toaster, it can
totally reflect the scene and other objects.

##  5 Conclusion and Limitations

In conclusion, our Ref-MC2 introduces multi-times Monte Carlo sampling into
the inverse rendering pipeline to model the indirect illumination. It improves
the performance in scenes with complex inter-reflections. However, increment
of sampling times significantly increases computational consumption and makes
the pipeline highly geometry-sensitive. To solve the challenge of
computational efficiency, based on the Lambert model, we pre-compute the
diffuse light that is irrelevant to the viewing direction as a map and
directly query it during multi-times sampling. To improve the geometry
quality, we adopt SDF-based architecture to get an initial geometry and refine
a design with Spherical Gaussian encoding for reflective objects. We further
use Flexicubes to take the initial mesh into the differentiable rendering
pipeline that learns disentangled materials. Our Ref-MC2 still has several
limitations. The major limitation is that the 2-time sampling cannot handle
extremely reflective objects, e.g., mirrors, as the specular energy hardly
degrades after reflections. Besides, the training time needs to further reduce
in future time.

## References

  * [1] Ronen Basri and David W. Jacobs.  Lambertian reflectance and linear subspaces.  IEEE Trans. Pattern Anal. Mach. Intell., 25(2):218–233, 2003.
  * [2] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P. A. Lensch.  Neural-pil: Neural pre-integrated lighting for reflectance decomposition.  In NeurIPS, pages 10691–10704, 2021.
  * [3] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In ECCV, volume 13692, pages 333–350, 2022.
  * [4] Zhang Chen, Zhong Li, Liangchen Song, Lele Chen, Jingyi Yu, Junsong Yuan, and Yi Xu.  Neurbf: A neural fields representation with adaptive radial basis functions.  In ICCV, pages 4159–4171, 2023.
  * [5] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In CVPR, 2022.
  * [6] Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, and Ying-Cong Chen.  Ref-neus: Ambiguity-reduced neural implicit surface learning for multi-view reconstruction with reflection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4251–4260, 2023.
  * [7] Yuan-Chen Guo, Yan-Pei Cao, Chen Wang, Yu He, Ying Shan, and Song-Hai Zhang.  Vmesh: Hybrid volume-mesh representation for efficient view synthesis.  In SIGGRAPH Asia, pages 17:1–17:11, 2023.
  * [8] Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Song-Hai Zhang.  Nerfren: Neural radiance fields with reflections.  In CVPR, pages 18388–18397, 2022.
  * [9] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg.  Shape, light, and material decomposition from images using monte carlo rendering and denoising.  Advances in Neural Information Processing Systems, 35:22856–22869, 2022.
  * [10] Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma.  Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields.  In ICCV, pages 19717–19726, 2023.
  * [11] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao.  2d gaussian splatting for geometrically accurate radiance fields.  In SIGGRAPH, 2024.
  * [12] Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma.  Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces.  arXiv preprint arXiv:2311.17977, 2023.
  * [13] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su.  Tensoir: Tensorial inverse rendering.  In CVPR, pages 165–174, 2023.
  * [14] James T Kajiya.  The rendering equation.  In Proceedings of the 13th annual conference on Computer graphics and interactive techniques, pages 143–150, 1986.
  * [15] James T Kajiya and Brian P Von Herzen.  Ray tracing volume densities.  ACM SIGGRAPH computer graphics, 18(3):165–174, 1984.
  * [16] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ACM Trans. Graph., 42(4):139:1–139:14, 2023.
  * [17] Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijaykumar.  ENVIDR: implicit differentiable renderer with neural environment lighting.  In ICCV, pages 79–89, 2023.
  * [18] Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang.  Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images.  ACM Transactions on Graphics (TOG), 42(4):1–22, 2023.
  * [19] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In Seminal graphics: pioneering efforts that shaped the field, pages 347–353. 1998.
  * [20] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai.  Scaffold-gs: Structured 3d gaussians for view-adaptive rendering.  CVPR, 2024.
  * [21] Jipeng Lv, Heng Guo, Guanying Chen, Jinxiu Liang, and Boxin Shi.  Non-lambertian multispectral photometric stereo via spectral reflectance decomposition.  In IJCAI, 2023.
  * [22] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In CVPR, pages 7210–7219, 2021.
  * [23] Stephen McAuley, Stephen Hill, Naty Hoffman, Yoshiharu Gotanda, Brian Smits, Brent Burley, and Adam Martinez.  Practical physically-based shading in film and game production.  In ACM SIGGRAPH 2012 Courses, pages 1–7. 2012.
  * [24] Stephen McAuley, Stephen Hill, Adam Martinez, Ryusuke Villemin, Matt Pettineo, Dimitar Lazarov, David Neubelt, Brian Karis, Christophe Hery, Naty Hoffman, et al.  Physically based shading in theory and practice.  In ACM SIGGRAPH 2013 Courses, pages 1–8. 2013.
  * [25] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [26] William J Morokoff and Russel E Caflisch.  Quasi-monte carlo integration.  Journal of computational physics, 122(2):218–230, 1995.
  * [27] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler.  Extracting triangular 3d models, materials, and lighting from images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8280–8290, 2022.
  * [28] Gregory M Nielson.  Dual marching cubes.  In IEEE visualization 2004, pages 489–496. IEEE, 2004.
  * [29] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589–5599, 2021.
  * [30] Michael Oren and Shree K Nayar.  Generalization of lambert’s reflectance model.  In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pages 239–246, 1994.
  * [31] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In ICCV, pages 5845–5854, 2021.
  * [32] Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P. Srinivasan, Ben Mildenhall, Andreas Geiger, Jonathan T. Barron, and Peter Hedman.  MERF: memory-efficient radiance fields for real-time view synthesis in unbounded scenes.  ACM Trans. Graph., 42(4):89:1–89:12, 2023.
  * [33] Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, and Jun Gao.  Flexible isosurface extraction for gradient-based mesh optimization.  ACM Transactions on Graphics (TOG), 42(4):1–16, 2023.
  * [34] Zixi Shu, Ran Yi, Yuqi Meng, Yutong Wu, and Lizhuang Ma.  Rt-octree: Accelerate plenoctree rendering with batched regular tracking and neural denoising for real-time neural radiance fields.  In SIGGRAPH Asia, pages 99:1–99:11, 2023.
  * [35] Peter-Pike J. Sloan, Jan Kautz, and John M. Snyder.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments.  ACM Trans. Graph., 21(3):527–536, 2002.
  * [36] Jiajun Tang, Haofeng Zhong, Shuchen Weng, and Boxin Shi.  Luminaire: Illumination-aware conditional image repainting for lighting-realistic generation.  In NeurIPS, 2023.
  * [37] Eric Veach and Leonidas J Guibas.  Optimally combining sampling techniques for monte carlo rendering.  In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 419–428, 1995.
  * [38] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan.  Ref-nerf: Structured view-dependent appearance for neural radiance fields.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5481–5490. IEEE, 2022.
  * [39] Fangjinhua Wang, Marie-Julie Rakotosaona, Michael Niemeyer, Richard Szeliski, Marc Pollefeys, and Federico Tombari.  Unisdf: Unifying neural representations for high-fidelity 3d reconstruction of complex scenes with reflections.  arxiv preprint arXiv:2312.13285, 2023.
  * [40] Jiaping Wang, Peiran Ren, Minmin Gong, John M. Snyder, and Baining Guo.  All-frequency rendering of dynamic, spatially-varying reflectance.  ACM Trans. Graph., 28(5):133, 2009.
  * [41] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  arXiv preprint arXiv:2106.10689, 2021.
  * [42] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  Nerf–: Neural radiance fields without known camera parameters.  arXiv preprint arXiv:2102.07064, 2021.
  * [43] Haoqian Wu, Zhipeng Hu, Lincheng Li, Yongqiang Zhang, Changjie Fan, and Xin Yu.  Nefii: Inverse rendering for reflectance decomposition with near-field indirect illumination.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4295–4304, 2023.
  * [44] Xiuchao Wu, Jiamin Xu, Zihan Zhu, Hujun Bao, Qixing Huang, James Tompkin, and Weiwei Xu.  Scalable neural indoor scene rendering.  ACM Trans. Graph., 41(4):98:1–98:16, 2022.
  * [45] Kun Xu, Wei-Lun Sun, Zhao Dong, Dan-Yong Zhao, Run-Dong Wu, and Shi-Min Hu.  Anisotropic spherical gaussians.  ACM Trans. Graph., 32(6):209:1–209:11, 2013.
  * [46] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  Advances in Neural Information Processing Systems, 34:4805–4815, 2021.
  * [47] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall.  Bakedsdf: Meshing neural sdfs for real-time view synthesis.  In SIGGRAPH, pages 46:1–46:9, 2023.
  * [48] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman.  Multiview neural surface reconstruction by disentangling geometry and appearance.  Advances in Neural Information Processing Systems, 33:2492–2502, 2020.
  * [49] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  In ICCV, pages 5732–5741, 2021.
  * [50] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger.  Mip-splatting: Alias-free 3d gaussian splatting.  CVPR, 2024.
  * [51] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely.  Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting.  In CVPR, pages 5453–5462, 2021.
  * [52] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv preprint arXiv:2010.07492, 2020.
  * [53] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron.  Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.  ACM Transactions on Graphics (ToG), 40(6):1–18, 2021.
  * [54] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou.  Modeling indirect illumination for inverse rendering.  In CVPR, pages 18622–18631, 2022.
  * [55] Youjia Zhang, Teng Xu, Junqing Yu, Yuteng Ye, Yanqing Jing, Junle Wang, Jingyi Yu, and Wei Yang.  Nemf: Inverse volume rendering with neural microflake field.  In ICCV, pages 22862–22872, 2023.


# Pioneering EEG Motor Imagery Classification
Through Counterfactual Analysis
††thanks: This work was supported by the Institute of Information &
Communications Technology Planning & Evaluation (IITP) grant, funded by the
Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate
School Program (Korea University)) and was partly supported by the Institute
of Information & communications Technology Planning & Evaluation (IITP) grant,
funded by the Korea government (MSIT) (No. 2021-0-02068, Artificial
Intelligence Innovation Hub).

Kang Yin
Hee-Dong Kim  Dept. of Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Dept. of Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Hye-Bin Shin
Seong-Whan Lee  Dept. of Brain and Cognitive Engineering
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Dept. of Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

The application of counterfactual explanation (CE) techniques in the realm of
electroencephalography (EEG) classification has been relatively infrequent in
contemporary research. In this study, we attempt to introduce and explore a
novel non-generative approach to CE, specifically tailored for the analysis of
EEG signals. This innovative approach assesses the model’s decision-making
process by strategically swapping patches derived from time-frequency
analyses. By meticulously examining the variations and nuances introduced in
the classification outcomes through this method, we aim to derive insights
that can enhance interpretability. The empirical results obtained from our
experimental investigations serve not only to validate the efficacy of our
proposed approach but also to reinforce human confidence in the model’s
predictive capabilities. Consequently, these findings underscore the
significance and potential value of conducting further, more extensive
research in this promising direction. The implementation is available at
https://github.com/Kang1121/EEG-Counterfactual.

Keywords–electroencephalography, counterfactual explanation;

##  I INTRODUCTION

The evolution of electroencephalograph (EEG) analysis and brain-computer
interfaces (BCI) has greatly benefited from emerging computational techniques.
Kim et al.[1] highlight the integration of convolutional neural network (CNN)
in single-trial electromyography (EMG) analysis. Thung et al.[2] delve into
predictive modeling for cognitive impairments using matrix completion
techniques. Ma et al.[3] integrate CNN with attention for motor imagery (MI)
classification. In parallel, Lee et al.[4] have showcased the potential
influence of auditory stimuli on brain functions, and Roy [5] fuses multi-
scale features for multi-classification. Such advancements coexist with
intriguing research into the neural underpinnings of emotion representation[6]
and state-of-the-art multi-view CNN architectures[7] for BCI.

But while these methods show promise, it’s unclear how they make predictions
about EEG data. The comprehension of deep learning, indeed, can often be
challenging and complex. Some research[8, 9] try to explain EEG models using
tools like shapley additive explanations (SHAP)[10] and layer-wise relevance
propagation (LRP)[11], focusing on the importance of certain features or
neurons. Yet, these tools may not offer clear or actionable insights. Another
approach, counterfactual explanation (CE), offers more direct advice by
suggesting specific changes to input data. While CE is popular in image
studies, it’s less common in brain-machine research. Our paper looks at how CE
might help clarify EEG model predictions, drawing from previous research[12]
in image analysis.

In this study, we apply CE to vividly illustrate how a model gives its
predictions from specific regions within EEG spectrogram. We emphasize the
connection between the counterfactuals and their original data samples. Our
findings indicate that implementing CE in EEG studies is not only feasible but
also holds considerable potential for further exploration. This research makes
two significant advancements:

  * •

We present an innovative approach by employing CE to shed light on EEG
classifications. As far as we’re aware, ours is a pioneering effort in this
domain with EEG signals.

  * •

We consciously bypass generative CE. Such a decision guarantees that the
resulting CE remains consistent with genuine, credible data and is
particularly apt for the more compact datasets common in EEG research.

##  II METHODOLOGY

###  II-A Problem Formulation

Consider an EEG spectrogram dataset $\mathcal{I}$ with $\mathcal{C}$ classes.
For a class $c$ with $n$ samples, let query set $Q=\\{I_{i}^{c}\\}_{i=1}^{n}$
and distractor set $D=\bigcup_{j=1,j\neq
c}^{\mathcal{C}}\\{I_{i}^{j}\\}_{i=1}^{n}$, such that $Q\cup D=\mathcal{I}$
and $Q\cap D=\varnothing$. Let $I$ and $I^{\prime}$ be random samples from $Q$
and $D$ with classes $c$ and $c^{\prime}$.

Building upon the work in[12], consider a feature extractor
$f:\mathcal{I}\to\mathbb{R}^{hw\times d}$ that transforms the EEG spectrogram
into a feature dimension of $h\times w\times d$. Alongside, there’s a
classifier $g:\mathbb{R}^{hw\times d}\to\mathbb{R}^{\mathcal{C}}$ that
associates the spatial feature to a set of $\mathcal{C}$-logits. We formulate
a counterfactual $I^{*}$ within the feature space $f(\cdot)$ by substituting
spatial cells in $f(I)$ with those from $f(I^{\prime})$, ensuring that the
classifier outputs $c^{\prime}$ for $I^{*}$. The counterfactual $I^{*}$ is
expressed as:

$f(I^{*})=(\mathds{1}-\mathbf{a})\circ f(I)+\mathbf{a}\circ Pf(I^{\prime}),$
(1)

where $\mathds{1}$ stands for a vector filled with ones. The sparse binary
gating vector $\mathbf{a}\in\mathbb{R}^{hw}$ adjusts the mix of
$f(I^{\prime})$ and $f(I)$ in the resultant $f(I^{*})$.
$P\in\mathbb{R}^{hw\times hw}$ is a permutation matrix ensuring the alignment
of $f(I^{\prime})$ and $f(I)$, and $\circ$ denotes the Hadamard product.

###  II-B Objective Function

Our objective is to alter the model’s prediction from $c$ to $c^{\prime}$.
Specifically, we aim to maximize the probability of the modified feature
$f(I^{*})$ being associated with class $c^{\prime}$. The objective is defined
as:

$\max_{P,a}g_{c^{\prime}}((\mathds{1}-\mathbf{a})\circ f(I)+\mathbf{a}\circ
Pf(I^{\prime})),$ (2)

where $P\in\mathcal{P}$, the collection of all $hw\times hw$ permutation
matrices. Drawing from the concepts in CE[12, 13], our interest lies in
identifying the minimal number of edit operations needed for a prediction
shift. As such, to avoid learning an identical mapping, Eq. 2 incorporates
regularization. This is characterized by the constraints
$||\mathbf{a}||_{1}=1,\mathbf{a}_{i}\in\\{0,1\\}$. Given that $\mathbf{a}$ is
binary, reducing its norm equates to minimizing the edit count between
$I^{\prime}$ to $I$.

##  III EXPERIMENT

###  III-A Dataset and Preprocessing

In this research, we utilize the BCI-C IV 2a EEG dataset[14]. This dataset
comprises EEG recordings from two sessions involving 9 healthy participants,
each performing 4 MI tasks. Each task consists of 72 trials per session. The
EEG recordings were captured through 22 channels at a sampling rate of 250 Hz,
with a 3-second MI period for every trial. In this study, we only choose left-
and-right-hand MI tasks in the following experiments.

To adapt the data for image processing suitable for network input[13], we
first downsample the signal by a factor of 3. Subsequently, we clip the data
and select 224 linearly spaced frequencies from 1 Hz to 41 Hz. We then employ
the continuous wavelet transform (CWT) to obtain the spectrogram. It is
important to note that the use of the EEG spectrogram as an input to the model
is a well-founded choice, substantiated by preceding research [15, 16, 17,
18], which has demonstrated that spatio-temporal attributes frequently yield
robust performance metrics. Once the preprocessing phase is complete, the EEG
data is reshaped into a 224 $\times$ 224 matrix, with the first axis
representing the frequency dimension and the second representing the temporal
dimension.

###  III-B Experimental Setup and Classification Results

Table I displays our approach to data categorization. While traditionally,
left-hand MI and right-hand MI are treated as two separate classes, we
introduce another configuration where each subject is considered its own
domain. This results in 2 $\times$ 9 = 18 classes, leading to a reduction in
sample size by a factor of 9. Besides, our training methodology diverges from
typical practices. We allocate half of each subject’s data for training, split
the remaining half between validation and test for CE. The ResNet-18
framework, embodying functions $f$ and $g$, serves as our primary model.
Instead of utilizing pretrained models, we train from scratch, leveraging
MixUp to enhance the training process. The Top-1 accuracy for 2-class and
18-class settings are 69.44 % and 54.78 %, respectively.

TABLE I: Overview of experimental setup and Top-1 accuracy. Dataset | Statistics | Top-1 (%)
---|---|---
| # Class | # Train | # Val | ResNet-18
BCI-C IV 2a (2-class) | 2 | 1,296 | 648 | 69.44
BCI-C IV 2a (18-class) | 18 | 1,296 | 648 | 54.78

Figure 1: Configuration of (left) keypoint annotations and (right) the
corresponding areas in the spectrogram.

###  III-C Evaluation Metrics

We adopt the evaluation metrics as outlined in [13]:

  * •

Near-KP assesses whether the spectrogram regions contain object keypoints.

  * •

Same-KP quantifies the frequency with which we select identical keypoints in
both the query and distractor spectrograms.

  * •

# Edits calculates the average count of modifications required for the
classification model to predict the distractor class $c^{\prime}$ on the
altered spectrogram $I^{*}$.

A crucial metric in vision CE is the keypoint (KP) evaluation, which relies on
manually annotated object coordinates within images. For our specific task, we
establish keypoints as detailed in Fig. 1 and assess our results based on
these keypoints. We select a total of 10 keypoints, centrally located in the
middle regions of the final two rows. Our rationale for choosing these
particular points is twofold. From a frequency perspective, they encompass the
entirety of the $\alpha$ band, which has been demonstrated to play a
significant role in MI tasks. Additionally, they touch upon the $\beta$,
$\theta$, and $\delta$ bands, all of which may be involved in the MI task to
some extent. Temporally speaking, under the assumption that the data follows a
normal distribution, we focus on the central portion, which is where MI
activity is most likely to occur.

###  III-D Quantitative Analysis

TABLE II: Counterfactual results on BCI-C IV 2a dataset. Setting |  | BCI-C IV 2a
---|---|---
|  | Near-KP (%) | Same-KP (%) | # Edits
Single | 2-class | 92.88 | 19.55 | -
Edit | 18-class | 74.50 | 13.80 | -
All | 2-class | 81.80 | 14.51 | 2.08
Edits | 18-class | 71.15 | 11.19 | 2.49

Table II provides results in two distinct settings: a ”Single Edit” scenario
and an ”All Edits” scenario. The former represents a one-time solution to Eq.
2, while the latter involves iteratively solving the same equation until a
shift in prediction is observed. A notable observation is the elevated scores
for Near-KP. This signifies alterations primarily at the annotated keypoint
regions, aligning with empirical evidence. Concurrently, we note an
exceptionally low score for Same-KP. This may imply that the model discerns
distinct, non-overlapping patterns to differentiate between classes.
Additionally, for the 2-class and 18-class categorizations, the average edit
times stand at 2.08 and 2.49 respectively. This suggests the model’s
capability in identifying distinctive features, enabling it to differentiate
between class variations.

###  III-E Qualitative Results

In this section, we offer visual insights into counterfactuals using Fig. 2,
which is representative of the overall data. It displays the query and
distractor pairs from a successful CE, with the left column representing the
query and the right the distractor. Replacements are shown top to bottom, with
two edits needed to change the model’s prediction. Each replacement is marked
by a red bounding box, predominantly located in the $\alpha$ band,
corroborated by Table II and aligning with our assumptions. These replacements
occur in distinct time periods, highlighting the model’s capability to discern
temporal differences. Interestingly, highlighted regions don’t always
correspond with the highest values, suggesting avenues for future research.

Figure 2: Visualization of the replacing regions in query and distractor
spectrograms. (# Edits = 2)

Furthermore, as depicted in Fig. 3, we present a marginal histogram of kernel
density estimation (KDE) for query and distractor spectrograms in 2-class and
18-class settings (first and second rows, respectively). We flatten the 7
$\times$ 7 square regions in Fig. 1 (left), with each axis having a length of
49, representing query and distractor, and each 7 corresponds to a frequency
range in Fig. 1 (right). The histogram indicates the frequency of regions
chosen, while the KDE shows the joint distribution of selected edit pairs. The
lighter an area, the more often it is selected as the replacement region. We
train the ResNet-18 classification model in three ways: underfitting, well-
trained, and overfitting (left to right), aiming to observe variations in CE
results. High Near-KP values suggest that highlighted areas should be near the
intersection of $\alpha$ band, while high Same-KP values indicate they should
align with the antidiagonal. In the underfitting case, the highlighted areas
are dispersed across all frequencies for query spectrograms, indicating that
the model has not learned distinctive classification features. Conversely, in
the overfitting case, the highlighted areas gravitate towards the visually
distinctive $\theta$ and $\delta$ bands, potentially biased by contradicting
empirical priors. The well-trained cases show compact distributions in the
$\alpha$ band, aligning with our expectations and suggesting that the model is
likely making predictions based on truly meaningful features.

Figure 3: Marginal histogram of KDE on edit pairs for 2-class and 18-class
settings (row 1 and 2), respectively. (Left to right: underfitting, well-
trained, and overfitting.)

##  IV CONCLUSION

In this study, we introduce a CE method tailored for EEG MI classification. We
have devised a comprehensive framework that adeptly adapts contemporary non-
generative CE techniques and evaluation criteria to EEG data. Furthermore, we
have developed specialized keypoints explicitly designed for EEG-based tasks.
Our empirical results carry significant implications: they not only affirm the
prowess of the pretrained model in modifying predictions with minimal edits,
but they also resonate with established human empirical insights, reinforcing
trust in the model’s predictions.

This work lays a foundational pathway for researchers to delve into the realm
of BCI counterfactual explanations, expanding beyond conventional explanatory
tools such as SHAP and LRP. Our contribution delineates a streamlined yet
robust procedure accompanied by a benchmark rooted in fundamental
configurations. We posit that such novel explorations may substantially aid in
elucidating intricate yet auspicious domains, such as decoding of imagined
speech [19, 20, 21]. However, we acknowledge the scope for refinement in our
approach. In future work, we aim to integrate various explanatory tools and
also enhance the adaptability of the framework to accommodate diverse input
dimensions. Furthermore, we intend to reassess and refine evaluation metrics
to identify approaches most fitting for BCI applications.

## Acknowledgement

We thank Seung-Hyup Na and Yeong-Joon Ju for their valuable discussions that
significantly contributed to the development of the research topic.

## References

  * [1] K.-T. Kim, C. Guan, and S.-W. Lee, “A subject-transfer framework based on single-trial EMG analysis using convolutional neural networks,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 28, no. 1, pp. 94–103, 2019.
  * [2] K.-H. Thung et al., “Conversion and time-to-conversion predictions of mild cognitive impairment using low-rank affinity pursuit denoising and matrix completion,” _Med. Image Anal._ , vol. 45, pp. 68–82, 2018.
  * [3] X. Ma et al., “A temporal dependency learning CNN with attention mechanism for MI-EEG decoding,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , 2023.
  * [4] M. Lee, C.-B. Song, G.-H. Shin, and S.-W. Lee, “Possible effect of binaural beat combined with autonomous sensory meridian response for inducing sleep,” _Front. Hum. Neurosci._ , vol. 13, pp. 425–440, 2019.
  * [5] A. M. Roy, “Adaptive transfer learning-based multiscale feature fused deep convolutional neural network for EEG MI multiclassification in brain–computer interface,” _Eng. Appl. Artif. Intell._ , vol. 116, p. 105347, 2022.
  * [6] J. Kim et al., “Abstract representations of associated emotions in the human brain,” _J. Neurosci._ , vol. 35, no. 14, pp. 5655–5663, 2015.
  * [7] R. Mane et al., “FBCNet: A multi-view convolutional neural network for brain-computer interface,” _arXiv preprint arXiv:2104.01233_ , 2021.
  * [8] D. Raab, A. Theissler, and M. Spiliopoulou, “XAI4EEG: Spectral and spatio-temporal explanation of deep learning-based seizure detection in EEG time series,” _Neural. Comput. Appl._ , vol. 35, no. 14, pp. 10 051–10 068, 2023.
  * [9] K. Yin, E. Y. Lim, and S.-W. Lee, “GITGAN: Generative inter-subject transfer for EEG motor imagery analysis,” _Pattern Recognit._ , p. 110015, 2023.
  * [10] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” _Adv. Neural Inf. Process. Syst. (NeurIPS)_ , vol. 30, 2017\.
  * [11] S. Bach et al., “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,” _PLoS One_ , vol. 10, no. 7, p. e0130140, 2015.
  * [12] Y. Goyal et al., “Counterfactual visual explanations,” _Int. Conf. Mach. Learn. (ICML)_ , pp. 2376–2384, 2019.
  * [13] S. Vandenhende, D. Mahajan, F. Radenovic, and D. Ghadiyaram, “Making heads or tails: Towards semantically consistent visual counterfactuals,” _Eur. Conf. Comput. Vis. (ECCV)_ , pp. 261–279, 2022.
  * [14] C. Brunner, R. Leeb, G. Müller-Putz, A. Schlögl, and G. Pfurtscheller, “BCI competition 2008–graz data set a,” _Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces), Graz University of Technology_ , vol. 16, pp. 1–6, 2008.
  * [15] S.-B. Lee et al., “Comparative analysis of features extracted from EEG spatial, spectral and temporal domains for binary and multiclass motor imagery classification,” _Inf. Sci._ , vol. 502, pp. 190–200, 2019.
  * [16] H.-I. Suk, S. Fazli, J. Mehnert, K.-R. Müller, and S.-W. Lee, “Predicting BCI subject performance using probabilistic spatio-temporal filters,” _PLoS One_ , vol. 9, no. 2, p. e87056, 2014.
  * [17] L. Farokhah, R. Sarno, and C. Fatichah, “Simplified 2D CNN architecture with channel selection for emotion recognition using EEG spectrogram,” _IEEE Access_ , 2023.
  * [18] J.-S. Bang, M.-H. Lee, S. Fazli, C. Guan, and S.-W. Lee, “Spatio-spectral feature representation for motor imagery classification using convolutional neural networks,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 33, no. 7, pp. 3038–3049, 2021.
  * [19] A. Kamble, P. Ghare, V. Kumar, A. Kothari, and A. Keskar, “Spectral analysis of EEG signals for automatic imagined speech recognition,” _IEEE Trans. Instrum. Meas._ , 2023.
  * [20] S.-H. Lee, M. Lee, and S.-W. Lee, “Neural decoding of imagined speech and visual imagery as intuitive paradigms for BCI communication,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 28, no. 12, pp. 2647–2659, 2020.
  * [21] S.-H. Lee, M. Lee, J.-H. Jeong, and S.-W. Lee, “Towards an EEG-based intuitive BCI communication system using imagined speech and visual imagery,” _Conf. Proc. IEEE Int. Conf. Syst. Man Cybern. (SMC)_ , pp. 4409–4414, 2019.