# Towards Understanding Variants of Invariant Risk Minimization through the
Lens of Calibration

Kotaro Yoshida
Tokyo Institute of Technology
<EMAIL_ADDRESS>
&Hiroki Naganuma∗
Mila, Université de Montreal
<EMAIL_ADDRESS>
equal contributions

###### Abstract

Machine learning models traditionally assume that training and test data are
independently and identically distributed. However, in real-world
applications, the test distribution often differs from training. This problem,
known as out-of-distribution generalization, challenges conventional models.
Invariant Risk Minimization (IRM) emerges as a solution, aiming to identify
features invariant across different environments to enhance out-of-
distribution robustness. However, IRM’s complexity, particularly its bi-level
optimization, has led to the development of various approximate methods. Our
study investigates these approximate IRM techniques, employing the Expected
Calibration Error (ECE) as a key metric. ECE, which measures the reliability
of model prediction, serves as an indicator of whether models effectively
capture environment-invariant features. Through a comparative analysis of
datasets with distributional shifts, we observe that Information Bottleneck-
based IRM, which condenses representational information, achieves a balance in
improving ECE while preserving accuracy relatively. This finding is pivotal,
as it demonstrates a feasible path to maintaining robustness without
compromising accuracy. Nonetheless, our experiments also caution against over-
regularization, which can diminish accuracy. This underscores the necessity
for a systematic approach in evaluating out-of-distribution generalization
metrics, one that beyond mere accuracy to address the nuanced interplay
between accuracy and calibration.

##  1 Introduction

In the evolving landscape of machine learning, the importance of out-of-
distribution (OOD) generalization (Arjovsky, 2020) and calibration (Guo et
al., 2017) cannot be understated, especially in real-world applications and
critical scenarios. Traditional machine learning approaches, grounded in the
assumption that data are independently and identically distributed (IID),
often struggle to cope with OOD scenarios. Additionally, a notable trend in
contemporary machine learning models is their tendency towards overconfidence,
which undermines the reliability of their confidence estimations. This issue
is primarily recognized as a calibration challenge (Guo et al., 2017).

Responding to these limitations, there has been a surge in research focusing
on methodologies like Invariant Risk Minimization (IRM) (Arjovsky et al.,
2020). IRM is designed to identify features that remain constant across
different environments, thereby facilitating more robust generalization in the
face of environmental variations. However, the computational demands of IRM
have led to the exploration of more feasible, approximate methods. Despite the
development of several IRM variants, their inability to consistently surpass
the performance of finely-tuned Empirical Risk Minimization (ERM) models has
been observed (Gulrajani & Lopez-Paz, 2020)(Zhang et al., 2023). This
performance gap raises critical questions about the inherent deficiencies of
these IRM variants. As we showed in Figure 1, even by leveraging IRM variants,
there is a trade-off between in-distribution accuracy and OOD accuracy. It
implies that models trained by IRM variants failed to obtain invariant
features since they sacrifice in-distribution accuracy to achieve better OOD
accuracy.

From another perspective of IRM, it can also be conceptualized as a
specialized form of multi-domain calibration (Wald et al., 2021). Our study
intersects with these findings, examining the extent to which IRM variants
mimic the original model in terms of calibration. We posit that effective
implementation of IRM should lead to reduced loss and a zero expected
calibration error (ECE) (Wald et al., 2021). Through comparative
experimentation, we assess various IRM variants in environments marked by
distributional shifts, such as diversity and correlation shifts (Ye et al.,
2021). Our evaluation focuses not only on accuracy but also on ECE. The
results highlight notable disparities in ECE among different methodologies,
even when OOD accuracy levels are similar. This suggests that a sole reliance
on OOD accuracy for method development might be inadequate. Our observations
include a range of calibration behaviors across methods, from overconfidence
to underestimation.

Our key contributions are as follows:

  * •

We investigate the reasons behind the gap between IRM formulation and its
variants through the lens of ECE.

  * •

We demonstrate that Information Bottleneck-based IRM (IB-IRM) effectively
lowers ECE, aligning it more closely with the original objectives of IRM
(Figure 3, Table 2).

  * •

We reveal that IRM variants with robust regularization typically achieve lower
ECE (Figure 3).

  * •

We identify a trade-off between ECE and accuracy, thereby underscoring the
intricate balance and compromises necessary for achieving both robust
generalization and accurate calibration (Figure 5).

Utilizing ECE as a metric, our study sheds light on the gap between the
original formulation of IRM and its variants. We recognize a notable trade-off
between ECE and accuracy, highlighting the complexities and necessary
compromises in striving for both robust generalization and precise
calibration.

(a)

ID IRMv1 OOD IRMv1

(b)

ERM IRMv1 BIRM IB-IRM IRM Game PAIR

Figure 1:  A graph observed in CMNIST showing the trade-off between in-
distribution (ID) and OOD accuracy in approximation methods of IRM. Figure (a)
visualizes the accuracy of the IRMv1 model ID and OOD, plotted against the
training steps on the horizontal axis. The two solid lines are approximately
symmetric around an accuracy of 50%. Figure (b) visualizes the ID accuracy
(horizontal axis) against the OOD accuracy (vertical axis) for typical
approximation methods of IRM. There is a clear trend that as the accuracy
improves OOD, the ID accuracy decreases.

##  2 Background

###  2.1 OOD Generalization

In traditional deep neural network scenarios, it is assumed that the learning
and test environments are IID. The most commonly used learning algorithm in
this assumption has been ERM(Vapnik, 1991), which has been successful in
various scenarios. Specifically, ERM aims to minimize the sum of losses of the
model $f:\mathcal{X}\to\mathcal{Y}$ in the training environment $E_{train}$.

$\min_{f:\mathcal{X}\to\mathcal{Y}}\sum_{e\in E_{train}}R^{e}(f)$ (1)

However, in the real world, the assumption of IID does not always hold, and
distribution shifts between training and testing environments can occur
(Quionero-Candela et al., 2009). When the model relies on features that are
only effective in the training environment, known as shortcut features, its
performance can decrease in the testing environment (Geirhos et al., 2020).
This issue is commonly known as OOD generalization and has become an urgent
problem in recent years. Therefore, it is necessary to train robust models
that are not dependent on specific environments and can perform consistently
across all environments. A robust model is defined as follows.

$\mathbb{E}[Y^{e}|f(X^{e})]=\mathbb{E}[Y^{e^{\prime}}|f(X^{e^{\prime}})],\forall
e,e^{\prime}\in E_{all}$ (2)

where $E_{all}$ represents the set of all possible environments. (2) indicates
that, for a robust model, the conditional probability of the label given the
model’s prediction should be equal between any two environments.

Since it is difficult to access all real-world environments during training,
various methods have been proposed to realize invariant models satisfying (2)
using limited learning environments.

###  2.2 Calibration

The alignment of a model’s predictive confidence with its actual accuracy is
extremely important in practical scenarios such as medical image diagnosis and
autonomous driving. For example, if there are 100 instances where a model
predicts with 80% confidence, then 80 of those predictions should be correct.
However, in recent deep neural networks, issues such as increased model
capacity have led to problems with the miscalibration and many methods have
been proposed to address this calibration issue (Guo et al., 2017). The
correct calibration in all environments is defined for any predictive
probability $\alpha$ of $f$ as follows:

$\mathbb{E}[Y^{e}|f(X^{e})=\alpha]=\alpha,\forall e\in E_{all}$ (3)

Equation (3) implies that the model’s predictive probability is always
consistent with its actual accuracy.

Furthermore, the quality of uncertainty calibration is often quantified as ECE
and is commonly used. It is defined as follows:

$ECE=\sum_{m=1}^{M}\frac{|B_{m}|}{N}|Acc(B_{m})-Conf(B_{m})|$ (4)

$Acc(B_{m})$ and $Conf(B_{m})$ are defined as follows

$\displaystyle Acc(B_{m})=$
$\displaystyle\frac{1}{\left|B_{m}\right|}\sum_{b\in
B_{m}}\mathbf{1}\left(\hat{y}_{b}=y_{b}\right)$ $\displaystyle Conf(B_{m})=$
$\displaystyle\frac{1}{\left|B_{m}\right|}\sum_{b\in B_{m}}\hat{p}_{b}$

where the predictive probabilities ranging from 0 to 1 are divided into $M$
bins, and for each bin, $Acc$ represents the accuracy of predictions within
the bin, while $Conf$ represents the average predictive probability
(confidence level) within the bin.

###  2.3 OOD Generalization and Calibration

It has been pointed out that OOD generalization and calibration are
interrelated. Notably, (3) can be a sufficient condition for (2),
theoretically demonstrating that calibration across all environments leads to
OOD generalization.

Additionally, it has been empirically demonstrated that there is a correlation
between a model’s accuracy in OOD (OOD) scenarios and its calibration
performance (Ovadia et al., 2019).

In this paper, we utilize this fact and hypothesize that models with high
calibration performance are likely to achieve better OOD generalization. We
evaluate the approximation methods of Invariant Risk Minimization (IRM),
introduced in 4, based on the model’s calibration performance to assess their
OOD generalization capabilities.

##  3 Preliminary

###  3.1 Domain Invariant Representation

One approach to making models more robust in OOD scenarios is through
representation learning. Representation learning is a method that teaches the
model to learn effective data representations from raw data for better
performance in predictions. Traditionally, features of data were designed by
human, but representation learning reduces this necessity and can also
compress the dimensions of input data, enabling faster computations.

In the context of OOD scenarios, there is an application for Representation
Learning, specifically aimed at extracting environment-invariant features from
raw data and reducing environment-specific spurious features to achieve more
robust predictions. For example, in a task of classifying animals from images,
if the model learns to associate parts of an animal with extraneous elements,
it may lead to high performance within the training domain but fail to make
robust predictions when the background distribution shifts in the test domain.
However, if the model discards spurious features (e.g., background) from the
raw data, it can consistently make robust predictions across all environments.
One particularly well-known method that applies representation learning to
out-of-distribution generalization scenarios is Domain-Adversarial Neural
Networks (DANN) (Ganin et al., 2016). This approach uses adversarial training
to adjust the learning process so that it is impossible to discern from the
extracted features which environment the input comes from, thereby aiming to
extract invariant features.

###  3.2 Invariant Learning

In the context of applying representation learning to OOD generalization, the
method we focus on in this paper is IRM(Arjovsky et al., 2020). IRM is
proposed with the aim of extracting environment-invariant features from input
data to enable consistent predictions across any environment. IRM is defined
with the purpose of realizing (2) as follows.

$\min_{\begin{subarray}{c}\Phi:\mathcal{X}\to\widehat{\mathcal{H}}\\\
\omega:\widehat{\mathcal{H}}\to\widehat{\mathcal{Y}}\end{subarray}}\sum_{e\in
E_{train}}R^{e}(\omega\circ\Phi)$ (5) $subject\ to\ \ \
\omega\in\arg\min_{\bar{\omega}:\widehat{\mathcal{H}}\to\widehat{\mathcal{Y}}}R^{e}(\bar{\omega}\circ\Phi),\forall
e\in E_{train}$

Here, $\mathcal{X}$ represents the input space of each data point,
$\widehat{\mathcal{H}}$ denotes the extracted invariant feature space, and
$\mathcal{Y}$ is the output space of the model. IRM defines the function as
$f=\omega\circ\Phi$, where $\Phi$ maps the input data to
$\widehat{\mathcal{H}}$, and based on these extracted invariant features, the
final prediction is made by $\omega$ . The aim of IRM is to achieve consistent
predictions as a form of OOD generalization.

The formulation in (5) represents a bi-level optimization problem, which is
challenging to solve exactly. As a result, various implementable approximation
methods have been proposed. However, despite numerous contributions to the
development of approximation methods for IRM, a method that generally
surpasses well-tuned ERM in terms of out-of-distribution generalization
accuracy has not yet been proposed. Therefore, it is essential to understand
why these approximation methods have not succeeded, and what approaches might
be more suitable for addressing the approximation problem.

##  4 Related Works

Table 1 presents an overview of IRM and its approximation methods. We discuss
the variants of IRM in the following section.

Methods | Featurizer | Predictor | Linearity of $\omega$ | Information Bottleneck | Game theory | Ensemble | Bayesian inference
---|---|---|---|---|---|---|---
IRM | $\Phi$ | $\omega$ | ✗ | ✗ | ✗ | ✗ | ✗
IRMv1 | $\omega\cdot\Phi$ | 1.0 | ✓ | ✗ | ✗ | ✗ | ✗
IB-IRM | $\omega\cdot\Phi$ | 1.0 | ✓ | ✓ | ✗ | ✗ | ✗
PAIR | $\omega\cdot\Phi$ | 1.0 | ✓ | ✗ |  ✓ (Pareto Optimality) | ✓ | ✗
IRM Game | $\Phi$ |  $\omega_{ensemble}$ 111$\omega_{ensemble}=\frac{1}{|E_{train}|}\left(\sum_{e\in E_{train}}w^{e}\right)$ | ✗ | ✗ |  ✓ (Nash Equilibrium) | ✓ | ✗
BIRM | $\omega\cdot\Phi$ | 1.0 | ✓ | ✗ | ✗ | ✗ | ✓

Table 1: Approximation methods of IRM

###  4.1 IRMv1

IRMv1(Arjovsky et al., 2020) constrains $\omega$ in (5) as a linear mapping
and allows the transformation $\Phi^{\prime}=(\omega\cdot\Phi)$,
$\omega^{\prime}=1.0$. The following one-variable optimization problem can be
relaxed and computed.

$\min_{\Phi^{\prime}:\mathcal{X}\to\mathcal{Y}}\sum_{e\in
E_{train}}R^{e}(\Phi^{\prime})\ +\
\lambda\parallel\nabla_{\omega^{\prime}\mid\omega^{\prime}=1.0}R^{e}(\omega^{\prime}\cdot\Phi^{\prime})\parallel^{2}$
(6)

###  4.2 Information Bottleneck based IRM(IB-IRM)

While IRMv1 has been successful with certain OOD generalization datasets such
as CMNIST, it has been noted that if not enough invariant features are
included in each environment in the training data, robust prediction is lost.
Therefore, IB-IRM(Ahuja et al., 2022) aims to improve IRMv1 by compressing the
entropy of $\Phi$ using the information bottleneck method, thereby preventing
excessive reliance on features that adversely affect the invariant prediction.
The variance of the features extracted from each data by $\Phi$ is added as a
regularization term.

$\lambda\parallel\nabla_{\omega^{\prime}\mid\omega^{\prime}=1.0}R^{e}(\omega^{\prime}\cdot\Phi^{\prime})\parallel^{2}+\
\gamma Var(\Phi)$ (7)

###  4.3 Pareto IRM(PAIR)

In general, there is a trade-off between ERM and OOD generalization, and
PAIR(Chen et al., 2023) focuses on the need to properly manage that trade-off
when generalizing a model out of distribution. To achieve more robust models,
PAIR is designed to find their Pareto optimal solutions in terms of multi-
objective optimization. In fact, it is implemented as a multi-objective
optimal problem for ERM, IRMv1 penalties, and vREx(Krueger et al., 2021)
penalties.

$\min_{\Phi^{\prime}:\mathcal{X}\to\mathcal{Y}}(\mathcal{L}_{ERM},\mathcal{L}_{IRMv1},\mathcal{L}_{vREx})$
(8)

where

$\displaystyle\mathcal{L}_{ERM}$ $\displaystyle=\sum_{e\in
E_{train}}R^{e}(\Phi^{\prime})$ $\displaystyle\mathcal{L}_{IRMv1}$
$\displaystyle=\parallel\nabla_{\omega^{\prime}\mid\omega^{\prime}=1.0}R^{e}(\omega^{\prime}\cdot\Phi^{\prime})\parallel^{2}$
$\displaystyle\mathcal{L}_{vREx}$
$\displaystyle=Var(\\{R^{e}(\Phi^{\prime})\\}_{e\in E_{train}})$

###  4.4 IRM Game

IRM Game(Ahuja et al., 2020) aims to achieve Nash equilibrium in terms of
inference accuracy by introducing a game-theoretic framework to the IRM
between each environment. Each environment $e$ in the training data is
assigned its own predictor $\omega^{e}$, each of which is trained to be
optimal in each environment, and the final $\omega_{ens}$ is the ensemble of
all predictors $\frac{1}{|E_{train}|}\sum_{e\in E_{train}}\omega^{e}$. By
learning environment-specific predictors, the limitation on linearity made in
IRMv1 is eliminated, and an implementation closer to (5) is aimed at.

$\min_{\begin{subarray}{c}\Phi:\mathcal{X}\to\widehat{\mathcal{H}}\\\
\omega_{ens}:\widehat{\mathcal{H}}\to\widehat{\mathcal{Y}}\end{subarray}}\sum_{e\in
E_{train}}R^{e}(\omega_{ens}\circ\Phi)$ (9) $s.t.\
\omega^{e}\in\arg\min_{\bar{\omega}^{e}:\widehat{\mathcal{H}}\to\widehat{\mathcal{Y}}}R^{e}\biggl{\\{}\bar{\omega}^{e}_{ens}\circ\Phi\biggr{\\}},\forall
e\in E_{train}$

where
$\bar{\omega}^{e}_{ens}=\frac{1}{|E_{train}|}\Bigl{(}\bar{\omega}^{e}+\sum_{e^{\prime}\neq
e}w^{e^{\prime}}\Bigr{)}$

###  4.5 Bayesian IRM(BIRM)

In BIRM(Lin et al., 2022), IRMv1 may overfit the training environment if the
training data are insufficient, so we introduced a Bayesian estimation
approach there. Noting that the posterior probability
$p(\omega^{e}|\Phi(X^{e}),Y^{e})$ is invariant in all environments given the
data mapped by $\Phi$ and the correct label if the model is able to acquire
invariant features, the penalty of IRMv1 is modified. By using Bayesian
estimation instead of point estimation, they aim to improve the generalization
performance of the model by taking into account the uncertainty of the data.

##  5 Experiments

Although the approximation methods in the previous chapter have achieved some
success in terms of OOD accuracy on OOD datasets (Arjovsky et al., 2020)(Ahuja
et al., 2022)(Chen et al., 2023)(Ahuja et al., 2020)(Lin et al., 2022), it is
not known how invariant features they actually acquire and how robust they are
to unknown environments. Therefore, we evaluated them by comparing their
calibration performance with ECE.

###  5.1 Setting

We compared ECE of each model on four different OOD-datasets. Specifically,
the datasets used were Colored MNIST (CMNIST)(Arjovsky et al., 2020), Rotated
MNIST (RMNIST)(Ghifary et al., 2015), PACS(Li et al., 2017), and VLCS(Fang et
al., 2013), sourced from the DomainBed benchmark(Gulrajani & Lopez-Paz, 2020).
These datasets can be distinguished based on the type of distribution shift
they represent(Ye et al., 2022). The distribution shift in CMNIST is known as
the Correlation Shift, where the conditional probability of labels given
spurious features changes across environments. The other three
datasets—RMNIST, PACS, and VLCS—are classified under Diversity Shift, where
the prior probability of spurious features changes across environments.

For optimization, Adam was used consistently across all models, and the tuning
of learning rates and hyperparameters for each method was conducted in
accordance with their respective papers.

###  5.2 Correlation Shift

####  5.2.1 Dataset

CMNIST is a dataset based on MNIST, containing 70,000 examples of dimension
(1, 28, 28). It is adapted for a binary classification task where digits less
than 5 are labeled as class 0, and those 5 or above as class 1. Spurious
correlations are added by assigning colors (red or green) to the digits, with
the proportion of the two colors varying in each class depending on the
environment. Denoting the proportion of green in class 0 as environment $e$,
then the training environments are set as $e=[0.1,0.2]$, and the test
environment as $e=[0.9]$.

####  5.2.2 Results

ERM IRMv1 BIRM IB-IRM IRM Game PAIR 100% PREDS 50% PREDS

Figure 2:  Comparison of OOD accuracy (horizontal axis) and OOD ECE (vertical
axis) in CMNIST. The red solid line represents the theoretical values when all
model’s predictive probabilities are 100%, and the blue solid line represents
the case when all predictive probabilities are 50%. IRMv1, IB-IRM, and BIRM,
which achieve relatively high inference accuracy, are distributed near the
blue line. This indicates that these models mitigate reliance on non-invariant
features by making predictions more ambiguous.

(a) OOD Accuracy

(b) OOD ECE

1 10 100 1000 10000

$\lambda$ in IRMv1 Figure 3:  Comparison of the model’s OOD ECE in CMNIST,
with various values of $\lambda$ in the formulation of IRMv1 6. It was
observed that increasing $\lambda$, thereby increasing the regularization
penalty in IRMv1, results in a lower ECE for the model in OOD environment.

Fig. 2 presents a scatter plot showing the relationship between OOD accuracy
(horizontal axis) and ECE (vertical axis) for various methods, each trained
with multiple hyperparameters. The red solid line represents the theoretical
value when all model’s predictive probabilities are 100%, indicating very high
confidence, while the blue solid line represents the case when all predictive
probabilities are 50%, indicating very low and ambiguous confidence. The
methods with relatively high accuracy (IRMv1(green), IB-IRM(light blue),
BIRM(red)) align with the blue solid line, showing that they can overcome
excessive fitting to non-invariant features by making predictions more
ambiguous. On the other hand, ERM(blue) and IRM Game(purple) tend to be
distributed near the red solid line, indicating predictions with relatively
high confidence. PAIR(yellow) is distributed between the two solid lines,
demonstrating characteristics that are intermediate between the two
aforementioned groups.

Figure 3 visualizes how the penalty introduced in IRMv1 for the purpose of
out-of-distribution generalization actually affects the ECE. It specifically
compares different values of $\lambda$ in IRMv1’s formulation 6, demonstrating
that as the impact of IRMv1’s penalty increases, the ECE decreases, suggesting
better calibration. Additionally, the effects of the two regularization
penalties in IB-IRM are similarly shown in the Appendix, Figures 6 and 7. In
each case, comparisons were made by varying the magnitude of the
regularization coefficient of the focused penalty while keeping the other
penalty sufficiently influential. These results, like those in Figure 3,
suggest that each penalty contributes to the improvement of the model’s ECE.

###  5.3 Diversity Shift

####  5.3.1 Datasets

The details for RMNIST, PACS, and VLCS are as follows:

RMNIST: RMNIST is a dataset based on MNIST, containing 70,000 examples of the
same dimension (1, 28, 28) and includes 10 classes. The environments are
distinguished by the rotation angle of the digits, denoted as $e$. The
training environments are set at
$e=[15^{\circ},30^{\circ},45^{\circ},60^{\circ},75^{\circ}]$, while the test
environment is at $e=[0^{\circ}]$.

PACS: PACS is a dataset containing 9,991 examples with dimensions (3, 224,
224) and includes 7 classes. The environments are distinguished by four
different styles of images. Specifically, the training environments are set as
$e=[Cartoons,Photos,Sketches]$, and the test environment is set as $e=[Art]$.

VLCS: VLCS is a dataset containing 10,729 examples with dimensions (3, 224,
224) and includes 5 classes. There are four types of environments. The
training environments are set as $e=[Caltech101,LabelMe,SUN09]$, and the test
environment is set as $e=[VOC2007]$.

####  5.3.2 Results

(a) RMNIST

(b) PACS

(c) VLCS

ERM IRMv1 BIRM IB-IRM IRM Game PAIR ID-ECE = OOD-ECE

Figure 4:  Comparison of the relationship between ECE in the training
environment (horizontal axis) and the test environment (vertical axis). The
red solid line represents the case where the ECE is equal in both
environments. It was observed that IB-IRM (in light blue) is distributed near
the red solid line, indicating a tendency not to overfit to the training
environment compared to other methods.

| ERM | IRMv1 | IB-IRM | BIRM | IRM Game | PAIR
---|---|---|---|---|---|---
RMNIST | ACC | 91.1$\pm$1.0 | 91.9$\pm$1.0 | 90.9$\pm$0.9 | 89.1$\pm$0.4 | 90.8$\pm$1.0 | 87.4$\pm$0.8
ECE | 5.27$\pm$1.11 | 3.98$\pm$0.69 | 1.58$\pm$0.44 | 6.93$\pm$0.56 | 7.17$\pm$0.98 | 7.88$\pm$0.79
PACS | ACC | 73.6$\pm$2.0 | 74.9$\pm$1.9 | 75.9$\pm$0.9 | 71.5$\pm$1.7 | 73.2$\pm$1.3 | 76.3$\pm$1.2
ECE | 11.99$\pm$1.86 | 13.35$\pm$1.45 | 12.85$\pm$0.60 | 13.73$\pm$1.93 | 12.50$\pm$1.45 | 12.46$\pm$0.95
VLCS | ACC | 70.3$\pm$1.0 | 70.5$\pm$1.2 | 69.4$\pm$0.7 | 68.8$\pm$0.8 | 69.9$\pm$1.2 | 71.1$\pm$1.2
ECE | 12.53$\pm$1.42 | 11.90$\pm$1.26 | 7.84$\pm$1.55 | 10.57$\pm$1.81 | 8.70$\pm$0.93 | 10.34$\pm$1.11
Avg. | ACC | 78.3$\pm$1.3 | 79.1$\pm$1.4 | 78.8$\pm$1.12 | 76.5$\pm$0.8 | 78.1$\pm$1.2 | 78.3$\pm$1.1
ECE | 9.93$\pm$1.46 | 9.74$\pm$1.13 | 7.42$\pm$0.86 | 10.41$\pm$1.43 | 9.46$\pm$1.12 | 10.23$\pm$0.95

Table 2:  Comparison of OOD accuracy and ECE across RMNIST, PACS, and VLCS
datasets. Following hyperparameter tuning for each method, measurements were
taken across three different random seeds, presenting the mean and variance
for each metric. It was observed that on average, IB-IRM exhibits a lower ECE
while maintaining comparable inference performance.  Figure 5:  The
relationship between out-of-distribution accuracy and ECE varies with the
$\lambda$ parameter of IRMv1 in the RMNIST dataset. As the color shifts from
blue to green, $\lambda$ increases, indicating stronger regularization in the
model. Points closer to blue, situated on the middle right of the figure,
correspond to intermediate ECE and high accuracy. In contrast, points nearer
to green, located towards the lower left, indicate better ECE but lower
accuracy. This suggests that in more practical datasets, such as those with
diversity shifts, stronger regularization may improve ECE, but this often
comes at the cost of a significant decrease in accuracy.

Considering that a model’s OOD accuracy affects its calibration (Ovadia et
al., 2019), a comparative evaluation of ECE combined with OOD accuracy was
conducted for a fairer comparison of ECE. A threshold was set for validation
accuracy in the training environment, and upon reaching this threshold, early
stopping of training was implemented to create conditions of comparable OOD
accuracy.

The average results from three different random seeds under the aforementioned
conditions are presented in Table 2, where although all methods achieved a
similar average OOD accuracy, notable differences were observed in OOD ECE.
Specifically, the ECE of IB-IRM was on average lower than that of any other
method. Additionally, Figure 4 visualizes the relationship between ECE in the
training and test environments, with the red solid line representing cases
where ECE is equal in both environments. Ideally, models should be distributed
near this red line from the perspective of cross-environmental calibration of
uncertainty. Specifically, in Figures 4(a) and 4(c), IB-IRM is distributed
near the red solid line compared to other methods, suggesting that it does not
overfit to the training environment and exhibits more ideal feature learning.

It was also suggested that regularization in models such as IRMv1 and IBIRM,
while reducing OOD ECE in cases of Diversity Shift, can adversely affect
accuracy. Figure 5 illustrates the relationship between OOD accuracy and ECE
in RMNIST when varying the magnitude of $\lambda$ in IRMv1. As regularization
strengthens, the model’s ECE improves; however, this is accompanied by a
drastic deterioration in accuracy, indicating a trade-off between the two.

##  6 Discussion and Conclusion

In this paper, focusing on IRM, which is challenging to realize due to the
difficulties of bi-level optimization, we conducted comparative evaluations
using ECE, a computationally feasible metric for calibration, to better
understand its approximation methods.

The experimental results suggested that IB-IRM, which applies the information
bottleneck method to IRM, is superior in terms of calibration. It is
hypothesized that IB-IRM prevents excessively high confidence levels caused by
dependence on environment-specific features by controlling the complexity of
the feature extractor $\Phi(X)$. Additionally, it was suggested that
regularization penalties introduced for the purpose of OOD generalization are
effective in achieving lower OOD ECE. However, on the other hand, it was
observed that in more practical datasets (Diversity Shift), while the ECE of
the model becomes lower, the OOD accuracy worsens. This indicates a trade-off
between OOD ECE and accuracy, suggesting that careful balancing of these two
metrics is important in aiming for OOD generalization.

As a future challenge, it has become clear that the degree of invariant
feature learning cannot be fully measured by accuracy in out-of-environment
data alone, necessitating the establishment of broader generalization metrics
and measurement methods. Furthermore, a better understanding of the
relationship between OOD generalization and calibration, and the development
of new approximation methods for IRM based on this understanding, are
anticipated.

## Acknowledgments

This work used computational resources TSUBAME3.0 supercomputer provided by
Tokyo Institute of Technology through the TSUBAME Encouragement Program for
Younger Users awarded to Kotaro Yoshida.

## References

  * Ahuja et al. (2020) Kartik Ahuja, Karthikeyan Shanmugam, Kush R. Varshney, and Amit Dhurandhar.  Invariant risk minimization games, 2020.
  * Ahuja et al. (2022) Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish.  Invariance principle meets information bottleneck for out-of-distribution generalization, 2022.
  * Arjovsky (2020) Martin Arjovsky.  _Out of distribution generalization in machine learning_.  PhD thesis, New York University, 2020.
  * Arjovsky et al. (2020) Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization, 2020.
  * Chen et al. (2023) Yongqiang Chen, Kaiwen Zhou, Yatao Bian, Binghui Xie, Bingzhe Wu, Yonggang Zhang, Kaili Ma, Han Yang, Peilin Zhao, Bo Han, and James Cheng.  Pareto invariant risk minimization: Towards mitigating the optimization dilemma in out-of-distribution generalization, 2023.
  * Fang et al. (2013) Chen Fang, Ye Xu, and Daniel N. Rockmore.  Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias.  In _2013 IEEE International Conference on Computer Vision_ , pp. 1657–1664, 2013.  doi: 10.1109/ICCV.2013.208.
  * Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky.  Domain-adversarial training of neural networks.  _Journal of Machine Learning Research_ , 17(59):1–35, 2016.  URL http://jmlr.org/papers/v17/15-239.html.
  * Geirhos et al. (2020) Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11):665–673, November 2020.  ISSN 2522-5839.  doi: 10.1038/s42256-020-00257-z.  URL http://dx.doi.org/10.1038/s42256-020-00257-z.
  * Ghifary et al. (2015) Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi.  Domain generalization for object recognition with multi-task autoencoders, 2015.
  * Gulrajani & Lopez-Paz (2020) Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization, 2020.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.  On calibration of modern neural networks, 2017.
  * Krueger et al. (2021) David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville.  Out-of-distribution generalization via risk extrapolation (rex), 2021.
  * Li et al. (2017) Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales.  Deeper, broader and artier domain generalization, 2017.
  * Lin et al. (2022) Yong Lin, Hanze Dong, Hao Wang, and Tong Zhang.  Bayesian invariant risk minimization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 16021–16030, June 2022.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift, 2019.
  * Quionero-Candela et al. (2009) Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence.  _Dataset Shift in Machine Learning_.  The MIT Press, 2009.  ISBN 0262170051.
  * Vapnik (1991) Vladimir Naumovich Vapnik.  Principles of risk minimization for learning theory.  In _Neural Information Processing Systems_ , 1991.  URL https://api.semanticscholar.org/CorpusID:15348764.
  * Wald et al. (2021) Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit.  On calibration and out-of-domain generalization.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=XWYJ25-yTRS.
  * Ye et al. (2021) Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li.  Ood-bench: Benchmarking and understanding out-of-distribution generalization datasets and algorithms.  _ArXiv_ , abs/2106.03721, 2021.  URL https://api.semanticscholar.org/CorpusID:235358632.
  * Ye et al. (2022) Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu.  Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization, 2022.
  * Zhang et al. (2023) Yihua Zhang, Pranay Sharma, Parikshit Ram, Mingyi Hong, Kush Varshney, and Sijia Liu.  What is missing in irm training and evaluation? challenges and solutions, 2023.

###  .1 Additional Results

Figure 3 in CMNIST compares the OOD ECE of the model when varying the
regularization penalty $\lambda$ in IRMv1. Additionally, it compares the OOD
ECE of the model when varying the coefficients of the two regularization
penalties in IBIRM 7. Similar to 3, as the regularization penalty increases,
the OOD ECE decreases.

(a) OOD Accuracy

(b) OOD ECE

1 10 100 1000 10000

$\gamma$ in IBIRM Figure 6:  Comparison of the model’s out-of-distribution ECE
in CMNIST, with various values of $\gamma$ and $\lambda$ fixed at a
sufficiently large value (10000) in the formulation of IBIRM 7. It was
observed that as $\gamma$ increases, i.e., as the regularization penalty from
the information bottleneck method becomes more significant, the ECE of the
model in out-of-distribution scenarios decreases.

(a) OOD Accuracy

(b) OOD ECE

1 10 100 1000 10000

$\lambda$ in IBIRM Figure 7:  Comparison of the model’s out-of-distribution
ECE in CMNIST, with various values of $\lambda$ and $\gamma$ fixed at a
sufficiently large value (10000) in the formulation of IBIRM 7. It was
observed that as $\lambda$ increases, i.e., as the regularization penalty from
IRMv1 in IB-IRM becomes more significant, the ECE of the model in out-of-
distribution decreases.


# Investigation of nodal domains in the chaotic microwave ray-splitting rough
billiard

Oleh Hul, Nazar Savytskyy, Oleg Tymoshchuk, Szymon Bauch and Leszek Sirko
Institute of Physics, Polish Academy of Sciences, Aleja Lotników 32/46, 02-668
Warszawa, Poland

(October 20, 2005)

###### Abstract

We study experimentally nodal domains of wave functions (electric field
distributions) lying in the regime of Shnirelman ergodicity in the chaotic
microwave half-circular ray-splitting rough billiard. For this aim the wave
functions $\Psi_{N}$ of the billiard were measured up to the level number
$N=415$. We show that in the regime of Shnirelman ergodicity ($N>208$) wave
functions of the chaotic half-circular microwave ray-splitting rough billiard
are extended over the whole energy surface and the amplitude distributions are
Gaussian. For such ergodic wave functions the dependence of the number of
nodal domains $\aleph_{N}$ on the level number $N$ was found. We show that in
the limit $N\rightarrow\infty$ the least squares fit of the experimental data
yields $\aleph_{N}/N\simeq 0.063\pm 0.023$ that is close to the theoretical
prediction $\aleph_{N}/N\simeq 0.062$. We demonstrate that for higher level
numbers $N\simeq 215-415$ the variance of the mean number of nodal domains
$\sigma^{2}_{N}/N$ is scattered around the theoretical limit
$\sigma^{2}_{N}/N\simeq 0.05$. We also found that the distribution of the
areas $s$ of nodal domains has power behavior $n_{s}\propto s^{-\tau}$, where
the scaling exponent is equal to $\tau=2.14\pm 0.12$. This result is in a good
agreement with the prediction of percolation theory.

###### pacs:

05.45.Mt,05.45.Df

In recent theoretical papers by Bogomolny and Schmit Bogomolny2002  and Blum
et al. Blum2002  the distributions of the nodal domains of real wave functions
$\Psi(x,y)$ in 2D quantum systems (billiards) have been considered. Nodal
domains are regions where a wave function $\Psi(x,y)$ has a definite sign. The
condition $\Psi(x,y)=0$ determines a set of nodal lines which separate nodal
domains. Bogomolny and Schmit Bogomolny2002  have proposed a very fruitful,
percolationlike, model for description of properties of the nodal domains of
generic chaotic system. Using this model they have shown that the distribution
of nodal domains of quantum wave functions of chaotic systems is universal.
Blum et al. Blum2002  have shown that the systems with integrable and chaotic
underlying classical dynamics can be distinguished by different distributions
of the number of nodal domains. In this way they provided a new criterion of
quantum chaos, which is not directly related to spectral statistics.

Theoretical findings of Bogomolny and Schmit Bogomolny2002  and Blum et al.
Blum2002  have been recently tested in the experiment with the microwave half-
circular rough billiard by Savytskyy et al. Savytskyy2004 .

In this paper we present the first experimental investigation of nodal domains
of wave functions of the chaotic microwave ray-splitting rough billiard. Ray-
splitting systems are a new class of chaotic systems in which the underlying
classical mechanics is non-Newtonian and non-deterministic BLUM96 ; SIR97 ;
BLUMEL2001 . In ray-splitting systems a wave which encounters a discontinuity
in the propagation medium splits into two or more rays travelling usually away
from the discontinuity. Ray splitting occurs in many fields of physics,
whenever the wave length is large in comparison with the range over which the
potential changes. Ideal model systems for the investigation of ray-splitting
phenomena are ray-splitting billiards COUC92 ; BLUMEL2001  and microwave
cavities with dielectric inserts SIR97 ; BAUCH98 ; HLUSH2000 ; Savytskyy2001 .
Measurements of wave functions of ray-splitting systems are very demanding
because in principle they require the direct access to the all parts of the
system Stoeckmann2001  including those filled with ray-splitting media, such
as dielectric in the case of ray-splitting microwave billiards. This is one of
the main reasons for which only low wave functions ($N\leq 100$) of ray-
splitting billiards have been measured so far Stoeckmann2001 . In this paper
we use a new method of the reconstruction of wave functions introduced by
Savytskyy and Sirko Savytskyy2002  which in the case of the half-circular
microwave ray-splitting rough billiard allowed for the reconstruction of wave
functions with the level numbers $N\leq 415$.

Figure 1: Sketch of the chaotic half-circular microwave ray-splitting rough
billiard which consists a half-circular Teflon insert of radius $R_{d}=8.465$
cm. Dimensions are given in cm. The cavity sidewalls are marked by 1 and 2
(see text). Squared wave functions $|\Psi_{N}(R_{c},\theta)|^{2}$ were
evaluated on a half-circle of fixed radius $R_{c}=19.25$ cm. Billiard’s rough
boundary is marked by $\Gamma$.

In the experiment we used the thin (height $h=8$ mm) aluminium cavity in the
shape of a rough half-circle (Fig. 1) which consisted a half-circular Teflon
insert of radius $R_{d}=8.465$ cm. The insert had the same height as the rough
cavity. The microwave cavity simulates the rough ray-splitting quantum
billiard due to the equivalence between the Schrödinger equation and the
Helmholtz equation BLUMEL2001 . This equivalence remains valid for frequencies
less than the cut-off frequency $\nu_{c}=c/2\eta h\simeq 13.1$ GHz, where c is
the speed of light and $\eta=1.425$ is the index of refraction of the Teflon
insert.

The cavity sidewalls were made of two segments. The rough segment 1 is
described by the radius function
$R(\theta)=R_{0}+\sum_{m=2}^{M}{a_{m}\sin(m\theta+\phi_{m})}$, where the mean
radius $R_{0}$=20.0 cm, $M=20$, $a_{m}$ and $\phi_{m}$ are uniformly
distributed on [0.084,0.091] cm and [0,2$\pi$], respectively, and
$0\leq\theta<{\pi}$. It is important to note that we used a rough half-
circular cavity instead of a rough circular cavity because in this way we
avoided nearly degenerate low-level eigenvalues Hlushchuk01b ; Hlushchuk01 .
Additionally, a half-circular geometry of the cavity was necessary for the
accurate measurements of the electric field distributions inside the billiard.

According to Frahm97  the roughness of a billiard may be characterized by the
function $k(\theta)=(dR/d\theta)/R_{0}$. The roughness parameter $\tilde{k}$
defined as the angle average of the function $k(\theta)$ was for our billiard
$\tilde{k}=(\left<k^{2}(\theta)\right>_{\theta})^{1/2}\simeq 0.200$. In such a
billiard the dynamics is diffusive in orbital momentum due to collisions with
the rough boundary because the roughness parameter $\tilde{k}$ is much larger
the chaos border parameter $k_{c}=M^{-5/2}=0.00056$ Frahm97 . The roughness
parameter $\tilde{k}$ determines also other properties of the billiard Frahm .
The eigenstates are localized for the level number
$N<N_{e}=1/128\tilde{k}^{4}=5$. The border of Breit-Wigner regime is given by
$N_{W}=M^{2}/48\tilde{k}^{2}\simeq 208$. It means that between $N_{e}<N<N_{W}$
Wigner ergodicity Frahm  ought to be observed and for $N>N_{W}$ Shnirelman
ergodicity should emerge. In the regime of Shnirelman ergodicity wave
functions have to be uniformly spread out in the billiard Shnirelman . In this
paper we focus our attention on Shnirelman ergodicity regime.

It is worth noting that rough billiards and related systems are of
considerable interest elsewhere, e.g. in the context of microdisc lasers
Yamamoto ; Stone , light scattering in optical fibers Doya2002 , ballistic
electron transport in microstructures Blanter , dynamic localization Sirko00
and localization in discontinuous quantum systems Borgonovi .

In order to measure the wave functions (electric field distributions inside
the microwave billiard), which are indispensable in investigation of nodal
domains, we used a new, very effective method described in Savytskyy2002 . It
is based on the perturbation technique and construction of the “trial
functions”.

Following Savytskyy2002  we will show that the wave functions
$\Psi_{N}(r,\theta)$ (electric field distribution $E_{N}(r,\theta)$ inside the
cavity) of the billiard can be determined from the form of electric field
$E_{N}(R_{c},\theta)$ evaluated on a half-circle of fixed radius $R_{c}$ (see
Fig. 1).

The first step in evaluation of $E_{N}(R_{c},\theta)$ is measurement of
$|E_{N}(R_{c},\theta)|^{2}$. For this purpose the perturbation technique
developed in Slater52  and used successfully in Slater52 ; Sridhar91 ;
Richter00 ; Anlage98  was applied. In this method a small perturber is
introduced inside the cavity to alter its resonant frequency according to

$\nu-\nu_{N}=\nu_{N}(aB_{N}^{2}-bE_{N}^{2}),$ $None$

where $\nu_{N}$ is the $N$th resonant frequency of the unperturbed cavity, $a$
and $b$ are geometrical factors. Equation (1) can be used to evaluate
$E_{N}^{2}$ only when the term containing magnetic field $B_{N}$ is
sufficiently small. In order to minimize the influence of $B_{N}$ on the
frequency shift $\nu-\nu_{N}$ a small piece of a metallic pin (3.0 mm in
length and 0.25 mm in diameter) was used as a perturber. The perturber was
attached to the micro filament line hidden in the groove (0.4 mm wide, 1.0 mm
deep) made in the cavity’s bottom wall along the half-circle $R_{c}$ and moved
by the stepper motor. Application of such a small pin perturber reduced the
largest positive frequency shifts to the uncertainty of frequency shift
measurements (15 kHz). It was verified that the presence of the narrow groove
in the bottom wall of the cavity caused only very small changes
$\delta\nu_{N}$ of the eigenfrequencies $\nu_{N}$ of the cavity
$|\delta\nu_{N}|/\nu_{N}\leq 10^{-4}$. Therefore, its influence into the
structure of the cavity’s wave functions was also negligible. A big advantage
of using the perturber that was attached to the line, was connected with the
fact that the perturber was always vertically positioned, which is crucial in
the measurements of the square of electric field $E_{N}$. The influence of the
thermal expansion of the Teflon insert and the aluminium cavity into its
resonant frequencies was eliminated by stabilization of the temperature of the
cavity with the accuracy of $0.05^{\circ}$.

Figure 2: Panel (a): Squared wave function $|\Psi_{415}(R_{c},\theta)|^{2}$
(in arbitrary units) measured on a half-circle with radius $R_{c}=19.25$ cm
($\nu_{415}\simeq 12.98$ GHz). Panel (b): The “trial wave function”
$\Psi_{415}(R_{c},\theta)$ (in arbitrary units) with the correctly assigned
signs, which was used in the reconstruction of the wave function
$\Psi_{415}(r,\theta)$ of the billiard (see Fig. 3).

The regime of Shnirelman ergodicity for the experimental rough billiard is
defined for $N>208$. Using a field perturbation technique we measured squared
wave functions $|\Psi_{N}(R_{c},\theta)|^{2}$ for 30 modes within the region
$215\leq N\leq 415$. The range of corresponding eigenfrequencies was from
$\nu_{215}\simeq 9.42$ GHz to $\nu_{415}\simeq 12.98$ GHz. The measurements
were performed at 0.36 mm steps along a half-circle with fixed radius
$R_{c}=19.25$ cm. This step was small enough to reveal in details the space
structure of high-lying levels. In Fig. 2 (a) we show the example of the
squared wave function $|\Psi_{N}(R_{c},\theta)|^{2}$ evaluated for the level
number $N=415$. The perturbation method used in our measurements allows us to
extract information about the wave function amplitude
$|\Psi_{N}(R_{c},\theta)|$ at any given point of the cavity but it doesn’t
allow to determine the sign of $\Psi_{N}(R_{c},\theta)$ Stein95 . However, the
determination of the sign of the wave function $\Psi_{N}(R_{c},\theta)$ is
crucial in the procedure of the reconstruction of the full wave function
$\Psi_{N}(r,\theta)$ of the billiard. The papers Savytskyy2002 ; Savytskyy2004
suggest the following sign-assignment strategy. First one should identify of
all close to zero minima of $|\Psi_{N}(R_{c},\theta)|$. Then the sign “minus”
is arbitrarily assigned to the region between the first and the second
minimum, “plus” to the region between the second minimum and the third one and
so on. In this way the “trial wave function” $\Psi_{N}(R_{c},\theta)$ is
constructed. If the assignment of the signs is correct the wave function
$\Psi_{N}(r,\theta)$ should be reconstructed inside the billiard with the
boundary condition $\Psi_{N}(r_{\Gamma},\theta_{\Gamma})=0$.

The wave function of a rough ray-splitting half-circular billiard outside of
the half-circular Teflon insert ($r\geq R_{d}$) may be expanded in terms of
Hankel functions

$\Psi^{out}_{N}(r,\theta)=\sum_{s=1}^{L}a_{s}\Omega_{s}(k_{N}r)\sin(s\theta),$
$None$

where $\Omega_{s}(x)=Re\\{H^{(2)}_{s}(x)+S_{ss}(k_{N}R_{d})H^{(1)}_{s}(x)\\}$
and $k_{N}=2\pi\nu_{N}/c$. $H^{(1)}_{s}(x)$ and $H^{(2)}_{s}(x)$ are Hankel
functions of the first and the second kind, respectively. The matrix
$S_{ss^{\prime}}(k_{N}R_{d})$ is defined as follows Hentschel2002

$S_{ss^{\prime}}(k_{N}R_{d})=-\frac{H^{(2)^{\prime}}_{s}(k_{N}R_{d})-\eta[J^{\prime}_{s}(\eta
k_{N}R_{d})/J_{s}(\eta
k_{N}R_{d})]H^{(2)}_{s}(k_{N}R_{d})}{H^{(1)^{\prime}}_{s}(k_{N}R_{d})-\eta[J^{\prime}_{s}(\eta
k_{N}R_{d})/J_{s}(\eta
k_{N}R_{d})]H^{(1)}_{s}(k_{N}R_{d})}\delta_{ss^{\prime}},$ $None$

where the derivatives of Hankel and Bessel functions are marked by primes. In
Eq. (2) the number of basis functions is limited to $L=k_{N}r_{max}+3$, where
$r_{max}=20.7$ cm is the maximum radius of the cavity.
$l_{N}^{max}=k_{N}r_{max}$ is a semiclassical estimate for the maximum
possible angular momentum for a given $k_{N}$. The functions with angular
momentum $s>l_{N}^{max}$ describe evanescent waves. We checked that the basis
of $L$ wave functions was large enough to properly reconstruct billiard’s wave
functions. The coefficients $a_{s}$ may be determined from the “trial wave
functions” $\Psi_{N}(R_{c},\theta)$ via

$a_{s}=[\frac{\pi}{2}\Omega_{s}(k_{N}R_{c})]^{-1}\int_{0}^{\pi}\Psi_{N}(R_{c},\theta)sin(s\theta)d\theta.$
$None$

The wave functions of the billiard inside the Teflon insert ($r\leq R_{d}$)
may be expanded in terms of circular waves

$\Psi^{in}_{N}(r,\theta)=\sum_{s=1}^{L^{\prime}}a^{{}^{\prime}}_{s}J_{s}(\eta
k_{N}r)\sin(s\theta).$ $None$

In Eq. (5) the number of basis functions was limited to $L^{\prime}=\eta
k_{N}R_{d}$. The coefficients $a_{s}$ given by Eq. (4) and the continuity
condition fulfilled at the border of the dielectric insert
$\Psi^{out}_{N}(R_{d},\theta)=\Psi^{in}_{N}(R_{d},\theta)$ may be used to
evaluate the coefficients $a^{{}^{\prime}}_{s}$ in Eq. (5) allowing in this
way to reconstruct the full wave function $\Psi_{N}(r,\theta)$ of the
billiard.

In the evaluation of the coefficients $a^{{}^{\prime}}_{s}$ in Eq. (5) an
important role plays the value of the refraction index $n$ of the Teflon
insert. We measured the refraction index $\eta=1.425\pm 0.002$ of Teflon by
measuring the set of resonant frequencies of a microwave circular cavity of
radius $R_{T}=3.25$ cm entirely filled by it.

Figure 3:  The reconstructed wave function $\Psi_{415}(r,\theta)$ of the
chaotic half-circular microwave rough billiard. The amplitudes have been
converted into a grey scale with white corresponding to large positive and
black corresponding to large negative values, respectively. Dimensions of the
billiard are given in cm. The position of the half-circular Teflon insert of
radius $R_{d}=8.465$ cm is marked with a solid line.

Using the method of the “trial wave function” we were able to reconstruct 30
experimental wave functions of the rough half-circular billiard with the level
number $N$ between 215 and 415\. The wave functions were reconstructed on
points of a square grid of side $4.2\cdot 10^{-4}$ m. As the quantitative
measure of the sign assignment quality we chose the integral
$\gamma\int_{\Gamma}|\Psi_{N}(r,\theta)|^{2}dl$ calculated along the
billiard’s rough boundary $\Gamma$, where $\gamma$ is length of $\Gamma$. In
Fig. 2 (b) we show the “trial wave function” $\Psi_{415}(R_{c},\theta)$ with
the correctly assigned signs, which was used in the reconstruction of the wave
function $\Psi_{415}(r,\theta)$ of the billiard (see Fig. 3). It is worth
noting that inside of the Teflon insert the size of nodal domains are much
smaller than outside of it. The remaining wave functions from the range
$N=215-415$ were not reconstructed because of the accidental near-degeneration
of the neighboring states or due to the problems with the measurements of
$|\Psi_{N}(R_{c},\theta)|^{2}$ along a half-circle coinciding for its
significant part with one or several of the nodal lines of
$\Psi_{N}(r,\theta)$. The problem of the near-degenerated states is important
because in the presence of the perturber the resonances are shifted, which may
cause the initially non-overlapping states to become near-degenerated at
certain positions of the perturber. Such a situation prevents us from the
reconstruction of the wave functions. The problems mentioned are getting much
more severe for $N>200$. Furthermore, the computation time $t_{r}$ required
for reconstruction of the ”trial wave function” scales like $t_{r}\propto
2^{n_{z}-2}$, where $n_{z}$ is the number of identified zeros in the measured
function $|\Psi_{N}(R_{c},\theta)|$.

Figure 4: Structure of the energy surface in the regime of Shnirelman
ergodicity. Here we show the moduli of amplitudes $|C^{(N)}_{nl}|$ for the
wave functions: (a) $N=215$, (b) $N=415$. The wave functions are delocalized
in the $n,l$ basis. Full lines show the energy surface (see text).

The structure of the energy surface Frahm97  of the billiard’s wave functions
plays an important role in the identification of their ergodicity. To check it
we extracted wave function amplitudes $C^{(N)}_{nl}=\left<n,l|N\right>$ in the
basis $n,l$ of a half-circular ray-splitting billiard (desymmetrized annular
ray-splitting billiard) Kohler1998  with radius $r_{max}$ and a half-circular
Teflon insert of radius $R_{d}$ . The normalized eigenfunctions of the half-
circular ray-splitting billiard are given by

$\Phi_{nl}(r,\theta)=\left\\{\begin{array}[]{cc}A_{ln}J_{l}(\eta\kappa_{ln}r)\sin(l\theta),&0\leq
r\leq R_{d},\\\
A_{ln}\left[C_{ln}J_{l}(\kappa_{ln}r)+D_{ln}Y_{l}(\kappa_{ln}r)\right]\sin(l\theta),&R_{d}\leq
r\leq r_{max},\end{array}\right.$ $None$

where
$A_{ln}=\left\\{\frac{\pi}{2}\left(\int_{0}^{R_{d}}rJ_{l}(\eta\kappa_{ln}r)^{2}dr+\int_{R_{d}}^{r_{max}}r\left[C_{ln}J_{l}(\kappa_{ln}r)+D_{ln}Y_{l}(\kappa_{ln}r)\right]^{2}dr\right)\right\\}^{-\frac{1}{2}}$.
$J_{l}(\kappa_{ln}r)$ and $Y_{l}(\kappa_{ln}r)$ are Bessel and Neumann
functions, respectively. The main quantum number $n=1,2,3\ldots$ enumerates
the zeros $y_{ln}=\kappa_{ln}r_{max}$ of the radial function

$C_{ln}J_{l}(y_{ln})+D_{ln}Y_{l}(y_{ln})=0,$ $None$

and $l=1,2,3\ldots$ is the angular momentum quantum number. The coefficients
$C_{ln}$ and $D_{ln}$ can be determined from the continuity conditions of the
wave function $\Phi_{nl}(r,\theta)$ and it’s derivative
$\Phi_{nl}^{{}^{\prime}}(r,\theta)$ on Teflon’s boundary $R_{d}$

$\left\\{\begin{array}[]{cc}J_{l}(\eta\kappa_{ln}R_{d})=C_{ln}J_{l}(\kappa_{ln}R_{d})+D_{ln}Y_{l}(\kappa_{ln}R_{d}),\\\
\eta
J_{l}^{{}^{\prime}}(\eta\kappa_{ln}R_{d})=C_{ln}J_{l}^{{}^{\prime}}(\kappa_{ln}R_{d})+D_{ln}Y_{l}^{{}^{\prime}}(\kappa_{ln}R_{d}).\end{array}\right.$
$None$

The moduli of amplitudes $|C^{(N)}_{nl}|$ and their projections into the
energy surface for the representative experimental wave functions $N=215$ and
$N=415$ are shown in Fig. 4. As expected, in the regime of Shnirelman
ergodicity the wave functions are extended over the whole energy surface
Hlushchuk01 . The full lines on the projection planes in Fig. 4(a) and Fig.
4(b) mark the energy surface of a half-circular annular ray-splitting billiard
$H(n,l)\simeq E_{N}=k^{2}_{N}$ estimated from the formula
$|H(n,l)-E_{N}|/E_{N}\leq 0.12$. The peaks $|C^{(N)}_{nl}|$ are spread almost
perfectly along the lines marking the energy surface.

Figure 5: Panel (a): The amplitude distribution $P(\Psi_{N}A^{1/2})$ for the
wave function $N=215$. Panel (b): The distribution $P(\Psi_{N}A^{1/2})$ for
the wave function $N=415$. The amplitude distributions were constructed as
histograms with bin equal to 0.2. The width of the distribution $P(\Psi)$ was
rescaled to unity by multiplying normalized to unity wave function by the
factor $A^{1/2}$, where $A$ denotes billiard’s area. Full lines show standard
normalized Gaussian prediction $P_{0}(\Psi
A^{1/2})=(1/\sqrt{2\pi})e^{-\Psi^{2}A/2}$.

Ergodic behavior of the measured wave functions can be also tested by
evaluation of the amplitude distribution $P(\Psi_{N})$ Berry77 ; Kaufman88 .
For irregular, chaotic states the probability of finding the value $\Psi_{N}$
at any point inside the billiard should be distributed as a Gaussian,
$P(\Psi_{N})\sim e^{-\beta\Psi_{N}^{2}}$. In Fig. 5(a) we show the amplitude
distribution $P(\Psi_{N}A^{1/2})$ for the wave function $N=215$ while in Fig.
5(b) the distribution $P(\Psi_{N}A^{1/2})$ for the wave function $N=415$ is
presented. The distributions were constructed as normalized to unity
histograms with the bin equal to 0.2. The width of the amplitude distributions
$P(\Psi_{N})$ was rescaled to unity by multiplying normalized to unity wave
functions by the factor $A^{1/2}$, where $A$ denotes billiard’s area (see
formula (23) in Kaufman88 ). For all measured wave functions lying in the
regime of Shnirelman ergodicity the distributions of $P(\Psi_{N}A^{1/2})$ were
in good agreement with the standard normalized Gaussian prediction $P_{0}(\Psi
A^{1/2})=(1/\sqrt{2\pi})e^{-\Psi^{2}A/2}$.

Figure 6: The number of nodal domains $\aleph_{N}$ (full circles) for the
chaotic half-circular microwave ray-splitting rough billiard. Full line shows
the least squares fit $\aleph_{N}=a_{1}N+b_{1}\sqrt{N}$ to the experimental
data (see text), where $a_{1}=0.063\pm 0.023$, $b_{1}=0.77\pm 0.40$. The
prediction of the theory of Bogomolny and Schmit Bogomolny2002  $a_{1}=0.062$.

The number of nodal domains $\aleph_{N}$ vs. the level number $N$ in the
chaotic microwave ray-splitting rough billiard is plotted in Fig. 6. The full
line in Fig. 6 shows the least squares fit $\aleph_{N}=a_{1}N+b_{1}\sqrt{N}$
of the experimental data, where $a_{1}=0.063\pm 0.023$, $b_{1}=0.77\pm 0.40$.
The coefficient $a_{1}=0.063\pm 0.023$ coincides with the prediction of the
percolation model of Bogomolny and Schmit Bogomolny2002  $\aleph_{N}/N\simeq
0.062$ within the error limits. The errors of the coefficients $a_{1}$ and
$b_{1}$ are relatively high because the number of nodal domains fluctuates
significantly in the function of the level number $N$, what was also
demonstrated in Blum2002  (see Fig .(5)). It is worth mention that in the
paper Savytskyy2004  the coefficient $a_{1}$ was estimated in the experiment
with the microwave rough billiard without the ray-splitting Teflon insert. Its
value $a_{1}=0.058\pm 0.006$ was also close to the theoretical prediction. The
second term in the least squares fit corresponds to a contribution of boundary
domains, i.e. domains that include the billiard boundary. Numerical
calculations of Blum et al. Blum2002  performed for the Sinai and stadium
billiards showed that the number of boundary domains scales as the number of
the boundary intersections, that is as $\sqrt{N}$. Present results together
with the results of Savytskyy2004  clearly suggest that in the rough billiards
(with and without ray-splitting), at low level number $N$, the boundary
domains also significantly influence the scaling of the number of nodal
domains $\aleph_{N}$, leading to the departure from the predicted scaling
$\aleph_{N}\sim N$.

Figure 7: The variance of the mean number of nodal domains divided by the
level number $\sigma^{2}_{N}/N$ for the chaotic half-circular microwave ray-
splitting rough billiard. Full line shows predicted by the theory limit
$\sigma^{2}_{N}/N\simeq 0.05$, Bogomolny and Schmit Bogomolny2002 .

Measured wave functions of the ray-splitting billiard may be also used for the
calculations of the variance $\sigma^{2}_{N}$ of the mean number of nodal
domains. It was predicted in Bogomolny2002  that for chaotic wave functions
the variance of the mean number of nodal domains should converge to the
theoretical limit $\sigma^{2}_{N}\simeq 0.05N$. In Fig. 7 the variance of the
mean number of nodal domains divided by the level number $\sigma^{2}_{N}/N$ is
shown for the microwave ray-splitting rough billiard. The variance
$\sigma^{2}_{N}=\frac{1}{N_{w}-1}\sum_{i=1}^{N_{w}}(\aleph_{N_{i}}-\langle\aleph_{N}\rangle)^{2}$
was calculated in the window of $N_{w}=5$ consecutive eigenstates measured
between $215\leq N\leq 415$, where the mean number of nodal domains was
defined as
$\langle\aleph_{N}\rangle=\frac{1}{N_{w}}\sum_{i=1}^{N_{w}}\aleph_{N_{i}}$.
For level numbers $N<300$ the variance $\sigma^{2}_{N}/N$ is above the
predicted theoretical limit, however, for $300<N\leq 415$ it is slightly below
it. A similar erratic behavior of $\sigma^{2}_{N}/N$ was also observed in
Bogomolny2002 .

Figure 8: Distribution of nodal domain areas. Full line shows the prediction
of percolation theory $\log_{10}(\langle
n_{s}/n\rangle)=-\frac{187}{91}\log_{10}(\langle s/s_{min}\rangle)$. The least
squares fit $\log_{10}(\langle n_{s}/n\rangle)=a_{2}-\tau\log_{10}(\langle
s/s_{min}\rangle)$ of the experimental results lying within the vertical lines
yields the scaling exponent $\tau=2.14\pm 0.12$ and $a_{2}=-0.06\pm 0.12$. The
result of the fit is shown by the dashed line.

The percolation model Bogomolny2002  allows for applying the results of
percolation theory to the description of nodal domains of chaotic billiards.
The percolation theory predicts that the distribution of the areas $s$ of
nodal clusters should obey the scaling behavior: $n_{s}\propto s^{-\tau}$. The
scaling exponent Ziff1986  is found to be $\tau=187/91$. In Fig. 8 we present
in logarithmic scales nodal domain areas distribution $\langle n_{s}/n\rangle$
vs. $\langle s/s_{min}\rangle$ obtained for the microwave ray-splitting rough
billiard. The distribution $\langle n_{s}/n\rangle$ was constructed as
normalized to unity histogram with the bin equal to 1. The areas $s$ of nodal
domains were calculated by summing up the areas of the nearest neighboring
grid sites having the same sign of the wave function. In Fig. 8 the vertical
axis $\langle
n_{s}/n\rangle=\frac{1}{N_{T}}\sum_{i=1}^{N_{T}}n_{s}^{(N)}/n^{(N)}$
represents the number of nodal domains $n_{s}^{(N)}$ of size $s$ divided by
the total number of domains $n^{(N)}$ averaged over $N_{T}=30$ wave functions
measured in the range $215\leq N\leq 415$. In these calculations we took into
account only the nodal domains which entirely lied outside or inside of the
Teflon insert for which percolation theory Ziff1986  should be applicable. The
horizontal axis in Fig. 8 is expressed in the units of the smallest possible
area $s_{min}^{(N)}$ Bogomolny2002 , $\langle
s/s_{min}\rangle=\frac{1}{N_{T}}\sum_{i=1}^{N_{T}}s/s_{min}^{(N)}$, where
$s_{min}^{(N)}=\pi(j_{01}/\eta k_{N})^{2}$ and $j_{01}\simeq 2.4048$ is the
first zero of the Bessel function $J_{0}(j_{01})=0$. For nodal domains lying
inside the Teflon insert the refraction index was according to our
measurements $\eta=1.425$ while outside of the insert we assumed $\eta=1$. The
full line in Fig. 8 shows the prediction of percolation theory
$\log_{10}(\langle n_{s}/n\rangle)=-\frac{187}{91}\log_{10}(\langle
s/s_{min}\rangle)$. In a broad range of $\log_{10}(\langle s/s_{min}\rangle)$,
approximately from 0.2 to 1.4, which is marked by the two vertical lines the
experimental results follow closely the theoretical prediction. The least
squares fit $\log_{10}(\langle n_{s}/n\rangle)=a_{2}-\tau\log_{10}(\langle
s/s_{min}\rangle)$ of the experimental results lying within the vertical lines
gives the scaling exponent $\tau=2.14\pm 0.12$ and $a_{2}=-0.06\pm 0.12$,
which is in a good agreement with the predicted $\tau=187/91\simeq 2.05$. The
result of the fit is shown in Fig. 8 by the dashed line.

In summary, for the first time we measured high-lying wave functions of the
chaotic microwave ray-splitting rough billiard. We showed that in the limit
$N\rightarrow\infty$ the least squares fit of the experimental data reveals
the asymptotic number of nodal domains $\aleph_{N}/N\simeq 0.063\pm 0.023$
that is close to the theoretical prediction $\aleph_{N}/N\simeq 0.062$
Bogomolny2002 . We demonstrate that for higher level numbers $N\simeq 215-415$
the variance of the mean number of nodal domains $\sigma^{2}/N$ is scattered
around the theoretical limit $\sigma^{2}/N\simeq 0.05$. Following the results
of percolationlike model proposed by Bogomolny2002  we confirmed that the
distribution of the areas $s$ of nodal domains has power behavior
$n_{s}\propto s^{-\tau}$, where scaling exponent is equal to $\tau=2.14\pm
0.12$. This result is in a good agreement with the prediction of percolation
theory Ziff1986 , which predicts $\tau=187/91\simeq 2.05$. The experimental
results presented in this paper strongly suggest that many properties of nodal
domains in chaotic ray-splitting billiards are the same, like in conventional
chaotic billiards without ray-splitting.

Acknowledgments. This work was supported by Ministry of Science and
Information Society Technologies grant No. 2 P03B 047 24.

## References

  * (1) E. Bogomolny and C. Schmit, Phys. Rev. Lett. 88, 114102-1 (2002).
  * (2) G. Blum, S. Gnutzmann, and U. Smilansky, Phys. Rev. Lett. 88, 114101-1 (2002).
  * (3) N. Savytskyy, O. Hul, and L. Sirko Phys. Rev. E 70, 056209 (2004).
  * (4) R. Blümel, T. M. Antonsen, B. Georgeot, E. Ott, and R. E. Prange, Phys. Rev. Lett. 76, 2476 (1996); Phys. Rev. E 53, 3284 (1996).
  * (5) L. Sirko, P. M. Koch and R. Blümel, Phys. Rev. Lett. 78, 2940 (1997).
  * (6) R. Blümel, P.M. Koch, and L. Sirko, Found. Phys. 31, 269 (2001).
  * (7) L. Couchman, E. Ott, and T. M. Antonsen, Jr., Phys. Rev. A 46, 6193 (1992).
  * (8) N. Savytskyy, A. Kohler, Sz. Bauch, R. Bl mel, and L. Sirko Phys. Rev. E 64, 036211 (2001).
  * (9) Sz. Bauch, A. Błȩdowski, L. Sirko, P. M. Koch, and R. Blümel, Phys. Rev. E 57, 304 (1998).
  * (10) Y. Hlushchuk, A. Kohler, Sz. Bauch, L. Sirko, R. Blümel, M. Barth, and H.-J. Stöckmann, Phys. Rev. E 61, 366 (2000).
  * (11) R. Schäfer, U. Kuhl, M. Barth, and H.-J. Stöckmann, Found. Phys. 31, 475 (2001).
  * (12) N. Savytskyy and L. Sirko, Phys. Rev. E 65, 066202-1 (2002).
  * (13) Y. Hlushchuk, A. Błȩdowski, N. Savytskyy, and L. Sirko, Physica Scripta 64, 192 (2001).
  * (14) Y. Hlushchuk, L. Sirko, U. Kuhl, M. Barth, H.-J. Stöckmann, Phys. Rev. E 63, 046208-1 (2001).
  * (15) K.M. Frahm and D.L. Shepelyansky, Phys. Rev. Lett. 78, 1440 (1997).
  * (16) K.M. Frahm and D.L. Shepelyansky, Phys. Rev. Lett. 79, 1833 (1997).
  * (17) A. Shnirelman, Usp. Mat. Nauk. 29, N6, 18 (1974).
  * (18) Y. Yamamoto and R.E. Sluster, Phys. Today 46(6), 66 (1993).
  * (19) J.U. Nöckel and A.D. Stone, Nature 385, 45 (1997).
  * (20) V. Doya, O. Legrand, and F. Mortessagne, Phys. Rev. Lett. 88, 014102 (2002).
  * (21) Ya. M. Blanter, A.D. Mirlin, and B.A. Muzykantskii, Phys. Rev. Lett. 80, 4161 (1998).
  * (22) L. Sirko, Sz. Bauch, Y. Hlushchuk, P.M. Koch, R. Blümel, M. Barth, U. Kuhl, and H.-J. Stöckmann, Phys. Lett. A 266, 331 (2000).
  * (23) F. Borgonovi, Phys. Rev. Lett. 80, 4653 (1998).
  * (24) L.C. Maier and J.C. Slater, J. Appl. Phys. 23, 68 (1952).
  * (25) S. Sridhar, Phys. Rev. Lett. 67, 785 (1991).
  * (26) C. Dembowski, H.-D. Gräf, A. Heine, R. Hofferbert, H. Rehfeld, and A. Richter, Phys. Rev. Lett. 84, 867 (2000).
  * (27) D.H. Wu, J.S.A. Bridgewater, A. Gokirmak, and S.M. Anlage, Phys. Rev. Lett. 81, 2890 (1998).
  * (28) A. Kohler and R. Blümel, Phys. Lett. A 238, 271 (1998).
  * (29) J. Stein, H.-J. Stöckmann, and U. Stoffregen, Phys. Rev. Lett. 75, 53 (1995).
  * (30) M. Hentschel and K. Richter, Phys. Rev. E 66, 056207 (2002).
  * (31) S.W. McDonald and A.N. Kaufman, Phys. Rev A 37, 3067 (1988).
  * (32) M.V. Berry, J. Phys. A 10, 2083 (1977).
  * (33) R. M. Ziff, Phys. Rev. Lett. 56, 545 (1986).