# Recollections about Växjö conferences. Preface to the special issue “Quantum
Information and Probability: from Foundations to Engineering” (QIP23)

Andrei Khrennikov
Linnaeus University, International Center for Mathematical Modeling
in Physics and Cognitive Sciences Växjö, SE-351 95, Sweden

###### Abstract

As the preface to the special issue for the conference “Quantum Information
and Probability: from Foundations to Engineering” (QIP23), I wrote these notes
with recollection about Växjö conferences. These conferences covered 25 years
of my life (2000-24) and played the crucial role in evolution of my own views
on the basic problems of quantum foundations. I hope to continue this
conference series as long as possible. Up to my understanding, this is the
longest conference series on quantum foundation in the history of quantum
physics. These notes contain my recollections of conversations with the
world’s leading experts on quantum foundations. I think that such notes may
have the historical value. My own views on quantum foundations are specific
and they evolved essentially during 25 years. Finally, I discovered the
practically forgotten pathway in physical foundations developed by von
Helmholtz, Hertz, Boltzmann, and Schrödinger and known as the Bild conception.
A scientific theory is a combination of two models, observational and causal.
Coupling between these models can be tricky. A causal model can operate with
hidden quantities which can’t be identified with observables. From this
viewpoint, Bell’s coupling of subquantum and quantum models is very special
and the violation of the Bell inequalities doesn’t close (local) ways beyond
quantum mechanics.

keywords: quantum foundations; Växjö conferences; quantum information
revolution; quantum-like revolution; foundational debates

##  1 Växjö series of annual conferences on quantum foundations

The quantum information revolution, also known as the second quantum
revolution, has influenced not only the development of quantum technology, but
it also has been accelerating studies on foundations of quantum theory.
Regardless the future perspectives of wide implementation, quantum computers,
networks, and cryptography are new powerful systems for testing quantum
foundations and elaborating novel insights on theoretical description of micro
and macro-phenomena and their interrelation. One of the main consequences of
the quantum information revolution is that quantum systems are treated as
information processors nowadays. Roughly speaking, the main theoretical
interest is to quantum soft-ware. Of course, the experimental implementation
is based on advanced hardware, including highly efficient photo-detectors.

The Nobel Prize (2022) given to Aspect, Clauser, and Zeilinger “for
experiments with entangled photons, establishing the violation of Bell
inequalities and pioneering quantum information science” was generated by the
quantum foundational studies. And I am proud that the series of the
conferences on quantum foundations, information, and probability arranged
annually in Växjö (Sweden) since year 2000 contributed essentially to critical
analysis of the“Bell project” [1, 2]. The hot debates on the possible
interpretations of quantum mechanics (QM) and especially on the violation of
the Bell type inequalities were supplemented by the talks of leading
experimentalists involved in testing of these inequalities (e.g., Aspect, de
Martini, Genovese, Gisin, Grangier, Hanson, Ramelow, Shalm, Zeilinger, Weihs,
Weinfurter,…). Such annual assembles energized studies on this topic, both
theoretical and experimental. One of my organizational contributions to this
project was arranging the special session Big event – Final Bell test at the
conference Quantum and Beyond in 2016. The session included the talks of the
representatives of all experimental groups performed the loophole free
experiments. The session was concluded by the talk of Zeilinger who presented
the future of experiments in quantum foundations.

  * •

A. Aspect (the talk was given by P. Grangier): Closing the door on Einstein
and Bohr’s quantum debate.

  * •

G. Weihs: Violation of Bell’s inequality under strict Einstein locality
conditions.

  * •

R. Hanson, From the first loophole-free Bell test to a quantum Internet.

  * •

M. Giustina and M. Versteegh: Significant-loophole-free test of local realism
with entangled photons.

  * •

K. Shalm: A strong loophole - free test of Bell’s inequalities.

  * •

H. Weinfurter: Event ready loophole free Bell test using heralded atom-atom
entanglement.

  * •

P. Grangier: Violation of Bell’s Inequalities in a quantum realistic
framework.

  * •

A. Zeilinger: The future of Bell experiments.

Videos of these talks can be found at lnuplay, see [4, 3, 5, 6, 7, 8]. It is
important to note that not all participants of the conference agreed with
Aspect and Grangier that the Einstein and Bohr debate is closed. During the
following conferences the debates on the possible interpretations of the
outputs of these loophole free experiments were continued. This is the good
place to recall that Bell by himself predicted this situation [2].

By discussing the completion of the project on the loophole free Bell tests we
should recall the contributions of Gill, Glancy, and Larsson who made crucial
contributions into statistical justification of these tests and defended their
statistical legacy during confrontation with the “anti-Bell opposition” which
was widely represented in Växjö. Personally I think that the existence of such
opposition played the crucial role in the evolution of the Bell project and
igniting general interest to Bell inequalities. The most active
representatives of the “anti-Bell opposition” in Växjö were Accardi, Adenier,
De Raedt and Michielsen, Hess and Philipp, Khrennikov, Kupczynski,
Nieuwenhuizen. The views of Nobel Prize Laureate ´t Hooft also deviated from
the conventional interpretation of the violation of the Bell inequality, but
in Växjö he didn’t present them openly (see below my recollection of our
conversations).

By completing this part of discussion the Bell project and the the Nobel Prize
2022, I would like to highlight once again the contribution of Gill, Glancy,
and Larsson: their mathematical (probabilistic) analysis was crucial in
completion of this project. Unfortunately, the role of mathematics in the
experiments is typically dimmed.

I have personally been very engaged in the foundational research and I enjoyed
the Växjö conferences, the talks and discussions with talented people.111Who
are generally talented not only in science. I think that many participants
remember the great evening when Daniel Greenberger played piano in Teleborg
Castle. Only one big dark cloud has always been at the quantum horizon, the
cloud of quantum nonlocality. Spooky action at a distance disturbs, but at the
same time stimulates my mind.

This metaphor “dark cloud at the quantum horizon” I borrowed from a lunch-talk
with Anton Zeilinger, where he told the story about how he became interested
in foundations. In 1950s he participated in a conference which assembled the
world-leading experts in quantum physics. The general attitude of participants
was that development of quantum theory is more or less completed. There was
just one dark cloud at the quantum horizon, the cloud of quantum gravity. As I
understood, he was curious whether there are other clouds at the quantum
foundational horizon and this curiosity led him to the Nobel Prize in 2022.

In this nonlocal atmosphere I tried to seek salvation in the circle of QBists
led by Fuchs with active support of Mermin and Schack (see [9]-[12] for some
references on QBism, especially Fuchs’ papers for Växjö proceedings). Although
QBism was widely represented at the conferences in Växjö since year 2002 and I
have always supported plenary talks and special sessions of QBists, I am a
strong antagonist of QBism for the subjective probability interpretation of QM
and highlighting the personal experiences of “quantum agents”. Educated within
the Soviet probability school and being lucky to have conversations with
Kolmogorov on foundations of probability, I treat the probability objectively.
The first time I heard about subjective probability from Fuchs and hearing
such nonsense annoys me a lot. However, QBism declares that in QM there is no
spooky action at a distance, QM is local. So, may be even QBism is better than
nonlocality!

This was precisely Hänsch’s message in his Växjö talk. I remind that he
received the Nobel Prize (2005) for “contributions to the development of
laser-based precision spectroscopy, including the optical frequency comb
technique”. Hence, he is excellent experimentalist and his appeal to QBism is
not due to attraction by the subjective viewpoint on QM. He told that he needs
a consistent local interpretation of QM and QBism provides it. Hänsch’s talk
induced hot debates and he was attacked by “quantum realists”. I remember the
critical comments of Vaidman and especially Elitzur. At one of the
conferences, we had the social opinion poll on acceptance of QBism as the
basic interpretation of QM. Surprisingly many world’s leading experimentalists
in quantum foundations (e.g., Weihs) supported QBism. QBism was less popular
among theoreticians and philosophers, it was rejected by the majority of the
participants. I suspect that Zeilinger is sympathetic to QBism, although he
did not declare this openly, but his talk in Vienna at the conference “Quantum
[Un]Speakables II: 50 Years of Bell’s Theorem” was so much in the spirit of
QBism.

I spent a lot of time by trying to proceed in quantum foundations and handle
the violation of the Bell inequalities without appealing to nonlocality. I
have understood that one of the barriers on the way towards liberation of QM
from nonlocality is operating with the term “local realism”. This is one of
the most misleading notions in the history of physics, but it is firmed in the
heads of the majority of physicists who are interested in fundamental
questions. Two totally different notions, namely, realism and locality, are
jointly mixed and one of them shadows another.

The first signal of this notion ambiguity I received from the answer of Gisin
to my comment to his talk that the notion of quantum nonlocality is
misleading. Gisin answered that the most misleading is the notion of local
realism and first one should eliminate it from discussions on violation of the
Bell inequalities: one can not keep one term and dismiss the other. Local non-
realism doesn’t mean anything, as Gisin wrote in article [13] (see also my
note [14]). I remark that Bell did not operate with the notion of local
realism, at least in his book [2] summarizing his main works he did not even
mention it.

In principle, one can reject both realism and locality. In particular, one
famous experimentalist who is well known for the rejection of realism (as many
representatives of the Vienna’s circle) often told to me that QM would finally
stimulate people to reject realism, but once he suddenly added that today it
is clear that one should also reject locality. The later statement was really
sudden for me in the light of all our previous conversations. I ignore the
double rejection possibility simply by using Occam’s razor. But I remember
Cromwell’s rule and I may be mistaken. By himself Gisin is definitely in favor
of rejection of locality.

Similar conversation was with famous supporter of the many worlds
interpretation. He always positioned himself as realist (being in one of
numerous worlds). But once he told that QM is nonlocal. I was really shocked,
I didn’t expect from him such hit below the belt. When I became a bit relaxed,
I asked him: “Why?!” And he pointed to nonlocality of the projection
postulate.

During twenty three Växjö conferences I have debated quantum foundations with
many bright scientists and often we disagreed, but this was in the spirit of
these conferences. I am especially thankful to the members of the “Växjö
club”, those who participated in the majority of the conferences, especially
to Bengtsson, D’Ariano, Fuchs, Jaeger, Elitzur, Larsson, Nieuwenhuizen,
Plotnitsky, Rauch, Vaidman, and Weihs.

At the beginning of this century Bohmians, especially Goldstein and Hiley,
strongly influenced my realist attitude - my first conference had the title
“Bohmian Machanics - 2000”. I was impressed by spiritual strength of Basil
Hiley who often struggled alone against a hundred of anti-Bohmians. This was a
good lesson of scientific struggle. I and other participants enjoyed the hot
battle between Basil Hiley and Marlan Scully on whether Bohmian mechanics can
be experimentally distinguished from conventional QM. Only strong pressure
from the chairman, Theo Nieuwenhuizen, led to termination of this battle.

As well the Bohmian interpretation, the many worlds interpretation was not so
widely represented in Växjö. But during 20 years Lev Vaidman advertized it
enthusiastically. This interpretation didn’t meet so strong criticism as the
Bohmian one, may be because the majority of participants didn’t take it
seriously. When I heard the speech of Lev at one of the first conferences
(year 2002?), I was really shocked and looked at Chris Fuchs who adviced to
invite Lev. But Chris nicked - “it is ok!” Later I started to think that the
many worlds interpretation is not so bad, as one of the consistent
alternatives to quantum nonlocality. The main problem for me is still
understanding of which experiment outcomes lead to world’s split? And the
personal contacts with Lev only increased my disappointment. During long time,
I was sure that each random experiment splits the world (where it is
performed). Then once I, Lev, and my daughter Polina (who was teenager at that
time) walked near the lake from the university to the downtown. Polina was
curious in the many worlds story that Lev told with enthusiasm. Suddenly she
said :“Lev, if I unexpectedly push you into this lake, then you will only fall
there in one of the worlds?” Lev looked at her with caution (you can expect
anything from a teenager) and (unexpectedly for me) said, that such a
splitting of worlds is impossible for macro-objects of such mass as he is; for
example, with an amoeba it would be possible…

Coming to Växjö of Gerard ‘t Hooft was the breath of the fresh air for
realists. His braveness in struggle against the orthodox Copenhagen views,
multiplied by the weight of the Nobel Prize, induced the wave of enthusiasm.
It was impression that his journey beyond quantum would soon lead to
establishing of the realist interpretation of quantum theory.

Unfortunately - or fortunately? - it didn’t happen. The first signal that ´t
Hooft proceeds too quickly without taking care of many basic foundational
problems came during one of our conversations, it happens that he “didn’t pay
attention” that his model confronts with the Bell theorem. But his reaction to
these news was plane and it seems that this theorem didn’t disturb him so
much. (I was really surprised!) In the post-conference email exchange his
attitude was Bohm-like and, as I understood, nonlocality was not a substantial
problem. Later he came with the argument based on superdeterminism and, in
particular, rejection of free will. From my present perspective, in principle
‘t Hooft need not appeal to superdeterminism. The latter, although logically
possible, is not so attractive for majority of the physicists. The model of ‘t
Hooft matches perfectly the Bild conception of Herz-Botzmann-Schrödinger on
the interrelation between observational and causal theoretic models (see
section 2). Coupling between two levels of the physical description can have a
variety of colors. As was already repeated a few times, on the way beyond
quantum one need not follow Bell with his specific rules for such coupling.

Contacts with Viennese quantum community were vital for forming of my later
views. I would always remember my conversations with Helmut Rauch during many
conferences in Växjö and my visits to the Atom Institute. He had his own
picture of the quantum phenomena, a kind of reality of both the wave function
and particle. Helmut Rauch strongly supported the Växjö series of conferences
and he attracted to Växjö the top experimentalists from Vienna. I spent a lot
of time with Johann Summhammer, both in Vienna and during his visits to Växjö
and I was really astonished by his idealist perspective of the physical world.
Spending time with him generated first doubts in my “naive realism”. I still
remember that once I found Johann staying near Växjö lake and looking at it.
He told me something as “Look to this beautiful picture which is created in
our brain by integrating the clicks of photo-detectors”.

I was lucky to visit the University of Vienna during realization of the final
Bell tests and enjoyed the atmosphere of coming great scientific event. I
spent a few months with Zeilinger’s group during both experiments closing the
last loopholes, in 2013 when the detection efficiency loophole was closed [15]
and in 2015 when the previous experiment was completed to the loophole free
experiment [16]. This is the good place to highlight the contribution of
Marissa Giustina in these breakthrough experiments. We spent a lot of time by
drinking tea and coffee and speaking about the experiments. I was curious in
understanding “experimental technicalities”. (In fact, I had my vested
interest, I was searching for experimental data that might show some kind of
deviations from quantum theory.) I was really surprised by Marissa’s
experience in the design and methodology of experimenting in quantum
foundations and her remarkable understanding of the basic problems of quantum
theory. She worked hardly on realization of both experiments and spent a few
hundred hours in the basement of Vienna Hofburg Palace (for the last
experiment) by collecting data and then analyzing these huge data-sets and
modifying the experimental setup. Definitely her contribution into closing all
loopholes in the Bell test was crucial. As a university professor, I was also
interested in the pedagogical practice of Anton Zeilinger: he put young person
in the heart of this fundamental experiment. And Marissa justified his hopes.

Relaxing conversations with Anton Zeilinger and the members of his
experimental group, especially with Sven Ramelow, Rupert Ursin, Johannes
Kofler, and Bernhard Wittmann were vital for my leave-taking from “naive
realism”. In particular, I was excited by Anton’s stories regarding his
debates with Dalai Lama. I suspect that recording of these debates has never
decrypted and published. And I think that it should be done, soon or later. In
Vienna I learned that a photon is simply a click of photo-detector and nothing
more, that the correlations in Bell experiment are natural and can be
influenced by a variety of technicalities, that some parameters of
experimental tests cannot be defined on the basis of quantum theory alone but
depend on a lot of experience of the experimentalists who select them by
hands. (Joint article [17] is nice scientific memory about the days spent with
Zeilinger’s group.) Consequently, experimentalists are less orthodox than
theoreticians and more open for beyond-quantum theories, especially if such
theories can be helpful for designing experiments. After my talk at one of the
Växjö conferences, Alain Aspect asked me a question about negative
probability, its possible interpretations and description of the Bell
argument. I was surprised, since I didn’t expect such question from
experimentalist. Moreover, by getting to know that I have a series of papers
and books on negative probability and its interpretation (e.g., [18]), he
continued the discussion during the lunch and told about his own model with
negative probabilities violating the Bell inequality, later he sent me this
paper, old and practically unknown, cf. with Zeilinger’s paper [19].

It is interesting that the intensity of the debates in Växjö is going down.
The last hot confrontation was between Ozawa and Lahti on Ozawa’s rethinking
of Heisenberg’s uncertainty relation and the possibility of its violation
[20]. The cause of debating decline is not clear. It might be that the basic
quantum foundational problems have been resolved or at least the community
came to the consensus. But it might be that one part of the quantum community
with its specific perspective on foundations became so powerful that this led
to disappearance of outsiders from the scene, at least in Växjö. It might be
also a signal of decreasing of interest to quantum foundations. This is the
good place to recall that in 1970s the interest to foundations of QM wasn’t so
high. During the PhD-defense of Alain Aspect one of jury’s members made the
remark: but, where would this bright student find a job? So, Bell’s test
strongly stimulated foundational research. Successful loophole free
experiments made the widespread illusion that, roughly speaking, there is no
any way beyond QM and it is frozen forever. I do not think so. One of the
messages of this note is that before trying to go “beyond quantum”, one should
analyze in more detail a variety of possible ways. And one should remember
that the main impact of Bell’s model was the possibility of its experimental
verification. So, new quantum foundational revolution would be connected to
new experimental tests. As an example of such tests, I point to Sorkin’s
equalities. Weihs’ group in Innsbruck spent a few years by trying to find
violations of these inequalities [21, 22], but they confronted with too
difficult technical problems for detectors’ functioning. Up to now, Sorkin’s
inequalities have not been violated.
Experimentalists working on quantum foundations need new theories and their
experimental tests as soon as possible, otherwise people would move to the
rapidly growing field of quantum technologies …

##  2 From realism to contextuality

Realism was so deeply in my mind that, for me, it was difficult to dismiss it
and imagine the acausal universe driven by irreducible quantum randomness
which was advertized by von Neumann [23]. I spent a few years of my life by
working on the mathematical models combining realism and locality and
constrained by the violation of the Bell inequalities. I interacted with
practically all leading experts in foundations who work on similar models,
e.g., Accardi, Kupczynski, Hess and Philipp, De Raedt and Michielsen,
Nieuwenhuizen, Volovich, de la Pena and Cetto. In spite of controversy in the
evaluation of such studies which was culminated in the famous bet of Accardi
with Gill, I have the impression that local realistic treatment of quantum
phenomena is possible. I think that it can be done by discarding hidden
assumptions not only of the Bell inequality project, but of quantum theory as
whole, e.g., the ergodicity assumption [24].

From my viewpoint, the main problem of the local realistic approach to quantum
phenomena is its clustering into special models working for concrete phenomena
(as at least claimed by the authors) without general theory. For example, the
probability interference can be modeled under the assumption that there exists
indivisible quantum of time.

My leave-taking from realism, at least in its naive form, started with
studying Bohr’s works and discussions with Plotnitsky on the complementarity
principle. After a while I understood, that the complementarity principle is
seeded in contexuality of measurements. Contextuality is understood very
widely here, as dependence on the complex of experimental conditions.
According to Bohr, measurements’ outcomes can’t be treated as objective
properties of a system; they are generated in the complex process of
interaction between a system and a measurement device. Bohr operated with the
term “conditions”, but not contexts. In the modern terminology, these are
experimental contexts. Such notion of contextuality is wider than so-called
joint measurement contextuality which was invented by Bell in the discussion
on possible interpretations of the violation of his inequality and nowadays it
is intensively studied in quantum information theory. We recall that Bell’s
contextuality is about measurement of an observable $A$ in the contexts of
joint measurements with other observables, say $B_{1},...,B_{n}$ which are
compatible with $A.$ As well as Bohr, Bell did not use the term contextuality.

During the recent years I worked a lot on Bohr’s contextuality and its place
in the formulation of the principle of complementarity [25, 26]. This is the
basic principle of QM. It is typically reduced to wave-particle duality.
However, careful reading of Bohr shows that this principle has a more complex
structure. It might be better to call it the principle of contextuality-
complementarity. This is an epistemic principle on structuring knowledge which
can be obtained about a quantum system. The ontological basis of the
complementarity principle is the quantum postulate declaring the existence of
indivisible quantum of action given by the Planck constant $h$ [27, 28, 29].
In articles [30, 31], the direct epistemic counterpart of the quantum
postulate is formulated in the form of the quantum action invariance. This
principle and the complementarity principle are treated as two basic
assumptions of QM, similar to Einstein’s principles of special relativity (cf.
with Zeilinger’s search for fundamental principles of QM [19]).

The Bohr complementarity and Heisenberg uncertainty principles are deeply
interconnected. Heisenberg also related his principle to the quantum
postulate. This is the good place to remark that recently the Heisenberg
uncertainty principle was reconsidered by Ozawa [20, 32] within theory of
quantum instruments.

Now we come to the issue of quantum nonlocality. As well as local realism,
this is an ambiguous notion. Two different sorts of nonlocality are often
mixed. One of them can be called intrinsic quantum nonlocality. This is
apparent nonlocality of quantum state update based on the projection
postulate, another nonlocality is nonlocality of the hidden variables models
violating the Bell inequalities. These two sorts of nonlocality should be
sharply distinguished, as was done in Aspect’s talk at one of the Växjö
conferences. He started his lecture with the statement that QM is
intrinsically nonlocal, since, for a compound system, the operation of the
state projection resulting from a local measurement is nonlocal. Then he told
that this nonlocality is so unpleasant for any physicist that one should
search for another theoretical formalism which would be free of projection
nonlocality. And in this way one introduces hidden variables. Finally, he
tells that, as is shown by the violations of the Bell inequalities, the
subquantum locality (Bell locality) also should be rejected. This is very
consistent presentation the Bell inequality project [33], much better than
original Bell’s motivation - to explain the perfect quantum correlations [2].
From my viewpoint, correlations are not mystical at all, irrelevantly to their
quantumness or classicality (see [34, 35] on probabilistic entanglement of
observables and [36] on classical Brownian entanglement). I prefer Aspect’s
motivation for going beyond QM as attempting to find a deeper theory which
would not suffer of projection nonlocality.

In paper [37] projection nonlocality is analyzed in connection to selection of
interpretations of QM. As is well known, QM is characterized by the diversity
of interpretations. Some experts consider it as a sign of the crisis in
quantum foundations (e.g., as Ballentine at the first Växjö conference in year
2001), others rigidly keep to one concrete interpretation without even paying
attention to the existence of other interpretations. But the majority of
researchers working in quantum physics are not interested in the
interpretation problem. I have asked the question:What is your interpretation
of the wave function? to hundreds of participants of the Växjö conference
series. Some of them were not able to say something consistent and others
automatically replied: I use the Copenhagen interpretation! However, the
meaning of the latter is another delicate problem. This is a good place to
recall Plotnitsky claiming that the term “Copenhagen interpretation” unifies a
bunch of interpretations; he suggested to speak about the interpretations in
the spirit of Copenhagen [38, 39]. My experience of discussions in Växjö
supports this suggestion.

Following Ballentine [40, 41, 42], we cluster the interpretations into two
classes:

  * •

SI the statistical (ensemble) interpretation,

  * •

II the individual interpretation.

By SI a quantum state describes the statistical features of an ensemble of
identically prepared systems; by II a quantum state is a state of an
individual system, say an electron. Ballentine identifies II with the orthodox
Copenhagen interpretation. Interpretation discussions, SI vs. II, flared up
during a few conferences, Ballentine and Nieuwenhuizen advocated consistently
SI and Lahti gave a few firing speeches in favor of II.

The projection nonlocality issue depends on an interpretation. By following II
one really confronts (as, e.g., Aspect) with the intrinsic quantum
nonlocality. However, by following SI one treats the projection postulate as
the quantum analog of the Bayes rule for classical probability update.

In fact, the EPR paper [43] was directed against the orthodox Copenhagen
interpretation. Unfortunately, it was not formulated precisely. Later in 1949
Einstein pointed out [44] that there are two choices:

  * •

a) the orthodox Copenhagen interpretation and nonlocality,

  * •

b) the statistical (ensemble) interpretation and locality.

We note that Einstein considered the intrinsic quantum nonlocality -
projection nonlocality.

By struggling against the orthodox Copenhagen interpretation Einstein did not
question the validity of quantum theory. Specially the reasoning in the EPR-
paper [43] is crucially based on the validity of the Heisenberg uncertainty
principle. I claim that Bell’s model with hidden variables contradicts this
principle as well as the complementarity principle. To follow Bell, one should
from the very beginning reject these principles and also the quantum
postulate, the existence of the indivisible quantum of action. This would be
the difficult decision, similar to the decision to reconstruct the special
relativity by rejecting the Einstein postulate on the constancy of the light
velocity.

What is the main problem of the Bell-like models with hidden variables? This
is the identification of the quantities of a subquantum model with quantum
observables, the identification of their outcomes and probability
distributions. This casualty of the Bell project was highlighted by De Broglie
[45], immediately after publication of the Bell works. Unfortunately, his
message was completely ignored. De Broglie interpreted his double solution
theory as a hidden variables theory. (We remark that so common identification
of De Broglie’s and Bohm’s theories is wrong. In particular, De Broglie did
not consider his theory as physically nonlocal; he just pointed to apparent
nonlocality of mathematical equations.) For De Broglie, the values and
probability distributions of subquantum quantities and quantum observables
should not be identified. Hence, in subquantum theories not only variables,
but even quantities (functions of hidden variables $\xi=\xi(\lambda))$ and
their probability distributions are hidden.

De Broglie’s approach to subquantum theories matches perfectly with the Bild
conception for structuring of scientific theories which was established by
Hertz and Boltzmann and later applied by Schrödinger to QM [46]-[50] (see also
[51] for recent development of the Bild conception). By this conception one
should distinguish an observational model and a causal theoretical model. As
was pointed out by Schrödinger, QM is an observational model and it is
characterized by acausality (as was emphasized by von Neumann [23]). One can
search for causal theoretical models beyond QM. Due to Hertz and Bolzmann,
coupling between causal theoretical and observational models can be tricky and
varying. From this viewpoint, Bell suggested just one special coupling. Hence,
employing the Bild conception opens the door to creation of a variety of
subquantum causal theoretical models, in spite the negative output of the Bell
project. Unfortunately, in quantum foundations the Bild conception is
practically unknown. Its partial resemblance can be found in ontic-epistemic
structuring of physical theories due to Primas and Atmanspacher [52, 53].

A promising causal theoretical model for QM is prequantum classical
statistical field theory, see [54]. This is model of classical random fields
reproducing quantum averages and correlations. Its coupling with QM is more
tricky than for the Bell model with hidden variables.

Bohr’s contextuality is formalized within the contextual measurement model
(CMM) developed in [55]. CMM is based on the calculus of contextual
probabilities which closely related to generalized probability models. CMM
serves as the basis for the Växjö interpretation [56] of QM, a contextual
statistical interpretation. By this interpretation QM is the machinery for
generally non-Bayesian probability update. One of the fundamental exhibitions
of non-Bayesianity is the violation of the classical formula of total
probability (FTP) [57]. Contextual generalization of FTP contains an
additional term, the interference term. In QM this is the formula for
interference of probabilities, e.g., in the two slit experiment (cf. Feynman
[58, 59]). Paper [61] exposes purely quantum approach to the Bell inequalities
which are exemplified as the tests of incompatibility of local variables, or
in the mathematical terms, as the tests of noncommutativity:
$[\hat{A}_{1},\hat{A}_{2}]\not=0$ and $[\hat{B}_{1},\hat{B}_{2}]\not=0$ for
Alice’s and Bob’s observables, respectively.

Recent article [62] is devoted to critical analysis of QBism’s private
experience perspective in light of Ozawa’s intersubjectivity theorem [63]
based on theory of quantum instruments and the indirect measurement scheme.
This paper is continuation of my challenging QBism during the last 20 years
[64, 65]. My intersubjectivity paper was carefully analyzed and responded by
the QBism community [66, 67]. But neither I nor Ozawa are satisfied by
QBists’s replies and the discussion will continue, may be at Växjö 2024
conference.

I was one of the pioneers in applications of the methodology and formalism of
quantum theory outside of physics [68, 69, 70, 71]. This area is known as
quantum-like modeling. I put a lot of efforts to its establishing and
advertising and I am thankful to all my friends contributed to the volume on
quantum-like revolution [60]. It is important to note that, in spite of my
opposition to QBism in quantum physics, I appeal to it [72] as a possible
interpretational basis for modeling cognition and decision making with quantum
formalism. I can’t say that the idea of applying the quantum methods outside
of physics was immediately welcomed by the participants of the first Växjö
conferences. I remember some hating comments on my talks devoted to quantum-
like modeling at the beginning of this century: “Please don’t touch our
beautiful quantum theory with your dirty hands!” But by opening the Overton
window wider and wider I was able to smooth the reactions of “rigid quantum
physicists” on the use the quantum methods outside of physics, at least in
decision making. Later a few special sessions on quantum-like modeling in
decision making, economics and finance were organized in cooperation with
Emmanuel Haven who contributed so much in advertising this area of research in
Växjö. Nowadays quantum-like modeling is the well established area of
research. One of its problems is attraction of numerous newcomers from various
domains of science who are not so well educated in quantum theory. Often
“Aladdin and the Magic Lamp”. But this is merely the illness of too rapid
growth.

I was lucky to speak as well with Kolmogorov, concerning foundations of
probability and possible generalizations of his axiomatics, as with Mackey
regarding the conditional probability structure of QM. Then this viewpoint on
quantum probability was discussed with Accardi, Ballentine, and Gudder and
these discussions were imprinted in my mind. My views on the causes of the
Bell inequalities violations were influenced by conversations with Lochak, De
Baere, and De Muynck.

My works in quantum foundations can be interesting for all who are not excited
by quantum mysteries and paradoxes (or scandals [73]) and consider QM as an
observational model describing quantum measurements. QM without nonlocality
and spooky action at a distance need not seem insipid. One can find a plenty
of interesting problems: theoretical, experimental, mathematical, and
philosophical. QM treated as a measurement model, works very well and there is
no reason to question it. Its apparent acausality is typical for observational
models as was highlighted already by Hertz, Boltzmann, and Schödinger
[46]-[50]. If one wants to go beyond QM, it is natural to appeal to the Bild
conception. In particular, the violation of the Bell inequalities should be
analyzed within this conception.

## References

  * [1] Bell, J.S. On the problem of hidden variables in quantum theory. Rev. Mod. Phys. 1966, 38, 450.
  * [2] Bell, J.S. Speakable and Unspeakable in Quantum Mechanics, 2nd ed.; Cambridge University Press: Cambridge, UK, 2004.
  * [3] G. Weihs, Violation of Bell’s inequality under strict Einstein locality conditions. Video, special session Big event – Final Bell test, A. Khrennikov (chairman), Conf. Quantum and Beyond, Växjö, 2016. https://play.lnu.se/media/t/0_6jkhpefo
  * [4] A. Aspect and P. Grangier, Closing the door on Einstein and Bohr’s quantum debate. Video, special session Big event – Final Bell test, A. Khrennikov (chairman), Conf. Quantum and Beyond, Växjö, 2016. https://play.lnu.se/media/t/0_61nm165n
  * [5] R. Hanson, From the first loophole-free Bell test to a quantum Internet. Video, special session Big event – Final Bell test, A. Khrennikov (chairman), Conf. Quantum and Beyond, Växjö, 2016. https://play.lnu.se/media/t/0_q2r30syq
  * [6] M. Giustina and M. Versteegh, Significant-loophole-free test of local realism with entangled photons. Video, special session Big event – Final Bell test, A. Khrennikov (chairman), Conf. Quantum and Beyond, Växjö, 2016. https://play.lnu.se/media/t/0_iasiwe06.
  * [7] H. Weinfurter, Event ready loophole free Bell test using heralded atom-atom entanglement. Video, special session Big event – Final Bell test, A. Khrennikov (chairman), Conf. Quantum and Beyond, Växjö, 2016. https://play.lnu.se/media/t/0_xprww7jj
  * [8] A. Zeilinger, The future of Bell experiments. Video, special session Big event – Final Bell test, A. Khrennikov (chairman), Conf. Quantum and Beyond, Växjö, 2016. https://play.lnu.se/media/t/0_hgz7034c
  * [9] Fuchs, C. A.: Quantum mechanics as quantum information (and only a little more). In: Khrennikov, A. (ed.), Quantum Theory: Reconsideration of Foundations, Ser. Math. Model. 2, pp. 463–543. Växjö Univ. Press, Växjö (2002). arXiv:quant-ph/0205039
  * [10] Fuchs, C. A.: The anti-Växjö interpretation of quantum mechanics. In: Khrennikov, A. (ed.), Quantum Theory: Reconsideration of Foundations, Ser. Math. Model. 2, pp. 99–116. Växjö Univ. Press, Växjö (2002). arXiv:quant-ph/0204146
  * [11] Fuchs, C. A., Mermin, N. D. and Schack, R. (2014). An Introduction to QBism with an Application to the Locality of Quantum Mechanics, _Am. J. Phys._ 82, p. 749.
  * [12] DeBrota, J. B., Fuchs, C. A., Pienaar, J. L., and Stacey, B. C. (2021). Born’s rule as a quantum extension of Bayesian coherence. Physical Review A, 104(2), 022207.
  * [13] Gisin, N. (2012). Non-realism: deep thought or a soft option?. Found. Phys., 42, 80-85.
  * [14] Khrennikov, A. Bell argument: Locality or realism? Time to make the choice. AIP Conf. Proc., 1424, pp. 160-175 (2012); https://arxiv.org/pdf/1108.0001v2.pdf
  * [15] M. Giustina1, A. Mech, S. Ramelow, B. Wittmann, J. Kofler, J. Beyer, A. Lita, B. Calkins, T. Gerrits, S. Woo Nam, R. Ursin, A. Zeilinger, Bell violation with entangled photons, free of the fair-sampling assumption. Nature 2013, 497, 227–230.
  * [16] Giustina, M., Versteegh, M. A. M., Wengerowsky, S., Handsteiner, J., Hochrainer, A., Phelan, K., Steinlechner, F., Kofler, J., Larsson, J.-A., Abellan, C., Amaya, W., Pruneri, V., Mitchell, M. W., Beyer, J., Gerrits, T., Lita, A.E., Shalm, L.K., Woo Nam, S., Scheidl, T., Ursin, R., Wittmann, B., and Zeilinger, A., A significant-loophole-free test of Bell’s theorem with entangled photons, Phys. Rev. Lett. 115, 250401 (2015).
  * [17] Khrennikov, A., Ramelow, S., Ursin, R., Wittmann, B., Kofler, J., and Basieva, I.: On the equivalence of the Clauser-Horne and Eberhard inequality based tests. Phys. Scr. T163, 014019 (2014)
  * [18] Khrennikov, A. Yu. _Interpretations of Probability_ ; VSP Int. Sc. Publishers: Utrecht/Tokyo, 1999; 2nd edn., De Gruyter: Berlin, 2009.
  * [19] Zeilinger, A.: A foundational principle for quantum mechanics. Found. Phys. 29 (4), 631–643 (1999)
  * [20] Ozawa, M.: Soundness and completeness of quantum root-mean-square errors. Quant. Inf. 5, Article number: 1 (2019)
  * [21] Sinha, U., Couteau, C., Medendorp, Z., Sillner, I., Laflamme, R., Sorkin, R., and Weihs, G.: Testing Born’s rule in quantum mechanics with a triple slit experiment. In: Accardi, L., Adenier, G., Fuchs, C., Jaeger, G., Khrennikov, A., Larsson, J.-A., and Stenholm, S. (eds). Foundations of Probability and Physics-5, 1101, pp. 200-207. American Institute of Physics, Ser. Conf. Proc., Melville, NY (2009)
  * [22] Sinha, U., Couteau, C., Jenewein, T. , Laflamme, R. and Weihs, G.: Ruling out Multi-Order Interference in Quantum Mechanics. Science 329, 418–421 (2010)
  * [23] Von Neumann, J. Mathematical Foundations of Quantum Mechanics; Princeton University Press: Princeton, NJ, USA, 1955.
  * [24] Khrennikov, A. (2017). Buonomano against Bell: Nonergodicity or nonlocality?. International Journal of Quantum Information, 15(08), 1740010.
  * [25] Khrennikov, A., Bohr against Bell: complementarity versus nonlocality. Open Phys. 2017, 15, 734-773.
  * [26] Khrennikov, A. After Bell. Fortschritte der Physik (Progress in Physics). 2017, 65, N 6-8, 1600014.
  * [27] N. Bohr, The quantum postulate and the recent development of atomic theory. Supplement to Nature, April 14 , 580-590 (1928).
  * [28] N. Bohr, Wirkungsquantum und Naturbeschreibung. Naturwissenschaft, 17, (1929).
  * [29] N. Bohr, The quantum of action and the description of nature. In: Foundations of Quantum Physics I (1926–1932). Niels Bohr collected works. J. Kalckar (ed.), v. 6, Elsevier B.V., 1985, pp. 201-217.
  * [30] Khrennikov, A. Quantum postulate vs. quantum nonlocality: on the role of the Planck constant in Bell’s argument. Found Phys 51, 16 (2021).
  * [31] Khrennikov, A. (2021). Is the Devil in h?. Entropy, 23(5), 632.
  * [32] Ozawa, M. (2023). Quantum Measurement Theory for Systems with Finite Dimensional State Spaces. In: Plotnitsky, A., Haven, E. (eds) The Quantum-Like Revolution. Springer, Cham.
  * [33] A. Aspect, Bell’s Theorem: The naive view of an experimentalist. quant-ph/0402001.
  * [34] Basieva, I., and Khrennikov, A. (2022). Conditional probability framework for entanglement and its decoupling from tensor product structure. J. Phys. A: Math. Theor., 55(39), 395302.
  * [35] Basieva, I., and Khrennikov, A. Entanglement of observables: Quantum conditional probability approach. Found. Phys., 53, Article number: 84 (2023).
  * [36] Allahverdyan, A. E., Khrennikov, A., and Nieuwenhuizen, T. M. (2005). Brownian entanglement. Phys. Rev. A, 72(3), 032102.
  * [37] Khrennikov, A.: Two faced Janus of quantum nonlocality. Entropy 22, 303 (2020)
  * [38] Plotnitsky, A. The Spirit and the Letter of Copenhagen: A Response to Andrei Khrennikov. arXiv:quant-ph/0206026v2 7 Jun 2002
  * [39] A. Plotnitsky, Reality without Realism. Matter, Thought, and Technology in Quantum Physics. Springer, Berlin-Heidelberg-New York, 2021.
  * [40] Ballentine, L. E. (1989). The statistical interpretation of quantum mechanics, _Rev. Mod. Phys._ , 42, 358–381.
  * [41] Ballentine, L. E. (2014). Quantum mechanics: a modern development. WSP, Singapore.
  * [42] Ballentine, L. E.: Interpretations of probability and quantum theory. In: Khrennikov, A. Yu. (ed) Foundations of Probability and Physics, Quantum Probability and White Noise Analysis 13, 71-84. WSP: Singapore, 2001.
  * [43] Einstein, A.; Podolsky, B.; Rosen, N. Can quantum-mechanical description of physical reality be considered complete? _Phys. Rev._ 1935, 47, 777–780.
  * [44] Einstein, A. Reply to criticism, In: P. A. Schilpp, Albert Einstein: Philosopher-Scientist. (The Library of living philosophers). Cambridge: Cambridge University Press, 1949, pp. 665-688
  * [45] De Broglie, L. (1974). Sur la réfutation du théoreme de Bell. Comptes Rendus de l’Academie des Sciences B, 278, 721-722.
  * [46] H. Hertz, Untersuchungen ueber die Ausbreitung der elektrischen Kraft (Leipzig: J.A. Barth, 1892); translated by D.E. Jones as Electric Waves being Researches on the Propagation of Electric Action with Finite Velocity through Space (London: Macmillan, 1893; reprinted New York: Dover, 1962), p. 21.
  * [47] H. Hertz, Die Prinzipien der Mechanik in neuen Zusammenhange dargestellt (Leipzig: J.A. Barth, 1894), pp. xxi-xxii; translated by D.E. Jones and J.T.Walley as The Principles of Mechanics presented in a new Form (New York: Dover, 1956).
  * [48] Boltzmann, L. (1905). Uber die Frage nach der objektiven Existenz der Vorgnge in der unbelebten Natur. In Barth, J.A. (Ed.) Populre Schriften: Leipzig.
  * [49] Boltzmann, L. (1974). On the development of the methods of theoretical physics in recent times. In McGuinness, B. (Ed.) Theoretical physics and philosophical problems. Vienna circle collection, Vol. 5. Dordrecht: Springer.
  * [50] E. Schrödinger, What is an elementary particle? Endeavour 9 (1950), 109-116. reprinted in: E. Schödinger, Gesammelte Abhandlungen. Band 4. Allgemein wissenschaftliche und populäe Aufsätze (Wien: Verlag der Österreichischen Akademie der Wissenschaften and Braunschweig/Wiesbaden: Friedr.Vieweg & Sohn, 1984), pp. 456-463.
  * [51] Khrennikov, A. The Bild-conception for Scientific Theory Structuring in Classical and Quantum Physics: from Hertz and Boltzmann to Schrödinger and De Broglie
  * [52] Atmanspacher, H. (1994). Is the ontic/epistemic-distinction sufficient to describe quantum systems exhaustively. In: Symposium on the Foundations of Modern Physics (pp. 15-32).
  * [53] Atmanspacher, H. and Primas, H. (2003). Epistemic and ontic quantum realities. In: Time, quantum and information (pp. 301-321). Springer, Berlin, Heidelberg.
  * [54] Khrennikov, A. (2014). Beyond quantum. Singapore: Pan Stanford Publ.
  * [55] Khrennikov, A. Contextual measurement model and quantum theory. Royal Society Open Science, 2024.
  * [56] Khrennikov, A. Växjö interpretation-2003: Realism of contexts. In _Quantum Theory: Reconsideration of Foundations_ ; Khrennikov A., Ed.; Växjö Univ. Press: Växjö, 2004, pp. 323–338. arXiv:quant-ph/0401072
  * [57] Khrennikov, A. (2009). _Contextual Approach to Quantum Formalism_ , (Springer, Berlin-Heidelberg-New York).
  * [58] Feynman, R. P. (1951). _The Concept of probability in quantum mechanics_ ; Berkeley Symp. on Math. Statist. and Prob. Proc. Second Berkeley Symp. on Math. Statist. and Prob. (Univ. of Calif. Press), 533–541.
  * [59] Feynman, R. and Hibbs, A.: Quantum Mechanics and Path Integrals. McGraw-Hill, New York (1965)
  * [60] A. Plotnitsky and E. Haven, The Quantum-Like Revolution. A Festschrift for Andrei Khrennikov with a foreword by 2022 Nobel Laureate Anton Zeilinger. Springer Nature, Berli, 2023.
  * [61] A. Khrennikov, Get rid of nonlocality from quantum physics. Entropy, 21(8), 806 (2019).
  * [62] A. Khrennikov, Ozawa’s Intersubjectivity Theorem as objection to QBism individual agent perspective. Int. J. Theor. Phys. 63, 23 (2024).
  * [63] Ozawa, M.: Intersubjectivity of outcomes of quantum measurements. arXiv:1911.10893 (2019)
  * [64] Khrennikov, A. (2018). External Observer Reflections on QBism, Its Possible Modifications, and Novel Applications, In: _Quantum Foundations_ , STEAM-H: Science, Technology, Engineering, Agriculture, Mathematics & Health; Khrennikov A. and Toni B. Eds.; Springer, Cham, pp. 93–118.
  * [65] Khrennikov, A. (2018). Towards better understanding QBism, _Found. Sc._ , 23 (1), 181–195
  * [66] Stacey, B. C. Whose Probabilities? About What? A Reply to Khrennikov. arXiv:2302.09475.
  * [67] Schack, R. When will two agents agree on a quantum measurement outcome? Intersubjective agreement in QBism. arXiv:2312.07728 [quant-ph]
  * [68] Khrennikov, A.: Quantum-like formalism for cognitive measurements. Biosystems 70, 211–233 (2003)
  * [69] Khrennikov, A.: On quantum-like probabilistic structure of mental information, OSID 11 (3), 267–275 (2004)
  * [70] Khrennikov, A.: Information Dynamics in Cognitive, Psychological, Social, and Anomalous Phenomena. Ser.: Fundamental Theories of Physics. Kluwer, Dordrecht (2004)
  * [71] Khrennikov, A.: Ubiquitous Quantum Structure: from Psychology to Finances. Springer, Berlin-Heidelberg-New York (2010)
  * [72] Khrennikov, A. (2016). Quantum Bayesianism as the basis of general theory of decision-making, _Phil. Trans. R. Soc._ 374(2068), 20150245.
  * [73] Nieuwenhuizen, T.M. (2023). Contra Multos Verbos: On Scandals of Quantum Mechanics. In: Plotnitsky, A., Haven, E. (eds) The Quantum-Like Revolution. Springer, Cham. for general quantum systems. Rep. Math. Phys. 18, 11–28 (1980).


# Learning Facial Liveness Representation
for Domain Generalized Face Anti-Spoofing

###### Abstract

Face anti-spoofing (FAS) aims at distinguishing face spoof attacks from the
authentic ones, which is typically approached by learning proper models for
performing the associated classification task. In practice, one would expect
such models to be generalized to FAS in different image domains. Moreover, it
is not practical to assume that the type of spoof attacks would be known in
advance. In this paper, we propose a deep learning model for addressing the
aforementioned domain-generalized face anti-spoofing task. In particular, our
proposed network is able to disentangle facial liveness representation from
the irrelevant ones (i.e., facial content and image domain features). The
resulting liveness representation exhibits sufficient domain invariant
properties, and thus it can be applied for performing domain-generalized FAS.
In our experiments, we conduct experiments on five benchmark datasets with
various settings, and we verify that our model performs favorably against
state-of-the-art approaches in identifying novel types of spoof attacks in
unseen image domains.

Index Terms—  Face anti-spoofing, domain generalization, representation
disentanglement, deep learning

##  1 Introduction

Face recognition technology has been widely applied in many interactive
intelligent systems such as automated teller machines (ATMs), mobile payments,
and entrance guard systems, due to their convenience and remarkable accuracy.
However, face recognition systems are still vulnerable to presentation attacks
ranging from print attacks, video replay attacks, and 3D facial mask attacks,
etc. Therefore, face anti-spoofing (FAS) plays a crucial role in securing the
robustness of face recognition systems.

Over the past few years, different FAS methods have been proposed by
researchers. Assuming inherent disparities between live and spoof faces,
studies have handled this problem from the perspective of detecting texture in
color space [1], image distortion [2], temporal variation [3], or deep
semantic features [4, 5]. Although promising results have been obtained by
these methods, to perform FAS on unseen image domains which are not observed
during training remains a challenging task. The performance of these FAS
methods trained from a particular source domain would drop dramatically when
different backgrounds, subjects, or shooting devices are encountered in a
different target domain of interest.

To address the domain shift problem mentioned above, studies have exploited
auxiliary information, such as face depth [6], for distinguishing live and
spoof faces. However, these approaches still have their limitations since they
highly depend on the accuracy of estimated auxiliary information. Therefore,
researchers start to improve the robustness of FAS from the perspective of
domain generalization, which aims to learn a generalized feature space by
aligning the distributions among multiple source domains. Shao [7] proposed a
multi-adversarial deep domain generalization (MADDG) framework to derive
domain-invariant feature spaces for real and fake images with a dual-force
triplet mining constraint. Extended from MADDG, Jia [8] proposed a single-side
domain generalization (SSDG) learning framework that groups spoof types across
domains together with a triplet-mining algorithm for the purpose of domain
generalization. However, the generalization ability of the mentioned
approaches might be limited. This is because the existing methods typically do
not distinguish between domain-independent facial liveness representations and
the domain-dependent ones which are irrelevant to FAS [9].

In light of the above issue, researchers handled FAS from the aspect of
feature disentanglement. A multi-domain disentangled representation learning
method is proposed by [9], aiming to obtain more discriminative liveness
features by disentangling domain-invariant representations from an image.
While [9] have disentangled the domain-independent FAS cues from the domain-
dependent representations, their approach might not be able to generalize to
real-world scenarios, in which novel (i.e., unseen) spoof attacks might be
presented. Although existing works focus on disentangling domain-relevant
features from other features, their extracted liveness features still contain
facial content information, which is irrelevant to liveness information. Thus,
the generalization ability of their methods to handle unseen spoof attacks is
still limited.

In this paper, we address the domain-generalized FAS problem by learning
domain-invariant facial liveness representation. We not only aim to handle FAS
in unseen data domains but unseen spoof attacks can also be detected, which
makes our proposed model more practical. As detailed later, we present a
representation disentanglement framework, which is designed to extract facial
liveness, content, and image domain representations. More specifically, the
liveness representation describes information for liveness detection. On the
other hand, the latter two representations, i.e., the facial content and image
domain representations are viewed as liveness-invariant features. The
disentanglement of such features from the liveness features allows our model
to better perform FAS in unseen domains with novel spoof attacks. The
contributions of this work can be highlighted below:

  * •

We propose a representation disentanglement network for domain-generalized
face liveness detection, which is able to recognize novel spoof attacks in
unseen domains/datasets during inference.

  * •

Our proposed network is designed to extract facial liveness, content, and
image domain representations. While the liveness representation would be
utilized for FAS, the latter two are liveness-invariant.

  * •

We conduct experiments on multiple FAS datasets in various settings, and
confirm that our method performs favorably against state-of-the-art approaches
in detecting novel spoof attacks in unseen image domains.

Fig. 1: Overview of our proposed network architecture. Our network aims to
extract liveness, facial content, and image domain representations from facial
input images. This is realized by the learning of liveness encoder $E_{L}$,
content encoder $E_{C}$, and domain encoder $E_{D}$, respectively. To further
ensure the disentanglement of liveness-irrelevant information, liveness
classifier $C_{L}$, content decoder $D_{C}$, and domain classifier $C_{D}$ are
jointly deployed. Once the training is complete, one can apply $E_{L}$ and
$C_{L}$ for domain-generalized FAS.

##  2 Proposed Method

###  2.1 Problem Definition and Annotations

For the sake of completeness, we first define the setting and notations
considered in this paper. During training, we have face images in _S_
different source domains denoted as $X=\\{X_{1},X_{2},...,X_{S}\\}$ and the
corresponding binary real/fake labels denoted as
$Y=\\{Y_{1},Y_{2},...,Y_{S}\\}$. For the $i$th source domain, we have
${N_{i}}$ images, i.e., $X_{i}=\\{x_{i,j}\\}_{j=1}^{N_{i}}$, and the
associated labels $Y_{i}=\\{y_{i,j}\\}_{j=1}^{N_{i}}$. As for the inference
stage, liveness of facial images in a disjoint and unseen target domain (with
seen attacks and unseen attacks) will be determined accordingly.

As shown in Fig. 1, our proposed network architecture has three encoders for
handling the facial input images: liveness encoder $E_{L}$, content encoder
$E_{C}$, and domain encoder $E_{D}$. The liveness encoder $E_{L}$ is designed
to extract liveness representation, followed by a liveness classifier $C_{L}$
for performing FAS prediction. The content encoder $E_{C}$ extracts the facial
content representation from the input, while the subsequent decoder $D_{C}$ is
deployed for reconstruction guarantees. As for the encoder $E_{D}$, it
extracts the domain representations from the training input images, so that
the domain classifier $C_{D}$ would classify the image domain accordingly.
Once the joint learning of the above network modules is complete, one can
simply take the liveness encoder $E_{L}$ and the liveness classifier $C_{L}$
for inference.

###  2.2 Learning Liveness-Irrelevant Representation

In order to address domain-generalized FAS, we propose to extract facial
content and image domain features from the liveness representation. Since the
disentangled content and image domain features are not utilized for FAS, they
can be viewed as liveness-irrelevant representations of face images.

As depicted in Fig. 1, the above two types of features are extracted by the
content encoder $E_{C}$ and domain encoder $E_{D}$. For the content encoder,
it is expected to retrieve facial content information, which is not regarding
the authenticity of the face input image. In our work, we specifically
consider content information described by PRNet [10], which is known for face
alignment information by observing a face image. The idea is that spoof image
or not, the facial input image is expected to contain facial contour and
landmark information, which suggest the recovery of the corresponding face
alignment. Thus, given the $j$th training image from source domain $i$, the
encoded content feature of $E_{C}(x_{i,j})$ would serve as the input to the
content decoder $D_{C}$, which is designed to recover the feature produced by
a pre-trained PRNet $\Phi(\cdot)$. In other words, we calculate the following
content loss $L_{cont}$ for updating both $E_{C}$ and $D_{C}$:

$\normalsize
L_{cont}=\sum_{i=1}^{S}\sum_{j=1}^{N}\|D_{C}(E_{C}(x_{i,j}))-\Phi(x_{i,j})\|_{2}^{2}.$
(1)

While the calculation of (1) ensures $E_{C}$ and $D_{C}$ for extracting and
recovering facial content information, we need additional supervision to
ensure the derived content features $E_{C}(x_{i,j})$ does not contain either
liveness or image domain information. As a results, with the deployment of the
liveness classifier $C_{L}$ and domain classifier $C_{D}$, we choose to
calculate the following content confusion loss $L^{cnf}_{cont}$:

$\displaystyle\normalsize{L^{cnf}_{cont}}=\sum_{i=1}^{S}\sum_{j=1}^{N}($
$\displaystyle\|C_{L}(E_{C}(x_{i,j}))-{\frac{1}{2}}\|_{2}^{2}$ (2)
$\displaystyle+$
$\displaystyle\|C_{D}(E_{C}(x_{i,j}))-{\frac{1}{S}}\|_{2}^{2}).$

The first and second terms in (2) are to confuse the liveness and domain
classifiers, respectively. It is worth noting that, $C_{L}$ predicts the
binary liveness label, and $C_{D}$ outputs the domain label (out of $S$). With
the observation of this content confusion loss, the training of our proposed
framework would further ensure the content encoder $E_{C}$ to produce liveness
and domain-irrelevant features.

As for the learning of image domain features, the modules of domain encoder
$E_{C}$ and domain classifier $C_{D}$ are deployed for achieving this goal.
Following the above example, assume that we have the $j$th training image from
source domain $i$, the encoded domain feature $E_{D}(x_{i,j})$ is expected to
describe illumination, image quality, etc., information. Thus, the subsequent
domain classifier $C_{D}$ is designed to recognize the image domain $i$ by
observing such domain features. In our work, we calculate the domain loss
$L_{dom}$ for updating both $E_{D}$ and $C_{D}$ as follows:

$\normalsize
L_{dom}=-\sum_{i=1}^{S}\sum_{j=1}^{N}m_{i}*log(C_{D}(E_{D}(x_{i,j}))).$ (3)

In the above equation, $m_{i}$ denotes the ground truth one-hot vector
representing the domain label.

Similar to the design of liveness-irrelevant facial content features, we now
discuss how we further ensure that the learning of $E_{D}$ and $C_{D}$ would
not contain liveness information. With the deployment of the liveness
classifier $C_{L}$ in Fig. 1, we have $C_{L}$ take the encoded domain features
$E_{D}(x_{i,j})$. To enforce the disentanglement of liveness-relevant
information, the liveness classifier $C_{L}$ is not expected to perform FAS on
$E_{D}(x_{i,j})$. Therefore, we calculate the domain confusion loss
$L^{cnf}_{dom}$, which is defined below:

$\normalsize
L^{cnf}_{dom}=\sum_{i=1}^{S}\sum_{j=1}^{N}\|C_{L}(E_{D}(x_{i,j}))-{\frac{1}{2}})\|_{2}^{2}.$
(4)

With facial content and domain losses, together with the corresponding
confusion losses, our proposed framework allows us to disentangle liveness-
irrelevant features from the input images. The deployment of $E_{C}$, $E_{D}$,
$D_{C}$, and $C_{D}$ would also facilitate the learning of liveness
representation, as we discuss next.

###  2.3 Learning Domain-Invariant Liveness Representation

To address domain-generalized FAS, learning of domain-invariant liveness
representation from the input images would be the major component of our
proposed framework. With facial content and image domain features properly
disentangled from the input image, we now discuss how the extraction of
liveness representation is realized by our network so that the derived
features can be applied to detect novel spoof attacks in unseen target
domains.

As depicted in Fig. 1, we have liveness encoder $E_{L}$ and the associated
classifier $C_{L}$ deployed in our framework. Given the $j$th training image
from source domain $i$, we have the encoded liveness feature $E_{L}(x_{i,j})$
expected to describe the liveness (i.e. real/fake) information. The subsequent
liveness classifier $C_{L}$ is designed to determine whether the input face
image is real or fake depending on the feature $E_{L}(x_{i,j})$. To better
separate real and fake facial images, we adopt the simplified Large Margin
Cosine Loss (LMCL) function [11] as the objective, which calculates
intra/inter-class angular distances for the corresponding input images with a
predetermined margin $m$. For the sake of clarity, we disregard the domain
index $i$ for the input image; thus, liveness loss $L_{live}$ for updating
$E_{L}$ and $C_{L}$ is calculated as follows:

| O&C&I to M | O&M&I to C | O&C&M to I | I&C&M to O
---|---|---|---|---
Method | HTER(%) | AUC(%) | HTER(%) | AUC(%) | HTER(%) | AUC(%) | HTER(%) | AUC(%)
Auxiliary(Depth Only) | 29.14 | 71.69 | 22.72 | 85.88 | 33.52 | 73.15 | 30.17 | 77.61
Auxiliary(All) [6] | 27.60 | - | - | - | 28.40 | - | - | -
MMD-AAE [12] | 31.58 | 75.18 | 27.08 | 83.19 | 44.95 | 58.29 | 40.98 | 63.08
MADDG [7] | 17.69 | 88.06 | 24.50 | 84.51 | 22.19 | 84.99 | 27.98 | 80.02
RFM [13] | 13.89 | 93.98 | 20.27 | 88.16 | 17.30 | 90.48 | 16.45 | 91.16
SSDG-M [8] | 16.67 | 90.47 | 23.11 | 85.45 | 18.21 | 94.61 | 25.17 | 81.83
SSDG-R [8] | 7.38 | 97.17 | 10.44 | 95.94 | 11.71 | 96.59 | 15.61 | 91.54
Cross [9] | 17.02 | 90.10 | 19.68 | 87.43 | 20.87 | 86.72 | 25.02 | 81.47
DRDG [14] | 12.43 | 95.81 | 19.05 | 88.79 | 15.56 | 91.79 | 15.63 | 91.16
Ours | 7.50 | 97.45 | 9.80 | 96.82 | 11.38 | 94.90 | 16.70 | 91.83

Table 1: Comparisons of FAS in unseen domains in terms of HTER and AUC. For
example, O&C&I to M denotes that the model is trained on the datasets of Oulu,
CASIA, & Idiap, and is evaluated on MSU. Note that the attack types are print
and replay in both source and target domains.

$\displaystyle
L_{live}=-\sum_{j=1}^{N}log(\frac{e^{\alpha(W^{T}_{y_{j}}E_{L}(x_{j})-m)}}{e^{\alpha(W^{T}_{y_{j}}E_{L}(x_{j})-m)}+e^{\alpha(W^{T}_{1-y_{j}}E_{L}(x_{j}))}}),$
(5)

where $\alpha$ is a hyperparameter, and $W=\\{W_{0},W_{1}\\}$ denotes the
parameters of liveness classifier $C_{L}$. We note that, $W_{0}$ represents
the model parameter for label $y_{j}=0$ (i.e., spoof attack), while $W_{1}$
denotes that for $y_{j}=1$ (i.e., real face). And, the margin $m$ is
introduced in Eq. (5), so that the separation between liveness representations
derived from real and fake images can be further enforced.

To further ensure that the liveness feature $E_{L}(x_{i,j})$ does not contain
any image domain information, we utilize the aforementioned domain classifier
$C_{D}$, and calculate the liveness confusion loss $L^{cnf}_{live}$ as
follows:

$\normalsize{L^{cnf}_{live}}=\sum_{i=1}^{S}\sum_{j=1}^{N}\|C_{D}(E_{L}(x_{i,j}))-\frac{1}{S})\|_{2}^{2}.$
(6)

With both liveness classification and confusion losses, we are able to train
our proposed framework for disentangling domain-invariant liveness
representation.

Together with Eq. (5) and Eq. (6), we maximize $W^{T}_{y_{j}}E_{L}(x_{j})$ and
minimize $W^{T}_{1-y_{j}}E_{L}(x_{j})$, which encourages the separation
between real and fake images (with margin $m$) across domains. On the other
hand, maximization of $W^{T}_{y_{j}}E_{L}(x_{j})$ further implies the
suppression of intra-class variation for images of the same label. In other
words, the learned $W_{1}$ and $W_{0}$ would represent the prototypes of
domain-invariant liveness representations of real and fake images,
respectively. With the disentanglement of image domain information, our
learning of liveness representation would not only separate real face images
and spoof attacks but also enforce the minimization of the associated intra-
class variations. Therefore, the generalization of our model to novel spoof
attacks across different domains can be expected.

The overall objectives, together with the detailed training process, are
summarized in the Algorithm 1 of our supplementary materials. Once the
training of our network architecture is complete, only $E_{L}$ and $C_{L}$ are
utilized for performing domain generalized FAS. That is, the liveness encoder
$E_{L}$ is applied to extract the domain-invariant liveness representation,
which is fed into $C_{L}$ for FAS prediction.

##  3 Experiment

###  3.1 Experimental Settings

Five public face anti-spoofing datasets are utilized to evaluate the
effectiveness of our method: OULU-NPU [15] (denoted as O), CASIA-FASD [16]
(denoted as C), Idiap Replay-Attack [chingovska2012effectiveness] (denoted as
I), MSU-MFSD [2] (denoted as M), and CelebA-Spoof [17] (denoted as Cb).

| O&C&I to Cb | O&M&I to Cb | O&C&M to Cb | I&C&M to Cb
---|---|---|---|---
Method | HTER(%) | AUC(%) | HTER(%) | AUC(%) | HTER(%) | AUC(%) | HTER(%) | AUC(%)
MADDG [7] | 42.46 | 62.26 | 44.96 | 57.01 | 50.08 | 57.18 | 48.92 | 51.86
RFM [13] | 41.49 | 60.19 | 43.32 | 65.96 | 42.83 | 59.37 | 35.93 | 67.32
SSDG-M [8] | 41.19 | 60.91 | 35.25 | 65.96 | 36.40 | 66.66 | 35.02 | 69.19
SSDG-R [8] | 20.29 | 86.87 | 20.58 | 86.54 | 25.05 | 82.11 | 19.86 | 88.58
Ours | 19.42 | 88.17 | 20.60 | 86.93 | 22.32 | 85.49 | 16.22 | 90.85

Table 2: Comparisons of FAS in unseen domains with novel spoof attacks in
terms of HTER and AUC. Note that the attack types are print and replay in the
source domains, while Cb contains the spoof attack of 3D masks for testing.

For the architecture of our model, we have ResNet-18 [18] pre-trained on
ImageNet for the encoders $E_{L}$, $E_{C}$ and $E_{D}$, where the weights of
the first layer are shared. The statistics of each dataset and the details of
the architecture are shown in Table A. and Table B. of the supplementary
materials, respectively. For the evaluation metrics, we have the Half Total
Error Rate (HTER) and Area under the Curve of ROC (AUC) following [7, 8, 9,
14] in all experiments.

###  3.2 FAS in Unseen Target Domains

Following [7, 8, 9, 14], we utilize four public datasets, i.e., O, C, M, and
I, to evaluate the effectiveness of our model adapting to unseen datasets. We
select three datasets as the source domains and the remaining one as the
target domain.

As shown in Table 1, our approach achieved impressive results and performed
against all the existing FAS approaches. This demonstrates that the liveness
encoder $E_{L}$ in our proposed model is able to extract generalized features
for the unseen domains by disentangling the liveness-irrelevant information.
While recent approaches including MADDG and SSDG consider extracting domain-
generalized liveness features, the generalization ability of their methods is
still limited. This is because that their extracted liveness features still
contain facial information, which is irrelevant to liveness information.

###  3.3 FAS with Novel Spoof Attacks in Unseen Target Domains

| O&C&I to Cb | O&M&I to Cb | O&C&M to Cb | I&C&M to Cb
---|---|---|---|---
Method | HTER(%) | AUC(%) | HTER(%) | AUC(%) | HTER(%) | AUC(%) | HTER(%) | AUC(%)
Baseline | 34.41 | 72.81 | 45.01 | 57.53 | 33.36 | 72.9 | 38.30 | 66.74
Baseline + $E_{C}$, $D_{C}$ | 33.26 | 73.23 | 41.94 | 61.63 | 28.13 | 77.35 | 39.15 | 68.78
Baseline + $E_{D}$, $C_{D}$ | 27.66 | 79.40 | 34.64 | 69.59 | 25.99 | 80.62 | 26.15 | 82.54
Ours | 19.42 | 88.17 | 20.60 | 86.93 | 22.32 | 85.49 | 16.22 | 90.85
Table 3: Analysis of our network architecture design. Note that Baseline
denotes the learning of only the liveness encoder and liveness classifier. The
modules for disentangling liveness-irrelevant representations (i.e.,
$E_{C}$&$D_{C}$ for content and $E_{D}$&$C_{D}$ for image domain) are
assessed, which are shown to support the effectiveness of our full model.

We evaluate the effectiveness of our model adapting to a real-world scenario
where the model encounters unseen spoof attacks which are not observed during
training. Five datasets are used in this setting: O, C, I, M, and Cb. We
choose Cb as the target domain since Cb contains a unique attack type, i.e. 3D
mask, which does not appear in other datasets.

As shown in Table 2, we can observe that our approach outperforms existing FAS
approaches in all of the four tasks. Our proposed method can be better
generalized to unseen spoof attacks because it learns the domain-invariant
liveness representations, which are able to generalize to unseen target
domains. As explained in the liveness loss $L_{live}$, our liveness classifier
$C_{L}$ could learn a better liveness feature prototype after seeing different
spoof attacks from the training data. Therefore, the learned fake prototype of
the liveness classifier can generalize better to unseen spoof attacks.

###  3.4 Qualitative Analysis

As discussed in the previous sections, recent DG approaches, including SSDG,
focus on extracting domain-invariant features but neglect that the derived
liveness features may still contain irrelevant facial content information. On
the other hand, our method disentangles such facial content for better
generalization performances. To verify the effectiveness of our
disentanglement model, we utilize the Grad-CAM [19] algorithm to obtain the
class activation map visualizations, where the activation map indicates the
regions that the model attends on when performing domain-generalized FAS.

We compare our model to SSDG-R and show the visualization results in Figure 2.
The first column of the figure shows the visualization result of a real face
and the other columns present the results of spoof faces. In the first column,
SSDG-R focused on the background of the face image while our proposed model
concentrated on the liveness part of the face image. For the spoof image in
the third column, with the facial content features disentangled, our proposed
model concentrated on the sunken part of the 3D mask, which could be regarded
as important liveness information. On the other hand, SSDG-R focuses more on
the liveness-irrelevant facial contours and background. The visualization
results support that disentanglement of the facial content enforce our model
on the liveness-related part of the face images and therefore facilitate the
domain-generalized FAS.

Fig. 2: FAS visualization examples on real and spoof attacks (in green and red
bounding boxes, respectively) using O&M&I to Cb. Note that the Grad-CAM is
utilized to visualize the activation maps for real and fake images. Comparing
to SSDG-R, our method offers explainable attention maps.

###  3.5 Ablation Study

As listed in Table 3, the baseline model (in the first row) contains only the
liveness encoder $E_{L}$ and liveness classifier $C_{L}$. The second and the
third row shows the results when the facial content disentanglement and domain
disentanglement is applied, respectively. The last row shows the results of
our proposed model, i.e., both content and domain disentanglement are applied.

We can see from the first two rows that the model with the disentanglement of
facial content performs better than the baseline. This confirms that such
disentanglement helps our model to focus on the liveness cues in the images
instead of liveness-irrelevant facial content. Comparing the first and third
row, the domain disentanglement mechanism helps our model extract domain-
invariant liveness features that can be applied on the unseen target domain
and therefore brings significant improvements. With both facial content and
domain feature disentanglement are utilized, our proposed method achieved the
best performance in all tasks.

##  4 Conclusion

In this paper, we address the challenging task of domain-generalized FAS
problem, in which novel spoof attacks in unseen target domains need to be
identified. Based on the idea of representation disentanglement, we present a
network architecture that is able to extract facial liveness, content, and
domain features. Aiming at performing domain-generalized FAS, the facial
liveness representation exhibits domain invariant properties while the content
and domain representations are viewed as liveness-irrelevant features, whose
derivations are enforced by our network module and objective designs.
Extensive experiments on benchmark datasets demonstrated the effectiveness of
our proposed network, which shows promising domain generalization in
addressing FAS, including the ability to handle novel types of spoof attacks
during inference. Since the current proposed model only observes images as
inputs, taking video inputs and thus dealing with visual-temporal information
would be among our future research directions for domain-generalized FAS.

Acknowledgement This work is supported in part by the Telecommunication
Laboratories of Chunghwa Telecom TL-110-L402. We also thank National Center
for High-performance Computing (NCHC) for providing computational and storage
resources.

## References

  * [1] Zinelabidine Boulkenafet, Jukka Komulainen, and Abdenour Hadid,  “Face anti-spoofing based on color texture analysis,”  in 2015 ICIP. IEEE, 2015.
  * [2] Di Wen, Hu Han, and Anil K Jain,  “Face spoof detection with image distortion analysis,”  IEEE Transactions on Information Forensics and Security, vol. 10, no. 4, 2015.
  * [3] Rui Shao, Xiangyuan Lan, and Pong C Yuen,  “Deep convolutional dynamic texture learning with adaptive channel-discriminability for 3d mask face anti-spoofing,”  in 2017 IJCB. IEEE, 2017, pp. 748–755.
  * [4] Jianwei Yang, Zhen Lei, and Stan Z Li,  “Learn convolutional neural network for face anti-spoofing,”  arXiv preprint arXiv:1408.5601, 2014.
  * [5] Keyurkumar Patel, Hu Han, and Anil K Jain,  “Cross-database face antispoofing with robust feature representation,”  in Chinese Conference on Biometric Recognition. Springer, 2016, pp. 611–619.
  * [6] Yaojie Liu, Amin Jourabloo, and Xiaoming Liu,  “Learning deep models for face anti-spoofing: Binary or auxiliary supervision,”  in CVPR, 2018, pp. 389–398.
  * [7] Rui Shao et al.,  “Multi-adversarial discriminative deep domain generalization for face presentation attack detection,”  in CVPR, 2019, pp. 10023–10031.
  * [8] Yunpei Jia et al.,  “Single-side domain generalization for face anti-spoofing,”  in CVPR, 2020, pp. 8484–8493.
  * [9] Guoqing Wang et al.,  “Cross-domain face presentation attack detection via multi-domain disentangled representation learning,”  in CVPR, 2020, pp. 6678–6687.
  * [10] Yao Feng et al.,  “Joint 3d face reconstruction and dense alignment with position map regression network,”  in ECCV, 2018, pp. 534–551.
  * [11] Hao Wang et al.,  “Cosface: Large margin cosine loss for deep face recognition,”  in CVPR, 2018.
  * [12] Haoliang Li et al.,  “Domain generalization with adversarial feature learning,”  in CVPR, 2018, pp. 5400–5409.
  * [13] Rui Shao, Xiangyuan Lan, and Pong C Yuen,  “Regularized fine-grained meta face anti-spoofing,”  in AAAI, 2020.
  * [14] Shubao Liu et al.,  “Dual reweighting domain generalization for face presentation attack detection,”  arXiv preprint arXiv:2106.16128, 2021.
  * [15] Zinelabinde Boulkenafet et al.,  “Oulu-npu: A mobile face presentation attack database with real-world variations,”  in IEEE international conference on automatic face & gesture recognition. IEEE, 2017.
  * [16] Zhiwei Zhang et al.,  “A face antispoofing database with diverse attacks,”  in 2012 5th IAPR international conference on Biometrics (ICB). IEEE, 2012.
  * [17] Yuanhan Zhang et al.,  “Celeba-spoof: Large-scale face anti-spoofing dataset with rich annotations,”  in ECCV. Springer, 2020.
  * [18] Kaiming He et al.,  “Deep residual learning for image recognition,”  in CVPR, 2016, pp. 770–778.
  * [19] Ramprasaath R Selvaraju et al.,  “Grad-cam: Visual explanations from deep networks via gradient-based localization,”  in ICCV, 2017, pp. 618–626.