Paper Title: Computational music analysis from first principles.





^1Department of Music, Princeton University, Princeton, NJ 19147, USA
^2Center for the Study of Complex Systems, University of Michigan, Ann Arbor, MI 48109, USA
^3Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501, USA

§ ABSTRACT 
We use coupled hidden Markov models to automatically annotate the 371 Bach chorales in the Riemenschneider edition, a corpus containing approximately 100 000 notes and 20 000 chords.  We give three separate analyses that achieve progressively greater accuracy at the cost of making increasingly strong assumptions about musical syntax.  Although our method makes almost no use of human input, we are able to identify both chords and keys with an accuracy of 85% or greater when compared to an expert human analysis, resulting in annotations accurate enough to be used for a range of music-theoretical purposes, while also being free of subjective human judgments.  Our work bears on longstanding debates about the objective reality of the structures postulated by standard Western harmonic theory, as well as on specific questions about the nature of Western harmonic syntax. 

Computational music analysis from first principles
    Dmitri Tymoczko^1,3 and Mark Newman^2,3
    August 5, 2024
==================================================






§ INTRODUCTION

More than two hundred years ago, Gottfried Weber argued that traditional Western harmony was pervasively ambiguous <cit.>.   One problem involves chords and keys: a single progression such as C major  F major can be experienced as a tension-reducing dominant-tonic in F major or a tension-increasing tonic-subdominant in C major (Fig. <ref>).  A second problem involves chords themselves: a voice moving from D6 to C6 over a stable {F4, A4} can signify either a change of harmony, from an inverted D minor to a root-position F major, or a “merely linear” passing motion that creates an F major chord as a nonsyntactical byproduct.  A third problem involves incomplete sonorities, for instance bare thirds that could plausibly belong to multiple triads, or triads belonging to multiple seventh chords.  Together these ambiguities ensure that even relatively simple passages can be analyzed in multiple ways, with no agreed-upon rules for choosing among them.  Music analysis thus requires complex holistic judgments, balancing vague and often-conflicting heuristics <cit.>.  It is not surprising, therefore, that we find widespread disagreement about analyses even among human experts—not only about general principles, but also specific passages (Fig. <ref>).  





This holism helps explain the somewhat disappointing performance of unsupervised computational musical analysis.  Micchi et al. <cit.>, for example, review a variety of attempts at automated analysis, none of which were able to assign Roman numerals with greater than 50% overall accuracy.  In a different vein, Quinn and White <cit.> used hidden Markov models to classify progressions in the Bach chorales but their results depart substantially from standard music-theoretical practice, for instance by including seemingly “nonharmonic” tones within chords, and by assigning tonic and subdominant to the same harmonic category.  Currently, no existing studies have managed to use unsupervised methods to  annotate substantial musical corpora in a way that closely captures the judgments of human annotators.  Some approaches have achieved a good match to human intuition in more limited contexts, such as the studies of Noland and Sandler <cit.> and Nápoles López et al. <cit.>, which both focused on the narrower question of identifying key centers rather than chords and were able to do so with about 90% accuracy.

Substantial human disagreement, coupled with the lack­luster performance of computational methods, together raise the skeptical possibility that human judgments might not be reducible to objective algorithms.  Weber himself argued that musical analysis required intuitive human judgment <cit.>.  Quinn <cit.> has argued that the notion of a chord is theoretically superfluous, suggesting that one can accurately capture the syntax of traditional Western tonal music without distinguishing harmonic from nonharmonic tones.  In a similar vein, Gjerdingen <cit.> proposes that the syntax of Western music is grounded in “schemas” that do not not require the apparatus of standard theory.  Perhaps the familiar theoretical picture, in which nonharmonic tones decorate tertian harmonies, is not so much an empirical claim as an interpretive grid we impose onto music.

Against this background, we present in this paper a simple and largely assumption-free computational method that does a good job at reproducing expert human analysis, achieving about 88% overall correctness at the task of identifying keys and chords in the 371 Bach chorales of the Riemenschneider edition, compared to a human analysis of the same corpus.  Our approach encompasses a family of related algorithms all sharing the same basic structure: for a given input, they find a fixed number of transposable probability distributions that best represent the occurrence of elements (notes or chords) in that input.  When fed raw notes and told to find a small number of probability distributions, the output can be interpreted as a series of chords; when fed these same chords as input, the output can be interpreted as a series of keys.  The significance of these results lies not so much in their accuracy alone but in the combination of accuracy, simplicity, and lack of presupposition: our algorithm performs well with very little information about musical structure, using well-known and straightforward techniques, and without the need for extensive training sets of expert human analysis.  In using familiar methods to reproduce the standard chord-and-key picture, it answers skeptics who have questioned the objectivity of Western musical concepts <cit.>.  Conversely, the close correspondence to human annotations suggests that those annotations do not  rest on large numbers of arbitrary decisions.  Finally, by retracing the steps required to approximate human analyses, our algorithm clarifies the assumptions underlying traditional analytical practices.

The structure of the paper is as follows.  Section <ref> describes the statistical model we use in detail and places it in the context of previous work on related models.  Section <ref> applies this model to the corpus of Bach chorales and compares the results to those of a human analysis, finding reasonably good agreement.  Section <ref> improves these already-good results by applying a few additional rules of thumb concerning modulatory tendencies, local harmonic “laws,” and the principles of nonharmonic tone usage.  Section <ref> considers some broader methodological issues arising from our investigation, as well as potential uses for the machine-annotated corpus generated by our calculations.  A number of technical issues are discussed in appendices.  Data and code are freely available online at .




§ HIDDEN MARKOV MODELS FOR ANALYZING

KEYS AND CHORDS


Our basic goal is to take a sequence of notes, represented as notation and divided among some number of voices, and infer from them the underlying chords and keys at each point in a work.  We ignore lengths of notes, rests, and other rhythmic elements of the music: all notes are given equal weight, whether short or long, and whether they fall on the beat or off it.  Whenever any voice begins a note or rest, we consider it a new data point for our analysis.  The key or chord need not change at every such data point however.

To perform our analysis we make use of hidden Markov models, a standard statistical tool for the analysis of discrete time-series data.  In a hidden Markov model it is assumed that a system of interest is governed by “hidden” factors, changing over time, which are not directly observed but which nonetheless influence the observed behavior of the system.  For example, the observed behavior might be the sung notes of a Bach chorale and the hidden factors the underlying chords, which one can model (for example) with Roman numerals.  The chords are hidden in the sense that the Roman numerals are not stated explicitly by the composer or written on the score, but they determine the observed data—the notes—in a probabilistic fashion.  If the underlying chord is C major, for example, then there is a high probability that we will hear certain notes {C, E, G}, a lower probability that we will hear some others {D, A, B}, and a still-lower probability for others {C, E, A}.

The hidden factors can change over time.  Our C major chord might transition to G major, then to A minor, and so forth, but some transitions are more common than others: C major is unlikely to be followed by C major for example.  To capture these differences, a hidden Markov model incorporates a set of transition probabilities that represent how likely every chord is to be followed by every other, and we expect that some of these probabilities will be  large and others small.

Collectively, these elements define the hidden Markov model: chords change over time with some transitions being more likely than others, and notes reflect the current chord in a probabilistic fashion, the probabilities of individual notes being dependent on the underlying chord.

The hidden Markov model can be thought of as a machine for generating sequences of notes from underlying chords.  In the present instance, however, we want to use the model in reverse, we want to infer the chords from the notes.  We ask, “If these are the notes we hear, what is our best guess about the chords that produced them?”  We can answer this question by combining the hidden Markov model with standard numerical techniques of Bayesian inference: the Baum-Welch and Viterbi algorithms, described below.

A central goal of our work is to perform these calculations while making as few assumptions as possible ahead of time.  We could for instance make use of guesses about the model parameters based on input from human experts, and analyses of this kind have been performed in the past <cit.>, but we will not do that here.  The parameters in our calculations are given random initial values, which are then automatically refined by the Baum-Welch algorithm to fit the data.  We give the calculation as little help as possible in performing its analysis; we are interested in what it comes up with when given free rein to take any approach it likes.

Once we have inferred a collection of chords we use them as input to a second layer of analysis in which we infer the prevailing key at each moment from the chords.  This second analysis is also performed using a hidden Markov model.  The hidden features of this model are now the keys and it is assumed that chords are produced with probabilities that depend on the current key.  If we are in C major for instance, then some chords will have a high probability of occurrence (C major, G7), while others will have lower probability (C minor, A major).  Again we make only the most basic prior assumptions about the hidden Markov model and we initialize the parameter values at random; we are interested to see what the algorithm comes up with on its own.  In effect, the computer “listens” to a large corpus and “learns” how music works from first principles, based on what it hears.

Our only assumptions about our models are the following:


  * Music is built from an ordered set of 12 octave-free pitch classes, laid out in a circular arrangement like the numbers on a clock. Musical transposition is represented by addition modulo 12.

  * The probability of a particular note at a particular time is dictated by an underlying “chord,” a transposable probability distribution anchored on one of the 12 tones, and belonging to a small number of types, which is fixed at three in our calculations.  In practice these types turn out to correspond to accepted classes such as major and minor chords, but we do not specify this: we specify only the number of chord types and not their structure.

  * The probability of transition from one chord to the next depends on chord type and the interval by which the chords are separated.  We do not specify what the probabilities are, only that different probabilities can exist for different transitions.

  * The probabilities of appearance of chords are dictated by an underlying “key,” which is another transposable probability distribution anchored on one of the 12 tones, and belonging to a small number of types, fixed at two in our calculations.  In practice these types turn out to correspond to major and minor keys, but again we do not specify this, only that different types exist.

  * The probabilities of transition between keys may vary depending on the type of key and the interval between successive keys, but we do not specify what those probabilities are.


Arguably the biggest assumption here is about the number of chord and key categories (three and two respectively), and these choices are certainly rooted in our pre-existing knowledge about the style.  The choice of two key categories, in particular, is based on the expectation that we will find major and minor keys only.  Other styles could employ a larger number of modes and demand a model with more key types.

The number of chords is more flexible.  Three categories gives reasonable results without artificially shoehorning highly dissimilar chords into a single category.  But we have performed analyses with larger numbers of chord types and the results are reasonable.  One can gain additional nuance by allowing more types, at the expense of more complex analytic output and also longer computation time.  Using three chord types, however, gives excellent results, so a larger number, while allowable, is not necessary.




 §.§ Related previous work

A number of previous authors have employed hidden Markov models for harmonic analysis, particularly for the task of inferring the key of a piece of music, a calculation similar to the second half of our analysis described above.  The goal of this work has generally been to create machinery that can identify keys with high accuracy.  To do this, most authors have tuned their algorithms using knowledge from human experts concerning occurrence of notes within keys and transitions between keys.  While this can indeed produce high accuracy, it conflicts with our goal of determining how much the model can achieve on its own.  The difference of approach is analogous in some ways to different philosophies in the design of programs for playing chess.  Early chess programs were trained with large amounts of expert human knowledge about the game, such as opening books and endgames, and these programs can perform well.  But more recent work focused on a new class of algorithms, exemplified by DeepMind's Alpha­Zero, which eschews human expertise and learns from scratch by playing against itself.  Starting with only the basic rules of the game, such as how the pieces move and what constitutes a win, the program is able in less than 24 hours to graduate from beginner chess to superhuman skill without outside help.

Most previous work on hidden Markov models for musical analysis has been analogous to the first approach, making use of human knowledge to build a working machine.  Noland and Sandler <cit.>, for example, made use of a hidden Markov model to learn keys from chord sequences in music by the Beatles, attaining 91% accuracy in key identification compared to a human analysis.  The parameters of the model were initialized by hand using values assigned by experts.  Nápoles López et al. <cit.> took a similar approach, but inferred keys directly from notes without using chords as input.  Their approach works best with symbolic input but can also be applied directly to audio data.  A number of other authors have also developed methods for analyzing audio data, most often in the form of pitch-class profiles <cit.>.  Chai and Vercoe <cit.> used this approach, for instance, to deduce keys directly from notes, while Sheh and Ellis <cit.> and Burgoyne and Saul <cit.> both trained hidden Markov models to infer chords from pitch-class profiles.  Burgoyne and Saul went further to infer keys as well, although again using hand-tuned parameters.

Our approach makes use of no hand-tuned parameters—the only assumptions are those laid out in Section <ref>.  In effect we tell the program only the “rules of chess” and ask whether it can teach itself, from scratch, to interpret harmony as a human being does, simply by listening to enough of it.  We also use a larger corpus, annotated in much more detail, than the corpora featuring in previous work: our annotations record not only global keys but also local key changes, and include approximately 20 000 chords.[Albrecht and Shanahan <cit.> use a corpus of 982 pieces, but each is annotated only with a single global key; our corpus instead includes within-piece key changes.]  The corpus has been checked for accuracy both by hand and automatically, and it has been used in the past for a variety of purposes <cit.>.

More recently, several authors have trained machine learning algorithms on corpora of hand-annotated scores <cit.>.  This strategy once again treats expert judgments as a capacity to be emulated.  Our work instead seeks to use machine learning to understand that capacity, and in particular to determine the extent to which it is guided by arbitrary conventions.



 §.§ Inferring chords from notes


We use two specific hidden Markov models in our calculations.  The first represents how notes are related to the underlying chords.  As applied to a complete corpus of musical works, such as a set of chorales, this model assumes a sequence of chords for each work, transition probabilities between chords, and note probabilities that depend on chords.  Chords are defined by an anchor pitch r and a type t.  The anchor can be thought of roughly as the root of the chord, although in some cases, as we will see, it does not correspond to the conventional root.  Moreover, we are at pains to avoid any presuppositions about chord structure, so we use the generic term “anchor” here.

As discussed in Section <ref>, we fix the number of chord types at three.  We do not specify what the types represent—that is for the algorithm to learn—but as we will see they end up being major and minor triads plus a combined third type that captures both diminished and dominant chords.  For convenience we will often denote a chord by a single symbol c, which is synonymous with the anchor/type pair (r,t).

As also mentioned previously, we identify a new data point every time any voice starts a note or rest.  We will refer to these data points as steps of the work.[Quinn and White <cit.> call them “salami slices.”] Different steps do not necessarily correspond to the same note value—some could be quarter notes while others are eighth notes for instance.  The underlying chord may change at any step, or it may stay the same.  All chords are considered a priori equally likely at the start of a work and we define a set of probabilities α_c_1c_2 for subsequent transitions between chords c_1=(r_1,t_1) and c_2=(r_2,t_2) thus:

    α_c_1c_2 = P( c_2 | c_1 ) = f(r_2-r_1,t_1,t_2).

That is, the probability that the next chord will be c_2 given that the previous one was c_1 is a function of the types of the chords and the interval between their anchors.  Here r_2-r_1 indicates the upward interval from the first anchor to the second, in semitones modulo 12.  Thus the interval from B3 to C4 is 1 semitone while the interval from C4 to B3 is 11 semitones.  The probability of not changing chord at a particular step is

    α_c_1c_1 = f(0,t_1,t_1).

Note that this does not depend on the anchor r_1 of the chord but does depend on the type t_1 and hence is potentially different for chords of different type.

We also define a set of probabilities for pitch classes to occur given the current chord.  Since pitch classes are octave free there are only 12 possibilities, which we represent by integers n = 0…11.  Then

    μ_cn = P( n | c )

is the probability that a sounding note has pitch n given that the current chord is c.

This fully defines our first model.  Our goal is now to take the notes of all works in a corpus and fit the model to them, meaning we want to infer the chords and probabilities given the observed notes.  We fit a single instance of the model to all works in the corpus, under the assumption that similar rules of harmony apply to all works.  There are standard techniques for performing this type of fit, which comprise two stages of analysis <cit.>.  In the first stage we use the Baum-Welch expectation-maximization algorithm to calculate the values of the probabilities.  Then in the second stage, given those values, we use the Viterbi dynamic-programming algorithm to calculate the best estimate of the chords.  A technical description of these two well-known algorithms is given in the appendices.

The Baum-Welch algorithm is an iterative algorithm that starts from an initial guess about the values of the parameters and repeatedly refines that guess until a locally optimal value is attained.  In previous work, authors have used input from human experts to make the initial guess, but in order to avoid relying on expert opinion we use random values produced by a high-quality random number generator.  Like all expectation-maximization algorithms, it is known that the Baum-Welch algorithm can converge to sub-optimal solutions from certain initial conditions.  To guard against this we perform 50 separate runs of the algorithm with different random starting conditions and take as our final solution the one that gives the highest value of the data likelihood—see Section <ref> and Appendix <ref>.

Note that, since our calculations are performed using pitch classes as input, the chords produced by our calculation do not include information about inversions.  In Section <ref> we discuss simple rules for deducing inversions automatically by referring back to the score.




 §.§ Inferring keys from chords


Once we have determined the chords, we use our second hidden Markov model to also determine the prevailing key at each step.  This model assumes a sequence of keys, one at each step, transition probabilities between keys, and probabilities of chords given the prevailing key.  Keys are represented by an anchor s and a type u, and again for brevity we will use a single symbol k to represent a key, which is equivalent to the anchor/type pair (s,u).  (Again, the anchor can be intuitively associated with a tonic, but we do not assume this to be the case.)  We allow two types of keys but do not stipulate what they should be, although as we will see they strongly resemble major and minor.

Transitions between keys are parameterized by probabilities

    β_k_1k_2 = P( k_2 | k_1 ) = g(s_2-s_1,u_1,u_2),

which is a function of the types of the keys and of the interval between their anchors (measured again as the upward interval in semitones, modulo 12).  All keys are considered a priori equally likely at the start of a work and the probability of not changing key is

    β_k_1k_1 = g(0,u_1,u_1),

which thus depends on the type of key (as it should) but not on the anchor.  The probabilities of chords are

    ν_kc = P( c | k ),

which is the probability that a sounding chord is c given the prevailing key k.

This second model is fitted to the chord sequences generated by the first model (Section <ref>), one sequence for each work in the corpus, using the Baum-Welch algorithm to determine the probabilities and the Viterbi algorithm to determine the specific keys.  Again we fit a single instance of the model to all works in the corpus, and again the initial parameter values are generated randomly and we repeat the entire calculation 50 times to guard against the possibility of converging to a sub-optimal solution.

The end result of applying both models to the corpus is a set of keys and chords at every step in every work.  As a by-product we also derive the values of the probability parameters, which are of interest in their own right, as discussed in Section <ref>.




§ ANALYSIS OF BACH CHORALES


Our analysis focuses on the 371 Bach chorales in the Riemenschneider edition.  As described in Section <ref>, our calculations are divided into two parts, the first focusing on the chords and the second on the keys.   Each of these is then further subdivided into the calculation of the probability parameters and the calculation of the chords and keys themselves.

Turning first to the chords, our algorithm looks for chords of three types, which are distinguished by the probability distributions of their notes.  Over the course of the 50 runs of the algorithm, the best fit we find to the data is one in which the three chord classes correspond to major, minor, and dominant seventh chords, as described below.  The next best fit has classes corresponding to major, minor, and dominant seventh with suspended fourth, but this fit is enormously worse than the first-place winner: the data likelihood—the probability of the notes given the inferred chord classes—is more than a million times worse.  This allows us to say with the some confidence that the major/minor/dominant seventh analysis is the best fit to the data in the present case.  (See Appendix <ref> for further discussion of this point.)

The distributions of notes within each chord class found by our method when applied to the chorales are shown in Fig. <ref>, with the convention that the highest-probability note is placed in the anchor position in each case.  The first two plots correspond clearly to major and minor chords—note the high probabilities assigned to the root, major third, and fifth in the first case and root, minor third, and fifth in the second case.  Other notes get much smaller probabilities, though nonzero, in part because these notes can occur as passing notes, neighbor notes, or other embellishments, and also because they occur as sevenths.  (See, for instance the minor seventh, 10 semitones above the anchor in the middle panel.  By the same token, the major sixth in the same panel, 9 semitones above the anchor, indicates the presence of a half-diminished seventh chord {D, F, A, B}.)





The third category of chord, represented by the bottom panel in Fig. <ref>, has high probabilities for the anchor pitch, the minor third, and the diminished fifth, suggesting a diminished triad, but also for the minor sixth (eight semitones above the anchor), suggesting a dominant seventh.[Note that the anchor in this case—the most probable note—is, from a conventional perspective, the third of the dominant seventh.]  Thus the third category captures both diminished and dominant seventh chords, a sensible marriage since diminished chords often play a role functionally similar to dominant sevenths, particularly in a major mode.  Note that this category can also represent a pure major triad, with notes 0, 3, and 8 semitones above the anchor—a dominant triad without a seventh.

Once the probabilities are determined, we calculate the chords themselves.  Figure <ref> shows a typical example, for Riemenschneider No. 88, along with a human analysis produced by one of us (DT).  Overall, our algorithm does a good but not perfect job of identifying chords.  Out of 45 chords, the calculation gets 40 correct and 5 wrong (highlighted in the figure).  Often the errors are understandable.  For instance, identifying a B dim at the start of measure 5 is certainly reasonable, though most human annotators would consider the G to represent a chord change.  (In part this is because the progression A minor-B diminished-A minor is rare.)  Less obvious is the A minor instead of D minor on the final beat of the same bar: here the algorithm takes the presence of the A-E fifth to suggest an A minor chord, whereas the human considers the E a nonharmonic suspension.

Turning to the keys, our calculation looks for keys of two types, characterized by the probability of occurrence of chords of each of the three types in each possible transposition relative to the anchor pitch.  Again the calculation finds the probabilities first.  The values of the probabilities are shown in Fig. <ref> and correspond closely to our expectations for major and minor keys.  In the first key type, shown in the upper panel of the figure and corresponding to major keys, major triads occur with high probability at zero, five, and seven semitones above the anchor and minor triads—with somewhat lower probabilities—two, four, and nine semitones above the anchor.  Diminished/dominant seventh chords are almost always 11 semitones above the anchor, corresponding to a dominant seventh on the fifth or a diminished on the leading tone (which play similar roles).

In the second key type, shown in the lower panel of Fig. <ref> and corresponding to minor keys, major triads occur mostly on the fifth scale degree, minor triads on the first and fourth, and diminished on the major seventh again.  All of these probabilities agree well with our expectations.

Over the 50 repetitions of the calculation with different random initial values of the parameters, these are the only probability distributions of chords found by our algorithm.  No other distribution was found even once, indicating with high certainty that this is the best fit to the data.

In passing, we note that although our algorithm operates exclusively in chromatic space, these chord probabilities allow us to infer the presence of familiar scales.  Taking the most probable notes in the most probable chords gives us the diatonic scale above the major-key tonic, (C, D, E, F, G, A, B) in C major, and the harmonic minor scale above the minor-key tonic, (A, B, C, D, E, F, G) in A minor.  This suggests the presence of an additional hierarchical scalar level intermediate between chord and chromatic aggregate.



Once the probabilities are determined, the algorithm makes an estimate of the prevailing key at each step in each work and, like the chords, the results are accurate but not perfect.  Again Fig. <ref> gives an example, with disagreements highlighted.  The main error—and one that we see repeatedly in our results—is that our calculation tends to locate key changes slightly earlier than the human analysis does.  For example, the first key change from A minor to G major is marked on the fourth beat of measure 1 by our algorithm but on the first beat of measure 2 by the human analysis.  The algorithm changes keys at the moment the chords become more probable in the new key than in the old, whereas humans typically default toward remaining in a single key, requiring more definitive confirmation that a new key has arrived (e.g., a V-I progression in the new key).  It is debatable whether this is really a disagreement: there are arguably two separate questions here, one about when the key-change process begins, and the other about when it ends; our algorithm answers the first question, whereas human analysts often answer the second.  Furthermore, even expert listeners may differ by a beat or two on exactly when a modulation occurs.

A more significant disagreement is seen at the end of measure 9, where our calculation identifies a modulation to A minor that the human analysis omits.  This also is a fairly common issue: the computer appears willing to identify brief excursions from the current tonal center as modulations, where the human analysis considers the key to be unchanged.  In this case the algorithm sees A minor and B dim chords as implying a temporary key change to A minor.  The issue here is that the algorithm has no notion of “being in one key while briefly using progressions characteristic of another”; it simply calculates the key most likely to produce each sequence of chords.

Once our calculations are complete we can make a quantitative estimate of the accuracy of our method by comparing all chords and keys between the human and computer analyses of all 371 chorales.  For the chords there are a number of ways to perform this comparison, as we discuss in Section <ref>, but the simplest is to take each human-annotated chord and translate it into one of the three categories used by the computer—major, minor, and diminished/dominant 7th—then compare this to the category found at the same point by our algorithm.  When we do this, we find agreement in 87.9% of cases, averaged over all 371 chorales.  Comparing keys is more straightforward, since the computer's two categories clearly correspond to standard major and minor, so no translation between human and machine language is necessary.  We do allow some flexibility about the timing of key changes: we do not penalize cases in which the computer analysis finds the keys correctly but there is disagreement about the timing of the change by up to two steps (as in measures 1 and 2 of Fig. <ref>).  With this provision, we find agreement of keys 84.3% of the time.  These numbers are comparable with the accuracy of previous automated approaches for finding chords and keys, and in the vicinity of what we estimate, intuitively, to be the level of agreement among human annotators.  Exact comparisons with previous work are difficult because other approaches have typically focused on determining either the chords or the keys, but not both, and have used a variety of data sets varying in size, content, and style.  Burgoyne and Saul <cit.> report 83% accuracy in determining chords from pitch-class profiles, a little worse than our result; Sheh and Ellis <cit.> report 75% on a similar task.  Nápoles López  <cit.> summarize a number of previous attempts at key identification, some of which achieve somewhat higher accuracy than ours, though without inferring chords, and in some cases using smaller or simpler data sets or hand-tuned parameter values.






 §.§ Probabilities for transitions between chords and keys

Each of our two hidden Markov models also contains probability parameters that represent the likelihood of transitions, between chords on the one hand and keys on the other.  Figure <ref> depicts the chord-transition probabilities.  The chords, represented by the dots in the figure, are arranged around a circle of fifths and divided into three concentric rings, corresponding to the three types, major triads in the outermost ring, then minor triads and diminished/dominant seventh chords.  Transitions, represented by the arrows, are transposed so as to begin at a chord with a C anchor.  Thus a transition up a fifth is represented by an arrow from C to G; down a fifth is represented by an arrow from C to F.  The varying probabilities themselves are represented by the thickness of the arrows.

Once again, the results found by the algorithm correspond closely to accepted understanding of chord movements, with strong transitions of major chords up and down a fifth (C major  F major and C major  G major) and minor chords down a fifth (C minor  F minor) and a third (C minor  A dim, the latter potentially standing in for F7).  The most frequent diminished-chord transitions are C dim  D major and C dim  D minor, with C dim here potentially standing for A7, so that these correspond to V7  I, V7  i, vii  I, or vii  i.

The corresponding results for transitions between keys are shown in Fig. <ref> and again coincide with intuition, the most frequent changes being between the relative major and minor (C minor  E major and C major  A minor), one step around the circle of fifths (C  G and C  F), and two steps (C major  D minor and C minor  B major).

While these probabilities are broadly consistent with standard music theory, the correspondence is somewhat indirect.  This is because the probabilities are defined in terms of the movement of the anchor—a C major chord goes to F major for instance, or the key of C major goes to D minor—but not relative to any key center.  By contrast, music theoretical concepts are typically defined in a key-relative fashion: the tendency of the dominant chord to progress to a tonic, or for the dominant key to modulate to the relative minor.  (In the case of chord progressions the relevant key is the local key while in the case of modulations it is the global tonic of the entire piece.)  Our model's probabilities thus average over syntactically distinct transitions: the probability of the chord progression C major  F major is an average of I-IV in C, V-I in F, and so on.






§ MACHINE AND HUMAN ANALYSES


In Section <ref> we evaluated our model's performance by translating human annotations into the language of the hidden Markov model.  In this section we do the reverse, translating the model output into standard harmonic labels, with the goal of both examining the algorithm's performance in greater detail and also of producing an annotated corpus potentially useful to other researchers.

There is some disagreement about the optimal terminology for harmonic analysis.  Generally speaking, music theorists have taken one of three approaches: functional theories use flexible chord categories that do not completely determine a chord's note content (e.g., “dominant”); scale-degree theories use Roman numerals that presuppose tertian (“stack of thirds”) structure (e.g., “V7”); and figured-bass theories simply list intervals above a bass scale degree (e.g., “a sixth, fourth, and third above the supertonic”).  These competing approaches complicate the attempt to find a ground truth against which we can compare our model, as the choice of terminology is itself somewhat controversial.

The target language we use here combines Roman numerals indicating a chord's pitch-class content relative to the key with figured-bass symbols that indicate both the bass note and the presence or absence of a seventh.  Table <ref> summarizes the labeling system, which is described more fully in <cit.>.  These categories are sharp, imposing strict limits on which notes can be considered harmonic at any time: in C major the symbol V65 corresponds to a harmony containing the notes {B, D, F, G} with B in the bass.  Notes may be omitted but no other notes can be included within the harmony.  (To be sure, the style permits “nonharmonic” tones—suspensions, neighbors, passing tones, pedals, and anticipations—but these are typically understood as extraneous to the harmony, and subject to additional contrapuntal laws.)  The sharpness of these labels contrasts with the more flexible categories used in jazz and other styles, which permit the addition of non-notated harmonic tones such as ninths, elevenths, and thirteenths.



Our two-level hidden Markov model approach labels both chords and keys with ordered pairs (x, y), where x represents the anchor pitch and y is a probability distribution denoting a class of chords or keys.  For keys, the interpretation is straightforward: the anchor is the tonic note and the two probability distributions of Fig. <ref> give the probabilities of occurrence of various chords in major and minor keys.  As mentioned, this approach matches our ground-truth keys 84.3% of the time.

The interpretation of chords is less obvious.  Each probability distribution in Fig. <ref> encompasses a number of different chords, so there is no straightforward translation from distributions to chord symbols.  An effective strategy is to use the score to disambiguate the analysis, inferring the harmonic symbol that best reflects the sounding notes at each step.  Essentially the hidden Markov model constrains the range of possibilities to the point where simple rules can fill in the remaining details.

In this section we describe three different translations of the hidden Markov model output into chord symbols, ordered by their degree of reliance on the score.  For the interested reader, the results are available for download in the RomanText format, which can be read by software packages such as music21 <cit.>.



  
*Method 1.  Our first translation assigns the first probability distribution in Fig. <ref> to the major triad and the second to the minor triad (ignoring the small probability of a sixth or seventh above the bass).  The third category, however, can correspond to five different chords: diminished triads, diminished sevenths, and half-diminished sevenths with root given by the anchor pitch and dominant sevenths or major triads with root a major third below the anchor.[In a major key these chords are fairly similar and can often be aggregated without much loss of information.  In a minor key, however, they can be quite different: in A minor, the diminished triad B-D-F is typically a predominant progressing to E7, whereas the dominant seventh B-D-F-G is an applied dominant typically progressing to C.]  We resolve this ambiguity by taking as harmonic all those notes present in the score that are either 0, 3, 6, or 8 semitones above the anchor.  This allows for diminished triads, major triads, and dominant sevenths.  Diminished and half-diminished sevenths are identified by first identifying a diminished triad and then the appropriate seventh from the score.  We identify bass notes as the lowest chord tone in the sounding harmony.  The resulting analyses are available in the file “TN2022raw.zip.”





This simple translation matches our ground-truth chords 79% of the time, conditional on getting the key correct.  The lower success rate reflects the fact that this comparison is more demanding than the one we performed in Section <ref>, where we asked only whether the hidden Markov model has assigned each chord to the correct one of the three categories.  Here a correct result requires identifying key, Roman numeral, bass note, and the presence of a seventh.  Table <ref> shows the most common mistakes made by the algorithm, relative to the human ground truth.  In this table “suspension” refers to the misanalysis of a chord produced by a suspension, “sixth/seventh” refers to the analysis of a seventh chord as a triad with added sixth, and “missing seventh” refers to the neglect of a harmonic seventh present in the music.



  
*Method 2.  We now consider several ways to improve the translation, by developing better rules that more accurately reflect human judgment.  As we will see, many of these rules reflect notational conventions rather than substantive syntactical questions.



  
*Keys.  One issue with the key identification is that the hidden Markov model tends to modulate more readily than a human analyst would.  We can improve the accuracy somewhat, relative to the human ground truth, by eliminating very small key regions that are sandwiched between two identical keys, such as a stretch of C major followed by a two-chord stretch of D minor, followed by a stretch of C major again.  This modestly improves the fit between the hidden Markov model and our human corpus, from 84.3 to 85.6%.  It is unclear how the results might be further improved: unlike chord identification, key identification does not seem to be amendable to simple algorithmic fixes. 



  
*Bass notes. In Bach's chorales the bass part normally articulates the true bass note even if it briefly crosses above the tenor.  This is because it was often doubled in a lower octave by the organ.  Thus rather than taking the lowest sounding note to be the bass, we should consider the bass voice to be the carrier of the bass note at all times.  Like the other musical voices, however, the bass sometimes sounds nonharmonic tones not belonging to the chord.  To deal with this issue, we distinguish the step or instant of musical time from the chord span, a maximal contiguous span of steps assigned identical analytical symbols, with the understanding that the true bass note may not be present in every step in a span.  We can define the initial bass note in a given chord span as the first harmonic note in the bass part.  This remains in force until it is displaced by another harmonic note in the bass part within the same span (Fig. <ref>).



  
*Chord labels. We have seen that the third chord category in Fig. <ref> is ambiguous, encompassing a diminished triad, diminished or half-diminished seventh, dominant triad, or dominant seventh a third below.  Close inspection shows that the other two categories are also somewhat ambiguous.  The first, shown at the top of Fig. <ref>, has 0, 4, and 7 semitones as its most probable intervals above the anchor, with smaller probabilities for 9 and 11 semitones.  This means that it could potentially represent {C, E, G}, {C, E, G, B}, {C, E, G, A}, or {C, E, A}.  Similarly, the second category could represent the minor triad {C, E, G}, the minor seventh {C, E, G, B}, or the half-diminished seventh {C, E, G, A}.  Thus, the same set of notes can be analyzed in multiple ways: {A, C, E, G} can be either category 1 or 2 (corresponding to either C major with an added sixth or A minor with an added seventh) and {A, C, E, G} can be analyzed as either category 2 or 3 (corresponding to C minor with an added sixth or A diminished with an added seventh).

As we discuss in the next section, this sort of multivalence is characteristic of some harmonic theories.  Here, however, our goal is to match a particular style of analysis in which chord symbols set a strict limit on the allowed harmonic scale degrees.  To do this, we start by distinguishing the sounding notes occurring within a particular step from the persistent notes, those sounding in every step of the chord span.  We then develop rules that determine whether a given note is harmonic or not, depending on whether it is found in the persistent notes. The best match to the human analysis requires a slightly different strategy for each of our three chord categories.  

For the first (the “major chord”) we consider the major sixth or major seventh to be harmonic only if they belong to the persistent notes, but the minor seventh to be harmonic if it belongs to the sounding notes.  This is because the major sixth and major seventh often appear as unlabeled passing tones, while the minor seventh signals a dominant seventh, which is very likely to be labeled in a human analysis.  The final possibility, the major triad with a minor sixth, is very rare in Bach's chorales.

For the second category (“minor chord”) we consider the sixth to be harmonic only if it belongs to the persistent notes, but the minor seventh to be harmonic if it belongs to the sounding notes.  (The major seventh over a minor triad is again very rare.)  For the third category we consider the minor sixth, diminished seventh, and minor seventh to be harmonic if they occur in the sounding notes.  

These rules reflect both notational conventions, such as when to label a seventh, and substantive beliefs about musical structure, e.g., that a major chord with a minor seventh almost always represents a dominant seventh chord.  Requiring that a note belong to the persistent notes sets a high standard for harmonicity: such notes only count as chord tones if they last for an entire chord span, in which case they are very unlikely to be nonharmonic.



  
*Cadential formulas. A final issue involves idiomatic cadential suspensions that human beings would normally consider nonharmonic, such as (G3, C4, D4)  (G3, B3, D4).  Our algorithm often analyzes this progression as two different chords, a C major (missing the E and with a D in the persistent notes) followed by G major.  We can eliminate most of these errors by hand-coding a small set of standard suspension patterns: for instance, if the persistent notes in a category-1 chord include the major second above the anchor (e.g.,{C, D, G}), and if the music progresses to another category-1 chord up by fifth (e.g., {B, D, G}), then we eliminate the initial chord, extending the second chord backward to cover its predecessor.  These rules are summarized in Table <ref>.  Here it is worth noting that four-chord versions of our model typically settle on suspension formations as a fourth probability distribution, underscoring their importance to the style.



Thus improved, the translation—our second of three—now agrees with the human annotations on about 87% of the chordal assignments, conditional on agreeing about the key.  (The annotations are available in the file “TN2022.zip.”)  We have obtained this agreement at the cost of using a modest amount of knowledge about the underlying musical style: without this it would be difficult to translate the three-category output of the hidden Markov model into the much richer symbol system of standard harmonic analysis.  Importantly, this knowledge is embodied by general rules that are independent of local musical context: relative to key, the same set of notes is always treated the same way. 



  
*Method 3.  Our third translation improves accuracy still further by allowing for “nonharmonic consonances”—triads and seventh chords produced by nonharmonic tones and not part of the harmonic narrative.  (Gottfried Weber called these “apparent chords” <cit.>.)  These nonharmonic consonances are an important source of the holism mentioned in the introduction: choosing between the analyses in Figs. <ref> and <ref> requires a sense of which harmonic progressions are likely to occur, expectations that must be in place prior to analyzing a given passage.  Our third method thus introduces a degree of context-dependency into the analyses.

We can simulate background knowledge using the results of our hidden Markov model analysis.  First we count every two-chord progression in the dataset annotated using our previous method and use these counts to estimate the frequency of each progression.  Second, we encode a basic set of rules about the nonharmonic tones permissible in Bach's style—the “contrapuntal grammar” that was largely inherited from the Renaissance.  (Thus we stipulate, for example, that passing and neighboring tones are out-of-chord notes that are preceded and followed by chord-tones to which they are connected by step.) These rules give us a set of contrapuntally permissible analyses for every step.  Third, whenever we find two eighth-note harmonies occurring within a single quarter-note beat, we ask whether either of those harmonies could be analyzed as nonharmonic.  This gives us a range of possible chord progressions, from which we choose the one whose chord progressions have the highest average frequency in our count of two-chord progressions.  The result can be thought of as the “most likely analysis,” given harmonic expectations based on the raw hidden Markov model output and the additional rules described here.

Figure <ref> shows how the algorithm treats four superficially similar passages.  Notes marked “p?” are flagged as potentially nonharmonic and we consider one reading in which the note is a passing tone and another in which it is harmonic.  The numbers underneath each two-chord progression show the number of occurrences of this progression in our previous analysis.  The algorithm chooses the reading with the highest average frequency, shown in bold and surrounded by a box.  This sometimes requires treating superficially similar passages in different ways.  Our third approach achieves this context-dependency in a rule-governed way, relying on first-pass statistics to generate a most-probable interpretation from among a set of contrapuntally permissible alternatives.  By contrast, human beings make context-dependent judgments whose origin and justification can be hard to understand.



This pruning of the analysis improves accuracy to the point where we match the human chord symbols 90% of the time, assuming agreement about the key.  The resulting data set, which is accurate enough for a range of music-theoretical purposes, is available as “TNpruned.zip.”  The high degree of agreement with the human analysis demonstrates that analytical competence does not require much training or experience.  Instead it can be learned from a suitable corpus of music, along with a modest amount of style-specific knowledge.  The subtle analytical judgments that go into Fig. <ref> can be simulated with relatively presupposition-free methods, making almost no assumptions about the specific harmonic syntax of the music.




§ TESTS OF WESTERN HARMONIC THEORY


The most obvious use for these datasets is as a neutral testing-ground for theories of Western harmonic structure.  The biggest question here is whether traditional music has a chord-to-chord harmonic structure akin to that of a first-order Markov model, articulated by rules of the form “V chords generally go to I chords but not IV chords.”  Many theorists believed such structure exists, including Rameau, Weber, Riemann, Piston, Kostka, Payne, and other contemporary pedagogues.  A more recent alternative is that harmony is characterized by a hierarchical structure rooted in melodic laws—a view originating with Heinrich Schenker <cit.> and endorsed by post-Schenkerians such as Lerdahl and Jackendoff <cit.>.  In recent years, this alternative perspective has been incorporated into introductory textbooks <cit.>.

Our data provide strong evidence for the existence of chord-to-chord syntax.  The most popular chords and transitions found in the major-mode passages of the chorales can be visualized as shown in Fig. <ref>.  In this figure, symbol size corresponds to frequency while arrow size indicates the relative frequency of the transitions.[The figure shows Roman numerals accounting for at least 5% of all chords in the chorales' major-key passages (disregarding inversion), along with all transitions accounting for at least 0.75% of all transitions.] This representation is reminiscent of the prescriptive maps found throughout the theoretical and pedagogical literature <cit.>.  While the presence of this chord-to-chord structure does rule out additional hierarchical organization, it does suggest that any such organization must either produce or complement the music's nonhierarchical organization.  This is significant insofar as Schenkerian theorists have sometimes rejected <cit.> or neglected <cit.> this sort of chord-to-chord structure.

Figure <ref> provides the same information for modulations, transposing all major-key pieces to C major and all minor-key pieces to A minor.[In making this graph we selected only those chorales that start and end in the same key, transposing them to so that the starting key was either C major or A minor.] Here the most common modulations are oscillatory: keys tend to move from the tonic to a subsidiary key and back, rather than on longer journeys through multiple subsidiary keys.  This is likely a feature of the chorales in particular, and not characteristic of genres such as the classical sonata.  In this respect the modulatory grammar is unlike the chord-to-chord grammar, which describes a much wider range of harmonic practices—indeed, a good portion of eighteenth- and nineteenth-century music.





Another longstanding area of theoretical controversy is the principles governing “doubling,” or the duplication of elements in four-part triadic harmony.  Aarden and von Hippel have surveyed the astonishing range of disagreement surrounding this subject <cit.>.  Table <ref> shows which notes Bach tends to double in major mode, as extracted from our automated analysis.  In all cases we find that the bass is the most frequently doubled, unless it is a leading tone in which case it is not doubled.  Note for instance that the bass of the first-inversion tonic chord is frequently doubled despite longstanding injunctions to the contrary.  For all the controversy surrounding the subject, Bach's practice is fairly clear.



There are other potential analytical uses for our data.  Combined with the musical score, our analysis allows one to algorithmically catalog nonharmonic tones (Table <ref>), identify the specific voice leadings used in a given harmonic progression, explore the resolution or nonresolution of tendency tones (e.g., leading tones and sevenths), detect parallel perfect intervals either on the musical surface or masked by nonharmonic tones, and search for examples of various phenomena of interest (e.g., unusual chord progressions).  In this way our data provides a quantitative tool for grounding theoretical assertions that have traditionally been supported intuitively.

Our work also suggests some more general theoretical lessons.  Theories of Western harmony are subject to a tradeoff between the determinacy of chord categories and the complexity of chord grammar.  This issue dates back to Jean-Philippe Rameau (1683–1784), celebrated as “the Newton of music” and in many ways the progenitor of contemporary harmonic theory.  Rameau is sometimes credited with being the inventor of “inversional equivalence,” the thesis that pitch-class content determines chordal identity.  From this perspective the notes {C, E, G} always constitute a C-major chord no matter how they are arranged in register.  When pitch classes are labeled relative to a tonal center, this implies that a chord's scale-degree content determines its harmonic category.  In fact, however, Rameau was not a pure scale-degree theorist, and his chord-categories had a flexible relation to pitch-class content.  For example, his 1722 Treatise on Harmony explained the deceptive cadence V7-vi as a descending-fifth progression in which the second chord was represented not by the root, third, and fifth but by root, third, and sixth <cit.>.[Rameau later abandoned this analysis of deceptive cadences, although followers such as d’Alembert continued to uphold it <cit.>.  However, Rameau never abandoned the analogous analysis of the predominant chord with added sixth, arguing that {F, A, C, D} in C major could have a fundamental bass of either F or D, or indeed both.]  Thus on this view the notes {A, C, E} could represent either a chord whose root is A or a chord whose root is C.



Here Rameau opted for a simpler harmonic grammar at the expense of a more complex categorization scheme: he described both the authentic and deceptive progressions with a single rule, “G7 chords progress down by a fifth to C chords.”  But this simplification came at the cost of a categorization scheme in which chordal identity could not be identified from note content alone.  Instead, context determined whether the notes {A, C, E} represented C major or A minor.  This can be considered a functional approach to chord categorization, as a chord's label depends on its local behavior.  Functional approaches can be controversial insofar as they identify chords on the basis of pre-existing theory: {A, C, E} might be identified as a kind of C chord when preceded by G7 but a kind of A chord when preceded by E7.  Thus the claim that chords descend by a fifth is not simply a hypothesis supported by data, but a rule that regulates analysis.  

Something similar can be said of Hugo Riemann, one of the most influential nineteenth-century theorists.  Riemann considered the basic chords to be tonic, subdominant, and dominant, or I, IV and V in Roman numeral notation, arguing that tonal phrases typically articulate a series of I-IV-V-I progressions <cit.>.  However these basic chords were subject to category-preserving transformations: the “parallel” transform replaces a chordal fifth with the sixth above the root, and the “leittonwechsel” replaces a chordal root with the note a semitone below.  This allowed Riemann to describe both G7  C and G7  Am as instances of a single underlying dominant-to-tonic pattern.  In G7  C, the tonic is represented by its primary triad, while in G7  Am it is represented by the primary triad's parallel variant.  Once again, the result is an ambiguity at the level of chord categories: the chord {A, C, E} can represent either a tonic (with A replacing the fifth of {C, E, G}) or subdominant (with E replacing the root of {F, A, C}).  

Our unsupervised hidden Markov model learns a series flexible chord categories that resemble those of Rameau and Riemann.  Like those theorists, it groups the diminished triad {B, D, F} with the dominant seventh {G, B, D, F} into a single category.  Like Rameau, it analyzes the notes {D, F, A, C} sometimes as an F major chord with added sixth and sometimes as a D minor chord with added seventh.  This is not because it considers these chords to “sound similar,” nor because it is committed to metaphysical claims about the priority of descending-fifth progressions.  Rather it does so solely for reasons of information-theoretic efficiency, mutually adjusting probability distributions and transition matrices so as to maximize its ability to predict the future.  What is surprising is that this process reproduces results familiar from the history of music theory.  Those claims have sometimes been denigrated as methodologically disreputable <cit.>, but our work suggests they can be given an algorithmic foundation: musical “function” can be understood information theoretically, as the maximally parsimonious description of the input data.[This result is consistent with work of Jacoby, Tishby, and Tymoczko <cit.>, who use the “information bottleneck” to cluster Roman numerals into categories, recovering something resembling traditional functional theory.]

Contemporary North American theory tends to prefer more determinate chord labels at the cost of a more complex harmonic grammar: in this approach, {C, E, G} is a C-major chord and {A, C, E} is an A minor chord regardless of context.  This in turn requires a more complex set of harmonic rules according to which G7 progresses most often to {C, E, G} (the “authentic” resolution) and less often to {A, C, E} (the “deceptive” resolution).  This approach generally allows for a fine-grained description of musical practice, at the cost of being less enlightening about why G7 progresses as it does.[There are some circumstances where contemporary North American theory uses functional rather than scale-degree labels.  The most common involves cadential _4^6 chords such as {G3, C4, E4} in C major.  Where a pure scale-degree theory would label this chord I64, many contemporary theorists use V64 to record the fact that it typically embellishes and precedes a dominant.  Other circumstances include the assimilation of vii to V (as a “dominant seventh lacking its root” <cit.>), the distinction between “plagal” and “predominant” functions of IV <cit.>, and the correct labeling of chords like {G3, B3, E4} in C major (which can be considered a dominant or a mediant <cit.>).]  In standard Roman-numeral theory this is just a fact about the grammar, whereas for Riemann and Rameau {A, C, E} is portrayed as being intrinsically similar to {C, E, G}.  In this way, the different approaches serve somewhat different purposes: the general categories of Rameau and Riemann provide a partial explanation for tonal practice, one that was useful for composers working in the style.  The more precise categories of contemporary theory, by contrast, may help scholars articulate a more detailed grammar of historical practice.

This suggests a pluralistic picture in which there is more than one way to describe harmony.  The most specific simply represents harmonies as collections of scale-degrees above a bass.  This is likely how the earliest functional composers thought—musicians like Corelli, Vivaldi, Handel, and J. S. Bach, who wrote before the widespread acceptance of inversional equivalence.  More general are the theory-laden harmonic categories of North American theory, Roman numerals presupposing stack-of-thirds harmonic structure.  These facilitate the description of general patterns that can appear over a variety of different bass notes, such as “V chords generally progress to I chords.”  The even more general theories of Rameau and Riemann use still more flexible categories that help explain the principles underlying the grammar.  Our hidden Markov model natively produces something very much like this last approach, though as we have seen it can be translated into Roman numerals.

These different approaches each have their partisans among contemporary theorists: Gjerdingen <cit.> and Holtmeier <cit.> have advocated for figured-bass representations; Tymoczko <cit.> for Roman numerals; and Agmon <cit.> for harmonic functions.  Yet the different analytical languages are arguably optimized for different musical purposes.  Instead of asking which approach is best, it might be better to ask which is better for a particular application.  In particular, it seems likely that more flexible categories are better suited for introductory or creative contexts, where the goal is to apply broad principles in novel contexts, while the more fine-grained language may be better for detailed syntactical investigation.  In other words, introductory textbooks might benefit from the functional approach, while more specialized investigations might need the language of Roman numerals.



§ CONCLUSIONS



There is at best a tenuous consensus about the basic concepts of standard harmonic theory.  Schenker and many of his successors deny that harmonies obey any local chord-to-chord grammar, proposing that music is instead governed by purely linear laws.  Functional theorists follow Rameau and Riemann in promoting flexible categories that can be represented by many different triads.  Schoenbergians, along with Ian Quinn and Christopher White, challenge the distinction between harmonic and nonharmonic—–the very picture of tertian harmonies decorated by “merely linear” tones.  And figured-bass theorists from J. S. Bach to Holtmeier and Gjerdingen have advocated for figured-bass symbols.

Against this background of widespread disagreement, it is remarkable that our largely assumption-free computation manages to reproduce so many familiar features of traditional theory.  Its two key categories are directly interpretable as major and minor.  Its three chordal categories are broadly synonymous with “major,” “minor” and “diminished or dominant.”  Yet at a more fine-grained level these categories admit a degree of flexibility that recall the function-theoretical ideas of Riemann and Rameau: the notes {F, A, C, D} can be interpreted either as a D minor chord with added seventh or an F major chord with added sixth.  We have seen that it is possible to translate these flexible categories into the more precise terminology of North American theory, where added sixths are unknown and there is a sharp distinction between D minor and F major.  Indeed, our automated analyses largely accord with the human ground-truth annotations.  

Our work results in several datasets that can be used for further music-theoretical and musicological inquiry.  Though perhaps less accurate than purely human analytical corpora, they are also less subjective, being largely independent of human biases: thus we can be fairly sure that regularities appearing in these data are not a mere byproduct of analytical preconceptions.  Furthermore, all the assumptions that go into producing the corpora are clearly spelled out, embodied in the computer programs we have used. Though this paper has not focused on detailed corpus study, we have already observed some non-obvious features of musical structure, such as oscillatory rather than progressive modulatory norms (Fig. <ref>).  One project for future work is to explore these datasets more thoroughly.

Another is to develop methods that deal with a broader range of music.  Bach's chorales are unusual in having a fixed number of voices, avoiding rests and incomplete chords, using an extremely regular harmonic rhythm, and explicitly delineating voices and phrases in their notation.   Classical-style harmonies, by contrast, change irregularly, are often incomplete, and feature a variable number of voices not explicitly embodied in the notation.  It is not obvious that we should be able to analyze this music with largely assumption-free methods.  Instead, it may be necessary to make further assumptions about classical syntax.  Solving this problem, however, would open the door to a new era for music theory and music history, one that exploits big data and large-scale statistical analysis.





6cm0.5pt









§ RESULTS OF INDIVIDUAL RUNS


Our calculations consist of two separate parts: the analysis of notes to identify chords and the analysis of the resulting chords to identify keys.  Over multiple repetitions of the calculation the second step—the identification of keys—reliably converges to only a single solution with key classes corresponding to major and minor keys.  No other solution is ever found.  The first step, in which the chords are identified, converges most often to the solution described in Section <ref>, with three chord classes corresponding to major, minor, and diminished/dominant seventh chords.  In 37 of the 50 runs we performed—roughly 75% of the time—we found these chord classes.  In the remaining 13 runs we found other solutions, typically with two of the conventional chords plus one alternate.  A common alternate, for instance, is the dominant seventh with suspended fourth {C, F, G, B}, which occurs in place of the dominant seventh.

These findings, however, somewhat understate our results, in that the alternate solutions, while occurring a moderate fraction of the time, do so with hugely diminished probability.  The probability of these solutions, as quantified by the data likelihood P(𝒞|θ) (see Appendix <ref>), is much lower than for the major/minor/dominant solution.  For example, the highest likelihood solution found overall—which has the major/minor/dominant form—has a data log-likelihood of -118926.5 (base e).  The best of the alternate solutions has a log-likelihood of -121413.4.  This means that even this best alternate is lower in probability by the stupendously large factor e^-2486.9≃ 10^1080.  Thus the major/minor/dominant classification is very heavily favored in our analysis.

Figure <ref> shows the note probabilities for the chord classes in the top 20 of our calculation runs, those with the highest data likelihoods—each panel is the equivalent for the given run of Fig. <ref> in the main text.  Of these twenty plots, every single one corresponds to the major/minor/dominant solution.  In fact, all of the top 31 runs give this solution; one has to go down to the 32nd-best run out of 50 to find one of the alternates.  Again the results speak clearly: the standard major/minor/dominant combination is the best way to describe the structure of these data.




§ THE BAUM-WELCH ALGORITHM


The Baum-Welch algorithm is an expectation-maximization (EM) algorithm for estimating the optimal values of the parameters of a hidden Markov model.  We describe the process here in the language of keys and chords, but the same algorithm applies to any hidden Markov model.

Suppose we have a corpus of N works labeled by i=1… N, with the ith work having M_i chords C_i = ( c^i_1, …, c^i_M_i ).  The parameters of the hidden Markov model, which are the same for every work, are the transition probabilities β = {β_kk'} and chord probabilities ν = {ν_kc} as defined in Section <ref>, plus the prior probabilities π = {π_k } that the first key in a work is k.  (In our calculations we assume π_k to be the same for all keys, so it plays no role, but we include it in the derivations for the sake of completeness.)  Denoting the parameters collectively by θ = {π, β, ν}, the probability within the model of a sequence of keys K_i = ( k^i_1,…,k^i_M_i ) for the M_i steps of work i is

    P(K_i|θ) = P(k^i_1|θ) ∏_t=2^M_i P(k^i_t|k^i_t-1,θ)
      = π_k^i_1∏_t=2^M_iβ_k^i_t-1k^i_t,

and the probability of the chords given the sequence of keys is

    P(C_i|K_i,θ) = ∏_t=1^M_i P(c^i_t|k^i_t)
      = ∏_t=1^M_iν_k^i_tc^i_t.

Hence the probability of both the keys and the chords is

    P(K_i,C_i|θ) = P(C_i|K_i,θ) P(K_i|θ)
      = π_k^i_1ν_k^i_1c^i_1∏_t=2^M_iβ_k^i_t-1k^i_tν_k^i_tc^i_t

and the probability for the entire corpus, with keys = { K_1,…, K_N } and chords = { C_1,…, C_N }, is the product of this expression over all works:

    P(,|θ) = ∏_i=1^N P(K_i,C_i|θ)
      = ∏_i=1^N π_k^i_1ν_k^i_1c^i_1∏_t=2^M_iβ_k^i_t-1k^i_tν_k^i_tc^i_t.


Using Bayes rule we write the probability of the unknowns  and θ as

    P(,θ|) = P(,|θ) P(θ) P(),

which for given chords  is proportional to P(,|θ) assuming a uniform prior P(θ)= on the parameters θ.

We find our best estimate of the parameter values θ by maximizing the log-probability

    log P(θ|) = log∑_ P(,θ|) ∝log∑_ P(,|θ).

We now apply Jensen's inequality, which says that for any set of positive quantities x_i,q_i>0 with i=1… n and ∑_i=1^n q_i = 1 we have

    log∑_i x_i ≥∑_i q_i logx_i q_i.

Applied to Eq. (<ref>) this tells us that

    log∑_ P(,|θ) ≥∑_ q() logP(,|θ) q(),

where q() is any probability distribution over  satisfying ∑_ q() = 1.  The exact equality is achieved when

    q() = P(,|θ)∑_ P(,|θ),

which necessarily maximizes the right-hand side of (<ref>), and a subsequent maximization with respect to θ will then find the optimal parameter values.  Put another way, a double maximization of the right-hand side gets us the answer we want.  In the EM approach we perform this double maximization by simply maximizing with respect to q() with θ held constant, then with respect to θ with q() constant, and repeating until we reach convergence.  Maximization with respect to q() is achieved by applying Eq. (<ref>) as we have said.  Maximization with respect to θ is achieved by differentiating the right-hand side of (<ref>) and setting the result to zero.

For instance, the derivative with respect to β_kk' takes the form

    ∂∂β_kk'[ ∑_ q() log P(,|θ) - λ∑_l β_kl] = 0,

where we have enforced the necessary constraint ∑_l β_kl = 1 with a Lagrange multiplier λ.  Substituting from Eq. (<ref>) for P(,|θ), taking the log, and performing the derivative, we then find that

    β_kk' = ∑_i=1^N ∑_t=1^M_i-1 q^i_t,t+1(k,k')∑_k'∑_i=1^N ∑_t=1^M_i-1 q^i_t,t+1(k,k'),

with

    q^i_t,t+1(k,k') = ∑_ q() δ_k^i_t,kδ_k^i_t+1,k',

where δ_ij is the Kronecker delta, which is 1 when i=j and 0 otherwise.  A similar derivative with respect to ν_kc, enforcing the constraint ∑_c ν_kc = 1 with another Lagrange multiplier, gives

    ν_kc = ∑_i=1^N ∑_t=1^M_i q^i_t(k) δ_c,c^i_t∑_i=1^N ∑_t=1^M_i q^i_t(k),

where

    q^i_t,t+1(k,k') = ∑_ q() δ_k^i_t,kδ_k^i_t+1,k'.


The EM algorithm involves choosing an initial set of parameter values θ = {π, β, ν} and applying Eq. (<ref>) to calculate q(), then Eqs. (<ref>), (<ref>), (<ref>), and (<ref>) to calculate new values of the parameters and iterating until the parameter values converge.  It can be proved that this procedure always converges to a local optimum of the parameter values, but not to the global optimum.  In practice, therefore, one typically runs the algorithm several times and, if different runs converge to different optima, accepts the solution with the highest log-probability, Eq. (<ref>).

Equations (<ref>) and (<ref>) are easily implemented.  The tricky part of the calculation is evaluating the probability distribution q() of Eq. (<ref>) and from it the quantities q^i_t(k) and q^i_t,t+1(k,k').  A naive version of this calculation would involve performing the sum in the denominator of (<ref>), which would be computationally exorbitant, but luckily there exists an efficient algorithm for computing q^i_t(k) and q^i_t,t+1(k,k') directly, the forward-backward algorithm.




 §.§ The forward-backward algorithm

The quantity q^i_t(k) defined in Eq. (<ref>) is equal to the probability that the prevailing key is k at step t of work i.  Similarly q^i_t,t+1(k,k') is the probability that the keys at consecutive steps t and t+1 are k and k'.  The forward-backward algorithm computes these probabilities using a dynamic programming approach.  Focusing on a single work i, we write the joint probability of the key and chord at the first step as

    P(k_1,c_1|θ) = P(k_1|θ) P(c_1|k_1,θ) = π_k_1ν_k_1c_1,

where we temporarily suppress the superscript i for the sake of clarity, since it will be the same throughout.

Given this starting point, we can now calculate the set of subsequent probabilities

    P(k_t,c_1… c_t) = ∑_k_t-1 P(k_t-1,c_1… c_t-1)
      P(k_t|k_t-1) P(c_t|k_t),

where we have also suppressed θ.  Defining the shorthand a_t(k_t) = P(k_t,c_1… c_t), this can then be written as

    a_t(k) = ν_kc_t∑_k'β_k'k a_t-1(k').

Equation (<ref>) tells us that a_1(k) = π_k ν_kc_1, then applying (<ref>) repeatedly allows us to quickly calculate a_t(k) for all keys k and steps t.  This is the “forward” half of the forward-backward algorithm.

The backward half defines a complementary set of quantities b_t(k_t) = P(c_t+1… c_M|k_t), where M is the total number of steps in the work.  We write

    P(c_t… c_M|k_t-1) = ∑_k_t P(c_t+1… c_M|k_t) P(k_t|k_t-1) P(c_t|k_t),

which can be written in the form

    b_t-1(k) = ∑_k'ν_k'c_tβ_kk' b_t(k'),

from which, starting with b_M(k)=1 for all k, we can quickly calculate all b_t(k).

Given the complete set of a_t(k) and b_t(k), we now have

    P(k_t,C) = P(k_t,c_1… c_t) P(c_t+1… c_M|k_t) = a_t(k_t) b_t(k_t),

where C = ( c_1,…,c_M ) as previously.  Thus

    P(k_t|C) = P(k_t,C) P(C) = P(k_t,C)∑_k_t P(k_t,C),

or equivalently

    q^i_t(k) = a_t(k) b_t(k)∑_k a_t(k) b_t(k).

Repeating this calculation for each work i we then have the full set of probabilities q^i_t(k).

Similarly we can write

    P(k_t,k_t+1,C)    = P(k_t,c_1… c_t) P(c_t+2… c_M|k_t+1) 
       × P(k_t+1|k_t) P(c_t+1|k_t+1) 
       = a_t(k_t) b_t+1(k_t+1) β_k_tk_t+1ν_k_t+1c_t+1,

so that

    P(k_t,k_t+1|C) = P(k_t,k_t+1,C) P(C)
      = P(k_t,k_t+1,C)∑_k_t,k_t+1 P(k_t,k_t+1,C),

or

    q_t,t+1(k,k') = a_t(k) b_t+1(k') β_kk'ν_k'c_t+1∑_kk' a_t(k) b_t+1(k') β_kk'ν_k'c_t+1.

Between them, Eqs. (<ref>) and (<ref>) now allow us to complete our EM algorithm and calculate the optimal values of the parameters.




§ THE VITERBI ALGORITHM


Using the Baum-Welch algorithm we can calculate the optimal values of the probability parameters β_kk' and ν_kc.  Given these values we can then calculate the best estimate of the hidden variables using the Viterbi algorithm.  Again we use the language of keys and chords, in which case the hidden variables are the keys.

We define x_t(k_t) to be the probability that the key is k_t on step t and the chords up to that point are c_1… c_t, given that the preceding keys k_1… k_t-1 take their most likely (i.e. highest probability) values and given the values of the parameters θ.  That is,

    x_t(k_t) = max_k_1… k_t-1 P(k_1… k_t,c_1… c_t|θ).

We will henceforth suppress θ, since the parameters are fixed throughout this part of the calculation.

Now we note that

    P(k_1… k_t,c_1… c_t) 
            = P(k_1… k_t-1,c_1… c_t-1) P(k_t|k_t-1) P(c_t|k_t),

and hence

    max_k_1… k_t-1 P(k_1… k_t,c_1… c_t|θ) 
        = max_k_t-1max_k_1… k_t-2
               P(k_1… k_t-1,c_1… c_t-1) P(k_t|k_t-1) P(c_t|k_t),

or

    x_t(k) = ν_kc_tmax_k'β_k'k x_t-1(k').

Starting from x_1(k) = π_k ν_kc_1, we can use Eq. (<ref>) to calculate the full set of quantities x_t(k) for all t and k.

The last of these quantities gives us the probability of k_M when all other keys take their most likely values, so the most likely value of k_M is simply

    k_M = _k x_M(k),

where _k x(k) indicates the value of k that gives the largest value of x.  Then the most likely values of the remaining keys can be calculated in reverse order from

    k_t = _k β_kk_t+1 x_t(k).

The end result is the complete sequence of keys k_1… k_M that has the highest probability for the given values of the probability parameters.




§ IMPLEMENTATION AND RUNNING TIME


Implementation of the Baum-Welch algorithm is a straightforward numerical realization of the equations.  We choose random initial values for each of the probability parameters β_kk' and ν_kc, and then for each work we repeatedly apply Eq. (<ref>) to calculate the quantities a_t(k), Eq. (<ref>) to calculate the b_t(k), and then Eqs. (<ref>) and (<ref>) to calculate q^i_t(k) and q^i_t,t+1(k,k').  Armed with these we then use Eqs. (<ref>) and (<ref>) to calculate updated values of β_kk' and ν_kc, repeating the whole process until the values converge within a predetermined tolerance.

A technical issue arises with applications to longer works because the quantities a_t(k) and b_k(t) can become very small and underflow the numerical resolution of the computer.  To avoid this, we normalize these quantities at each step within a work.  For instance, after calculating a_t(k) in Eq. (<ref>) we compute a normalized version

    ã_t(k) = a_t(k) A_t,

with

    A_t = ∑_k a_t(k),

then use this version in the next step of Eq. (<ref>).  We also perform a similar normalization of b_k(t).  These changes have no effect on Eqs. (<ref>) and (<ref>) since any overall rescaling of a_t(k) or b_t(k) cancels in numerator and denominator of both equations, but they prevent underflows by ensuring that a_t(k) and b_t(k) both sum to 1, so that their average values cannot become arbitrarily small even for long works.

Implementation of the Viterbi algorithm is a similarly straightforward realization of Eq. (<ref>).  No iteration is needed here: one applies the algorithm just once to compute the optimal key sequence for each work.  Potential underflows can be prevented by normalizing the values of x_t(k) while having no effect on the keys computed in Eq. (<ref>).

Most of the running time is taken up by the calculation of the optimal parameter values using the Baum-Welch algorithm, and most of that time is taken by the forward-backward algorithm.  If there are R distinct keys (where R=24 in our case) and if the total number of steps in all works in the corpus is T = ∑_i=1^N M_i, then the calculation of a_t(k) from Eq. (<ref>) for all t and k takes time (TR^2), and similarly for the calculation of b_k(t).  This time gets multiplied by the number of iterations of the Baum-Welch algorithm needed to achieve convergence.  In our calculations we iterate until the values of the parameters change by less than one part in 10 million on each iteration, and this point is typically reached after about 100 iterations.  Total running time in practice for the full corpus of Bach chorales is about five minutes on a standard laptop (circa 2024).

For practical applications to finding keys and chords in written music, however, one would not need to run the Baum-Welch algorithm at all.  In such applications one would normally pre-compute the optimal values of the probability parameters then apply those to infer keys and chords for any given work.  This would require running only the Viterbi algorithm for each work of interest, which is very fast, taking only a fraction of a second on a typical computer.




§ CALCULATION OF THE DATA LIKELIHOOD


The Baum-Welch algorithm maximizes the probability of the model parameters given the data, Eq. (<ref>), but is not guaranteed to find a global maximum.  Like all expectation-maximization algorithms the Baum-Welch algorithm may erroneously converge to a local maximum of the probability.  To guard against this possibility we perform multiple runs of the algorithm with different random initial conditions and then use the results from the run that gives the highest probability.  This does not guarantee that we find the global maximum, but in practice we see the same best result occurring repeatedly, which strongly suggests that there is no better result to be found.

Calculating the value of the posterior probability P(θ|𝒞) is equivalent to calculating the data likelihood P(𝒞|θ), since we assume a uniform prior on the parameters θ, and the likelihood can be calculated conveniently from results we already have.  As described in Appendix <ref>, we calculate as part of the Baum-Welch algorithm the quantities ã_t(k) defined in Eq. (<ref>).  Applying this equation repeatedly we have

    a_M(k_M) = P(k_M,c_1… c_M|θ) = ã_M(k_M) ∏_t=1^M A_t

with A_t as in Eq. (<ref>) and M being the total number of steps in the work.  Summing over k_M, we then have

    P(c_1… c_M|θ)    = ∑_k_M P(k_M,c_1… c_t|θ) 
       = ∑_k_Mã_M(k_M) ∏_t=1^M A_t
      = ∏_t=1^M A_t ,

where we have made use of the fact that ∑_k_Mã_M(k_M) = 1 because of the normalization in Eq. (<ref>).  The total likelihood P(𝒞|θ) of all works in the entire data set is then the product of this quantity over all works.


10
url<#>1urlprefixURL 

WEBER1846
G. Weber, Theory of Musical Composition. Wilkins, Carter, and Company,
  and O. C. B. Carter, Boston ([1817–1821] 1846).

TAOM
D. Tymoczko, Tonality: An Owner's Manual. Oxford University Press, New
  York (2023).

Micchi
G. Micchi, M. Gotham, and M. Giraud, Not all roads lead to Rome: Pitch
  representation and model architecture for automatic harmonic analysis.
  Transactions of the International Society for Music Information
  Retrieval 3, 42–54 (2020).

QUINNWHITE
I. Quinn and C. White, Chord context and harmonic function in tonal music.
  Music Theory Spectrum 40, 314–337 (2018).

NOLAND2006
K. C. Noland and M. B. Sandler, Key estimation using a hidden Markov model.
  In Proceedings of the 7th International Conference on Music
  Information Retrieval (ISMIR 2006), pp. 121–126 (2006).

NAPOLESLOPEZ2019
N. Nápoles López, C. Arthur, and I. Fujinaga, Key-finding based on a hidden
  Markov model and key profiles. In Proceedings of the 6th
  International Conference on Digital Libraries for Musicology, pp. 33–37
  (2019).

QUINN
I. Quinn, Tonal harmony. In A. Rehding and S. Rings (eds.), The Oxford
  Handbook of Critical Concepts in Music Theory, pp. 467–497, Oxford
  University Press, New York (2018).

GJERDINGEN2007
R. Gjerdingen, Music in the Galant Style. Oxford University Press, New
  York (2007).

BURGOYNE2005
J. Burgoyne and L. Saul, Learning harmonic relationships in digital audio with
  Dirichlet-based hidden Markov models. In Proceedings of the 6th
  International Conference on Music Information Retrieval (ISMIR 2005), pp.
  438–443 (2005).

FUJISHIMA1999
T. Fujishima, Realtime chord recognition of musical sound: A system using
  common Lisp music. In Proceedings of the International Computer
  Music Conference, pp. 464–467 (1999).

CHAI2005
W. Chai and B. Vercoe, Detection of key change in classical piano music. In
  Proceedings of the 6th International Conference on Music Information
  Retrieval (ISMIR 2005), pp. 468–473 (2005).

SHEH2003
A. Sheh and D. P. Ellis, Chord segmentation and recognition using EM-trained
  hidden Markov models. In Proceedings of the 4th Internation
  Conference on Music Information Retrieval (ISMIR 2003), pp. 185–191 (2003).

ALBRECHTSHANAHAN
J. Albrecht and D. Shanahan, The use of large corpora to train a new type of
  key-finding algorithm: An improved treatment of the minor mode. Music
  Perception 31, 59–67 (2013).

napoleslopez21augmentednet
N. Nápoles López, M. Gotham, and I. Fujinaga, AugmentedNet: A Roman
  numeral analysis network with synthetic training examples and additional
  tonal tasks. In Proceedings of the 22nd International Conference on
  Music Information Retrieval, pp. 404–411 (2021).

ZUCCHINI2021
W. Zucchini, I. L. MacDonald, and R. Langrock, Hidden Markov Models for
  Time Series. Chapman & Hall, London (2021).

TYMOCZKO2019
D. Tymoczko, M. Gotham, M. S. Cuthbert, and C. Ariza, The RomanText format: A
  flexible and standard method for representing Roman numeral analyses. In
  A. Flexer, G. Peeters, J. Urbano, and A. Volk (eds.), Proceedings of
  the 20th International Conference on Music Information Retrieval, pp.
  123–129 (2019).

CUTHBERT2010
M. S. Cuthbert and C. Ariza, Music21: A toolkit for computer-aided musicology
  and symbolic music data. In J. S. Downie and R. C. Veltkamp (eds.),
  Proceedings of the 11th International Conference on Music Information
  Retrieval, pp. 637–642 (2010).

SCHENKER1997
H. Schenker, The Masterwork in Music: Volume 3, 1930: A Yearbook.
  Cambridge University Press, Cambridge ((1930) 1997).

JACKENDOFF1983
F. Lerdahl and R. Jackendoff, A Generative Theory of Tonal Music. MIT
  Press, Cambridge, MA (1983).

LAITZ2008
S. Laitz, The Complete Musician. Oxford University Press, New York
  (2008).

CLENDINNING2016
J. P. Clendinning and E. W. Marvin, The Musician’s Guide to Theory
  and Analysis. Norton, New York, 3rd edition (2016).

PISTON1941
W. Piston, Harmony. Norton, New York (1941).

KOSTKA2003
S. Kostka and D. Payne, Tonal Harmony. Alfred A. Knopf, New York, 4th
  edition (2016).

TYMOCZKO2011
D. Tymoczko, A Geometry of Music. Oxford University Press, New York
  (2011).

LERDAHL2001
F. Lerdahl, Tonal Pitch Space. Oxford University Press, New York
  (2001).

AARDEN2004
B. Aarden and P. T. von Hippel, Rules for chord doubling (and spacing): Which
  ones do we need? Music Theory Online 10(2) (2004).

RAMEAU1971
J. P. Rameau, Treatise on Harmony. Dover, New York ([1722] 1971).

CHRISTENSEN1993
T. Christensen, Rameau and Musical Thought in the Enlightenment.
  Cambridge University Press, Cambridge (1993).

BURNHAM1992
S. Burnham, Method and motivation in Hugo Riemann's History of Harmonic
  Theory. Music Theory Spectrum 14(1), 1–14 (1992).

JACOBY2015
N. Jacoby, N. Tishby, and D. Tymoczko, An information theoretical approach to
  chord classification. Journal of New Music Research 44,
  219–244 (2015).

DAHLHAUS1990
C. Dahlhaus, Studies in the Origins of Harmonic Tonality. Princeton
  University Press, Princeton (1990).

SWINDEN2005
K. Swinden, When functions collide: Aspects of plural function in chromatic
  music. Music Theory Spectrum 27, 249–282 (2005).

ROTHGEB1996
J. Rothgeb, Re: Eytan Agmon on functional theory. Music Theory
  Online 2(1) (1996).

HOLTMEIER2007
L. Holtmeier, Heinichen, Rameau, and the Italian thoroughbass tradition:
  Concepts of tonality and chord in the rule of the octave. Journal of
  Music Theory 51, 5–49 (2007).

AGMON1995
E. Agmon, Functional harmony revisited: A prototype-theoretic approach.
  Music Theory Spectrum 17(2), 196–214 (1995).





 Paper Title: ReSi: A Comprehensive Benchmark for Representational Similarity Measures.



MUFASA: Multi-View Fusion and Adaptation Network with Spatial Awareness for Radar Object Detection
    Xiangyuan Peng1,2 Miao Tang3 Huawei Sun1,2Kay Bierzynski2Lorenzo Servadei1 Robert Wille1

    August 5, 2024
==================================================================================================



§ ABSTRACT 
Measuring the similarity of different representations of neural architectures is a fundamental task and an open research challenge for the machine learning community.
This paper presents the first comprehensive benchmark for evaluating representational similarity measures based on well-defined groundings of similarity.
The representational similarity () benchmark consists of (i) six carefully designed tests for similarity measures, (ii) 23 similarity measures, (iii) eleven neural network architectures, and (iv) six datasets, spanning over the graph, language, and vision domains.
The benchmark opens up several important avenues of research on representational similarity that enable novel explorations and applications of neural architectures.
We demonstrate the utility of the  benchmark by conducting experiments on various neural network architectures, real world datasets and similarity measures.
All components of the benchmark are publicly available[<https://github.com/mklabunde/resi>] and thereby facilitate systematic reproduction and production of research results.
The benchmark is extensible, future research can build on and further expand it.
We believe that the  benchmark can serve as a sound platform catalyzing future research that aims to systematically evaluate existing and explore novel ways of comparing representations of neural architectures.





§ INTRODUCTION



Representations are fundamental concepts of deep learning, which have garnered significant interest due to their ability to cast light on the opaque inner workings of neural networks.
Studying and analyzing them has enabled insights into numerous problems, e.g., understanding learning dynamics <cit.>, catastrophic forgetting <cit.>, and language changes over time <cit.>.
Such analyses commonly involve measuring similarity of representations, which resulted in a plethora of similarity measures proposed in the literature <cit.>.
However, these similarity measures have often been proposed in an ad-hoc manner, without comprehensive comparison to already existing similarity measures. 
Moreover, they have often been proposed in conjunction with new quality criteria that were deemed desirable, with previously defined quality criteria being ignored.
So far, only the few most popular measures have been compared <cit.> or analyzed in more detail <cit.>.

In this work, we present the first comprehensive benchmark for representational similarity measures.
It comprises six tests that postulate different ground truth assumptions about the similarities between representations that measures could capture.
We implement these tests over several architectures and datasets in the graph, language, and vision  domains.
The  benchmark enables tests for 23 similarity measures that have been proposed in literature and illustrate how the results can provide insights into the properties and strengths and weaknesses of these measures, as well as guide researchers and practitioners in choosing an appropriate measure for an application at hand.
All benchmark code and the corresponding models are openly accessible online. 





§ GROUNDING REPRESENTATIONAL SIMILARITY



Before presenting the  benchmark, we briefly introduce key terms and notations for representational similarity, and discuss how ground-truths for representational similarity can be established.



 §.§ Representational Similarity



The  benchmark is designed to evaluate the quality of measures that aim to quantify the similarity of neural representations.
Such representations can be derived from applying a neural network model 

    f = f^(L)∘ f^(L-1)∘…∘ f^(1),

where each function f^(l):^D'⟶^D denotes a single layer,
on a matrix of N inputs X∈^N× p – for simplicity, we assume each input to be in vector format even though, as in the vision domain, these can also be multidimensional matrices.
Then, at any layer l one can extract the model's representation 

    := ^(l) = (f^(l)∘ f^(l-1)∘…∘ f^(1))(X)
        ∈^N× D.

Unless noted otherwise, we always consider representations from the final inner layer of a neural network.
We define a representational similarity measure as a mapping m: ^N× D×^N× D'⟶ that assigns a similarity score m(,) to a pair of representations ,.
For brevity, throughout this work we will often simply denote these measures as similarity measures.






 §.§ Grounding Similarity




All representational similarity measures quantify representational similarity with a single scalar, but the opinions when two representations should be similar diverge:
For example, similarity can be related to comparing the pairwise distances in a representation <cit.>, the ability to directly align two representations <cit.>, or their topology <cit.>.

While these motivations are usually theoretically connected to differences in models, in practice, differences in models are not necessarily captured by these approaches.
However, these differences, e.g., in their behavior, are often of interest because they cannot be easily measured.
Similarity measures present a potential shortcut to identifying these differences.
Complicating this is that model difference is multi-faceted:
One can distinguish models by their behavior like accuracy, robustness to augmentations or domain shifts, and their preference on texture or shapes, or by aspects like differences in training data or their human-likeness to name a few examples.
This multi-dimensionality leads to a crucial problem in measuring representational similarity, namely the lack of a general ground-truth that measures should reflect.


Due to this plurality in model behavior, a number of possible targets to ground representational similarity have been proposed in literature.
For our purposes, we focus on the following two broad approaches to establish a ground truth for similarity.

1. Grounding by Prediction.
A straightforward way to obtain a ground-truth for model similarity is to consider the differences in the predictive behaviors of a pair of models: when a two models yield different predictions, they should also differ in their representations. 
This approach allows one to ignore where the source of difference between the models originates from, but simultaneously implies that one cannot be certain whether ground truth similarities stems from differences in the classifier or the representations.
Previous efforts that have used this way to ground similarity include the study by <cit.>, who correlated representational similarity with accuracy difference, or the work by <cit.>, who ground similarity to differences in individual predictions.


2. Grounding by Design.
Through careful design, one can construct groups of representations for which one can impose a ground truth about similarity by relation.
For instance, one can demand that representations from the same group should be more similar to each other than representations from different groups.
One example for such a design is displayed by <cit.>, who trained multiple models of the same architecture, and formed groups of representations based on the depth of their producing layer.
Then, they demanded that representations of the same layer should be more similar to each other than to representations from other layers.
These relative comparisons have the advantage, that no deeper insights about the often opaque and non-linear similarity measures are needed.
However, the validity of the evaluation hinges on the validity of the assumption that representations actually follow the expected groups, which requires clear justification.









§ RESI: A BENCHMARK FOR REPRESENTATIONAL SIMILARITY MEASURES



We present the main components of the  benchmark. It consists of (i) six carefully designed tests for similarity measures, (ii) 23 similarity measures, (iii) eleven neural network architectures, and (iv) six datasets, spanning over graph, language, and vision domains.
Further, we discuss how the results for our benchmark are evaluated.











 §.§ Tests

We provide descriptions of six tests implemented within the  benchmark.
We begin with discussing two tests which are grounded by predictions, and then describe four tests which are grounded by design.
An overview of all tests is given in <Ref>.



Test 1: Correlation to Accuracy Difference.
When two models are trained under similar conditions but yield differences in their performance, this can be seen as a signal that the underlying representations are different.
Following this intuition, we correlate the representational similarity of a model pair with the absolute difference in their accuracies, thus similarity is grounded by predictions.


Training Protocol.
We roughly follow the protocol established by <cit.>, for which ten models are trained on each dataset, varying only by the training seeds.
Afterward, we compute the accuracy of the models on the test set. 


Test 2: Correlation to Output Difference.
This test follows the same intuition as Test 1, with the difference that it focuses on the individual output differences of each model rather than the aggregated accuracy of models. 
Given that models with similar accuracy may still yield very different predictions on individual instances, this provides a  more fine-grained signal which can serve as a more robust grounding for the similarity of the underlying representations. 
Thus, similarity is again grounded in predictions. 


Training Protocol.
Due to the similarities to Test 1, we use the same models for this experiment.

Test 3: Label Randomization.
In this experiment, we train models on the same input data, but with labels randomized to different degrees. 
The expectation is that models that learn to predict the true labels learn different representations compared to the models that learn to memorize random labels.
These differently trained models are grouped by their degree of label randomization during training. We evaluate if similarity within a group is higher than between groups.  
Therefore, representational similarity is grounded by design of these groups.


Training Protocol.
For each domain, we create at least two groups of models. 
We always train one group on fully correct labels and one group on fully random labels.
Additional groups have partially randomized labels (25%, 50%, 75%).
Across all domains, we always trained five models per group for each architecture.

Test 4: Shortcut Affinity.
The ability to identify whether two models use similar or different features can be a desirable property for a similarity measure. 
Hence, we create a scenario, in which we control feature usage by introducing artificial shortcut features to the training data. 
Specific features correspond to each label, thus leaking them, but not necessarily perfectly—we can add incorrect shortcut features for some instances.
Groups of models are then formed based on the degree to which the shortcut features in the training data match with the true labels. 
This way, the additional features can be useful or irrelevant for classification, resulting in some models learning to use them and others learning to ignore it.
We can verify the degree to which models rely on the shortcuts by evaluating their performance on data with uncorrelated shortcut features. 
Similar to Test 3, we evaluate if similarities within the same groups are higher than between groups.
That way, representational similarity is grounded by design of these groups.


Training Protocol.
For each domain, we construct one group of models trained on data with shortcuts that always leak the correct label, one group trained on data with completely random shortcut features, and additional groups in which the labels were leaked on a fixed percentage of training labels.
Representations are then extracted on a fixed test set, on which random shortcuts were introduced.
Each group consists of five models, in which training seed affected the model and the shortcuts.


Test 5: Augmentation.
Augmentation is commonly used to "teach" models to become invariant to changes in the input domain which do not affect the labels.
In this test, we evaluate if the similarity measures are able to capture robustness to such changes, by developing models with varying amounts of augmentation.
Similarly to before, we train groups of models with different degrees of augmentation. 
Models of the same group should yield more similar representations than models from other groups trained on differently augmented data – again, similarity is grounded by design of the models.


Training Protocol.
In all domains, we trained one group of reference models on standard data, and additional groups of models with varying degree of augmentation on the training data. 
Each group consists of five models, which only vary in their training seed.
Representations are computed on the standard, non-augmented test data.








Test 6: Layer Monotonicity.
As noted in <Ref>, the individual layers f^(l) of a neural network f all yield representations ^(l) that can be compared. 
Given that these layers also represent a sequence of transformations of the input data , i.e., ^(l) = f^(l)(^(l-1)), it seems intuitive that representations of neighboring layers should be more similar than representations of layers that are further away, given the greater amount of transformations between the further-apart layers.
Thus, in this experiment, we extract the layer-wise representations of a neural network and test whether a representational similarity measure can distinguish pairs of layers based on their distance to each other.
Similarity of representations is thereby grounded by design.

Training Protocol. 
We reuse the trained models of Test 1 and Test 2, as no training scenario changes were made – only for graph neural networks, we increased the number of layers to five inner layers to enable a sufficient number of comparisons.



 §.§ Representational Similarity Measures

 evaluates 23 different similarity measures. A brief overview is provided in <Ref>.
We directly used the reference implementations of the measures where possible.
Otherwise, we closely followed the given definitions and recommendations, also in the preprocessing of representations.

Hyperparameters.
For all neighborhood-based measures (Jaccard, RankSim, 2nd-cos), we set the neighborhood size to k=10 and determined nearest neighbors based on cosine similarity.
For CKA, we chose the commonly used linear kernel.
For RSA, we applied Pearson correlation as inner and Spearman correlation as outer similarity function, and we computed the RSM in RSMDiff based on Euclidean distance.
In GULP, we set the regularization weight to λ=0.
Finally, for the IMD score, we use 8,000 approximation steps and five repetitions.
All other measures did not provide any hyperparameters.






 §.§ Models

Overall, the  benchmark utilizes a range of various graph, language, and vision models.
Details on parameter choices can be found in Appendix <ref>.
All trained models are publicly available.


Graphs.
As graph neural network architectures, we chose the classic GCN <cit.>, GraphSAGE <cit.>, and GAT <cit.> models, using their respective implementation from  <cit.>.
For each experiment and dataset, we trained these models from scratch. 


Language.
We focus on the popular BERT architecture <cit.>.
<cit.> provide 25 BERT models trained with different seeds.
In our experiments, we fine-tuned these models on the given datasets to avoid computationally expensive pretraining.


Vision.
For the  benchmark, we focused on prominent classification architecture families, namely ResNets <cit.>, ViT's <cit.> and older VGG's <cit.>. 
To capture the effect of scaling architecture sizes, we included 
ResNet18, ResNet34, ResNet101, VGG11, VGG19, ViT-B/32 and the ViT-L/32 architectures. 
All models were trained from scratch on the (possibly augmented) dataset, apart from the ViTs which were initialized with pre-trained weights from IN21k.




 §.§ Datasets

We provide a brief overview of the datasets used within the  benchmark. 
More detailed descriptions of the datasets can be found in Appendix <ref>.


Graphs.
We focus on graph datasets that provide multiclass labels for node classification, and for which dataset splits into training, validation and test sets are already available. 
Specifically, we select the Cora <cit.>, Flickr <cit.>, and OGBN-Arxiv <cit.> datasets.
For the Cora graph, we extract representations for the complete test set of 1,000 instances, whereas for Flickr and OGBN-Arxiv, we subsampled the test set to 10,000 instances for computational reasons.


Language.
We use two classification datasets: SST2 <cit.> is a collection of sentences extracted from movie reviews labeled with positive or negative sentiment.
MNLI <cit.> is a dataset of premise-hypothesis pairs labeled with the relation between premise and hypothesis. 
We use the validation and validation-matched subsets to extract representations for SST2 and MNLI, respectively.


Vision.
For the vision domain, we use the ImageNet100 dataset. 
This dataset is derived from Imagenet1k <cit.> by subsampling 100 randomly chosen classes. 
This reduces total sample size by 90%, 
reducing overall training time, while keeping image resolution and content similar to the prominent ImageNet1k dataset.
For all models, image resolution was fixed to 224×224.




 §.§ Evaluation

Lastly, we describe how we evaluate and quantify the performance of representational similarity measures within the  benchmark tests.
Due to the different nature of the two approaches we use to ground representational similarity, we present the corresponding evaluation approaches separately.




  
Grounding by Prediction.

When grounding similarity of representations to the predictions of their corresponding classification models f, we specifically consider the outputs := f()∈^N× C,
where we assume that each row _i=f(X_i)∈^C consists of the instance-wise class probability scores for C given classes.

In Test 1, we leverage these outputs to correlate representational similarity to absolute differences in accuracy.
Thus, letting q_acc() := q_acc(, y) denote the accuracy of an output  with respect to ground-truth labels y∈^N,  we compute the absolute difference in accuracies 

    Δ_acc(, ) = | q_acc() - q_acc()|.

Then, given a similarity measure m, and letting ℱ denote the set of models trained for this test, for all model pairs f, f'∈ℱ we collect the representations similarity scores m(, ) as well as the accuracy differences Δ_acc(, ), and report the Spearman correlation between these sets of values.

In Test 2, we more specifically consider differences in the instance-wise predictions _i, _i rather than differences in aggregate performance scores.
Toward that end, for all pairs of models f, f'∈ℱ, we both compute the disagreement

    Δ_Dis(, ) = 1N∑_i=1^N1{max_j _i,j≠max_j _i,j},

between their hard predictions, and the average Jensen-Shannon divergence (JSD)

    Δ_JSD(, ) 
        = 12N∑_i=1^N JSD(_i _i')

of the class-wise probability scores, and report the Spearman correlation of both measures with the corresponding set of representational similarities m(, ).




  
Grounding by Design.
For all the tests of this category, we do not consider functional outputs  of individual models as a ground-truth for representational similarity anymore.
Instead, we have created multiple groups of representations 𝒢, such as groups of representations from models that were trained with the same degree of label randomization as in Test 3, through which we impose a ground truth about similarity by relation.
In Test 3, 4 and 5, we postulate that for any similarity measure m it should hold that representations , ∈𝒢 from the same group should be more similar to each other than representations ∈𝒢, ^*∈𝒢^* from different groups, i.e.

    m(,^*) ≤ m(,),

where we assume that for m, higher values indicate more similarity.
Then, one trivial way to evaluate the performance of a similarity measure m is be to count the number of times inequality (<ref>) holds over all combinations of groups and representations, and then report the relative conformity rate over all comparisons. 
Given that this measure can, however, be biased by the ratio of inter-group vs. intra-group pairs, we additionally adapt the Area under the Precision-Recall curve (AUPRC) measure to our context.
This is done by assigning a label y(, ) := 1{𝒢 = 𝒢' }
to each pair of representations ∈𝒢, ∈𝒢' that can be compared within the given set of groups, and then interpreting the representational similarities m(, ) as decision scores based on which one should be able to "predict" the label y(, ).




Finally, in Test 6, we group representations ^(l) by the layers l they were extracted from.
However, in contrast to the previous tests, we postulate that an ordinal relationship between the layers has to hold.
Specifically, given a model with L layers, for  all tuples 1≤ i≤ j<k≤ l≤ L we require a measure m to satisfy

    m(^(i),^(l)) ≤ m(^(j),^(k)),

assuming that for m, higher values indicate more similarity.
As in the previous tests, we report the corresponding conformity rate. 
Further, we report the Spearman correlation between the similarities m(^(i),^(j)) and the distance j-i of the corresponding layers over all tuples 1≤ i < j ≤ L.












§ BENCHMARK RESULTS


In the following, we illustrate selected benchmark results. Due to limited space, we focus on presenting (a) an aggregated overview and (b) an exemplary detailed result for a single dataset and a single architecture.

For an aggregated result overview, we rank all measures for each combination of test, dataset, and architecture. The rank distribution across tests in each domain is shown in <Ref>.
We observe that no measure outperforms the others consistently across the domains, with most measures having high variance in their rankings.

However, one can identify some domain-specific trends, such as the neighborhood-based second-order cosine similarity (2nd-cos), Jaccard similarity, or rank similarity measures performing well in the graph domain. On the other hand, angle-based measures performing well on language models.
Further, the popular CKA measure appears to perform particularly well in the vision domain.
Orthogonal Procrustes, which has overall performed well in the analysis by <cit.>, also appears to consistently rank among the top third of measures. 
Conversely, other prominent measures such as RSA or SVCCA do not appear to stand out in general.



The  benchmark can also provide more detailed results for analysis.
To provide some examples, <Ref> presents the outcomes for a single dataset and architecture in each domain.
All values are denoted such that higher values correspond to better adherence to similarity groundings.

Again, we observe that even for a single dataset there is no single measure that outperforms the others in all tests.
Instead, most measures appear to perform well in some tasks but do not in others. 
For instance, measures such as the Eigenspace overlap score or GULP appear to be able to perfectly distinguish layers of a GraphSAGE model, but appear incapable to identify whether they have learned specific shortcuts, which most other measures appear to be capable of.
One can also identify model-specific trends, such as SVCCA correlating particularly well with predictions from BERT models, but not as much with other domains.
In addition, considering the performance of descriptive statistics-based measures such as uniformity difference or concentricity difference may provide additional geometric insights on representations. For instance, the uniformity of BERT representations appears to inform relatively well about the degree to which it was trained on augmented data.











§ RELATED WORK

Despite the large amount of representational similarity measures that have been proposed in the literature <cit.>, only few efforts have focused on comparatively analyzing existing measures. 
In the following, we briefly provide an overview of existing work.


Grounding by Prediction.
<cit.> first proposed grounding similarity of representations to performance differences of probes trained on them.
They benchmarked three similarity measures with tests focusing on identifying effects of seed variation or manually manipulating the representations.
Compared to , they focus only on a single test in our framework.
Their work was extended by <cit.> with a focus on measuring similarity of high-dimensional representations in CNNs.
Similar to Test 2 in , <cit.> tested the ability of their similarity measure to capture prediction disagreement.


Grounding by Design.
Similar to Tests 3-5 in the  benchmark, a number of previous works have also tested whether similarity measures could distinguish representations based on differences in their underlying models.
<cit.> considered representations of models from different architectures, and required similarity measures to distinguish representations by model family.
<cit.> tested whether measures could distinguish models based on differences in the data they were trained on.
In a setting similar to Test 6, <cit.> grouped representations of models that only differ in their training seed by layer depth, and tested whether measures assign more similarity to representations from corresponding layers.

Finally, a number of analyses have tested whether similarity measures can detect model-independent changes in representations.
Specifically, it has been explored whether measures could distinguish representations based on different synthetic clusters <cit.>, based on different levels of noise <cit.>, or based on subsampled dimensions <cit.>. 



§ DISCUSSION AND CONCLUSIONS



In this paper, we have presented , the first comprehensive benchmark for representational similarity measures. By applying its six tests on a wide range of 23 similarity measures to evaluate their capabilities on graph, language, and vision representations, we have demonstrated that it can yield rich insights both for researchers who may be looking for theoretical understanding of representational similarity and for practitioners that are looking for an appropriate measure for a problem at hand.
We provide all code publicly online, and invite the machine learning community to use it to test (potentially new) measures, or to extend it to include even more models or even additional tests.


We have put considerable effort into properly grounding the benchmark tests. 
Specifically, when grounding by design, we have not only carefully set up distinct training conditions, but also verified from the validation performance of the models under consideration that these indeed follow the intended behavior and adapted the generation process if necessary.
Still, we cannot fully control what models learn. Even if two models behave identically, their internal processes, and thereby their representations, could still differ, potentially confounding the resulting test scores.
When grounding by predictions, we have deliberately restricted the tests to models that only vary in their training seed, to reduce the potential for additional confounders as much as possible.

Furthermore, in our tests we solely focus on last-layer representations.
Checking the validity of the existing tests at other layers could be done by evaluating whether probes follow the same behavioral patterns as the full model.

While  implements 23 similarity measures, we do not claim to include all existing measures. Additionally, we did not explore the effects of different preprocessing approaches or parameter choices of individual measures. Given that such factors can lead to different results <cit.>,  opens up the way for a systematic evaluation of such steps in future work.
Similarly, the six tests implemented in  should be considered a comprehensive but extensible foundation of tests that users can build upon.
We see a lot of potential in augmenting  with additional tests, aiming for a unifying test suite that can inform the whole community of researchers and practitioners.
Finally, we have observed vast differences in the run time of the implemented similarity measures, which can limit large scale comparisons.
Thus, one could additionally consider computational performance of similarity measures as an interesting criterion for future evaluations of similarity measures.



§ ACKNOWLEDGEMENTS


This work is supported by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation under Grant No.: 196649072.

The authors acknowledge support by the state of Baden-Württemberg through bwHPC
and DFG through grant INST 35/1597-1 FUGG.

abbrvnat










§ EXPERIMENT DETAILS


Complementing the higher level  experiment descriptions in the main manuscript, a more detailed description of them is provided in this section. This includes general information about chosen model architectures, datasets and hyperparameters for each domain, with additional details that are unique to each experiments provided later. 






 §.§ General Experiment Details


Across all modalities and experiments, efforts were made to keep architectures, training hyperparameters or dataset choices as static as possible, while minimizing overall compute effort needed to compare representations with all these metrics. The settings unique to each domain are reported in their dedicated  following sections, while shared choices are explained here:

For all tests except Test 6 (layer monotonicity), we only utilize the representations of the last inner layer, directly before the linear layers of the classifier for our comparisons.
We chose to do so as we can verify our assumptions of a model learning shortcuts or learning to be more robust to augmentations with the predictions directly derived from these representations.
In an earlier layer, for example, we would not be able to verify whether a model learned the shortcut or not.
For the output correlation, a similar argument can be made, as predictions are directly derived from these representations. 
Only for the layer monotonicity test, intermediate representations were used. 







  §.§.§ Graph Settings




  
Models and Parametrization. 
As graph neural network architectures, we chose the GCN <cit.>, GraphSAGE <cit.>, and GAT <cit.> models due to their widespread prominence.
In our implementation, we used the respective model classes as provided in the  <cit.> package.

For each experiment and dataset, we trained these models from scratch.
Regarding the hyperparameters, we roughly followed values that were used for these models in existing benchmarks. 
An overview of the hyperparameter choices for each algorithm and dataset are presented in <Ref>. 
The layer count also includes the classification layer, thus the number of inner layers is one less than the number given in the table.
For the GAT model, we always used h=8 attention heads.
We always used the Adam optimizer <cit.> as implemented in  <cit.> to optimize a cross-entropy loss on the given multiclass datasets.
To validate the performance, we plot the training, validation and test accuracies of the final models, in <Ref>.

All these parameter choices were consistently applied in all our benchmark tests, except for Test 6 (layer monotonicity), where we increased the number of layers to six, yielding five inner layers. 



  
Datasets. 

We focus on graph datasets that provide multiclass labels for node classification, and for which dataset splits into training, validation and test sets are already available. 
Thus, we selected 

    
  * the citation network Cora, for which nodes represent documents and edges represent citation links,
    
  * the  Flickr <cit.>  network, where nodes represent one images uploaded to Flickr, and edges are formed if two images share some common properties,
    
  * the OGBN-Arxiv citation network between Computer Science (CS) arXiv papers. Each node is an arXiv paper, and each directed edge indicates that one paper cites another one.


Specifics on the datasets can be found in Table <ref>.












  §.§.§ Language Settings. 



  
Models. 
We focus on BERT models <cit.> as <cit.> provide 25 reproductions of the original models with different seeds.
Using these models, we avoid training BERT models, which is not feasible for us.
Other models like LLMs would be interesting as well and can be explored in the future.



  
Representation Extraction. 
BERT produces one representation per token of each input.
As we focus on the last-layer representations, we use the representation of the CLS token as the representation for the whole input.
As this token is used by the final classifier, we argue that all relevant information for that input will be contained in this representation.

While it is possible to extract representations for all tokens, inputs with many tokens would take up more rows in the final representation matrix, which could bias similarity estimates towards similarity of long inputs.
Additionally, the runtime of many representational similarity measures scale quadratically in the number of rows of the representations, which limits the size of the representations given the many comparisons that we run in our benchmark.



  
Datasets. 
We use two classification datasets: SST2 <cit.> is a collection of sentences extracted from movie reviews labeled with positive or negative sentiment.
MNLI <cit.> is a dataset of premise-hypothesis pairs. 
They are labeled according to whether the hypothesis follows from the premise.
We use the validation and validation-matched subsets to extract representations for SST2 and MNLI, respectively.




  
Training Hyperparameters. 
As we use the pretrained models from <cit.>, we only finetune the models for our experiments.
While the number of epochs varies between 3 and 10 depending on the experiment, we always use the checkpoint with best validation performance.
We generally use a linear learning rate schedule with 10% warmup to a maximum of 5e-5, evaluate every 1000 steps, and use batch size of 64.
Otherwise, we use default hyperparameters of the transformers library[<https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/trainer#transformers.TrainingArguments>].







  §.§.§ Vision Settings



  
Architecture Choices. 
In the vision domain a plethora of architectures exist which could have been used in the scope of the benchmark.
To narrow the scope and to keep computational overhead manageable only architectures for classification were considered. Moreover a subset of architecture of prominent classification architecture families were evaluated, namely ResNets <cit.>, ViT's <cit.> and older VGG's <cit.>. 
To capture if architecture size influences the measures, different sizes for the architectures were evaluated, resulting in 

    
  * ResNet18,
    
  * ResNet34,
    
  * ResNet101,
    
  * VGG19,
    
  * ViT-B/32 and
    
  * ViT-L/32

as the final architecture choice. Pre-trained checkpoint for ViT-B/32 and ViT-L/32 use are taken from  provided under the Apache 2.0. Specific checkpoints are 'google/vit-base-patch32-224-in21k' and 'google/vit-large-patch32-224-in21k'.


  
Representation extraction. 
In order to measure representational similarity, representations need to be extracted at different positions in the architecture.
In order to capture the full system behavior of a model, representations are only extracted at positions where no skip-connections exist.
For VGG networks this would not narrow down choices due to lack of skip connections, yet for ResNets and ViT's this narrows the possibility of choices where representations can be extracted.




  
Representation Extraction. 
Across all experiments representations are extracted on unseen test images that models have not seen during training.
For ImageNet100 the 50 validation cases per class are used, resulting in N=5k samples. 



  
Training Hyperparameters. 
The  benchmark requires models that can be differentiated by their training setting.
Hence, it is mandatory to either train models from scratch or to fine-tune them accordingly.
For all models of the vision domain the choice was made to train the models from scratch for each respective dataset, with the exception of ViT's whose encoders were initialized from a pre-trained IN21k checkpoint. The chosen training hyperparameters are kept static across Tests, with different data being the only difference. A selection of hyperparameters is displayed in <Ref>. While the parameter choices worked well for ResNets and VGGs, the ViT performance was lower than the ResNets, despite attempts to optimize their training settings. We assume this may originate from the fewer overall steps taken due to the 10× lower dataset size relative to full ImageNet1k.





  §.§.§ Compute Resources


To conduct the experiments, a broad spectrum of hardware was used: For the model training GPU nodes with up to 80GB VRAM were employed. Depending on domain, representations were either extracted on GPU nodes and saved to disk for later processing, or extracted on demand on CPU nodes. Lastly, the representational similarity measures were calculated between representations on CPU nodes with 6-256 CPU cores and working memory between 80 and 1024 GB.

To extend the current tests with additional similarity measures, practitioners need to have CPU compute resources to compute their own similarity measures on the existing model's representations. 
To introduce novel tests, practitioners would need to provide GPU compute resources for network training and subsequent CPU resources to evaluate all similarity measures on the new models.



 §.§ Test 3: Label Randomization - Details











  
Graphs. 
We trained groups of five models each, for which r∈{25%, 50%, 75%, 100%} of all labels are randomized during training.
The models in each group were initialized with different training seeds, and different seeds also imply different randomization of labels.
Given that we exclusively consider predictions on datasets with L≥ 7 classes, for all labels that were drawn to be randomized, we randomly drew a new label from the set of all existing labels, excluding the actual ground-truth label, with uniform probability.
After observing the performance of the models with randomized labels on validation and test data, we decided to only include models for which r∈{25%, 100%} were altered during training for this test, along with five models trained on the regular data.
THis is mainly due to the models already having shown behavior indicating that for higher degrees of randomization than 25%, there was hardly any relationship left between features and true labels that the models would have learned.
We illustrate the performance of the models trained under varying degrees of labels in <Ref>, where we can see that the models are strongly affected by such randomization.






  
Language. 
We create datasets where r∈{25%, 50%, 75%, 100%} of all labels are randomized so they can only be learned by memorization.
If an instance should get a randomized label, we pick uniformly at random from a new set of 5 labels that are entirely unrelated to the data.
After validating behavior, we use models without randomization as well 75% and 100%.



  
Vision. 
We create datasets where r∈{25%, 50%, 75%, 100%} of all labels in the training set are randomized, forcing models to memorize. Whenever a label is chosen to be randomized, we randomly sample from all 100 existing labels, and replace the true label with the sampled one. Moreover we re-use the first 5 models from Test 1 and 2 representing a datasets with r=0.
After validating behavior, we use models with 0%, 50%, 100%, with clusters visualized in <ref>.





 §.§ Test 4: Shortcut Affinity - Details





  
Graphs. 
For all graph datasets, shortcuts are added by appending a one-hot encoding of the corresponding labels to the node features.
Nodes which are not assigned a true level shortcut obtain a one-hot encoding of a random label instead.
We trained models with a ratio of ρ∈{0%, 25%, 50%, 75%, 100%} true shortcuts in the training data, but only considered the models trained on with ρ∈{0%, 50%, 100%} shortcuts in the training data, as otherwise there would not have been a very clear separation of the models.
We illustrate the performance of models influenced by the remaining degrees of shortcut likelihood in the training data in <Ref>, where we can see that most models are indeed affected. In particular, GraphSAGE appears very sensitive to shortcuts, whereas other methods appear more robust - this may also be due to differences in the way node information is updated in the convolutional layers.

Overall, on Cora the effect was the smallest, which is likely due to the models fitting easily to the training labels even without a shortcut.

For all groups and seeds, we extracted representations based on the same fixed dataset, which contained one-hot encodings of random labels.











  
Language. 
We add shortcuts to the data by prepending a new special token to each sentence.
The embedding matrix of BERT is extended with one token per class and trained during finetuning.
We create datasets with 5 levels of shortcut strength, i.e., the rate at which the shortcut leaks the correct label.
As completely incorrect shortcut tokens would perfectly leak the label in binary problems like SST2, the lowest rate is the relative frequency of the majority class.
Then we make 4 equally sized steps up to perfect shortcuts.
This means that SST2 datasets have shortcut rates r∈{55.8%,66.8%,77.9%,88.9%,100%} and for MNLI r∈{35.4%,67.7%,51.55%,83.85%,100%}.
We train models with 5 different seeds for each shortcut dataset, but after validating sufficiently different behavior (see <Ref>) only use the models with the lowest and the two highest rates.



  
Vision. 
In the vision domain, shortcuts are created through insertion of colored dot.
Each dot has a static color, a diameter of 5 and is placed randomly in the image. To turn this into a shortcut, each class is assigned a unique color code, which is uniformly drawn once and shared across all experiments. By varying how often the pre-determined color appears on an image containing the corresponding class label, one can control how useful the color dot feature is to the class in the image. A few exemplary images of the shortcuts are shown in <ref>.


In our experiments, we train models under 5 degrees of correlation 
ρ∈{0%, 25%, 50%, 75%, 100%}, providing a more or less potent shortcut. To validate that these different ratios lead to sufficiently different behavior that our grouping assumption holds, we measure the test set accuracy of all models with r = 0%. Models that learned to utilize the shortcut will show a stronger decrease in accuracy as the shortcut does not provide any meaningful signal anymore.
In order to provide clearly distinguishable clusters, we discarded models trained with r ∈{25%, 50%}.
The accuracy clusters after removing the two groups are shown in <ref>.









 §.§ Test 5: Augmentation - Details










  
Graphs. 
To augment the graph data during training, we applied the DropEdge approach <cit.>, which, at each training epoch, samples a fixed fraction of edges that are removed from the network for this specific epoch.
Based on this approach, we trained four groups of augmented models, for which r ∈{20%, 40%, 60%, 80%} of all edges were removed at each epoch.
Again, after considering the performance of the models on validation and test data, we chose to only include the models with 
 r ∈{20%, 80%}  for this test along with a group of models trained with standard parameters, which was due to the overall weak effect impact that the intermediate steps appeared to have.
The performance of these remaining models is shown in <Ref>, where we overall observe weak, but visible effect from augmentation.

 



  
Language. 
BERT is trained on large-scale data of varying quality and, as preliminary experiments showed, highly robust to augmentation like word casing, synonyms, and typos[We tested most of the augmentations from <https://langtest.org/docs/pages/tests/robustness>.].
Therefore, we chose more aggressive augmentation that can also scale to large amounts of data.
We randomly replace words with synonyms, delete and swap words, and randomly insert a synonym of a word in the sentence in a random position, as described by <cit.> and implemented in the textattack library[<https://textattack.readthedocs.io/>] <cit.>.
To create datasets with different level of augmentation, we augment r ∈{0%, 25%, 50%, 75%, 100%} of all words per sentence.
We train 5 models with different seeds per dataset.
For SST2, we use the 0% and 100% models; for MNLI, we use 0%, 25%, and 100%.



  
Vision. 
For the augmentation test we utilize increasing levels of additive gaussian noise with 5 different noise variance limits var ∈{(0,0), (0, 1000), (0, 2000), (0, 3000), (0, 4000) }, we refer to these levels as Off, S, M, L and Max. Exemplary images of the augmented images are displayed in <ref>. Aside from the noise, only RandomResizeCrop and HorizontalFlip augmentations are used.
After validation model behavior we use Max, S and M, see <ref> for the resulting clusters. 







 §.§ Test 6: Layer Monotonicity - Details






  
Graphs. 
Since for the graph neural network models, we never used more than three layers in the previous experiments, for this test we trained a new set of five graph neural network models, each with five inner layers.
That way, we can obtain a sufficient number of comparisons, while still preserving decent performance—with higher number of layers, the performance began to strongly deteriorate. 
As all inner layer shared the same number of channels, ne preprocessing was necessary to compare the layer-wise representations.



  
NLP. 
We use ten models trained on standard data with varying seeds for each dataset.
We use the CLS token representations across all 12 transformer layers and the embedding layer.





  
Vision
For the layer monotonicity test, we use the 10 models from Test 1 and 2. Differently to those tests we extract representations not only before the classifier, but we also need to extract representations in intermediate layers.

For CNN architecture representations will need to be compared that may be of different spatial extent. 
This is problematic for some metrics that use the spatial extent to increase their sample size N, as they assume features at the same spatial position correspond to each other. Moreover it is problematic, as for width and height (w,h) of (112,112) this increases the total number of values by >10.000. To avoid memory issues, we only extract representations from layers of the architecture with global stride ≥8 resulting in (w,h) (28, 28) and overall we limit the number of layers to the last 6.
The representations of these layers are then downscaled to a final 7×7 through AveragePooling, resulting in the same spatial extent for all representations, with the exception of the classifier layer without spatial extent.
When comparing representations of spatial extent, we rearrange w and h into the channel dimension c – ('n c h w' -> 'n (c h w)') – unless the metric either, requires n > p or, if computation time of the measure scales prohibitively. For these measures we move it into the sample dimension – ('n c h w' -> 'n (c h w)'), but reduce the sample dimension (n h w) -> (n) by only using every 49th value due to memory and time constraints.

For comparisons between non-spatial and spatial representations, we employ average pooling, removing any spatial extent.

For ViT's we follow the setup of the language domain and only use the classifier token, offering a simple and compute efficient solution.

We denote, that this resampling and sub-sampling of the representations in the CNN case is suboptimal, yet this issue originates from the limitations of the similarity measure, be it needing n > p or scaling prohibitively. 




§ FULL BENCHMARK RESULTS

In the following, we provide the full results of all experiments within our benchmark.



 §.§ Graph Results

We begin with presenting the results from the graph domain.





























 §.§ Language Results


All results are shown in <Ref>.








 §.§ Vision Results

We present all tables containing the results of the vision domain. Results are presented in test order, with each test featuring one table. 
Some entries in the result tables feature  values. This was caused by various reasons: (i) Numerical instability due to operations like Singular Value Decompositions or negative values in square roots and (ii) identical similarity values, leading to failure of correlation values. Whenever such failure occured for an entire group of models the entire measure was excluded for this case, as e.g. removing the model group of gauss noise M may simplify the overall task of distinguishing models.  
















 Paper Title: IN-Sight: Interactive Navigation through Sight.



[
    R. Sasportes^†
    August 5, 2024
==================

empty
empty








§ ABSTRACT 





Current visual navigation systems often treat the environment as static, lacking the ability to adaptively interact with obstacles. This limitation leads to navigation failure when encountering unavoidable obstructions. In response, we introduce IN-Sight, a novel approach to self-supervised path planning, enabling more effective navigation strategies through interaction with obstacles. Utilizing RGB-D observations, IN-Sight calculates traversability scores and incorporates them into a semantic map, facilitating long-range path planning in complex, maze-like environments. To precisely navigate around obstacles, IN-Sight employs a local planner, trained imperatively on a differentiable costmap using representation learning techniques. The entire framework undergoes end-to-end training within the state-of-the-art photorealistic Intel SPEAR Simulator. We validate the effectiveness of IN-Sight through extensive benchmarking in a variety of simulated scenarios and ablation studies. Moreover, we demonstrate the system's real-world applicability with zero-shot sim-to-real transfer, deploying our planner on the legged robot platform ANYmal, showcasing its practical potential for interactive navigation in real environments. 

Supplementary Video: <https://youtu.be/ja0Vjm72ZDw>






§ INTRODUCTION


In robotics, navigating real-world environments from vision sensors has been a long studied subject <cit.>. Yet, most traditional navigation approaches treat the environment as a passive entity, not capturing the dynamic nature of the real world where obstacles can be moved deliberately. Visual Interactive Navigation (VIN) takes this into account by engaging with the surrounding, actively moving obstacles out of the way to enable more effective traversal strategies (cf. Figure <ref>).



Engaging with the environment introduces several challenges. Firstly, an agent must discern whether an obstacle is movable and whether moving it is beneficial for reaching the goal. This involves predicting how obstacles will respond to interaction, a task complicated by the intricate nature of such dynamics. As a result, the field gravitates towards learning-based approaches. Reinforcement Learning (RL) has shown promise in enabling navigation in interactive indoor environments using vision sensors, but it can suffer from poor sample efficiency, especially with high-resolution visual inputs. Therefore, many studies resort to fast non-photorealistic physics simulators <cit.><cit.><cit.> or vector observations rather than raw visual inputs <cit.>. While successful in simulation, these approaches often fall short in real-world application.

Real-world environments, characterized by maze-like structures such as buildings or public spaces, pose a significant challenge to agents that rely on purely reactive policies without utilizing past observations, leading them to easily get lost. <cit.> attempts to mitigate this by integrating a transformer-based memory into the RL policy, however causing stability complications during training and limited sim2real transfer. In contrast, the supervised learning approach <cit.> incorporates memory using explicit mapping. The authors fine-tune a pretrained object pose estimation module to localize paper boxes, combining it with explicit path planning on a pre-computed map. This approach alleviates the memory problem and avoids training on non-photorealistic data, improving applicability to real-world environments. However, this work does neither address the complexities related to reasoning whether an obstacle can be moved, nor how generalization to unseen obstacles with similar properties can be achieved. Relying on a trial-and-error approach, the robot has to actively push the obstacle to determine whether its movable, resulting in many undesired interactions. Given that the visual appearance of real-world environments varies drastically and physical sensors are affected by sensing artifacts, deploying VIN systems to robotic hardware remains a major complication.

In this work we introduce a hierarchical planner, continuously integrating traversability estimates into a global map, which is then used to perform high-level path planning. From the high-level path plan we extract subgoals, which are then tracked by a local planner. This allows us to navigate cluttered and maze-like environments at scale. Contrary to previous RL-based approaches, we train our system in a self-supervised fashion, achieving fast convergence to a robust policy. We generate a diverse photorealistic training set consisting of sequential data using the Intel SPEAR simulator. We further employ various robustification techniques in order to mitigate the effects of sensing artifacts and imperfect photorealism.























Our main contributions are as follows:



    
  * We propose a new training paradigm by extending the established imperative approach <cit.> to accommodate interactive environments, leveraging sequential observations to support interactive navigation.
    
  * We introduce a scalable approach to generate self-supervision for path prediction and traversability estimation, based on different heuristics such as obstacle mass, friction, and terrain inclination.
    
  * We employ a blend of robustification strategies and accelerate the ICL-NIUM depth noise algorithm <cit.> by two orders of magnitude, allowing to simulate truly random depth sensor artifacts without slowing down training.
    
  * We demonstrate successful zero-shot transfer of the learned policy to the ANYmal legged robot.




§ RELATED WORK

Visual navigation is a widely explored research field with a substantial amount of influential contributions over the past two decades. A variety of relevant concepts, developed to address the different challenges of visual navigation, can be directly applied to visual interactive navigation. 
As mentioned previously, memory allows the agent to utilize past observations in decision processes. Not only limited to classical planning pipelines, memory also becomes an essential building block for successful learning-based architectures <cit.>. Often encountered in hierarchical planners, mapping allows to leverage explicit path planning methods such as A-star, Dijkstra's algorithm, or fast marching methods <cit.>. Beyond simple occupancy maps used in classical pipelines, learning-based approaches enable to encode more complex characteristics such as goal proximity <cit.>, semantic classes <cit.>, or goal reachability costs <cit.>. Mostly these systems encode heuristics which align sub-optimally with the local planners objective, as they are optimized using different labels. The labels we use to train our global mapping module are generated on the fly from the same cost-map the local planner is optimized on. This way we achieve full consistency between global- and local path planning. For the mapping, we rely on traversability estimation, determining which parts of the visible terrain can be safely traversed. Geometry-based methods such as <cit.> using Lidar or depth cameras suffer from poor terrain estimates as deformable instances such as grass, shrubs, or snow cannot be reliably classified. Semantic-based methods such as <cit.> successfully circumvent this problem by incorporating RGB information, but often suffer from a lack of robustness against appearance changes in real-world environments. In <cit.>, the authors utilize a visual transformer as a strong feature extractor and train a small network online for traversability adaption. We adopt this idea and fuse our representation learner with RGB information encoded by the pretrained DinoViT-V2 <cit.>. 

Various techniques in representation learning aim to extract the most relevant features from the input data. <cit.> explores this challenge by using a Variational Auto-Encoder (VAE) <cit.> in conjunction with an RNN to encourage the learning of a Latent Dynamics Model (LDM), showing that the RNN is able to learn a compact representation of the world that it can propagate forward in time. Similarly, <cit.> applies this concept for dimensionality reduction, pre-training a world model to then train a policy network on the outputs of the world model using RL. Both works however rely on complicated training schedules with the different modules trained disjointly, leading to sub-optimal representations potentially containing information not relevant for the downstream task. We address this shortcoming by training our entire representation learner end-to-end. Using variational inference, we learn regularized representations of the environment, implicitly encoding information about the dynamic obstacles such as shape, position, or velocity.




§ PLANNER ARCHITECTURE






 §.§ System Overview

The global planner utilizes an online-generated global map to search for a potentially feasible path, which is then tracked by the local planner. The workflow of our planning algorithm is illustrated in Figure <ref>. Specifically, the perception module encodes the RGB observation R_k and the depth observation D_k at timestep k into a compact representation z_k and predicts a pixel-wise traversability mask T_k. The traversability mask is then fed to the mapping module where it is integrated into a 2D grid map. The grid map is used to compute a minimum-cost path to the goal by executing A* path search. From the discrete path, a subgoal is selected which is then fed to the local planner together with the embedding z_k. The local planner then predicts a path to the subgoal, consisting of N intermediate waypoints. The idea is that the global planner uses explicit memory (grid-map) to capture the macroscopic structure of the environment and is therefore well suited to provide a long-term planning strategy. However, the semantic map has limited spatial resolution and contains temporary artifacts (decaying over time) in case of non-stationary obstacles. Therefore, we introduce a local planner that takes the responsibility of locally navigating around obstacles and deciding if an interaction is desirable. The local planner comprises a Recurrent Neural Network (RNN) which allows the planner to grasp short-term context from previous time steps. Combining both concepts enables the system to efficiently explore large environments, while locally considering interactions with obstacles to reduce the traversal effort. 




 §.§ Long-horizon planning

The traversability mask T encodes for each pixel whether it belongs to free space, a movable obstacle or non-traversable space. In order to integrate the mask into the map, we first compute the point cloud from the depth image using the pinhole projection model, then transform the points onto the world frame. We then project the points on the 2D plane and compute the respective grid cell index. Finally, we sample for each active grid cell the number of previous observations m from the weight map to compute the update weight α (cf. Eq. <ref>) and then update the grid map (cf. Eq. <ref>) in a weighted average fashion

    α   =    2 / (1 + m) , 
    
        c    =   α· c + (1 - α) ·c,

where c denotes the observed traversability class of the point and c denotes the average traversability class of the grid cell. The number of observations is clipped between [0,100] such that the influence of new measurements does not eventually converge to zero. This leads to sufficiently fast decay of map-artifacts due to moving obstacles. For each updated grid cell the associated cell in the weight map is incremented. 

For global path planning, we resort to the A* algorithm with L2 heuristic. Due to the weighted averaging of the measurements during the mapping, our map no longer contains discrete area classes (0:free-space, 1:obstacle, 2:non-traversable, 3:unexplored) but a continuous belief value over the area classes. To obtain the discrete area classes back, we threshold the continuous values with the following ranges: [0, 0.5)→ free space, [0.5, 1.5)→ obstacles, [1.5, 3.0)→ non-traversable, [3]→ unexplored. We assign different penalties for planning through grid cells belonging to movable obstacles and non-traversable space. After the global path is found, we extract the element from the path in direct line of sight with the robot
we determine from the global path a set of points, in which each two consecutive points can be connected by a straight line without touching neither obstacle space nor non-traversable space. From this set, we use the closest point as the next subgoal g_k for the local planner.



 §.§ Short-horizon planning





Figure <ref> illustrates our planner on an implementation level. The components can be categorized into perception (left) and planning (right).  We aim to learn a compact representation of the world from a sequence of visual observations, encoding relevant information about the environment such as position, velocity, and semantic information of surrounding obstacles. For this we let the perception network model the depth inputs D by a small set of latent factors z, using a Latent Variable Model (LVM) of the form

    log p_θ(D)    = log∫ p_θ(D|z)p_θ(z)dz
       = [t]
            𝔼_z∼ q_ϕ(z)[logp_θ(D|z)p_θ(z)/q_ϕ(z)]    
    
            + 𝕂𝕃(q_ϕ(z)||p_θ(z|x))_≥ 0.

Reformulating the LVM using parameterized families of distributions we can resort to parameter optimization instead of variational calculus. Following <cit.> we choose the following parametrization

    p_θ(z)    →𝒩(0,I) , 
        
        p_θ(D|z)    = 𝒩(D| f_ψ(z), I) = p_ψ(D|z) , 
        
        q_ϕ(z)    = 𝒩(z|ϕ=(μ, Σ))  with ϕ = g_λ([D,R]]),


where the parameters θ and ϕ are computed using armortized inference. For this the functions g_λ and f_ψ are learned from a dataset consisting of synthetic image sequences mixed with real-world images. To inject semantic information into the LVM, RGB information R_k is used as an additional input for estimating the latent parameters ϕ. We encode the RGB images using the DinoViT V2 <cit.> with frozen parameters, serving as a robust feature extractor. The depth input D_k is encoded using a residual CNN. The inputs are then concatenated and fed through another residual CNN to estimate the latent parameters ϕ_k. Both μ and Σ are of shape [23,40,8], thus we model the latent space with 7360 latent variables. We sample a latent tensor z_k∼ q_ϕ_k(z) and decode it using the shared-parameter transposed CNN f_ψ to get a reconstruction D̂_k of the depth input D_k. Tracing the information flow from D_k to D̂_k it becomes visible that the parameterized models involved in the computation form a Variational Auto-Encoder (VAE).
We also decode the mean tensor μ_k of the latent distribution to get the traversability mask T_k, using a transposed CNN. The traversability mask is used for the mapping as described earlier. The parameters of the latent distribution ϕ_k = [μ, Σ]_k are passed to the convolutional GRU which additionally takes the subgoal G_k as an input. The GRU propagates its hidden state h_k→ h_k+1. The mean of the latent μ_k, the subgoal G_k, and the old hidden-state h_k are fed to the Path-CNN which predicts N waypoints wp_k, forming the local path to the subgoal. The new hidden-state h_k+1 is fed through the CNN g_η(h_k+1) to predict the parameters χ of the distribution over the next latent state q_χ(z_k+1). We again sample a latent vector z_k+1∼ q_χ(z_k+1) and pass it through the shared-parameter transposed CNN f_ψ. Tracing the information flow from D_k to D̂_k+1 it becomes visible that parameterized models involved in the computation form a predictive VAE.
Intuitively the GRU in the planner maintains a (world) belief state h and updates it based on the current (world) observation ϕ_k, resulting in the updated prediction of the belief state h_k+1. We then use this prediction to infer the distribution over the next observation z_k+1. With the structural bias of the RNN, our system is encouraged to learn the dynamics of the latent variables. As commonly done in LVMs, optimizing the Evidence Lower-Bound (ELBO, cf. Equation <ref>) introduces latent-regularization by minimization of the KL Divergence. Using an uncorrelated Unit-Normal prior encourages more disentangled representations eventually promoting decoupling in the learned system dynamics <cit.><cit.>. With this in mind, our side-objective can be understood as optimizing an LDM, which empirically has been shown to improve the expressiveness and robustness of the learned representations <cit.>.
As a second side-objective, we use the tensors h_k and h_k+1 to predict the change in pose (position, heading) between the current and next time step, P_k. 



§ TRAINING




Our training set consists of sequential data, synthetically generated using Intel SPEAR. During data collection, the agent has access to the navigation mesh <cit.> encoding all traversal- and obstacle-interaction costs in the scene. This privileged information allows the agent to traverse along the minimum-cost path to the goal. Once the goal is reached, the environment resets. To demonstrate domain randomization, the positions and orientations of the agent and all obstacles are randomized on reset.
To account for sensor imperfections, we add white noise to the RGB images and simulate common depth sensor artifacts <cit.> (cf. Figure <ref>). We accelerate the sampling procedure of <cit.> using GPU vectorization (x500 speedup on i7-8750H w. RTX2070 mobile) allowing us to sample unique artifacts on a per-sample basis. This way, all noise artifacts are truly random between epochs. We also inject independent real-world images of multiple depth cameras from the SUN-RGBD dataset to robustify against unmodeled sensor intrinsics.


The training loss follows Figure <ref>

    ℒ   =    c_nav· (c_path·ℒ_path + ℒ_odom) + 
           c_trav·ℒ_trav + c_repr·ℒ_repr

with

    ℒ_repr = ℒ_reco,1 + ℒ_KLD,1 + ℒ_reco,2 + ℒ_KLD,2.

where the coefficients c are hand-tuned parameters. The path loss follows

    ℒ_PATH = α·ℒ_o + (1-α) · (ℒ_m + ℒ_g),


where ℒ_o penalizes proximity towards obstacles, ℒ_m penalizes irregular distances between consecutive waypoints and ℒ_g penalizes a high L2 distance between last waypoint and goal. ℒ_o is computed by projecting the waypoints onto the 2D costmap and sampling the cost values <cit.>. The differentiable cost map (cf. Figure <ref>, Training) is derived from the navigation mesh, which we modify to encode obstacle locations and semantic classes (free space, light-weight / medium-weight / heavy-weight obstacle, non-traversable space) during simulation. The cost value of a grid-cell depends on the distance to the next obstacle and the obstacles semantic class. Using the tuning value α we manually balance the preference of the agent to be more goal-directed or more obstacle avoidant. 
ℒ_trav is computed using the Multi-class Cross-Entropy. We compute the traversability labels by unprojecting the true depth image into a point-cloud, transforming it into world frame, and then projecting it onto the 2D grid map. We can then sample the cost values from the costmap and associate each cost value to each pixel from the depth image, such that we obtain the 2D ego-perspective traversability mask. 






For the auxiliary loss ℒ_odom we take sum of the L2 norm of the translational error and the
L2 norm of the rotational error between the prediction of the change in pose and the actual change in pose.
We train the system on an NVIDIA RTX3090 for 10 epochs using PyTorch's AdamW optimizer with a constant learning rate of 1e-4 and weight decay of 1e-8. We also apply gradient scaling and train in half-precision (16 bit). We resort to a sequence length of 5 time instances where each time instance has a batch of size 10. Therefore, we use 50 samples to compute the gradients. The DinoViT parameters are frozen. The dataset is shuffled after each epoch.



§ EXPERIMENTS











We evaluate our planner across three distinct simulated environments: The maze-like House 1, featuring manipulable obstacles that either appear in the dataset or are entirely new (cf. <ref>); House 2, populated exclusively with new obstacles that have textures different from those in the training set; and the cluttered Forest, which contains no movable obstacles.



 §.§ Simulation Benchmarks


We conduct each experiment over 100 epochs. At the start of each epoch, the agent is respawned, and we sample five random start/goal pairs, choosing the one with the greatest integral path length. This method eliminates overly simplistic start/goal pairs that could be solved with a direct path. An epoch is considered successful if the agent reaches its goal within 500 iterations (equivalent to 50 seconds at a rate of 10Hz). In our simulations, the agent navigates straight between local waypoints, executing in-place rotations upon reaching a target waypoint. An episode is deemed successful if the agent comes within 1 meter of the goal. Every 10 epochs, we reposition all non-stationary obstacles to a new location within a small area around their original, manually selected positions. For benchmarking, we employ the following metrics: the Success Rate (SR) (Equation <ref>) and Success Weighted by Path Length (SPL) (Equation <ref>)

    SR   =   num_success/num_success + num_failure
    SPL   =   1/N∑_i=0^N-1 S_i L_i/max(L_i, L_i).


where L_i is the length of the traversed path and L_i is the length of the least-cost path computed with respect to the navigation mesh (only available in simulation, cf. <ref>). We also capture the number of collisions with static elements η_static and movable obstacles η_obstacle.
The performance of the proposed planner, along with its ablated versions, is indicated in Table <ref>. We observe that the agent manages to reach the goal in time with consistently high SR and SPL across all three test environments. We further observe that the agent successfully inherits the navigation preferences encoded into the costmap, as a result of optimizing the policy with respect to the ℒ_o loss (cf <ref>).
An example of interactive navigation is provided in Figure <ref>, demonstrating the planner's ability to manage obstacles blocking the shortest path to the goal. In this scenario, the agent encounters a wooden crate on its direct route. It correctly identifies the crate as movable and determines that displacing the crate to proceed along the obstructed pathway is more resource-efficient than detouring through an alternative room. It then tips the crate over and proceeds with the traversal. This behavior demonstrates the planner's ability to intelligently negotiate obstacles, enhancing navigation strategies and, crucially, averting navigation failures by addressing obstructions directly when no unobstructed alternative paths are available.



 §.§ Ablation Study

We ablate the planner to examine the impact of each design choice on the final planning performance. We also measure the performance of a simplistic baseline configuration inspired by <cit.>, where we take the encoder architecture from the proposed planner but instead predict the waypoints directly from the embeddings Φ_k using a convolutional residual network. All ablated configurations are trained under the same conditions as the proposed planner, as outlined in Section <ref>.


The Deterministic configuration has the stochastic blocks removed (blue blocks, Figure <ref>) and is not optimized for the KL-Divergence. While having comparable SR and SPL to the proposed planner in the House 1 environment, it shows inferior performance on House 2 and Forest. We also observe that the number of collisions η_static increases for the scenes comprising a high obstacle diversity. This confirms the hypothesis that the latent regularization improves generalization by allowing for better interpolation in the latent space (cf. Section <ref>).

In the noRNN configuration the local planner lacks memory. Thus, the network is not able to maintain a temporally consistent representation of the nearby obstacles. If an obstruction drops out of sight when circumventing, the agent turns prematurely towards the obstacle again. This reduces the path margin and increases the risk of a collision. Consequently, η_static increases and the agent times out more often (reduced SR). 

In the onlyVAE configuration, having both the RNN and predictive VAE branch removed, the SR significantly deteriorates in both the Forest and House 2 environments, together with an increase in η_static in all test environments. Similar observations can be made when removing the odometry network (cf. noAUX). This confirms the hypothesis that optimizing an LDM or odometry estimator as a co-objective benefits the learned representations and leads to improved generalization (cf. Section <ref>). 

The minimal Baseline configuration performs equally good to the proposed planner in the less complex, non-interactive Forest environment. However, in the interactive, labyrinth-like environments the performance drops significantly, underlining the efficacy of our architectural design choices.







 §.§ Real-World Experiments






We test our planner at 5 Hz on the quadrupedal ANYbotics ANYmal D robot equipped with a NVidia Jetson AGX PC and an Intel Realsense D435 camera. The robot autonomously navigates from a start point to a predetermined goal without human intervention. We intentionally obstruct the direct route with unfamiliar obstacles not encountered during training (cf. Figure <ref> [A,E]), including both static (tables, heavy chairs, walls) and movable objects (paper boxes, lightweight office chairs, foam bricks). Our observations reveal that the planner effectively distinguishes between traversable and non-traversable areas (cf. Figure <ref> [B]) and consistently generates paths within the navigable space (cf. Figure <ref> [D]). It identifies movable obstacles (cf. Figure <ref> [A1,B1]), pushing them aside when bypassing is not feasible (cf. Figure <ref> [A2:3,B2:3]). Using the depth reconstructions (cf. Figure <ref> [C]) we can confirm that the obstacle shapes are successfully captured by the latent. From the local path visualization (cf. Figure <ref> [D]) we can observe that the local path correctly bends around static obstacles, confirming that the learned planner correctly reasons about obstacles. Moreover, the stability of local path predictions over time, even when obstacles are no longer visible, underscores the planner's temporal consistency.



§ CONCLUSION AND FUTURE WORK

We presented IN-Sight: a visual path planner for interactive navigation in cluttered and maze-like environments, demonstrating successful transfer to robotic hardware. We introduced a self-supervised training approach, leveraging the navigation mesh to auto-generate consistent path- and traversability supervision. This significantly reduces the manual labor required to create large photorealistic datasets suitable for planner and traversability estimator training, opening new opportunities for interactive navigation research. For future work, we aim to enhance our current setup in terms of sim-to-real transfer. By using low Level of Detail (LOD) collision meshes for the obstacles, we can generate depth reconstruction labels D̂_k with low LOD. These labels are expected to facilitate the learning of more basic structural representations, thereby improving the system's ability to generalize. Different from that, we seek to explore the application of Large Vision-Language Models for reasoning about the movability of obstacles. Utilization of the extensive amount of data encapsulated within these foundation models combined with their ability to reason, could potentially allow for the consideration of minute details in the environment, leading to more accurate predictions of whether an obstacle can be displaced.



-12cm  
                                  
                                  
                                  
                                  
                                  













§ ACKNOWLEDGMENT

This work was supported by Intel Labs Munich, in a collaboration between the Robotic Systems Lab at ETH Zurich, and the Smart Robotics Lab at TUM. Moreover, this work has been conducted as part of ANYmal Research, a community to advance legged robotics.



bibliography/IEEEtranN

tocchapterBibliography



 Paper Title: Graph Representation Learning via Causal Diffusion for Out-of-Distribution Recommendation.






Graph Representation Learning via Causal Diffusion for Out-of-Distribution Recommendation





    Chu Zhao
Northeastern University 

Shenyang, China 

chuzhao@stumail.neu.edu.cn
Enneng Yang
Northeastern University 

Shenyang, China 

ennengyang@stumail.neu.edu.cn
Yuliang Liang
Northeastern University 

Shenyang, China 

liangyuliang@stumail.neu.edu.cn
Pengxiang Lan
Northeastern University 

Shenyang, China 

2390222@stu.neu.edu.cn
Yuting Liu
Northeastern University 

Shenyang, China 

yutingliu@stumail.neu.edu.cn
Jianzhe Zhao
Northeastern University 

Shenyang, China 

zhaojz@swc.neu.edu.cn
Guibing Guo 
Northeastern University 

Shenyang, China 

guogb@swc.neu.edu.cn 
Xingwei Wang
Northeastern University 

Shenyang, China 

wangxw@mail.neu.edu.cn

    August 5, 2024
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================













§ ABSTRACT 
Graph Neural Networks (GNNs)-based recommendation algorithms typically assume that training and testing data are drawn from independent and identically distributed (IID) spaces. However, this assumption often fails in the presence of out-of-distribution (OOD) data, resulting in significant performance degradation. In this study, we construct a Structural Causal Model (SCM) to analyze interaction data, revealing that environmental confounders (e.g., the COVID-19 pandemic) lead to unstable correlations in GNN-based models, thus impairing their generalization to OOD data. To address this issue, we propose a novel approach, graph representation learning via causal diffusion (CausalDiffRec) for OOD recommendation. This method enhances the model's generalization on OOD data by eliminating environmental confounding factors and learning invariant graph representations. Specifically, we use backdoor adjustment and variational inference to infer the real environmental distribution, thereby eliminating the impact of environmental confounders. This inferred distribution is then used as prior knowledge to guide the representation learning in the reverse phase of the diffusion process to learn the invariant representation. In addition, we provide a theoretical derivation that proves optimizing the objective function of CausalDiffRec can encourage the model to learn environment-invariant graph representations, thereby achieving excellent generalization performance in recommendations under distribution shifts. Our extensive experiments validate the effectiveness of CausalDiffRec in improving the generalization of OOD data, and the average improvement is up to 10.69% on Food, 18.83% on KuaiRec, 22.41% on Yelp2018, and 11.65% on Douban datasets. Our implementation code is available at <https://github.com/user683/CausalDiffRec>.



Graph Neural Networks, Out-of-distribution, Invariant Learning, Recommender Systems















Graph Representation Learning via Causal Diffusion for Out-of-Distribution Recommendation





    Chu Zhao
Northeastern University 

Shenyang, China 

chuzhao@stumail.neu.edu.cn
Enneng Yang
Northeastern University 

Shenyang, China 

ennengyang@stumail.neu.edu.cn
Yuliang Liang
Northeastern University 

Shenyang, China 

liangyuliang@stumail.neu.edu.cn
Pengxiang Lan
Northeastern University 

Shenyang, China 

2390222@stu.neu.edu.cn
Yuting Liu
Northeastern University 

Shenyang, China 

yutingliu@stumail.neu.edu.cn
Jianzhe Zhao
Northeastern University 

Shenyang, China 

zhaojz@swc.neu.edu.cn
Guibing Guo 
Northeastern University 

Shenyang, China 

guogb@swc.neu.edu.cn 
Xingwei Wang
Northeastern University 

Shenyang, China 

wangxw@mail.neu.edu.cn

    August 5, 2024
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ INTRODUCTION


Graph Neural Networks <cit.>, due to their exceptional ability to learn high-order features, have been widely applied in recommendation systems.

GNN-based recommendation algorithms <cit.> learn user and item representations by aggregating information from neighboring nodes in the user-item interaction graph and then computing their similarity to predict user preferences. In addition, researchers have introduced various other techniques to continuously improve GNN-based recommendation algorithms. For example, integrating attention mechanisms <cit.> with knowledge graphs <cit.> led to improving recommendation accuracy. Furthermore, the introduction of contrastive learning aims to improve the robustness of recommendation algorithms <cit.>.

Despite the significant progress these methods have made in improving recommendation accuracy, most of them assume that the distribution of the test dataset and the training dataset is independently and identically distributed (IID) and focus on enhancing recommendation performance based on this assumption. Unfortunately, methods based on the above assumption fail to generalize excellent recommendation performance to out-of-distribution (OOD) data <cit.>, where the distribution of the test data significantly differs from that of the training data. In Figure <ref>, we present a simple example to illustrate how the popularity of medical supplies changes with the environmental factor of COVID-19. Specifically, due to the pandemic, people may be required by the government to stay at home, leading to reduced outdoor activities. During this period, while purchasing medical supplies, people may also increase their demand for fitness equipment and electronic products. The recommender system might learn that users who purchase masks also frequently buy fitness equipment and electronic products, significantly increasing the popularity of these items. This correlation is driven by the common factor of the `pandemic' rather than a direct causal relationship between the items. When the pandemic ends (i.e., the environmental factors change), the popularity of masks decreases, and people's demand for fitness equipment and electronic products diminishes. The recommender system, relying on the unstable correlations learned during the training phase, may incorrectly recommend fitness equipment and electronic products to users who purchase masks, resulting in poor performance under the new distribution. Furthermore, we construct IID and OOD test sets on the Yelp2018 dataset to illustrate the changes in model performance. As shown in Figure <ref>, LightGCN's <cit.> performance on the OOD dataset, compared to the IID dataset, shows an average decline of 29.03% across three metrics. Such issue of GNN-based models lacking robustness on OOD datasets inspires us to propose a recommendation framework with strong generalization capabilities for distribution shifts.



Several works <cit.> have focused on improving the generalization ability of recommender systems on OOD datasets. Some researchers use causal inference to counteract shifts in the data distribution. For example, CausPref <cit.> 
is based on the NeuMF<cit.> method and designs invariant user preference causal learning and anti-preference negative sampling methods to improve model generalization.
COR <cit.>  uses the
Variational Auto-Encoder for
causal modeling by incorporating an encoder to infer unobserved
user features from historical interactions.
Although the aforementioned methods have improved the performance of recommendation models on out-of-distribution datasets to some extent, they are not specifically designed for GNNs, making direct migration to GNN-based methods difficult. 

Other researchers employ techniques such as graph contrastive learning and graph data augmentation to enhance the robustness of GNN-based recommendation algorithms, such as SGL <cit.>, SimGCL <cit.>, and LightGCL <cit.>. These methods mainly address noise or popularity bias in the data, but they struggle to achieve good recommendation performance when the test data distribution is unknown or has multiple distributions. As shown in the experimental results of Tables <ref>, these methods perform worse than other baselines in other types of data distribution shifts. In recent literature, few GNN-based methods <cit.>  have been proposed to improve generalization when faced with multiple distributions. However, these methods lack solid theoretical support. 

Given these limitations, there is an urgent need to design theoretically grounded GNN-based methods to address distribution shifts. In this paper, we use invariant learning to improve the generalization of the OOD dataset. 
Utilizing insights from the prior knowledge of environment distribution and invariant learning <cit.>  enhances model stability across varied environments. This is achieved by acquiring invariant representations, which in turn boosts the model's generalization capabilities and overall robustness.
However, designing models based on invariant learning still faces the following two challenges:

    
   1) How to infer the distribution of underlying environments from observed user-item interaction data? 
    
   2) How to recognize environment-invariant patterns amid changing user behaviors and preferences?


To address the aforementioned challenges, in this paper, we first construct the Structural Causal Model (SCM) to analyze the data generation process in recommender systems and investigate the learning process of GNN-based recommendation algorithms under data distribution shifts. We conclude that latent environmental variables can lead GNN-based algorithms to capture unstable correlations related to the environment, which is the key reason for the failure of GNN-based models to generalize on OOD data. Furthermore, we propose a novel approach called graph representation learning via Causal Diffusion (CausalDiffRec) for OOD recommendation, which leverages causal inference to eliminate unstable correlations caused by environmental variables. This approach aims to learn invariant representations across different environments, thereby achieving OOD generalization in recommender systems. Specifically, CausalDiffRec consists of three main components: an environment generator, an environment inference, and a diffusion module. The environment generator is used to create K significantly different graphs to simulate data distributions under various environments. The environment inference module then infers the environment components from these generated graphs and uses them as input for the diffusion reverse stage to guide invariant graph representation learning. Finally, we provide theoretical proof that CausalDiffRec, under the conditions of invariant learning theory, can identify invariant graph representations across different environments, thereby improving generalization performance on OOD data. The contributions of this paper are concluded as follows:

    
  * Causal Analysis. We construct the SCM and analyze the generalization ability of GNN-based recommendation models on OOD data from the perspective of data generation. Based on our analysis and experimental results, we conclude that environmental confounders lead the model to capture unstable correlations, which is the key reason for its failure to generalize under distribution shifts.
    
  * Methodology. We propose a novel GNN-based method, CausalDiffRec, for OOD recommendation. CausalDiffRec primarily consists of three modules: environment generation, environment inference, and the diffusion module. The environment generation module simulates user data distributions under different conditions/environments; the environment inference module employs causal inference and variational approximation methods to infer the environment distribution; and the diffusion module is used for graph representation learning.
    Our theoretical analysis guarantees that optimizing the objective function of CausalDiffRec enables the model to achieve great generalization. 
    
  * Experimental Findings. We constructed three common types of distribution shifts across four datasets and conducted comparative experiments. The experiments demonstrate that CausalDiffRec consistently outperforms baseline methods. Specifically, when dealing with OOD data, CausalDiffRec exhibits enhanced generalization capabilities, achieving a maximum metric improvement rate of 36.73% compared to the baseline methods.






§ PRELIMINARY



 §.§ GNN-based Recommendation


Given the observed implicit interaction matrix ℛ∈{0,1}^m × n, in which 𝒰 = {u_1, u_2, …, u_m} represents the set of users, ℐ = {i_1, i_2, …, i_n} represents the set of items, m and n denote the number of users and items, respectively. For the elements in the interaction matrix, r_ui = 1 indicates an interaction between user u and item i, otherwise 0. In GNN-based recommendation algorithms, the user-item interaction matrix ℛ is first transformed into a bipartite graph G={𝒱,ℰ}. We employ 𝒱 to represent the node set and ℰ={(u,i)|u ∈𝒰, i ∈ℐ,r_ui = 1} denotes the edge set.
Given a user-item interaction graph 𝒢_u and the true user interactions y_u with respect to (w.r.t) user u, the optimization objective of GNN-based methods can be expressed as:

    arg min_θ 𝔼_(𝒢_u,y_u) ∼ P(G,Y)[l(f_θ(𝒢_u;θ),y_u)],

where f_θ(·) is a learner that learns representations by aggregating high-order neighbor information from the user-item interaction graph. l denotes the loss function and P(G,Y) represents the joint distribution of the interaction graph G and true label Y. 





 §.§ Denoising Diffusion Probabilistic Models


Denoising Diffusion Probabilistic Models (DDPM) <cit.> have been widely used in the field of image and video generation. The key idea of DDPM is to achieve the generation and reconstruction of the input data distribution through a process of gradually adding and removing noise. It leverages neural networks to learn the reverse denoising process from noise to real data distribution.


The diffusion process in recommender system models the evolution of user preferences and item information through noise addition and iterative recovery. Initially, data 𝐱_0 sampled from q(𝐱) undergo a forward diffusion to generate noisy samples 𝓍_1, …, 𝐱_T over T steps. Each step adds Gaussian noise, transforming the data distribution incrementally <cit.>:

    q(𝐱_t|𝐱_t-1) = 𝒩( 𝐱_t;√(1 - β_t)𝐱_t-1,β_t𝐈),

where β_t ∈ (0, 1) controls the level of the added noise at step t. In the reverse phase, the aim is to restore the original data by learning a model p_θ to approximate the reverse diffusion from 𝐱_T to 𝐱_0. The process, governed by p_θ(𝐱_t-1|𝐱_t), uses the mean μ_θ and covariance Σ_θ learned via neural networks:

    p_θ(𝐱_t-1|𝐱_t) = 𝒩( 𝐱_t-1; μ_θ(𝐱_t, t), Σ_θ(𝐱_t, t) ).

The reverse process is optimized to minimize the variational lower bound (VLB), balancing the fidelity of reconstruction and the simplicity of the model <cit.>:

    ℒ_VLB    = 𝔼_q(𝐱_1:T|𝐱_0)[ ∑_t=1^T D_KL(q(𝐱_t-1|𝐱_t, 𝐱_0) || p_θ(𝐱_t-1|𝐱_t)) ] 
         - log p_θ(𝐱_0|𝐱_1),

where D_KL denotes the Kullback-Leibler (KL) divergence. Following <cit.>, we mitigate the training instability issue in the model by expanding and reweighting each KL divergence term in the VLB with specific parameterization. Therefore, we have the following mean squared error loss:

    ℒ_simple = 𝔼_t, 𝐱_0, ϵ_t[ ϵ_t - ϵ_θ( √(α)_t 𝐱_0 + √(1 -α_t)ϵ, t ) ^2 ],

where ϵ_t ∼𝒩 (0,I) is the noise for injection in forward process, ϵ_θ(·) denotes a function approximator that neural networks can replace, and α_t = ∏_t=1^T 1 - β_t.
This framework allows the model to effectively learn noise-free representations and improve recommendation accuracy.





§ METHODOLOGY

In this section, we first construct SCM and identify environmental confounders as the key reason for the failure of GNN-based models to generalize on OOD (out-of-distribution) data. Subsequently, we introduce the variational inference to infer the true distribution of the environment. We use the diffusion model to learn the representation based on invariant learning. Finally, we provide rigorous theoretical proof of CausalDiffRec that can achieve great generalization. The model framework is illustrated in Figure <ref>.




 §.§ SCM of GNN-based Recommendation


To explore the reasons behind the failure of GNN-based models to generalize on OOD data, we follow previous works <cit.> <cit.> and first construct the SCM for data generation and data modeling in recommendation systems, as shown in Figure <ref> (a) and Figure <ref> (b). We find that environmental confounding factors are the key reason for the generalization failure of GNN-based methods. Finally, we design an intervention model in Figure <ref> (c) to eliminate the impact of environmental confounding factors.



  §.§.§ Causal View in GNN-based Recommendation

In Figure <ref>, the three causal relations are derived from the definitions of data generation. The detailed causal analysis behind them is presented as follows:




  * E denotes the unobserved environmental factors (i.e., sudden hot events or policies). G and Y denote the user-item interaction graph and the true label, respectively. I is the invariant attribute of users and items unaffected by environmental factors, such as user gender and item category information. Previous work <cit.> has indicated that leveraging these invariant features can effectively enhance the model's generalization capability in OOD environments.

  * E → G. The direct influence of the environment on user-item interactions can be defined as P(G|E). For example, if the environmental variable is the weather, users might interact more frequently with warm clothing in a cold environment.

  * G → Y. This represents the influence of the user-item interaction graph G on the user behavior label Y. The GNN-based recommendation model Y = f_θ(G) defines the relation. When the model parameters θ are fixed, the mapping between G and Y is also deterministic.

  * I → Y. This indicates that the invariant user-item attributes directly influence the user behavior label Y. For example, a user may consistently prefer to dine at a particular restaurant, and attributes such as the restaurant's location and name generally do not change.

  * E → Y. This denotes that the environment directly influences the user behavior label Y, independent of user interactions between users and items. For example, in a specific holiday environment, users may be more inclined to purchase holiday-related items, regardless of whether they have interacted with them. 
 
In real-world scenarios, training data is collected from heterogeneous environments. Therefore, the environment directly influences the distribution of the data and the prediction result, which can be explicitly represented as P(Y, G|E)=P(G|E)P(Y|G, E). If we employ D_tr(E) to represent the training data distribution for unobserved environments, the GNN-based model, when faced with OOD data, can rewrite Eq. (<ref>) as:

    arg min_θ 𝔼_e ∼ D_tr(E), (𝒢_u,y_u) ∼ P(G,Y|E=e)[ l (f_θ(𝒢_u;θ), y_u)|e],

Eq. (<ref>) shows that environment E affects the data generation used for training the GNN-based recommendation model.





  §.§.§ Confounding Effect of E

Figure <ref> (a) and Figure <ref> (b)  illustrate the causal relationships in data generation and model training for graph-based recommendation algorithms. E acts as the confounder and directly optimizing P(Y|G) leads the GNN-based recommendation model to learn the shortcut predictive relationship between 𝒢_u and y_u, which is highly correlated with the environment E. During the model training process, there is a tendency to use this easily captured shortcut relationship to model user preferences. However, this shortcut relationship is highly sensitive to the environment E. When the environment of the test set is different from that of the training set (i.e., D_tr(E) ≠ D_ts(E)), this relationship becomes unstable and invalid. The recommendation model that excessively learns environment-sensitive relationships in the training data will struggle to accurately model user preferences when faced with out-of-distribution data during the testing phase, resulting in a decrease in recommendation accuracy.



  §.§.§ Intervention

Through the above analysis, we can improve the generalization ability of GNN-based recommendation models by guiding the model to uncover stable predictive relationships behind the training data, specifically those that are less sensitive to environmental changes. Thus, we can eliminate the influence of environmental confounders on model predictions.
Specifically, we learn stable correlations between user item interaction 𝒢_u and ground truth y_u by optimizing P_θ(Y | do(G)) instead of P_θ(Y | G). In causal theory, the do-operation signifies removing the dependencies between the target variable and other variables. As shown in Figure <ref> (c), by cutting off the causal relationship between the environment variables and the user interaction graph, the model no longer learns the unstable correlations between 𝒢_u and y_u. The do-operation simulates the generation process of the interaction graph G, where environmental factors do not influence the user-item interactions. This operation blocks the unstable backdoor path G ← E → Y, enabling the GNN-based recommendation model to capture the desired causal relationship that remains invariant under environmental changes.  

Theoretically, P_θ(Y|do(G)) can be computed through randomized controlled trials, which involve randomly collecting new data from any possible environment to eliminate environmental bias. However, such physical interventions are challenging. For instance, in a short video recommendation setting, it is impossible to expose all short videos to a single user, and it is also impractical to control the environment of data interactions. 
In this paper, we achieve a statistical estimation of P_θ(Y|do(G)) by leveraging backdoor adjustment.
The derivation process is shown as follows:

    P_θ(Y|do(G)) 
       
        = ∑_e P_θ(Y|do(G), E=e, I)P_θ(E=e|do(G))P_θ(I) 
       
        = ∑_e  P_θ(Y|G, E=e, I)P_θ(E=e|do(G))P_θ(I) 
       
        = ∑_e  P_θ(Y|G, E=e, I)P_θ(E=e)P_θ(I)   
       
        = ∑_e P_θ(Y|G, E=e, I)P_θ(E=e, I)   
       
        = 𝔼_e ∼ D_tr(E)[P_θ(Y|G, E, I)],

through the aforementioned backdoor adjustment, the influence of the environment E on the generation of G can be eliminated, enabling the model to learn correlations independent of the environment. However, in recommendation scenarios, environmental variables are typically unobservable or undefined, and their prior distribution P(E=e) cannot be computed. Therefore, directly optimizing the Eq. (<ref>) is challenging.




 §.§ Model Instantiations



  §.§.§ Environment Inference

This work introduces a variational inference method and proposes a variational inference-based environment instantiation mechanism. The core idea is to use variational inference to approximate the true distribution of environments and generate environment pseudo-labels as latent variables. The following tractable evidence lower bound (ELBO) can be obtained as the learning objective:

    log P_θ(Y|do(G)) ≥ℒ_envInf = 𝔼_Q_ϕ(E|G, I)[logP_θ(Y|G, E, I)] 
    - D_KL (Q_ϕ(E|G, I)∥ P_θ(E)),

where Q_ϕ (E|G, I) denotes environment estimation, which draws samples from the true distribution of the environment E. D_KL represents the Kullback–Leibler (KL) divergence of the volitional distribution Q_ϕ(E|G, I) and the prior distribution P_θ(E).
P_θ(Y|G, E, I) is the graph representation learning module that employs the user-item interaction graph and the node attributes of users and items as input to learn invariant representations. Section <ref> will provide a detailed introduction to the graph representation learning module. We present the derivation process of Eq. (<ref>) as follows:

Taking the logarithm on both sides of Eq. (<ref>) and according to Jensen’s Inequality, we have:

    logP_θ(Y|do(G)) 
        
        =log 𝔼_e ∼ D_tr(E)[P_θ(Y|G, E, I)] 
        =
        log∑_e P_θ(Y|G, E=e, I)P_θ(E=e, I)Q_ϕ(E=e|G, I)/Q_ϕ(E=e|G, I)
       ≥∑_e Q_ϕ(E=e|G, I) 
           log P_θ(Y|G,E=e,I)
        P_θ(E=e,I)1/Q_ϕ(E=e|G, I)
       
        = ∑_e [Q_ϕ(E=e|G, I) log P_θ(Y|G,E=e, I) -
           log Q_ϕ(E=e|G, I)P_θ(E=e,I)/Q_ϕ(E=e|G, I)] 
       
        = 𝔼_Q_ϕ(E=e|G, I)[logP_θ(Y|G, E=e, I)] 
       -  D_KL (Q_ϕ(E=e|G, I)∥ P_θ(E=e)).




  §.§.§ Invariant Pattern Recognition Mechanism


Following previous invariant learning studies <cit.>, this work proposes an invariant pattern recognition mechanism to explore invariant graph representations, encouraging the model to learn invariant correlations under distribution shifts. We make the following assumption:


Assumption: For a given user-item interaction graph (i.e., data distribution D), these interaction data are collected from K different environments E. User behavior patterns exist independently of the environment and can be used to generalize out-of-distribution user preference prediction. There exists an optimal invariant graph representation learning F^*(·) satisfying:

    
  * Invariance Property. ∀ e ∈ D(E), P_θ(Y|F^*(G),E=e, I) = P(Y|F^*(G), I). 
    
    
  * Sufficiency Condition. Y = F^*(G) + ϵ, ϵ E, where  indicates statistical independence and ϵ is random noise.


The invariance property assumption indicates that a graph representation learning model exists capable of learning invariant user-item representations across different data distribution environments. The sufficiency condition assumption means that the learned invariant representations enable the model to make accurate predictions.



  §.§.§ Invariant Representation Learning
  This section mainly consists of an environment generator, a diffusion-based graph representation learning module, and a recommendation module. Next, we will detail how they collaborate to enhance the generalizability of GNN-based models on OOD data and improve recommendation accuracy.

Environment Generator.
In real-world recommendation scenarios, training datasets are collected in various environments. However, for a single user-centric interaction graph, the training dataset comes from a single environment. We need to learn environment-invariant correlations from training data originating from different environments to achieve the generalization capability of GNN-based recommendation models under distribution shifts. To circumvent this dilemma, this paper designs an environment generator g_ω_k(·) (1 ≤ k ≤ K ), which takes the user's original interaction graph G as input and generates a set of K interaction graphs {G_i}_i=1^K to simulate training data from different environments. The optimization objective is expressed as follows:

    ℒ_generator = [ Var(ℒ(g_ω_k(G)): 1 ≤ k ≤ K )],

where Var(·) denotes the variance and ℒ(·) is the loss function. Following existing work <cit.>, we modify the graph structure by adding and removing edges. Given a Boolean matrix B_k, the adjacency matrix A of the graph, and its complement A', the k-th generated view for the original view is A_k = A + B_k ⊙ (A - A'). Since B_k is a discrete matrix and not differentiable, it cannot be optimized directly. To address this issue, we borrow the idea from <cit.> and use reinforcement learning to treat graph generation as a decision process and edge editing as actions. Specifically, for view k, we consider a parameter matrix θ_k = {θ_nm^k}. For the n-th node, the probability of exiting the edge between it and the m-th node is given by:

    h(a_nm^k) = exp(θ_nm^k)/∑^m'=m_m'=1exp(θ_nm'^k).


We then sample s actions {b_nmt^k}_t=1^s from a multinomial distribution M(h(α_n1^k), …, h(α_nm^k)), which give the nonzero entries in the n-th row of B_k. The reward function R(G_k) can be defined as the inverse loss. We can use the reinforcement algorithm to optimize the generator with the gradient:

    ∇_θ_klog h_θ_k(A_k) R(G_k),

where θ_k is the model parameters  and h_θ_k(A_k) = ∏_n ∏_t=1^s h(b_nmt^k). Optimizing Eq. (<ref>) ensures that the generated graphs have large differences.



Causal Diffusion. Given the generated interaction graph G_k=(A_k, I) where A is the adjacency matrix, and I is the feature matrix of users or items, instead of directly using G_k as input for diffusion, we use the encoder from the Variational Graph Autoencoder (VGAE) to compress G_k to a low-dimensional vector 𝐱^0_k ∼𝒩(μ_k,σ_k) for subsequent environment inference and graph invariant representation leanring. The encoding process is as follows:
 
    q_ψ(𝐱^0_k|A_k,I) = N(𝐱^0_k|μ_k,σ_k),

where μ_k = GCN_μ(A_k, I) is matrix of mean vectors and σ_k = GCN_σ(A_k, I) denotes standard deviation. The GCN(·) is the graph convolution network in the graph variational autoencoder. According to the reparameterization trick, x^0_k can be calculated  as follows:

    𝐱^0_k = μ_k + σ_k ⊙ϵ,

where ϵ∼ N(0,I) and ⊙ is the element product. Latent embedding 𝐱^0_k will be used as the input for the environment inference module to generate environment pseudo-labels. Meanwhile, 𝐱^0_k will do the forward and reverse processes in the latent space to learn the user/item embeddings in DDPM.  The forward process can be calculated as:

    q(𝐱^1:T_k |𝐱^0_k) = ∏_t=1^T q(𝐱^t_k |𝐱^t-1_k).

After obtaining the environment approximation variable  z_causal = Q(E|G_k, I) according to Eq. (<ref>), the pair of latent variables (z_causal, 𝐱^T_k) to learn the invariant graph representation. We approximate the inference distribution by parameterizing the probabilistic decoder through a conditional DDPM p_θ(𝐱^t-1_k | 𝐱^T_k, z_causal). Using DDPM, the forward process is entirely deterministic except for t = 1.  We define the joint distribution of the reverse generative process as follows:

    p_θ(𝐱^0:T_k | z_causal) = p(𝐱^T_k) ∏_t=1^T p_θ(𝐱^t-1_k |𝐱^t_k, z_causal).



    p_θ(𝐱^t-1_k|𝐱) = 𝒩( 𝐱^t-1_k; μ_θ(𝐱^t_k,z_causal, t), Σ_θ(𝐱^t_k, z_causal, t) ).

The loss function in Eq. (<ref>) can be rewritten as:

    ℒ_Invsimple = 𝔼_t, 𝐱_0, ϵ_t[ ϵ_t - ϵ_θ( 𝐱^t_k, z_causal, t ) ^2 ],

where 𝐱^t_k = √(α)_t 𝐱^0_k + √(1 -α_t)ϵ. 

After obtaining the reconstructed output vector ℛ = 𝐱^0'_k from DDPM, it will be used as the input for the decoder of the variational graph decoder, which then reconstructs the input graph. The entire process is illustrated as follows:

    Â_k = ϕ(ℛℛ^⊤),



where ϕ(·) is the activation function (the sigmoid function is used in this paper). The VGAE is optimized by the variational lower bound:

    ℒ_VGAE = 𝔼_q_ψ(𝐱^0_k|A_k,I)[ log p_θ(Â_k|𝐱^0_k) ]    
    - D_KL( q_ψ(𝐱^0_k|A_k,I) ∥ p(𝐱^0_k) ).


Prediction and Joint Optimization. Using the well-trained diffusion model to sample the final embeddings for user preference modeling:

    r̂_u,i =  e^⊤_ue_i,

where e_u and e_i denote the final user embedding w.r.t u-th user and item embedding w.r.t i-th item, respectively. Without loss of generality, LightGCN is used as the recommendation backbone, and Bayesian Personalized Ranking (BRP) loss is employed to optimize the model parameters:

    ℒ_rec = ∑_u, v^+, v^- -logσ(r̂_u, v^+ - r̂_u, v^-),

where (u, v^+, v^-) is a triplet sample for pairwise recommendation training. v^+ represents positive samples from which the user has interacted, and v^- are the negative samples that are randomly drawn from the set of items with which the user has not interacted, respectively. 

We use a joint learning strategy to optimize CausalDiffRec:

    ℒ =    ℒ_rec + λ_1 ·ℒ_generator
        +λ_2 · (ℒ_VGAE + ℒ_Invsample) +  λ_3 ·ℒ_envInf,

where λ_1, λ_2, and λ_3 are hyper-parameters. 

In section <ref>,
we provide the theoretical proof that our proposed CausalDiffRec can get rid of the unstable correlation and learn OOD generalized representations when satisfying the Assumption. 

The pseudocode for the model training process is shown in Algorithm <ref>.



§ THEORETICAL ANALYSIS
 
CausalDiffRec aims to learn the optimal generator F^*(·) as stated in the assumption in section <ref>, thereby obtaining invariant graph representations to achieve OOD generalization in recommendation performance under data distribution shifts. 
Before starting the theoretical derivation, let's do some preliminary work. For the convenience of theoretical proof, we rewrite Eq. (<ref>) as:

    argmin_θ  (ℒ_task +  ℒ_infer),

where ℒ_task = ℒ_rec + ℒ_generator + ℒ_VGAE+ ℒ_Invsample and  ℒ_infer = ℒ_envInf, and for the derivation convenience, we temporarily ignore the penalty coefficient. The ℒ_task and ℒ_infer can be further abstracted as:

    ℒ_task  = arg min_θ𝔼_e ∼ D_tr(E), (𝒢_u,y_u) ∼ P(Y,G|E=e)
          [  l (f_θ(𝒢_u;θ), y_u)] 
       ℒ_infer 
            = min_q(Y|z_causal)Var{𝔼_e ∼ D_tr(E), (𝒢_u,y_u) ∼ P(Y,G|E=e)
          [l (f_θ(𝒢_u;θ), y_u)|do(𝒢_u)]}.

We follow the proof technique from <cit.> and show the optimality of the Eq. (<ref>) with the following two propositions that can achieve OOD recommendation.

 
Minimizing Eq. (<ref>) promotes the model's adherence to the Invariance Property and the Sufficient Condition outlined in Assumption (in Sec. <ref>).



 
Optimizing Eq. (<ref>) corresponds to minimizing the upper bound of the OOD generalization error described in Eq. (<ref>).



Proposition <ref> and Proposition <ref>, respectively, avoid strong hypotheses and ensure that the OOD generalization error bound of the learned model is within the expected range. In fact, this can also be explained from the perspective of the SCM model in Figure <ref>. Optimizing Eq. (<ref>) eliminates the negative impact of unstable correlations learned by the model, which are caused by latent environments, on modeling user preferences. At the same time, it enhances the model's ability to learn invariant causal features across different latent environments. Proofs for Proposition <ref> and Proposition <ref> are shown as follows:

Before starting the proof, we directly follow previous work <cit.> to propose the following lemma, using information theory to interpret the invariance property and sufficient condition in Assumption and to assist in the proof of Proposition <ref>.


Using the Mutual Information 𝕀(;), the invariance property and sufficient condition in Assumption can be equivalently expressed as follow lemma:

(1) Invariance: ∀ e ∈ D(E), P_θ(Y|𝒫^*_Inv, E=e, I) = P(Y|𝒫^*_Inv, I) ⇔𝕀(Y; E|𝒫^*_Inv, I) = 0 where 𝒫^*_Inv =  F^*(G). 
(2) Sufficiency: 𝕀(Y; 𝒫^*_Inv, I) is maxmized.





    For the invariance property, it is easy to get the following equation:
    
    𝕀(Y;E|𝒫^*_Inv, I) 
       = 𝔼_𝒫^*_Inv, I[ 𝔻_KL( P(Y, E | 𝒫^*_Inv, I)  P(Y | 𝒫^*_Inv, I) P(E | 𝒫^*_Inv, I) ) ]

    For the sufficient condition, we use the method of contradiction and prove it through the following two steps:
    
First, we prove that for Y, 𝒫^*_Inv, and I satisfying 𝒫^*_Inv = max_𝒫_Inv𝕀(Y; 𝒫_Inv, I), they also satisfy that 𝕀(Y; 𝒫^*_Inv, I) is maximized. We use the method of contradiction to prove this.
Assume 𝒫^*_Inv≠max_𝒫_Inv𝕀(Y; 𝒫_Inv, I), and there exists 𝒫^'_Inv = max_𝒫_Inv𝕀(Y; 𝒫_Inv, I), where 𝒫^'_Inv≠𝒫^*_Inv. We can always find a mapping function M such that 𝒫^'_Inv = M(𝒫^*_Inv, R), where R is a random variable. Then we have:

    𝕀(Y; 𝒫^'_Inv, I)     = 𝕀(Y; 𝒫^*_Inv, R, I) 
        = 𝕀(Y; 𝒫^*_Inv, I) + 𝕀(Y; R | 𝒫^*_Inv, I).

Since R is a random variable and does not contain any information about Y, we have 𝕀(Y; R | 𝒫^*_Inv, I) = 0. Therefore:

    𝕀(Y; 𝒫^'_Inv, I) = 𝕀(Y; 𝒫^*_Inv, I).

This leads to a contradiction. 

Next, we prove that for Y, 𝒫^*_Inv, and I satisfying 𝒫^*_Inv = max_𝒫_Inv𝕀(Y; 𝒫_Inv, I), they also satisfy that 𝕀(Y; 𝒫^*_Inv, I) is maximized. Assume 𝒫^*_Inv≠max_𝒫_Inv𝕀(Y; 𝒫_Inv, I), and there exists 𝒫^'_Inv = max_𝒫_Inv𝕀(Y; 𝒫_Inv, I), where 𝒫^'_Inv≠𝒫^*_Inv. We have the following inequality:

    𝕀(Y; 𝒫^*_Inv, I) ≤𝕀(Y; 𝒫^'_Inv, I).

From this, we can deduce that:

    𝒫^'_Inv = max_𝒫_Inv𝕀(Y; 𝒫_Inv, I),
 where contradicts 𝒫^*_Inv = max_𝒫_Inv𝕀(Y; 𝒫_Inv, I).
Since the assumption leads to a contradiction, and the assumption does not hold. Therefore, 𝒫^*_Inv = max_𝒫_Inv𝕀(Y; 𝒫_Inv, I) holds. This proves that 𝕀(Y; 𝒫^*_Inv, I) is maximized. The lemma <ref> is complicated proven.


Proof of Proposition 
<ref>. 

    First, optimizing the first term ℒ_task in Eq. (<ref>) enables the model to satisfy the sufficient condition. Analyzing the SCM in Figure <ref>(c), we have the fact that max_q(z|G, I)𝕀(Y,z_causal) is equivalent to min_q(z|G, I)𝕀(Y, G|Z_causal), as we use do(G) to eliminate the unstable correlations between Y and G caused by the latent environment. We have:
    
    𝕀(Y,G|z_causal)     = D_KL(p(Y|G,E)p(Y|z_causal,E)) 
        = D_KL(p(Y|G,E)p(Y|z_causal)) 
        - D_KL(p(Y|z_causal, E)  q(Y|z_causal)) 
       ≤ D_KL(p(Y|G,E)p(Y|z_causal)).

    Based on the above derivation, we have:
    
    𝕀(Y,G|z_causal) ≤
       min_q(Y|z_causal) D_KL(p(Y|G,E)p(Y|z_causal)).

    Besides, we have:
    
    D_KL(p(Y|G,E)p(Y|z_causal))
               
       = 𝔼_e∈ D_tr(E)𝔼_(G,Y) ∼ p(G,Y|e)
         𝔼_z_causal∼ q(z_causal|G,I)[logq(Y|G,e)/p(Y|z_causal)] 
       ≤𝔼_e∈ D_tr(E)𝔼_(G,Y) ∼ p(G,Y|e)
       [logp(Y|G,e)/𝔼_z_causal∼ q(z_causal|G,I)q(Y|z_causal)] (Jensen Inequality).

    Finally, we reach:
    
    min_q(Y|z_causal) D_KL(p(Y|G,E)p(Y|z_causal)) 
             
       ⇔arg min_θ𝔼_e ∼ D_tr(E), (𝒢_u,y_u) ∼ P(Y,G|E=e) [  l (f_θ(𝒢_u;θ), y_u)].

    Thus, we have demonstrated that minimizing the expectation term (ℒ_task) in Eq. (<ref>) is equivalent to minimizing the upper bound of 𝕀(Y; G | z_causal). This results in maximizing 𝕀(Y; 𝒫^*_Inv, I), thereby helping to ensure that the model satisfies the Sufficient Condition.
    
    Next, we prove that optimizing the first term ℒ_task in Eq. (<ref>) enables the model to satisfy the Invariance Property.
    Similar to Eq. (<ref>), we have:
    
    𝕀(Y; E =e | z_causal) 
       = D_KL(p(Y | z_causal, e) ∥ p(Y | z_causal)) 
       = D_KL(p(Y | z_causal, E) ∥𝔼_e ∈ D(E)[p(Y | z_causal, e)]) 
       = D_KL(q(Y | z_causal) ∥𝔼_e ∈ D(E)[q(Y | z_causal)]) 
        - D_KL(q(Y | z_causal) ∥ p(Y | z_causal, e)) 
        - D_KL(𝔼_e ∈ D(E)[p(Y | z_causal, e)] ∥𝔼_e ∈ D(E)[q(Y | z_causal)]) 
       ≤ D_KL(q(Y | z_causal) ∥𝔼_e ∈ D(E)[q(Y | z_causal)]).

    Besides, the last term in Eq. (<ref>) can be further expressed as:
    
    D_KL(q(Y | z_causal) ∥𝔼_e ∈ D(E)[q(Y | z_causal)]) 
        = 𝔼_e∈ D_tr(E)𝔼_(G,Y) ∼ p(G,Y|e)𝔼_z_causal∼ q(z_causal|G,I)
         [logp(Y|G,e)/𝔼_e ∈ D(E)q(Y|z_causal)] (Jensen Inequality)
       ≤𝔼_e∈ D(E)[| l (f_θ(𝒢_u;θ), y_u)-  𝔼_e ∈ D(E)[l (f_θ(𝒢_u;θ), y_u)]|]

    where the last term in Eq. (<ref>) is the upper bound for the D_KL(q(Y | z_causal) ∥𝔼_e ∈ D(E)[q(Y | z_causal)]). 
    Finally, we have:
    
    min_q(Y|z_causal) D_KL(q(Y | z_causal) ∥𝔼_e ∈ D(E)[q(Y | z_causal)])  
       ⇔min_q(Y|z_causal)Var{𝔼_e ∼ D_tr(E), (𝒢_u,y_u) ∼ P(Y,G|E=e)
        [l (f_θ(𝒢_u;θ), y_u)|do(𝒢_u)]}.

    Hence, minimizing the variance term (ℒ_risk) in Eq. (<ref>) effectively reduces the upper bound of 𝕀(Y; E =e | z_causal). Thereby ensuring the model adheres to the Invariance Property.


Proof of Proposition <ref>. Optimizing Eq. (<ref>) is tantamount to reducing the upper bound of the OOD generalization error in Eq. (<ref>).
   Let q(Y | G) represent the inferred variational distribution of the true distribution p(Y | G, E). The OOD generalization error can be quantified by the KL divergence between these two distributions:
   
    D_KL(p(Y|G,E)q(Y|G)) 
        =𝔼_e ∈ D_tr(E)𝔼_(Y,G) ∼ p(Y,G|E)logp(Y|G,E=e)/q(Y|G).

   Following previous work, we use information theory to assist in the proof of Proposition <ref>.
   We propose the lemma <ref> to rewrite the OOD generalization, which is shown as follows:
    
       The out-of-distribution generalization error is limited by:
       
    D_KL(p(Y|G,E)q(Y|G)) ≤ D_KL[p(Y|G,E)q(Y|z_causal)],

   
   where q(Y|z_causal) is the inferred variational environment distribution. The proof of Lemma <ref> is shown as:
   
    D_KL(p(Y|G,E=e)q(Y|z_causal)) 
        = 𝔼_e ∈ D(E)𝔼_(Y,G) ∼ p(G, Y|E=e)[logp(Y|G,E=e)/q(Y|G)] 
        = 𝔼_e ∈ D(E)𝔼_(Y,G) ∼ p(G, Y|E=e)
         [logp(Y|G,E=e)/𝔼_z_causal∼ q(z_causal|G,I)q(Y|z_causal)] 
       ≤𝔼_e ∈ D(E)𝔼_(Y,G) ∼ p(G, Y|E=e)
         𝔼_z_causal∼ q(z_causal|G,I)logp(Y|G,e)/q(Y|z_causal)
        = D_KL[p(Y|G,E)q(Y|z_causal)],

   The Lemma <ref> has been fully proven. Based on Lemma <ref> and Proposition <ref>, the Eq. (<ref>) can be adapted as:
   
    min_q(z_causal|G,I),q(Y,z_causal)
        D_KL(p(Y|G,E=e)q(Y|z_causal)) + 𝕀(Y,E=e|z_causal)

   Hence, according to Lemma <ref>, we confirm that minimizing Eq. (<ref>) is tantamount to minimizing the upper bound of the OOD generalization error in Eq. (<ref>), meaning that:
   
    argmin_θ(ℒ_task + ℒ_infer)  ⇔min_q(z_causal|G,I),q(Y,z_causal)
          D_KL(p(Y|G,E=e)q(Y|z_causal)) 
        + 𝕀(Y,E=e|z_causal) (𝕀(Y,E=e|z_causal) is  non-negative)  
       ≥min_q(z_causal|G,I),q(Y,z_causal)
          D_KL(p(Y|G,E=e)q(Y|z_causal)) 
       ≥  D_KL(p(Y|G,E)q(Y|G)).

   The Proposition <ref> is completely proven.








§ EXPERIMENTS

In this section, we conducted extensive experiments to validate the performance of CausalDiffRec and address the following key research questions:


  * RQ1: How does CausalDiffRec compare to the state-of-the-art strategies in both OOD and IID test evaluations?

  * RQ2: Are the proposed components of CausalDiffRec effective for OOD generalization?

  * RQ3: How do hyperparameter settings affect the performance of CausalDiffRec?








 §.§ Experimental Settings


Datasets. We evaluate our proposed CausalDiffRec on four real-world datasets: Food[https://www.aclweb.org/anthology/D19-1613/], KuaiRec[https://kuairec.com] Yelp2018[https://www.yelp.com/dataset], and Douban[https://www.kaggle.com/datasets/] comprise raw data from the Douban system. Detailed statistics of the datasets are presented in Table 1. More data processing details can be seen in the code implementation link.
Following exiting work <cit.>,
we constructed OOD test sets for three common scenarios of distribution shift:


  * Temporal shift. We sort the dataset in descending order by timestamp and used each user's most recent 20% of interactions as the OOD test set. 
The first 60% of the data based on interaction time is used for the training set. Food is used for this type of shift.

  * Exposure shift. In KuaiRec, the small matrix is fully exposed and used for the OOD test set, and the big matrix is collected from the online platform and used for the training dataset, resulting in a distributional shift. 

  * Popularity shift. We randomly select 20% of the interactions to create the OOD test set, ensuring that item popularity follows a uniform distribution. The training set maintains a long-tail distribution. This type of shift was applied to the Yelp2018 and Douban datasets.




Baselines. We compare the CusalDiffRec with the state-of-the-art models:
LightGCN<cit.>, 
SGL <cit.>,
SimGCL<cit.>, LightGCL<cit.>, InvPref <cit.>, 
InvCF <cit.>, 
AdvDrop <cit.>, AdvInfo <cit.>, and DRO <cit.>.



Hyperparameter Settings. We implement our CausalDiffRec in Pytorch. All experiments are conducted on a single RTX-4090 with 24G memory. Following the default hyperparameter search settings of the baselines, we expand their hyperparameter search space and tune the hyperparameters.
For our CausalDiffRec, we tune
the learning rates in {1e-3, 1e-4, 1e-5}. The number of diffusion steps varies between 10 and 1000, and the diffusion embedding size is chosen in {8,16,32,64}. Additional hyperparameter details are available in our released code.





























































 §.§ Overall Performance (RQ1)

This section compares CausalDiffRec's performance and baselines under various data shifts and conducts a performance analysis.




Evaluation on temporal shift: Table <ref> shows that CausalDiffRec significantly outperforms SOTA models on the Food dataset, with improvements of 1.99%, 24.89%, 6.03%, and 9.68% in Recall and NDCG. This indicates CausalDiffRec's effectiveness in handling temporal shift. DRO also excels in this area, with a 15% improvement over LightGCN in NDCG@20, due to its robust optimization across various data distributions. CDR surpasses GNN-based models thanks to its temporal VAE-based architecture, capturing preference shifts from temporal changes.

Evaluation on exposure shift: In real-world scenarios, only a small subset of items is exposed to users, leading to non-random missing interaction records. Using the fully exposed KuaiRec dataset, CausalDiffRec consistently outperforms baselines, with improvements ranging from 6.90% to 28.83%, indicating its capability to handle exposure bias. DRO and AdvInfoNce also show superior performance in NDCG and Recall metrics, enhancing the generalization of GNN-based models and demonstrating robustness compared to LightGCN.


Evaluation on popularity shift. We compare model performance on the Yelp2018 and Douban datasets, showing that our model significantly outperforms the baselines. On Douban, CausalDiffRec achieves 8.22% to 17.96% improvement, and on Yelp2018, the improvements range from 11.24% to 36.73%.
Methods using contrastive learning (e.g., SimGCL, LightGCL, AdvInfoNce) outperform other baselines in handling popularity shifts. This is because the InfoNCE loss helps the model learn a more uniform representation distribution, reducing bias towards popular items. InvPref performs best among the baselines on Yelp2018, using clustering for contextual labels, unlike our variational inference approach. Our method, tailored for graph data, aggregates neighbor information for better recommendation performance than matrix factorization-based methods.

Additionally, in Table <ref>, we report the performance of CausalDiffRec compared to several baseline models that use LightGCN as the backbone. From the table, we observe the following: 1) These baseline models outperform LightGCN on IID datasets; 2) CausalDiffRec outperforms all baseline models across all metrics. This indicates that CausalDiffRec also performs well on IID datasets. We attribute the performance improvement to our use of data augmentation and the incorporation of auxiliary information in modeling user preferences.

In summary, the analysis of experimental results demonstrates that our proposed CausalDiffRec can handle different types of distribution shifts and achieve good generalization.










 §.§ In-depth Analysis (RQ2)

In this section, we conduct ablation experiments to study the impact of each component of CausalDiffRec on recommendation performance. The main components include the environment generator module and the environment inference module. Additionally, we use t-SNE to visualize the item representations captured by the baseline model and CausalDiffRec, to compare the models' generalization capabilities on OOD data.



Ablation studies. Table <ref> presents the results of the ablation study that compares LightGCN, CausalDiffRec, and its two variants: 'w/o Gen.' (without the environment generator) and 'w/o Env.' (without the environment inference). The results show that removing these modules causes a significant drop in all metrics across four datasets. For example, on Yelp2018, Recall, and NDCG decreased by 148.15% and 126.83%, respectively, demonstrating the effectiveness of CausalDiffRec based on invariant learning theory for enhancing recommendation performance on OOD datasets.
Additionally, even without the modules, CausalDiffRec still outperforms LightGCN on popularity shift datasets (Yelp2018 and Douban) due to the effectiveness of data augmentation and environment inference. However, on the Food and KuaiRec datasets, removing either module results in worse performance than LightGCN, likely due to multiple biases in these datasets. Without one module, the model struggles to handle multiple data distributions, leading to a performance drop.
Overall, the ablation experiments highlight the importance of all modules in CausalDiffRec for improving recommendation performance and generalizing on OOD data.

Visualization analysis. In Figure <ref> and Figure <ref>, we used t-SNE to visualize the item representations learned by LightGCN, SimGCL, and CausalDiffRec on Douban and Yelp2018 datasets to better observe our model's ability to handle distribution shifts. Following previous work <cit.>, we recorded the popularity of each item in the training set and designated the top 10% most popular items as 'popular items' and the bottom 10% as 'unpopular items'. It is obvious that the embeddings of popular and unpopular items learned by LightGCN still exhibit a gap in the representation space. In contrast, the embeddings learned by CausalDiffRec are more evenly distributed within the same space. This indicates that CausalDiffRec can mitigate the popularity shift caused by popular items.  Additionally, we found that the embeddings of popular and unpopular items learned by SimGCL are more evenly distributed compared to LightGCN. This is because contrastive learning can learn a uniform representation distribution.

















 §.§ Hyperparameter Investigation (RQ3)

Effect of Diffusion Step T.
We conducted experiments to investigate the impact of the number of diffusion steps on performance; in CausalDiffRec, we used the same number of steps in the forward and reverse phases.
we compare the performance with T changing from 10 to 500. We present the results in Figure <ref>, and we have the following findings:

    
  * When the number of steps is chosen within the range {10, 50, 100}, CausalDiffRec achieves the best performance across all datasets. We find that appropriately increasing the number of steps significantly improves Recall@20 and NDCG@20. These performance enhancements are mainly attributed to the diffusion enriching the representation capabilities of users and items.
    
  * Nevertheless, as we continue to increase the number of steps, the model will face overfitting issues. For example, on the food and yelp2018 datasets, Recall@20 and NDCG@20 consistently decrease. Although there is an upward trend on KuaiRec, the optimal solution is not achieved. Additionally, it is evident that more steps also lead to longer training times. We should carefully adjust the number of steps to find the optimal balance between enhancing representation ability and avoiding overfitting.


Effect of Penalty Coefficient.
Figure <ref> reports the experimental results of varying the coefficients λ_1, λ_2, and λ_3 on food and KuaiRec. We compare the performances with all coefficients changing from 1e-1 to 1e-4. Further, we have the following conclusions: 

    
  * λ_1 ∈ [1e-1, 1e-2] typically strikes a good balance and delivers excellent performance. When further reducing the value of λ_1, the model's performance declined. We speculate that this led to overfitting. Similarly, we found that larger values of λ_3 can also achieve better recommendation performance. Therefore, the value range of λ_3 is the same as that of λ_1.
    
  * λ_3 directly regularizes the capability of the graph representation learning module. We found that for different datasets, values that are too large or too small do not allow the model to learn optimal representations. Taking everything into consideration, we set λ_3 to 1e-3 to achieve good generalization.



Effect of the number of Environments. Figure <ref> shows the impact of the number of environments on the model's performance. We can see that as the number of environments increases, the performance of CausalDiffRec improves across the three datasets. This indicates that more environments help enhance the model's generalization on OOD (Out-of-Distribution) data. However, as the number of environments further increases, performance declines, which we believe is due to the model overfitting to too many environments.




§ RELATED WORK



 §.§ GNN-based Recommendation

Recent developments in graph-based recommender systems have leveraged graph neural networks to model user-item interactions as a bipartite graph, enhancing recommendation accuracy through complex interaction capture <cit.>. Notably, LightGCN focuses on neighborhood aggregation without additional transformations, while other approaches employ attention mechanisms to prioritize influential interactions <cit.>. Further research explores non-Euclidean spaces like hyperbolic space to better represent user-item relationships <cit.>. Knowledge graphs also enhance these systems by integrating rich semantic and relational data directly into the recommendation process <cit.>. Despite these advancements, graph-based systems often struggle with out-of-distribution data due to the IID assumption and are challenged by multiple distribution shifts <cit.>. Additionally, contrastive learning methods in these systems rely on a fixed paradigm that lacks robust theoretical support, limiting adaptability to varied data shifts. However, the aforementioned models are trained on datasets where the training and test data distributions are drawn from the same distribution, leading to generalization failure when facing OOD data.



 §.§ Diffusion based Recoomendation

The integration of diffusion processes into recommender systems leverages diffusion mechanisms to model dynamic propagation of user preferences and item information through interaction networks, enhancing recommendation accuracy and timeliness <cit.>. These models capture evolving user behaviors and have shown potential in various recommendation contexts, from sequential recommendations to location-based services. For instance, DiffRec <cit.> applies diffusion directly for recommendations, while Diff-POI <cit.> models location preferences. Furthermore, approaches like DiffKG <cit.> and RecDiff <cit.> utilize diffusion for denoising entity representations in knowledge graphs and user data in social recommendations, respectively, enhancing the robustness and reliability of the systems. These studies underscore diffusion's suitability for advanced representation learning in recommender systems. However, these methods cannot solve the OOD problem.




 §.§ Out-of-Distribution Recommendation


Researchers have focused on recommendation algorithms for out-of-distribution (OOD) data. COR <cit.> infers latent environmental factors in OOD data. CausPref <cit.> learns invariant user preferences and causal structures using anti-preference negative sampling. CaseQ <cit.> employs backdoor adjustment and variational inference for sequential recommendations. InvPref <cit.> separates invariant and variant preferences by identifying heterogeneous environments. However, these methods don't directly apply to graph-based recommendation models and fail to address OOD in graph structures. AdaDrop <cit.> uses adversarial learning and graph neural networks to enhance performance by decoupling user preferences. DRO <cit.> integrates Distributionally Robust Optimization into Graph Neural Networks to handle distribution shifts in graph-based recommender systems. Distinct from these GNN-based methods, this paper explores how to use the theory of invariant learning to design GNN-based methods with good generalization capabilities.





§ CONCLUSION

In this paper, we propose a novel GNN-based model for OOD recommendation, called CausalDiffRec, designed to learn environment-invariant graph representations to enhance the generalization of recommendation models on OOD data. CausalDiffRec employs the backdoor criterion from causal inference and variational inference methods to eliminate the influence of environmental confounders. It uses a diffusion-based sampling strategy to learn graph representations. Our approach is grounded in invariant learning theory, and we provide theoretical proof that by optimizing the objective function in CausalDiffRec, the model is encouraged to identify invariant graph representations across environments, thereby improving its generalization performance on OOD data. Experiments conducted on four real-world datasets demonstrate that the proposed CausalDiffRec framework outperforms baseline models. In addition, ablation studies further validate the effectiveness of the model.



00
wu2022graph
S. Wu, F. Sun, W. Zhang, X. Xie, and B. Cui,
"Graph neural networks in recommender systems: a survey,"
ACM Computing Surveys, vol. 55, no. 5, pp. 1–37, 2022.

wang2019neural
X. Wang, X. He, M. Wang, F. Feng, and T.-S. Chua,
"Neural graph collaborative filtering,"
in SIGIR, 2019, pp. 165–174.

he2020lightgcn
X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, and M. Wang,
"Lightgcn: Simplifying and powering graph convolution network for recommendation,"
in SIGIR, 2020, pp. 639–648.

wang2020disenhan
Y. Wang, S. Tang, Y. Lei, W. Song, S. Wang, and M. Zhang,
"Disenhan: Disentangled heterogeneous graph attention network for recommendation,"
in CIKM, 2020, pp. 1605–1614.

chang2021sequential
J. Chang, C. Gao, Y. Zheng, Y. Hui, Y. Niu, Y. Song, D. Jin, and Y. Li,
"Sequential recommendation with graph neural networks,"
in SIGIR, 2021, pp. 378–387.

wang2021session
J. Wang, K. Ding, Z. Zhu, and J. Caverlee,
"Session-based recommendation with hypergraph attention networks,"
in SDM, 2021, pp. 82–90, SIAM.

wang2020disentangled
X. Wang, H. Jin, A. Zhang, X. He, T. Xu, and T.-S. Chua,
"Disentangled graph collaborative filtering,"
in SIGIR, 2020, pp. 1001–1010.

niu2021review
Z. Niu, G. Zhong, and H. Yu,
"A review on the attention mechanism of deep learning,"
Neurocomputing, vol. 452, pp. 48–62, 2021.

fukui2019attention
H. Fukui, T. Hirakawa, T. Yamashita, and H. Fujiyoshi,
"Attention branch network: Learning of attention mechanism for visual explanation,"
in CVPR, 2019, pp. 10705–10714.

zhang2022geometric
Y. Zhang, C. Li, X. Xie, X. Wang, C. Shi, Y. Liu, H. Sun, L. Zhang, W. Deng, and Q. Zhang,
"Geometric disentangled collaborative filtering,"
in SIGIR, 2022, pp. 80–90.

sun2021hgcf
J. Sun, Z. Cheng, S. Zuberi, F. Pérez, and M. Volkovs,
"Hgcf: Hyperbolic graph convolution networks for collaborative filtering,"
in WWW, 2021, pp. 593–601.

wang2019kgat
X. Wang, X. He, Y. Cao, M. Liu, and T.-S. Chua,
"Kgat: Knowledge graph attention network for recommendation,"
in KDD, 2019, pp. 950–958.

cao2019unifying
Y. Cao, X. Wang, X. He, Z. Hu, and T.-S. Chua,
"Unifying knowledge graph learning and recommendation: Towards a better understanding of user preferences,"
in WWW, 2019, pp. 151–161.

wu2021self
J. Wu, X. Wang, F. Feng, X. He, L. Chen, J. Lian, and X. Xie,
"Self-supervised graph learning for recommendation,"
in SIGIR, 2021, pp. 726–735.

yu2022graph
J. Yu, H. Yin, X. Xia, T. Chen, L. Cui, and Q. V. Hung,
"Are graph augmentations necessary? simple graph contrastive learning for recommendation,"
in SIGIR, 2022, pp. 1294–1303.

jiang2023adaptive
Y. Jiang, C. Huang, and L. Huang,
"Adaptive graph contrastive learning for recommendation,"
in KDD, 2023, pp. 4252–4261.

xia2023automated
L. Xia, C. Huang, C. Huang, K. Lin, T. Yu, and B. Kao,
"Automated self-supervised learning for recommendation,"
in WWW, 2023, pp. 992–1002.

li2023graph
C. Li, L. Xia, X. Ren, Y. Ye, Y. Xu, and C. Huang,
"Graph transformer for recommendation,"
in SIGIR, 2023, pp. 1680–1689.

wang2022causal
W. Wang, X. Lin, F. Feng, X. He, M. Lin, and T.-S. Chua,
"Causal representation learning for out-of-distribution recommendation,"
in WWW, 2022, pp. 3562–3571.

he2022causpref
Y. He, Z. Wang, P. Cui, H. Zou, Y. Zhang, Q. Cui, and Y. Jiang,
"CausPref: Causal preference learning for out-of-distribution recommendation,"
in WWW, 2022, pp. 410–421.

yang2022towards
C. Yang, Q. Wu, Q. Wen, Z. Zhou, L. Sun, and J. Yan,
"Towards out-of-distribution sequential event prediction: A causal treatment,"
NIPS, vol. 35, pp. 22656–22670, 2022.

wang2022invariant
Z. Wang, Y. He, J. Liu, W. Zou, P. S. Yu, and P. Cui,
"Invariant preference learning for general debiasing in recommendation,"
in KDD, 2022, pp. 1969–1978.

zhang2024general
A. Zhang, W. Ma, P. Wei, L. Sheng, and X. Wang,
"General Debiasing for Graph-based Collaborative Filtering via Adversarial Graph Dropout,"
in WWW, 2024, pp. 3864–3875.

wang2024distributionally
B. Wang, J. Chen, C. Li, S. Zhou, Q. Shi, Y. Gao, Y. Feng, C. Chen, and C. Wang,
"Distributionally Robust Graph-based Recommendation System,"
in WWW, 2024, pp. 3777–3788.

ho2020denoising
J. Ho, A. Jain, and P. Abbeel,
"Denoising diffusion probabilistic models,"
NIPS, vol. 33, pp. 6840–6851, 2020.

wang2023learning
Y. Wang, X. Wang, A.-D. Dinh, B. Du, and C. Xu,
"Learning to schedule in diffusion probabilistic models,"
in KDD, 2023, pp. 2478–2488.

wu2022handling
Q. Wu, H. Zhang, J. Yan, and D. Wipf,
"Handling distribution shifts on graphs: An invariance perspective,"
ICLR, 2022.

yuan2024environment
H. Yuan, Q. Sun, X. Fu, Z. Zhang, C. Ji, H. Peng, and J. Li,
"Environment-Aware Dynamic Graph Learning for Out-of-Distribution Generalization,"
NIPS, vol. 36, 2024.

wangDro
B. Wang, J. Chen, C. Li, S. Zhou, Q. Shi, Y. Gao, Y. Feng, C. Chen, and C. Wang,
"Distributionally Robust Graph-based Recommendation System,"
in WWW, 2024.

zhang2024robust
A. Zhang, W. Ma, J. Zheng, X. Wang, and T.-S. Chua,
"Robust collaborative filtering to popularity distribution shift,"
TOIS, vol. 42, no. 1, pp. 1–23, 2024.
cai2023lightgcl
X. Cai, C. Huang, L. Xia, and X. Ren,
"LightGCL: Simple yet effective graph contrastive learning for recommendation,"
arXiv preprint arXiv:2302.08191, 2023.

wang2023causal
W. Wang, X. Lin, L. Wang, F. Feng, Y. Ma, and T.-S. Chua,
"Causal disentangled recommendation against user preference shifts,"
TOIS, vol. 42, no. 1, pp. 1–27, 2023.

zhang2023invariant
A. Zhang, J. Zheng, X. Wang, Y. Yuan, and T.-S. Chua,
"Invariant collaborative filtering to popularity distribution shift,"
in WWW, 2023, pp. 1240–1251.

zhang2024empowering
A. Zhang, L. Sheng, Z. Cai, X. Wang, and T.-S. Chua,
"Empowering Collaborative Filtering with Principled Adversarial Contrastive Loss,"
NIPS, 2024.

zhou2020graph
J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun,
"Graph neural networks: A review of methods and applications,"
AI open, vol. 1, pp. 57–81, 2020, Elsevier.

wu2020comprehensive
Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu,
"A comprehensive survey on graph neural networks,"
TNNLS, vol. 32, no. 1, pp. 4–24, 2020, IEEE.

zheng2022graph
X. Zheng, Y. Wang, Y. Liu, M. Li, M. Zhang, D. Jin, P. S. Yu, and S. Pan,
"Graph neural networks for graphs with heterophily: A survey,"
arXiv preprint arXiv:2202.07082, 2022.

guo2020survey
Q. Guo, F. Zhuang, C. Qin, H. Zhu, X. Xie, H. Xiong, and Q. He,
"A survey on knowledge graph-based recommender systems,"
TKDE, vol. 34, no. 8, pp. 3549–3568, 2020, IEEE.

wang2021graph
S. Wang, L. Hu, Y. Wang, X. He, Q. Z. Sheng, M. A. Orgun, L. Cao, F. Ricci, and P. S. Yu,
"Graph learning based recommender systems: A review,"
arXiv preprint arXiv:2105.06339, 2021.

guo2022attention
M.-H. Guo, T.-X. Xu, J.-J. Liu, Z.-N. Liu, P.-T. Jiang, T.-J. Mu, S.-H. Zhang, R. R. Martin, M.-M. Cheng, and S.-M. Hu,
"Attention mechanisms in computer vision: A survey,"
Computational visual media, vol. 8, no. 3, pp. 331–368, 2022, Springer.

ji2021survey
S. Ji, S. Pan, E. Cambria, P. Marttinen, and P. S. Yu,
"A survey on knowledge graphs: Representation, acquisition, and applications,"
TNNLS, vol. 33, no. 2, pp. 494–514, 2021, IEEE.

jing2023contrastive
M. Jing, Y. Zhu, T. Zang, and K. Wang,
"Contrastive self-supervised learning in recommender systems: A survey,"
TOIS, vol. 42, no. 2, pp. 1–39, 2023.

yu2023self
J. Yu, H. Yin, X. Xia, T. Chen, J. Li, and Z. Huang,
"Self-supervised learning for recommender systems: A survey,"
TKDE, vol. 36, no. 1, pp. 335–355, 2023, IEEE.

creager2021environment
E. Creager, J.-H. Jacobsen, and R. Zemel,
"Environment inference for invariant learning,"
in ML, 2021, pp. 2189–2200, PMLR.

li2022learning
H. Li, Z. Zhang, X. Wang, and W. Zhu,
"Learning invariant graph representations for out-of-distribution generalization,"
NIPS, vol. 35, pp. 11828–11841, 2022.

zhang2024spectral
Z. Zhang, X. Wang, Z. Zhang, Z. Qin, W. Wen, H. Xue, H. Li, and W. Zhu,
"Spectral invariant learning for dynamic graphs under distribution shifts,"
NIPS, vol. 36, 2024.

yang2024generalized
J. Yang, K. Zhou, Y. Li, and Z. Liu,
"Generalized out-of-distribution detection: A survey,"
IJCV, pp. 1–28, 2024, Springer.

liu2021towards
J. Liu, Z. Shen, Y. He, X. Zhang, R. Xu, H. Yu, and P. Cui,
"Towards out-of-distribution generalization: A survey,"
arXiv preprint arXiv:2108.13624, 2021.

hsu2020generalized
Y.-C. Hsu, Y. Shen, H. Jin, and Z. Kira,
"Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data,"
in CVPR, 2020, pp. 10951–10960.

yang2022learning
N. Yang, K. Zeng, Q. Wu, X. Jia, and J. Yan,
"Learning substructure invariance for out-of-distribution molecular representations,"
NIPS, vol. 35, pp. 12964–12978, 2022.

lee2023exploring
S. Lee, J. Jo, and S. J. Hwang,
"Exploring chemical space with score-based out-of-distribution generation,"
in ICML, 2023, pp. 18872–18892, PMLR.

chen2022learning
Y. Chen, Y. Zhang, Y. Bian, H. Yang, K. Ma, B. Xie, T. Liu, B. Han, and J. Cheng,
"Learning causally invariant representations for out-of-distribution generalization on graphs,"
NIPS, vol. 35, pp. 22131–22148, 2022.

wang2023diffusion
W. Wang, Y. Xu, F. Feng, X. Lin, X. He, and T.-S. Chua,
"Diffusion recommender model,"
in SIGIR, 2023, pp. 832–841.

li2023diffurec
Z. Li, A. Sun, and C. Li,
"Diffurec: A diffusion model for sequential recommendation,"
TOIS, vol. 42, no. 3, pp. 1–28, 2023.

ma2024plug
H. Ma, R. Xie, L. Meng, X. Chen, X. Zhang, L. Lin, and Z. Kang,
"Plug-in diffusion model for sequential recommendation,"
in AAAI, vol. 38, no. 8, pp. 8886–8894, 2024.

wu2023diff4rec
Z. Wu, X. Wang, H. Chen, K. Li, Y. Han, L. Sun, and W. Zhu,
"Diff4rec: Sequential recommendation with curriculum-scheduled diffusion augmentation,"
in CIKM, 2023, pp. 9329–9335.

qin2023diffusion
Y. Qin, H. Wu, W. Ju, X. Luo, and M. Zhang,
"A diffusion model for poi recommendation,"
TOIS, vol. 42, no. 2, pp. 1–27, 2023.

jiang2024diffkg
Y. Jiang, Y. Yang, L. Xia, and C. Huang,
"Diffkg: Knowledge graph diffusion model for recommendation,"
in WSDM, 2024, pp. 313–321.

li2024recdiff
Z. Li, L. Xia, and C. Huang,
"RecDiff: Diffusion Model for Social Recommendation,"
arXiv preprint arXiv:2406.01629, 2024.

X.HE2017
 X. He,  L. Liao,  H. Zhang, L, Nie,  X. Hu, and T.S. Chua. "Neural collaborative filtering," in WWW, 2017, pp. 173–182.

Lin2024
X. Lin, W. Wang, J. Zhao, Y. Li, F. Feng, and T.S. Chua. "Temporally and distributionally robust optimization for cold-start recommendation," in AAAI, 2024, pp. 8750-8758.







 Paper Title: Contrasting Deep Learning Models for Direct Respiratory Insufficiency Detection Versus Blood Oxygen Saturation Estimation.
















































1]Marcelo Matheus Gauy[Corresponding author: marcelo.gauy@usp.br]
1]Natália Hitomi Koza
1]Ricardo Mikio Morita
2]Gabriel Rocha Stanzione
3]Arnaldo Cândido Júnior
4]Larissa Cristina Berti
5]Anna Sara Shafferman Levin
5]Ester Cerdeira Sabino
6]Flaviane Romani Fernandes Svartman
1]Marcelo Finger

[1]organization=IME - University of São Paulo,
            addressline=Rua do Matão 1010, 
            city=São Paulo,
            postcode=05508-090, 
            state=São Paulo,
            country=Brazil

[2]organization=Federal Technological University of Parana,
            addressline=Avenida Brasil, 4232, 
            city=Medianeira,
            postcode=85884-000, 
            state=Paraná,
            country=Brazil

[3]organization=IBILCE - São Paulo State University,
            addressline=Rua Cristóvão Colombo 2265, 
            city=São José do Rio Preto,
            postcode=15054-000, 
            state=São Paulo,
            country=Brazil
            
            
[4]organization=São Paulo State University,
            addressline=Av. Hygino Muzzi Filho, 737, 
            city=Marília,
            postcode=17525-900, 
            state=São Paulo,
            country=Brazil


[5]organization=FMUSP - University of São Paulo,
            addressline=Av. Dr. Arnaldo, 455, 
            city=São Paulo,
            postcode=01246-903, 
            state=São Paulo,
            country=Brazil

[6]organization=FFLCH - University of São Paulo,
            addressline=Rua do Lago, 717, 
            city=São Paulo,
            postcode=05508-080, 
            state=São Paulo,
            country=Brazil


§ ABSTRACT 


We contrast high effectiveness of state of the art deep learning architectures designed for general audio classification tasks, refined for respiratory insufficiency (RI) detection and blood oxygen saturation (SpO2) estimation and classification through automated audio analysis. 
Recently, multiple deep learning architectures have been proposed to detect RI in COVID patients through audio analysis, achieving accuracy above 95% and F1-score above 0.93.
RI is a condition associated with low SpO2 levels, commonly defined as the threshold SpO2 <92%. While SpO2 serves as a crucial determinant of RI, a medical doctor's diagnosis typically relies on multiple factors. These include respiratory frequency, heart rate, SpO2 levels, among others. 
Here we study pretrained audio neural networks (CNN6, CNN10 and CNN14) and the Masked Autoencoder (Audio-MAE) for RI detection, where these models achieve near perfect accuracy, surpassing previous results. Yet, for the regression task of estimating SpO2 levels, the models achieve root mean square error values exceeding the accepted clinical range of 3.5% for finger oximeters. Additionally, Pearson correlation coefficients fail to surpass 0.3. As deep learning models perform better in classification than regression, we transform SpO2-regression into a SpO2-threshold binary classification problem, with a threshold of 92%. However, this task still yields an F1-score below 0.65. Thus, audio analysis offers valuable insights into a patient's RI status, but does not provide accurate information about actual SpO2 levels, indicating a separation of domains in which voice and speech biomarkers may and may not be useful in medical diagnostics under current technologies. 













    < g r a p h i c s >






  * We improve literature results on audio-based respiratory insufficiency detection.

  * Blood oxygen saturation estimation from audio is hard under current technologies.

  * We analyze features making the former effective but not the latter.




Respiratory Insufficiency detection Blood Oxygen Saturation Estimation Deep Neural Networks Voice features Speech features.















§ INTRODUCTION



Respiratory insufficiency (RI) is a condition commonly associated with low blood oxygen saturation levels (SpO2), which can be broadly defined as the impairment of respiratory gas exchange between the ambient air and circulating blood <cit.>. Interest in its automated detection has intensified during the pandemic, as it is a COVID-19 associated condition that leads to hospitalization <cit.>.  However, for the purposes of this study, RI is defined as blood oxygen saturation level below a certain threshold, most usually  SpO2 <92% <cit.>. 

It's important to acknowledge that there is no universally definitive gold standard for diagnosing respiratory insufficiency. Medical practitioners typically integrate various factors into their diagnostic process, including respiratory frequency, heart rate, and SpO2 levels, among others. Consequently, in clinical practice, scenarios arise where patients may exhibit SpO2 levels potentially above the threshold, yet other clinical indicators point towards a positive diagnosis of RI. Conversely, instances occur where SpO2 levels fall below the threshold, but other clinical factors do not support a diagnosis of RI. An extreme example of the latter could be measuring the SpO2 of a swimmer immediately following an extensive swimming exercise.


Recent works analyze audios of COVID-19 induced RI patients <cit.> as well as general RI <cit.> and determined that deep learning architectures — such as Convolutional Neural Networks, or CNNs, and Transformers based architectures — are very effective at distinguishing audio recordings of RI patients from non-RI control voices. The most effective architectures achieve accuracy between 95% and 97.5% on either COVID-19 induced RI dataset or general RI dataset <cit.>. Factors believed to be important in the decision process of those models are altered speech pause distribution in patients <cit.> as well as signal energy levels throughout speech <cit.> and altered F_0 related parameters <cit.>. These analyses have been carried out as part of the SPIRA project <cit.>, which aims to provide cheap and mobile artificial intelligence (AI) tools for the triage of RI patients via audio analysis.


In this paper, we study current state of the art (SOTA) audio classification models and their proficiency for detecting RI. The models studied are the Pretrained Audio Neural Networks (PANNs <cit.>) in addition to a novel unsupervised pretrained model called Masked Autoencoder (Audio-MAE <cit.>). These models are pretrained on a 5000 hours dataset of Youtube videos called AudioSet <cit.>, and are known to be extremely effective at a large array of audio classification tasks <cit.>. Moreover, while RI detection has been extensively studied, no previous works have analyzed whether it is possible to directly estimate SpO2 (a key defining characteristic of RI) from voice  and speech[We distinguish between the analysis of voice, namely acoustic properties of human utterances, from speech, which involves natural language emissions.]. Thus, we investigate whether the previously mentioned SOTA audio classification models can be used for estimating patient SpO2 solely from voice and speech audios. The architectures considered are extremely effective at RI detection, and surpass previous works, with the best model for RI detection being the Audio-MAE, achieving near perfect accuracy (99.9%). However, despite the significant improvement in RI detection, as well as their widespread effectiveness at multiple audio classification tasks, the SOTA models studied are not capable of attaining good performance when estimating SpO2. The 4 networks are used in a regression task of estimating patient SpO2 from a patient's voice and speech audios. Moreover, as a regression task may be challenging for the networks due to the relatively small size of the patient dataset, we also use them in a quantized classification task performed over classes of low SpO2 and high SpO2 for a given threshold, e.g. 92%.


For the regression task, no model achieves Pearson correlation between the oximeter SpO2 values and the predicted SpO2 values above 0.3, with the mean absolute error being around 4.4% for the best model [It is known that oximeter error, depending on the instrument, varies between 1% and 2%.].
For the classification task, all models display F1-score below 0.65. Note that this is significantly lower than the result obtained for RI detection, as the F1-score would also be above 0.99.


Thus, this paper presents the following contributions:


  * Audio-MAE and the PANNs improve the RI detection accuracy to near perfection, significantly surpassing previous models.

  * Despite that, none of the studied models is capable of estimating SpO2 accurately through voice and speech. This showcases the potential limits of using both as biomarkers.

  * We present a list of reasons why models might be able to detect RI but struggle with SpO2.


















§ RELATED WORK



This work is developed as part of the SPIRA project <cit.>. Previous works within that context have built effective models for the detection via audio analysis (voice and speech) of both COVID-19 induced RI <cit.>, and general RI <cit.>, where the detection is aimed at Brazilian Portuguese speech. Variants of some of those models were also proposed for the detection of COVID-19 through voice <cit.>. In addition to the development of deep learning models, the SPIRA project also aims at studying the acoustic properties of RI compromised speech, that is, it aims at investigating the characteristics present in voice and speech which support those models to make their classification, thus aiming to generate explanations for its decisions. It has been found that RI patients have altered speech pause distribution <cit.>, different F_0 related parameters <cit.> as well as distinct signal energy throughout speech <cit.>. Thus, previous works from the SPIRA project focused on analyzing RI and its effects on voice and speech directly, either through building deep learning models or through determining explainable features. No previous analysis from the project focused on analyzing SpO2 levels directly and estimating those from voice and speech.


Outside the SPIRA project, we have not found works studying the relations between RI and voice or speech. However, there are initiatives which try to detect COVID-19 from voice or cough <cit.>, as well as recent works studying COVID-19 disease progression via longitudinal cough, breathing and voice data <cit.>.  We must also mention that <cit.> have recently proposed employing acoustic analysis for the detection of type-2 diabetes, but further studies seem  necessary to validate this approach.

While no previous works studying the problem of determining SpO2 by means of voice and speech analysis have been found, it is common to use machine learning techniques to convert the raw PPG signal from an oximeter into the SpO2 values <cit.>. This process is reported to have a mean absolute error below 1% and an above 95% accuracy for a ± 2% error band for SpO2 values in the range 81 to 100.




§ METHODS




 §.§ SPIRA Dataset



Here, we use the original dataset from <cit.> for RI detection and an extended version of the corresponding patient dataset for SpO2 estimation. The original dataset consisted of 566 patient audios (and about 6000 control audios). These audios mainly consisted in patient's uttering a predetermined sentence which they would read from a prompt on a cellphone [A sentence was designed by linguists to contain simple but large words, as well as no obvious speech pauses during its utterance, leading to: “O amor ao próximo ajuda a enfrentar o coronavírus com a força que a gente precisa”/“Love of neighbor helps in strengthening the fight against Coronavirus.”]. These patient voices were collected in COVID-19 wards in Brazil during the peak of the COVID-19 pandemic. The control audios were collected from voluntary donors with an app over the internet. Collection was absolutely anonymous, so no one knows who the patients and controls were, and no ethnographic information is available. More than 600 patient audios were collected, but a number of distortions, most commonly the whispering of collectors being heard in the audios, led to several audios being discarded, leading to a 566 item dataset <cit.>. 

In this work, we use the exact same dataset as <cit.> with the same division in training, validation and test sets proposed there. This is to allow comparisons between our models and previous studies. For SpO2 estimation, we use the patient audios (as they are the ones with SpO2 available) and, in this case, do not need to contrast them with the control audios (collected in another environment). We can use the full original patient dataset for SpO2 estimation. In total, we have 566 patient audios with their respective SpO2 measured with an oximeter at the time of voice recording. Alternatively, we could have used the more general RI dataset provided by <cit.>, which includes SpO2 data for both patients and controls. However, that dataset contains less than 200 files and is too small to use on its own in a regression task. In addition, it may not be a good idea to combine both datasets as the voice features from COVID-19 RI and general RI are quite different <cit.>. As such, we found it better to use the COVID-19 induced RI audios we had available. As it might still be possible to identify patient's by their voice, and as healthcare datasets are generally confidential, we prefer to not publish the entirety of our patient dataset. Note that the dataset from <cit.> is found on https://github.com/SPIRA-COVID19/SPIRA-ACL2021/tree/masterGithub.




In the full 566 audios dataset, we have 307 men, of which 103 have SpO2 below or equal to 92%, and 259 women, of which 82 have SpO2 below or equal to 92%. We perform a random split of the dataset in training, validation and test sets in the following way: first, among the 103 men and 82 women with SpO2 below or equal to 92%, randomly select 8 men and 8 women for the validation set and 16 men and 16 women for the test set, the remaining files with SpO2 below or equal to 92% go to the training set; second, among the 204 men and 177 women with SpO2 above 92%, randomly select 16 men and 16 women for the validation set and 32 men and 32 women for the test set, with the remaining files with SpO2 above 92% going to the training set. This is to ensure that our validation and test sets are balanced by sex and have similar proportions of low and high SpO2 present between the three dataset parts.



 §.§ RI detection task
 

As mentioned previously, we use the exact same training, validation and test sets division as the original work <cit.>. In the original dataset, patients who suffered from RI were collected in 
COVID-19 hospital wards and controls were collected via a web app. The different collection environments require that the audios are properly preprocessed so as to avoid overfitting to the different noise sources <cit.>. We perform the same preprocessing steps for RI detection as suggested by previous works <cit.>. Our models receive spectrograms as opposed to MFCC-grams as was common in past studies. This is due to their effective pretraining on AudioSet, which, through transfer learning, allows the models to perform extremely well despite using spectrograms (which were found to be typically less effective than MFCC-grams in previous works).



 §.§ SpO2 Estimation tasks


We perform two types of SpO2 estimation tasks. The first is a regression task where the models predict a value between 0 and 100 that should match as closely as possible the SpO2 level of the corresponding patient. We use the MSE (mean squared error) loss for gradient computation, though we also made preliminary experiments with the MAE (mean absolute error) loss obtaining similar results for the models we tested. The second task is a classification one where the models are asked to classify whether the audios come from a patient with SpO2 above 92% or not. This is a binary classification task that is likely to be much easier for the models than performing a complete SpO2 regression and also likely to be much less data intensive, which is important as healthcare datasets are usually small in size. This allows us to check whether the apparent difficulty from the regression task did not come solely from the size of the dataset. The loss used is a binary cross entropy loss, as is standard for binary classification.


Lastly, we have observed, especially for regression, that there is some reasonably large performance difference (measured in terms of the Pearson coefficient for regression) among different training, validation and test splits. This did not seem significant in our preliminary experiments for the classification task but did seem significant for regression. To more accurately represent the performance of the models, we repeat each experiment on each model 10 times with the added caveat that each experiment gets a different (random) training, validation, test set split (as proposed in Section <ref>). Thus, our average performance (say, Pearson coefficient for regression or F1-score for classification) on the test set more closely resembles the average one from a random test set and not the particular test set for any of the experiments. Alternatively, we could have used cross-validation to avoid the problem of selecting the test set, but that would often require changing the internal logic of the data feeding for each model as the windowing augmentation technique we use needs to be performed after the fold division (see Section <ref> for more details). As a result, we chose to just resample the test set with each experiment and get an average result.



 §.§ Model Architectures



In general, the models described here were used exactly as originally proposed, with the exception of the last layer, which consists of 1 unit for regression and 2 units for classification (see Figure <ref> for the general classification architecture). In the case of regression, we add an extra 10 units fully-connected feed-forward (FC) layer before the final layer to facilitate the information extraction. See Figure <ref> for the general SpO2 regression architecture. We find that applying dropout at the intermediary FC layer is not helpful (so increasing its size and regularizing with dropout does not lead to improved results). The codes used are the same as the original models with the few changes mentioned in Figures <ref> and <ref>.














Masked Autoencoder (Audio MAE)
This model was proposed in <cit.>. Similar variants of this approach can be found in <cit.>. Masked Autoencoders were originally proposed for image processing <cit.> and later found to also perform extremely well in audio classification tasks <cit.>. The input to the Masked Autoencoder is typically the spectrogram of the audio. The key idea here is to do masked reconstruction of the frames, an idea inspired by the pretraining technique of <cit.> for NLP. The authors propose to erase a large proportion of the audios (80% in this case). Additionally, the authors do not erase frames per se but typically 16× 16 blocks of frames and channels, borrowing from Vision Transformers <cit.> method of feeding the spectrogram in blocks as tokens to the Transformers. The models are (unsupervisedly) pretrained to reconstruct this large share of masked blocks back into the original spectrogram. In order to deal with a large masking proportion, the authors make use of a standard Transformer both for encoding and decoding. The encoder is a vanilla 12-layer Vision Transformers <cit.>. The decoder is a 16-layer Transformers with shifted local attention [This is a variant of local attention which shifts the attention windows from one layer to the next. The reason for doing this instead of standard global attention, as is typically used for images, is that the relevant information in audio spectrograms is predominantly local.].

We use the pretrained Audio-MAE model on AudioSet from <cit.>. AudioSet is a 5000 hour dataset of Youtube audios distributed in 527 classes. In principle, one could consider the version of this model also finetuned (that is, trained with supervision) on the AudioSet. However, as we deal with a regression task, we did not expect there to be a significant advantage from doing this type of finetuning on top of the pretraining [Moreover, the Pretrained Audio Neural Networks described below were trained on AudioSet with supervision, so we already explore whether that has some inherent advantage.]. As such, we have settled for the model that received unsupervised pretraining on AudioSet.
Lastly, in our experiments with the COVID-19 induced RI detection task (dataset from <cit.>), the accuracy of the model is above 99.9% and even in the harder general RI detection task (dataset from <cit.>) the accuracy is above 98%, without employing any changes to Audio-MAE. This showcases the effectiveness of the pretraining technique used as previous models had lower accuracy and typically only achieved it with MFCC-gram as the input.


Pretrained Audio Neural Networks - PANNs
We make use of 3 CNN-based PANNs from <cit.>, namely CNN6, CNN10 and CNN14. These are pretrained (with supervision) on the AudioSet dataset to classify the audios among the 527 classes. They have been effectively used for multiple audio based tasks, such as, audio set tagging <cit.>, speech emotion recognition <cit.>, automated audio captioning <cit.> 
and COVID-19 detection <cit.>. These 3 PANNs are similar in architecture but have varying levels of complexity. We find this useful as it gives us an idea on how hindered our models are by the size of our dataset.

CNN6 is a 6 layer CNN; CNN10 has 10 layers; lastly CNN14 has 14 layers. They have convolutional layers, with kernel 5× 5 for CNN6, and kernel 3× 3 for CNN10 and CNN14. Each such layer is followed by batch normalization <cit.> and ReLu nonlinearity <cit.>, for faster and more stable training convergence <cit.>. These layers are present 4 times in CNN6 and, in between two layers, an average pooling 2× 2 (AVG) is used, which works better than max pooling <cit.>. For CNN10 and CNN14, the convolutional layers appear in pairs before AVG is applied. CNN10 contains 4 pairs of such blocks, for a total of 8 layers, and CNN14 contains 6 pairs of such blocks, for a total of 12 layers. As for the number of kernels in each block pair, they start at 64 for the first pair and double for each subsequent pair. In the last block pair, instead of AVG, we apply Global pooling (sum of average and max pooling). All networks have a penultimate FC layer to increase representation ability, containing 512 units on CNN6 and CNN10 and 2048 units on CNN14; and a final 527 unit FC sigmoid layer, to obtain the class probabilities. To prevent overfitting, dropout is applied after every downsampling operation. All networks take as input a sequence of frames of log mel Spectrogram with 64 mel bins each. 

We use the pretrained models from <cit.>. These were pretrained to classify the AudioSet audios in 527 classes. 
Finally, the 3 pretrained PANNs are also superior to the MFCC Transformers on both standard RI detection tasks (COVID-19 induced RI and general RI), so they are known to be effective for RI detection <cit.>.




 §.§ Preprocessing



We perform the same windowing data augmentation technique used in <cit.>. Namely, our audios are split in 4 second windows with 1 second hop. That is, a 5 second audio is split in two, one from seconds 0 to 4 and the other from seconds 1 to 5. This is a simple data augmentation that increases the amount of audios we have available. Moreover, as Audio-MAE uses all-to-all attention, decreasing the length of the audios leads it to be less computationally and memory intensive. Furthermore, there may be some correlation between lower SpO2 levels and audio length, as patients with lower SpO2 may have more difficulty breathing (as was observed in previous works when contrasting healthy controls and RI patients <cit.>), and we want to prevent the model from excessively concentrating on the audio lengths. Lastly, observe that windowing needs to be performed after the training, validation and test set split as otherwise we would risk biasing our results as the validation and test sets would potentially share a lot of audio parts with the training set.


After the windowing step is performed, we resample the audio waveforms with a 16kHz sample rate [Performance difference by resampling the audios is minimal.] and convert them into either a spectrogram, via a Fast Fourier transform (FFT) <cit.>, feeding them to the models as an image. The parameters for conversion (such as, hop length, window length, number of mel coefficients and size of FFT) vary with the model as we are typically using pretrained versions of models, which were pretrained with a particular set of parameters. Apart from being required to use the same parameters when extracting the spectrogram as the original pretrained models, we do not expect there to be major differences in performance due to the particular choice of those parameters as long as they are within reason.

Specifically for the COVID-19 RI detection task, one needs to add hospital ward noises to both patient and control audios. This is the same process as in previous works <cit.> and is a necessary step to avoid overfitting to such noises. This step is after windowing, but naturally before spectrogram transformation.




§ RESULTS





 §.§ COVID-19 RI detection task


We performed an experiment to evaluate the accuracy of the 4 SOTA models studied here in the COVID-19 RI detection task. We also briefly report the same results for the general RI detection task using the dataset from <cit.>. We use the binary cross entropy loss to train the 4 models described in Section <ref> for COVID-19 RI detection on the training set from <cit.>, performing early stopping according to the best accuracy measured on the validation set at the end of each epoch. All models were run with a batch size of 16, with Adam optimizer <cit.> with the same parameters as the original works for the SOTA models. The learning rate for the PANNs was 0.0001 without any scheduler and was trained, with early stopping, for 100 epochs. Audio-MAE used the original learning rate with the original Noam scheduler and was trained, with early stopping, for 40 epochs. Experiments with PANNs were repeated 5 times, while the Audio-MAE experiment was repeated 10 times to obtain an average behavior. We only report accuracy as the dataset is balanced between patients and controls as well as sex. We also report the accuracy on the dataset from <cit.>. Since that is a smaller dataset, the Audio-MAE performance has larger variance, which is probably a consequence of some overfitting in specific runs. Perhaps this could be avoided with hyperparameter optimization but it is nevertheless a complex model on a tiny dataset. The results can be found on Table <ref>.







 §.§ SpO2 estimation analysis


We perform two experiments to showcase that voice and speech features do not contain significant information regarding a patient's SpO2 level. The first experiment is a direct SpO2 regression task, where models are tasked with predicting the SpO2 level of a patient from audio samples. We use the MSE loss to train the 4 models described in Section <ref> on the training set, performing early stopping according to the best MSE loss measured on the validation set at the end of each epoch. We evaluate the models according to 4 metrics: root mean squared error (RMSE), MAE, R^2 and Pearson correlation. As we observe large variance in performance depending on the run and dataset split, we perform 10 random dataset splits into training, validation and test sets and perform train-test cycles 10 times. All models were run with a batch size of 16. MOdels were trained with AdamW optimizer <cit.>, under default weight decay of 0.01, except for the Audio-MAE, which used the original value of 0.0005.
PANNs used a learning rate of 0.0001 without any scheduler. They were trained for 10 epochs. The Audio-MAE used the original initial learning rate with the original Noam scheduler parameters and was trained for 20 epochs.

The 4 models used the originally proposed hyperparameter sets and we then performed a search around those, going one by one and searching for the best value while fixing all others. No significant advantage was observed by varying parameters.

Table <ref> shows the results of each model according to the 4 metrics. The average across 10 experiments with the sample standard deviation is depicted. No model achieved Pearson above 0.3 and R^2 above 0.1 on average, showcasing the difficulty of the task and the seemingly little information on the SpO2 level available on a patient's voice. Models typically stayed in the range of 0.22 and 0.26 for the average Pearson and, considering the reported sample standard deviation, they can all be considered almost equivalent to one another, except for CNN14.





As the previous regression task may have been too hard for the models given the amount of data we had available for training, we proposed a second experiment to support our result that the voice of patient's does not contain much information on their SpO2 level. For the second experiment we reduce the task of estimating SpO2 levels to a simple binary classification task: whether the SpO2 of a patient is above a given threshold or not. We have chosen the threshold at 92% as that is commonly the defining SpO2 level for an RI patient. With this threshold, we can see the task as distinguishing between more critical RI patients (those still following the textbook definition for RI based on SpO2) and less critical ones, where criticality depends solely on the SpO2 level. Note that considering only SpO2 is likely ill-advised when defining criticality as the data comes from COVID-19 wards. However, this task does allow us to measure whether voice features contain information on the SpO2 of a patient, or whether the information they contain refers more generally to a patient's condition and has little influence from the SpO2.

We use a binary cross-entropy loss to train the 4 models described in Section <ref> on the training set and perform early stopping according to the best loss measured on the validation set at the end of each epoch.
We evaluate the models according to their classification accuracy and more importantly their F1-score. The F1-score is effectively the correct measure to use as the number of patients with SpO2 above 92 is much larger than the number of patients with SpO2 below or equal to 92.

As in the regression task, the variance is quite large among experiments. Though the influence of the dataset split is smaller, for symmetry with the regression task, we consider the same 10 dataset splits into training, validation and test sets and perform 10 experiments on each model, one on each dataset split. This gives us an average performance of the models for a random test set. All models were run with a batch size of 16. All models used Adam as the optimizer with the default weight decay of 0.01, with the exception of the Audio-MAE which used AdamW with a weight decay of 0.0005 as was the case in the original paper.
The PANNs used a fixed learning rate of 0.0001 as was the case for regression and were trained for 10 epochs. The Audio-MAE used the original learning rate and original Noam scheduler parameter and was trained for 40 epochs. For the second task it was not helpful to have an intermediary layer in the models. Moreover, changing dropout in the inner layers of the pretrained models is likely ill-advised as it would require pretraining to be done again. We have attempted, without success, to vary the weight decay parameter to increase L_2 regularization. It does not seem that one can improve the results substantially by simply exploring the parameter space and to get better results, one would likely need substantially more data. Even then, it seems unlikely that this task can ever reach as high accuracy and F1-score as the RI detection task.

Table <ref> shows the results of each model according to accuracy and F1-score. The average across 10 experiments is shown with the sample standard deviation. No model achieved an F1-score above 0.65 on average, which is considerably lower than the above-mentioned more than 0.97 F1-score achieved by these models on the RI detection task. Due to the overlapping intervals given by mean standard deviation, models based on Masked Autoencoders and the 3 PANNs can be considered almost equivalent in performance.







§ DISCUSSION



With the three experiments, we show that it is harder to estimate SpO2 levels based solely on a patient's voice, in comparison to RI detection. The chosen models are close to state of the art for a large array of audio classification tasks (see <cit.>). Moreover, all 4 models are known to be very effective at the related RI detection task. We observe that we have attempted, without success, to also use the original models for RI detection from <cit.> for SpO2 estimation. This suggests that a patient's voice and speech has plenty of features pertaining to whether one suffers from RI, even though it has little information on the patient's SpO2 at the time of collection; so important information is lost in the SpO2 estimation process which is nonetheless used in the classification process. We also hypothesize that the patient's treatment in the hospitals leads to an improvement of SpO2 levels (which seemingly pushes them out of the RI range), but their voice and speech retains multiple traces found in RI patients, so that the models cannot distinguish lower SpO2 from higher SpO2 in the investigated models.

Note that for the regression task, the obtained levels of RMSE for all the models were above the clinical level of 3.5 <cit.>. The Pearson correlation coefficient is below 0.3, as well as the R^2 being below 0.1 imply that there is little correlation between the predicted SpO2 and the oximeter SpO2, making the models very poor SpO2 predictors. For the classification task, the F1-score being below 0.65 is a strong indication of the difficulty of the task and specially when contrasted with the high (larger than 0.98) F1-score attained in the RI detection task. Thus, there are multiple features present in voice and speech which enable RI detection but cannot be used for estimating SpO2, even if the task is as simple as identifying SpO2 above or below 92%.
Such features are, potentially, the patient's more numerous pauses at unexpected locations hypothesized in <cit.>. Another potential feature would be different F_0 related parameters <cit.> as well as different energy levels throughout speech <cit.>. Note that none of these features observed to be relevant for RI detection can correlate strongly to SpO2 levels (as the networks would learn such simple features) even if they correlate with RI. Much like medical doctors are aware of the fact that SpO2 cannot be used as the sole determining factor of RI, the same holds for deep learning models.

In terms of criticism, our work could be contested on the fact that we did not use Wav2Vec <cit.> type networks. However, Wav2Vec main advantages lie on transcription tasks, and Masked Autoencoders typically outperform them for standard classification tasks <cit.>. Another important criticism is that our data consists solely of hospitalized COVID-19 patients. While it is possible that the results would change for other respiratory conditions, we currently possess no means of verifying that hypothesis as the only dataset available with SpO2 values for multiple types of diseases is too small in size <cit.> (the total number of cases with SpO2 below 92 is less than 10).
Lastly, it could be said that our dataset is just too small for the networks to learn to identify SpO2. While the dataset leans on the small size for complex models, even the simpler models (e.g. CNN6 and CNN10) failed to extract much meaningful information, while all these models could identify RI with a similar dataset.



§ CONCLUSIONS


We analyzed the performance of 4 models, which we show to be superior to previous state of the art RI detection through voice. The best model, Audio-MAE, achieves near perfect accuracy on COVID-19 RI detection (99.9%). In terms of general respiratory insufficiency detection (dataset from <cit.>), all 4 models achieve accuracy above 97%. In addition, as these models are known to perform extremely well for a large array of audio classification tasks, we have decided to analyze their performance on two SpO2 estimation tasks from voice.

The first task is SpO2 regression from voice, where the best RMSE is above 3.5 which is the standard for clinical use of SpO2. Even more crucially, Pearson correlation between oximeter SpO2 values and even the best model SpO2 estimate does not reach 0.3. Then we considered a binary classification task for predicting from voice whether the SpO2 is above 92% or not. None of the considered models achieve an F1-score above 0.65. This is in sharp contrast with RI detection, be it COVID-19 induced or general, where all models achieve F1-score above 0.97. We can conclude that voice and speech features are strong indicators of RI presence (such as the patient's pause distribution <cit.>), despite not containing much information about the SpO2. As a possibility, we hypothesize that the treatment patients received in the hospitals led to temporary improvement in their SpO2 levels, bringing those closer to healthy levels, but their voice and speech still retains traces found in RI patients. As a result, models can pick up those traces to identify RI but they are not very informative when it comes to the actual SpO2 level.

Models perform better for end-to-end classification than for regression. As SpO2 regression is ineffective, any attempts to do RI classification intermediated by regression are also ineffective.  Lastly, some of the pretrained models using supervised learning are only pretrained for classification tasks, which may lead to poorer performance in downstream regression tasks. While some models use generic pretraining tasks, our experience, as well as that of other researchers, shows that current self-supervision techniques do not translate well for regression. As Transformers are known to perform well in time series forecasting tasks <cit.> which involve regression, it seems that current self-supervision techniques do not capture dynamic information on how the data space changes and how those changes might affect downstream tasks. If it is possible for the models to learn such information, they might become more effective for regression tasks, which could improve our results for that case. 
Keep in mind, though, that the binary classification task used shows that SpO2 estimation from voice is inherently hard. As a consequence of the binary classification task performance, our results show the limits of using voice and speech as biomarkers.




§ ACKNOWLEDGEMENTS

This work was supported by FAPESP grants 2022/16374-6 (POSTDOC)  and 2023/00488-5 (SPIRA), and by Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001. Carried out at the Center for Artificial Intelligence (C4AI-USP), supported by FAPESP grant 2019/07665-4 and by the IBM Corporation. The sponsor's involvement in the work was limited to the provided funding. Marcelo Finger was partly supported by CNPq grant PQ 302963/2022-7.

















elsarticle-num 




 Paper Title: Automated Software Vulnerability Static Code Analysis Using Generative Pre-Trained Transformer Models.



Extracting doubly-excited state lifetimes in helium directly in the time domain with attosecond noncollinear four-wave-mixing spectroscopy
    Stephen R. Leone
    August 5, 2024
==========================================================================================================================================



§ ABSTRACT 
Generative Pre-Trained Transformer models have been shown to be surprisingly effective at a variety of natural language processing tasks – including generating computer code. However, in general GPT models have been shown to not be incredibly effective at handling specific computational tasks (such as evaluating mathematical functions). 
In this study, we evaluate the effectiveness of open source GPT models, with no fine-tuning, and with context introduced by the langchain and localGPT Large Language Model (LLM) framework, for the task of automatic identification of the presence of vulnerable code syntax (specifically targeting C and C++ source code). This task is evaluated on a selection of 36 source code examples from the NIST SARD dataset, which are specifically curated to not contain natural English that indicates the presence, or lack thereof, of a particular vulnerability (including the removal of all source code comments). The NIST SARD source code dataset contains identified vulnerable lines of source code that are examples of one out of the 839 distinct Common Weakness Enumerations (CWE), allowing for exact quantification of the GPT output classification error rate. A total of 5 GPT models are evaluated, using 10 different inference temperatures and 100 repetitions at each setting, resulting in 5,000 GPT queries per vulnerable source code analyzed. 
Ultimately, we find that the open source GPT models that we evaluated are not suitable for fully automated vulnerability scanning because the false positive and false negative rates are too high to likely be useful in practice. However, we do find that the GPT models perform surprisingly well at automated vulnerability detection for some of the test cases, in particular surpassing random sampling (for some GPT models and inference temperatures), and being able to identify the exact lines of code that are vulnerable albeit at a low success rate. The best performing GPT model result found was Llama-2-70b-chat-hf with inference temperature of 0.1 applied to NIST SARD test case 149165 (which is an example of a buffer overflow vulnerability), which had a binary classification recall score of 1.0 and a precision of 1.0 for correctly and uniquely identifying the vulnerable line of code and the correct CWE number. 
Additionally, the GPT models are able to, with a rate quantifiably better than random sampling, identify the specific line of source that contains the identified CWE for many of the NIST SARD test cases. 









§ INTRODUCTION



Generative Pre-Trained Transformer (GPT) models, built using the attention architecture <cit.>, have been shown to be surprisingly effective at a variety of information processing tasks including computer code generation and natural language text summarization <cit.>. GPT models are intended to work very well with language, and are typically referred to as a class of Large Language Model (LLM). Notably however, GPT models can, in certain settings, also perform algorithmic tasks such as learning the greatest greatest common divisor algorithm <cit.>. The surprising capability of GPT models to perform well at tasks involving computer code leads to the natural question of whether GPT models could be used for determine security properties of static software samples. In this study, we examine whether current open source GPT models can be used to correctly identify software vulnerabilities and weaknesses in source code correctly. 


The evaluation of our study is based on a set of 36 labeled examples from the highly credible NIST SARD dataset <cit.>. Each source code example in this dataset is meticulously mapped to the Common Weakness Enumeration (CWE) framework <cit.>, providing a solid foundation for our research on software vulnerability detection. The Common Weakness Enumeration (CWE) is a categorization of known types of software weaknesses, hardware weaknesses, and software vulnerabilities. 



Static code vulnerability scanners are an important field of study because they speed up the process of identifying and thus patching software and even potentially hardware vulnerabilities and flaws that cause security problems in computer systems. There are a wide variety of static code analysis scanners that have been developed, usually focusing on specific domains or specific languages <cit.>. A challenge with static vulnerability analysis is that the outputs typically have high false positive rates because the tools usually use string pattern matching to identify known syntax representing an instance of a specific code flaw. The main advantage of static code analysis is that it is much faster and easier to apply than dynamic program analysis. 



Ref. <cit.> studies the task of repairing CWE examples using GPT models. Ref. <cit.> examined source code vulnerability detection using several code language models (namely BERT variants) on a large dataset of software examples. 


Our study introduces novel methods compared to existing research, particularly in the use of highly curated datasets that include the exact vulnerable line of source code. The structured GPT prompt we employ with these GPT models enables the automated extraction of vulnerability detection data from the output, facilitating large-scale, repeatable, accuracy analysis. This dataset combined with structured JSON prompting and parsing allows us to fully leverage the capabilities of GPT models for automated vulnerability detection, requiring no manual processing of the GPT output. In total, we generate and process 100 · 10 · 36 · 5 = 180000 discrete GPT text outputs (each of these being a separate, independent, and complete text inference computations. All of the GPT model computations are performed locally, on a local GPU compute server, using open source GPT models. 












§ METHODS



The five open-source GPT models used in this study are summarized in Table <ref>. These GPT models were selected for their general performance among available GPT models - and all of these models can generate and parse computer code. Although a large number of different GPT models could be utilized for this study - due to the high computation time required to perform these experiments, a small number of GPT models were selected. These GPT models are from the huggingface GPT repository <cit.> and the GPT models are run using the Python3 modules  <cit.> and , using a local compute server that has four Nvidia A100 GPU's <cit.> (each with 82 Gigabytes of memory), and CUDA Version 12.4. 


GPT models can be used to predict the most likely token, given the previous N tokens (where N is the context window for current GPT models on the order of 4096-32768). When used in this manner, the text produced by the GPT models is deterministic, given that the previous tokens are the same. However, GPT models can learn and generate probability distributions of the most likely subsequent tokens - and we can add stochasticity in the text generation process by sampling from that distribution, weighted by their respective probabilities. The parameter that controls this sampling is known as temperature. This stochastic text generation is part of what makes generative machine learning models able to produce a wide variety of outputs. In the context of investigating the usage of these models as code-scanning tools, these models will only perform somewhat well at this task. Moreover, we need outputs to vary somewhat to determine the possible range of characteristics of the output since the text generation process of GPT models can be stochastic. Therefore, we vary the text generation inference temperature from 0.1 to 1.0 in increments of 0.1. The huggingface <cit.> python 3 module used to execute this GPT model inference allows temperatures up to 100, where 0 denotes deterministic output (with no sampling), and 100 denotes closer to uniform token sampling. We choose relatively low temperatures because we need the output to conform to parsable data structures. Critically, because the outputs vary significantly, we view this task as a stochastic sampling problem, where we characterize the proportion of samples with specific required properties (the most basic being whether the data could be automatically parsed). 100 samples (GPT outputs) are generated for each inference temperature, GPT model, and source code test case. 










To ground the GPT model output in the definitions of the Common Weakness Enumeration (CWE) framework, we utilize a technique known as retrieval augmented generation <cit.>. Specifically, we use a codebase known as  <cit.> that uses langchain <cit.> to augment the capabilities of GPT models by using additional word embeddings of reference documents (not as part of the model pre-training). Specifically, the source documents are split into character maximum chunk sizes of 1000, with chunk overlap of 200 characters. 

The localGPT prompt template is a combination of context and then the user prompt question. No chat history was used, meaning that each text generation call is independent. The prompt construction uses a system prompt <cit.> followed subsequently by the context of the generated word embeddings and then the user prompt. The system prompt used in this study, which is from a version of the localGPT codebase <cit.>, is given below:



You are a helpful assistant, you will use the provided context to answer user questions.
Read the given context before answering questions and think step by step. If you can not answer a user question based on the provided context, inform the user. Do not use any other information for answering user. Provide a detailed answer to the question.



The reference documents that are used to provide context for the GPT model inference calls are the following: one text file for each of the 839 CWEs that are currently defined, where the text file contains the CWE number, the name of the software weakness, and the complete description of the CWE from Mitre [<https://cwe.mitre.org>]. Next is a text file with a brief definition of the CWE framework. And lastly, two books (PDF files of the books) on software, security, and vulnerabilities (specifically for the C family languages) <cit.>. 



Table <ref> outlines the exact code test cases that are analyzed in this study. Note that NIST SARD test case 148871 is the vulnerable code that was tracked by CVE-2009-3549, and test case 149199 is an example of an implementation of CVE-2002-1869. All of the selected NIST SARD test cases have a few specific properties that we manually selected these for. First, each test case is an example of a different CWE, which makes the results representative of a variety of different possible software weaknesses. Second, each test case contains exactly one line of source code that is an example of a particular CWE; this simplifies the parsing and accuracy metrics. Third, each test case contains no variable names, text, or function names that indicate whether the source code contains a vulnerability or not. In cases where there is a string or function name that indicates what type of example the source code is, we manually remove, or change, that string before parsing by the GPT models. Fourth, for no test cases does the vulnerable line of code occur more than once in the entire source code; this eliminates ambiguity for measuring the accuracy rate of the GPT models for identifying specific lines of source code. Any comments in the source code examples are removed before parsing by the GPT models so as to not give the GPT models natural language that describes the test cases. 

Appendix <ref> shows the exact source code for 10 of these NIST SARD test cases. The phrase vulnerable line of source will be used throughout the text to refer to the exact syntax of the vulnerable line of code – not line number. Line numbers can be somewhat ambiguous, especially for these C syntax test cases, and moreover having the GPT model output the exact source code makes checking the validity of that output much clearer. 


The exact prompt that is used for all of the GPT models is given in the text box below. Note the use of the triple backticks to delineate where the source code is, and where the JSON datastructure is. The use of the triple backticks in the prompt allows the GPT outputs to be automatically parsed by using the triple backticks, and possibly the markdown code identifier, as the delimiter. Note that this triple backtick parsing does not always work, in particular if the JSON is not correctly formatted or the triple backticks are not correctly used in the GPT output. The prompt is intended to be very source code agnostic; there are no details given of the source code in the prompt. The prompt is also intended to not bias the output towards specifying a specific number of vulnerabilities. In the case of the 36 NIST SARD test cases used in this study, the number of CWEs is always exactly 1, however the goal is for this method to more general than only identifying a single CWE and moreover we want the GPT model to be able to fail in a variety of ways in order to rigorously evaluate its accuracy (in this case, identifying more CWEs than are present would constitute a high false positive rate). 



I am seeking assistance to analyze provided source code for the purpose of detecting potential Common Weakness Enumeration (CWE) vulnerabilities. 
The key requirement is for the analysis results to be exclusively formatted in JSON enclosed by triple backticks, listing only the CWEs that are detected as 'Present' in the code, thereby omitting any 'Not Present' entries for a cleaner output. 
The exact line, or lines, of code which have this CWE vulnerability needs to be specified in the returned JSON structure, along with a succinct description of what the vulnerability category is for this source code. 
The JSON output should enumerate each identified CWE with its identification number, the exact source code syntax that causes the CWE, and a description of the vulnerability, as shown in the example structure below:




Source code:


Re-check that your output contains the required JSON data-structure before emitting text. 


The actual GPT prompt is generated by replacing the vulnerable source code in place of the text . 


The usage of requesting the vulnerable code syntax and the description of the vulnerability has two uses:


    
  * Ground the GPT response to only respond based on the provided source code
    
  * To check whether the provided syntax is correct. Because of potential ambiguity with line numbers, only exact syntax string matches are checked for, not integer line number responses. 


The description field is not parsed for accuracy or correctness, but the text contained in that field is an interesting indicator of how correct (or incorrect) the GPT output is. For this reason, when reporting results (in Section <ref>), we provide specific instances of GPT outputs, including the description fields. 


Some of the raw GPT output includes characters that are not ascii and can cause issues with saving and parsing the output. Therefore, the raw GPT output is encoded as utf-8 where character parsing errors are ignored and decoded into a parsable string using utf-8. The field in the JSON output  has repeated excess whitespace removed, as is done with the reference vulnerable line of source code from each of the NIST SARD samples; this ensures that when the string equality check is performed, the only relevant characteristic with respect to whitespace in the string that is being checked is that the whitespace is delineating characters (for example, repeated spaces, tabs, newlines do not impact this accuracy check). 

The GPT output is (attempted to be) parsed in several stages, as follows:


    
  * First, if there are triple backtick delimiters in the output, then the output is split at those characters and the first enclosed set of strings is attempted to be parsed as a valid JSON structure. Interestingly, occasionally the GPT model will add in markdown code identifiers immediately after the triple backticks - in these cases we automatically check for these and remove them if they occur so that the structure can be parsed. These markdown identifiers are given in Appendix <ref>. 
    
  * If the previous step failed, then we attempt an unstructured JSON parsing approach, where output strings are incrementally deleted from the beginning and from the end until a JSON structure can be parsed (or, not). 


Using this parsing, any valid JSON strings that are produced as part of the GPT output will be automatically parsed. This allows this study to perform a systematic evaluation of the accuracy rates of the GPT inference output. This type of data parsing is also highly scalable to a large number of static code samples - and importantly does not require any manual input, tuning, or processing. In other words, this analysis pipeline is entirely automated. 





 §.§ CWE Identification Accuracy Measures



This section lists several measures for quantifying the success rate of the GPT models for identifying known vulnerable source code. 


The first four metrics that are checked are defined as follows:


    
  * Count of parsable GPT outputs (being able to extract a JSON structure and parse it into Python, with not necessarily correct JSON fields. For example a JSON with an empty list is a valid parse. However, this step does require that there is at least one JSON key, which is correctly named as . The subsequent value is allowed to be empty or have any sort of incorrect structure (which could denote that the model is returning no found CWEs) but we need at least this key to validate that a JSON datastructure was parsed. 
    
  * The CWE field is labeled as  is correct (meaning, the correct CW number is listed), but the identified source code is not correct (meaning, it is not the labeled line of code that has the vulnerability) and does not contain the vulnerable line of code either. 
    
  * The CWE is correct and identified as  and the vulnerable code is contained within the identified code syntax but also includes other lines of code from the test case that are not themselves vulnerable as defined by the NIST SARD dataset. These instances can be interpreted as being partially correct. 
    
  * Count of fully correct instances where the CWE is correct (and identified as ) and the identified line of code is correctly identified as the vulnerable code. 


This last metric (metric number 4) is the most important accuracy measure because it counts out of the 1,000 GPT outputs how many correctly performed the computation that we are attempting to prompt the GPT model to perform. 

For none of these four metrics is the Description field, or any other additional field in the JSON, parsed for correctness or even considered. All of these measures are integers, where the larger the integer is the better the GPT model performed, but the absolute correct metric is measure 4. Note that the quantities measured in metrics 2, 3, and 4 (in the immediately preceding list of measures) are always disjoint; there are no instances that are counted in two or more of these measures. An important, and informative, aspect of these metrics is finding cases where the GPT output correctly identified the vulnerable line of code. However, since line numbers of the code are not checked or requested in the prompt (since they are somewhat ambiguously defined especially for C-family languages), a potential source of error in this quantification is if there exist multiple lines of code with the same syntax, but only one is vulnerable. This does happen, but only in one of the test cases. This is NIST SARD test case 149107, which is CWE-415 (Double Free) which by its definition contains two lines of memory free calls. Therefore, for this one test instance the identification of the vulnerable line of code is ambiguous, but for all others the unique identification is valid and correct. 


Another set of metrics that are measured are binary classification error rates. This is computed by considering the GPT output for identified CWEs as a vector of True or False entries, with length equal to the number of valid CWEs (which is currently 839). This measure disregards both the requested  and  fields (and in particular does not consider whether these entries even exist in the JSON structure. For all  entries (that identifies a valid CWE number), and whose  is exactly  (allowing for different capitalizations), that is measured as a binary state of True. All other entries (including instances of identified CWE numbers that are not valid) are measured as binary classifications of False. JSON dictionaries that are completely empty are a valid GPT output that is identifying no CWEs (meaning a binary classification vector of length 839 that is all False entries). These measures are only computed on the outputs that can be parsed; all outputs that can not be parsed are not counted towards any classification accuracy.[In the case where a JSON entry contains multiple entries with the same CWE number, that case is treated as the GPT model having identified that CWE number only once; in other words, repeated entries are ignored. This specifically works for this accuracy metric of binary classification since all other fields (except  are ignored. ] These binary classification measures allow rates of false negatives, false positives, true negatives, and true positives to be measured, and then standard machine learning classification measures such as recall and precision can thus be computed. Precision is intuitively defined as the classifier (which in this case is a GPT language model) being able to not label as positive a sample that is negative. Recall is intuitively defined as the classifier (which in this case is a GPT language model) being able to find all of the positive samples. These metrics being 0 corresponds to the highest possible error rate, and 1 is the best measure these metrics can be. 

All parsing steps require the field names in the JSON datastructure to be correct (correct spelling, no ancillary text), but differing capitalizations are always allowed. 









§ RESULTS



For each of the 36 NIST SARD test cases in Table <ref>, a total of 5,000 inference calls are performed using the methods described in Section <ref> for the 5 different open-sourced GPT models. The accuracy measures from these outputs are then summarized using several high-level metrics. Tables <ref> and <ref> present two different accuracy measure summaries. Table <ref> counts how many of the GPT outputs were parsable and correctly identified the CWE number and vulnerable line of code. Table <ref> shows the binary classification measures recall and precision. Table <ref> reports the absolute integer count of how many of the GPT outputs correctly identified the single line of vulnerable source code; importantly, these counts are independent of correct identification of the relevant CWE. These counts, as a proportion out of 1000 should be compared against uniform random sampling of the total lines of source code (shown in Table <ref>). Table <ref> shows that the correct identification of vulnerable lines of code is much higher than the correct identification of the CWE number and the vulnerable line of source code, as measured in Table <ref>. Notably in Table <ref>, only for a few of the NIST SARD test cases is the count of ever correctly identifying the syntax of vulnerable line of source code zero across all GPT models and settings; namely test cases 149203, 148871, and 2046. For some of the test cases, the proportion of correctly identified vulnerable line of source code is very high. The test cases where the vulnerable line of source was not uniquely identified in Table <ref> does not necessarily mean that the correct CWE identification was always 0 (when analyzing strictly whether the identified CWE was correct) - an example of this is test case 2046. 

Table <ref> shows that for the code test cases where the accuracy is 0, the 5 different language models do not perform uniformly bad – for some of the models the accuracy is always 0 for a given test case, but then when analyzed with a different model the accuracy is non-zero (an example of this is NIST SARD test case 500757). This means that some of the GPT models do perform better than other GPT models for specific test cases. And moreover, there is not consistently a single model that always outperforms the other models. 


Table <ref> shows the best performing GPT model and inference temperature in terms of recall and precision measures. Finally, Table <ref> counts the most frequently identified false positives. The best performing GPT model and inference temperature shows that there is no specific model, or temperature setting, that consistently performs the best - and in in particular, every one of the tested GPT models was the best performer for at least one of the CWE test cases. This shows that this task does require diversity of the model architecture, and very likely the type of text on which the models were trained. Lastly, even with these best performing GPT models and inference temperatures some of the CWE test cases were never correctly identified, meaning that the accuracy rates for all GPT models and settings were always 0 - that happened for 11 out of 36 of the test cases as denoted by asterisks in Table <ref>. 


Section <ref> then gives several examples of correct JSON output from the GPT models where the vulnerable line of source code was identified, and where the description is correct in some cases and incorrect in other cases. 



Table <ref> quantifies the exact counts of false positive CWE id occurrences for the full matrix of the 5 GPT models and NIST SARD test cases. Specifically, for each NIST SARD test case and GPT model, Table <ref> reports the most frequently mis-identified CWE number. Table <ref> demonstrates two important trends. First, the proportion of false positives is very high, and this is consistent across the different GPT models. Second, for most test cases and GPT models the distribution of incorrect CWE ids is not uniformly distributed across the possible CWE numbers - it is heavily biased towards a few specific CWE numbers. For example,  shows a clear bias towards predicting CWE-124 for multiple test cases. Similarly,  shows a consistent bias for (incorrectly) predicting CWE-123, and  shows a consistent bias for (incorrectly) predicting CWE-476. 



































 §.§ Example Correctly Identified Vulnerable Lines of Code



This section enumerates several parsed JSON outputs from the various GPT models and code test cases. These outputs were hand selected to be representative of cases that are correct (in the sense that the CWE number is correctly identified, as is the vulnerable line of source code), but whose description fields are in some cases correct and in other cases incorrect. Showing explicit examples of the output is intended to primarily show the variability of the generated Description field, and also to show concretely what successful static code analysis identification looks like from the GPT models. 

No text changes were made to these GPT outputs except for formatting to fit within the text boxes. Note that these outputs are specifically examples of text from the enclosed triple backtick fields - any additional prose or output from the model is not shown. 



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 500843. Produced from the Turdus GPT model with temperature 0.8. Note that the generated description is partially correct - CWE-476 is NULL pointer dereference but the description also mentions Divison-by-Zero which is not correct. ]






[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 500843. Produced from the zephyr-7b-alpha GPT model with temperature 0.7. Note that the generated description is entirely incorrect.]



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 500843. Produced from the Turdus GPT model with temperature 1.0. Note that the generated description is entirely incorrect. ]





[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 500843. Produced from the Llama-2-70b-chat-hf GPT model with temperature 0.8. Note that the provided description is mostly correct in that the correct vulnerability name was identified but there is not a function named f in the source code. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1792. Produced from the Turdus GPT model with temperature 0.3. Note that the generated description is correct.]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1792. Produced from the Turdus GPT model with temperature 0.4. Note that the generated description is correct.]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1792. Produced from the Turdus GPT model with temperature 0.4. Note that the generated description is correct except that the identified print statement in the description text is incorrect.]





[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1645. Produced from the Turdus GPT model with temperature 0.5. The generated Description field in this example is correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1645. Produced from the zephyr-7b-alpha GPT model with temperature 0.9. The generated Description field in this example is correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149165. Produced from the Llama-2-70b-chat-hf GPT model with temperature 0.9. The generated Description field in this example is entirely incorrect. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149165. Produced from the Llama-2-70b-chat-hf GPT model with temperature 0.1. The generated Description field in this example is correct. ]



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149165. Produced from the Mistral-7B-Instruct-v0.1 GPT model with temperature 0.6. The generated Description field in this example is correct. ]



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149165. Produced from the zephyr-7b-beta GPT model with temperature 0.2. The generated Description field in this example is correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149165. Produced from the zephyr-7b-beta GPT model with temperature 0.4. The generated Description field in this example is not correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 500757. Produced from the Turdus GPT model with temperature 0.8. The generated Description field in this example is correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 1989. Produced from the Turdus GPT model with temperature 0.9. The generated Description field in this example is correct. ]






[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149111. Produced from the Llama-2-70b-chat-hf GPT model with temperature 0.3. The generated Description field in this example is correct. ]




[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149111. Produced from the Mistral-7B-Instruct-v0.1 GPT model with temperature 0.7. The generated Description field in this example is correct. ]



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 149241. Produced from the zephyr-7b-alpha GPT model with temperature 0.2. The generated Description field in this example is correct. ]



[title=Example of correct CWE number and code line identification in a JSON datastructure format from NIST SARD Test Case 313. Produced from the Llama-2-70b-chat-hf GPT model with temperature 0.6. The generated Description field in this example is correct. ]









§ DISCUSSION AND CONCLUSION




This study has examined the task of correctly identifying software weaknesses and vulnerabilities in software using open source Generative Pre-Trained Transformer (GPT) models. This was able to be examined because of two specific reasons - the first is the existence of the NIST SARD dataset, which is a very well labelled dataset that includes the vulnerable line of code and the relevant CWE. The second is the specific prompting template usage that allows fully automated extraction of JSON output for the vulnerability detection - and therefore error rates can be quantified for a large range of NIST SARD test cases, GPT models, and GPT model settings. Thus, this study reports a systematic analysis of automated vulnerability detection using GPT models. 

In general, our findings are that the correct CWE identification rate varies significantly depending on the specific code test case that is analyzed. For some test cases the correct CWE identification (and relevant line code of syntax) is surprisingly high, but for other test cases the identification accuracy rate is always 0 for all GPT models and temperature settings. In general, the error rates of this vulnerability detection is too high – in particular, the false positive rate is very high. However, it is notable that for some cases, seemingly the more popular classes of CWEs that are mentioned in the type of internet text on which these GPT models were trained, the identification rates are quite good. As specific examples of the lowest error rate results (from Table <ref>), we list several specific examples of CWE identification binary classification accuracy rates (without considering rates for correctly identifying the vulnerable line of source code). Test case 1989 had a best recall of 0.928, a best precision of 0.709 (with a parsability rate of 0.42) when analyzed with  GPT model and temperature 0.1. Test case 149165 had a best recall of 1.0, a best precision of 1.0 (with a parsability rate of 1.0) when analyzed with  GPT model and temperature 0.1. Test case 149111 had a best recall of 0.727, a best precision of 0.675 (with a parsability rate of 0.77) when analyzed with  GPT model and temperature 0.5. Test case 149241 had a best recall of 1.0, a best precision of 0.5 (with a parsability rate of 1.0) when analyzed with  GPT model and temperature 0.1. Given the high accuracy rates of at least some of these test cases, it seems plausible that some of the current open source GPT models, if thorough verification is used, could correctly identify certain classes of vulnerabilities, such as buffer overflows, with very low error rates. 


It should be emphasized that this particular software security analysis task requires a number of semantic connections to be made in the GPT model inference process (and of course, the correct specification of the JSON syntax) in order for the (automated) vulnerability identification to be correct. It is not clear to what extent a pre-trained language model can learn these semantic connections - but it is possible least for some of the CWEs, these connections were learned in the standard open source GPT models that we tested. This is is a non-trivial finding, and suggests that GPT models do have some capability to identify security vulnerabilities. However, it also needs to be emphasized that such usage of current GPT models should not be used in cases where the accuracy can not be tested. In other words, these GPT models and vulnerability identification prompting should not be used in software engineering tasks – they should be used in cases where careful and precise testing of the GPT output can be performed for the purpose of examining specific characteristics of source code of interest. The primary reason for this is that the false positive rate is high - and not only is it high but the false positives are biased towards incorrectly identifying specific CWEs. In other words, simply measuring which CWE is predicted most frequently in a GPT output distribution (for a non-zero inference temperature) will typically not be correct. 


The primary limitation with increasing the scope of more code test cases to more thoroughly test the capabilities of GPT models for software security analysis is the total amount of compute time that is required for performing these inference calls, especially in a systematic study of varying inference parameters. Future studies that increase the number of labeled source code examples (e.g., from the extensive amount of data available in the NIST SARD dataset) that are used for testing the accuracy GPT automated vulnerability identification would be very relevant. 


An aspect of this study that is unclear is what details of the source code are the GPT models using to infer relevant CWE's in the source code. In particular, what semantic details are used to parse whether a CWE is more likely or less likely. In some cases, it could be that context, such as what types of libraries are used or what types of strings are being constructed (such as SQL query strings), is sufficient to reduce the total number of possibly applicable CWEs so that random guessing among those feasible candidates accounts for some of the GPT model's performance. The metrics in this study do not determine what components of the source code are important; this would require quantifying the explainability of GPT models <cit.> for this task. We leave this study open to future research, and emphasize that this type of detailed explainability analysis is absolutely a critical component of study before these GPT models are incorporated into any type of vulnerability detection in production software development and testing. 






§ ACKNOWLEDGMENTS



Sandia National Laboratories is a multi-mission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC (NTESS), a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration (DOE/NNSA) under contract DE-NA0003525. This written work is authored by an employee of NTESS. The employee, not NTESS, owns the right, title and interest in and to the written work and is responsible for its contents. Any subjective views or opinions that might be expressed in the written work do not necessarily represent the views of the U.S. Government. The publisher acknowledges that the U.S. Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this written work or allow others to do so, for U.S. Government purposes. The DOE will provide public access to results of federally sponsored research in accordance with the DOE Public Access Plan.














§ MARKDOWN COMPUTER CODE LANGUAGE IDENTIFIERS











§ NIST SARD TEST CASES



This section shows the exact source code for 10 of the NIST SARD test cases that are used in this study. The single vulnerable line of source code is highlighted as red text for each test case. 


.49
[caption= NIST SARD Test case 500843, instance of CWE-476 ,captionpos=b,label=SARD_source_code:500843,language=C,showstringspaces=false,style=base,keywordstyle=]

struct Token 
    const Token* nextArgument() const;
    const Token* next() const;
    int varId() const;
    void str() const;;
void f(const Token *first) 
    first = first->nextArgument();
    if (first)
        first = first->next();
    @first->str();@



.49
[caption= NIST SARD Test case 1792, instance of CWE-79. Note that some of the strings in this test case have been modified so that terms related to cross site scripting are removed. ,captionpos=b,label=SARD_source_code:500843,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdio.h>
#include <cgic.h>
#include <string.h>
#include <stdlib.h>

int cgiMain() 

	cgiHeaderContentType("text/html");
	 
	fprintf(cgiOut, "<html><head>");
	fprintf(cgiOut, "<title>Text: 1</title></head>");
	fprintf(cgiOut, "<body><h1>Text</h1>");
	 
	char q[4][1024];
	unsigned int i = 0;
	for (; i < 4; ++i)
		char name[2];
		sprintf(name,"q
		cgiFormString(name, q[i], sizeof(q[i]));
		if (strlen(q[i]))
		
			@fprintf(cgiOut, "Value number 
		
			
	 
	fprintf(cgiOut, "</body></html>");
	return 0;





.49
[caption= NIST SARD Test case 1779, instance of CWE-463. ,captionpos=b,label=SARD_source_code:1779,language=C,showstringspaces=false,style=base,keywordstyle=]

main ()

char *foo;
int counter;
foo=malloc(sizeof(char)*10);
for (counter=0;counter!=14;counter++)
@foo[counter]='a';@
printf("









.49
[caption=NIST SARD Test case 1645, instance of CWE-20. ,captionpos=b,label=SARD_source_code:1645,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdio.h>
#include <stdlib.h>

#define	MAXSIZE		40

void test(char *str)
	char buf[MAXSIZE];
	snprintf(buf, sizeof buf, "/bin/echo 
	buf[MAXSIZE-1] = 0;
	@system(buf);@


int main(int argc, char **argv)
	char *userstr;
	if(argc > 1) 
		userstr = argv[1];
		test(userstr);
	
	return 0;









.49
[caption=NIST SARD Test case 149165, instance of CWE-121. ,captionpos=b,label=SARD_source_code:149165,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#include <string.h>

unsigned int getRand()

	unsigned int r;
	FILE *f;
	f = fopen("/dev/urandom", "rb");
	if(f == NULL)
	
		fprintf(stderr, "Error opening file");
		exit(-1);
	
	if(fread(   r, sizeof r, 1, f) != 1)
	
		fprintf(stderr, "Error reading file");
		fclose(f);
		exit(-1);
	
	if(fclose(f) != 0)
		fprintf(stderr, "Error closing file");

	return r;


unsigned plop() 
	return getRand() 


int main(int argc, char *argv[])

	char buffer[256];
	memset(buffer, 0, sizeof(buffer));
	@buffer[plop()] = '!';@		 
	printf("
	return 0;









.49
[caption=NIST SARD Test case 2015, instance of CWE-329. ,captionpos=b,label=SARD_source_code:2015,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <openssl/evp.h>

main()
EVP_CIPHER_CTX ctx;
char key[EVP_MAX_KEY_LENGTH];
char iv[EVP_MAX_IV_LENGTH];
int b=8;
RAND_bytes(key, b);
memset(iv,0,EVP_MAX_IV_LENGTH);
@EVP_EncryptInit(   ctx,EVP_bf_cbc(), key,iv);@
return;









.49
[caption= NIST SARD Test case 500757, instance of CWE-787 ,captionpos=b,label=SARD_source_code:500757,language=C,showstringspaces=false,style=base,keywordstyle=]
#include <cstdlib>
void f()

  char *p; p = (char *)malloc(10);
  @p[10] = 7;@
  free(p);





.49
[caption=NIST SARD Test case 149135, instance of CWE-489. ,captionpos=b,label=SARD_source_code:149135,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdio.h>
#include <stdlib.h>
#include <stdbool.h>
#include <string.h>

static bool debug = false;

void promote_root() 
	if (debug) 
		 
		printf ("# You are root now...");
	


int main(int argc, char *argv[])

	if (argc > 1)
	
		unsigned i;
		const unsigned nbArgs = argc;

		for (i = 1; i < nbArgs; ++i)
		@
			if (!strncmp(argv[i],"-debug",6)) @
			
				debug = true;
				printf("Move to debugging mode");
				 
				if (strlen(argv[i]) >= 11        !strncmp(argv[i]+6,":root",5))
				
					promote_root();
				
			
			
	
	else
	
		printf("No args...");
	

	return 0;










.49
[caption=NIST SARD Test case 149203, instance of CWE-416. ,captionpos=b,label=SARD_source_code:149203,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

typedef struct str_t str;
struct str_t 
	union 
		int   a;
		char *b;
	 foo;
;

int main()

	str container;
	char *str1;
	container.foo.a = 0;
	
	if ((container.foo.b = (char *)malloc(256*sizeof(char))) != NULL)
	
		strcpy(container.foo.b, "Falut!");
		container.foo.b[0] = 'S';
		printf("
		free(container.foo.b);
		str1 = malloc(65536); 
 		if (str1 == NULL)
        		printf("Memory allocation problem"); return 1;

		free(str1);
		
		if (container.foo.b)
			@container.foo.b[0] = 'S'; printf("
	
	return 0;









.49
[caption=NIST SARD Test case 149111, instance of CWE-134. ,captionpos=b,label=SARD_source_code:149111,language=C,showstringspaces=false,style=base,keywordstyle=]

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#define MAX_SIZE 255

typedef struct cont_t cont;
struct cont_t 
	char fmt[MAX_SIZE];
;

int main(int argc, char *argv[])

	if (argc > 1) 
		cont container = .fmt="";
		strncpy(container.fmt, argv[1],MAX_SIZE-1);
		container.fmt[MAX_SIZE - 1] = '';
		@printf(container.fmt);		@		 
	
	return 0;


















 Paper Title: Unsupervised Pairwise Causal Discovery on Heterogeneous Data using Mutual Information Measures.


          








A]Alexandre Trilla,
B]Nenad Mijatovic

[A]Alstom, Santa Perpètua de la Mogoda, Barcelona, 08130, Spain
	
[B]Alstom, Saint Ouen, Paris, 93482, France

§ ABSTRACT 

A fundamental task in science is to determine the underlying
causal relations because it is the knowledge of this functional
structure what leads to the correct interpretation of an effect 
given the apparent associations in the observed data.
In this sense, 
Causal Discovery is a technique that tackles this
challenge by analyzing the statistical properties of the 
constituent variables.

In this work, we target the generalizability of the 
discovery method by following a reductionist approach that 
only involves two variables, i.e., the pairwise or bi-variate
setting.

We question the current (possibly misleading)
baseline results on the basis that they were obtained through 
supervised learning, which is arguably contrary
to this genuinely exploratory endeavor. In consequence, we
approach this problem in an unsupervised way, using robust Mutual
Information measures, and observing the impact of the different variable
types, which is oftentimes ignored in the design of solutions.
Thus, we provide a novel set of standard unbiased results
that can serve as a reference to guide future discovery tasks 
in completely unknown environments.



Causal Discovery, Functional Causal Model, Unconditional Independence Test, Mutual Information, Unsupervised Learning, Heterogeneous Data








§ INTRODUCTION

Causal Discovery methods are able to identify the causal structure
from the joint distribution of the data by introducing assumptions 
that restrict the model of their generating process <cit.>.
In a multivariate setting, the traditional constraint-based and 
score-based methods exploit conditional independence 
relationships in the data <cit.>.
These approaches have found a great deal of success in 
challenging environments such as biology and the Earth
sciences <cit.>. Nevertheless,
they do not necessarily provide complete causal information 
because they output Markov Equivalence Classes, i.e., a set of
causal structures that satisfy the same conditional independence
statements. Moreover, they cannot handle isolated cause-effect
settings that lack the diversity of other variables for the 
conditional tests. Therefore, the single reduced cause-effect association,
also known as the pairwise or bivariate scenario, constitutes an 
essential building block of more complex causal structures. 

To distinguish cause from effect in this pairwise setting, 
one needs to find a way to capture the asymmetry between the
two variables <cit.>. In this sense,
computational methods based on properly defined Functional Causal 
Models (FCMs) are able to distinguish different structures in the 
same equivalence class.
A FCM represents the effect variable Y as a function of its direct
causes X and some noise term ε, i.e., 
Y = f(X, ε), where ε is
independent of X. Thanks to the restricted functional classes,
the causal direction between X and Y is generally identifiable because
the independence between the noise and the cause holds
only for the true causal direction and is violated for the wrong
direction <cit.> (unless the variables are jointly Gaussian,
which render the orientation unidentifiable). Additionally, the
identifiability of pairwise FCM generalizes to the identifiability 
of multivariate FCM <cit.>.

In this work, we argue that some reference benchmark results for pairwise 
causal discovery could be misleading because they tackle this genuinely 
exploratory objective as a classical supervised classification problem, 
inducing them to possibly learn specific contextual details beyond causality 
that tend to overestimate their actual performance <cit.>.
Moreover, there exists a tight relationship between the nature of data 
and the specific tools from statistics that could also accrue the estimation 
of skewed results, which was not originally addressed. Consequently, we 
propose a reevaluation of the benchmark driven by the heterogeneity 
of the data and also by following an unsupervised 
learning approach. Our method leverages linear functional causal models and 
combines different unconditional independence tests based on robust
Mutual Information measures.
The paper is organized as follows: Section <ref>
introduces the fundamental concepts for pairwise causal
discovery, Section <ref> describes the enhanced method
proposed in this work, Section <ref> shows the novel
benchmarked results, Section <ref>
discusses the approach from a causal perspective, along with its
limitations, and
Section <ref> concludes the article.






§ BACKGROUND
 

This section reviews the fundamental concepts that support the
discovery of causality in the bivariate or pairwise setting.



 §.§ Identifiability of a Linear Model with Non-Gaussian Noise

The fundamental principle that enables the identification of cause
and effect lies in the asymmetry of the association between them: 
the causal direction, i.e., from cause to effect, is functionally 
simpler than the anticausal one, i.e., from effect to cause, which
requires a more complex application <cit.>.
Thus, a given model with limited capacity will be able to 
faithfully capture the expressiveness 
of only one of these two causal orientations.

Linear additive noise models assume that the effect Y is a linear 
function of the cause X plus a noise term ε that
is independent of the cause, formally defined as: Y = bX + ε.
The linear FCM is learnable if at most one of X and ε
is Gaussian <cit.>. Pairwise discovery approaches such as this one are regarded to be very flexible for identifying causal models in the general case because they don't require a third variable for conditioning.
In specific scenarios such as multivariate time series, they can even
improve the performance of traditional constraint-based
approaches such as the Peter-Clark algorithm and others that
utilize Granger causality <cit.>.

The linear model with non-Gaussian noise can be estimated from 
(unconfounded) observational data by exploiting the inherent 
asymmetry between cause and effect through a regression as 
follows: for both directions (i.e., causal and anticausal), the 
FCM is fit; then, the amount of association between the estimated 
noise term (i.e., the regression residual) and the hypothetical 
cause is computed; finally, the direction assignment which gives an 
independent noise term is considered plausible <cit.>.

Obviously, the cornerstone of this regression-based method is the
measure of dependence between the regression residual and its
associated potential cause. The next section 
describes a solution to estimate the strength of this association.





 §.§ Mutual Information as a Measure of Association

The amount of association between the regression residual RR and
the hypothetical cause HC signals the causal direction. However,
these variables are uncorrelated by construction of the regression. To 
properly quantify this dependence, the Mutual Information (MI) measure 
provides a reliable indicator, which is defined as


    MI(RR,HC) = ∑_RR,HC P_RR,HClog ( P_RR,HC/P_RR  P_HC )  ,


where P_RR,HC represents the (discrete) joint probability between
RR and HC, and P_RR and P_HC represent their marginal
distributions, respectively.

For illustrative purposes, the following variable dependencies 
are defined. First, a random Uniform noise sample U is
independently assigned to Z, X_ind and Y_ind. Then,
the following structural variable associations are created: 
Y_dep← X_ind + Z, X_conf← X_ind + Z,
and Y_conf← Y_ind + Z, which yield the subsequent relationships:


  Causal Between X_ind and Y_dep: X → Y

  Anticausal Between Y_dep and X_ind: Y ← X

  Independent Between X_ind and Y_ind: X  Y

  Confounded Between X_conf and Y_conf: X ↔ Y


Figure <ref> shows the distribution of MI between
the regression residual and the hypothetical cause considering
the preceding types of structural variable associations for a random
sample of 1000 instances. The different density plots
show how the causal orientation can be reliably recovered using
MI when the model assumptions are respected. Note that
the overlap between Causal and Independent structures 
will be disambiguated when both experiments are conducted regarding 
the different causal direction hypotheses.
Finally, MI is also able to detect latent confounding, which is 
oftentimes assumed (and required) not to occur for 
discovery <cit.>, i.e., the causal 
sufficiency principle <cit.>. This feature 
adds value to the robustness of the MI indicator.



While the distribution of MI is a useful tool to visualize the
different causal structures, there is still a missing criterion to 
make a classification decision in order to construct the underlying
causal structure from the data. The next section addresses this point.




 §.§ Pearson's χ^2 as an Unconditional Independence Test


In a real-word scenario, one cannot expect to get a null measure
of association because of sampling noise, the sample size, etc.
In this case, the use of an (unconditional) independence
test is the proper way to handle this situation. In general,
this is known as the REgression with Subsequent Independence Test
(RESIT) procedure <cit.>. 
Once the independence test is in place, a confidence interval needs 
to be observed to obtain actionable results. To this end, Pearson's 
χ^2 test is here introduced as a fundamental 
method to inform the decision process <cit.>.

The Pearson's χ^2 test is a statistical hypothesis test used 
in the analysis of contingency tables for categorical variables.
It is used here to determine whether there is a statistically 
significant difference between the expected frequencies E
for independence (i.e., the marginal probabilities)
and the observed frequencies O in one or more categories i
of the contingency table (RR,HC), formally defined as

    χ^2(RR,HC) = ∑_i ∈ (RR,HC)(O_i - E_i)^2/E_i .


The test is valid as long as its statistic is χ^2 distributed 
under the null hypothesis.
Note that χ^2(RR,HC) is in its essence related to the MI measure of 
association as is used for the discovery of causal structure.

While the χ^2 test is most suitable for discrete variables <cit.>,
a general
causal discovery method should be able to seamlessly deal with 
continuous variables. Typically, a uniform grid is applied to discretize
a real-valued space prior to conducting the test.
Following the former illustrative example, Table <ref>
shows the p-values for the different causal structures.
Once a threshold is introduced through the confidence interval,
e.g., p<0.05 for a given number of degrees of freedom related to
the resolution of the grid,
passing the test becomes the rule for making further decisions.
Note that in all cases, there is a unique combination of results
for the two hypothesized causal directions that identifies the true
underlying causal structure.
Also note that the threshold may be adjusted to require further decision strength because the closer the variables get to a Gaussian distribution, the harder it is to distinguish the direction of causation <cit.>.



While these results
may be sufficient as an indication, they may also have some 
limitations for discovering causal associations between numerical variables.
The next section addresses this shortcoming and describes
an enhanced solution for this type of variables.







 §.§ Total Information Coefficient as a Robust Independence Test



The Total Information Coefficient (TIC) is a robust independence
test based on MI for real-valued variables 
that features the two important heuristic properties of generality and 
equitability <cit.>, which are defined as follows:



  Generality With sufficient sample size the statistic 
should capture a wide range of interesting associations, not limited 
to specific function types. 

  Equitability The statistic should give similar scores to 
equally noisy relationships of different types.


For a pair of (RR,RC) variables, the TIC algorithm applies a search
procedure to partition their joint probability function and find the
grid with the highest induced MI.
TIC operates by summing over optimal grids G on the joint 
density distribution of the two variables, such that it exhibits 
a stronger power 
against independence <cit.>, formally expressed as

    TIC(RR,HC) = ∑_G MI((RR,HC)|_G)/log∥ G ∥ ,

where ∥ G ∥ denotes the minimum of the number 
of rows and columns of G. This can be viewed as a regularized 
version of MI that penalizes complicated grids.
Figure <ref> shows qualitatively the impact of the
optimum grids G.




The performance of TIC has been positively rated with
other successful approaches for causal discovery dealing with
continuous variables, such as the
Hilbert-Schmidt independence criterion <cit.>.
Finally, an additional motivation for
considering TIC in this work is to increase the odds of success
in case of model hypothesis violations, i.e., the functional form
of the relationship Y = f(X), which is crucial for causal
identification.






§ METHOD


In the pursuit of a general approach to causal discovery
combining several methods <cit.>,
one common point of success is found in the use
of information-theoretic measures, such as Mutual Information,
to quantify the amount of regularity in the 
data <cit.>. 

However, the nature of the variables, i.e., discrete or continuous, 
can be problematic in an heterogeneous context if the statistical 
methods are confused <cit.>.


Moreover, supervised learning approaches are excluded from the
general objective because these methods do not yet work as standalone 
techniques for causal learning <cit.>. Therefore,
the desideratum in causal discovery is to have methods that work on a 
broad range of problems under different conditions with relaxed
assumptions <cit.>, ideally showing a 
certain degree of robustness regarding violations of the model 
hypothesis <cit.>.


In this work we propose a general method to discover the causal 
structure in heterogeneous data based on a bivariate linear FCM by
implementing a flexible RESIT procedure, see 
Section <ref>, where the
unconditional independence test, e.g., χ^2 or TIC,
also see Section <ref>,
is driven by the nature of the variables involved. The proposed 
strategy is described with comments in Algorithm <ref>.
Note that the orientation decision rules follow from the evidence 
given by the illustrative setting in Section <ref>, 
which is regarded as self-evident.
Such explicitly-stated logic rules are meant to increase the 
interpretability and explainability of the proposed method.
Also note that discrete variables are assimilated to real-valued variables for computing the regression residuals on which the RESIT method is based. Finally,
since our contribution targets the problem of applying one single discovery technique on data with different types of variables, and we focus our effort at the integration level rather than at the fundamental level, we refer the interested reader to the references for the specific methods being integrated for further details about their comparison to other methods from the literature.
We rely on the advice from Peters, Reshef, and 
colleagues <cit.>, to select the best methods that
we integrate in RESFIT in order to cover a broad range of techniques, 
and we provide an analysis on how these separate tools perform when
they are put together.















§ RESULTS


This section describes the reference benchmark dataset that was used
to evaluate the introduction of the flexible independence test selection, 
the results that were obtained, and the tools for the implementation 
of the research.



 §.§ Reference Benchmark Dataset



The “ChaLearn cause-effect pair (SUP2)” is taken for reference as
the recommended benchmark dataset to evaluate the performance
of pairwise causal discovery <cit.>.
It comprises pairs of artificially generated dependent 
variables with different types 
(numerical, categorical and binary) for the full 
causal discovery task (orientation, independence and confounding).
The dataset features a balanced number of unique values across all 
classes and includes around 6000 pairs. In terms of data type balance,
the majority is comprised of numerical and mixed variable pairs. Finally, 
the average median length of an instance is around 2000 values.



 §.§ Performance Scores


The accuracy classification score, i.e., the total rate of correct
predictions statistically given by the overall amount of
true positives and true negatives,
is used in this research following the previous benchmark
evaluation approaches <cit.>, yielding a baseline around 
65±2% for the supervised learning setting <cit.>.
To further assess the stability of our scores in the unsupervised
scenario, statistical
bootstrapping is introduced in line with this former work.
Moreover, Gaussianity is asserted with the Lilliefors normality
test <cit.>,  and the main descriptive
statistics are extracted despite the low amount of available performance
samples, well under 30, which are commonly
required to obtain reliable estimations <cit.>.
Finally, in terms of statistically significant comparisons, the t-test has been
used <cit.>.
Table <ref> shows the results of the experiments.
Note that they are presented separately by data type and independence
test, whereas the proposed RESFIT algorithm automatically integrates
them. This is done for illustrative purposes and for enriching 
the ensuing discussion.




The first straightforward conclusion that the results show is that,
in all cases, there is a statistically significant difference between
the accuracy averages for the two unconditional independence tests
within each data type stratum. Therefore, introducing a selection 
action on the test function is expected to impact 
on the discovery of causal associations.
Additionally, TIC yields a slightly smaller variance in all
the scores. Finally, when numerical data types are present,
the standard deviation in accuracy drops an order of magnitude.

The code for this research is available 
here[Code repository: https://github.com/atrilla/ccia24].
The next section discusses the global results with respect to the
perspective of a causal effect, and addresses the limitations
of the proposed approach.





§ DISCUSSION


This section delves into the finer details of the results that
were obtained, and sheds light on the actual value introduced by
the flexibility in selecting the unconditional independence test
for causal discovery.



 §.§ Average Causal Effect of the Flexible Test Selection


To properly quantify the impact of introducing the flexibility
on the unconditional independence test, this section 
treats this selection action as a “treatment” variable X 
and studies its impact on the “outcome” discovery accuracy
score Y.
The Average Causal Effect (ACE) of X → Y is
expressed using the following counterfactual notation 


    ACE =    𝔼 [Y_1 - Y_0] = 𝔼 [Y_1] - 𝔼 [Y_0] = 𝔼 [Y(X=1)] - 𝔼 [Y(X=0)] 
    
    	=    𝔼 [Y|X=1] - 𝔼 [Y|X=0]  ,


where Y_x refers to the value the Y accuracy result would have
if X was set to x, i.e., Y(X=x). Here, X
could take the values x=1 to indicate
flexibility of independence test, and x=0 to indicate no
preference (i.e., the null random choice). The conditionals that
eventually follow are the values that are actually observed in the results.
Note that all these expected quantities can be exactly calculated 
because the different experiments can be conducted on the same data
(also dispelling any doubts about latent confounding).
This setting is thus not subject to the fundamental problem of 
causal inference where only one of the potential outcomes
can be observed <cit.>.

Equation (<ref>) is shown to be an unbiased estimator for the
ACE <cit.>. For computing the expectations in the
causal effect comparison, a weighted mixture of Gaussian random
variables is required.
The quantity of the reference potential outcome 
𝔼 [Y_0], where no smart selection occurs, is given by
the arithmetic mean value of the total results, and it yields
an accuracy score of 0.3531±0.0357. Alternatively, the potential outcome
𝔼 [Y_1], where the smart flexible test selection is introduced, 
is given by the average of the best performance results among 
the different variable
types, i.e., a sum weighted by their balance in the dataset,
and it yields a value of 0.3866±0.0690. Therefore, the ACE
is of 3.36%, and this estimated difference is statistically significant.
However, the absolute results are far from the 65% baseline, which
suggests that the original supervised approach may have leaked
statistical patterns beyond causation. Although the unsupervised
learning approach does not seem to offer any pragmatic quantifiable
advantage, we believe that its adoption is a must for tackling the
inherently hard exploratory objective of causal discovery and avoid
any qualms with regards to learned biases.




 §.§ Limitations

The realization that TIC, which was conceived for
numerical data, performed better on discrete data is surprising.
Also, χ^2, which is defined for discrete data, performed 
better on numerical data. This counterintuitive behavior 
where theory and practice disagree may
suggest that the independence tests that were used are subject 
to some inherent limitations such as the sample size. 
We regard the shortage of samples to be the main limitation
of our approach on real-world data applications such as the
Tuebingen dataset <cit.>.

Finally, while the approach that we propose is able to welcome
any testing technique, we selected RESIT flexibly paired with TIC and 
χ^2 following the advice of Peters, Reshef, and 
colleagues <cit.>. To the best
of our knowledge, this limited choice should be sufficient to 
generally cover the spectrum of approaches.





§ CONCLUSION



Environments with heterogeneous data pose challenging questions to
the discovery of causal structure, and leveraging one single method may lead to erroneous results in general.
In this work, we propose an
unsupervised pairwise approach using linear functional causal 
models and different unconditional independence tests based on 
Mutual Information measures, the selection of which 
is driven by the nature of data and the empirical results 
from a reference benchmark.
The introduction of this independence test selection flexibility is
estimated to have a positive and statistically significant average
causal effect over 3% in accuracy.
These scores may establish the first 
standard baseline for this kind of flexible focus, but more
research is needed because the limitations of the
data size and the statistical techniques can result in 
counterintuitive results.







vancouver





 Paper Title: Extended Fiducial Inference: Toward an Automated Process of Statistical Inference.


 



 
 

 









 


 


 
Extended Fiducial Inference: Toward an Automated Process of Statistical Inference
    Faming LiangCorrespondence author: Faming Liang, email: fmliang@purdue.edu, 
 Department of Statistics, Purdue University, West Lafayette, IN 47907, USA; 
 ^† Department of Population Medicine, Harvard Medical School/ Harvard Pilgrim Health Care Institute, Boston, MA 02215, USA; 
^ Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania, Philadelphia, PA 19104, USA. 
 ,   Sehwan Kim^†, and Yan Sun^
    
====================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ ABSTRACT 







While fiducial inference was widely considered a big blunder by R.A. Fisher, the goal he initially set –`inferring the uncertainty of model parameters on the basis of observations' – has been continually pursued by many statisticians. To this end, we develop a new statistical inference method called extended Fiducial inference (EFI). The new method 
achieves the goal of fiducial inference by leveraging advanced statistical computing techniques 

while remaining scalable for big data. EFI involves jointly imputing random errors realized in observations using stochastic gradient Markov chain Monte Carlo 
 and estimating the inverse function using a sparse deep neural network (DNN). The consistency of the sparse DNN estimator ensures that the uncertainty embedded in observations is properly propagated to model parameters through the estimated inverse function, thereby validating downstream statistical inference.
Compared to frequentist and Bayesian methods, EFI offers significant advantages in parameter estimation and hypothesis testing. Specifically, EFI provides higher fidelity in parameter estimation, especially when outliers are present in the observations; and eliminates the need for theoretical reference distributions in hypothesis testing, thereby automating the statistical inference process. EFI also provides an innovative framework for semi-supervised learning.



  
Keywords:  Complex Hypothesis Test, Markov chain Monte Carlo, Semi-Supervised Learning, Sparse deep learning, Uncertainty Quantification



 








§ INTRODUCTION












 
 Statistical inference is a fundamental task in modern data science, which studies how to propagate the uncertainty embedded in data to model parameters. 
 During the past century, frequentist and Bayesian methods have evolved 
 as two major frameworks of statistical inference. 
  However, due to some intrinsic issues (see Section <ref>), these methods may lack one or more features — such as fidelity, automaticity, and scalability —
 necessary for performing statistical inference on complex models in modern data science. Specifically, the frequentist methods often estimate
 model parameters using the maximum  likelihood approach and test hypotheses by comparing a test statistic with a  known theoretical reference distribution. 
 It is well-known that the maximum likelihood estimator (MLE) can be significantly 
 influenced by outliers, 
 which reduces the fidelity of parameter estimates. For hypothesis testing, the required theoretical reference distribution is test statistic-dependent, making statistical inference difficult to automate. Although this issue can be partially mitigated by asymptotic normality, the sample size required to achieve asymptotic normality can be very large especially in high-dimensional scenarios.  
For Bayesian methods, their dependence on prior distributions has been a subject of criticism throughout the history of Bayesian statistics, often raising concerns about their fidelity.

 

As a possible way to overcome the drawbacks of frequentist and Bayesian methods, the fiducial
method has been proposed by R.A. Fisher in a series of papers starting from 1930s 
(see <cit.> for a review), which quantifies uncertainty of model parameters by the so-called fiducial distribution. 
Fisher originally introduced this method, motivated by the observation that pivotal quantities permit uncertainty quantification for an unknown parameter  
 in the same way as the frequentist method. However, he encountered difficulties in extending this pivotal quantity-based method to models with multiple parameters. 
It is worth noting that for some models, the fiducial distribution is the same as the posterior distribution derived with Jeffreys' prior, but Fisher argued that the logic behind the Bayesian method is unacceptable because the use of prior is unjustifiable <cit.>. This argument also distinguishes the fiducial method from objective Bayesian methods, even though non-informative priors are used in the latter. 

Fiducial inference was generally regarded as a big blunder by Fisher. However, the goal he initially set, making inference about unknown parameters on the basis of observations <cit.>, has been continually pursued by many statisticians.  Building on early works in sparse deep learning 
<cit.> and adaptive stochastic gradient Markov chain Monte Carlo (MCMC) <cit.>, 
 this paper develops a new statistical inference framework called the extended fiducial inference (EFI), 
 which achieves the initial goal of fiducial inference while possessing
 necessary features like fidelity, automaticity, and scalability that are essential for statistical inference in modern data science.  
 
 
 Our contributions in this work are in three folds: 
 
    
  * Development of the EFI framework: We develop a scalable and effective method for conducting fiducial inference. Our method involves jointly imputing the random errors contained in the data and estimating the inverse function for the model parameters. It ensures that the uncertainty embedded in the data is properly propagated to the model parameters through the estimated inverse function, thereby validating downstream statistical inference. Compared to frequentist and Bayesian methods, EFI provides higher-fidelity inference, especially in the presence of outliers.


  * Innovative statistical framework for semi-supervised learning: EFI provides an innovative   framework of statistical inference for missing data problems, especially in scenarios where missing values are present in response data, as encountered in semi-supervised learning problems. This innovation 
can have profound implications for modern data science, particularly in biomedical research where obtaining labeled data can be costly.   
     

  * Automaticity of statistical inference:  EFI enables automatic statistical inference for complex  models, at least conceptually. It can be as flexible 
as frequentist methods in parameter estimation. However, unlike frequentist methods,  
it eliminates the requirement for theoretical reference distributions  (including asymptotic normality as a special case) in hypothesis testing. 
Compared to Bayesian methods, EFI eliminates the requirement for prior distributions, which can vary depending on the problem or analyst's choice, thus enhancing the fidelity of statistical inference.
 

  
 In summary, with the aid of advanced statistical computing techniques, EFI holds the potential to significantly advance modern data science. Specifically, it provides higher-fidelity inference, introduces an innovative statistical framework for semi-supervised learning, and automates statistical inference for 
 complex models.

 
   
The remaining part of the paper is organized as follows. Section 2 
distinguishes the concepts of frequentist, Bayesian and EFI from the perspective of structural inference  <cit.>. 
Section 3 provides a theoretical framework for EFI. 
Section 4 describes an effective algorithm for performing EFI 
and studies its theoretical properties. 
Section 5 presents some numerical examples validating EFI as a statistical inference method. 
Section 6 presents applications of EFI on semi-supervised learning.
Section 7 presents applications of EFI for complex hypothesis tests. 
Section 8 concludes the paper with a brief discussion. 





§ FREQUENTIST, BAYESIAN, AND EXTENDED FIDUCIAL INFERENCE
 

This section elaborates the conceptual difference between frequentist, Bayesian, and EFI methods from the perspective of structural inference <cit.>. Consider a regression model: 

    Y=f(X,Z,),

where   Y∈ℝ and X∈ℝ^d represent the response and explanatory variables, respectively; ∈ℝ^p represents the vector of unknown parameters; 
and Z∈ℝ represents a scaled random error that follows  

 a known distribution denoted by π_0(·). 
Suppose that a random sample of size n has been collected from the model, denoted by {(y_1,x_1), (y_2,x_2),…,(y_n,x_n)}, and our goal is to quantify uncertainty of  based on the collected samples (also known as observations). 

In the view of structural inference <cit.>, we can express the observations  {(y_1,x_1), (y_2,x_2),…, (y_n,x_n)} 
in the data generating equation as follow:  

    y_i=f(x_i,z_i,),    i=1,2,…,n.



This system of equations consists of n+p unknowns, namely, {, z_1, z_2, …, z_n
}, while there are only n equations. Therefore, the values of  cannot be uniquely determined by the data-generating equation, which gives the source of uncertainty of the parameters as illustrated by Figure <ref>. 
For convenience, we will refer to z_1, z_2,…, z_n as latent variables in the context of data-generating equations, while still calling them random errors when appropriate.

 




















  
Frequentist Methods
The frequentist methods treat  as fixed unknowns. To solve for  from the undetermined system (<ref>), they  often impose a  constraint on the system such that the latent variables can be dismissed and  can be uniquely determined. 


For example, the maximum likelihood estimation method works under the constraint that the joint likelihood function of the samples, or equivalently, the likelihood of 
 {z_1,z_2,…,z_n}, is maximized. 
As an illustration, let's consider the linear regression model:
 
    y_i=x_i^T + σ z_i,    i=1,2,…,n,

 where ∈ℝ^p-1 is the regression coefficient vector, 
 σ∈ℝ^+ is a positive scale parameter, and z_1,z_2,…,z_n are  
 i.i.d standard Gaussian random variables. For this model, 
the maximum likelihood estimation method is to solve  for :=(,σ) subject to the constraint 

    ∏_i=1^n ϕ(z_i)=max_(z̃_1,z̃_2,…,z̃_n)∈ℝ^n∏_i=1^n ϕ(z̃_i),

where ϕ(·) denotes the standard Gaussian density function. As it turns out, this is equivalent to solving the optimization problem:

    max_(,σ)∑_i=1^n logϕ( y_i-x_i^T/σ),

and the resulting estimator is given by 

    =(_n^T _n)^-1_n^T _n,   σ̂^2=1/n∑_i=1^n (y_i-x_i^T)^2,
 
where  _n=(y_1,…,y_n)^T and _n=(x_1,x_2,…,x_n)^T. 

Another example of frequentist methods is  moment estimation, which solves for  under the constraint that the sample moments are equal to the population moments. 
For the model (<ref>), the moment constraint can be expressed as   

    ∑_i=1^n y_i^k= ∑_i=1^n ∫ [f(x_i,z,)]^k π_0(z) dz,    i=1,2,…, p,

where the latent variables z_1,z_2,…,z_n are dismissed via integration. 













 

Let  denote an estimator of . The frequentist method assesses the uncertainty of  in an unconditional mode, where the distribution of  is derived based on the preassumed distribution π_0(z) instead of the random errors z_1,z_2,…,z_n realized in the observations.
For example, considering the MLE given in (<ref>), one can derive that 
∼ N(, σ^2 (_n^T _n)^-1) and n σ̂^2/σ^2∼χ^2(n-p+1) based on the preassumed Gaussian distribution for the random errors.
This unconditional mode makes the inference procedure challenging to automate; in particular, the distribution of  is problem-dependent and generally difficult to derive. Additionally, the constraints used for the solution of  might be violated by observations. For example, when outliers exist, the maximum likelihood constraint might not hold, and the resulting MLE can significantly differ from the true value of .

 
 Refer to Section <ref> for numerical examples. 
  



  
Bayesian Methods
  In contrast to frequentist methods, Bayesian methods treat  as random variables and circumvent the issue of latent variables 
  by adopting a conditional approach. Specifically, Bayesian methods assume that  follows a prior distribution, and quantify uncertainty of  based on the  conditional distribution (also known as the posterior distribution):

    π(|(y_1,x_1),(y_2,x_2),…,(y_n,x_n)) = ∏_i=1^n p(y_i|x_i,) π()/∫∏_i=1^n p(y_i|x_i,) π() d,
 
where  p(y_i|x_i,) denotes the likelihood function of y_i, and π() represents the prior distribution of . 
The dependence of the inference on the prior distribution has been subject to criticism throughout the history of Bayesian statistics, as the prior distribution introduces subjective elements that may affect the fidelity of statistical inference.
 


  
Extended Fiducial Inference
 Let _n:={z_1,z_2,…,z_n} denote the collection of latent variables, and let G(_n,_n,_n) denote an inverse function for the solution of  in the system (<ref>). 
 
 
 
 As a general computational procedure, EFI jointly imputes  _n and estimates G(_n,_n,_n), and then quantifies the uncertainty of  
 based the estimated inverse function and the imputed values of _n, where 
 the estimated inverse function serves as an uncertainty propagator from _n to . 
Technically, EFI approximates G(_n,_n,_n) using a sparse deep neural network (DNN) <cit.>, and employs an adaptive stochastic gradient MCMC algorithm  <cit.> to jointly simulate the values of _n and estimate the parameters of the sparse  DNN.



 While treating  as fixed unknowns, EFI distinguishes itself from frequentist methods by conducting inference for  in a conditional mode and sidestepping the imposition of any constraints on the latent variables. Additionally, unlike Bayesian methods, EFI eliminates the need for 
 placing a prior distribution on . In summary, EFI aims to make statistical inference of   based solely on observations.
 

 


  
Related Works

During the past several decades, there have been quite a few works on statistical inference with the attempt to achieve the goal of fiducial inference, although some gaps remain. These works are briefly reviewed in what follows. 



Generalized Fiducial Inference (GFI).
  Like EFI, GFI <cit.> also attempts to solve the data generating equation, but employs an acceptance-rejection procedure 
  
  similar to the approximate Bayesian computation (ABC) algorithm <cit.>. As an illustration, let's consider model  (<ref>), for which the acceptance-rejection procedure 
   consists of the following steps: 
  
  
  (a) (Proposal) Generate _n=(z̃_1, z̃_2,…, z̃_n)^T from the Gaussian distribution N(0,I_n).
  
  (b) (-fitting) Find the best fitting parameters =min__n-_n-σ_n, where · denotes an appropriate norm, and compute  the fitted value _n=_n +σ̃_n.
  
  (c) (Acceptance-rejection) Accept   if _n-_n≤ϵ for some pre-specified small value ϵ, and reject otherwise. 

Subsequently, statistical inference is made based on the accepted samples of . However, 
as n increases, this procedure can become 
extremely inefficient due to its decreasing   acceptance rate. 


As a potential solution to resolving this computational issue, the limiting distribution of accepted  (as ϵ→ 0) was derived in <cit.>. However, as shown in <cit.>, the limiting 
distribution  depends on the norm used in the above procedure.
Furthermore, for many problems, direct simulation of the limiting distribution might be challenging, 
  as shown in <cit.>, which involves the calculation of the determinant of an n× n matrix 
 at each iteration.

Quite recently, <cit.> proposed replacing the -fitting step of the acceptance-rejection procedure 
with a mapping G̃: (_n,_n,_n)→ pre-learned using a DNN. However, this replacement cannot improve the acceptance rate of , since  _n is still proposed from an independent trial distribution. Other concerns about the replacement include the consistency of the DNN estimator and its difficulty in dealing with  cases where _n and _n contain missing data. 


Compared to GFI, EFI provides a more feasible computational scheme for conducting fiducial inference, in addition to some conceptual differences in defining the fiducial distribution as discussed later.



  
Structural Inference.
Fraser <cit.> introduced the concept of modeling the data as a function of  parameters and random errors through a structural equation (also known as data generating equation), under which statistical inference would be conditioned on the realized random errors. The structural inference approach  has successfully addressed some difficulties suffered by the pivotal quantity-based fiducial method. In particular, it avoids the issues of improper normalization <cit.> and non-uniqueness <cit.>. 
However, like the Bayesian method, the structural inference method can suffer from the marginalization paradox <cit.> 

that can cause inconsistency of inference. 
It is important to note that the structural equation concept 
has led to a fruitful framework for statistical inference. 
Both GFI and EFI are developed based on it. 


However, EFI differs significantly from structural inference in its treatment of 
. EFI regards  as fixed unknowns, whereas structural inference treats  as variables. As a result, EFI successfully sidesteps the marginalization paradox like a frequentist method.

In EFI,  can be determined only in the limit n→∞, where the inverse function G(_n,_n,_n) derived with finite samples can be understood as a stochastic estimator of  with a random component formed by _n.

 
The Dempster–Shafer theory (see e.g., <cit.>, <cit.>, and <cit.>) and the inferential model (see e.g., <cit.>, <cit.>, and <cit.>)  provide interesting frameworks for statistical reasoning with uncertainty. However, they are not primarily concerned with fiducial inference in the form Fisher conceived. The inferential model method 
 avoids imposing any constraints on the latent variables _n but instead conducts inference for  in an unconditional mode, as discussed in <cit.>. 
 It achieves this by working with a low-dimensional association model,  which is built upon the sufficient or summary statistics for  and includes only a limited number of latent variables. Leveraging this association model, it subsequently constructs a confidence set for  using the Dempster-Shafer theory by considering a set of plausible random errors pre-constructed for the association model in an unconditional mode.
 
  For many statistical models, it yields the same confidence set as the maximum likelihood estimation method. To maintain conciseness of  this review, we omit detailed descriptions for them. 
   

























 





§ EXTENDED FIDUCIAL INFERENCE





 §.§ Extended Fiducial Distribution
 

 
Before introducing the EFI method, we first define the extended fiducial distribution (EFD) as a confidence distribution (CD) estimator <cit.> of b(), 
where b(·) is a function of interest. Let's revisit the data generating equation (<ref>) and begin by making several assumptions.

 
There exists an inverse function G: ℝ^n ×ℝ^n× d×ℝ^n→ℝ^p: 

    =G(_n,_n,_n).
 


In this context, “inverse” implies that if (_n, _n, _n) satisfies 
_n = f(_n, _n, ) for some , 
then   G(_n, _n, _n) = follows.





 From the perspective of parameter estimation, Assumption <ref> implies that the parameters are identifiable given the random error-augmented data 
{_n,_n,_n}. This is generally true when n≥ p, as in this case the system (<ref>) has no more  unknowns than the number of equations by considering _n as known. For the case p>n, we recommend reducing the dimension of the problem through an  application of a model-free sure independence screening procedure. More discussions on this issue can be found at the end of the paper. 


It is worth noting that the inverse function is not necessarily constructed using all n samples. For example, it can be simply constructed by solving any p equations in (<ref>) 
for . This raises an issue about non-uniqueness of G(·).

In what follows, we will study how the non-uniqueness of G(·) 
impacts the statistical inference for the unknowns _n and .


For a given inverse function, we define an energy function: 

    U_n():=U(_n,_n,,G(·)).











To ensure proper inference for the unknowns, the energy function U_n(·) needs to satisfy certain regularity conditions as outlined in Assumptions <ref>-<ref>.

 
  
 The energy function U_n(·) is  non-negative, 
 min_ U_n() exists and equals 0, 
  and U_n()=0 if and only if  _n=f(_n,,G(_n,_n,)). 
 


Let 𝒵_n denote the zero-energy set 


    𝒵_n={∈ℝ^n: U_n()=0 }.



 
If Assumptions <ref>-<ref> hold, then the zero-energy set 
  𝒵_n is invariant to the choice of G(·).
 

Suppose that there exist two inverse functions G_1(·) and G_2(·). Let 
𝒵_n^(1) and 𝒵_n^(2) denote their respective zero-energy sets. 
For any ∈ℝ^n, if ∈𝒵_n^(1), then
_n=f(_n,,G_1(_n,_n,)) holds by Assumption <ref>. 
Let =G_1(_n,_n,). Hence,  (_n,_n,) satisfies the 
data generating equation (<ref>) with the parameter . 

Since G_2(·) is also an inverse function for the data generating equation, we have 
G_2(_n,_n,)= by Assumption <ref>. 
This implies _n=f(_n,,G_2(_n,_n,)),  
and thus ∈𝒵_n^(2) according to Assumption <ref>.
That is, 𝒵_n^(1)⊆𝒵_n^(2) holds. Vice versa, we can show 
𝒵_n^(2)⊆𝒵_n^(1). Therefore,  
𝒵_n^(1)=𝒵_n^(2) and the zero-energy set is invariant 
to the choice of the inverse function. 


 


Let p_n^*(|_n,_n) denote the extended fiducial density function of _n 
on 𝒵_n.
To properly define  p_n^*(|_n,_n), we adopt a
limiting way. More precisely, we first define the conditional distribution 

    p_n^ϵ(|_n,_n) ∝exp{-  U_n()/ϵ}π_0^⊗ n(),

where ϵ>0 represents the temperature, and π_0^⊗ n()=π_0(z_1)×π_0(z_2) ×⋯×π_0(z_n) serves as 
the marginal distribution in the construction of this conditional distribution. 
Then, we define p_n^*(|_n,_n) as the limit 

    p_n^*=lim_ϵ↓ 0 p_n^ϵ.


This type of convergence  has been studied in <cit.>. 
Specifically, the convergence can be studied in  two cases: (a) 
Π_n(𝒵_n)>0 and (b) Π_n(𝒵_n)=0, where
 Π_n(·) denotes a probability measure on (ℝ^n, ℛ)  with ℛ being the Borel σ-algebra and 
  the corresponding density function given by π_0^⊗ n.  




  §.§.§ Case (a): Π_n(𝒵_n)>0

For this case, we follow <cit.> to further assume that U_n() satisfies:
 


Π_n(U_n()<a)>0 for any a>0.

Then, following Proposition 2.2 of <cit.>, it can be shown that  
the limiting probability measure of p_n^ϵ exists and is uniformly distributed 
on 𝒵_n with respect to Π_n.  This is summarized in the following 
Theorem: 


If Assumptions <ref>-<ref> hold and Π_n(𝒵_n)>0, then 
p_n^*(|_n,_n) is invariant to the choice of the inverse function G(·) and the energy function U_n(·), and it is given by   

    d P_n^*(|_n,_n)/d= 1/Π_n(𝒵_n)π_0^⊗ n(),   ∈𝒵_n,

where P_n^* represents the cumulative distribution function (CDF) corresponding to p_n^*. 


The proof of Theorem <ref> follows Proposition 2.2 of 
<cit.> and Lemma <ref> directly, and it is thus omitted.
An example of this case is the logistic regression as discussed in 
Section <ref> of the supplement, for which the energy function is defined as   
 
    U_n()= ∑_i=1^n ρ((z_i-x_i^T G(_n,_n,_n))(2y_i-1) ),

where z_1,z_2,…,z_n iid∼ Logistic(0,1) with the CDF given by F(z)=1/(1+e^-z), and ρ(·) is the ReLU function: ρ(s)=s if s>0 and 0 otherwise.




  §.§.§ Case (b): Π_n(𝒵_n)=0


For this case,  we assume that 𝒵_n forms a manifold in ℝ^n with the highest dimension p.
 Following <cit.>, by the tubular neighborhood theorem <cit.>,  we can decompose ∈𝒵_n as follows: 

    =m(u_1,u_2,…,u_p)+t_1 𝒩(1)+⋯+ t_n-p𝒩(n-p),

where m(u_1,u_2,…,u_p) is local coordinates, and 
𝒩(1), …, 𝒩(n-p) are normalized smooth normal vectors perpendicular to 𝒵_n. Let =(t_1,t_2,…,t_n-p)^T. 
In addition to Assumptions <ref>-<ref>, we assume the following conditions hold: 

























 
    
  (i)  There exists a>0 such that {U_n() ≤ a} is compact. 
    
  (ii) π_0^⊗ n() is continuous, and U_n() ∈ C^3(ℝ^n) is three-time continuously differentiable. 
    
  (iii) 𝒵_n has finitely many components and each component is a compact smooth manifold with the highest dimension p. 
    
  (iv)  π_0^⊗ n() is not identically zero on the p-dimensional manifold,
    and det( ∂^2 U/∂^2())  0 for ∈𝒵_n.
 

  (Theorem 3.1;  <cit.>) 
If Assumptions <ref>-<ref> hold, then  
 the limiting probability measure p_n^* concentrates on the highest dimensional manifold and is given by 
 
    d P_n^*(|_n,_n)/d ν()= π_0^⊗ n() (  det( ∇^2_ U_n()))^-1/2/∫_𝒵_nπ_0^⊗ n() (  det(∇_^2 U_n() )^-1/2 d ν,   ∈𝒵_n,
 
 where ν is the sum of intrinsic measures on the p-dimensional manifold in 𝒵_n.


Lemma <ref> is a restatement of Theorem 3.1 of 
<cit.> and its proof is thus omitted.  
We note that the distribution P_n^* can also be derived using the co-area formula (see e.g., <cit.>, section 3.2.12; <cit.>, Proposition 2; <cit.>, Theorem 1) under similar conditions.


























 
 Given the inverse function G(·), we define the parameter space 
 
    Θ={∈ℝ^p: =G(_n,_n,), ∈𝒵_n},

which represents the set of all possible values of  that G(·) takes when  runs over 𝒵_n. Then 
for any function b(), its EFD associated with the inverse function G(·) can be defined as follows: 

  [EFD of b()] Consider the data generating equation   (<ref>) and an inverse function =G(_n,_n,). 
For any function b() of interest, its EFD associated with the inverse function G(·) is defined as  

    μ_n^*(B|_n,_n)    =∫_𝒵_n(B) d P_n^*(|_n,_n),     ,

where 𝒵_n(B)={∈𝒵_n: b(G(_n,_n,)) ∈ B}, 
and P_n^*(|_n,_n) is given by (<ref>). 
 

Essentially, b(G(_n,_n,_n)) can be considered as an estimator of b(), and 
Eq. (<ref>) represents the CD estimator of b() associated with the inverse function G(·). 
Here we would like to emphasize that viewing μ_n^* as a distribution function of b() (i.e., regarding  as a variable) is not appropriate, as in this context it will easily lead to a Bayesian approach for jointly simulating of (,_n). The resulting sample pair (,_n) will break the inverse mapping (<ref>) and, in consequence, the uncertainty of _n will not be properly propagated to .



For an effective implementation of EFI, we propose the following 
importance resampling procedure: 

 
  (a) (Manifold sampling) For any given inverse function G̃(·), simulate M samples, denoted by 𝒮_M={_1, _2, …,_M}, from  π_0^⊗ n() subject to the constraint U_n ()=0. This can be done using a constrained Monte Carlo algorithm such as constrained Hamiltonian Monte Carlo <cit.>. 
 
  (b) (Weighting) Calculate the importance weight ω_i= (  det( ∇^2_ U_n(_i)))^-1/2 for each sample _i∈𝒮 using an inverse function G(·) of interest.
 
  (c) (Resampling) Draw m samples from 𝒮_M without replacement according to the probabilities: ω_i/∑_j=1^M ω_j for i=1,2,…,M.
 
  (d) (Inference) For b(), find the EFD  associated with G(·) according to 
  (<ref>) based on the m samples obtained in step (c). 


This procedure involves two inverse functions: G̃(·) and 
G(·). By Lemma <ref>, any inverse function G̃(·) 
can be used in step (a) to generate samples from 𝒵_n, 
and this greatly facilitates comparisons of the inference results from 
different choices of G(·). 
If G(·) and G̃(·) are chosen to be the same,  p_n^*(|_n,_n) ∝π_0^⊗ n() (  det( ∇^2_ U_n()))^-1/2 
can also be directly simulated on 𝒵_n 
using a constrained Monte Carlo algorithm. 














 
  (On the flexibility of EFI)

EFI provides a flexible framework of statistical inference. One can adjust the inverse function G(·) and the energy function U_n(·) to ensure that the resulting CD estimator of b() satisfies desired properties, such as efficiency, unbiasedness, and robustness. 
This mirrors the flexibility of frequentist methods, where different estimators of b() can be designed for different purposes. However, its conditional inference nature makes EFI even more attractive than frequentist methods, as it circumvents the need for  derivations of theoretical distributions of the estimators.


Lastly, we note that the fiducial distribution defined above conceptually differs from that defined in GFI <cit.>. 
 Specifically, GFI interprets the fiducial distribution as the -marginal of a distribution defined on the manifold formed by the data generating equations in the joint space of (, _n)∈ℝ^p ×ℝ^n, while EFI interprets it as the -transformation of a distribution 
defined on a subset or manifold formed by the data generating equations in the sample space of _n ∈ℝ^n. Our definition is consistent with the EFI algorithm developed in this paper. 























































 



 §.§ EFI for the Models with Additive Noise


The importance resampling procedure proposed in Section <ref> is general for simulations of p_n^*,  but computing the importance weights can be challenging 
when the sample size n is large. 
Specifically, it involves calculating the determinant of an (n-p)× (n-p)-matrix at each iteration.  
To address this issue, we consider models with additive noise, which represent a broad class of models and have been extensively studied in the context of causal inference (see e.g., <cit.> and <cit.>). 
Additionally, we suggest setting the energy function as 
prescribed in Assumption <ref>-(i), with 
the L_2-norm        
U_n()=_n-f(_n,,G(_n,_n,))^2 as a special case. 
Consequently, for these models, we show that 
the importance weight is reduced to a constant and, therefore, one can simulate from p_n^* 
by directly simulating from π_0^⊗ n() 
using a constrained Monte Carlo algorithm.

   
 (i)  U_n(·) is specified in the form: U_n()=h(J())=∑_i=1^n h(e_i) for 
some function h(·) satisfying ∂ h(J)/∂ J()=0 for any ∈𝒵_n, where J()=_n-f(_n,,G(_n,_n,))=(e_1,e_2,…, e_n)^T, and e_i=y_i-f(x_i,z_i,) for i=1,2,…,n; 
and (ii) the model noise is additive; i.e., the function f(X,Z,) in model (<ref>) is a linear function of Z.  











 
If Assumptions <ref>-<ref> hold, then  
P_n^*(|_n,_n) given in (<ref>) is invariant to the choices of G(·) and U_n(·). Furthermore, P_n^* reduces to a truncated distribution of 
π_0^⊗ n on the manifold 𝒵_n.
 

Under Assumptions <ref>-<ref>, the uniqueness of 𝒵_n 
has been established in Lemma <ref>.  
If U_n(·) is specified as in Assumption <ref>, 
the condition ∂ h(J)/∂ J()=0 on 𝒵_n implies 
 
    ∇_^2 U_n()= (∇_ J())^T ∇_J^2 h(J()) ∇_ J().

Furthermore, with the aid of Assumption <ref>-(iv) 
and the symmetric form of h(·) (with respect to e_i's),
 ∇_J^2 h(J()) reduces to a diagonal matrix of ς I_n-k
for some positive constant ς>0. 
This ensures the factor ς to be canceled out for the numerator and denominator in (<ref>). Consequently,  
P_n^* is invariant to the choice of U_n(·).

To further establish the invariance of P_n^* with respect to the choice of G(·), 
we consider an inverse function 

    G(_n,_n,_n)=(z_1,z_2,…,z_p),

where (z_1,z_2,…,z_p) is obtained by solving the first p equations 
in (<ref>).  Here we assume the solution (z_1,z_2,…,z_p) is unique for the p equations. 
Let =(z_p+1,z_p+2,…,z_n)^T, which corresponds to a transformation 
of  in (<ref>). 
Then, it is easy to verify that at any point ∈𝒵_n, 
 the first p rows of the matrix ∇_ J() ∈ℝ^n × (n-p) are all zero, and 
 the remaining (n-p)-rows forms a (n-p)× (n-p)-diagonal matrix for which 
 the diagonal elements are nonzero and expressed as a function of (_n,) 
 (i.e., a constant function of ) by the assumption 
 that f(X,Z,) is a linear function of Z. 
 
 Therefore, at any point ∈𝒵_n,  ∇_^2 U_n() forms a positive-definite constant matrix with rank n-p;
 and P_n^* in (<ref>) reduces to  a truncated distribution of π_0^⊗ n on 𝒵_n.

 In the same way, we can construct np different inverse functions, 
 each obtained by choosing  a different set of p equations to solve for . Therefore, each of them results in a positive definite matrix  ∇_^2 U_n() and the same distribution P_n^*. 
 For any appropriate linear combination of these inverse functions, which still forms an inverse function, 
 the above result still holds. For the combination case, the desired result can be established via appropriate matrix operations, as illustrated using a linear regression example 
 in Section <ref> of the supplement. 
 
 Finally, we note that for any inverse function, since it solves all n equations, it must also be a solver for a selected set of p equations. 
 By the uniqueness of the solution for p equations, the inverse function 
 can be regarded as  a linear combination
 of these np basis inverse functions.  
 






































 






   
























  
Example 1 Consider the linear regression model (<ref>) again. Let =(,σ^2).
To conduct EFI for , we set G(_n,_n,)=(z_1,z_2,…,z_p):=(^T,σ̂)^T,  a solver for the first p equations in (<ref>), and set the energy function

    U_n()=_n-f(_n,,G(_n,_n,))^2.

  Consequently, we  have 

    J()  =[             Y_1:p-X_1:p-σ̂ Z_1:p; Y_(p+1):n-X_(p+1):n-σ̂ Z_(p+1):n ],    ∇_Z_(p+1):n J()  = [        0; σ̂ I_n-p ],

where Y_1:p, X_1:p and Z_1:p to denote the response, explanatory, and noise variables of the first p samples in the dataset, respectively; likewise, 
Y_(p+1):n, X_(p+1):n and Z_(p+1):n denote the response, explanatory, and noise variables of the last n-p samples. 

 At any ∈𝒵_n, we have =, i.e.,  
 can be treated as constants, and thus 
 ∇_^2 U_n()=2 D̃^T D̃ for some matrix D̃ of rank
 n-p. This yields the result that p_n^*(|_n,_n) is a truncation of 
 π_0^⊗ n on 𝒵_n. 
































































  
Example 1 (continuation) 
 For simplicity, let's first consider the case where σ^2 is known. In this scenario, we set 
 
    G(_n,_n,_n)= (_n^T _n)^-1_n^T (_n-σ_n),

 which utilizes all available data. Then the EFD of , denoted by μ_n^*(|_n,_n,σ^2), is given by N((_n^T_n)^-1_n^T_n, σ^2 (_n^T_n)^-1) after normalizing p_n^*(|_n,_n) on 
𝒵_n,   which coincides with 
 the posterior distribution of  under Jeffery's prior π() ∝ 1.  
Furthermore, the resulting confidence set for  is identical to those obtained 
by the GFI and ordinary least square (OLS) methods. 


Next, let's consider the case where σ^2 is unknown. To find the EFD of σ^2, 
we solve the first p-1 equations for , resulting in the solution: 

    = (_1:(p-1)^T_1:(p-1))^-1_1:(p-1)^T(_1:p-1-σ_1:(p-1)),

where  Y_1:(p-1), X_1:(p-1) and Z_1:(p-1) denote the response, explanatory, and noise variables of the first p-1 samples in the dataset, respectively. 
By substituting  into each of the remaining n-p+1 equations and adjusting with the covariance of the Z-terms,
we obtain a combined 
solution for σ^2: 

    σ̃^2    = (_p:n-_p:n (_1:(p-1)^T_1:(p-1))^-1_1:(p-1)^T_1:p-1)^T Σ^-1  (_p:n-_p:n (_1:(p-1)^T_1:(p-1))^-1_1:(p-1)^T_1:p-1)/ (_p:n-_p:n (_1:(p-1)^T_1:(p-1))^-1_1:(p-1)^T_1:p-1)^T Σ^-1  (_p:n-_p:n (_1:(p-1)^T_1:(p-1))^-1_1:(p-1)^T_1:p-1), 
        := A/W,

where   
and  Y_p:n, X_p:n and Z_p:n denote the response, explanatory, and noise variables of the last n-p+1 samples in the dataset, respectively; and Σ=I_n-p+1+_p:n (_1:(p-1)^T_1:(p-1))^-1_p:n^T, representing the covariance matrix of (_p:n-_p:n (_1:(p-1)^T_1:(p-1))^-1_1:(p-1)^T_1:p-1). Here, A forms an unbiased estimator of (n-p+1)σ^2, 
and W follows 
a χ^2-distribution with a degree-of-freedom of n-p+1. 
Therefore, if we set σ̃^2 as the inverse function G(·) 
for σ^2, the resulting EFD of σ^2 is given by 

    μ_n^*(σ^2|_n,_n)=π_χ^-2_n-p+1(σ^2/A) 1/A,

where π_χ^-2_k(u) denotes 
the density function of an inverse-chi-squared distribution with a degree-of-freedom of k.   
If we use the mean of μ_n^*(σ^2|_n,_n) as an estimator of σ^2, 
 it can be shown that it has a bias of 2/n-p-3σ^2. In contrast, the MLE
 of σ^2 has a bias of -p-1/nσ^2. Therefore, the EFD results in a smaller bias than the MLE when n>(p+3)(p-1)/(p-3). Note that, as stated in Remark <ref>, we 
 can adjust σ̃^2 by the factor n-p-3/n-p-1 to make the mean of the EFD 
 unbiased for σ^2, if desired. 









Finally, we can obtain the EFD of  by completing the integration:

    μ_n^*(|_n,_n)= ∫μ_n^*(|_n,_n,σ^2) μ_n^*(σ^2|_n,_n) dσ^2,

which is a multivariate non-central t-distribution t(_, _, ν_) with the parameters given by 

    _=(_n^T_n)^-1_n^T_n,   _=A/n-p+1 (_n^T_n)^-1,
      ν_=n-p+1.

The mean and covariance matrix of the EFD is given by (_n^T_n)^-1_n^T_n and A/n-p-1 (_n^T _n)^-1. 




It is worth noting that our EFD (<ref>) matches the result obtained with OLS. The latter often presents the result as 

    _n^T(I_n-_n(_n^T _n)^-1_n^T)_n/σ^2∼χ^2_n-p+1,

where the numerator forms an unbiased estimator of (n-p+1)σ^2. Similarly, in EFI,  A serves as 
an unbiased estimator of (n-p+1)σ^2. Also, the EFD (<ref>) can be represented 
as an inverse-Gamma distribution IG(α_g, β_g) 
with α_g=n-p+1/2 and β_g=A/2, which is the same as the GFI solution <cit.> except for the expression of A. 

   The EFD derivation procedure described in  
Example 1 can be extended to general nonlinear regression problems with additive noise. 
Consider the model _n=f(_n,)+σ_n:=_y+σ_n, where _n is
assumed to follow a known distribution symmetric about 0. First, let's assume that 
σ^2 is known. 
Let T(_n) be the OLS estimator of , which makes use of all n samples. 
Let π_T denote the distribution of T(_n), and  
 let μ_T^(i)=η^(i)(_y) denote its i^th moment for i=1,2,…, k.  
 We can regard T(_n- σ) as an 
 inverse function for . Consequently, by (<ref>), 
 the EFD of , associated with T(_n- σ), has the i^th moment given by η^(i)(_n) for i=1,2,…,k. Furthermore, EFI shares the same distribution π_T as the frequentist method for quantifying the uncertainty of .
 
 If σ^2 is unknown, we can follow the same procedure as described in Example 1 to find 
 the EFD for σ^2. Finally, we can obtain the EFD for  by completing an integration similar to (<ref>).
 




















































































































 





 

























 


































 









































































 




























































































Theorem <ref> implies that for an additive noise model, if any 
inverse function G̃(·) is known, then 
p_n^*(|_n,_n) can be directly simulated from π_0^⊗ n() subject to the constraint U_n()=0 using a constrained Monte Carlo algorithm.  
Furthermore, for b(), the empirical EFD 
associated with a known inverse function G(·) can be constructed based on the samples simulated from p_n^*(|_n,_n).




§ EXTENDED FIDUCIAL INFERENCE WITH A SPARSE DNN INVERSE FUNCTION
 

As implied by Theorem <ref>, Lemma <ref>, 
and Theorem <ref>,  conducting fiducial inference requires finding appropriate inverse functions. However, in practice, these inverse functions are typically very difficult to determine. 
To address this issue,  we propose approximating G̃(·) with a sparse DNN and employing an adaptive stochastic gradient MCMC algorithm to simultaneously simulate from p_n^*(|_n,_n) and train the sparse DNN.  Then an empirical EFD of b() associated with G(·)=G̃(·) can be constructed based on the -samples simulated from p_n^*(|_n,_n).  
We call this proposed algorithm the EFI-DNN algorithm. 




 §.§ The EFI-DNN Algorithm
 















 

The entire structure of the algorithm is depicted by the so-called EFI network, as shown in Figure <ref>. 
Let _i:=ĝ(y_i,x_i,z_i,) denote the DNN prediction function parameterized by the weights  in the EFI network, and let 

    _n:=1/n∑_i=1^n _i=1/n∑_i=1^n ĝ(y_i,x_i,z_i,),

which works as an estimator for the inverse function G(_n,_n,_n). 
Henceforth, we will call _n an EFI-DNN estimator of G(_n,_n,_n). 
The EFI network has two output nodes defined, respectively, by 

    e_i1   :=_i-_n^2, 
    
    e_i2   :=d(y_i,ỹ_i):=d(y_i,x_i, z_i, _i),

where ỹ_i=f(x_i,z_i,_i),  the function f(·) is as defined in (<ref>), and d(·) is a function that measures the difference between y_i and ỹ_i. With a slight abuse of notation, we rewrite 
d(y_i,ỹ_i) as a function of
y_i, x_i, z_i, and _i. For example, for  normal linear/nonlinear regression, we define 

    d(y_i,x_i,z_i,_i)=y_i-f(x_i,z_i,_i)^2.

For logistic regression, we define 
d(y_i,x_i,z_i,_i) via a ReLu function, see Section <ref> of the supplement.
  
For the EFI network, we consider  as the parameters to estimate, _n as the latent variable (or missing data) to impute, (_n,_n) as the observed data (or incomplete data), and (_n,_n,_n) as the complete data. Regarding the EFI network, we have a few further remarks.  

 
The DNN in the EFI network is a fully-connected feedforward neural network, which maps (y_i,x_i,z_i) to _i for each i ∈{1,2,…,n}. Both the depths and widths of the DNN can increase with the sample size n but under a constraint as given in Assumption <ref>-(ii-1) (in the supplement). To ensure Assumption <ref>-(ii), the activation function needs to be continuously differentiable and,   
therefore, can be chosen from options like tanh, softplus or sigmoid. In practice, the ReLU activation function can also be used, as the resulting energy function is non-continuously differentiable at isolated points only. Consequently, for case (b), we will have
(<ref>) holding almost surely, 
as implied by the proof provided in  
<cit.>; for case (a),   (<ref>)  still holds as the continuously differentiability condition is not required. 



To address the potential overfitting issue in the DNN, we treat  in a Bayesian approach. We impose a sparse prior on , as given in (<ref>), based on the sparse deep learning theory in <cit.>.
However, this Bayesian treatment is optional, as   can still be consistently estimated 
within the frequentist framework when the training sample size n is sufficiently large.
Furthermore, as discussed in Remark <ref>, the prior hyperparameters can be entirely determined by the data through cross-validation, aligning the EFI-DNN algorithm with the principle of fiducial inference. 
Sparse learning enables the EFI-DNN algorithm to exhibit robust performance 
across a wide range of DNNs with different depths and widths, provided they possess sufficient capacity to approximate desired inverse functions. Regarding the interpretability of the sparse DNN, we refer to <cit.> and <cit.>. 
In the context of EFI networks, the sparse DNN provides a parsimonious approximation to the inverse function. If the inverse function is a sparse neural network function, then its structure 
can be consistently recovered (up to  some loss-invariant transformations).



While the EFI network shares a similar structure with the fiducial autoencoder used in <cit.>, the DNNs in the two works are trained in  different ways. In <cit.>, the DNN is pre-trained using data simulated from the model with a wide range of parameter values. In the present work, the DNN is trained concurrently with the imputation of latent variables. 








 Let π() denote the prior density function of , and let π(_n,_n|_n,) denote 
 the conditional density function of (_n,_n) given (_n,).  
 The form of π() will be detailed later; as discussed in Remark <ref>, π() should be chosen such that _n forms a consistent estimator for the inverse mapping  G(_n,_n,_n).
 We propose to estimate  by maximizing the posterior distribution 
 π(|_n,_n) ∝π() ∫π(_n,_n|_n,) d_n. 
 This can be done by solving the equation
 
    ∇_logπ(|_n,_n)=0.

  
  
 Further, by the Bayesian version of Fisher's identity, see Lemma 1 of <cit.>,  (<ref>) can be expressed as  
 
    ∇_logπ(|_n,_n)=∫∇_logπ(|_n,_n,_n) π(_n|_n,_n,) d=0.
 
 To define  π(|_n,_n,_n) and π(_n|_n,_n,), we first define a scaled energy function for the distribution  π(_n|_n,_n,), up to an additive constant and a multiplicative constant:  
 
    Ũ_n(_n,;_n,_n)=η∑_i=1^n _i-_n^2 + ∑_i=1^n 
     d(y_i,x_i,z_i,_i),

where the first term serves as 
a penalty function enforcing _i's to converge to the same value, 
 and η>0 is a regularization parameter.
This penalty allows us to address possible non-uniqueness  of  
the inverse functions _i=ĝ(y_i,x_i,z_i,) for i=1,2,…,n. 
Let 

    π(_n|_n,_n,) = C e^-λŨ_n(_n,;_n,_n),

for some constants C>0 and λ>0. 
Then we have the following conditional distributions: 


    π(|_n,_n,_n)    
    ∝π() e^- λŨ_n(_n,;_n,_n) , 
    
     π(_n|_n,_n,)    
     ∝π_0^⊗ n(_n) e^- λŨ_n(_n,;_n,_n),
 
where λ is a tuning  parameter resembling the inverse of the temperature 
in (<ref>), and π_0^⊗ n(_n) is the marginal distribution of _n in the space  ℝ^n. 
With respect to the EFI network, we call π(_n|_n,_n,), π(|_n,_n,_n), and π(_n|_n,_n,)
 the complete-data likelihood function, the complete-data posterior distribution, and the missing-data predictive distribution, respectively. 




 





Alternative to (<ref>), we can define the energy function as

    Ũ_n^'(_n,; _n,_n) = η∑_i=1^n_i- _n ^2 +  ∑_i=1^n d(y_i,x_i,z_i,_n).

Without confusion, we will refer to the EFI-DNN algorithm with the energy functions (<ref>) and (<ref>) as EFI-a (alternative version) and EFI (default version), respectively, in the remaining of the paper. Compared to (<ref>), (<ref>) is more regular, where the fitting errors are assumed to be mutually independent given . 
As λ→∞,
EFI-a and EFI are asymptotically equivalent, leading to the same zero-energy set. 



With the distributions given in (<ref>), Eq. (<ref>) is now well defined and 
can be solved using an adaptive stochastic gradient MCMC algorithm 
<cit.>. 

The algorithm works by iterating between the following two steps, where k indexes the iterations: 
 

  (a) (Latent variable sampling) Generate _n^(k+1) from a transition kernel induced by a stochastic gradient MCMC algorithm. For example, we can simulate _n^(k+1) 
using the stochastic gradient 
Langevin dynamics (SGLD) algorithm <cit.>: 

    _n^(k+1)   = _n^(k) +ϵ_k+1∇__nlogπ(_n^(k)|_n,_n,^(k)) +√(2 τϵ_k+1)^(k+1),

where ^(k+1)∼ N( 0, I_d_) is a standard Gaussian random vector of dimension d_, ϵ_k+1 is the learning rate,  
∇__nlogπ(_n^(k)|_n,_n,^(k)) 
denotes an unbiased estimator of ∇__nlogπ(_n^(k)|_n,_n, ^(k)), 
and τ is the temperature that is generally set to 1 in simulations. 
 
 
  (b) (Parameter updating) Update the estimate of  by stochastic gradient descent (SGD):  
 
    ^(k+1)=^(k)+ γ_k+1/n∇_logπ(^(k)|_n,_n,_n^(k+1)),

 where γ_k+1 denotes the step size of 
 stochastic approximation <cit.>, and 
 ∇_logπ(^(k)|_n,_n,_n^(k+1)) denotes an unbiased estimator of ∇_logπ(^(k)|_n,_n,_n^(k+1)).
 
 
 The algorithm is referred to as “adaptive” as the transition kernel in step (a) changes along with the update of .
 Applying the adaptive SGLD algorithm to the EFI network leads to Algorithm <ref>, where the parameter updating step is implemented with mini-batches, and a fiducial sample collection step is added. Note that, given the current estimate of ,
 the latent variable sampling step can be executed in parallel for each observation (x_i,y_i).
 
  Therefore, the whole algorithm is scalable  
  with respect to big data. 

 
  
  

 
 



 §.§ Convergence Theory of the EFI-DNN Algorithm
 

To indicate the dependency of  on the sample size n, we rewrite   as _n in this subsection.
We note that the theoretical study is conducted under the assumption 
that the EFI network has been correctly specified such that there exists a sparse solution _n^*, at which  (_n,_n,_n^*) can be generated from the EFI network; specifically, _n^* ∼π(|_n,_n,_n^*) holds, where _n^* represents the values of the latent variables realized in the observations. 
The convergence of the EFI-DNN algorithm is studied in a few steps. 
First, we show in Theorem <ref> that _n^(k) -_n^* p→ 0 as k→∞, where _n^* is a solution to (<ref>) and p→ denotes convergence in probability. 
Second, we show in Theorem <ref> 
that _n^(k) converges weakly to π(_n|_n,_n,_n^*) 
in 2-Wasserstein distance as k →∞. 




 Third,  we show in Theorem <ref> and the followed discussions that  with 
an appropriate choice of the prior distribution π(_n) and as n→∞ and λ→∞, ĝ(y_i,x_i,z_i,_n^*) constitutes a consistent estimator of ^* and, 
subsequently, the EFI-DNN estimator 

    _n^*:=1/n∑_i=1^n ĝ(y_i,x_i,z_i,_n^*),

constitutes a consistent estimator for the inverse mapping G(_n,_n,_n).
By summarizing the three theorems, we conclude that the EFI-DNN algorithm 
 leads to valid uncertainty quantification for .  
Finally, we show that if _n^* is consistent, π(|_n,_n,_n^*)
is reduced to the extended fiducial distribution of _n as defined in Section <ref>.





  §.§.§ Convergence of Algorithm <ref>


  Suppose Assumptions <ref>-<ref> (in the supplement) hold. If we set the learning rate sequence {ϵ_k: k=1,2,…} and the step size sequence {γ_k: k=1,2,…} in the form ϵ_k=C_ϵ/c_ϵ+k^α and 
γ_k=C_γ/c_γ+k^β for some constants C_ϵ>0, c_ϵ>0, C_γ>0 and c_γ>0, α,β∈ (0,1], and β≤α≤min{1, 2 β}, then there exists a root _n^* ∈{: ∇_logπ(|_n,_n)=0} such that 

    𝔼_n^(k)-_n^*^2 ≤ξγ_k,    k ≥ k_0,

for some constant ξ>0 and iteration number k_0>0.
 

Since the adaptive SGLD algorithm can be viewed as a special case of the adaptive pre-conditioned SGLD algorithm <cit.>, Theorem <ref> can be proved by following the proof of 
Theorem A.1 of <cit.> with minor  modifications. Regarding the convergence rate of the algorithm, <cit.> provides an explicit form of ξ. To make the presentation concise, we omit it in the paper.  
 

Let π^*=π(_n|_n,_n,_n^*), let T_k = ∑_i=0^k-1ϵ_i+1,  and let μ_T_k denote the probability law of _n^(k).
   Theorem <ref> establishes convergence of μ_T_k in 2-Wasserstein distance. 

   
Suppose Assumptions <ref>-<ref> (in the supplement) hold, and 
{ϵ_k} and {γ_k} are set as in Theorem <ref>. Then,   
for any k ∈ℕ, 

    𝕎_2(μ_T_k, π^*) ≤ (Ĉ_0 δ_g^1/4+C̃_1 γ_1^1/4)T_k + Ĉ_2 e^-T_k/c_LS,

for some positive constants Ĉ_0, Ĉ_1, and Ĉ_2, where 𝕎_2(·,·) denotes the 2-Wasserstein distance, c_LS denotes the logarithmic Sobolev constant of π^*, and δ_g is a coefficient 
as defined in Assumption <ref> and reflects the variation of the stochastic gradient ∇__nlogπ(_n^(k)|_n,_n,^(k)).


We use the full data in the sampling step such that δ_g=0, choose α∈ (0,1], 
and choose γ_1 ≺1/T_k^4 for any T_k, which ensures 𝕎_2(μ_T_k, π^*) → 0 as k→∞. 




  §.§.§ On the Consistency of _n^*


 Let 𝒲_n ⊂ℝ^d_w denote the space of _n, where d_w denotes the dimension of _n. Let each component of _n be subject to a truncated mixture Gaussian distribution with the density function given by
 
    π(w_n^(i)) =  ρ_n f(w_n^(i);0,σ_1,n^2)+(1-ρ_n) f(w_n^(i);0,σ_0,n^2),    w_n^(i)∈𝒲_n^(i),    i=1,2,…,d_w,

 where  𝒲_n^(i)⊂ℝ denotes the ith component of 𝒲_n, 
 ρ_n is the mixture proportion,  σ_0,n<σ_1,n,  the density function of each component of the mixture distribution is given by 
 
    f(w;0,σ^2)=ϕ(w/σ)/∫_𝒲_n^(i) [ρ_n ϕ(w/σ_1,n)+(1-ρ_n) ϕ(w/σ_0,n)] dw,

 and ϕ(·) denotes the standard Gaussian density function.
 All components of _n are a priori independent. 
 In our experience, the weights of DNNs often cluster around a small subset near the origin 0 in the space ℝ^d_w. Therefore, it is reasonable to constrain 𝒲_n to a compact set, as stipulated in Assumption <ref>.
 
 To establish the consistency of _n^*, we first define 
 
    (_n|_n^*):=1/nlogπ(_n,_n^*|_n,_n)+1/nlogπ(_n),

 where _n^* ∼π(|_n,_n,_n^*) as defined previously.  
Therefore,
 
    _n^*:=max__n∈𝒲_n(_n|_n^*),

 is also the global maximizer of  the log-posterior logπ(_n|_n,_n,_n^*), given the pseudo-complete data. Further, we define 
 
    (_n|_n^*)    :=1/n∫logπ(_n,_n^*|_n,_n) dπ(_n^*|_n,_n,_n^*) +1/nlogπ(_n) 
       =  1/n{logπ(_n|_n,_n) - ∫logπ(_n^*|_n,_n,_n^*)/π(_n^*|_n,_n,_n) d π(_n^*|_n,_n,_n^*) 
        + ∫logπ(_n^*|_n,_n,_n^*) d π(_n^*|_n,_n,_n^*)+c },

 where c=log∫_𝒲_nπ(_n|_n,_n) π(_n)d_n is the log-normalizing constant of the posterior π(_n|_n,_n).
 In the derivation of (<ref>), _n can be ignored for simplicity as it is constant. For simplicity of notation, we let 
 D_KL(_n)= ∫logπ(_n^*|_n,_n,_n^*)/π(_n^*|_n,_n,_n) d π(_n^*|_n,_n,_n^*) be the Kullback-Leibler divergence between π(_n^*|_n,_n,_n^*) and 
 π(_n^*|_n,_n,_n) in what follows. 
 
 
 
 
 
   

 Let Q^*(_n)=𝔼(logπ(Y,Z|X,_n))+1/nlogπ(_n), where the expectation is taken with respect to the joint distribution of (X,Y,Z). 
 Further, by Assumption <ref> and the weak law of large numbers, 

    1/nlogπ(_n|_n,_n,_n)-Q^*(_n)p→ 0,

holds uniformly over the parameter space 𝒲_n. Assumption <ref> restricts the shape of Q^*(_n) around the global maximizer, which cannot be discontinuous or too flat. Given nonidentifiability of the neural network model, see e.g. <cit.>, we have implicitly assumed that each _n  is unique up to the loss-invariant transformations, e.g., reordering the hidden neurons of the same hidden layer and simultaneously changing the signs of some weights and biases. The same assumption has often been used in theoretical studies of neural networks, see e.g. <cit.> and  <cit.>. 

On the other hand,

by Theorem 1 of <cit.>, under some regularity conditions 
we have 
 
    sup__n∈𝒲_n| (_n|_n^*)-(_n|_n^*) | p→ 0,   .
 
Putting (<ref>) and (<ref>) together and assuming 
that Q^*(_n) satisfies Assumption <ref>, then   

we have the following lemma, whose proof is given in the supplement. 


   Suppose Assumptions <ref>-<ref> (in the supplement) hold, and
 π(_n,_n|_n,_n) is continuous in _n. 
 If _n^* is unique, then _n^* that 
 maximizes π(_n|_n,_n) and  minimizes D_KL(_n) is unique and, subsequently, 
_n^*-_n^*p→ 0 holds as n →∞.
 




















The uniqueness of _n^*, up to some loss-invariant transformations, can be ensured by the consistency of the posterior 
π(_n|_n,_n,_n) as established in Theorem <ref> with an appropriate prior π(_n). 
The condition minimizing D_KL(_n) is generally 
implied by Ũ_n(_n,_n;_n,_n)=0 provided the consistency 
of _n^*, and the convergence of _n^* to a maximum of π(_n|_n,_n) is generally implied 
by the Monte Carlo nature of Algorithm <ref>. 
Therefore, by Theorem <ref>, if _n^(k) converges and 
Ũ_n(_n,_n^(k);_n,_n) converges to 0, we would 
 have _n^*-_n^*p→ 0, provided that  
the prior has been appropriately chosen such that the posterior consistency holds and ĝ(y_i,x_i,z_i,_n^*) constitutes a  consistent estimator 
of ^*.

Suppose our choice of the prior π(_n) ensures that  
the posterior consistency holds and 
ĝ(y_i,x_i,z_i,_n^*) is consistent for ^*.

By Lemma <ref>, ĝ(y_i,x_i,z_i,_n^*) 

would also be consistent for ^*, provided  
ĝ(·) is continuous.

The posterior consistency and the consistency of ĝ(y_i,x_i,z_i,_n^*) can be proved based on the results of <cit.>.  This is summarized in Theorem <ref>, whose proof
can be found in the supplement. Note that working on _n^* is simpler than working on _n^*, as the former is based on the complete data.  

 



















 








   
  
 Suppose that π(_n) is a truncated mixture Gaussian prior distribution 
  as specified in (<ref>) and  Assumptions <ref>-<ref> (in the supplement) hold. Then, under the limit  λ→∞, 
  the posterior consistency holds for   π(_n|_n,_n,_n) 
  and the inverse mapping estimator ĝ(·) (with  either the energy function (<ref>) or (<ref>)) constitutes a consistent estimator for the model parameters, i.e., 

    ĝ(y,x,z,_n^*) -^*p→ 0,   ,

 where ^* denotes the fixed unknown parameter values, and (y,x,z) denotes a generic element of (_n,_n,_n). 


Following from Theorem <ref>, we immediately have 
1/n∑_i=1^n ĝ(y_i,x_i, z_i, _n^*)-^*p→ 0 as n→∞. As a slight relaxation of Assumption <ref>, we can write (<ref>) as 

    ^*=lim_n→∞ G(_n,_n,_n),

where _n is assumed to be known.




For example,  consider the normal mean model

    y_i=+z_i,    z_i ∼ N(0,1),    i=1,2,…,n,

for which G(_n,_n)=∑_i=1^n (y_i- z_i)/n ≡^*   and, therefore,  (<ref>) holds trivially. By combining the above two limits, we have

    1/n∑_i=1^n ĝ(y_i,x_i, z_i, _n^*)-G(_n,_n,_n)p→ 0,   ,

i.e., the EFI-DNN estimator _n^*:=1/n∑_i=1^n ĝ(y_i,x_i, z_i, _n^*) is consistent for the inverse mapping G(_n,_n,_n).  
Further, by Slutsky's theorem, the uncertainty of _n can be propagated to  via
the EFI-DNN estimator. Therefore, 
the confidence distribution of  can be approximated by 

    μ̃_n(d)= 1/∑_k=1^δ__n^*,k (d),   ,

where δ_a stands for the Dirac measure at a given point a, 
_n^*,k:= 1/n∑_i=1^n ĝ(x_i,y_i, z_i^*,k, _n^*), and 
_n^*,k:=(z_1^*,k,z_2^*,k,…,z_n^*,k) for k=1,2,…,  denote  random draws from the distribution 
π(_n|_n,_n,_n^*) under the limit setting of λ.


In this paper, although we set both the learning rate and step size sequences to decay with iterations, for which we particularly set 0.5<β≤α<1, we can still treat (_n^(k),_n^(k)) approximately equally weighted by Theorem 2 of <cit.> and some classical results of stochastic approximation MCMC (see e.g., Theorem 3.3 of <cit.>). That is, we can approximate 
the confidence distribution of  by 

    μ̂_n(d)= 1/∑_k=1^δ__n^k (d),   ,

where _n^k:= 1/n∑_i=1^n ĝ(x_i,y_i, z_i^k, _n^(k)), 
_n^(k):=(z_1^k,z_2^k,…,z_n^k), and (_n^(k),_n^(k)) denotes 
the sample and parameter estimate produced by Algorithm 
<ref> at iteration k.  Some weighted estimation schemes, see e.g. <cit.>, 
also work, but involve extra computation. 
 
 
   
 To obtain a consistent EFI-DNN estimator for the inverse mapping, we impose a truncated mixture Gaussian prior (<ref>) on _n.  It is worth noting that the hyperparameters of the prior distribution can be entirely determined from the data. Specifically, we can employ cross-validation to determine their values while constraining their orders to meet Assumption <ref>-(iv). We refer to <cit.> for the setup of the cross-validation procedure which, together with the sparse DNN approximation theory established above, ensures consistency of 
 the inverse mapping estimator.
 This consistency property significantly mitigates the impact 
 of the prior distribution on downstream inference, aligning the EFI-DNN algorithm with the principle of fiducial inference. When the sample size n is 
 much larger than the dimension of , we can treat  in a frequentist way. Mathematically, this is equivalent to setting π(^(k)) ∝ 1 in (<ref>) when running Algorithm <ref>.
 





  §.§.§ On the Property of π(|_n,_n,_n^*)


  We are now to study the property of π(|_n,_n,_n^*).
  Consider the energy function defined in (<ref>) again. For convenience, we rewrite it as
   
    Ǔ_n()= ηĝ(y_i,x_i,z_i,_n^*) -
       1/n∑_i=1^n ĝ(y_i,x_i,z_i,_n^*) ^2 
       + ∑_i=1^n 
       d(y_i, x_i, z_i, ĝ(y_i,x_i,z_i,_n^*)),

   where we replace _i's and _n with their DNN expressions. Define 
   
    𝒵_Ǔ_n={∈ℝ^n: Ǔ_n()=0 }.

   
   Let Π_n denote a probability measure on (ℝ^n, ℛ), where ℛ is the Borel σ-algebra, 
   and let π_0^⊗ n be the corresponding density function. 
   Further, we rewrite  π(_n|_n,_n,_n^*) 
   as the following:   
   
    p_n,λ(|_n,_n) ∝π_0^⊗ n()
        e^-λǓ_n().

   A direct application of the theory in 
   <cit.> to (<ref>) leads to the following lemma, for 
   which  Assumptions <ref>-<ref> will be be justified in Remark  <ref>. 
   
  (Proposition 2.2 and Theorem 3.1 of <cit.>)  Suppose that the EFI network, the energy function Ǔ_n(), the probability measure Π_n, and the zero-energy set 𝒵_Ǔ_n satisfy Assumptions <ref>-<ref>.
 
 
  (a) If Π_n(𝒵_Ǔ_n)>0, then 
 lim_λ→∞ p_n,λ(|_n,_n) is given by 
 
    P_n^*(|_n,_n)/d=1/Π_n( 𝒵_Ǔ_n)π_0^⊗ n(), 
       ∈𝒵_Ǔ_n.

  
  (b) If Π_n(𝒵_Ǔ_n)=0, 
 
 then  lim_λ→∞ p_n,λ(|_n,_n) is given by 
  
    P_n^*(|_n,_n)/dν= π_0^⊗ n() (  det∇_^2 Ǔ_n()) () )^-1/2/∫_𝒵_Ǔ_nπ_0^⊗ n() ( 
        det∇_^2 Ǔ_n()) () )^-1/2 d ν,   ∈𝒵_Ǔ_n,

   where ν is the sum of intrinsic measures on 
   the  p-dimensional manifold in 𝒵_Ǔ_n.
  
  

     
 The conditions specified in Assumptions <ref>-<ref> are readily met by the EFI network. The existence of the minimum min_Ǔ_n()=0 is asymptotically guaranteed by the consistency of ĝ(y_i,x_i,z_i,_n^*). 
 In particular, we have Ǔ_n(_n^*) p→ 0 as n →∞.
    The condition Π_n(𝒵_Ǔ_n)>0 is  satisfied for logistic regression as discussed in Section <ref> of the supplement. While  the condition Π_n(𝒵_Ǔ_n)=0 is naturally satisfied for normal linear/nonlinear regression problems, as 𝒵_Ǔ_n forms a manifold in ℝ^n in this case. In the model (<ref>), 
 if the function f satisfies the continuity condition as required in 
 Assumption <ref>-(ii), we can ensure that the EFI network also satisfies it by employing appropriate activation functions, such as sigmoid, tanh and softplus. The other conditions are standard and generally hold.
 

  
 
 
 Lemma <ref> implies that the choice of η is not critical for the convergence of the EFI-DNN algorithm, as long as λ→∞. Specifically, different choices of η will result in the same zero-energy set as λ→∞. 
 
 In practice, to enhance the convergence of the EFI-DNN estimator to the desired inverse function, one can set η to a moderate value such as 2, 5, or 10, and set λ to be reasonably large. Recall that 
 η represents a regularization parameter as defined in (<ref>).  
 An appropriate value of λ can be determined by gradually increasing it until the resulting confidence intervals of the model parameters cease to shrink.
 
















In summary, we have developed a valid algorithm for conducting fiducial inference for general statistical models by leveraging a sparse DNN for the inverse function approximation. 
The EFI-DNN algorithm is computationally efficient. When simulating the latent variables, it essentially samples from (<ref>) with a small value of ϵ rather than directly  from the limiting distribution (<ref>). This circumvents the need to compute the determinant det(∇^2_ U_n()), thereby significantly enhancing computational efficiency. On the other hand, since the algorithm
is designed  to sample from the limiting 
distribution of (<ref>), it can be applied to models 
with any type of noise, whether additive or non-additive. 
Furthermore, thanks to the universal approximation capability of DNNs, the EFI-DNN algorithm is highly versatile and can be applied to 
statistical models of various complexities.  

 
 
 

 §.§ Some Variants of the EFI-DNN Algorithm

 
 In Algorithm <ref>, the latent variable sampling step is performed using a SGLD algorithm. This can be replaced with an advanced stochastic gradient MCMC algorithm, such as stochastic gradient Hamiltonian Monte Carlo (SGHMC) <cit.>, 
 momentum SGLD <cit.>, 
 or preconditioned SGLD <cit.>. 
 The convergence of adaptive SGHMC has been studied in <cit.>, 
 where similar theoretical results to Theorem <ref> and Theorem <ref> were achieved.
 Compared to SGLD, SGHMC includes an extra momentum term, which 
 enables faster exploration of the sample space <cit.>. 

  
 Other than adaptive SGHMC, we also recommend replacing SGLD   
 with tempering SGLD in Algorithm <ref>. 
 In this tempering algorithm, the temperature τ in (<ref>) is replaced by a decreasing sequence τ_k that converges to 1 
 along with iterations. Such a tempering algorithm is particularly useful for outlier 
 detection problems, as illustrated in Section <ref>. With the tempering technique, random errors of large magnitudes can be easily drawn for some observations, accelerating the convergence of the simulation. 

Similar to the tempering technique discussed above, using an increasing sequence of {λ_k}  that converges to a target value along with iterations can also improve the convergence of the simulation. As λ_k increases, the latent variable samples gradually shift toward the set 𝒵_Ǔ_n. In this setup, Algorithm <ref> possesses a dual adaptive mechanism, adapting both the values of λ_k and ^(k). The convergence properties of such an algorithm will be investigated in future work, following a framework similar to <cit.>.

  
 


§ ILLUSTRATIVE EXAMPLES
 



 §.§ Linear Regression
 
We begin by considering a linear regression model given by

    y_i=x_i^T + σ z_i,     i=1,2,…,n,

where z_i ∼ N(0,1), x_i=(x_i,0,…,x_i,9)^T, x_i,0=1, x_i,k∼ N(0,1) for k=1,…,9,  σ=1, and the regression coefficient =(θ_0,θ_1,…,θ_9)^T=(1,1,1,1,1,0,0,0,0,0)^T. 
For convenience, we refer to (θ_0,θ_1,…,θ_4) as signal parameters and (θ_5,θ_6,…,θ_9) as noise parameters. We simulated 100 datasets from this model, each with a sample size of n=500. 


EFI-a and EFI were applied to this example with σ assumed to be known.  For EFI, we have also tried different activation functions, including ReLU, softplus, tanh, and sigmoid.
Refer to the supplement for the settings of the experiment. 
The numerical results are summarized in Table <ref>. 
Figure <ref> illustrates the concept of EFI. The left plot displays a scatter plot of _n versus _n, where _n represents the true random errors realized for the observations and _n represents a set of random errors imputed by EFI. The scatter plot highlights the presence of uncertainty in the random errors 
 contained in the data.  
 According to the theory of EFI, the uncertainty in _n propagates to , giving rise to uncertainty in . 
 The middle plot is a quantile-quantile (Q-Q) plot for _n and _n,  indicating 
 that they follow the same distribution. 
 The right plot compares the confidence intervals of β_1 produced by 
 EFI and the OLS method. 
 For this dataset, the two methods produced nearly identical confidence intervals 
 for β_1. This complies with our theoretical result presented 
 in Example 1 of Section <ref>.


 




For comparison, we have applied OLS and GFI to this example. The OLS method is simple, whose implementation is available in many statistical packages such as R Studio.  

There are two ways to implement GFI as described in Section <ref>. 
One is to use the acceptance-rejection procedure as described in Section 2. However, due to its importance sampling nature, this procedure becomes highly inefficient 
for the problems with a large value of n. 
For instance, in this example, we attempted to generate 50,000,000 samples of _n from N(0,I_n) for n=500, but none of them was accepted. The other way 
involves direct simulations from the limiting distribution as given in Theorem 1 of <cit.>.  
For this example, the limiting distribution is given by 
∼ N(_n^T _n)^-1_n^T _n, σ^2(_n^T _n)^-1), where _n represents the design matrix of (<ref>) and _n=(y_1,y_2,…,y_n)^T, which is identical to the extended fiducial distribution. 





Table <ref> shows that both versions of EFI work very well for this example. 
In our experience, EFI-a often requires a larger value of η to control the variability of _̂î than EFI. Additionally, EFI tends to be more robust to parameter settings than EFI-a, as it directly use the average _n  
in generating the fitted values ỹ_i's. 
Since EFI and EFI-a are asymptotically equivalent, as mentioned in Remark <ref>, we will only present the results of EFI in the following analysis. Furthermore, Table <ref> 
shows that EFI is robust to the choice of the activation 
functions. Note that each of these activation 
functions is Lipschitz continuous (with a 
Lipschitz constant of 1) and can result in a consistent estimator 
for the inverse function. 


For a comprehensive treatment of the model (<ref>), we applied EFI to the simulated datasets with σ^2 assumed to be unknown. The results are summarized in 
Table <ref>,  which demonstrates the validity of EFI for performing  
statistical inference on the model. In this case, we experimented different settings 
of η and λ,  and EFI 
proved to be robust to these settings.  




































In summary, EFI performs as expected for this example, yielding similar results to OLS and GFI. This is consistent with our analytic results in Example 1, where we showed that EFI results in the same theoretical confidence distribution as OLS and GFI for the linear regression model. 
It is worth noting that in this particular example, the observations precisely follow the presumed model. In Section <ref>, we will demonstrate that EFI can outperform likelihood-based methods when this situation is altered.   




 §.§ Behrens-Fisher problem
 

Consider two Gaussian distributions N(μ_1,σ_1^2) and N(μ_2,σ_2^2).
Suppose that two independent random samples of sizes n_1 and n_2 are drawn from them, respectively. The structural equations are given by 

    y_1i   =μ_1+σ_1 z_1i,    i=1,…,n_1, 
    
        y_2i   =μ_2+σ_2 z_2i,    i=1,…,n_2,

where z_i1, z_i2∼ N(0,1) independently. 
The Behrens-Fisher problem pertains to the inference for the difference μ_1-μ_2 when the ratio σ_1/σ_2 is unknown.





Behrens <cit.> proposed the first solution to the problem 
in the context of testing the hypothesis H_0: μ_1=μ_2 versus H_1: μ_1 μ_2, 
based on the pivot: 

    T= (Y̅_1-Y̅_2)-(μ_1-μ_2)/√(S_1^2/n_1+S_2^2/n_2),
 
where Y̅_i and S_i^2 denote, respectively, the sample mean and sample variance 
of population i for i=1,2. 
Fisher <cit.> pointed out that this solution could be justified using the fiducial theory. Jeffreys <cit.> showed that a Bayesian calculation with the prior 
π() ∝ (σ_1 σ_2)^-1 yields the same confidence interval as the fiducial method.
From a frequentist perspective, Bartlett <cit.> noted that inverting Behrens' test can lead to a conservative confidence interval for μ_1 - μ_2, i.e., its coverage probability is greater than the nominal level.
Later, based on the same statistic T, Welch <cit.> proposed a t-test for which the resulting confidence interval for μ_1 - μ_2 has a coverage probability nearly equal to the nominal level. However, Fisher <cit.> criticized Welch's test for its negatively biased relevant selections, i.e.,  the coverage rate of its confidence interval can be lower than the nominal level for some instances.

As shown in <cit.>, there are no exact fixed-level tests based on the complete sufficient statistics for this problem. However, exact solutions based on other statistics and approximate solutions based on the complete sufficient statistics do exist.

Recently,  Martin and Liu <cit.> applied the inferential model method to this problem, resulting in the same confidence interval as Hsu-Scheffé's <cit.>, but which is known to be conservative <cit.>.  Wang and Jia <cit.> developed a non-asymptotic t-test for the problem based on a statistic different from T, but the efficiency of the test is still unclear.
 
























We applied EFI to this problem by solving the two structural equations in (<ref>) 
separately: one for (μ_1, σ_1) and the other for (μ_2,σ_2).  
 Let {μ̂_1^(k): k=1,2,…, } and {μ̂_2^(k): k=1,2,…,} denote, respectively, the fiducial samples for the population means produced by the two EFI solvers. Then, the 95% confidence interval for μ_1-μ_2 can be directly constructed by finding the 2.5th and 97.5th percentiles of the samples  {μ̂_1^(k) - μ̂_2^(k): k=1,2,…,}. 









 This confidence interval construction method sets EFI significantly apart from existing methods, as it doesn't directly seek the distribution of a test statistic. This advantage of EFI will be further illustrated in Section <ref>. 
 






















 In our first simulations, we set n_1 = n_2 = 50, μ_1 = 1, μ_2 = 0,  
   and varied the values of (σ_1^2, σ_2^2) as provided in Table <ref>. The widths and coverage rates of the resulting confidence intervals are reported in Table <ref>, where the results were obtained with =10,000 and by averaging over 200 independent datasets. 
   For comparison, we also report the results from the Behrens-Fisher method (available in the R package `asht' <cit.>), Welch's method, Hsu-Scheffé's method, and  Te-test <cit.>.
 The comparison suggests that for this example, EFI tends to be more efficient than the existing methods, yielding shorter confidence intervals while maintaining the same level of coverage rates. 
 























































To explain the efficiency of EFI, we present in Figure <ref> 
the Q-Q plots of {μ̂_i^(k): k=1,2,…,}  
versus {t̃_i^(k): k=1,2,…,} for i=1,2.
Here,  t̃_i^(k)=y̅_i - s_i/√(n_i) t_n_i-1^*(k), 
and t_n_i-1^*(k) denotes the kth sample randomly drawn from a 
student t-distribution with n_i-1 degrees of freedom.   
Since the sample size n is finite and thus the samples can be viewed as drawn from 
a tail-truncated distribution, EFI imputes the latent variables essentially from a tail-truncated distribution, due to its conditional inference nature. As a result, the Q-Q plots in Figure <ref> display a tail-cut phenomenon. Therefore, when the sample size n is small, the EFI confidence intervals can be shorter than those from unconditional inference methods, even they have the same coverage rates. 
However, when the sample size becomes large, this feature of conditional inference can disappear as illustrated by Table <ref> 
with the results of n_1=n_2=500. 
We refer to this feature as the finite-sample effect for conditional inference.  
It is worth noting that since EFI solves for (μ_1,σ_1) and (μ_2,σ_2) separately, the Behrens-Fisher problem essentially becomes 
a linear regression problem with unknown variances for EFI. Therefore, it is not surprising that the empirical distribution of μ̂_i closely matches 
 a location-scale student t-distribution. 























































































































 §.§ Bivariate Normal Distribution
 

Let _1, _2,…,_n, with _i=(y_i,1,y_i,2)^T for i=1,2,…,n, be  
independent samples from a bivariate normal distribution 
with the mean vector and covariance matrix given as follows: 

    [ μ_1; μ_2 ]= [ 1; 0 ],   [   σ_1^2 ρσ_1σ_2; ρσ_1σ_2   σ_2^2 ]=[   1 0.5; 0.5   1 ],

where ρ is the coefficient of correlation between two components of the bivariate normal vector. 
To perform EFI, we consider the following decomposition: 

    y_i,1   =μ_1+l_1z_i,1, 
    
        y_i,2   =μ_2+l_2z_i,1+l_3z_i,2,

where l_1>0 and l_3>0, and z_i,k's (for k=1,2 and i=1,2,…,n) are i.i.d standard normal random variables.  
It is easy to derive that σ_1=l_1, σ_2=√(l_2^2+l_3^2), and  
ρ=l_2/√(l_2^2+l_3^2).  Based on this decomposition, we set 
=(μ_1,μ_2,log(l_1),l_2,log(l_3))^T for EFI. The results are presented 
in Table <ref>, where we  
calculated the coverage rates and confidence interval widths based on 100 replications of the data set. 
The sample size is n=100 for each dataset. 











Inference for the parameters of the bivariate normal distribution has served as a classical example of fiducial inference. This can be seen in works such as Fisher <cit.>, Segal <cit.>, and Bennett <cit.>. Their derivations have yielded the following established results:


  * The marginal fiducial distribution of either μ_k is given by  
 √(n) (y̅_k-μ_i)/s_k ∼ t(n-2), where y̅_k=1/n∑_i=1^n y_i,k, s_k=1/√(n-1)√(∑_i=1^n (y_i,k-y̅_k)^2), and t(n-2) denotes a student-t distribution with the degree of freedom n-2. 


  * The marginal fiducial distribution of either σ_k^2 is given by  
 (n-1) s_k^2/σ_k^2 ∼χ^2_n-2, where χ^2_n-2 denotes a chi-squared distribution with the degree of freedom being n-2. 
 

According to <cit.>, the marginal fiducial distribution of ρ that was derived by Fisher  <cit.> is the same as its marginal posterior distribution 
when the parameters are subject to the right-Haar prior π(μ_1,μ_2,σ_1,σ_2,ρ) ∝σ_1^-2(1-ρ^2)^-1. More precisely, the marginal fiducial distribution of ρ has a stochastic representation as 

    ψ( - √(χ_1^2*/χ_n-1^2*)+ 
     √(χ_n-2^2*/χ_n-1^2*)
    r/√(1-r^2)),   ,

r=1/n-1∑_i=1^n (y_i,1-y̅_1)(y_i,2-y̅_2)/(s_1 s_2) is the sample correlation coefficient,  χ_1^2^*, χ_n-1^2* and χ_n-2^2* are chi-squared random variables with the indicated degrees of freedom, and all the random variables 
are mutually independent. 







  

The comparison suggests that for this example, EFI tends to produce shorter confidence intervals than the Fiducial method for the scale parameters σ_1, σ_2, and ρ, while the two methods tend to yield similar results for the location parameters μ_1 and μ_2. 
Once again, we attribute the efficiency of EFI in this example to the finite-sample effect, similar to 
the Behrens-Fisher problem. 



























































 








 














  

























  






  














 

 §.§ Fidelity in Parameter Estimation
 



The frequentist methods often conduct parameter estimation under the maximum likelihood principle.
As implied by the constraint (<ref>), 
the MLE can be easily contaminated by outliers. In contrast, as implied by (<ref>) and (<ref>), 


EFI essentially estimates  by maximizing the  predictive likelihood function  π(_n|_n,_n,)∝π_0^⊗ n(_n) e^-λ∑_i=1^n 
d(y_i,x_i,z_i,),

which balances the fitting errors and the likelihood of random errors. Compared to the MLE  _MLE= max_π_0^⊗ n(_n), 
where _n can be expressed as a function of (_n,_n,), 
the EFI estimator tends to be more robust to outliers and provides higher fidelity in parameter estimation. However, if the model is correctly specified, no outliers exist, and the sample size is reasonably large, maximizing 
π_0^⊗ n(_n) leads to an approximate minimization of the fitting error  ∑_i=1^n d(y_i,x_i,z_i,). 
Specifically, when _MLEp→^*, _n^* can be recovered in probability and thus  ∑_i=1^n 
d(y_i,x_i,z_i,) p→ 0. 
In such cases,  the two methods will yield similar estimates, refer to Table <ref> for an illustrative example 
of this issue. 

To illustrate EFI's robustness to outliers, we consider the model (<ref>) again. In this new simulation, we set n=600 and generated random errors from a mixture Gaussian distributions: z_1,z_2,…,z_540∼ N(0,1) and z_541,z_542,…,z_600∼ N(4,1). The latter cases were considered as outliers, although some of them might be indistinguishable from the former ones. 


 Figure <ref> compares the performances of EFI and OLS on a simulated dataset. It suggests that EFI only slightly shrank 
the random errors and led to a more accurate estimate of σ^2 (≈ 1.0) and narrower 
confidence intervals for , while the OLS estimate of σ^2 (≈ 1.8) was significantly enlarged by outliers and the resulting confidence intervals of  were much wider.
The Bayesian method performs similarly to the maximum likelihood estimation method, 
as they both are likelihood-based.
 



In Section <ref> of the supplement, we present another example which shows that the EFI estimator is less prone to overfitting compared to those from the maximum likelihood or ordinary least square method. This is again attributed to its emphasis on balancing the fitting errors and the likelihood of random errors.  
 



§ EFI FOR SEMI-SUPERVISED LEARNING

 
As mentioned previously, the incorporation of computer technology into science and daily life
has enabled scientists to collect massive volumes of data during the past two decades. 
However, many of the data are unlabeled, as acquisition of labeled data 
for many problems can be expensive.


In such situations, semi-supervised learning (SSL), which is to combine a small amount of labeled data with a large amount of unlabeled data to enhance the learning of a classifier, 
can be of great practical value. However, to make use of unlabeled data, some assumptions 
about the distribution of the data are needed <cit.>. 
For example, one often makes i) the smoothness assumption that the points closing to each other are more likely to share a label,  ii) the cluster assumption that the points form some clusters and those in the same cluster are more likely to share a label (although the data share a label may spread across multiple clusters), or iii) the manifold assumption that the high-dimensional data lie roughly on a low-dimensional manifold.  
 The existing SSL methods  can be roughly divided into categories such as  consistency regularization, proxy-label, generative models, and graph-based methods. See <cit.> and <cit.> for overviews.  

 For a better explanation of the idea behind the current SSL methods, let's consider a text classification problem.
   Let _l denote labeled text data, let _l denote the labels, and let _u denote unlabeled text data. 
 <cit.> modeled the text data using a mixture multinomial 
 distribution as a generative model. 
 By treating  _u, the labels of _u, as missing data, 
 they derived the incomplete data posterior: 
 
    logπ(|_l, _l, _u)     = Const+logπ() 
      + ∑_x_i ∈_llog( p(y_i=c_j|) p(x_i|y_i=c_j,) ) 
        +∑_x_i ∈_ulog(∑_c_j ∈ S p(c_j|) p(x_i|c_j, ) 
       ),

where S denotes the set of classes,  denotes the set of parameters 
of the mixture distribution, and π() denotes the  prior of . 
As implied by (<ref>), the key for SSL is to model the text data (_l,_u) for its 
class-wise distribution, i.e., p(x_i|c_j,). 
 Otherwise, under the conventional regression setting  where (_l,_u) is treated 
as constants, the last term in (<ref>) will be dropped and the unlabeled data 
will not be able to help to improve the estimate of . 

In contrast, as indicated by Figure <ref>, EFI uses both the text data  and labels  as input, and models the distribution of  in an implicit way. 
Moreover, such an implicit model is general and user friendly due to the universal approximation power of deep neural networks. Therefore, EFI can be easily adapted to SSL by treating _u as missing data, which
will be sampled along with the latent variable _n in step (ii) of Algorithm <ref>. 
To illustrate the potential of EFI in SSL, we consider some classification problems
taken at UCI machine learning repository.

For binary classification, the second term (i.e.,  fitting error term)  
in (<ref>) 
can be replaced by 

    ∑_i=1^n_lρ((u_i-x_i^T_i)(2y_i-1))+∑_j=1^n_uρ((u_j^miss-x_j^T_j)(tanh(v_j^miss/τ)),

where ρ(·) is a ReLU function, n_l denotes the number of labeled data, n_u denotes the number of unlabeled data, u_i and u_j^miss are latent variables, 
v_j^miss is defined through the equation P(y_j^miss=1)=1/1+e^-v_j^miss/τ for the missed label, and τ is a scale parameter. 
In simulations, we set τ=1/50, ensuring the probability 1/1+e^-v_i^miss/τ  is dichotomized to either 1 or 0, and treat { u_i: i=1,2,…,n_l} and
{ u_j^miss, v_j^miss: j=1,2,…,n_u} as latent variables to simulate at each iteration.
For EFI, (<ref>) can be changed by replacing _i's and 
_j's with _n. 
 For multiclass classification problems, (<ref>) can be slightly modified.
 




For each dataset, EFI was run in 5-fold cross-validation,  where the labels were removed from 50% of the training samples. The results are summarized in Table <ref>, where the results of supervised 
learning were obtained with the classical logistic regression.
For comparison, the self-training algorithm <cit.> and label-propagation algorithm <cit.>, which both 
belong to the category of proxy-label methods and are available in 
the package scikit-learn 1.2.0, were applied to the datasets. 
In self-training, a model is first trained on labeled data, 
this trained model is then used to predict the classification probabilities of unlabeled data, and predictions with high confidence are added to the training set to retrain the model.
In label-propagation learning, a graph is first created to 
connect the training samples, and then the known labels are propagated through the edges of the graph to unlabeled samples 
in the training set.
A drawback of these methods is that the model is unable to correct its own mistakes, potentially amplifying wrong classifications or biases through the training process. The supervised learning methods are to learn a logistic regression model for each of the datasets.
 







The comparison shows the superiority of EFI in SSL, which can generally perform much 
better than the self-training and label propagation algorithms. 
For the dataset “Raisin”, EFI even outperforms supervised learning, 
and we would attribute this performance of EFI to its fidelity in parameter estimation.
For these datasets, we have also applied EFI to the full training set and labeled data only. 
The results are similar to those from the logistic regression. Refer to the supplement for the detail. 

 

  


§ EFI FOR COMPLEX HYPOTHESIS TESTS
 

 As the scale and complexity of scientific data grow, there is often an interest in testing more complex hypotheses. However, within the frequentist framework, it is usually challenging to derive the theoretical reference distributions for the corresponding test statistics. In contrast, EFI operates in the mode of conditional inference, circumventing the need for theoretical reference distributions and enabling easy hypothesis testing based on collected fiducial samples. In this sense, EFI is driving statistical inference toward an automated process.

  To illustrate the automaticity of EFI in hypothesis testing, we consider the following mediation  analysis model <cit.>: 

 
    Y   =β_T T+β M+_x^T X +ϵ_Y,   ϵ_Y∼ N(0,σ_Y^2), 
    
          M   =γ T +_x^T X + ϵ_M,   ϵ_M∼ N(0,σ_M^2),

 where Y, T, M and X denote the outcome, treatment, mediator and design matrix, 
 respectively.  The mediator effect can be inferred by testing the hypothesis H_0:βγ=0 against H_A:βγ≠ 0 with the natural 
 test statistic β̂γ̂. As mentioned by <cit.>,
 this is a challenging inferential task due to the non-uniform asymptotics of 
 the univariate test statistic. Specifically, the null hypothesis consists of three cases: (i) β=0,γ≠ 0, (ii) β≠ 0, γ=0, and (iii) 
 β=γ=0, while the theoretical reference distribution of β̂γ̂ under case (iii) is different from that under cases (i) and (ii). 
 It is known that traditional statistical tests such as Sobel's test <cit.> 
  and Max-P test <cit.> are conservative under case (iii).
 Recently, with a fine theoretical analysis, <cit.> derived a test that is minimax optimal with respect to local power over the alternative parameter space while preserving type-I error. 
 
 In contrast, applying EFI to such a composite hypothesis test is straightforward. 
 The mediator effect can be directly inferred based on the fiducial samples of 
 β and γ, which can be collected along with iterations of Algorithm <ref>. 
 We note that the bootstrap method <cit.> works in a similar way to EFI, which performs conditional inference for the model parameters and approximates their confidence distributions in an 
 empirical way. In this paper, we implemented the bootstrap method for the model (<ref>) using the R package “mediation” <cit.> under the default setting.

 

  
Simulation Studies For illustration, we simulated 100 datasets from the model 
 (<ref>) under each of the cross settings of n∈{500,1000, 2000} and  
 (β,γ)∈{(0.2,0), (0,0.2), (0,0)}, 
 where X=(X_1,X_2) consists of two independent standard Gaussian random variables, 
 σ_Y=√(2), σ_M=1, _x=(0.2,0.4)^T, β_T=1, _x=(0.4,0.6)^T.  
 The results are summarized in Table <ref>, which 
 indicates the validity and superiority of EFI in testing complex hypotheses. 
 Compared to the other methods, the type-I errors
 of EFI are much closer to the nominal level 0.05, see Figure <ref> in the supplement for a graphical view of the results. 

 



 
  Further, we simulated datasets for comparison of the powers of these tests, where 
 (β,γ)∈{(0.1,0.4), (-0.1,0.4), (0.2,0.2)} and other parameters were as set 
 in the type-I error experiments. The results are summarized in Table <ref>, see also Figure <ref> in the supplement for a graphical view of the results. 
 The comparison indicates that the EFI test has higher power than the other methods. The superiority of EFI over the Bootstrap method is particularly encouraging, highlighting the great potential of EFI in conditional inference and advancing the automation of statistical inference.
  
 





 
  This example demonstrates the potential of  EFI in hypothesis testing. 
Due to its conditional inference nature, EFI eliminates the need for theoretical reference distributions, thereby automating the process of hypothesis testing.
Moreover, compared to frequentist methods, EFI lowers the requirement for sample size. In particular, under high-dimensional scenarios where the model dimension p grows with the sample size n,  frequentist methods typically require p^2/n → 0  for achieving asymptotic normality  (see e.g. <cit.> and <cit.>). For EFI, we believe that p/n → 0 is sufficient for achieving valid fiducial inference, which ensures   Assumption (<ref>) holds for many data generation equations.   
A further theoretical study on this issue will be reported elsewhere.



 




§ DISCUSSION
 





 
We have developed EFI as a novel and flexible framework for statistical inference,  applicable to general statistical models regardless of the type of noise, whether additive or non-additive. 
We have also introduced the EFI-DNN algorithm for effective implementation of EFI, which jointly imputes the realized random errors in observations
using stochastic gradient Markov chain Monte Carlo 
and estimates the inverse function using a sparse DNN based on all available data. The consistency of the sparse DNN estimator ensures that the uncertainty embedded in the observations is properly propagated to the model parameters through the estimated inverse function, thereby validating downstream statistical inference. The EFI-DNN algorithm has demonstrated appealing properties in parameter estimation, hypothesis testing, and semi-supervised learning. 
Additionally, thanks to the conditional inference nature of EFI and the universal approximation power of DNNs, the EFI-DNN algorithm holds  great potential to automate statistical inference.  
Toward this direction, further study on the theoretical properties 
of the EFI-DNN inference is of great interest. 




The EFI-DNN algorithm is scalable, which can handle very large-scale datasets with the use of adaptive stochastic gradient MCMC algorithms. Specifically, its parameter updating step can be accelerated by the mini-batch strategy; and the latent variable sampling step can be executed separately for each observation, enabling straightforward implementation in a parallel architecture. Theoretical guarantees for the convergence of the algorithm have been studied; we established the weak convergence of the imputed random errors and the consistency of the inverse function estimator.
  
 This paper has considered only the problems
 where  p is either fixed or grows with n slowly enough to satisfy Assumption <ref>-(ii).  
 Extending the EFI-DNN algorithm to high-dimensional problems, where p> n and/or p grows with n at a higher rate, is possible.  
 For instance, if the high-dimensional issue arises from including an excessively large number of covariates,  a model-free sure independence screening procedure (see e.g.,<cit.>) can be performed 
  on the data before applying the algorithm.
Furthermore, if one aims to examine the uncertainty of a parameter for an individual covariate, the Markov neighborhood regression (MNR) approach  <cit.> can be applied. This approach decomposes the high-dimensional inference problem into a sequence of low-dimensional inference problems based on the graphical model formed by the covariates.

 

   
 



 
 
 
 
 
 




§ AVAILABILITY
 

The code that implements the EFI method can be found at <https://github.com/sehwankimstat/EFI>.



§ ACKNOWLEDGMENTS
 
Liang's research is support in part by the NSF grants DMS-2015498 and DMS-2210819, and the NIH grants R01-GM126089 and R01-GM152717. The authors thank the editor, associate editor, and three referees for their constructive comments, which have led to significant improvement of this paper.





 


Appendix: Supplement for “Extended Fiducial Inference:  Toward an Automated Process of Statistical Inference” 























This supplement is organized as follows. Section <ref> provides the proofs for Theorem <ref> and Theorem <ref>. Section <ref> provides the proof for Theorem <ref>. 
Section <ref>  provides the proof for Example 1 of Section 3.1 of the main text. 
Section <ref> provides more numerical results. Section <ref> presents detailed parameter settings used in the numerical experiments. 

 




§ PROOF OF THEOREM <REF> AND THEOREM <REF>
 
 
 Notation: For both Theorem <ref> and Theorem <ref>, the sample size n is fixed. For simplicity of notation, we will replace the dataset notation (_n,_n,_n) by (_n,_n,_n) and further drop the subscripts of _n, _n, 
 _n, _n and 𝒲_n in the remaining part of this section. 
 Additionally, for convenience, we redefine h():=∇_logπ(|,), ℋ(,):=∇_logπ(|,,), 
 π_D(|):=π(|,,), and F_D(,):=logπ_D(|), where D represents a training dataset. Furthermore, 
 with a slight abuse of notation, we use _k and _k to denote the latent variable sample and parameter estimate obtained at iteration k of Algorithm <ref>. 


 

 §.§ Proof of Theorem <ref>

 
 With the simplified notation, 
 the equation (<ref>) of the main text can be rewritten as  
 
    h()=∫(,) π_D(|) d=0,

where ∈ℝ^d_w, ∈ℝ^d_z, and d_w and d_z denote the dimensions of  and , respectively. 
The adaptive SGLD algorithm used for solving  equation (<ref>) can be written in a general form as 

    _k+1    =_k+ϵ_k+1 g(_k,_k,u_D,k)+√(2ϵ_k+1)_k+1, 
    _k+1   =_k+γ_k+1(_k,_k+1),

where k ∈ℕ indexes iterations, ϵ_k+1∈ℝ^+ denotes the learning rate, γ_k+1∈ℝ^+ denotes the step size, _k ∼ N(0,I_d_) is a zero mean standard Gaussian random vector,  g(_k,_k,u_D,k): ℝ^d_×ℝ^d_w×𝒰→ℝ^d_ denotes an unbiased estimator of ∇_ F_D(_k,_k), 𝒰={1,2,…,n} is the index set of the observations in D,  and {u_D,k: k=1,2,…} is a sequence of i.i.d random elements of 𝒰 with probability measure 𝒬_D. In general, u_D,k can be understood as the index set of a
mini-batch sample. In the case that the full dataset is used at each iteration, we have  
u_D,k=𝒰 for all k. 

  
  
 To prove the convergence of the adaptive SGLD algorithm (<ref>), 
 we make the following assumptions. 
   
 
 The step size sequence {γ_k}_k ∈ℕ is a positive decreasing 
 sequence of real numbers such that 

    lim_k→∞γ_k=0,   ∑_k=1^∞γ_k= ∞.

 There exist δ>0 and a stationary point ^* such that for any ∈𝒲,
 
    ⟨-^*, h() ⟩≤ -δ-^*^2,

 and, in addition, 
  
    lim inf_k→∞ 2δγ_k/γ_k+1+γ_k+1-γ_k/γ_k+1^2>0,

where · denotes the default l_2-norm. 


 

  F_D(,) is M-smooth on  and  with M>0, and (m,b)-dissipative on  for some constants m>1 and b>0. In other words, for any , ^', ^''∈𝒳 and , ^'∈𝒲, the following  conditions are satisfied:

    ∇_F_D(,^')- ∇_F_D(^',^'') ≤ M ^'-^''  + M-^',  
         ⟨∇_F_D(^*,), ⟩≤ b-m^2,

where 
^* is a stationary point  as defined in Assumption <ref>. 



 

Let (^*,^*) be a minimizer of F_D(,) and ^* be a stationary point such that ∇_F_D(^*,^*)=0. By (<ref>),  we have ^*^2≤b/m. Therefore,

    ∇_F_D(,)   ≤∇_F_D(^*,^*)+M^*-+M-^*
       ≤ M-^*+M+B,

where B=M √(b/m), and 

    ∇_ F_D(,)^2≤ 3M^2^2+3M^2-^*^2+3 B^2.




    Let R_k=g(_k,_k,u_D,k)- ∇_ F_D(_k,_k). Assume that R_k's are mutually independent white noise, 
and they satisfy the conditions

    (R_k|ℱ_k)=0,     R_k^2≤δ_g (M^2 _k^2+M^2 _k-^*^2+B^2),

where δ_g and B are positive constants, and ℱ_k=σ{_1,x_1,_2,x_2,…, _k,x_k} denotes a σ-filtration.  


 
There exist positive constants M and B such that 

    (,)^2    ≤ M^2 -^*^2+M^2 ^2+B^2.



  
(Uniform L^2 bounds; Lemma A.2 of  <cit.>)  Suppose Assumptions <ref>-<ref> hold, and the learning rate sequence {ϵ_k: k=1,2,…} and the step size sequence {γ_k: k=1,2,…} are set in the form:

    ϵ_k=C_ϵ/c_ϵ+k^α,   γ_k=C_γ/c_γ+k^β,

for some constants C_ϵ>0, c_ϵ>0, C_γ>0, c_γ>0, α,β∈ (0,1], and β≤α≤min{1,2 β}.
Then there exist constants G_z and G_ such that _k^2 ≤ G_z and _k-^*^2 ≤ G_ for all k=0,1,2,….

 


  (Solution of Poisson equation) 
For any ∈𝒲, ∈, and a function V()=1+, 
there exists a function μ_ on  that solves the Poisson equation μ_()-𝒯_μ_()=ℋ(,)-h(), 
where 𝒯_ denotes a probability transition kernel with
𝒯_μ_()=∫_μ_(')𝒯_(,') d ',  such that 

    ℋ(_k,_k+1)=h(_k)+μ__k(_k+1)-𝒯__kμ__k(_k+1),    k=1,2,….

Moreover, for all , '∈𝒲 and ∈, 
μ_()-μ_'()  + 𝒯_μ_()-𝒯_'μ_'()≤ς_1 -' V() and
μ_()  + 𝒯_μ_() ≤ς_2 V() for some constants ς_1>0 and ς_2>0. 



 [Theorem A.1 of <cit.>] Suppose Assumptions <ref>-<ref> hold, and the learning rate sequence {ϵ_k: k=1,2,…} and the step size sequence {γ_k: k=1,2,…} are chosen as in 
Lemma <ref>. Then there exists a root ^*∈{: h()=0} such that

    𝔼_k-^*^2≤ξγ_k,    k≥ k_0,
 
 where ξ and k_0 are some constants determined by  the sequences {ϵ_k} and {γ_k} and
 the constants (δ,δ_g,M,B, m,b,ς_1,ς_2).




  
Proof of Theorem <ref>
 <cit.> proved the result (<ref>) for the adaptive 
Langevinized ensemble Kalman filter (LEnKF) algorithm, which is equivalent to an adaptive pre-conditioned SGLD algorithm. Since the SGLD algorithm (<ref>) is a special case 
of the pre-conditioned SGLD algorithm, this theorem can be proved by following the proof of 
<cit.> with minor modifications. 
We omit the details of the proof.


 

  
Regarding the convergence rate of _k, we note that <cit.> gives an explicit form of ξ. Refer to Theorem A.1 of <cit.> for the detail. 

 




 §.§ Proof of Theorem <ref>



Let T_k = ∑_i=1^kϵ_k. 
Let μ_D,T_k=ℒ(_k|_k,D) denote the probability law of _k at iteration k of Algorithm <ref>, let ν_D,T_k 
=ℒ((T_k)|^*,D) denote the probability law of a continuous time diffusion process, 
and let π^*=π_D(|^*). 

   Suppose the conditions of Lemma <ref> hold. Then there exist 
some constants C_0>0 and C_1>0 such that 

    D_KL(μ_D,T_kν_D,T_k) ≤ (C_0 δ_g+C_1 γ_1) T_k,

where D_KL|(··) denotes the Kullback-Leibler divergence. 


Our proof follows the proof of Lemma 7 in <cit.> closely, but changing from a constant learning rate sequence to a decaying learning rate sequence. 
Similar developments can also be found in  Appendix D of <cit.>.


 Let T̅(s) = T_k for T_k ≤ s < T_k+1, k = 1,…, ∞. Conditioned on D and , {_k} forms a Markov process. Consider the following continuous-time interpolation of this process: 

    (t)=_0-∫_0^t g((T̅(s)),(s), u̅_D(s)) d s+√(2)∫_0^t d B(s),    t ≥ 0,

where (s)= _k for  T_k ≤ s < T_k+1, and {B(s)}_s≥ 0 is the standard Brownian motion in ℝ^d_z.
Note that, for each k, (T_k) and _k have the same probability law μ_D,T_k. Moreover, by a result of <cit.>, the process (t) has the same one-time marginals as the Itô process

    '(t)=_0-∫_0^t g_D,s('(s)) d s+√(2)∫_0^t d B(s),

where 

    g_D,s(z):=𝔼[g((T̅(s)),(s),u̅_D(s))  |(s)=z].
 
Crucially, '(t) is a Markov process, while (t) is not. 

Let _'^t:=ℒ('(s): 0 ≤ s ≤ t | D, (s)) and _^t:=ℒ((s): 0 ≤ s ≤ t | D, ^*).  The Radon-Nikodym derivative of _^t w.r.t. _Z'^t is given by the Girsanov formula 

    d_^t/ d_'^t(')   =exp{1/2∫_0^t(∇ F_D('(s), ^*)-g_D,s('(s)))^* d B(s).
       .-1/4∫_0^t∇ F_D('(s), ^*)-g_D, s('(s))^2 d s }.

Using (<ref>) and the martingale property of the Itô integral, we have

    D_KL(_'^t_^t)    =-∫d_'^tlogd_^t/ d_'^t
       =1/4∫_0^t𝔼∇ F_D('(s), ^*)-g_D, s('(s))^2 d s 
       =1/4∫_0^t𝔼∇ F_D((s),^*)-g_D, s((s))^2 d s,

where the last line follows from the fact that ℒ((s)) = ℒ('(s)) for each s. 


Recall that T_0 = 0 and T_k = ∑_i=1^kϵ_k.
Then, by the definition of g_D, s, Jensen's inequality and the M-smoothness of F_D, we have

    D_KL(_'^T_k_^T_k)   =1/4∑_j=0^k-1∫_T_j^T_j+1𝔼∇ F_D((s),^*)-g_D, s((s))^2 d s 
       ≤1/2∑_j=0^k-1∫_T_j^T_j+1𝔼∇ F_D((s), ^*)-∇ F_D((T̅(s)), ^*)^2 d s 
           +1/2∑_j=0^k-1∫_T_j^T_j+1𝔼∇ F_D((T̅(s)), ^*)-g((T̅(s)),(s), u̅_D(s) )^2 d s 
       ≤ M^2/2∑_j=0^k-1∫_T_j^T_j+1𝔼(s)-(T̅(s))^2 d s 
           +1/2∑_j=0^k-1∫_T_j^T_j+1𝔼∇ F_D((T̅(s)), ^*)-g((T̅(s)),(s), u̅_D(s) )^2 d s.

To estimate the first summation on the right side of (<ref>), we consider some s∈[T_j,T_j+1). By (<ref>), we have

    (s)-(T_j) =-(s-T_j) g(_j, _j, u_D,j)+√(2)(B(s)-B(T_j)) 
       =-(s-T_j) ∇ F_D(_j, ^*)+(s-T_j)(∇ F_D(_j, ^*)-g(_j,  _j, u_D,j))+√(2)(B(s)-B(T_j)).

Therefore, by (<ref>), Assumption <ref>, Lemma <ref> and Lemma <ref>, we have 

    𝔼(s)-(T_j) ^2
       ≤ 3 ϵ_j+1^2𝔼∇ F_D(_j, ^*)^2+3 ϵ_j+1^2𝔼∇ F_D(_j, ^*)-g(_j, _j, u_D,j)^2+6 ϵ_j+1 d_z
       ≤ 3 ϵ_j+1^2𝔼∇ F_D(_j, ^*)^2+
    6 ϵ_j+1 d_z
        + 6 ϵ_j+1^2( 𝔼∇ F_D(_j, ^*)
    -∇ F_D(_j, _j)^2+ 𝔼∇ F_D(_j, _j)-g(_j, _j, u_D,j)^2) 
       ≤ 9 ϵ_j+1^2(M^2𝔼_j^2+B^2)+6 d_ϵ_j+1 + 6 ϵ_j+1^2 (ξ M^2 γ_j + δ_g(M^2 G_z+ξ M^2 γ_j+B^2)) 
       ≤ 9 ϵ_j+1^2(M^2G_z + B^2 )+6 d_ϵ_j+1  + 6 ξ M^2γ_j ϵ_j+1^2+ 6 δ_g ϵ_j+1^2 (M^2 G_z+ ξ M^2 γ_j+B^2).

Consequently, we can bound the first summation on the right-hand side of (<ref>) 
as follows: 

    ∑_j=0^k-1∫_T_j^T_j+1𝔼(s)-(T̅(s))^2 d s 
       ≤ 9(M^2G_z + B^2) ∑_j=0^k-1ϵ_j+1^3 +6 d_z∑_j=0^k-1ϵ_j+1^2 + 6ξ M^2∑_j=0^k-1γ_jϵ_j+1^3 + 
     6 δ_g ∑_j=0^k-1ϵ_j+1^3 (M^2 G_z+ ξ M^2 γ_j+B^2).

Similarly, by Lemma <ref>, the second summation on the right-hand side of (<ref>) can be bounded as follows:

    ∑_j=0^k-1∫_T_j^T_j+1𝔼∇ F_D((T̅(s)), ^*)-g((T̅(s)),(s),u̅_D,s)^2 d s 
       = ∑_j=0^k-1ϵ_j+1𝔼∇ F_D(_j, ^*)-g(_j, _j,u_D,j)^2
       ≤ 2 ∑_j=0^k-1ϵ_j+1(𝔼∇ F_D(_j, ^*)-
    F_D(_j, _j) ^2+ 𝔼 F_D(_j, _j)-
    g(_j, _j,u_D,j)^2 ) 
       ≤ 2 ξ M^2∑_j=0^k-1γ_j ϵ_j+1+ 2 δ_g ∑_j=0^k-1ϵ_j+1(M^2 G_z+ξ M^2 γ_j+B^2).

Substituting Equations (<ref>) and (<ref>) into (<ref>), we obtain

    D_KL(𝐏_'^T_k𝐏_^T_k)    ≤9/2(M^4G_z + M^2B^2) ∑_j=0^k-1ϵ_j+1^3 + 3 M^2 d_z∑_j=0^k-1ϵ_j+1^2 + 3ξ M^4 ∑_j=0^k-1γ_jϵ_j+1^3  
    + ξ M^2 ∑_j=0^k-1γ_j ϵ_j+1
        + δ_g ∑_j=0^k-1ϵ_j+1 (3M^2 ϵ_j+1^2+1) (M^2 G_z+ξ M^2 γ_j+B^2).

Since μ_D, _k,T_k=ℒ(_k|D, _k) and ν_D, ^*, T_k=ℒ((t)|D, ^*), the data-processing inequality for the Kullback-Leibler divergence gives

    D_KL(μ_D, _k, T_kν_D, ^*, T_k) 
    ≤  D_KL(𝐏_'^T_k𝐏_^T_k) 
    ≤   9/2(M^4G_z + M^2B^2) ∑_j=0^k-1ϵ_j+1^3 + 3 M^2 d_z∑_j=0^k-1ϵ_j+1^2 + 3ξ M^4 ∑_j=0^k-1γ_jϵ_j+1^3  
    + ξ M^2 ∑_j=0^k-1γ_j ϵ_j+1
        + δ_g ∑_j=0^k-1ϵ_j+1 (3M^2 ϵ_j+1^2+1) (M^2 G_z+ξ M^2 γ_j+B^2) 
    
       ≤ (C_0 δ_g + C_1 γ_1) T_k,

for some constants C_0>0 and C_1>0. 


 
The probability law μ_0 of the initial hypothesis _0 has a bounded and strictly positive density 
p_0 with respect to the Lebesgue measure on ℝ_d_, and 

    κ_0:=log∫_ℝ^d_ e^^2 p_0() d < ∞.



   Suppose Assumption <ref> and the conditions of Lemma <ref> hold. Then there exist 
some constants C̃_0>0 and C̃_1>0 such that 

    𝕎_2^2(μ_D,T_k, ν_D,T_k) ≤ (C̃_0 √(δ_g)+C̃_1 √(γ_1)) T_k^2,

where 𝕎_2(·,·) denotes 2-Wasserstein distance. 


The proof of Lemma <ref> follows that of Proposition 8 of <cit.> closely. 
First, we apply Corollary 2.3 of <cit.> to get the inequality 

    𝕎_2^2(μ_D,T_k, ν_D,T_k) ≤ C T_k ( D_KL(μ_D,T_k, ν_D,T_k)+ 
    √( D_KL(μ_D,T_k, ν_D,T_k))),

for some constant C, for which we assume both μ_D,T_k and ν_D,T_k 
have finite second moments.  Further, by substituting (<ref>) into (<ref>), we can complete the proof. 

 
   Suppose Assumption <ref> and the conditions of Lemma <ref> hold. Then there exist 
some constants Ĉ_0>0, Ĉ_1>0 and Ĉ_2 such that 

    𝕎_2(μ_D,T_k, π^*) ≤ (Ĉ_0 δ_g^1/4+Ĉ_1 γ_1^1/4)T_k + Ĉ_2 e^-T_k/c_LS,

where c_LS denotes the logarithmic Sobolev constant of π^*=π_D(|^*). 

The proof of Lemma <ref> follows that of Proposition 10 in <cit.> closely, 
and it is thus omitted. 





§ PROOF OF THEOREM <REF>
 

Notation: We use (x,y,z) to denote a generic observation in the dataset 
(_n,_n,_n).

   The EFI network satisfies the conditions:
 
     
  (i) The parameter space 𝒲_n (of _n) is convex and compact. 
     
     
  (ii) 𝔼 (logπ(y,z|x,_n))^2 < ∞ for any _n∈𝒲_n.

     
     
 
  










 
 
 For any positive integer n, the following conditions hold:
  
    
    
    
  (i) Q^*(_n) is continuous in _n and uniquely maximized at some point _n^b;
    
  (ii) for any ϵ>0, sup_∈𝒲_n\ B(ϵ)Q^*(_n) exists, where B(ϵ)={_n:_n-_n^b<ϵ}, and δ=Q^*(_n^b)-sup__n ∈𝒲_n \ B(ϵ)Q^*(_n)>0.


 



  
Proof of Lemma <ref>  





















 


 























Suppose that π(_n|_n,_n) has a different maximizer that minimizes D_KL(_n) as well. Let _n^† denote such a maximizer, which is different from 
_n^* but maintains _n^* ∼π(|_n,_n, _n^†). 
For _n^†, similar to (<ref>), we have 

    (_n|_n^†)    :=1/n∫logπ(_n,_n^*|_n,_n) dπ(_n^*|_n,_n,_n^†) +1/nlogπ(_n) 
       =  1/n{logπ(_n|_n,_n) - ∫logπ(_n^*|_n,_n,_n^†)/π(_n^*|_n,_n,_n) d π(_n^*|_n,_n,_n^†) 
        + ∫logπ(_n^*|_n,_n,_n^†) d π(_n^*|_n,_n,_n^†)+c },

 which, by the non-negativeness of the Kullback-Leibler divergence, implies that 
 _n^† is also the maximizer of  (_n|_n^†). 
 By (<ref>), (<ref>), and Assumption <ref>, we would have 
 
    _n^*-_n^†p→ 0,   ,

 following the proof of Lemma 2 in <cit.>.
 This contradicts with the uniqueness of _n^*. 
 
Therefore, if _n^* is unique, then  _n^* is unique. Subsequently, 
we have _n^*-_n^*p→ 0 as n→∞, which completes the proof. 


 
  
 We follow <cit.> to make the following assumption on the DNN model embedded in the EFI network, for which the random errors _n are assumed to be known. 
The sparse DNN has H_n-1 hidden layers, and each layer consists of L_j hidden units. 
Specifically, we use L_0 and L_H_n to denote the input and output dimensions, respectively. 
The weights and biases of the sparse DNN are specified by 
_n,  and the structure of the sparse DNN is specified  by Λ_n, 
a binary vector corresponding to the elements of _n. 

 

 
  (i) The complete data (x,y,z) is bounded by 1 entry-wisely, i.e. (x,y,z)∈Ω=[-1,1]^p_n, and the density of (x,y,z) is bounded in its support Ω uniformly with respect to n.
 
 
  (ii) The underlying true sparse DNN (_n^*,Λ̃_n^*) satisfies the following conditions:  
 
 
  (ii-1) The network structure satisfies: 
 r_n H_nlog n+r_n logL+s_nlog p_n≤ C_0 n^1-ε, where 0<ε<1 is a small constant, 
 r_n denotes the connectivity of Λ̃_n^*, L=max_1≤ j≤ H_n-1 L_j denotes the maximum hidden layer width,  and s_n denotes the input dimension of Λ̃_n^*.
 
 
  (ii-2) The network weights are polynomially bounded:  _n^*_∞≤ E_n, where 
  E_n=n^C_1 for some constant C_1>0. 
 
 
  (iii) The activation function ψ used in the DNN is Lipschitz continuous with a Lipschitz constant of 1.
 
 
 
  (iv) The mixture Gaussian prior (<ref>) satisfies the conditions:
 ρ_n = O( 1/{K_n[n^H_n(L p_n)]^τ'}) for some constant τ'>0,
  E_n/{H_nlog n+logL}^1/2≲σ_1,n≲ n^α' for some 
  constant α'>0, and 
σ_0,n≲min{ 1/{√(n) K_n (n^3/2σ_1,0/H_n)^H_n}, 
 1/{√(n) K_n (n E_n/H_n)^H_n}}, where K_n=∑_h=1^H_n (L_h-1× L_h+L_h) denotes the total number of parameters  of the fully connected DNN. 

 
  (v) For the normal regression case, y=f(x,_0)+σ z with 
 z∼ N(0,1), the function f(x,_0) is Lipschitz continuous with respect to _0; and for the logistic regression case,  log(P(Y=1)/(1-P(Y=1)))=μ(x,), the logit link function μ(x,) is Lipschitz continuous with respect to .   
 
  


 
 
If we further assume that the exponent 0≤ C_1< 1/2 and the connectivity r_n=O(n^ζ') for some 0<ζ'<1/2-C_1-ε' and 0<ε'<1/2-C_1-ζ'.
Then, based on the proof of  Theorem 1  and the followed remark in  <cit.>, it is easy to figure out that K_n is allowed to increase with the sample size n in an exponential rate: K_n ≺exp(n^2ε').
 
 By the arguments provided in <cit.>, the sparse DNN approximation under the above assumptions is achievable for quite a few classes  of functions, such as 
 bounded α-Hölder smooth functions <cit.>,  piecewise smooth functions with fixed input dimensions <cit.>, and the functions that can be represented by an affine system <cit.>.
  

   
Assumption <ref>-(i) restricts Ω, the domain of the complete data (x, y, z), to a bounded set [-1, 1]^p_n. To satisfy this condition, we can add a data transformation/normalization layer to the DNN model, ensuring that the transformed input values fall within the set Ω.
In particular, the transformation/normalization layer can form an 1-1 mapping and contain no tuning parameters. For example, when dealing with the standard Gaussian random variable, we can transform it to be uniform over (0,1) via the probability integral transformation Φ(z), where Φ(·) denotes the CDF of the standard Gaussian random variable.   
 


















 

For the EFI network, we define

    h_n(_n)=1/nlogπ(_n,_n^*|_n,_n)+1/nlogπ(_n),

where _n^* is the true random errors realized in the data (_n,_n) and it is thus independent of _n. 
Then the posterior density of _n is given by 
 π(_n|_n,_n,_n^*) =  e^nh_n(_n)/∫ e^nh_n(_n)d_n 
and, for a function b(_n), the posterior expectation is given
by ∫ b(_n)e^nh_n(_n)d_n/∫ e^nh_n(_n)d_n.
Recall that we have  defined _n^*=max__nπ(_n|_n,_n,_n^*), which is also the global maximizer of h_n(_n). 
Let B_δ(_n) denote an Euclidean ball of radius δ
centered at _n. Let h_i_1,i_2, …,i_d(_n)
 denote the d-th order partial derivative ∂^dh(_n)/∂ w_n^i_1∂ w_n^i_2⋯∂ w_n^i_d, 
 let H_n(_n) denote the Hessian matrix of h_n(_n), let
 h_ij denote the (i,j)-th component of the Hessian matrix, and  
 let h^ij denote the (i,j)-component of the inverse of the Hessian matrix. 
 Recall that Λ̃^*  denotes the set of indicators for the 
 connections of the true sparse DNN,  
 r_n denotes the size of the true sparse DNN, 
 and K_n denotes the size of the fully connected DNN. 
  
 
There exist positive numbers ϵ, M, and n_0 such that for any n>n_0,  the function h_n(_n) in (<ref>) satisfies the following conditions:

 
  (i)  |h_i_1,…,i_d(_n^*)|<M hold for any _n∈ B_ϵ(_n^*) and any 1≤ i_1,…,i_d≤ K_n, where 3 ≤ d ≤ 4. 
 

  (ii) |h^ij(_n^*)|<M if Λ̃_n,i^*=Λ̃_n,j^*=1 and |h^ij(_n)| = O(1/K_n^2) otherwise, where Λ̃_n,i^* denotes the i-th element of Λ̃_n.


  (iii) (-n/2πH_n(_n))^1/2∫_ℝ^K_n∖ B_δ(_n)
e^n(h_n(_n)-h_n(_n))d_n=O(r_n^4/n)=o(1) for any 
0<δ<ϵ.



 Assumption <ref>-(i)&(iii) are typical conditions for Laplace approximation, see e.g., <cit.>. Assumption <ref>-(ii) requires the inverse Hessian to 
 have very small values for the elements corresponding to the false connections. Refer to 
 <cit.> for its justification. 
 



  
Proof of Theorem <ref> 
 
As discussed in Section <ref>, we have the likelihood function for the EFI network as 

    π(_n|_n,_n,) = C e^-λ U_n(_n,;_n,_n).


In the context of this proof, we assume that _n=(z_1,z_2,…,z_n)^T is known. 




Additionally, we use d_t(p,p^*)=t^-1 (∫ p^*(p^*/p)^t -1) to denote a divergence measure for two 
distributions p and p^*. It is easy to see that 
d_t converges to the KL-divergence as t↓ 0.  


































  
Normal Regression
For the normal linear/nonlinear regression, we essentially have
 the energy function:  
 
    U_n(_n,;_n,_n)= ∑_i=1^n y_i-f(x_i,_0)-σ z_i^2,

as λ→∞, where :=(_0,log(σ))=G(x_i,y_i,z_i,) is a constant function    
over the observations {(y_i,x_i,z_i): i=1,2,…, n}. 

Therefore, as λ→∞, (<ref>) enforces 
the output  of the DNN to satisfy the relationship:

    y_i=f(x_i,_0)+σ z_i,    i=1,2,…,n,

i.e., y ∼ N(f(x,_0),σ^2). 







A direct calculation shows that the divergence d_1(·,·) of two Gaussian distributions p(x):=N(f(x,_0), σ^2) 
and q(x):=N(f(x,_0'), ς^2) is given by

    d_1(q,p)=ς^2/σ/√(2 ς^2-σ^2) e^f(x,_0)-f(x,_0')^2/2ς^2-σ^2-1,

provided that 2 ς^2-σ^2>0. 
The divergence d_1(·,·) is a function of the two factors f(x,_0)-f(x,_0')^2
and |log (σ)-log (ς)|. In particular, if both the factors goes to 0, then 
d_1(·,·) goes to 0.





Therefore, to bound the value of d_1(·,·), one can bound 

    f(x,_0)-f(x,_0')^2+|log (σ)-log (ς)|^2 = O(-'^2) =O(G(x,y,z,)-G(x,y,z,')^2),

provided that f(x,_0) is Lipschitz continuous with respect to _0, where 
'= G(x,y,z,') and ' denotes the corresponding DNN weights. 
This result implies that as λ→∞, 
the posterior consistency for the DNN model in the EFI network can be studied as for a conventional normal regression DNN model with input variables (x,y,z) 
and the output variable , provided that f(x,_0) is Lipschitz continuous with respect to _0. Therefore, by Theorem 2.1 of <cit.>, the posterior consistency holds for the DNN model  under Assumption <ref>.  




  
Logistic Regression 
For logistic regression,  the reasoning is similar. As implied by (<ref>), we essentially have the following probability mass function for a generic observation (x,y,z):

    p_λ(y|z, x, )  ∝exp{-λρ((z-μ)(2y-1))},

where =G(x,y,z,) is a constant function  over the observations {(y_i,x_i,z_i): i=1,2,…, n}, and μ=μ(x,)=x^T for the linear case. 
As λ→∞, the two events 
{Z < μ} and {Y=1} are asymptotically  equivalent, i.e., 
{Z < μ}⟺{Y=1} with probability 1. 
Due to the monotonicity of the function 1/1+e^-z,   
{1/1+e^-Z< 1/1+e^-μ(x,)}⟺{Y=1} with probability 1 as λ→∞.
Furthermore,  since Z follows the logistic distribution,  1/1+e^-Z is uniform on (0,1).  Therefore, as λ→∞, 
(<ref>) enforces the output  of the DNN model to 
 satisfy the following relationship:  

    μ(x,)=log(P(Y=1)/(1-P(Y=1))).
 
Following the calculation in <cit.>, the divergence d_1(·,·) (up to a  multiplicative constant) of two logistic  distributions, with respective logit link functions  μ(x,) and μ(x, '), is given by 

    μ(x,)-μ(x,')^2= O(-'^2)=O(G(x,y,z,)-G(x,y,z,')^2),

provided that μ(x,) is Lipschitz continuous with respect to , where 
'= G(x,y,z,') and ' denotes the corresponding DNN weights.  
This result implies that as λ→∞, the posterior consistency for the DNN model in the EFI network can be studied as for a conventional logistic regression 
DNN model with input variables (x,y,z) and the output variable , provided that the logit link function μ(x,) is Lipschitz continuous with respect to . Therefore,  by Theorem 2.1 of <cit.>, the posterior consistency holds for the EFI network under Assumption <ref>.



Furthermore, by Assumption <ref>, the parameter space 𝒲_n is compact and convex.  
Therefore, for any bounded function b(_n), the posterior mean 𝔼 (b(_n)) is a consistent 
 estimator of b(_n^*) under posterior consistency. 
For the inverse mapping estimator ĝ(x,y,z,_n),  by Assumption <ref>-(i) and 
Assumption (<ref>)-(i), it is bounded and 

    |ĝ_i_1,…,i_d(x,y,z,_n)|= 
    |∂^dĝ(x,y,z,_n)/∂_n^i_1∂_n^i_2⋯∂_n^i_d|<M,

holds for 
some constant M, for any 1 ≤ d ≤ 2 and 1≤ i_1,…,i_d≤ K_n. 
Then, under Assumption <ref> and by Theorem 2.3 of <cit.>,  ĝ(x,y,z,_n^*) (as an approximator to the posterior mean 𝔼 g(x,y,z,_n)) 

 forms a consistent estimator of ^*.
  
 
 Finally, by Lemma  <ref>, _n^*-_n^*p→ 0 holds,  which implies ĝ(x,y,z,_n^*) is also  
 a consistent estimator of ^*. This completes the proof. 





















 
























§ DERIVATION OF EFD FOR A REGRESSION EXAMPLE
 


Consider the linear regression model as defined in equation  (<ref>), where ∈ℝ^p-1. For an illustrative purpose, we assume that σ^2 is known. We set  

    G(_n, _n, ) =G̃(_n, _n, )= (_n^T _n)^-1_n^T(_n - σ),

and the energy function

    U_n()=_n-f(_n,,G(_n,_n,))^2.

Let _n = I_n - _n(_n^T _n)^-1_n^T, which is an idempotent matrix of rank n-p+1. Then 

    J() =    _n-f(_n,,G(_n,_n,)) 
    
    =    _n - _n G(_n, _n, ) - σ
    
    =    _n(_n - σ).

Let (_1, …, _p-1) be the eigenvectors corresponding to the zero eigenvalues of _n, i.e. _n _i =  0∈ℝ^n for i=1,2,…,p-1. Let (_p, …, _n) be the eigenvectors corresponding to the nonzero eigenvalues of _n. 
Let _1 = (_1, …, _p-1) ∈ℝ^n × (p-1) and _2 = (_p, …, _n) ∈ℝ^n × (n-p+1). Then it is clear that 

    {: J() = 0} = {1/σ_n - _1 : ∈ℝ^p-1}.

For any vector ∈ℝ^n, 
we can write down the exact form of the decomposition in (<ref>) as:

    = 1/σ_n - _1  - _2 ,

where ∈ℝ^p-1 and ∈ℝ^n-p+1. Then, for U_n() = J() ^2, we have

    ∇^2_ U_n() =2 _2^T_n^T _n _2.

Note that 

    rank(∇^2_ U_n()) = rank(_n _2) = rank(_n(_1, _2)) = rank(_n) = n-p+1.

Therefore, ∇^2_ U_n() is a positive constant. 
Furthermore, for any ∈𝒵_n, it can be written as = 1/σ_n - _1 for some ∈ℝ^p-1, and the limiting measure 
has the form 

    p_n^*(|_n,_n)=   p_n^*( 1/σ_n - _1 |_n,_n) ∝π_0^⊗ n( 1/σ_n - _1 ),

which corresponds to a truncation of π_0^⊗ n(·) on the manifold 𝒵_n. Therefore,  

    1/σ_n - _1 ∼ N( 0, I_n).


For  any ∈𝒵_n, we set

    G̃(_n, _n, )     = (_n^T _n)^-1_n^T (_n-σ (1/σ_n-_1)),

and the resulting EFD is given by 

    μ_n^*(|_n,_n)=N(, σ^2 (_n^T_n)^-1),

where =  (_n^T _n)^-1_n^T _n. 
























§ MORE NUMERICAL RESULTS
 



 §.§ Nonlinear Regression
  

Nonlinear least squares regression problems are intrinsically hard due to their complex energy landscapes, which may contains some saddle points, local minima or pathological curvatures. 
To test the performance of EFI on nonlinear regression, we took a benchmark dataset, Gauss2, at 
NIST Statistical Reference Datasets (<https://www.itl.nist.gov/div898/strd/nls/nls_main.shtml>),
which consists of 250 observations. 

The nonlinear regression function of the example is given by 

    y=β_1 exp{-β_2 x}+ β_3 exp{-(x-β_4)^2/β_5^2}+β_6 exp{-(x-β_7)^2/β_8^2}+ϵ:=f(x,)+ϵ,   ϵ∼ N(0,6.25),

where =(β_1,β_2,…,β_8) denotes the vector of unknown parameters. The nonlinear regression function represents two slightly-blended Gaussian density 
curves on a decaying exponential baseline plus normally distributed zero-mean noise with known variance 6.25.  For this example, the “best-available” OLS solution has been given as shown in 
Table <ref>, which was obtained using 128-bit precision and confirmed by at least two different algorithms and software packages using analytic derivatives.


The EFI method was applied to this example with the experimental settings given in Section <ref> of this supplement. 
Table <ref> compares the parameter estimates and confidence intervals by the OLS and EFI methods. For OLS, the confidence intervals are constructed by Wald's method with the estimates' standard deviations given in the website.  
For EFI, the parameter estimates are obtained by averaging the fiducial -samples collected in the simulation, and the confidence intervals are constructed with 2.5% and 97.5% quantiles of the fiducial samples. Therefore, the EFI confidence intervals are not necessarily symmetric about the parameter estimates. The comparison shows that the EFI confidence intervals 
tend to be shorter than the OLS confidence intervals. 
More importantly, since EFI and OLS employ different objective functions, they actually converge to different solutions. This can be seen from the confidence intervals of β_3 resulting from the two methods,
 which have no overlaps. 



To further explore the difference of the EFI and OLS solutions, we examined their fitting 
and residual plots in Figure <ref>.
The right plot indicates that EFI tends to have larger 
residuals than OLS. A simple calculation shows that the OLS has a mean-squared-residuals of 4.99, while the EFI has a mean-squared-residuals of 5.72, which is closer to the ideal value 6.25. 
This comparison implies that OLS, which simply minimizes the sum of squared fitting errors, can lead to an overfitting issue even for this reasonably large dataset. EFI performs better in this regard by striking a balance between fitting errors and the likelihood of random errors, as discussed in Section 3.4 of the main text. This balance potentially results in a solution of higher fidelity.

For this example, we have also tried the Bayesian method, which leads to almost the same solution as OLS, as they essentially employ the same objective function. 
 


 



 §.§ Logistic Regression
  

The EFI method can be easily extended to discrete statistical models via approximation or  transformation. For example, for the logistic regression, whose response variable y_i ∈{0,1} is discrete, making the fitting-error term in (<ref>) 
 and (<ref>) less well defined.  To address this issue, we define ReLU function:
ρ(δ)=δ if δ>0 and 0 otherwise, and then replace the fitting-error term 
in (<ref>) and (<ref>) by   
   
    ∑_i=1^n ρ((z_i-x_i^T_i)(2y_i-1) ),

   where z_1,z_2,…,z_n iid∼ Logistic(0,1) with the CDF given by 
   F(z)=1/(1+e^-z). That is, z_i represents the random error realized in the observation 
   (x_i,y_i). Correspondingly, the fitted value ỹ_i is defined by
   ỹ_i= 1 if F(z_i) ≤ F(x_i^T_i) and 0 otherwise, where F(z_i)∼ Uniform(0,1) by the probability integral transformation theorem.
   It is easy to see that (<ref>) penalizes the cases {i: y_i ỹ_i} and the resulting energy function satisfies 
   Assumption <ref>.  
   In particular, we can have 
   Π_n(𝒵_Ǔ_n)>0 for this problem, 
   because each z_i can take any value in an interval 
   (-∞,a] or [b,∞) (for some a,b∈ℝ) while maintaining
   the zero total-fitting-error given in (<ref>). 
    
   
  




 

  
  We simulated 100 datasets from a logistic regression consisting of 4 
 covariates independently drawn from N(0,1). The true regression coefficients were   
 =(θ_0,θ_1,…,θ_4)=(1,1,1,-1,-1), including the 
 interpret θ_0. The sample size of each dataset was n=1000. 
 The numerical results are summarized in Table <ref>. 
 The comparison with the MLE results indicates the validity of EFI for statistical inference of logistic regression. 
  

 For comparison, we applied GFI to this example by running the R package gfilogisreg <cit.>, but which did not produce results for this example due to a computational instability issue suffered by the package.
 
 For IM, we refer to <cit.>, where the likelihood function is used for inference of  and the confidence intervals are constructed by inverting the Monte Carlo hypothesis tests conducted on a lattice of grid points in Θ. For example, if we take 50 grid points in each dimension of Θ and simulate 1000 samples at each grid point, then we need to simulate a total of 3.125× 10^11 samples. This is time consuming even for such a 5-dimensional problem.  

For multiclass logistic regression, similar to (<ref>), 
the fitting-error term in (<ref>) and (<ref>) can be defined as 

    ∑_i=1^n [   ∑_j  m_iρ(x_i^T _i,j - x_i^T _i,m_i) +ρ(z_i-x_i^T _i,m_i) ],

where m_i denotes the true class of  the training sample x_i, 
and _i,j denotes the parameter corresponding to class j for the training sample x_i. 






 §.§ Semi-Supervised Learning


Table <ref> presents more examples for the semi-supervised learning. 






 §.§ EFI for Complex Hypothesis Tests








 
 


§ EXPERIMENTAL SETTING
 


To enforce a sparse DNN to be learned for the inverse function g(·), we impose 
the following mixture Gaussian prior on each element of _n: 

    π(w)∼ρ  N(0,σ_1^2)+(1-ρ)N(0,σ_0^2),

where w denotes a generic element of _n and, unless stated otherwise, 
we set ρ=1e-2, σ_0=1e-5 and σ_1=0.02.
The elements of _n are a priori independent. 

In all experiments of this paper, we use ReLU as the activation function, and 
set the learning rate sequence {ϵ_k} and the step size sequence {γ_k} in the forms given in Theorem <ref>. Specifically, we set α=13/14 and β=4/7 unless stated otherwise, and set different values of C_ϵ, c_ϵ, C_γ and c_γ for different experiments as given below. 






 §.§ Linear regression

For both cases with known and unknown σ^2, we use a DNN with structure  
12-300-100-d_ for inverse function approximation, where d_ denotes the dimension of .   

 

  
Known σ^2
For EFI-a, we set η=100 and λ=10; and for EFI, we set η=10 and λ=10. EFI-a and EFI share the same learning rate and step size sequences with (C_ϵ,c_ϵ,C_γ,c_γ)=(50000,10000,5000,100000). 
We set the burn-in period =1000 and the iteration number M=100,000.
The Markov chain is thinned by a factor of B=10 in sample collection, i.e., M/B=10,000 
-samples were collected for calculation of the coverage rates and CI-widths. For EFI, we employ the same parameter settings 
for different activation functions. 


 

  
Unknown σ^2
 For EFI, we used SGHMC in latent variable sampling, i.e., we simulate ^(k+1) in the following formula: 
  
    _n^(k+1)   = (1-ζ) _n^(k) +ϵ_k+1∇__nlogπ(_n^(k)|_n,_n,^(k)) +√(2 ζτϵ_k+1)^(k+1), 
    _n^(k+1)   =_n^(k)+_n^(k+1),

where τ=1, 0<ζ≤ 1 is the momentum parameter, ^(k+1)∼ N(0,I_d), 
and ϵ_k+1 is the learning rate.
It is worth noting that the algorithm is reduced to SGLD if one sets ζ=1.

In simulations, we set the decaying parameters 
α=β=4/7, and the Markov chain is thinned by a factor of B as below for M/B=10,000 samples were used for calculation of the coverage rates and CI-widths. 
 

 

  * (η=2,λ=30): We set ζ=0.025 and (C_ϵ,c_ϵ,C_γ,c_γ)=(6500,100000,1700,100000), (,M)=(10000,50000) thinned by B=5; 


  * (η=2,λ=40): We set ζ=0.025 and (C_ϵ,c_ϵ,C_γ,c_γ)=(5600,100000,1400,100000), (,M)=(10000,90000) thinned by B=9; 


  * (η=2,λ=50):   We set ζ=0.05 and (C_ϵ,c_ϵ,C_γ,c_γ)=(4000,100000,1000,100000),, (,M)=(10000,200000) thinned by B=20; 



  * (η=4,λ=50): We set ζ=0.005 and (C_ϵ,c_ϵ,C_γ,c_γ)=(1950,80000,490,80000),, (,M)=(10000,120000) thinned by B=12; 

 


   

 §.§ Behrens-Fisher problem

   We use a DNN with structure 2-20-10-2 and set η=5 and λ=20. The burn-in period =10000,  the iteration number M=40000 and 60000 for n=50 and 500, respectively. The Markov chain is thinned by a factor of B=4 and 6 for n=50 and 500, 
   respectively, in sample collection. This makes that M/B=10,000 samples are used in  calculation of the coverage rates and CI-widths for each case. 
 

  * σ_1^2=0.25, σ_2^2=1: (i) for n=50, we set ζ=0.01, and (C_ϵ,c_ϵ,C_γ,c_γ)=(2500,100000,2500,100000); 
(ii) for n=500,  we set ζ=0.005, and (C_ϵ,c_ϵ,C_γ,c_γ)=(3000,100000,3000,100000); 


  * σ_1^2=1, σ_2^2=1:  (i) for n=50, we set ζ=0.05, and (C_ϵ,c_ϵ,C_γ,c_γ)=(2800,100000,2800,100000); 
(ii) for n=500,  we set ζ=0.028, and (C_ϵ,c_ϵ,C_γ,c_γ)=(3100,100000,3100,100000).




 
 

 
 


 §.§ Bivariate normal

 
  We used a DNN with structure 4-80-20-5 for inverse function approximation, and we set η=2 and λ=50. 
 We used SGHMC in latent variable sampling as in (<ref>)
. In simulations, we set the momentum parameter ζ=0.1,  the decaying parameters 
α=β=4/7, (C_ϵ,c_ϵ,C_γ,c_γ)=(4500,100000,1100,100000). 
the burn-in period =10000,  the iteration number M=50000, and the Markov chain is thinned by a factor of B=5 in sample collection, i.e., M/B=10,000 samples were used for calculation of the coverage rates and CI-widths. 
 
 


 §.§ Fidelity in Parameter Estimation

 
We used a DNN with structure 12-300-100-11 for inverse function approximation, and set (η,λ)=(2,50).
The tempering SGLD algorithm is used in the latent variable sampling step, where we set 
the temperature sequence τ_t=max(100*(0.9999)^t,1). 
For the learning rate and step size sequences, we set (C_ϵ,c_ϵ,C_γ,c_γ)=(50000000,10000000,50,10000).
For sample collections, we set =50,000, M=150,000, and B=15. 

 

 

 §.§ Nonlinear Regression in the Supplement
 
 
  
  We used a DNN with structure 3-150-50-8 for inverse function approximation, and we set (η,λ)=(500,0.2) in order to avoid a local trap of fitting _n to _n. 
  For the learning rate and step size sequences,  we set (C_ϵ,c_ϵ,C_γ,c_γ)=(1,10000000,1,100) for iterations t<50,000,
  and set (C_ϵ,c_ϵ,C_γ,c_γ)=(1000,100000,10,10000) 
  for t≥ 50,000. For sample collections, we set 
   =60,000, M=150,000 and B=15. 

 

  
  



 §.§ Logistic regression in the Supplement



 For EFI, we set η=2 and λ=1000. 
 We used SGHMC (<ref>) in latent variable sampling.









In simulations, we set the momentum parameter ζ=0.01,  the decaying parameters 
α=β=2/7, 
  (C_ϵ,c_ϵ,C_γ,c_γ)=(50000,100000,30000,100000). 
the burn-in period =10000,  the iteration number M=50000, and the Markov chain is thinned by a factor of B=5 in sample collection, i.e., M/B=10,000 samples were used for calculation of the coverage rates and CI-widths. 



 
 
 
   
  
  
   
   


 








 §.§ EFI for Semi-Supervised Learning


We used a DNN with structure (p+2)-90-30-p for inverse function approximation, where p corresponds to the dimension of  for all cases. 
For EFI on both full label cases, and labeled-data only cases (use 50% of training data), we set α=β=2/7,η=5,λ=200,=10000,M=40000,B=4 with ζ=0.1, (C_ϵ,c_ϵ,C_γ,c_γ)=(100000,100000,2000,100000). 
For semi-supervised EFI, the same parameter settings have been used with the exceptions given as follows:





  * Beast-Cancer:
 (η,λ)=(5,200);


  * Diabetes:
 (η,λ)=(2,500) and (C_ϵ,c_ϵ,C_γ,c_γ)=(200000,100000,1000,100000);

  * Divorce:
 (η,λ)=(10/3,300);

  * Raisin:
 (η,λ)=(2,500).
 

 



 §.§ EFI for Complex Hypothesis Tests


We used a DNN with structure 7-180-30-9 for inverse function approximation, and set α=β=4/7,η=10,λ=10,=10000,M=50000,B=5. In addition, we varied the values
of other parameters according to the problem and sample size. 




  
Type-I error For different sample sizes, we set the parameters as follows:



  * n=500. For case 1, we set ζ=0.1 and (C_ϵ,c_ϵ,C_γ,c_γ)=(290000,100000,4000,100000); for case 2, we set ζ=0.1 and  (C_ϵ,c_ϵ,C_γ,c_γ)=(100000,100000,2000,100000); for
case 3, we set ζ=0.1 and (C_ϵ,c_ϵ,C_γ,c_γ)=(100000,100000,2000,100000).


  * n=1000. For case 1, we set ζ=0.1 and (C_ϵ,c_ϵ,C_γ,c_γ)=
(100000,100000,4000,100000); for case 2, we set ζ=1 and (C_ϵ,c_ϵ,C_γ,c_γ)=(200000,100000,4000,100000); 
for case 3, we set ζ=0.1 and  (C_ϵ,c_ϵ,C_γ,c_γ) =(2000,100000,1000,100000). 


  * n=2000. For case 1 and case 2, we set ζ=1 and  (C_ϵ,c_ϵ,C_γ,c_γ)=(200000,100000,4000,100000); and for case 3, 
we set ζ=0.1 and (C_ϵ,c_ϵ,C_γ,c_γ)=(2000,100000,1000,100000).
 








  
Power For all cases, we set the parameters 
ζ=0.1 and  (C_ϵ,c_ϵ,C_γ,c_γ)=(2000,100000,1000, 100000). 

 
 

asa






