Paper Title: Unveiling land use dynamics: Insights from a hierarchical Bayesian spatio-temporal modelling of Compositional Data.



Article Title]
Unveiling land use dynamics: Insights from a hierarchical Bayesian spatio-temporal modelling of Compositional Data


[1]Mario FigueiraMario.Figueira@uv.es
1]Carmen Guarner
1]David Conesa
1]Antonio López-Quílez
2]Tamás Krisztin


*[1]Department of Statistics and Operations Research, University of Valencia, Carrer del Dr. Moliner, 50, 46100 Burjassot, Valencia, Spain

[2]International Institute for Applied Systems Analysis (IIASA), Schlossplatz 1, 2361, Laxenburg, Austria






Changes in land use patterns have significant environmental and socioeconomic impacts, making it crucial for policymakers to understand their causes and consequences. This study, part of the European LAMASUS (Land Management for Sustainability) project, aims to support the EU's climate neutrality target by developing a governance model through collaboration between policymakers, land users, and researchers. We present a methodological synthesis for treating land use data using a Bayesian approach within spatial and spatio-temporal modeling frameworks.

The study tackles the challenges of analyzing land use changes, particularly the presence of zero values and computational issues with large datasets. It introduces joint model structures to address zeros and employs sequential inference and consensus methods for Big Data problems. Spatial downscaling models approximate smaller scales from aggregated data, circumventing high-resolution data complications.

We explore Beta regression and Compositional Data Analysis (CoDa) for land use data, review relevant spatial and spatio-temporal models, and present strategies for handling zeros. The paper demonstrates the implementation of key models, downscaling techniques, and solutions to Big Data challenges with examples from simulated data and the LAMASUS project, providing a comprehensive framework for understanding and managing land use changes.








[
    [
    July 31, 2024
=================





§ INTRODUCTION











Changes in land use patterns have significant environmental and socioeconomic impacts, making it crucial for policymakers to understand their causes and consequences <cit.>. Studies on land use are valuable for identifying the determinants of these changes, which affect biodiversity, water pollution, soil erosion, climate change, and economic and social welfare <cit.>. These changes are driven by a combination of socioeconomic factors, pedo-climatic conditions, and policy variables. Land use models are crucial for analysing these influences and their effects <cit.>. Moreover, the decision-making processes surrounding land use are complex, shaped by both local and global biophysical and socioeconomic factors. Therefore, comprehensive knowledge of these influences aids in evaluating and formulating environmentally friendly public policies.




Various disciplines, including economics, statistics, geography, and land use science, have developed empirical land use modelling approaches using aggregate or individual data <cit.>. However, many studies overlook spatial autocorrelation in modelling land use or use ad hoc methods <cit.>, despite its prevalence in economic decisions. This underscores the need for more sophisticated spatial econometric models that account for spatial heterogeneity and interdependence to accurately measure and analyse land use patterns <cit.>.




This work is part of the European https://www.lamasus.eu/LAMASUS (Land Management for Sustainability) project. LAMASUS develops an innovative governance model through collaboration between policymakers, land users, and researchers. This Horizon Europe project aims to support the European Union’s climate neutrality target by creating an open-access modelling toolbox for designing effective land use policies within the framework of the European Green Deal. In the context of the LAMASUS project, this paper presents a methodological synthesis for the treatment of land use data using a Bayesian approach and in the spatial and spatio-temporal modelling framework. This approach is integrated in conjunction with those that aim to provide a better understanding of the spatial dynamics of the processes that determine and drive land use changes. Enhanced understanding also enables the implementation of land use policies based on models that improve policy formulation.

LAMASUS faces different problems related to land use and land use change analysis. In this case, there are numerous land use categories, which are aggregated into a small set to facilitate the determination of the main drivers of land use at different scales, both administrative (NUTS scales) and high-resolution levels, that is, small spatial scales. From the aggregated spatial scales, to approximate a smaller scale, spatial downscaling models or disaggregation models are applied, avoiding dealing with high-resolution data that may present inconveniences related to the systematic presence of 0's or being large data banks, which would pose a Big Data problem. On the other hand, to work with the aggregated data with many disaggregated land use categories, joint model structures are proposed to deal with the systematic presence of 0's and 1's, while for problems that emerge from Big Data, a combined procedure of sequential inference and consensus is proposed.







In particular, the aim of this paper is to present different procedures and methods to analyse land use data in different contexts of spatial and spatio-temporal modelling. In this sense, section 2 introduces land use data and its analysis by Beta regression, when only one land use category is available, or within the framework of Compositional Data Analysis (CoDa), if multiple land use categories are available. In Section 3 we briefly review a selection of those spatial and spatio-temporal models that are of interest for assessing the spatial and spatio-temporal structure of land use data. Section 4 sets out the proposed methodology for dealing with the presence of 0's and 1's in land use data, both when a single category is available and when several categories are available. In Section 4 is briefly shown the implementation of the main spatial and spatio-temporal models for land use data. Section 5 illustrates how to deal with the presence of 0's and 1's in Beta regression and CoDa analysis. In Section 6 we present downscaling models, also known as disaggregation models, applied to land use and compositional data. Section 7 is related to Big Data in the framework of high-resolution land use data. Section 8 stands for some simulated data and real data from LAMASUS project. Finally, we conclude in Section 9.



§ LAND USE AND COMPOSITIONAL DATA


In this section, we focus on presenting the treatment of compositional data for the analysis of land use shares, providing a brief exposition of the theoretical framework for CoDa and the implementation of such analysis to address the drivers and underlying process in land use shares. 




Land use data is usually presented as proportions of land use shares in a given area, which implies that land use shares, by definition, have a compositional nature <cit.>. Therefore, one approach to capture the variability of land use shares would be through Beta regression if we focus on a specific category, or through CoDa to analyse the joint variability of several land use categories, e.g. cropland, grassland, forest, urban and other. The joint analysis of the categories in the composition can be considered by different approaches: transforming the logratios <cit.>, and modelling those logratios using multivariate Gaussian models, or using Dirichlet regression models <cit.> for the proportions without needing to transform them.













In general, compositional data consists in a set of parts that identifies the constituents of the composition {1, 2, …, D}, while components {𝐲_1, 𝐲_2, ..., 𝐲_D} are numerical proportions in which individual parts occur <cit.>. Therefore, CoDa can be defined by a matrix 𝐘_n × D of n observations times D parts of the composition. Each row of the matrix 𝐘_n × D satisfies the following closure condition ∑_d=1^D 𝐘_id=1, and for each component is usually to assume for each value that 𝐘_id∈ (0,1). The vector 𝐲_1× D for the i-th observation, related to each row of 𝐘_n × D, is called composition and it pertains to the simplex sample space. A simplex space with dimension d, denoted by 𝕊^d, is defined as:


    𝕊^D = {𝐲∈ℝ^D : y_d∈ (0,1), ∑_d=1^D y_d = 1 },

where due to the closure condition and the property y_i ∈ (0,1), the entire set of observations of the composition 𝐘_n × D  does not follow the usual Euclidean geometry ℝ^D, but rather the Aitchison geometry 𝕊^d with dimension d = D - 1 <cit.>. Additionally, it is also possible to defined a subcomposition 𝐲_1 × C of 𝐲_1 × D. This refers to a subset of C parts of the referring composition of D parts, where C < D and the subcomposition is also subject to the closure condition ∑_i=1^C y_i = 1.












 §.§ Logratio transformations


In Aitchison's approach to compositional data analysis <cit.>, the focus was on distributional issues, such as finding a way to transform compositions into interval-scale multivariate vectors that could validly use the multivariate normal distribution. The proposed logistic transformations, which aimed to create “transformed-normal” models, relied on logarithmic transformations. These transformations allowed the use of standard unconstrained multivariate statistics applied to transformed data, with inferences translatable back into compositional statements <cit.>. However, this reliance on logarithms precluded zero data values, leading to the ongoing debate about zero replacement and treatment, a topic often overlooked in publications <cit.>.

There are several proposals for logratio transformations to analyse CoDa, including those from Aitchison's original work <cit.>, such as pairwise logratios (LR), additive logratios (ALR), and centred logratios (CLR). Later proposals <cit.> include isometric logratios (ILR) and pivot logratios (PLR). These transformations attempt to address various shortcomings of the alternative methods. However, each logratio transformation has its own strengths and weaknesses, making it impossible to propose a single transformation suitable for all CoDa datasets <cit.>. 

In this work, we will focus on two logratios of the proposed by <cit.>: ALR and CLR. The ALR transformation implies defining a subset of d=D-1 logratios with the same denominator, called the reference (r) part. Therefore, the ALR with respect to a reference part of the composition is written as 

    ALR(y_i | y_r) = log(y_i/y_r),  i ∈{1,..., D}:i≠ r,

where is possible to define D alternative ALR transformations, depending on the part chosen as reference. In this case, it is important to use a reference part that satisfy either statistical or meaningful objective. In fact, if the chosen reference part is almost constant, then the corresponding ALRs are approximately the logarithm of the parts, up to a nearly constant amount.













The CLR transformation is defined as the logratios of the parts with respect to their geometric mean g(𝐲)=∏_i=1^Dy_i^1/D:

    CLR(y_i) = log(y_i/g(𝐲)),  i ∈{1,...,D},

transformation that turns the sum-to-one constraint into a sum-to-zero constraint ∑_i=1^D CLR(y_i) = ∑_i=1^D [log(y_i) - log(g(𝐲)) ] = 0. The usefulness of this transformation is that Euclidean distances between the CLRs are identical to the distances using all possible logratios; in other words, the CLR is an isometric transformation.

Both logratio transformations allow us to model the logratios using multivariate Gaussian distributions 𝐲^* ∼MVN(μ, Σ), where 𝐲^* represents the logratios, either from CLR or ALR transformations. This multivariate model not only accounts for complex structures in the linear predictors, including spatial and spatio-temporal dependencies, but also captures the correlations between the logratios through the variance-covariance matrix Σ of the multivariate Gaussian distribution <cit.>.









 §.§ Dirichlet distribution


Dirichlet distribution is a generalisation of the Beta distribution for more than two proportions, and its probability density function is defined as 

    π(𝐲|α) = 1/B(α)∏_i=1^D y_i^α_i - 1,

where α = (α_1,...,α_D) is the vector of shape parameters α_i>0  ∀ i, and B(α) is the Multinomial Beta function, being the normalizing constant of the Dirichlet. The sum of the shape parameters α_0 = ∑_i=1^Dα_i is usually interpreted as a precision marginal parameter τ. In the Dirichlet distribution the sum-to-one constraint ∑_i=1y_i = 1 is also satisfied by the data vector 𝐲 = (y_1,..., y_D), y_i ∈ (0,1). Therefore, if 𝐲∼𝒟(α) denote a variable that is Dirichlet distributed, the expected values are 𝔼(y_i) = α_i/α_0, and the relation between the shape parameters and the linear predictor is given by the log-link function log(α_i)=η_i. Thus, it is also possible to parameterise the Dirichlet distribution in terms of the mean as μ_i = exp(η_i)/(∑_i=1^D exp(η_i)).







The parameterization of the Dirichlet distribution in terms of the shape parameters (or the mean) linked to the linear predictor allows us to use complex structures, as in the INLA framework, by defining Latent Gaussian Models (LGM). These structures can include temporal components, spatial components such as the SPDE-FEM approach, or more complex spatio-temporal processes. The implementation of the Dirichlet distribution in  relies on methodology and R-package presented by <cit.>. This provides the basis to implement complex spatial and spatio-temporal structures for Dirichlet distributed data inside the INLA framework.












§ SPATIAL AND SPATIO-TEMPORAL MODELS


The land use may present changes along space and time. Consequently, to assess the variability it is possible to construct spatio-temporal models that can provide a further description and insights <cit.>. In this section, we will provide a brief overview of the different structures that can be used in spatio-temporal modelling for land use data.







In most cases, land use data is provided as areas, i.e., values related to polygons that typically represent administrative boundaries. This areal data naturally arises when a fixed domain is partitioned into a finite number of subregions where outcomes are aggregated. Therefore, an appropriate way to model this kind of spatial dependency is by building a structure of adjacency, or an adjacency matrix 𝐖, between the different connected areas and setting a multivariate Gaussian density with a precision matrix based on a modulation of this adjacency structure <cit.>, which is quite common in disease mapping as well <cit.>. This prior distribution defines the dependence between the connected areas and allows the assessment of spatial structures, from simpler ones like the Besag model <cit.> or the Besag-York-Mollié model <cit.> to more complex multidimensional spatial structures <cit.>. 







In spatial econometrics, the Spatial Error model (SEM), Spatial Lag model (SLM), and Spatial Durbin model (SDM) are commonly used to analyse spatial dependency <cit.>. These models also define a spatial dependence structure for the explanatory covariates and the error term. The general framework that synthesises these models is the general nesting spatial model equation <cit.>

    [ 𝐲 = δ𝐖𝐲 + α1_N + 𝐗β + 𝐖𝐗θ + 𝐮,;                   𝐮 = λ𝐖𝐮 + ε, ]

where 1_N denotes a vector (N × 1) of ones, α is a global intercept, 𝐖 s an adjacency matrix of positive defined weights usually normalized such that each row sums to unity <cit.>, 𝐖𝐲 represents the endogenous interaction effects of the response variables 𝐲, 𝐖𝐗 represents the exogenous interaction effects among the explanatory variables 𝐗, 𝐖𝐮 denotes the interaction among the spatial units, β and θ represent unknown parameters. Finally, δ and λ are the spatial autoregressive parameter and spatial autocorrelation parameter, respectively. These models can be reassembled to fit within the INLA framework <cit.> by fixing some parameters and performing their analysis outside INLA, using Monte Carlo (MC, ) or Markov Chain Monte Carlo (MCMC, ).




The temporal terms can be incorporated into the linear predictor of the model in an additive way, i.e. as a new term evaluating a purely temporal trend. This can be synthesised into a model with one spatial component and one temporal component 𝐲 = 𝐮_s + 𝐮_t + ε, where 𝐮_s is the purely spatial component, 𝐮_t is the purely temporal term and ε is an independent and identically distributed error term. As an example, the temporal component can be integrated into a rewritten form of the SLM to fit within the INLA framework in the following way:

    𝐲 = (𝐈 + ρ𝐖)^-1𝐗β + 𝐮_t + 𝐮_s

where u_t,i = u_t,i-1 + ϵ(σ) follows a first-order random walk, and 𝐮_s ∼MVN(0, 𝐐):𝐐 = τ (𝐈-ρ_l𝐖)^T · (𝐈-ρ_l𝐖) represents the rewritten spatial structure for the spatial error component.




In addition to being able to evaluate additive spatio-temporal models, it is possible to consider different structures for spatial and temporal interaction components. In <cit.> four general structures are proposed to define spatio-temporal interaction models, in which the precision matrix of the spatio-temporal effect is constructed by means of the Kronecker product of the precision matrix of the spatial component by the precision matrix of the temporal component 𝐐_st=𝐐_s⊗𝐐_t <cit.>. Four natural interactions arise from this formulation: (a) Type I interaction, this is defined as the Kronecker product of the precision matrices of an unstructured spatial 𝐐_s=τ_s𝐈 and temporal effect 𝐐_t=τ_t𝐈, where the structure for the spatio-temporal component is 𝐐_st=𝐐_s⊗𝐐_t=τ_st𝐈⊗𝐈. This interaction makes sense when space and time have a discretised structure, otherwise this specification may not make sense. (b) Type II interaction assumes an unstructured spatial effect 𝐐_s=τ_s𝐈 interacting with a structured spatial effect 𝐐_t, then the spatio-temporal precision matrix is 𝐐_st=𝐈⊗𝐐_t. (c) Type III interaction is the interaction of a structured spatial effect 𝐐_s by an unstructured temporal effect 𝐐_t=τ_t𝐈, obtaining that 𝐐_st=𝐐_s⊗𝐈. (d) Type IV interaction is the interaction between a structured spatial effect 𝐐_s and a structured temporal effect 𝐐_t, which implies that the interacting spatio-temporal component is defined by 𝐐_st=𝐐_s⊗𝐐_t. Moreover, it is also possible to construct non-separable spatio-temporal interaction models <cit.>. In these models, the precision matrix cannot be decomposed into the Kronecker product of a spatial precision matrix and a temporal precision matrix, i.e., 𝐐_st≠𝐐_s ⊗𝐐_t.



§ IMPLEMENTING SPATIAL CODA MODELS


Implementing spatial models in compositional data can be challenging due to the multivariate nature of the response variables. Each category or log-ratio of land use may have its own spatial or spatio-temporal component with a distinct set of hyperparameters. To address this complexity, rather than defining a separate component for each variable, two alternatives can be considered. One approach involves replicating some spatial or spatio-temporal components across different land use categories or log-ratios, resulting in identical hyperparameters for these replicated components <cit.>. Another approach is to share spatial or spatio-temporal components across some categories, either scaled by parameters to be estimated or sharing the same value <cit.>. 

This can be exemplified with the following toy model for three logratios 𝐘 = (𝐲_1, 𝐲_2, 𝐲_3), where 𝐘∼MVN(μ, Σ), and the mean vector μ=(μ_1, μ_2, μ_3) is modelled as:

    [ μ_1 = β_101 + 𝐗β_11 + 𝐮_1s(τ_s, λ_s) + 𝐮_t(τ_t),; μ_2 = β_201 + 𝐗β_21 + 𝐮_2s(τ_s, λ_s) + 𝐮_t(τ_t),; μ_3 = β_301 + 𝐗β_31 + 𝐮_3s(τ_s, λ_s) + 𝐮_t(τ_t),;                                                  ]

where β_∙0 are the intercepts, β_∙1 represents the fixed effect vector related to the matrix of explanatory variables 𝐗, 𝐮_∙ s denotes the spatial effects, which are three different replicas from the same Leroux distribution <cit.>, given by 𝐮∼MVN(0, 𝐐_s):𝐐_s = τ_s · [𝐈 + λ_s · (𝐃 - 𝐈 - 𝐖)], and 𝐮_t: u_t,i = u_t,i-1 + ε_i(τ_t) is a temporal component shared through the three linear predictors with the same values for each logratio. Finally, Σ is the variance-covariance matrix between of logratios, with Σ^-1_ii = τ_i and Σ^-1_ij = ρ_ij/(τ_i ·τ_j)^1/2, i ≠ j. 

In  it is not possible to model multivariate models directly, so the correlation structure for the multivariate normal observations can be represented by a random effect with such a structure in the linear predictor of the model. To illustrate this, suppose we have n observations of a D-dimensional multivariate distribution. To evaluate the correlations of this multivariate distribution in , we can construct a component of the latent field 𝐮 such that 𝐮∼MVN(0, 𝐐), whose precision matrix is defined as 𝐐_ii=τ_ii and 𝐐_ij= ρ_ij/√(τ_iiτ_jj). Each realisation of this component produces a vector of D elements, so if its realisation is replicated for each observation, we will have n replicas of this effect evaluated with the same hyperparameters {τ, ρ}. This allows us to express the D-dimensional multivariate distribution with n components as a univariate distribution with n × D observations and fixed precision in the univariate Gaussian likelihoods to a high value (log(τ)≫ 1), where in each term of the linear predictor we will have a realisation from this effect u corresponding to a category d ∈{1,...,D} and an observation i ∈{1,...,n}.




This demonstrates the flexibility to define various spatio-temporal structures for modelling compositional data, or more generally, multivariate data. It also highlights the combinatorial challenge of defining models within such a framework, where testing all possible combinations implies defining 2^D models for each component of the model, with D being the multivariate dimensionality. This issue suggests that constructing models in these contexts can benefit significantly from expert knowledge, which can help narrow down the feasible set of reasonable models. This approach avoids the need for extensive model selection procedures among numerous sets, which can become intractable with even a few components. Therefore, implementing spatial models in the context of compositional data requires establishing criteria based on expert knowledge or other relevant factors to limit the set of models to be evaluated. This is especially important given the various possibilities for defining spatio-temporal components that we have mentioned in the previous section.



§ DEALING WITH 0'S IN LAND USE DATA


In the first section, we presented various approaches to analyse land use data: beta models when focusing on a single category, and multivariate normal models for some transformation of logratios (additive logratios, centred logratios, isometric logratios) or Dirichlet models when dealing with multiple categories. These are all different approaches to CoDa analysis in general, and to land use data provided in proportions in particular. However, a significant challenge in dealing with compositional data across all these approaches is the presence of zeros and ones <cit.>. Zeros can be problematic because they often indicate the absence of a component, complicating the application of standard statistical techniques that assume strictly positive data. Similarly, ones can signify the complete dominance of a single component (inducing zero values in the remaining components), which can distort correlations and overall analysis. Addressing these issues typically requires specific methods, such as zero-replacement strategies or transformations, to ensure that the compositional nature of the data is properly accounted for and that the analyses yield meaningful results.







In general, different types of zeros can be found in compositional data (CoDa): (i) rounded zeros or below-detection values, (ii) count zeros, and (iii) essential zeros. The first two types of zeros are associated with “false” zeros, where the structure of the experimental design, the sample size, or the sensitivity of the measuring instruments results in null values, even though the underlying phenomenon is not necessarily null. In contrast, the third type of zeros, also called structural or absolute zeros, assumes that the measurements indeed capture the real absence or null value of the phenomenon <cit.>. For the first two cases of rounded zeros and count zeros, replacement procedures or transformations have been proposed <cit.>, while for essential zeros, likelihood modifications such as zero-inflated models or hierarchical structures can be used <cit.>.

In line with <cit.>, a joint hierarchical model similar to Hurdle models <cit.>, which are widely used in the environmental and ecological sciences, can be proposed to deal with null values in likelihoods that do not allow them to be evaluated. This approach involves conditional modelling of the components according to whether each particular component has a zero or non-zero value. In order to apply a structure similar to that of a Hurdle model, there must be a correspondence in the modelling of the composition and subcompositions with that of the null and non-null values of the subcompositions. This implies that we can use it in the case of dealing with data distributed according to a Dirichlet or when we use the centred logratios, since it allows us to make the joint analysis of the zeros with the process that generates the logratios for the CLR, or the shape parameters in the case of the Dirichlet.










Let's assume that we have a matrix of n observations and D compositions, 𝐘_n × D where each element Y_ij is the value for the i-th observation and j-th composition. Then, we can define an incidence matrix 𝐈_n × D, that for each element I_ij we have I_ij = 0  Y_ij = 0 and I_ij = 1  Y_ij≠ 0. Each column of the incidence matrix will be the values that we will model according to a Bernoulli distribution, and in principle this modelling will be independent between the different compositions. Thus, for each composition, the vector 𝐢_d = 𝐈_1:n,d we will have:

    [                                𝐢_1                                  ∼                           Ber(π_1);                         logit(π_1)                                  = 𝐗β_1 + ∑_k=1^K f_k,1(𝐳_k) + 𝐮_st,1;                                                                     ⋮                                   ;                                𝐢_D                                  ∼                           Ber(π_D);                         logit(π_D)                                  = 𝐗β_D + ∑_k=1^K f_k,D(𝐳_k) + 𝐮_st,D;                                    ]

where β_d is the vector of linear coefficients, 𝐟_d are non-linear components and 𝐮_st,d the spatio-temporal components for the d-th component. While for the models for the values of the compositions, if we use the CLR we can write 𝐘^*=CLR(𝐘), we have the following multivariate Gaussian model for the logratio of the compositions 𝐘^* ∼MVN(0,Σ). However, in this case, it is important to distinguish between rows where there is a component with a zero value, or where there is one component with a unit value and the rest are zeros, and rows where no values are zero, in order to analyse them analogously to how it is done with Hurdle models.

























The multivariate model can be rewritten as D univariate Gaussian models in which the marginal precision is fixed at a very high value and the variance-covariance structure is evaluated by a random effect as described in the previous section in order to implement these models in :

    [                                  𝐲^*_1                                      ∼                            N(μ_1, τ^*);                                    μ_1                                      = 𝐗β_1 + ∑_k=1^K f_k,1(𝐳_k) + 𝐮_st,1 + 𝐮;                                                                             ⋮                                       ;                                  𝐲^*_D                                      ∼                            N(μ_D, τ^*);                                    μ_D                                      = 𝐗β_D + ∑_k=1^K f_k,D(𝐳_k) + 𝐮_st,D + 𝐮;                                        ]

where τ^* is a fixed precision with a high value (log(τ^*) ≫ 1), 𝐮 is the random effect that accounts for the variance-covariance structure, 𝐮∼MVN(0, 𝐐), such that 𝐐=Σ^-1. In addition, for each observation row with a zero value component, the CLR transformation is re-evaluated to exclude those null values. This procedure is the same whether we use the Dirichlet distribution or the Beta distribution, which is a simplification of the Dirichlet when only two categories are available.



§ DOWNSCALING MODELS


Empirical studies involving land use change often use aggregate data for regions, countries, or other geographic scales <cit.>. However, some land use studies have had access to individual parcel-level data for analysing water quality, the extent of urban sprawl, carbon sequestration costs, and habitat fragmentation <cit.>. The spatial pattern of land use is of particular interest and has been useful in identifying factors that drive changes in land use at the disaggregated level <cit.>. In a more general sense, this issue of aggregated, disaggregated, and different scales of data availability and analysis can be related to fusion models <cit.>, data misalignment <cit.>, change of support <cit.> and disaggregation modelling or downscaling <cit.>. In this general context, different issues arise concerning the availability at different scales of information concerning the response variable, as in data fusion models <cit.>, or availability at different scales of relative information, as in downscaler models <cit.>. This implies extracting information from an aggregated scale to a disaggregated or continuous scale using an effect with a continuous spatial or spatio-temporal structure, structure that is commonly used in geostatistics to account for the data spatial dependence. This is possible to implement in INLA thanks to the definition of spatial models using the SPDE-FEM approach, which is integrated in the  package <cit.>.



 §.§ SPDE-FEM


The SPDE-FEM approach (which stands for Stochastic Partial Differential Equations and Finite Element Methods) allows for expressing the continuous spatial structure by approximating a Gaussian Field with a Gaussian Markov Random Field. This makes the inferential process efficient by reducing computational costs. This approach was initially proposed in <cit.> by means of an approximate stochastic weak solution to the SPDE in Equation (<ref>), proving that a stationary SPDE solution has a Matérn covariance function and allowing to calculate directly its precision matrix, bypassing the need for resource-intensive inverse processes. Thus, formally we get that the structured spatial effect ξ with precision τ is related to a non-structured Gaussian random effect (white noise) through a filtering operator (κ - Δ)^α/2: 

    [     (κ^2-Δ)^α/2τ·ξ(𝐬)=𝒲(𝐬),; 𝐬∈𝒟,  α=ν+ d/2,  κ>0,  ν>0, ]

where 𝐬 are the locations; κ is a spatial scale parameter positive defined and related to ρ and ν through κ=√(8ν)/ρ; α controls the smoothness of the performances, which by default is 2; τ regulates the variance, so it is also a parameter defined positive as τ^2=Γ(ν)[ Γ(α)(4π)^d/2κ^2σ^2]^-1; and 𝒟 is the spatial domain 𝒟⊂ℝ^d, being d the Euclidean spatial dimension of such spatial domain. Additionally, Δ=∑_i∂^2/∂ s_i^2 is the Laplacian operator and 𝒲 denotes a spatial stochastic Gaussian process with unit variance. Therefore, the solution for ξ(𝐬_i) leads to a Gaussian field with a covariance matrix defined by the Matérn function correlation 𝒞(h), where the covariance is the correlation multiplied by the marginal deviation:

    𝒞(h)=σ^2·2^1-ν/Γ(ν)(√(2ν)· h /ρ)𝒦_ν(√(2ν)· h/ρ).

In the above equation, σ is the marginal standard deviation, ρ is the spatial range, 𝒦_ν is a modified Bessel function of the second kind of order ν, where ν is a parameter of smoothness defined as ν=α-d/2. In INLA, α=2 is taken by default, and in our cases, the dimension of the analysis space is the plane (d=2), resulting in a constant smoothness parameter value ν=1. Furthermore, under the SPDE approximation, the spatial range ρ is considered as the distance at which the correlation value is close to 0.1 <cit.>. 

This approach allows defining a spatial effect whose precision structure is sparse and approximates the solution of a continuous Gaussian field with a variance-covariance structure belonging to the Matérn family; where one particular case of this family is the exponential covariance function when ν = 1/2. This spatial effect is defined by a Gaussian Markov Random Field (GMRF), 𝐮∼GMRF(0, 𝐐), whose precision matrix for the standard case of d=2 and ν = 1 can be expressed as 𝐐 = τ^2 (κ^4 𝐂 + 2κ^2 𝐆 + 𝐆𝐂^-1𝐆). Here, 𝐂 and 𝐆 are two matrices that can be computed through the structure of a constrained refined Delaunay triangulation 𝒯 by Finite Element Methods. The GMRF realisations at the node locations 𝐬_n can be projected to any other point within the mesh domain, usually to the observation locations 𝐬_o, through a linear basis approximation synthesised in the matrix product of the projection matrix 𝐀(𝐬_n, 𝐬_o) with the GMRF realisation at the mesh nodes 𝐮(𝐬_n), 𝐮(𝐬_o)=𝐀(𝐬_n, 𝐬_o)𝐮(𝐬_n). This allows the spatial effect 𝐮(𝐬_o) to be rewritten as 𝐮(𝐬_o) ∼GMRF(0, 𝐀^T𝐐𝐀), which is clearly equivalent to 𝐮(𝐬_o) = 𝐀(𝐬_n, 𝐬_o) 𝐮(𝐬_n).



 §.§ Downscaling model with SPDE-FEM


The definition of a spatial effect whose structure is continuous in space and space-time allows for spatial and spatio-temporal downscaling. Let's assume we have the following structure for the linear predictors of the model of our data: 

    [                                      η_i,1                                          = 𝐗β_1 + ∑_k=1^Kf_k,1(z_i) + u_st,1(C_i,t_i);                                                                                     ⋮                                           ;                                      η_i,D                                          = 𝐗β_D + ∑_k=1^Kf_k,d(z_i) + u_st,D(C_i,t_i) ]

the downscaling is defined for the spatial structure when, for each area C_i, the spatially structured effect is calculated as follows:

    u_st(C_i, t_i) = ∫_𝐬∈ C_i u_st(𝐬,t_i) d𝐬/|C_i|≈∑_i=1^n(𝐬_i∈ C_i) u_st(𝐬_i,t_i)/n(𝐬_i∈ C_i) = 𝐀(𝐬_n, s_i)𝐮_st(𝐬_n,t_i)/n(𝐬_i∈ C_i)

where s_i are integration points, allowing the approximation of the integral in the region C_i by averaging the sum of the field at the integration points, and n(s_i ∈ C_i) is the number of integration points in C_i; being the area of C_i determine as |C_i| = ∫_𝐬∈ C_id𝐬. If a spatio-temporal downscaling is implemented, then the integration would be performed in both dimensions, spatial s and temporal t, using integration points for each corresponding volume C_i × t_i. Additionally, this procedure can also be applied to establish sub-models for those covariates with a spatial structure that are given in areas. These sub-models would involve the modelling of such covariates by incorporating the integral of the covariate scaled by a linear regression coefficient, analogous to how it is done in error models <cit.> in cases without downscaling. The linear predictors in Equation (<ref>) can be associated with either a Dirichlet distribution or a multivariate normal distribution related to some of the logratio transformations previously shown. If they correspond to a logratio transformation, it is necessary to re-write the linear predictors to incorporate the variance-covariance structure Σ of the multivariate normal distribution, as explained in previous sections.

The downscaling approach not only provides a way to obtain estimates on a continuous scale of the spatial effect but, due to its structure and implementation, also allows us to consider the same spatial effect for different spatial supports. This has several implications, as detailed in the initial references. One particular implication we would like to highlight is that when the spatial support varies over time—such as in the case of small areas for disease mapping or NUTS3—the use of a downscaled spatial or spatio-temporal effect ensures a coherent structure across the different supports, maintaining the spatial or spatio-temporal dependence of the data. Furthermore, this does not depend on the specific use of the SPDE-FEM approach, as it can be implemented with any other Gaussian field, such as spline and two-dimensional spline bases, or any other continuous field.



§ BIG DATA




Public smart card data, Wi-Fi access point data, wireless sensor networks, and data with spatial location information —such as social media data, mobile phone tracking, and other sensing information from Internet of Things devices— can provide useful ancillary data for LULC (Land Use and Land Cover) mapping. Compared to traditional geospatial data acquisition, these geospatial big data (GBD) are typically obtained at a lower cost and offer different coverages and better spatio-temporal resolutions <cit.>. They contain abundant human activity information, which can compensate for the lack of socioeconomic data <cit.>. By leveraging both Remote Sensing (RS) and GBD, it is possible to examine the physical and socioeconomic characteristics of the urban land system <cit.>. Despite the great potential of integrating RS and GBD for enhanced insights into urban land use, significant challenges remain in storing, managing, analysing, and visualising these data due to differences in spatial data quality (e.g., semantics, timestamps, and scale), technical formats, and data structures <cit.>. Therefore, a large amount of land use information is accessible, particularly with global and low-scale land use data, which gives rise to a big data problem. 


1.



This entails dealing with very heterogeneous and high dimensional data, particularly when data are available at global scale <cit.> or when LULC data at regional scales are available at fine grid <cit.>. The joint analysis of data that are at different scales and whose spatial structure is in different supports, as well as combining different sources of information, means that modelling structures can be particularly complex <cit.>. In order to deal with these large databases and complex modelling structures, sequential inference procedures can be implemented in INLA, which reduces the computational cost. In particular, a sequential consensus inferential procedure can be established and implemented in  <cit.>. This procedure, summarised in Figure <ref>, involves the marginal updating of fixed effects and hyperparameters over a given partition of the data. The partition of the data can be done along the various sources of information available, such as different likelihoods or groups of likelihoods, or by leveraging the structure of the latent field (spatial, temporal, or spatio-temporal). Once the sequential inference is performed, the random effects information is combined using a consensus approach according to their (i) marginal or (ii) multivariate distributions. This approach also accommodates complex models with shared components between different likelihoods.

This procedure reduces computational burden in terms of memory and CPU usage by solving the dataset or the original model through partitions. While it provides good estimates for the latent field, the evaluation of hyperparameters may differ from that obtained by analysing the full model without partitioning. This is because only the marginal information is updated, and hyperparameters can exhibit non-negligible correlations in the posterior distribution.



§ EXAMPLES


In this section, several examples are provided to illustrate the implementation of the various methodological approaches described in the previous sections. These range from handling zeros and ones using adaptations of Hurdle models—widely used in environmental and ecological sciences—to methods for downscaling and procedures for managing large databases or complex models.

The first example demonstrates how to handle compositional data with 0's and 1's, using simulated data. The second example illustrates the results of a downscaling model applied to real land use data from the European LAMASUS project. Finally, the third example uses a large simulated dataset to showcase the effectiveness of the sequential consensus procedure in analysing extensive datasets and accurately estimating the underlying process.



 §.§ Dealing with 0's example


In this first example, we present two cases that illustrate how to handle the presence of zeros in CoDa. The first case uses data simulated from a Beta distribution to exemplify the simplest scenario, proving the Hurdle model for dealing with zeros. The second case also addresses CoDa, analysing 3 categories through a multivariate model of the CLR transformation.

In the Beta case, we simulate the data from the following hierarchical conditional model:

    [                              Z_i                                ∼                        Ber(π_i),;                       logit(π_i)                                = β_B + α·(β_0 + β_1 · x_i + u_i),;                    Y_j | Z_j = 1                                ∼                    Beta(μ_j, ϕ),;                       logit(μ_j)                                =          β_0 + β_1 · x_j + u_sj, ]

where Z_i are Bernoulli random variables explaining the process that generates the incidence vector (incidence matrix for CoDa), such that z_i=0  y_i=0 and z_1  y_i ≠ 0. Y_j are Beta random variables conditioned on Z_j=1, which implies that Y_j≠ 0. The latent field comprises an intercept β_0, a linear coefficient β_1 related to a covariate 𝐱, and a spatial effect 𝐮_s following a Leroux structure. The spatial adjacency structure is built by partitioning mainland Spain with a Voronoi diagram, as shown in Figure <ref>. This latent field is shared in the linear predictor of the Bernoulli process, scaled by α and including an intercept β_B that controls the overall proportion of 0's. Finally, ϕ stands for the precision of the Beta distribution.


Figure <ref> shows the simulated spatial structure and the distribution of null and non-zero values, along with the inferential results of the beta model without incorporating the zeros and the Hurdle model incorporating the null values. It can be seen that, in general, the Hurdle model better captures the true values of the spatial effects where there are many zeros, as well as the posterior distributions of the model parameters.


The CoDa simulation is performed considering that only one (𝐘_1) of the 3 categories (𝐘_1, 𝐘_2, 𝐘_3) can present zero values. This model serves as a conceptual example of how it can be implemented for CoDa, dealing with zeros and ones in several categories. Therefore, the model structure when no null values are present is as follows:

    [                Z_i1∼Ber(π_i),; Y_j1| Z_j1 = 1 ∼N(μ_j1, τ^*),; Y_j2| Z_j1 = 1 ∼N(μ_j2, τ^*),; Y_j3| Z_j1 = 1 ∼N(μ_j3, τ^*),;                               ]

where the linear predictor for the CLRs are defined with the same structure as the one used for the Beta example with the addition of the 𝐮 effect to take into account the correlation between the categories, η_id = β_0d + β_1d x_i + u_sid + u_i. However, when Z_i1 = 0 the CLR change to the remaining two categories (𝐘_2, 𝐘_3) using the same underlying process in the linear predictors as before.


Figure <ref> shows the simulated spatial effects and the distribution of the zero and non-zero values for the first composition (Comp. 1). The figure also shows the inferred spatial effects (mean and stdev.) for the different compositions, along with the distributions for the fixed parameters and hyperparameters. 


In both cases we can observe an improvement in the estimation of the model by incorporating the Hurdle-type structure to integrate the analysis of the zeros.



 §.§ Downscaling example: LAMASUS data




In this example, we present the implementation of the downscaling procedure for real land use data from the European LAMASUS project.  In the context of the LAMASUS project, we aim to identify the components that determine or drive land use and land use change in Europe over the last decades. The data used in this example is spatially structured at the NUTS3 level between the years 2007 and 2018; where NUTS (Nomenclature of Territorial Units for Statistics) is the hierarchical system for dividing the economic territory of the European Union. The land use data analysed in this example is classified into five aggregated categories to avoid the presence of zero and unit values. These categories are: (i) cropland, (ii) grassland, (iii) forest, (iv) urban, and (v) other natural land. A wide range of explanatory variables are available to analyse the underlying processes that determine land use, including socio-economic and demographic variables such as GDP, GVA, and employment, as well as environmental variables like aspect and elevation.  



We begin by focusing on the proportion of agricultural land use, specifically the cropland category. To model exclusively the proportion of cultivated land use, we employ a downscaling model using a Beta distribution. Specifically, we use a spatio-temporal hierarchical Bayesian model, as outlined below:

    [                                   Y_it                                      ∼                          Beta(μ_it, ϕ);                                       ;                            logit(μ_it)                                      = βX_it + ∫_𝐬∈ C_iu(s) d𝐬/| C_i | + u_t,;                                        ]

where Y_it represents the random variable for the proportion of cropland use in a specific region i and year t. The conditional distribution of Y_it follows a Beta distribution with mean μ_it and precision ϕ. The mean is linked through the logit function to the linear predictor. The covariates are represented by 𝐗_it, and the vector of the linear regression coefficients are β. The downscaling spatial effect is u(𝐬), while the temporal effect is modelled as a first-order autoregressive process u_t=ϕ_t u_t-1 + ϵ(τ_t), where ϕ_t and τ_t are the temporal autoregressive parameter and the marginal precision of the temporal autoregressive effect, respectively. The selection of covariates to be included in the model was performed by exploring all possible combinations of models without spatial and temporal effects, selecting the combination with lower Watanabe-Akaike Information Criterion (WAIC, ) is selected. 


Some results can be seen in Figure <ref>. As the figure shows, the temporal effect is small. As for the spatial effect, it captures the variability not explained by the covariates. We can observe that, in certain areas, the spatial effect has a greater impact on the overall average effect than in other regions, where its impact is less pronounced. Regarding the covariates, some notable ones include decoupled payments (in euros) normalized by the total used agricultural area (in hectares), and less favoured area subsidies (in euros) normalized by the total used agricultural area (in hectares). Additionally, other relevant variables are the area of the region, elevation of the terrain, GVA, and total output (in euros).





The analysis for the 5 aggregated land use categories is performed using a multivariate Gaussian model with the ALR transformation of the categories. To select the covariates, we implemented a stepwise search algorithm driven by the WAIC of the models after a pre-selection where high correlated (ρ>0.75) variables were extracted. This algorithm involves the following steps: (i) a forward's search in the covariates until no new covariate can be included, then (ii) a backward step until no covariate can be deleted, and (iii) repeating steps (i) and (ii) until there is no change. The base model into which the algorithm is implemented encompasses a specific intercept, spatial, and temporal component for each ALR. The final model includes 9 explanatory covariates, and the model for the mean μ = (μ_1, …, μ_4) of the multivariate Gaussian distribution MVN(μ, Σ) is written as follows:

    [                                 μ_i1                                    = β_01 + 𝐗_iβ_1 + u_si1 + u_ti1 + u_i,;                                 μ_i2                                    = β_02 + 𝐗_iβ_2 + u_si2 + u_ti2 + u_i,;                                 μ_i3                                    = β_03 + 𝐗_iβ_3 + u_si3 + u_ti3 + u_i,;                                 μ_i4                                    = β_04 + 𝐗_iβ_4 + u_si4 + u_ti4 + u_i,;                                      ]

where u_sid is the downscaled spatial effect, u_sid = ∫_𝐬∈ C_i u_sd(𝐬)d𝐬 /|C_i|, and the component 𝐮 accounts for the correlation Σ^-1 between the categories, as explained in the corresponding section.

Figure <ref> shows the downscaled continuous spatial effects for the different ALRs, together with the temporal effects. 




 §.§ Big Data example


In this final example, we present the implementation of the algorithm summarised in Figure <ref> for evaluating large compositional data databases. The purpose of the simulation is not merely to demonstrate the results of the algorithm implementation for handling large databases, but also to showcase its versatility and its application to spatio-temporal downscaling models. This enables the processing of structured data in areas with spatial support that varies over the years.

The simulation is conducted for three categories evaluated using the ALR transformation, so the multivariate Gaussian structure will have a dimension of 2. The spatial structure is simulated using a continuous Gaussian field aggregated over two different spatial supports structured in areas, making the downscaling model purely spatial. The Gaussian field is a separable spatio-temporal effect, defined by the following precision matrix: 𝐐_st = 𝐐_s ⊗𝐐_t. The spatial precision matrix is defined using an SPDE in two dimensions, while the temporal part is determined by a first-order autoregressive structure with 600 (temporal) nodes. To construct the different spatial supports, the territory of mainland Spain has been partitioned using Voronoi diagrams.







The model used to simulate and infer the data is as follows:

    [ μ_it1 = β_01 + ∫_𝐬∈ C_iu_1(𝐬,t)d𝐬 + u_it,; μ_it2 = β_02 + ∫_𝐬∈ C_iu_2(𝐬,t)d𝐬 + u_it,;                                           ]

where β_0d are the intercepts, 𝐮_d(𝐬,t) are the spatio-temporal effects aggregated over the different spatial supports for each ALR. Finally, the effect 𝐮 accounts for the correlation structure between the different ALRs, allowing for the implementation of the multivariate Gaussian distribution in .

Figure <ref> shows the mesh for the spatial effect and the different spatial supports in which the data was aggregated over the various years. It also presents the aggregated spatial effect using simulated data, as well as the desegregated spatial effects obtained from the inferential process of the two ALRs for the first temporal node. The data set  was divided along the temporal scale, grouping into 6 temporal nodes, as larger grouping would exceed the server's memory capacity, which had 253 GB of RAM and 93 cores. Additionally, the complete analysis by the sequential consensus procedure for the entire dataset took 314.28 minutes. 





§ CONCLUSIONS


In the field of land use science, the application of spatial or spatio-temporal models is essential for evaluating the key factors that modify and direct land use and land use changes. Assessing spatial patterns alongside temporal patterns provides a precise understanding of their distribution in space and their temporal evolution. To achieve this, approaches often use either large-scale aggregated data or small-scale disaggregated data. In the latter case, when small-scale data is available, two natural problems may arise: the presence of null values and computational complications when handling large databases. Conversely, when dealing with large-scale data, the issue of changing the spatial aggregation support over time may occur. This study presents methodologies to address the problem of null values, downscaling procedures for spatial or spatio-temporal structures, and big data challenges. These methodologies are illustrated with various examples to demonstrate their implementation.

The results presented in the examples show the improvement when the presence of zeros is integrated into the analysis, along with the ability to perform dowscaling or disaggregation models using continuous Gaussian fields, specifically through the SPDE-FEM approach. Additionally, the implementation of a dowscaling model on a large dataset across different aggregation supports was shown. This highlights the possibility of encountering aggregated data in different spatial structures over the years, and how to deal with them in a Big Data context. This can be achieved by implementing the sequential consensus algorithm; otherwise, computational limitations would render the analysis unfeasible. 







§ ACKNOWLEDGMENTS




DC and ALQ thank support by the grant PID2022-136455NB-I00, funded by Ministerio de Ciencia, Innovación y Universidades of Spain (MCIN/AEI/10.13039/501100011033/FEDER, UE) and the European Regional Development Fund. DC also acknowledges Grant CIAICO/2022/165 funded by Generalitat Valenciana. 

This study is part of the ThinkInAzul program and is financed by the MCIN with funds from the European Union (NextGenerationEU-PRTR-C1711) and by the Generalitat Valenciana GVA-THINKINAZUL/2021/021. 









 Paper Title: A Ball Divergence Based Measure For Conditional Independence Testing.



On the use of calibration data in error-aware compilation techniques for NISQ  devices
    Handy Kurniawan


Universidad Complutense de Madrid

Madrid, Spain 

handykur@ucm.es
Laura Rodríguez-Soriano


Universitat Politècnica de València

Valencia, Spain 

lrodsor1@disca.upv.es
Daniele Cuomo


Universitat Politècnica de València

Valencia, Spain 

cuomo.daniele@outlook.com
Carmen G. Almudever


Universitat Politècnica de València
Valencia, Spain 

cargara2@disca.upv.es
Francisco García Herrero


Universidad Complutense de Madrid
Madrid, Spain 

francg18@ucm.es

    August 5, 2024
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT  

In this paper we introduce a new measure of conditional dependence between two random vectors  X and Y given another random vector Z using the ball divergence. Our measure characterizes conditional independence and does not require any moment assumptions. We propose a consistent estimator of the measure using a kernel averaging technique and derive its asymptotic distribution. Using this statistic we construct two tests for conditional independence, one in the model-X framework and the other based on a novel local wild bootstrap algorithm. In the model-X framework, which assumes the knowledge of the distribution of X| Z, applying the conditional randomization test we obtain a method that controls Type I error in finite samples and is asymptotically consistent, even if the distribution of X| Z is incorrectly specified up to distance preserving transformations. More generally, in situations where X| Z is unknown or hard to estimate,  we design a double-bandwidth based local wild bootstrap algorithm that asymptotically controls both Type I error and power. We illustrate the advantage of our method, both in terms of Type I error and power, in a range of simulation settings and also in a real data example. A consequence of our theoretical results is a general framework for studying the asymptotic properties of a 2-sample conditional V-statistic, which is of independent interest.  


Keywords:  Ball divergence, conditional independence testing, kernel density estimation, model-X framework, local wild bootstrap. 





§ INTRODUCTION

Conditional independence testing plays a central role in a variety of statistical applications such as graphical modeling <cit.>, causal discovery <cit.>, variable selection <cit.>, dimensionality reduction <cit.>, and computational biology <cit.>. Formally, given random vectors X,  Y,  Z, the conditional independence problem involves testing the hypothesis:   

    H_0:  X Y |  Z versus  H_1: X Y |  Z ,

based on i.i.d. samples {( X_i,  Y_i,  Z_i)}_1 ≤ i ≤ n from the joint distribution of ( X,  Y,  Z). This problem is well understood for parametric models, such as the multivariate Gaussian (where the classical partial correlation characterizes conditional independence) and for discrete data <cit.>. However, testing conditional independence is a particularly challenging  problem for continuous distributions in the nonparametric setting. For instance,  
<cit.> showed that if Z is continuous, then there is no unbiasedly estimable measure that characterizes conditional independence. Moreover, <cit.> (see also <cit.> and <cit.>) showed that for absolutely continuous (with respect to the Lebesgue measure) random variables it is not possible to have a non-trivial test with level asymptotically uniformly bounded by some given α∈ (0, 1) over the entire null hypothesis space H_0. Hence, pointwise control of Type I error is the best one can hope for, unless additional assumptions are made on the distribution of ( X,  Y,  Z). 

Numerous nonparametric conditional independence measures, with different moment and smoothness assumptions on the underlying distribution, have been proposed over the years. These include measures based on estimating conditional cumulative distribution functions <cit.>, conditional characteristic functions <cit.>, conditional probability density functions <cit.>, empirical likelihood <cit.>, kernel methods <cit.>, mutual information and entropy <cit.>, Rosenblatt transformations  <cit.>, copulas <cit.>, maximal nonlinear conditional correlation <cit.>, distance correlation <cit.>, estimating regression functions <cit.>, geometric graph-based measures <cit.>, among several others (see <cit.> for a more detailed review). 

To use a conditional dependence measure for testing the hypothesis (<ref>) one has to choose a rejection threshold for the corresponding  test statistic, which, as mentioned above, is a delicate matter that usually requires additional assumptions. An approach that has gained popularity recently is the model-X framework <cit.>, where one assumes that  _ X|  Z, the conditional distribution of X|  Z, is known. In this case, the conditional randomization test (CRT) can be implemented by repeatedly resampling from _ X|  Z to uniformly control the finite sample Type I error of any conditional dependence measure. This, in particular, has been applied to calibrate the Azadkia-Chatterjee conditional dependence measure <cit.> and the Kernel Partial Correlation (KPC) of <cit.>. <cit.> characterized the Type I error inflation of the CRT when an estimate of _ X|  Z is considered, and also demonstrate situations where this estimation error is asymptotically negligible. They also introduced a variant based on non-uniform permutations which is known as the Conditional Permutation Test (CPT). The CRT, however, is inapplicable in applications where _ X|  Z is unknown. In such situations there are two common approaches: (1) approximating the asymptotic null distribution <cit.> and (2) local bootstrap algorithms <cit.>. The former provides pointwise asymptotic Type I error control, although strong assumptions on the  estimation errors of the nuisance parameters are sometimes required for the results to hold. The latter, in its discrete variant  <cit.>, proceeds by sampling, for each Z_j, the observations in { X_i}_1 ≤ i ≤ n with probability proportional to their `distance' (in terms of some kernel) from Z_j, for 1 ≤ j ≤ n. However, theoretical understanding of the discrete local bootstrap for conditional independence testing is limited. Another approach that circumvents the estimation of _ X| Z are local permutation methods <cit.>, where one randomly perturbs the observations in { Y_i }_1 ≤ i ≤ n that are `close' to the conditioning variable Z_j, for 1 ≤ j ≤ n.  


In this paper, we propose a new measure of conditional dependence for Euclidean random vectors using the ball divergence <cit.>, a recently introduced robust measure of difference between 2 probability distributions. Our construction relies on the observation that X  Y|  Z if and only if the distributions of X |  Y,  Z and X |  Z are the same almost surely. Hence, a natural approach to constructing a conditional dependence measure is to quantify the difference between _ X|  Y,  Z and _ X|  Z, and averaging it (with a possible weight function) over the marginal distribution of ( Y,  Z). In this paper, we operationalize this approach using the ball divergence to measure the difference between the corresponding conditional distributions. We refer to this measure as the conditional ball divergence  (cBD) which is zero if and only if X  Y|  Z, that is, it characterizes conditional independence. Moreover, the cBD, unlike several other conditional dependence measures, such as the conditional distance correlation <cit.> and the generalized covariance measure <cit.>, does not require any moment assumptions, hence it is robust against outliers and extreme values. We illustrate the efficacy of the cBD in measuring and testing conditional independence through the following results: 

 


  * In Section <ref>, we propose an estimate of the cBD using a kernel averaging technique and study its large sample properties. To begin with we show that the cBD estimate is consistent, that is, the estimate converges to the population measure as the sample size increases (Theorem <ref>). Next, we derive the asymptotic distribution of both the pointwise and the averaged cBD estimates (see Theorem <ref> and <ref>, respectively). The latter, in particular, shows that the cBD estimate with an appropriate weight function is asymptotically normal under H_0. 


  * In Section <ref>, we calibrate the cBD estimate to develop tests for conditional independence in the model-X framework and using a novel local wild bootstrap algorithm. In the model-X framework, by repeatedly resampling from the known distribution of X |  Z one can implement the conditional randomization test (CRT) which controls level in finite samples and is also consistent with only finite number of resamples (Proposition <ref>). Furthermore, since the cBD estimate depends only on the ordering of the pairwise distances between the observations, our test remains consistent even if _ X |  Z is misspecified up to distance preserving transformations, such as, location shift, rotation, or homogeneous scaling.  The same also holds for the conditional permutation test (CPT) (see Proposition <ref>). In general, when _ X|  Z is not known, we propose a novel local wild bootstrap for calibrating the cBD statistic. Here, given Z_j, instead of directly resampling an X_i (based on their distance from Z_j), for 1 ≤ i, j ≤ n, we sample from a Gaussian distribution with mean X_j and a small variance (bandwidth), for 1 ≤ j ≤ n. This allows us to design a double-bandwidth strategy, where a larger bandwidth is used for computing the test statistic and a smaller bandwidth is used for the sampling, such that both Type I error and power are asymptotically controlled (Theorem <ref>). 


  * In Section <ref>, we compare the performance of the cBD test (calibrated using the local wild bootstrap) with other competing state-of-the-art methods both in simulations and on a benchmark data set. The cBD outperforms the other tests in a variety of simulation settings. Our method is especially powerful in detecting non-linear geometric dependencies and negative correlation between the pairwise distances. Our experiments also show that the discrete local bootstrap for calibrating conditional independence tests do not always control the Type I error in finite samples. In the real data analysis we are also able to achieve higher power with a smaller sample size compared to competing methods. 



A consequence of our theoretical analysis, is a general result about the asymptotic properties of a 2-sample conditonal V-statistic (Section <ref>). This framework can be applied more generally for studying the limiting properties of kernel density based estimates for conditional inference. As an illustration, we outline in Remark <ref> the derivation of the pointwise limit of the conditional distance correlation. We discuss possible extensions and directions for future research in Section <ref>. The proofs of the results are deferred to the Appendix. 



§ CONDITIONAL BALL DIVERGENCE

 

Suppose X,  Y, Z are random vectors in ℝ^d_X, ℝ^d_Y, and ℝ^d_Z, respectively. Throughout we will assume that the joint distribution of ( X,  Y,  Z) has density p_ X,  Y,  Z in ℝ^d_X+d_Y+d_Z, with marginal densities denoted by p_ X, p_ Y, p_ Z, p_ X,  Y, p_ Y,  Z, and p_ X,  Z, with respect to Lebesgue measures in the appropriate dimensions. We begin by recalling the definition of the ball divergence, specialized to the case of probability measures on Euclidean spaces <cit.>. Throughout we will assume that the Euclidean space is equipped with a metric ρ such that it becomes a separable metric space. 


The ball divergence between two probability measures P and Q in ^d is defined as

    Θ^2(P, Q)=∫∫ (P-Q)^2 B̅(  u, ρ(  v,  u)) {dP(  u) d P ( v)+ d Q( u) d Q(  v)},

where B̅(  u, ε) is the closed ball of radius ε centered at u in ^d. (Here, for any measurable set A ⊆^d, (P-Q)^2 (A) = (P(A) - Q(A))^2.)
 

A key property of the ball divergence is that Θ^2(P, Q) = 0 if and only if P=Q <cit.>. Now, note that X Y |  Z if and only if the distributions of X |  Y,  Z and X|  Z are the same almost surely. This motivates the following measure of conditional dependence.  
 
 
Suppose X,  Y, Z are random vectors in ℝ^d_X, ℝ^d_Y, and ℝ^d_Z, respectively, and a : ^d_Y×^d_Z→ [0, ∞) is a bounded, measurable weight function which is positive on the support of p_ Y, Z. Then the conditional ball divergence (cBD) between X and Y given Z, with weight function a, is defined as 

    ζ_a( X,  Y|  Z)
        = [ Θ^2(_ X|  Y,  Z, _ X|  Z) a( Y,  Z) ] ,

where the expectation is with respect to the joint distribution of ( Y,  Z). 
 
  
 
The following proposition shows that the cBD characterizes conditional dependence without any additional assumptions on the underlying distribution.

  For any weight function a as in Definition <ref>, ζ_a( X,  Y|  Z)≥ 0. Moreover, the equality holds if and only if X Y |  Z.



    The non-negativity of ζ_a( X,  Y|  Z) is obvious. Also, note that if X Y |  Z, then the distributions _ X|  Y,  Z and _ X|  Z are the same almost surely. Hence, by the distribution characterization property of the ball divergence <cit.>, Θ^2(_ X|  Y,  Z,_ X|  Z) = 0 almost surely. This implies, recalling (<ref>), ζ_a( X,  Y |  Z)=0, whenever X Y |  Z. 
    
    Conversely, if ζ_a( X,  Y |  Z)=0, then by the positivity of a we have, Θ^2(_ X|  Y,  Z, _ X|  Z)=0 almost surely. This implies, by the property of the ball divergence, _ X|  Y,  Z = _ X|  Z almost surely, hence, X Y |  Z.




Note that the measure ζ_a( X,  Y|  Z) is asymmetric in X and Y. It is easy to symmetrize the measure by considering the average/maximum of ζ_a( X,  Y|  Z) and ζ_a( Y,  Z|  Z). The subsequent results will continue hold for the symmetrized variants, however, for analytic simplicity, we study the asymmetric measure in this article. 
 


 
Similar to the Chatterjee-Azadkia conditional dependence measure <cit.>  and the Kernel Partial Correlation (KPC)  <cit.>, the cBD can be normalized to be within [0, 1], where 0 characterizes conditional independence and 1 characterizes complete conditional dependence (see Section <ref> and Proposition <ref> for further details).  Hence, the cBD, unlike covariance-based measures such as the conditional distance correlation <cit.>, has the ability to measure the `strength' of (possibly) non-linear dependence between X and Y given Z. 








 §.§ Consistent Estimation
 
 


We now describe a procedure for estimating the cBD based on i.i.d. samples {( X_i,  Y_i,  Z_i)}_1 ≤ i ≤ n from the joint distribution p_ X,  Y,  Z, using a kernel averaging method. Hereafter, we will choose  ρ to be the Euclidean metric, to be denoted by ·, although most of our results will continue to hold for general L_p metrics. Also, suppose K: [0,∞) → [0,∞) is a non-negative symmetric kernel function and h_1, h_2 > 0 are bandwidth parameters. Throughout, unless specified otherwise, vectors will be denoted in row format. Then for any Borel set A ⊂ℝ^d_X define, 

    ℙ_ X|  Y =  y,  Z =  z(A) :=∑_i=1^n K(( y,  z) -( Y_i,  Z_i) /h_1)I_A{ X_i}/∑_i=1^n K(( y,  z)  - ( Y_i,  Z_i)  /h_1)

and 

    ℙ_ X|  Z =  z(A) :=∑_i=1^n K( z-  Z_i/h_2)I_A{ X_i}/∑_i=1^n K( z -  Z_i/h_2)  ,
 
as the sample analogues of ℙ_ X|  Z =  z(A) and ℙ_ X| Y =  y,  Z =  z(A), respectively. Here, I_A( t) =  1 { t ∈ A} denotes the indicator function of the event A. Then the cBD measure (recall (<ref>)) can be estimated as follows:  

    ζ̂_a, n = 1/n∑_s=1^n Θ^2(ℙ_ X|   Y_s,  Z_s,ℙ_ X|  Z_s)a(   Y_s,  Z_s).

The estimate (<ref>) can be expanded out using the definition of the ball divergence (recall (<ref>)). For this, define 

    w_( Y_i,  Z_i)( y,  z) := K(( y,  z)  - ( Y_i,  Z_i)  /h_1)  and  w_ Z_i( z) := K( z -  Z_i /h_2) ,

and w( y,  z) := ∑_i=1^n w_( Y_i,  Z_i)( y,  z) and w( z) := ∑_i=1^n w_ Z_i( z). 
Then denoting δ( x,  y,  z) =  1 { z ∈B̅ (  x,  x -  y )}, the indicator that z lies inside in the ball  B̅ (  x,  x -  y ),  we can express (<ref>) as:  

    ζ̂_a, n    = 1/n∑_s=1^n ( A_n, s + C_n, s) a( Y_s,  Z_s) ,

where 

    A_n, s := ∑_1 ≤ u, v ≤ n( ∑_r = 1^n {w_( Y_r,  Z_r)( Y_s,  Z_s)/w( Y_s,  Z_s)   - w_ Z_r( Z_s)/w( Z_s)}δ( X_u,  X_v,  X_r) )^2
     w_ ( Y_u,  Z_u)  ( Y_s,  Z_s) w_ ( Y_v,  Z_v) ( Y_s,  Z_s) / w( Y_s,  Z_s)^2 ,
 
 and 
 
    C_n, s := ∑_1 ≤ u, v ≤ n( ∑_r = 1^n {w_( Y_r,  Z_r)( Y_s,  Z_s)/w( Y_s,  Z_s)   - w_ Z_r( Z_s)/w( Z_s)}δ( X_u,  X_v,  X_r) )^2
     w_ Z_u  ( Z_s) w_ Z_v( Z_s) / w( Z_s)^2 .
 

To derive the asymptotic properties of ζ̂_a, n we will need the underlying distributions to be smooth in a certain sense. This is formalized in the following definition: 



Let f( x,  y) be a density function with marginal densities denoted as f_ X and f_ Y, respectively. For any sequence δ_n that converges to zero define

    R_n( x,  y;  u) := 1/δ_n^2{f( x,  y+δ_n u)/f( x,  y)-1-δ_n ( ∂/∂ y f( x, y)/f( x, y))  u ^⊤}.

We say f is nice with respect to Y, if 

    sup_ u: u=1lim sup_n→∞∫(R_n( x,  y; u))^2 f( x, y)d x   is uniformly bounded in  y.




A sufficient condition for f( x,  y) to be nice in Y is that the variation of the first and second order partial derivatives of the logarithm of f_ Y| X are bounded (see Lemma <ref> in Appendix <ref> for details). 

To derive the large sample properties of ζ̂_a, n we will make the following assumptions on the distributions, the kernels, and the bandwidths: 


    
  *  The joint densities p_ X,  Z and p_ X,  Y,  Z are nice with respect to Z and ( Y, Z), respectively.  Moreover, p_ X,  Y,  Z and its marginals are uniformly bounded.
    
    
  *  The kernel K is uniformly bounded, non-negative, symmetric about zero, and compactly supported. Moreover,
       
    ∫_ℝ^d K( u) d u = 1  and ∫_ℝ^d u^⊤ u  K( u) d u= τ I_d ,
 
    where I_d is denotes the d × d identity matrix and τ > 0 is some constant. Here, d=d_Y+d_Z or d=d_Z depending on the context. 

    
  *  The bandwidths h_1,h_2 depend on the sample size n in such a way that 
    
    h_1^d_1/h_2^d_2→ 0, nh_i^d_i→∞,  and  nh_i^d_i+4→ 0,   for i = 1,2
 
    as n →∞, where d_1 = d_Y+d_Z and d_2 = d_Z.  (Here, d_Y and d_Z denote the dimensions of Y and Z, respectively). 




Assumption <ref> is standard in the nonparametric kernel regression and density estimation literature. For example, it is satisfied by the commonly used Epanechnikov kernel which we will also use in our experiments. Assumption <ref> ensures that h_1 and h_2 are neither too large nor too small, and they maintain a relative ordering between themselves. Under these assumptions we have the following result (see Appendix <ref> for the proof). 

 Suppose Assumptions <ref>, <ref>, and <ref> hold. Then, n →∞, 

    ζ̂_a, nP→ζ_a( X,  Y| Z),
  that is, ζ̂_a, n is a consistent estimator of ζ_a( X,  Y| Z). 








 §.§ Asymptotic Distribution
 



We now derive the asymptotic distribution of the cBD estimate. We begin by  reviewing some properties of the ball divergence. Towards this, let δ( x,  y,  z) =  1 { z ∈B̅ (  x, ρ( x,  y)} =  1 {ρ(  x,  z) ≤ρ( x,  y) } and define 
    η( x,   y,  z,  z') := δ( x,  y,  z) δ( x,  y,  z').
 
Then given 2 distributions P and Q and i.i.d. samples U_1,  U_2,  U_3,  U_4 and V_1,  V_2,  V_3,  V_4 from P and Q, respectively, one can write the ball divergence (<ref>) as (from the proof of <cit.>): 

    Θ^2(P, Q) = 𝔼 [ ϕ( U_1,  U_2,  U_3,  U_4;  V_1,  V_2,  V_3,  V_4)  ] ,
 
where 

    ϕ( u_1,  u_2,  u_3,  u_4;  v_1,  v_2,  v_3,  v_4) = ϕ_A( u_1,   u_2,  u_3,  u_4,  v_3,  v_4) + ϕ_C( v_1,  v_2,   v_3,  v_4,  u_3,  u_4) ,
 
with 

    ϕ_A( u_1,   u_2,  u_3,  u_4,  v_3,  v_4)    := η( u_1,   u_2,  u_3,  u_4) + η( u_1,   u_2,  v_3,  v_4) - η( u_1,   u_2,  u_3,  v_3) - η( u_1,   u_2,  u_4,  v_4) , 
    ϕ_C( v_1,  v_2,   v_3,  v_4,  u_3,  u_4)     := η( v_1,   v_2,  v_3,  v_4) + η( v_1,   v_2,  u_3,  u_4) - η( v_1,   v_2,  v_3,  u_3) - η( v_1,   v_2,  v_4,  u_4).
 
Note that |ϕ| ≤ 2. This shows that Θ^2(F, G) is a 2-sample divergence measure with a bounded core function ϕ of degree (4,4). Although function ϕ is not symmetric, it can be easily symmetrized as follows: 

    Θ^2(μ,ν) = 𝔼 [ ϕ'( U_1,  U_2,  U_3,  U_4;  V_1,  V_2,  V_3,  V_4)  ],

where ϕ' is the symmetrized version of the core function ϕ defined as, 

    ϕ'(  u;  v) = 1/4!^2∑_τ∈ S_4∑_σ∈ S_4ϕ( u_τ;  v_κ ) .

Here, S_4 denotes the set of all permutations of {1, 2, 3, 4}, u_τ = (  u_τ(1),  u_τ(2),  u_τ(3),  u_τ(4) ), and v_σ =( v_σ(1),  v_σ(2),  v_σ(3),  v_σ(4) ), for τ, σ∈ S_4. <cit.> showed that when P=Q, the function ϕ' is a first-order degenerate kernel, that is, 

    𝔼[ϕ'( U_1,  U_2,  U_3,  U_4;  V_1,  V_2,  V_3,  V_4)|  U_1] = 0  and 𝔼[ϕ'( U_1,  U_2,  U_3,  U_4;  V_1,  V_2,  V_3,  V_4)|  V_1] = 0 ,
 
almost surely, but [ϕ^'( U_1,…, U_4;  V_1,…,  V_4)| U_1, U_2] ≠ 0.



Using (<ref>), the cBD measure (<ref>) can be expressed as: 

    ζ_a, n = [
    ϕ'( X_1,  X_2,  X_3,  X_4;  X'_1,  X'_2,  X'_3,  X'_4)  a( Y,  Z) ],

where ( X,  Y,   Z) is distributed according to p_ X,  Y,  Z, X_1,  X_2,  X_3,  X_4 are i.i.d. _ X|  Y ,  Z, and X_1',  X_2',  X_3',  X_4' are i.i.d. _ X|  Z. 
We know from Theorem <ref> that ζ_a, n can be consistently estimated using   

    ζ̂_a, n = 1/n∑_s=1^n Θ^2(_ X|  Y_s, Z_s, _ X|  Z_s) a( Y_s, Z_s).

In the following theorem we obtain the limiting distribution of the individual terms in (<ref>), conditioned on Y=  y and Z =  z,  both under H_0 and H_1. Throughout, we interpret stochastic integrals in the Wiener sense <cit.> (see also <cit.>), where, unlike in the Wiener-Itô integral, the `diagonal' is not removed. 




  Assume the conditions of Theorem <ref> holds. Then we have the following results:

  
      
  (1) Under H_0, that is, X Y| Z, for any fixed y∈ℝ^d_Y and z∈ℝ^d_Z  with p_ Y,  Z( y,  z) > 0,
       
    nh_1^d_Y+d_ZΘ^2(ℙ_ X| Y =  y,  Z =  z, ℙ_ X |  Z=  z) 
       D→4 2∫ Q_0 ( x,  x';  y,  z) d𝔾_ℙ_ X|  Y=  y,  Z =  z( x) d𝔾_ℙ_ X|  Y=  y,  Z =  z( x') ,
 
       where 
        
       
       
  * Q_0 ( x,  x';  y,  z) = [ϕ'( x, x',  X_1, X_2 ;  X_3, X_4,  X_5, X_6)], 
       where the expectations are taken with respect to X_1,…,  X_6 ∼ℙ_ X| Y =  y,  Z =  z and ϕ' is defined in (<ref>).
       
       
  * 𝔾_ℙ_ X|  Y=  y,  Z =  z is a Gaussian process indexed by L^2 functions in ℝ^d_X with covariance function as in (<ref>). 
       

        


       
  (2) Under H_1, that is, X Y| Z, for any fixed y∈ℝ^d_Y and z∈ℝ^d_Z  with p_ Y,  Z( y,  z) > 0, we have 
       
    √(nh_1^d_Y+d_Z)(Θ^2(ℙ_ X| Y =  y,  Z =  z, ℙ_ X |  Z=  z)-Θ^2(ℙ_ X| Y =  y,  Z =  z, ℙ_ X |  Z=  z)) 
       D→ 4∫ Q_1 ( x;  y,  z) d𝔾_ℙ_ X|  Y=  y,  Z =  z( x) D = 4 N(0, Var__ X| Y=  y,  Z=  z[Q_1 ( X;  y,  z)) ] ,
 
       where 𝔾_ℙ_ X|  Y=  y,  Z =  z is as defined before and 
       
    Q_1 ( x;  y,  z) = [ϕ'( x, X_1, X_2, X_3;  X_1', X_2', X_3', X_4')]   ,

       with X_1,  X_2, X_3 ∼ℙ_ X| Y =  y,  Z =  z, X_1',…,  X_4' ∼ℙ_ X|  Z= z. 
  
  


The proof of Theorem <ref> is given in Appendix <ref>. The proof follows from a more general result about the limiting distribution of conditional V-statistics that might be of independent interest (see Section <ref>). 


 
Theorem <ref> shows that the pointwise limit of the cBD estimate has a Gaussian distribution under H_1 (see (<ref>)). 
However, under H_0, the asymptotic variance of the Gaussian is zero and the limit in (<ref>) is degenerate. This is because the ball divergence core function ϕ' is conditionally first-order degenerate under H_0 (see Definition <ref>). As a result, the limiting distribution in this case is given by a bivariate stochastic integral  (see (<ref>)). To express the limit in (<ref>) in an alternative form, note that by the spectral theorem, for y ∈ℝ^d_Y,  z ∈ℝ^d_Z fixed, 

    Q_0 ( x,  x';  y,  z) = ∑_i=1^∞λ_i ϕ_i ( x) ϕ_i( x'),

where {λ_i}_i ≥ 1 and {ϕ_i}_i ≥ 1 are the eigenvalues and the eigenvectors of the operator 

    ℋ(f)( x):= ∫ Q_0 ( x,  x';  y,  z) f(  x') d_ X| Y=  y,  Z=  z( x') ,

for f ∈ L_2(^d_X, _ X| Y=  y,  Z=  z). 
Denote U_i := ∫ϕ_i ( x) d𝔾_ℙ_ X|  Y=  y,  Z =  z( x), for i ≥ 1. By the orthogonality of the eigenvectors, {U_i}_i ≥ 1 is a collection of independent Gaussian random variables with mean zero and variance Var_ℙ_ X|  Y=  y,  Z =  z[ϕ_i ( X)]. Hence, from (<ref>), 

    ∫ Q_0 ( x,  x';  y,  z) d𝔾_ℙ_ X|  Y=  y,  Z =  z( x) d𝔾_ℙ_ X|  Y=  y,  Z =  z( x') D = ∑_i=1^n λ_i U_i^2 ,
 
which is the familiar representation of the distribution of a degenerate V-statistics as an infinite weighted sum of independent χ^2_1 random variables. 



Now, we proceed to derive the limiting distribution of the cBD estimate (<ref>). One challenge towards this is the random denominators in (<ref>) arising from the kernel density estimates. This issue is usually handled either by assuming the density of Y,  Z is bounded below or by choosing an appropriate weight function. Here, we implement the latter; as in <cit.>. The idea is to choose a weight function such that the random denominator in the corresponding estimate is cancelled. This leads us to define the following cBD measure with weight function a( y,  z) = a^*( y,  z) := p_ Y,  Z( y,  z)^4 p_ Z( z)^4: 

    ζ^*( X, Y| Z):=ζ_a^*( X, Y| Z) = [ Θ^2(_ X|  Y,  Z, _ X|  Z) p_ Y,  Z( Y,  Z)^4 p_ Z( Z)^4 ].

The natural empirical estimate of ζ^* is given by: 

    ζ̃_n^*    := 1/n∑_s=1^n Θ^2(ℙ_ X|   Y_s,  Z_s,ℙ_ X|  Z_s)p̂_ Y, Z^4( Y_s,  Z_s)p̂_ Z^4 ( Z_s) ,

where ϕ' is as defined in (<ref>) and p̂_ Y, Z( y, z) =w( y, z)/(nh_1^d_Y+d_Z) and p̂_ Z = w( z)/(nh_2^d_Z) are the kernel density estimators of p_ Y, Z and p_ Z, respectively (recall the discussion after (<ref>)). 
This estimate can be expanded out as a V-statistic, which incurs some bias  in the estimation procedure, due to the repetition of the indices in the summation. To avoid this, for the asymptotic results it is convenient to work with the following estimator, which takes the form of an U-statistic, 

    ζ̂_n^* := 1/(n)_9  ∑_ i ∈ ([n])_9φ_n(( X_i_1, Y_i_1, Z_i_1),( X_i_8, Y_i_8, Z_i_8),…,( X_i_9, Y_i_9, Z_i_9)) ,

where (n)_9 = n(n-1)⋯(n-8), (n)_9 is the set of all elements in [n]^9 with distinct indices, 

    φ_n(( X_1, Y_1, Z_1),…,( X_9, Y_9, Z_9)) 
        = 1/h_1^4(d_Y+d_Z) h_2^4d_Z∏_ℓ=2^5 w_( Y_ℓ, Z_ℓ)( Y_1, Z_1) ∏_ℓ=6^9w_ Z_ℓ( Z_1) ϕ' ( X_2,…,  X_5;  X_6, … X_9) ,
 
and ϕ' is the ball divergence core function (<ref>). The following theorem establishes the large sample properties of ζ̂_n^*.



 
  Assume the conditions of Theorem <ref> holds. Then, we have the following results: 
  
 
  (1) ζ^*( X, Y| Z)≥ 0 and the equality holds if and only if X Y| Z.
 
  (2) ζ̂_n^* is a consistent estimator of ζ^*( X, Y| Z), that is, 
 ζ̂_n^*P→ζ^*( X, Y| Z), as n→∞.

  (3) Under H_0:  X Y|  Z, if additionally we have n^2h_1^d_Y+d_Z+4→ 0 and h_2/h_1→ 0 as n→∞, then   
    
    nh_1^(d_Y+d_Z)/2ζ̂_n^*D→ N(0, 144  𝒞(K) σ^2),

    where 𝒞(K) := ∫(∫ K(( s,  t))K(( u,  v) - ( s,  t)) d s d t)^2 d ud v, 
    
    σ^2= ∫ Q_0^2( x_1,  x_2; y,  z) p_ X,  Y,  Z( x_1,  y,  z )p_ X,  Y,  Z( x_2,  y,  z ) p_ Y,  Z^6( y,  z) p_ Z^8( z) d x_1d x_2d yd z,
  
    with Q_0 as defined in Theorem <ref>. 
     
    


The proof of Theorem <ref> is given in Appendix <ref>. Note from (<ref>) that ζ̂_n^* is a U-statistics of order 9 with core function φ_n. The proof of Theorem <ref> proceeds by writing out the Hoeffding's projections of ζ̂_n^*. Under H_0, the first-order projection turns out to be asymptotically negligible and the limiting distribution is determined by terms arising from the second-order projection, which we analyze using results from <cit.> on degenerate U-statistics of order 2. For large samples, by estimating σ^2, we can use (<ref>) to obtain a test for conditional independence with precise asymptotic level. However, this approximation might not be very accurate for small samples. Moreover, although the U-statistic based estimate (<ref>) has a tractable limiting distribution, its computational complexity is very high. On the other hand, the  estimator proposed in Section <ref>, which can be expanded as a V-statistic, is computationally more efficient and it can be calibrated using a bootstrap or permutation method as discussed in Section <ref>, leading to a test that can detect conditional dependence even in moderately large samples. 



Another approach to reduce the computational complexity of the U-statistic based estimate (<ref>) is to consider the following estimator of ζ^*( X, Y| Z) which averages φ_n only over disjoint 9-tuples:

    ζ̂_n^L = 1/⌊ n/9 ⌋∑_i=1^⌊ n/9 ⌋φ_n(( X_9i-8, Y_9i-8, Z_9i-8),…,( X_9i, Y_9i, Z_9i)).

Clearly, ζ̂_n^L can be computed in 𝒪(n) time. Moreover, using similar arguments as in the proof of Theorem <ref> it can be shown that ζ̂_n^L is a consistent estimator of ζ^*( X, Y| Z) (under the same set of assumptions). However, the resulting test for conditional independence will be less powerful than the tests based on (<ref>) or (<ref>).  






§  LIMITING DISTRIBUTION OF CONDITIONAL V-STATISTICS

 
As is evident from the discussion in the previous sections, to derive the asymptotic properties in the cBD estimate (<ref>) one has to understand the limiting distribution of U/V-statistics that involve empirical estimates of the conditional distributions ℙ_ X|  Z =  z and ℙ_ X|  Y =  y,  Z =  z. Similar quantities appear in estimation of other nonparametric measures for conditional inference, such as the conditional distance covariance <cit.>, the tests in <cit.>, the conditional energy distance <cit.>, among others. In this section we distill the main techniques from our proofs and present a general result about the limiting distribution of a 2-sample conditional V-statistic, which can be broadly applied for deriving large sample properties of kernel density based estimates for conditional inference. 


We begin with a convergence result about a conditional empirical process. As before, we assume ( X_1,  Y_1,  Z_1), …, ( X_n,  Y_n,  Z_n) are i.i.d. samples with joint distribution _ X,  Y,  Z with density p_ X,  Y,  Z. Fixing y ∈^d_Y and z ∈^d_Z, let ℙ_ X|  Z =  z and ℙ_ X|  Y =  y,  Z =  z be the estimates of the conditional distributions ℙ_ X|  Z =  z and ℙ_ X| Y =  y,  Z =  z, respectively, as defined in Section <ref>. For bandwidths h_1 and h_2 satisfying Assumption <ref>, consider the conditional empirical processes, 

    S_n^ z :=    {√(nh_2^ d_Z )(∫ f( x) dℙ_ X|  Z =  z( d x)-_ℙ_ X|  Z =  z [ f( X) ] )| f∈ℱ_ z} , 
        
    
        S_n^ y, z :=    {√(nh_1^d_Y+d_Z)(∫ f( x) dℙ_ X |  Y= y,  Z =  z( d x)-_ℙ_ X|  Y=  y,  Z =  z [ f( X) ] )| f∈ℱ_ y, z},

where ℱ_ z := {f : ℝ^d_X→ℝ | ∫ f ( x )^2 dℙ_ X|  Z= z ( x) < ∞} and 
ℱ_ y,  z := {f : ℝ^d_X→ℝ | ∫ f ( x )^2 dℙ_ X|  Y =  y,  Z= z ( x) < ∞}. The following proposition establishes the finite-dimensional convergence of these processes. 



Suppose Assumptions <ref>, <ref>, and <ref> hold. Fix y ∈^d_Y and z ∈^d_Z such that p_ Z( z)>0 and p_ Y, Z( y, z)>0. Then the finite dimensional distributions of the processes
(S_n^ z, S_n^ y,  z) jointly converge to the finite dimensional distributions of 

    ( {∫ f( x) d𝔾_ℙ_ X|  Z =  z( x)| f∈ℱ_ z} , {∫ f( x) d𝔾_ℙ_ X|  Y=  y,  Z =  z( x)| f∈ℱ_ y,  z}) ,

where 𝔾_ℙ_ X|  Z =  z and 𝔾_ℙ_ X|  Y=  y,  Z =  z are independent Gaussian processes defined as follows: 




  * 𝔾_ℙ_ X|  Z =  z is a centered Gaussian process with covariance kernel, for f, g ∈ℱ_ z, 

    Cov[∫ f d𝔾_ℙ_ X|  Z =  z,∫ g d𝔾_ℙ_ X|  Z =  z] = c_K ( z) Cov_ℙ_ X|  Z =  z[ f( X), g( X)] ,
  
and c_K( z) := ∫_^d_Z K^2( u) d u/p_ Z( z).  


  * 𝔾_ℙ_ X|  Y=  y,  Z =  z is a centered Gaussian process with covariance kernel, for f, g ∈ℱ_ y,  z, 

    Cov[∫ f d𝔾_ℙ_ X|  Y=  y,  Z =  z,∫ g d𝔾_ℙ_ X|  Y=  y,  Z =  z]     = c_K ( y,  z) Cov_ℙ_ X|  Y=  y,  Z =  z[ f( X), g( X)] ,

where c_K( y, z) := ∫_^d_Y+d_Z K^2( u) d u/p_ Y, Z( y, z). 

   
 



The proof of Proposition <ref> is given in Appendix <ref>. 
Invoking this result we can derive the asymptotic properties of functionals of the conditional empirical process.  Specifically, given a core function ϕ:(^d_X)^r_1× (^d_X)^r_2→, for integers r_1, r_2≥ 0, consider the problem of estimating 

    θ := 𝔼[ϕ(  X_[r_1+r_2]) |{ Y_s =  y }_ 1  ≤ s ≤ r_1, { Z_s =  z }_1 ≤ s ≤ r_1+r_2  ] ,
 
where X_[r_1+r_2] = (  X_1, …,  X_r_1+r_2). 
     The plug-in empirical estimator of θ is given by: 
     
    θ̂_n     = ∫ϕ(x_1,…, x_r_1 ;  x_r_1+1,…, x_r_1+r_2) ∏_i=1^r_1d_ X| Y =  y, Z =  z ( x_i) ∏_j=r_1+1^r_1+r_2d_ X|  Z= z  ( x_j) .

     This is a 2-sample conditional V-statistic with a core function of order (r_1, r_2). To derive the asymptotic distribution of θ̂ we need to compute its Hoeffding's decomposition. For this, denote the symmetrization ϕ by: 
    
    ϕ'( x_1,…,  x_r_1+r_2) = 1/r_1! r_2!∑_τ∈ S_r_1∑_σ∈ S_r_2ϕ(  x_τ;  x_ r_1+ σ) ,
 
    where, for any integer r ≥ 1, S_r is the set of all permutations of {1, 2, …, r}, x_τ = (  x_τ(1), … x_τ(r_1) ), for τ∈ S_r_1, 
    and x_r_1+ σ = (  x_r_1+σ(1), … x_r_1+ σ(r_2) ), for σ∈ S_r_2. (Note that if ϕ is symmetric in the first r_1 and last r_2 coordinates, then ϕ' = ϕ.)  

 For 0 ≤ i ≤ r_1 and 0 ≤ j ≤ r_2, the (i,j)-th order (conditional) Hoeffding's projection of ϕ  is defined as: 

    ϕ_i, j ( x_1, …,  x_i;  x_1', …,  x_j') 
        := [ϕ'( x_1, …,  x_i,  X_i+1, …,  X_r_1;  x_1', …,  x_j',  X_r_1+j+1, … X_r_1+r_2)| 𝒢_ y,  z^(i, j)] ,

where 𝒢_ y,  z^(i, j) := {{ Y_s =  y,  Z_s =  z }_i+1 ≤ s ≤ r_1, { Z_s =  z }_ r_1+j+1 ≤ s ≤ r_1 + r_2} and ϕ' is the symmetrization of ϕ. 

 


  * The function ϕ is said to be conditionally non-degenerate, if either ϕ_1, 0 0 or ϕ_1, 0 0, almost surely. 


  * For k ≥ 1, the function ϕ is said to be conditionally degenerate of order k, if ϕ_i, j = 0, almost surely, for all a+b ≤ k and there exists a_0, b_0 such that a_0+b_0 = k+1 and ϕ_a_0, b_0≠0. 
 

  * The function ϕ is said to be conditionally completely degenerate, if ϕ_i, j = 0, almost surely, for all a+b < r_1+r_2. 

 




The following result gives the asymptotic distribution of θ̂, depending on the order of degeneracy of ϕ. Analogous results for one-sample conditional  U-statistics are derived in <cit.>. 

 Let θ and θ̂ be as defined in (<ref>) and (<ref>). Further, suppose 

    𝔼[ϕ^2(  X_[r_1+r_2]) |{ Y_s =  y }_ 1  ≤ s ≤ r_1, { Z_s =  z }_1 ≤ s ≤ r_1+r_2  ] < ∞,

and Assumptions <ref>, <ref>, <ref> hold. Then we have the following: 




  (1) θ̂_n = θ + R_n, where R_n=O_P((nh_1^d_Y+d_Z)^-1/2). 
          
          
  (2) If ϕ is conditionally non-degenerate, then 
              
    (nh_1^d_Y+d_Z)^1/2 (θ̂_n - θ ) D→  r_1∫ϕ_1, 0( x)  d𝔾_ℙ_ X|  Y=  y,  Z =  z( x)  D = N(0, r_1^2 σ^2) ,
 
     where σ^2 = c_K ( y,  z) Var_ℙ_ X|  Y=  y,  Z =  z [ ϕ_1, 0( X)]  and c_K( y, z) is defined in Proposition <ref>. 


          
          
  (3) If ϕ conditionally degenerate of order k, for some k ≥ 1, then 
     
    (nh_1^d_Y+d_Z)^(k+1)/2θ̂_n    D→r_1 k+1∫ϕ_k+1, 0( x_1…,  x_k+1)∏_i=1^k+1𝔾_ℙ_ X|  Y=  y,  Z =  z( x_i) ,
 
          where ϕ_k+1, 0 is the (k+1, 0)-th order Hoeffding's projection of ϕ. 
    

    



The proof of Proposition <ref> is given in Appendix <ref>. Proposition <ref>, in particular, gives us the asymptotic distribution of pointwise ball divergence Θ^2(ℙ_ X| Y =  y,  Z =  z, ℙ_ X |  Z=  z), both under the null and the alternative, as in Theorem <ref>. Moreover, as mentioned before, we expect this result to be useful in obtaining the pointwise limiting distribution of other nonparametric conditional independence or 2-sample estimates. We illustrate this in the following remark for the conditional distance covariance <cit.>. 



    The conditional distance covariance between 2 random vectors X and Y given another random vector Z =  z is defined as the square root of (see <cit.>): 
    
    𝒟( X, Y| Z= z) = 1/c_d_X c_d_Y∫_ℝ^d_X + d_Y|ϕ_ X, Y | Z =  z( t, s) - ϕ_ X| Z =  z( t)ϕ_ Y| Z =  z( s)|^2/ t^p+1 s^q+1d t d s ,
 
    where ϕ_ X |  Z =  z  (·), ϕ_ Y|  Z =  z  (·), and ϕ_ X,  Y|  Z =  z  (·),  
  are the conditional characteristic functions of X|  Z=  z, Y|  Z=  z, and ( X,  Y)|  Z=  z, respectively; and c_d_X = π^d_X/2/Γ((d_X+1)/2) and c_d_Y = π^d_Y/2/Γ((d_Y+1)/2). Following the derivation of <cit.>, (<ref>) can be alternatively expressed as:
    
    𝒟( X, Y| Z= z) = [φ(( X_1, Y_1),( X_2, Y_2),( X_3, Y_3),( X_4, Y_4))|{ Z_i =  z}_1≤ i≤ 4] ,
 
    where ( X_1, Y_1,  Z_1),( X_2, Y_2,  Z_2),( X_3, Y_3,  Z_3) are i.i.d. _ X, Y,  Z and 
   
    φ(( X_i, Y_i)_1≤ i≤ 4) =  X_1- X_2 Y_1- Y_2 +  X_1- X_2 Y_3- Y_4 - 2 X_1- X_2 Y_1- Y_3 .

Hence, the natural plug-in estimate of (<ref>) is: 
  
    𝒟_n^ z    = ∫φ(( x_i, y_i)_1≤ i≤ 4) ∏_i=1^4d_ X,  Y |  Z =  z ( x_i,  y_i)  .
 
It is known that 𝒟_n^ z is a consistent estimator of 𝒟( X, Y| Z= z) (see <cit.>), however, to the best of our knowledge, the asymptotic distribution of 𝒟_n^ z has not been previously explored. This can be addressed using Proposition <ref>.  
To  this end, note that the symmetrization φ^' of φ (as in (<ref>) with r_1=4 and r_2=0) is first-order degenerate under H_0 and is non-degenerate under H_1. Hence, using Proposition <ref> we have the following results. For this, we assume 
    
    𝔼[φ^2(( X_i, Y_i)_1≤ i≤ 4) |{ Z_s =  z }_1 ≤ s ≤ 4  ] < ∞

    and z ∈ℝ^d_Z is such that p_ Z( z) > 0.  
    
    
        
  * Under H_0: X Y| Z =  z we have,
        
    nh_1^d_Y+d_Z𝒟_n^ zD→4 2∫φ_2(( x_1,  y_1), ( x_2,  y_2)) ∏_i=1^2 d𝔾__ X, Y |  Z= z ( x_i,  y_i) D=6∑_i=1^∞λ_i U_i^2,

        where φ_2 is the second-order Hoeffding's projection of φ^' with respect to the distribution _ X, Y| Z= z as in Definition <ref>, {λ_i}_i ≥ 1 is a square integrable sequence, and {U_i}_i ≥ 1 is a sequence of i.i.d standard normal random variables. The distributional equality of the limiting random variable follows from spectral decomposition theorem as in Remark <ref>.  

        
  * Under H_1: X Y|  Z =  z we have,
        
    √(nh_1^d_Y+d_Z)(𝒟_n^ z - 𝒟( X, Y| Z= z))D→ 4 ∫φ_1( x ,  y )d𝔾__ X, Y| Z= z( x ,  y )D=𝒩(0,16σ^2),

        where φ_1 is the first-order Hoeffding's projection of φ^' with respect to the distribution _ X, Y| Z= z and σ^2= c_K( z) Var__ X, Y| Z= z[φ_1(  X ,  Y )].   
    




§ TESTING CONDITIONAL INDEPENDENCE
 



In this section, we will discuss methods to calibrate the cBD estimate for testing conditional independence. Throughout, we fix a weight function a and, suppressing the dependence on a,  denote the corresponding cBD measure by ζ( X,  Y| Z) and the estimate by ζ̂_n. 



 §.§ Tests under the Model-X framework



In the model-X framework <cit.> since the distribution of X|  Z is assumed to be known, one can implement the conditional randomization test (CRT) by repeatedly resampling from the distribution X|  Z to approximate the null distribution of the cBD estimate ζ̂_n. To describe this formally, denote the observed sample by 𝒟_0 := { (X_i, Y_i, Z_i)}_i=1^n.   The CRT calibration of the cBD estimate then proceeds as follows: 



  * Generate new observations X_1',  X_2',…,  X_n', where X_i^'∼ X|  Z_i, for 1 ≤ i ≤ n, and construct a new data set 𝒟' = { (X_i^',  Y_i,  Z_i)}_i=1^n. Repeat this M times to obtain resampled data sets 𝒟'_1,𝒟'_2…,𝒟'_M.  


  * Report the p-value 

    p_CRT = 1+∑_j=1^M 1{ζ̂_n(𝒟_j')≥ζ̂_n(𝒟_0)}/1+M,

where ζ̂_n(𝒟_j') denotes the test statistic computed on the j-th resampled data set 𝒟_j', for  1 ≤ j ≤ M. 

This procedure controls Type I error in finite samples and is also asymptotically consistent with a finite number of resamples as shown below (see Appendix <ref> for the proof). Throughout, we fix α∈ (0, 1). 


Suppose the assumptions of Theorem <ref> hold. Then the test function ϕ_CRT = 1{p_CRT<α}, controls Type I error in finite samples, that is, _H_0(ϕ_CRT) ≤α. Moreover, lim_n →∞_H_1(ϕ_CRT) =1,  for M>(1-α)/α. 



We can also calibrate the test using the conditional permutation test <cit.>, a variant of the CRT where one permutes the observations X_1,  X_2, …,  X_n non-uniformly in such a way that the dependence between X and Z is preserved. Specifically, we generate M permutations π_1,π_2,…,π_M of {1,2,…,n} independently from the distribution: 

    (Π = π| X,  Y,  Z) = ∏_ℓ = 1^n p( X_π(ℓ)|  Z_ℓ)/∑_π'∈ S_n∏_ℓ=1^n p( X_π'(ℓ)|  Z_ℓ) ,

where S_n is the set of all permutations of {1,2,…,n}, and p( x|  z) is the density function of the distribution of X|  Z= z. Then we construct resampled data sets 𝒟^π_j = {( X_π_j(i),  Y_i,  Z_i)}_i=1^n, for each 1 ≤ j ≤ M and compute the p-value: 

    p_CPT = 1+∑_j=1^M 1{ζ̂_n(𝒟^π_j)≥ζ̂_n(𝒟_0)}/1+M.

Here, as before, 𝒟_0 := { (X_i, Y_i, Z_i)}_i=1^n denotes the original data set and ζ̂_n(𝒟^π_j) is the test statistic computed on 𝒟^π_j, for 1 ≤ j ≤ M. 
This method also has the desired level and power properties as stated in the following proposition (see Appendix <ref> for the proof).  


Suppose the assumptions of Theorem <ref> hold. Then the test function ϕ_CPT = 1{p_CPT<α}, controls Type I error in finite samples, that is, _H_0(ϕ_CPT) ≤α. Moreover, lim_n →∞_H_1(ϕ_CPT) =1,  for M>(1-α)/α. 




 

In the model-X framework the knowledge of X|  Z plays a crucial role in controlling the Type I and Type II error rates. Naturally, misspecification of the distribution of X|  Z may lead to incorrect inferences. To investigate the effect of model misspecification on the CRT, suppose one resamples from the model ^*_ X|  Z, where _ X|  Z^* ≠_ X|  Z. Note that even if the model is misspecified, the CRT will have the desired error control properties if the distribution of ζ̂_n(𝒟') matches with the null distribution of ζ̂_n(𝒟_0). Towards this, note that the function ϕ' in (<ref>) depends only on the ordering of the pairwise distances of the observations { X_1,  X_2, …,  X_n}. Hence, the resampled test statistic and the observed test statistic will have the same distribution if _X |  Z can be obtained from _X |  Z^*  using a distance preserving transformation, that is, a translation, rotation or homogeneous scale transformation (the same change in scale along all coordinate axes). This is because the joint distribution of 1{ X_i- X_j≤ X_k- X_j}, for 1≤ i< j< k≤ n, w_( Y_i, Z_i)( Y_j,  Z_j), for 1≤ i< j≤ n, and w_ Z_i( Z_j), for 1≤ i< j≤ n,  will remain unchanged under such transformations. To illustrate this consider the following numerical example: Suppose 

    Z∼ N(0,1),   Y = Z + ε_1,  and  X = Z + rY+ ε_2,
 
where ε_1,ε_2 are i.i.d N(0,1) random variables. It is easy to see that when r=0, X Y| Z and when r≠0, X Y| Z. Here, we consider the cases r=0 and r=1. Note that for r=0, X| Z∼ Z + N(0,1) and for r=1, X| Z∼ 2Z+N(0,2). 
To compute our test statistic we set the bandwidths as: h_1 = c_1 n^-1/(d_Y+d_Z+2) and h_2 = c_2n^-1/(d_Z+2), where c_1 and c_2 are the averages of the coordinate-wise IQR of the observations { (Y_1, Z_1), …, (Y_n, Z_n) } and { Z_1,  …, Z_n}, respectively. Throughout the nominal level is set to α= 0.05. 




 Here, we consider _ X| Z^* ∼ 5Z + η, where η∼ N(10,25/(r+1)), to generate the resampled data when r=0 and r=1. Note that this is a scale and location shift of the actual distribution _ X| Z, hence the distributions of ζ̂_n(𝒟_0) and ζ̂_n(𝒟') should be the same. To check this we use the Kolmogorov-Smirnov (KS) test to evaluate the closeness of the distributions of ζ̂_n(𝒟_0) and ζ̂_n(𝒟'). The empirical power of the KS-test (computed based on 500 trials) for different sample sizes is reported in Figure <ref>. The power was close to the size 0.05 for all sample size for both r=0 and r=1, confirming that the distributions of ζ̂_n(𝒟_0) and ζ̂_n(𝒟') are indeed the same in the case, despite model misspecification. 





 
In this case we consider ^*_ X | Z =Unif(-|Z|,|Z|) to generate the resampled data. As before the empirical power of the KS-test (computed based on 500 trials) for different sample sizes is reported in Figure <ref>. The plots clearly shows that here the resampled test statistic had a significantly different distribution than that obtained from correctly specified _ X| Z. 









The above discussion shows that although the CRT with the cBD statistic is robust to distance preserving transformations, if the model is completely misspecified the CRT may lead to incorrect conclusions. In such cases one has to rely on a data-driven approach for approximating _ X| Z. This is discussed in the following section.  





 §.§ Local Wild Bootstrap Tests



In this section we propose a new bootstrap algorithm that approximates the density of _ X|  Z using kernel smoothing methods and generates the resampled data from this estimated distribution. 
Towards this, fix bandwidths h_0 and h_2' and consider the following estimate of the probability density function of X| Z, based on the data { ( X_i,  Z_i) }_1 ≤ i ≤ n: 

    p̂_ X|  Z= z( x|  z) = p̂_ X, Z( x, z)/p̂_ Z( z)    = 1/nh_0^d_Xh_2^' d_Z∑_s=1^n K(z - Z_s /h_2') ϕ( x- X_s/h_0)/1/nh_2^' d_Z∑_s=1^n K(z - Z_s /h_2')
        = ∑_s=1^n κ_s( z) 1/h_0^d_Xϕ( x- X_s/h_0) ,

where ϕ denotes the probability density function of the standard normal distribution and 

    κ_s( z) :=  K(z - Z_s /h_2')/∑_j=1^n K( z- Z_ j/h_2')  ,

for 1 ≤ s ≤ n. Let _ X|  Z= z be the probability distribution function corresponding to p̂_ X|  Z= z( x| z). Note that (<ref>) takes the form of a Gaussian mixture model. Specifically, the law of a random variable X̂∼_ X|  Z= z can be described as follows:  X̂ is distributed as 𝒩( X_s, h_0^2  I_d_X) with probability κ_s( z), for 1 ≤ s ≤ n. Now, the local wild bootstrap method for calibrating the cBD statistic can be described as follows: 





  * Given the observed data 𝒟_0, generate X̂_i∼_ X|  Z_i, independently for each 1 ≤ i  ≤ n,  to get the resampled data set 𝒟̂ = {(X̂_i,  Y_i,  Z_i)}_i=1^n. Repeat this procedure M times to obtain resampled data sets 𝒟̂_1,𝒟̂_2,…,𝒟̂_M.



  * Report the p-value 

    p_LWB = 1+∑_j=1^M 1{ζ̂_n(𝒟̂_j)≥ζ̂_n(𝒟_0)}/1+M,

where ζ̂_n(𝒟̂_j) is our test statistic computed based on the data 𝒟̂_j, for 1 ≤ j ≤ n.  


Note that although the random variables {ζ̂_n(𝒟̂_j)}_1 ≤ j ≤ M are exchangeable, they do not have the same distribution as ζ̂_n(𝒟̂_0) under H_0. However, we can choose the bandwidths h_0 and h_2' in a such a way that the distributions of ζ̂_n(𝒟̂_1) and ζ̂_n(𝒟_0) become asymptotically close (in the total variation distance). As a consequence, the level of the local wild bootstrap test can be controlled at a desired level asymptotically and the test is also consistent in large samples.  This is formalized in the following theorem. 

Before proceeding with statement of the theorem, we consider a slight generalization. Note that one can consider implementing the local wild bootstrap method with any collection of (random) weight functions {γ_s}_1 ≤ s ≤ n where γ_s: ℝ^d_Z→ [0, 1] is such that ∑_s=1^n γ_s( z) = 1, for all z ∈ℝ^d_Z and 1 ≤ s ≤ n, and the weight functions only depend on the data through { Z_i}_1 ≤ i ≤ n. (Choosing γ_s = κ_s as in (<ref>) gives the method described above.) Of course, not all weight functions γ_s are meaningful, only those which `concentrate' around the data point Z_s and decay with distance from Z_s are statistically relevant. In the following theorem we show the asymptotic validity of the local wild bootstrap for weight functions satisfying a general condition. Implications of this condition are discussed after the theorem. 
 


    Suppose Assumptions <ref>, <ref>, <ref> hold, nh_0^2 → 0, 
    and p_ X, Z is nice with respect to X. Also, let {γ_s}_1 ≤ s ≤ n be a collection of weight functions such that  ∏_s=1^n γ_s( Z_s) P→ 1, 
    as n →∞. Then the test function ϕ_LWB = 1{ p_LWB<α} satisfies the following: 
    

  (a) lim_n, M →∞_H_0(ϕ_LWB) ≤α, that is, ϕ_LWB controls Type I error at α.

    
  (b) If M>(1-α)/α, lim_n →∞_H_1(ϕ_LWB) =1,  that is, the power of ϕ_LWB converges to 1, as n →∞. 
  
   


The proof of Theorem <ref> is given in Appendix <ref>. The result shows that the local wild bootstrap asymptotically controls Type I error and is consistent  when the weights functions satisfy ∏_s=1^n γ_s( Z_s) P→ 1. To understand this condition, consider, independently for each 1 ≤ s ≤ n, a random variable Ẑ_̂ŝ which takes value Z_i with probability γ_s( Z_i), for 1 ≤ i ≤ n. Then ∏_s=1^n γ_s( Z_s) is the probability that {Ẑ_s=  Z_s: 1≤ s ≤ n}. Hence, the condition ∏_s=1^n γ_s( Z_s) P→ 1 ensures that the resampled and the observed configurations are asymptotically unchanged, that is, each Z_s is resampled exactly once with high probability, for 1 ≤ s ≤ n. There are 2 natural ways to ensure the condition ∏_s=1^n γ_s( Z_s) P→ 1 holds: (1) choosing γ_s= κ_s as in (<ref>) with the bandwidth h_2' satisfying h_2' = o(n^-2/d_Z) (see Lemma <ref> in Appendix <ref>), and (2) choosing γ_s = δ_ Z_s, for 1 ≤ s ≤ n, where δ_ z denotes the point mass at the point z. We adopt the latter choice in our numerical studies in Section <ref>. 


Another interesting consequence of Theorem <ref> is that it allows us to choose the bandwidth h_0 separately from the bandwidths h_1, h_2 which are used for computing the test statistic.  This double-bandwidth approach is necessary to resolve the dichotomy between Type I error and power that is inherent in permutation/bootstrap calibration of conditional independence measures. The finer choice of h_0 ensures the Type I error control, while the coarser choices of h_1, h_2 ensure efficient estimation of the cBD estimate, leading to better power. This also bears resemblance with the double-binning local permutation method in  <cit.>, where a similar condition on bin-widths appears (see the following remark).

 
Recently, <cit.> proposed a local permutation method for conditional independence testing by partitioning the space of the conditioning random variable Z and permuting the variables ( X,  Y) within each bin, to approximate the null distribution. They proved asymptotic Type I error rate control of the local permutation method, in the total variation sense, when the bin width h is such that nh^2→ 0. However, to have good power a larger bin size is required for computing the test statistic. This leads to a double-binning approach, where they used a larger bin size to estimate the test statistic and a smaller bin size to calibrate the test. However, their method requires Z to have bounded support and only  the case where Z is univariate is considered. In contrast, our resampling algorithm is applicable even when Z has unbounded support or is multidimensional. 


To further investigate how well the local wild bootstrap performs under the null hypothesis, 
we test the closeness of the distribution of the resampled test statistic ζ̂_n(𝒟̂) and the distribution of the test statistic obtained using the CRT method ζ̂_n(𝒟^') (which assumes the knowledge of _ X |  Z) using the following simulation. For comparison we also implement the same for the discrete local bootstrap. Specifically, we consider the same setup as in Remark <ref>: 


    Z∼ N(0,1),   Y = Z+ϵ_1,  and X = Z +ϵ_2,
 where ϵ_1,ϵ_2 are i.i.d. N(0,1) random variables. Here, also we take the bandwidths h_1 = c_1n^-1/(d_Y+d_Z+2) and h_2 = c_2 n^-1/(d_Z+2), where c_1 and c_2 are the averages of the coordinate-wise IQR of the observations {(Y_1,Z_1),…, (Y_n,Z_n)} and {Z_1,…, Z_n}, respectively. For every n, we repeat the resampling algorithms 200 times and use the Kolmogorov-Smirnov (KS) test to evaluate the closeness of the respective distributions. The nominal level is kept fixed at α = 0.05. Our findings are reported in Figure <ref>. We observe that the null distribution of the local wild bootstrapped test statistic is close to the distribution of the CRT resampled test statistic. 
On the other hand, the distribution of the discrete local bootstrapped test statistic is drastically different. 

This difference diminishes with increasing sample size, but one may require a very large sample size for this difference to be negligible. 










 §.§ Choice of the Weight Function



Recall that in Section <ref> we considered the cBD measure with weight function a( y, z) = p_ Z( z)^4 p_ Y, Z( y, z)^4. This choice is convenient because the corresponding cBD estimate is asymptotically Gaussian under H_0. Although the limiting null distribution of the cBD estimate might be intractable for general weight functions, Theorem <ref> provides a way to calibrate the cBD estimate that is asymptotical consistent for any choice of the weight function a(·, ·). This raises the question: How does the choice of the weight function effect the finite-sample power of the cBD based tests? We explore this through a small simulation with three possible weight functions (a) a( y, z) =1, (b) a( y, z) = p_ Y, Z( y, z)^2, and (c) a( y,  z) = p_ Z( z)^4p_ Y, Z( y, z)^4. For ease of computation, here we calibrate the test using the V-statistic cBD estimates. 




Consider the model 
    Y = Z + η_1  and  X = Z + rY+η_2,
 
where η_1,η_2 and Z are i.i.d. N(0, 1). For each of the 3 weight functions, the empirical power of the corresponding cBD test is computed for different values of r keeping the sample size fixed at n=50. The bandwidths h_1, h_2 for computing the cBD statistic are set as in Example <ref>. Also, the bandwidth for the local bootstrap is chosen to be h_0 = 20 c_2 n^-1/1.95,  where c_2 is the IQR of {Z_1, … , Z_n}, which satisfies the assumption nh_0^2→ 0 as in Theorem <ref>. The results are reported in Figure <ref>. We see that the test corresponding to a( Y, Z) = 1 has the highest power, followed by a( y, z) = p_ Y, Z( y, z)^2 and a( Y, Z) = p_ Z( z)^4p_ Y, Z( y, z)^4. This shows that although the weight function a( Y, Z) = p_ Z( z)^4p_ Y, Z( y, z)^4 leads to a simple asymptotic null distribution, other choices can have better finite-sample performance. In particular, this example shows that choosing a( y, z)=1 might be beneficial in practice. 




§ ANALYSIS OF SIMULATED AND BENCHMARK DATA SETS
 


In this section, we evaluate the empirical performance of our test on simulated experiments and also on a benchmark data set. We compare our test with the tests based on the Conditional Distance Covariance <cit.>, the Generalized Covariance Measure <cit.>, and the weighted Generalized Covariance Measure <cit.>, which are referred to as the cDC test, GCM tests, and wGCM test, respectively. There are multiple ways to carry out the GCM test depending upon the method used to estimate the regression function. Here, we use the generalized additive model for that purpose. <cit.> proposed two different tests using the wGCM where one can either choose a weight function apriori and use it for the entire data or estimate the optimal weight function based on a training data set and apply the test on the validation data set. Here, also we use generalized additive models to estimate the regression functions and the corresponding tests are referred to as wGCM.fix test and wGCM.est test, respectively. 

In our experiments we choose h_1 = c_1 n^-1/(d_Y+d_Z+2) and h_2 =c_2 n^-1/(d_Z+2) to estimate the test statistic, where c_1 and c_2 are the averages of the coordinate-wise IQR of the observations { ( Y_1,  Z_1), …, ( Y_n,  Z_n) } and { Z_1,  …,   Z_n}, respectively.  For the local smooth bootstrap we let γ_s = δ_ Z_s, for 1 ≤ s ≤ n, which ensures ∏_s=1^n γ_s( Z_s) = 1 and choose h_0 = 20   c_2 n^-1/1.95. We refer to our test as the cBD test. In our simulated experiments, all p-values are computed based on 500 replications and the empirical power is estimated by the proportion of times H_0 is rejected over 500 repetitions. Throughout the nominal level is set at 5%. 




 §.§ Analysis of Simulated Data


In this section we compare the performance of the proposed test with existing methods in simulation settings. We begin with a simple one dimensional regression model:  

 Consider the regression model 
    Y = Z + η_1  and  X = Z + rY+η_2,
 where η_1 and η_2 are independent error terms and Z follows a univariate distribution. Here, X depends on both Y and Z when r≠0 and the dependence becomes stronger as |r| increases. We consider two cases: 


  (a) Z,η_1,η_2 are i.i.d. N(0,1) random variables and 

  (b) Z,η_1,η_2 are i.i.d. standard Cauchy random variables. 
 
Powers of different tests are computed as a function r with sample size fixed at n=50. The results are reported in Figure <ref>. In Example <ref> (a), the GCM test has the best performance, closely followed by the wGCM tests and the cDC test. Our test also has satisfactory performance, but its power is relatively lower compared to other tests. On the other hand, in Example <ref> (b), our test and the cDC test have competitive performance and they significantly outperform the regression-based tests (the GCM and wGCM tests). This shows the lack of robustness of the regression-based tests against outliers and extreme values generated from heavy-tailed distributions. While the cDC test has more power when |r| is small, the cBD test outperforms the cDC test for large |r|. 












 
Here, we consider bivariate distributions with geometric dependence structures.  Specifically, suppose Z∼ Unif(0,1), X = Z + η_1, and Y= Z + η_2, where (η_1,η_2) are generated from one of the six unusual bivariate distributions described in <cit.>. The scatter plot of observations from these six bivariate distributions (referred to as (a) Four Clouds, (b) W, (c) Diamond, (d) Parabola, (e) Two Parabolas, and (f) Circle)  are shown in Figure <ref>. It is easy to see that in the Four Clouds example, X Y| Z while for the rest we have XY| Z. Here we compute the power of the different tests for varying sample sizes, which are reported in Figure <ref>. Note that in each of these examples, the Pearson's correlation coefficient between (η_1,η_2) is zero. Hence, the residuals in the GCM and wGCM tests are expected to be uncorrelated, rendering these tests powerless. This is validated in Figure <ref>, where the GCM and wGCM tests have power close to the nominal level of 0.05 for all the examples. In the Four Clouds example, the powers of all tests are close to the nominal level α=0.05, as expected. In the W example, none of the tests have satisfactory performance. In the Diamond example, the power curve of our test gradually increases with the sample size, whereas those for the other tests remain near the nominal level. For the Parabola and Two Parabolas examples, the power curves of our test and cDC test both increase with the sample size, but our test significantly outperforms the cDC test. In the Circle example, our test also has significantly higher power compared to the other tests even when the sample size is moderately large. Overall, our test has the highest power among competing methods in detecting these unusual dependencies. 
 

 Suppose X, Y and Z are as in Example <ref>, with (η_1,η_2) generated from an equal mixture of two bivariate normal distributions. These bivariate distributions have mean zero and variance-covariance matrix Σ_1 and Σ_2, respectively. We consider two cases: 
    (a) Σ_1 = [ 1 0; 0 1 ], Σ_2 = [ 10  0;  0 10 ]   and    (b) Σ_1 = [  1  0;  0 10 ],  Σ_2 = [ 10  0;  0  1 ].
 
Figure <ref> shows the power of the tests for varying sample sizes. Here, the regression-based tests have very poor performance, because the  covariance between (η_1,η_2) is zero. In Example <ref> (a) our test slightly outperforms the cDC test. However, in Example <ref> (b) the cBD test significantly outperforms the cDC and the other tests. Note that in this case the correlation between pairwise distances of η_1 and  η_2 is negative, where distance correlation based methods tend to perform poorly. For instance, in the context of independence testing <cit.> noted that if the correlation between the pairwise distances is negative, the distance correlation test performs poorly even for moderately large sample size. 

 



 Here, we consider the bivariate analogue of Example <ref>. Specifically, suppose 
    Y =  Z + η_1 and  X=  Z+r Y+η_2,
 
where η_1 and η_2 are independent error terms and Z follows a bivariate distribution. We consider two scenarios: 
 

  (a) Z,η_1,η_2 are i.i.d. observations from a bivariate standard normal distribution, 

  (b) Z,η_1,η_2 are i.i.d. and the components are independent standard Cauchy random variables. 

We compute the power of the tests for different values of r and report them in Figure <ref>. In Example <ref> (a) our test has slightly lower power compared to the other tests (similar to the univariate case considered in Example <ref> (a)). However, in Example <ref> (b) only our test and the cDC test have satisfactory performances. Although, the cDC test has an edge over our test in this case, the probability of Type I error (the power at r=0) of the cDC test is 0.192, which is significantly higher than the nominal level of 0.05. On the other hand, our test controls Type I error at the nominal level. In that sense, the power of cDC and cBD tests are not comparable in this scenario. 
 



 


 
Now, we consider a multivariate analogue of Example <ref>. In particular, suppose Z = (Z_1,Z_2,Z_3) is generated uniformly from the unit cube [0,1]^3 and  
    (X,Y) = (max{Z_1,Z_2,Z_3}+ε_1, max{Z_1,Z_2,Z_3}+ε_2),
 where (ε_1,ε_2) are generated from the (a) Circle, (b) Parabola, and (c) Two parabola distributions as in Example <ref>. We compute the power of the tests for varying sample sizes and report them in Figure <ref>. Here, the cDC test has extremely poor performance with power equal to zero in all instances. The regression-based tests also perform poorly, since the errors are uncorrelated. Only our test is able to successfully detect this form of conditional dependence, and its power increases with the sample size. 



 





 §.§ Analysis of Benchmark Datasets
 


In this section, we analyze the `marks' data set available in the R package `bnlearn'. This data set contains marks obtained by students in the subjects Mechanics (M), Vectors (V), Analysis (An), Algebra (Al), and Statistics (S), all measured on a scale of 0-100. This dataset has been previously analyzed in <cit.> and <cit.> where the authors found that the marks obtained by the students in Statistics is conditionally dependent on that of Analysis given the marks on the rest of the subjects. They also found dependence of the marks in Mechanics and Vectors given the marks on the rest of the subjects. Therefore, considering these studies we consider testing (a) H_0: S An | (M,V,Al) and (b) H_0: M V| (S,An,Al) where we have evidence against the null. To compare the performance of the different methods we generate random sub-samples from the original data and carry out the tests based on those sub-samples. This procedure is repeated 500 times to estimate the powers of the tests by the proportions of times they reject  H_0. We evaluate the performance for different sub-sample sizes, and the results are given in Figure <ref>. In case (a), the GCM test has a better performance when the sample size is small, but our cBD test outperforms all other tests for relatively larger sample size. In case (b), when the sample size is small, the cDC test and GCM test has a better performance than our test, but when the sample size is large the cBD test significantly outperforms its competitors.    









§ EXTENSIONS AND FUTURE DIRECTIONS




In this paper, we propose a new measure of conditional dependence using the ball divergence. We estimate the measure using a kernel averaging method, study its asymptotic properties, and develop a test for conditional independence using a novel bootstrap approach. Our test exhibits improved performance both in simulations and real-data examples. 

Many extensions and variations of our method are possible. For instance, there has been a series of recent results, beginning with the breakthrough work of <cit.>, on measures of  dependence which are normalized between [0, 1], and takes the values 0 or 1, respectively, if and only if the corresponding pair of random variables is independent or one is a measurable function of the other almost surely (see <cit.> for a survey of recent developments). For  measuring conditional dependence, <cit.> proposed a correlation coefficient with similar properties, that is, it lies between 0 and 1, takes value 0 if and only if X  Y|  Z, and takes value 1 if and only if X is almost surely a measurable function of Y given Z. The kernel partial correlation (KPC) of <cit.> extends this to general topological spaces. In a similar vein, we can also construct a normalized measure for conditional dependence using the ball divergence. Specifically, taking the weight function a=1 in (<ref>), define the normalized cBD as follows: 

    ℛ( X,  Y|  Z) :=   [Θ^2(_ X|  Y,  Z, _ X|  Z) ) ]/𝔼 [∫Θ^2(_δ_ x, _ X|  Z) d_ X| Y, Z ( x)   ] .

Similar to the conditional dependence measures in <cit.> and <cit.> the normalized cBD satisfies the following trifecta of properties (see Proposition <ref>): (1) ℛ( X,  Y|  Z) ∈ [0, 1], (2) ℛ( X,  Y|  Z) = 0 if and only if X Y |  Z, and (3)  ℛ( X,  Y|  Z) = 1
if and only if X is a measurable function of Y and Z almost surely.

One can also consider defining a conditional dependence measure using the ball covariance of <cit.>. Given a pair of random vectors ( X,  Y) in ℝ^d_X+d_Y with joint distribution _ X,  Y
and marginal distributions _ X and _ Y, respectively, the ball covariance 
with weight functions ω_1 : ℝ^d_X×ℝ^d_X→ℝ_≥ 0 and ω_2 : ℝ^d_Y×ℝ^d_Y→ℝ_≥ 0, is defined as: 

    BCov_ω_1, ω_2^2( X,  Y) = ∫ (P_ X,  Y -P_ X⊗ P_ Y )^2    ( B̅(u_1, ρ(u_1, u_2)) ×B̅(v_1, ρ(v_1, v_2)) ) 
       ω_1(u_1, u_2) ω_2(v_1, v_2) {dP_ X,  Y(u_1, v_1) dP_ X,  Y(u_2, v_2)  } ,

where B̅(  u, ε) is the closed ball of radius ε centered at u in the appropriate dimension. The BCov characterizes (unconditional) independence under certain assumptions on the support of the distribution (see <cit.>). Using the BCov, one can define a conditional dependence measure between X and Y, given Z as follows: 

    cBCov_ω_1, ω_2, a^2( X,  Y |  Z) = [ BCov_ω_1, ω_2^2( X |  Z,  Y |  Z) a ( Z) ] ,

for some weight function a : ℝ^d_Z→ℝ_≥ 0. It can be shown that cBCov = 0 if and only if X Y |  Z, however, the conditions required on the support for this to hold can be restrictive in the context of conditional independence testing.  The cBCov can be estimated using a kernel averaging method similar to that in Section <ref>,  and its large sample properties can be studied using the framework developed in Section <ref>. While the cBCov resembles a classical covariance measure, the cBD can
measure the strength of dependence, as discussed in the previous paragraph. Exploring how the cBCov compares with the cBD is an interesting future direction. 
 






Acknowledgements: B. B. Bhattacharya was supported by NSF CAREER grant DMS 2046393, NSF grant DMS 2113771, and a Sloan Research Fellowship. 



apalike














§ PROOF OF THEOREM <REF>




Recall, from (<ref>), that ζ( X,  Y |  Z) = [Θ^2( ℙ_ X|  Y,  Z, ℙ_ X|  Z) a( Y,  Z)], where Θ^2(F, G) is the ball divergence between 2 probability distributions F and G as defined in (<ref>). Let ϕ be the core function of the ball divergence as in (<ref>). For notational convenience denote u = ( u_1,  u_2,  u_3,  u_4) and v = ( v_1,  v_2,  v_3,  v_4), and note that,

    Θ^2(_ X|  Y=  y,  Z =  z, _ X|  Z =  z) = ∫ϕ( u;  v) ∏_i=1^4 d_ X|  Y =  y ,  Z =  z ( u_i) ∏_i=1^4 d_ X|  Z =  z ( v_i).

The corresponding estimate is given by

    Θ^2(_ X|  Y =  y , Z =  z , _ X|  Z =  z ) = ∫ϕ( u;  v) ∏_i=1^4 d_ X|  Y =  y ,  Z =  z ( u_i) ∏_i=1^4 d_ X|  Z =  z ( v_i),

where 
ℙ_ X|  Y =  y ,  Z =  z (A) and ℙ_ X|  Z =  z (A) are as defined in (<ref>) and (<ref>), respectively. Now, using Proposition <ref> we have,

    Θ^2(_ X|  Y =  y , Z =  z , _ X|  Z =  z ) = Θ^2(_ X|  Y =  y , Z =  z , _ X| Z =  z ) + O_P(1/√(nh_1^(d_Y+d_Z))) .















































Therefore,   

    Θ^2(ℙ_ X|  Y =  y,  Z =  z,ℙ_ X|  Z =  z) a( y,  z) P→Θ^2( ℙ_ X|  Y =  y,  Z =  z, ℙ_ X|  Z =  z)a( y,  z) .
 
Then by the Dominated Convergence Theorem (DCT),

    Θ^2(ℙ_ X|  Y_1 ,  Z_1 ,ℙ_ X|  Z_1) a(  Y_1,  Z_1) P→Θ^2( ℙ_ X|  Y_1 ,  Z_1 , ℙ_ X|  Z_1 )a( Y_1,  Z_1) .
 
Moreover, since |Θ^2(ℙ_ X|  Y_1,  Z_1,ℙ_ X|  Z_1)| ≤ 2 and a( y, z) is bounded, 

    𝔼[ (
    Θ^2(ℙ_ X|  Y_1 ,  Z_1 ,ℙ_ X|  Z_1 ) a( Y_1,  Z_1) )^2 ]  ≤ 
    4[ a^2( Y_1,  Z_1) ] <∞.
 Hence, the sequence Θ^2(ℙ_ X|  Y,  Z,ℙ_ X|  Z) a( Y,  Z) is asymptotically uniformly integrable and from (<ref>) we have,   

    [ζ̂_a, n] = [ Θ^2(ℙ_ X |  Y_1,  Z_1,ℙ_ X|  Z_1) a( Y_1,  Z_1) ] →[Θ^2( ℙ_ X|  Y_1,  Z_1, ℙ_ X|  Z_1)a( Y_1,  Z_1)] = ζ_a( X, Y|  Z),
 
as n→∞. Therefore, the estimate ζ̂_a, n is asymptotically unbiased. 

To complete the proof of consistency, we will show that [ζ̂_a, n ]→ 0, as n→∞. To begin with, observe that 

    [ζ̂_a, n] = T_n^(1) + T_n^(2) ,
 
where 

    T_n^(1)    := 1/n[Θ^2(ℙ_ X|  Y_1,  Z_1,ℙ_ X|  Z_1) a( Y_1,  Z_1) ] , 
     
       T_n^(2)    : =  n(n-1)/n^2Cov[Θ^2(ℙ_ X|  Y_1,  Z_1,ℙ_ X|  Z_1) a( Y_1,  Z_1),Θ^2(ℙ_ X|  Y_2,  Z_2,ℙ_ X|  Z_2) a(  Y_2,  Z_2) ] .

Using |Θ^2(ℙ_ X|  Y_1,  Z_1,ℙ_ X|  Z_1)| ≤ 2 and the boundedness of a( y, z), it follows that 
    [Θ^2(ℙ_ X|  Y_1,  Z_1,ℙ_ X|  Z_1) a( Y_1,  Z_1)] = O(1),
 and hence, T_n^(1)→ 0. Therefore, it suffices to show that T_n^(2)→ 0. For this, using boundedness note that 

    [ ( Θ^2(ℙ_ X|  Y_1,  Z_1,ℙ_ X|  Z_1) a( Y_1,  Z_1) Θ^2(ℙ_ X|  Y_2,  Z_2,ℙ_ X|  Z_2) a( Y_2,  Z_2) )^2 ] ≤ 16 ( [ a^2( Y_1,  Z_1) a^2(  Y_2,  Z_2)] ) <∞.

Hence, (<ref>), uniform integrability, and DCT gives, 

    lim_n→∞𝔼[Θ^2(ℙ_ X|  Y_1,  Z_1,ℙ_ X|  Z_1) a( Y_1,  Z_1) Θ^2(ℙ_ X|  Y_2,  Z_2,ℙ_ X|  Z_2) a(  Y_2,  Z_2) ]  =  ζ_a( X,  Y |  Z)^2 .

Combining the above with (<ref>) gives, T_n^(2)→ 0. This implies, [ζ̂_a, n] → 0, as n→∞, which together with (<ref>) completes the proof of the Theorem <ref>.  




§ PROOF OF THEOREM <REF>






    
  (1) 
Note that under H_0:  X Y| Z, the core function of the ball divergence ϕ' (as in (<ref>)) is conditionally degenerate of order 1, in the sense of Definition <ref>. Hence, by Proposition <ref> (3) with r_1=r_2=4 and k=1, under H_0, 
    
    nh_1^d_Y+d_ZΘ^2(_ X| Y =  y,  Z =  z , _ X| Z= z) D→4  2∫ϕ_2, 0( x,  x') d𝔾__ X| Y=  y,  Z =  z( x)d𝔾__ X| Y=  y,  Z =  z( x') ,

    where ϕ_2, 0 is the (2, 0)-th order Hoeffding's decomposition of ϕ. Recalling (<ref>), it easy to see that 
ϕ_2, 0( x,  x') = Q_0( x,  x';  y, z),  
where Q_0 is as defined in Theorem <ref>. 

    
    
  (2) Under H_1: X Y| Z, the core function of the ball divergence ϕ' is conditionally non-degenerate.  Hence, by Proposition <ref> (2), 
    
    √(nh_1^d_Y+d_Z)(Θ^2(_ X| Y =  y,  Z =  z , _ X| Z= z)-Θ^2(_ X| Y =  y,  Z =  z, _ X| Z= z)) 
       D→ 4 ∫ϕ_1, 0( x) d𝔾__ X| Y=  y,  Z =  z( x) .

    From (<ref>),we have ϕ_1, 0( x) = Q_1( x;  y, z), where Q_1 is as defined in Theorem <ref>. 
    




§ PROOF OF THEOREM <REF>
 





 §.§ Proof of Theorem <ref> (1)
 

Recall from (<ref>)    

    ζ^*( X, Y| Z) = [Θ^2(_ X| Y, Z,_ X| Z)p_ Z^4( Z)p_ Y, Z^4( Y, Z)].

    Since Θ^2(_ X|  Y, Z,_ X| Z)≥ 0 we have ζ^*( X, Y| Z)≥ 0. The equality holds if and only if Θ^2(_ X|  Y, Z,_ X| Z)=0 almost surely, that is, if and only if _ X|  Y, Z=_ X| Z almost surely. This proves the result in Theorem <ref> (1).  
    
    


 §.§ Proof of Theorem <ref> (2)
 
    
   Recall the definition of ζ̂_n^* from (<ref>). The core function φ_n in (<ref>) can be expressed as: 
   
    φ_n(( X_1, Y_1, Z_1),( X_2, Y_2, Z_2),…, ( X_9, Y_9, Z_9))
        = 1/ h_1^4(d_Y+d_Z) h_2^4d_Z K_12⋯ K_15K_16'⋯ K_19' ϕ'( X_2,…,  X_5;  X_6,…,  X_9),

   where K_ts = K(( Y_t, Z_t)-( Y_s, Z_s)/h_1) and K_ts' = K( Z_t- Z_s/h_2). For notational convenience, let U_i:=( X_i, Y_i, Z_i). Now note,
    
    [ζ̂_n^*] = [φ_n( U_1, U_2,…,  U_9)]

    and
    
    [ζ̂_n^*] = n 9^-1∑_c=1^9 9 cn-9 9-c[φ_n^(c)( U_1,…,  U_c)] ,

    where
    
    φ_n^(c)( u_1,…,  u_c) = [1/9!∑_π∈ S_9φ_n( U_π(1),…,  U_π(9))| U_1= u_1,…,  U_c =  u_c ] ,

    is the c-th order Hoeffding's projection of φ_n (see Section 1.3 in <cit.> for the derivation of the above expressions). By Lemma <ref> we have, 
    
    [ζ̂_n^*]→ζ^*( X, Y| Z).
 
    Therefore, the result in Theorem <ref> (2) will follow if we can show [ζ̂_n^*] → 0, as n→∞. This is established in the following lemma: 
    
     For 1 ≤ c ≤ 9,  
    
    1/n^c[φ_n^(c)( U_1,…,  U_c)] → 0 ,

    as n→∞. Consequently, from (<ref>), [ζ̂_n^*] → 0. 
    
    
     Suppose U = ( X,  Y,  Z) is distributed with density p_ X,  Y,  Z. For c=1, 
    
    [φ_n^(1)( U)]     = [φ_n^(1)( U)^2] - ([φ_n^(1)( U)])^2.
 
    By Lemma <ref>, 
    ([φ_n^(1)( U)])^2 = ([ζ̂_n^*])^2→(ζ^*( X,  Y| Z))^2,  as n→∞. 
    Therefore, we need to find the rate of convergence of [φ_n^(1)( U)^2]. Note that, from (<ref>), we can write,
    
    φ_n^(1)( u)     = 1/9{[φ_n( u,  U_2,…,  U_9)] + 4 [φ_n( U_1,  u, U_3,…,  U_9)] + 4 [φ_n( U_1,  U_2,…,  U_5,  u,  U_7,…,  U_9)]}
        := 1/9{ϑ_1^(1)( u) + 4 ϑ_2^(1)( u)+ 4 ϑ_3^(1) ( u)} .
 
    By Lemma <ref>,
    
    [ϑ_1^(1)( U)^2]     = [[φ_n( U,  U_2,…,  U_9)][φ_n( U,  U_2',…,  U_9')]]*(where U,  U_2,…,  U_9, U_2',…,  U_9' are i.i.d.)
        =  [φ_n( U,  U_2,…,  U_9)φ_n( U,  U_2',…,  U_9')] 
       →∫[[ϕ'( X_2,…,  X_5;  X_6,…,  X_9)] p_ Z( z)^4 p_ Y, Z( y, z)^4]^2 p_ Y, Z( y, z)d y d z ,
 
    where the expectation is with respect to X_2,…,  X_5 i.i.d. from ℙ_ X |  Y= y,  Z =  z and X_6,…,  X_9 i.i.d. from _ X|  Z = z. Similarly, it can shown that [ϑ_2^(1)( U)^2] = O(1) and [ϑ_3^(1)( U)^2] = O(1). Hence, from (<ref>) and using (a+b+c)^2≤ 3(a^2+b^2+c^2), it follows that     
     [φ_n^(1)( U)^2] = O(1). Hence, 1/n[φ_n^(1)( U)] → 0, as n→∞. 
    
    Next, suppose c=2. Note from (<ref>) that,
    
    φ_n^(2)( u, u') = 1/72{ϑ_1^(2)( u, u') + ϑ_2^(2)( u, u') } ,

    where
    
    ϑ_1^(2)( u, u') :=      4 ϑ_1, 1^(2)( u, u') + 4 ϑ_1, 2^(2)( u, u') + 16 ϑ_1, 3^(2)( u, u') + 6 ϑ_1, 4^(2)( u, u') + 6 ϑ_1, 5^(2)( u, u') ,
 
    where 
    
    ϑ_1, 1^(2)( u, u')     := [φ_n( u,  u',  U_3,…,  U_9)] , 
    ϑ_1, 2^(2)( u, u')     := [φ_n( u,  U_2,…,  U_5;  u',  U_7,…,   U_9)] , 
    ϑ_1, 3^(2)( u, u')     :=  [φ_n( U_1,  u, U_3,…,  U_5;  u',  U_7,…,   U_9)] , 
    ϑ_1, 4^(2)( u, u')     := [φ_n( U_1,  u,  u',  U_4,  U_5;  U_6, …,   U_9)] ,
    ϑ_1, 5^(2)( u, u')     :=  [φ_n( U_1,  U_2, …,  U_5;  u,  u',  U_8,   U_9)]  ;

        and ϑ_2^(2)( u, u') = ϑ_1^(2)( u', u). For U,  U' are i.i.d. with density p_ X,  Y,  Z, it follows from (<ref>), (<ref>), and (<ref>),  
        
    h_1^(d_Y+d_Z)[φ_n^(2)( U, U')] →τ_2^2 ,
 
  for some constant τ_2^2 > 0. Hence, 1/n^2[φ_n^(2)( U,  U')] → 0, as n→∞. 

    Similarly, for any c=3,…, 9 it can be shown that there exists a constant τ_c^2 > 0 such that 
    
    h_1^(c-1)(d_Y+d_Z)[φ_n^(c)( U_1,…,  U_c)]→τ_c^2 ,

    as n→∞. This implies 1/n^c[φ_n^(c)( U_1,…,  U_c)] → 0, as n→∞. 
    
        
        The above lemma together with (<ref>) completes the proof of Theorem <ref> (2). 
        
        
        


 §.§ Proof of Theorem <ref> (3)
 

    
        
        To begin with, note that ζ̂_n^* is a U-statistics with kernel φ_n of degree 9. Let δ_n^(1)( u_1) = φ_n^(1)( u_1) and, for 2 ≤ c ≤ 9, 
  
    δ_n^(c)( u_1,…,  u_c) = φ_n^(c)( u_1,…,  u_c) - ∑_a=1^c-1∑_1≤ i_1≤⋯≤ i_a≤ cδ_n^(a)( u_i_1,…, u_i_a) ,
 
   where φ_n^(c)( u_1,…,  u_c) is as defined in (<ref>). From Section 1.6 in <cit.> we have,
   
    ζ̂_n^* = ∑_c=1^9 9 cΔ_n^(c) ,

   where Δ_n^(c) = n c^-1∑_1≤ i_1≤⋯≤ i_c ≤ nδ_n^(c)( U_i_1,…,  U_i_c) satisfies the following properties: For 1 ≤ c ≤ 9, 
   
       
  (a) The U-statistics Δ_n^(1), Δ_n^(2),…, Δ_n^(9) are uncorrelated.
       
  (b) [Δ_n^(c)] = n c^-1[δ_n^(c)( U_1,…,  U_c)].
       
  (c) [δ_n^(c)( U_1,…,  U_c)] = ∑_a=1^c (-1)^c-ac a[φ_n^(a)( U_1,…,  U_a)], for 1 ≤ c ≤ 9.
   
   
   We begin by showing that δ_n^(1) is asymptotically negligible. 
   
    Under H_0:  X  Y| Z, δ_n^(1)(  u) = O(h_1^4), almost surely. Moreover, 
    nh_1^(d_Y+d_Z)/2Δ_n^(1)L_2→ 0.

    

 
Note that 
   
    δ_n^(1)( u)     =φ_n^(1)( u) = 1/9{ϑ_1^(1)( u) + 4 ϑ_2^(1)( u)+ 4 ϑ_3^(1) ( u)} ,
 
   where ϑ_1^(1), ϑ_2^(1), and ϑ_3^(1) are as defined in (<ref>). 
   
    Note that 

    ϑ_1^(1)( u)     = [ φ_n( u, U_2,…, U_5;  U_6,…,  U_9)] 
        = 1/ h_1^4(d_Y+d_Z) h_2^4d_Z[∏_s=2^5 w_ Y_s, Z_s( y, z) ∏_s=6^9 w_ Z_s( z)ϕ'( X_2,…,  X_9)] 
        = p_ Y,  Z (  y,  z)^4  p_ Z ( z)^4[ ϕ'( X_2,  X_3,…,  X_9) |{ Y_s=  y }_2 ≤ s ≤ 5, { Z_s=  z }_2 ≤ s ≤ 9 ] +  O(h_1^2 h_2^2)  * (by Remark <ref>)
        = O(h_1^2 h_2^2) = O(h_1^4) ,
 
since ϕ' is conditionally degenerate of order 1 under H_0 and h_2/h_1→ 0 by assumption. Similarly, it can be shown that ϑ_2^(1)( u) = O(h_1^4) and ϑ_3^(1)( u) = O(h_1^4), almost surely. This implies, δ_n^(1)(  u) = O(h_1^4). 

Also, we have from Lemma <ref>, 

    [ϑ_1^(1)( U)]     = [ φ_n ( U, U_2,…, U_5;  U_6,…,  U_9)] 
       →∫ p_ Y,  Z (  y,  z)^5  p_ Z ( z)^4[ ϕ'( X_2,  X_3,…,  X_9) |{ Y_s=  y }_2 ≤ s ≤ 5, { Z_s=  z }_2 ≤ s ≤ 9 ] d y d z +  O(h_1^2 h_2^2)   
        = O(h_1^2 h_2^2) = O(h_1^4) .
 
Similarly, [ϑ_2^(1)( U)] = O(h_1^4) and [ϑ_3^(1)( U)] = O(h_1^4). Hence, 

    nh_1^(d_Y+d_Z)/2[Δ_n^(1)(  U)] = nh_1^(d_Y+d_Z)/2[ δ_n^(1)(  U) ] = O(nh_1^(d_Y+d_Z)/2 + 4) → 0 ,

since n^2 h_1^d_Y+d_Z+4→ 0.  
Next, from (<ref>) and Lemma <ref> we have [ϑ_1^(1)( U)^2] = O(h_1^8). Similarly, it can be shown that [ϑ_2^(1)( U)^2] = O(h_1^8) and [ϑ_3^(1)( U)^2] = O(h_1^8). This implies, 

    n^2h_1^(d_Y+d_Z)Var[ Δ_n^(1)(  U)] = nh_1^(d_Y+d_Z)Var[ δ_n^(1)(  U) ] = O(nh_1^(d_Y+d_Z + 8) )  → 0,

since n^2 h_1^d_Y+d_Z+4→ 0. Combining (<ref>) and (<ref>) shows that nh_1^(d_Y+d_Z)/2Δ_n^(1)L_2→ 0. 
  
   
   The above lemma shows that ζ̂_n^* is an approximately first-order degenerate U-statistics. Now, we consider the second-order Hoeffding's projection. 
   
       Under H_0:  X  Y| Z, 
             
    nh_1^(d_Y+d_Z)/2Δ_n^(2)D→ N(0, 144  σ_2^2 𝒞_2(K)) ,

              where σ_2^2 = σ^2 and 𝒞_2(K) = 𝒞(K) are as in the statement of Theorem <ref>. 
         


  
We have from (<ref>), δ_n^(2)( u,  u') = φ_n^(2)( u,  u') -  δ_n^(1)( u) - δ_n^(1)( u'). This means, 
   
    Δ_n^(2)    = 1/n  2∑_1 ≤ i < j ≤ nφ_n^(2)( U_i,  U_j) -  2/n∑_i=1^n δ_n^(1)( U_i) 
        = 1/n  2∑_1 ≤ i < j ≤ nφ_n^(2)( U_i,  U_j) -  o_P(1/nh_1^(d_Y+d_Z)/2) ,
 
by Lemma <ref>. Hence, it suffices to derive the limiting distribution 
  of n  2^-1∑_1 ≤ i < j ≤ nφ_n^(2)( U_i,  U_j). Towards this, recall from (<ref>), 
   
    φ_n^(2)( u, u') = 1/72{ϑ_1^(2)( u, u') + ϑ_2^(2)( u, u') } ,

where ϑ_1^(2) and ϑ_2^(2) are as defined in (<ref>).  Note that ϑ_1^(2)( u,  u') has 5 terms. The proof of Lemma <ref> now proceeds as follows: 



  * First we show that the U-statistics corresponding to the first 2 terms in (<ref>) are asymptotically negligible:   

    nh_1^(d_Y+d_Z)/2/n(n-1)∑_1 ≤ i  j ≤ nϑ_1, 1^(2)( U_i,  U_j) = o_L_2(1) ,  nh_1^(d_Y+d_Z)/2/n(n-1)∑_1 ≤ i  j ≤ nϑ_1, 2^(2)( U_i,  U_j) = o_L_2(1) ,
 
where o_L_2(1) is a term that converges to zero in L_2, as n →∞. 



  * Next, we show the following about the U-statistics corresponding to the last three terms in  (<ref>): 

    nh_1^(d_Y+d_Z)/2/n(n-1)∑_1≤ i  j ≤ nϑ_1, 4^(2) ( U_i, U_j)    → N(0, σ_2^2 𝒞_2(K)) ,
 
where σ_2^2 and 𝒞_2(K) are as in Lemma <ref>;   
and 

    nh_2^d_Z/2/n(n-1)∑_1≤ i  j ≤ nϑ_1, 3^(2) ( U_i, U_j)    → N(0, τ_3^2),  nh_2^d_Z/2/n(n-1)∑_1≤ i  j ≤ nϑ_1, 5^(2) ( U_i, U_j) → N(0, τ_5^2) ,
 
for some positive constants τ_3^2 and τ_5^2. Here, the convergence in (<ref>) and (<ref>) hold in distribution and in the first 2 moments. 



The proofs of the above results are given below. To complete the proof of Lemma <ref>, observe that (<ref>) implies, since h_2/h_1 → 0, 

    nh_1^(d_Y+d_Z)/2/n(n-1)∑_1 ≤ i  j ≤ nϑ_1, 1^(3)( U_i,  U_j) = o_P(1) ,  nh_1^(d_Y+d_Z)/2/n(n-1)∑_1 ≤ i  j ≤ nϑ_1, 2^(5)( U_i,  U_j) = o_P(1) .
 
Combining this with (<ref>) and (<ref>) the result in Lemma <ref> follows. 

 


  
    Recall the definition of ϑ_1, 1^(2) from (<ref>). Then by Lemma <ref> and Remark <ref>, 
       
    1/n(n-1)∑_1 ≤ i  j ≤ n[ ϑ_1, 1^(2)( U_i,  U_j) ] 
        =  [ϑ_1, 1^(2)( U_1,  U_2) ] 
    
               = 1/h_2^4d_Zh_1^4(d_Y+d_Z)∫[  ∏_s=2^5 w_ Y_s, Z_s( y_1,  z_1) ∏_s=6^9 w_ Z_s( z_1)ϕ'( X_2,  X_3,…,  X_9)] p_ Y,  Z( y_1,  z_1) d y_1 d z_1 
       →∫ p_ Y,  Z (  y,  z)^5  p_ Z ( z)^4[ ϕ'( X_2,  X_3,…,  X_9) |{ Y_s=  y }_2 ≤ s ≤ 5, { Z_s=  z }_2 ≤ s ≤ 9 ]  d y d z +  O(h_1^2 h_2^2)  
        = O(h_1^2 h_2^2) = O(h_1^4) .
 
This means, 

    nh_1^(d_Y+d_Z)/2/n(n-1)∑_1 ≤ i  j ≤ n [ ϑ_1, 1^(2)( U_i,  U_j) ] = O(nh_1^(d_Y+d_Z)/2 +4 ) → 0.
 




Next, we compute the variance of ∑_1 ≤ i  j ≤ nϑ_1, 1^(2)( U_i,  U_j). As before, by Lemma <ref> and Remark <ref>, 

    h_1^(d_Y+d_Z)/n^2∑_1 ≤ i  j ≤ n[ ϑ_1, 1^(2)( U_i,  U_j) ] 
       ≲ h_1^(d_Y+d_Z)[ ϑ_1, 1^(2)( U_1,  U_2) ^2 ] 
        = h_1^(d_Y+d_Z)[[φ_n( U_1,  U_2,  U_3, …,  U_9)][φ_n( U_1,  U_2,  U_3', …,  U_9')]]*(where U_1,…,  U_9, U_3',…,  U_9' are i.i.d.)
        = h_1^(d_Y+d_Z)[φ_n( U_1,  U_2,  U_3, …,  U_9)φ_n( U_1,  U_2,  U_3', …,  U_9')] 
       →{ C_K ∫([ γ'( X_2,  X_3, …,  X_9;  X_3', …,  X_9') ] p_ Z( z)^8 p_ Y, Z( y, z)^7 ) p_ Y, Z( y, z) d y d z  + O(h_1^2 h_2^2) }
        = O(h_1^2 h_2^2) = O(h_1^4) → 0 ,
 
where γ' is the symmetrized version of the function γ : (^d_X)^15→, 

    γ( x_2,  x_3, …,  x_9;  x_3', …,  x_9') :=ϕ^'( x_2,  x_3, …,  x_9) ϕ^'( x_2,  x_3', …,  x_9'),

the expectation is with respect to X_2,…,  X_5,  X_3', …,  X_5' i.i.d. from ℙ_ X |  Y= y,  Z =  z and X_6,…,  X_9,  X_6',…,  X_9' i.i.d. from _ X|  Z = z, and C_K is some constant depending on K. Since ϕ^' is conditionally first order degenerate under H_0, 

    [ γ'( X_2,  X_3, …,  X_9;  X_3', …,  X_9') ] = 0 ,

which explains the last step in (<ref>). Now, we consider the covariance terms, 

    h_1^(d_Y+d_Z)/n^2∑_1 ≤ i  j  k ≤ nCov[ ϑ_1, 1^(2)( U_i,  U_j),  ϑ_1, 1^(2)( U_j,  U_k)  ] 
       ≲ n h_1^(d_Y+d_Z)[ ϑ_1, 1^(2)( U_1,  U_2) ϑ_1, 1^(2)( U_1,  U_3) ] 
        = n h_1^(d_Y+d_Z)[[φ_n( U_1,  U_2,  U_3', …,  U_9')][φ_n( U_1,  U_3,  U_3”, …,  U_9”)]]  *(where U_1,  U_2,  U_3,  U_3', …,  U_9',  U_3”, …,  U_9” are i.i.d.)
        = n h_1^(d_Y+d_Z)[φ_n( U_1,  U_2,  U_3', …,  U_9') φ_n( U_1,  U_3,  U_3”, …,  U_9”) ]  
        = O(nh_1^(d_Y+d_Z)+4) → 0,
 
by Lemma <ref> and Remark <ref>.  Collecting (<ref>) and (<ref>) it follows that

    h_1^(d_Y+d_Z)/n^2[∑_1 ≤ i  j ≤ nϑ_1, 1^(2)( U_i,  U_j) ] → 0 .

 From (<ref>) and (<ref>) the first result in (<ref>) follows. The second result in (<ref>) can be proved similarly. 


   

We will only prove (<ref>). The results in (<ref>) can proved similarly. To begin with recall the definition of ϑ_1, 4^(2) from (<ref>). Then 

    ϑ_1, 4^(2)( u,  u') 
        = [φ_n( U_1,  u,  u',  U_4,  U_5;  U_6, …,   U_9)] 
        = 1/h_1^4(d_Y+d_Z)h_2^4d_Z[ w_ y, z( Y_1, Z_1) w_ y', z'( Y_1, Z_1)∏_s = 4^5 w_ Y_s,  Z_s( Y_1, Z_1) ∏_s = 6^9 w_ Z_s( Z_1) ϕ'( x, x', X_4, …, X_9)] ,
 
       where u = ( x,  y,  z) and u' = ( x',  y',  z'). Then note that, 
       
    [ϑ_1, 4^(2)( U,  U')^2]
        = 1/h_1^8(d_Y+d_Z)h_2^8d_Z
           [ w_ Y, Z( Y_1, Z_1) w_ Y', Z'( Y_1, Z_1) ∏_s = 4^5 w_ Y_s,  Z_s( Y_1, Z_1) ∏_s = 6^9 w_ Z_s( Z_1) ϕ'( X, X', X_4, …, X_9)
                w_ Y, Z(Ỹ_1,Z̃_1) w_ Y', Z'(Ỹ_1,Z̃_1)∏_s = 4^5 w_Ỹ_s, Z̃_s(Ỹ_1,Z̃_1) ∏_s = 6^9 w_Z̃_s(Z̃_1) ϕ'( X, X',X̃_4, …,X̃_9)] ,
 
       where {Ũ_i = (X̃_i, Ỹ_i, , Z̃_i) }_1 ≤ i ≤ 9 are i.i.d. with density p_ X,  Y,  Z which is independent of  { U_i = (  X_i,  Y_i, ,  Z_i) }_1 ≤ i ≤ 9. To compute (<ref>), we first condition on U_1,Ũ_1, U, and U' to get, 
              
    [ϑ_1, 4^(2)( U,  U')^2| U, U',  U_1, Ũ_̃1̃]
        = 1/h_1^8(d_Y+d_Z)h_2^8d_Z w_ Y, Z( Y_1, Z_1) w_ Y', Z'( Y_1, Z_1) w_ Y, Z(Ỹ_1,Z̃_1) w_ Y', Z'(Ỹ_1,Z̃_1) 
       ∫{∏_s = 4^5 w_ y_s,  z_s( Y_1, Z_1) ∏_s = 4^5 w_ỹ_s, z̃_s(Ỹ_1,Z̃_1) ∏_s = 6^9 w_ z_s( Z_1) ∏_s = 6^9 w_z̃_s( Z_1) ϕ'( X, X', x_4, …, x_9) ϕ'( X, X',x̃_4, …,x̃_9) 
             ∏_s=4^5 p_ X,  Y,  Z( x_s,  y_s,  z_s) ∏_s=4^5 p_ X,  Y,  Z(x̃_s, ỹ_s, z̃_s)  ∏_s=6^9 p_ X,  Z(  x_s,  z_s)  ∏_s=6^9 p_ X,  Z(x̃_s, z̃_s) }d𝒱 ,
 
         where d𝒱 := ∏_s=4^9 d x_s ∏_s=4^9 dx̃_s ∏_s=4^5 d y_s ∏_s=4^5 dỹ_s ∏_s=4^9 d z_s ∏_s=4^9 dz̃_s. Now, by the change of variables: 
    y_s =  Y_1 + h_1  u_s,  z_s =  Z_1 + h_1  v_s, ỹ_s = Ỹ_1 + h_1 ũ_s, z̃_s = Z̃_1 + h_1 ṽ_s,
 for s ∈{4, 5}, and 
         
    z_s =  Z_1 + h_2  v_s, z̃_s = Z̃_1 + h_2 ṽ_s,
 
         for s ∈{6, 7, 8, 9} we get, 
              
    [ϑ_1, 4^(2)( U,  U')^2| U, U',  U_1, Ũ_̃1̃] 
        = 1/h_1^4(d_Y+d_Z) w_ Y, Z( Y_1, Z_1) w_ Y', Z'( Y_1, Z_1) w_ Y, Z(Ỹ_1,Z̃_1) w_ Y', Z'(Ỹ_1,Z̃_1) 
       ∫{∏_s = 4^5 K(  (  u_s,  v_s )   ) ∏_s = 4^5 K(  ( ũ_s, ṽ_s )   ) ∏_s = 6^9 K(  v_s  ) ∏_s = 6^9 K( ṽ_s  )  ϕ'( X, X', x_4, …, x_9) ϕ'( X, X',x̃_4, …,x̃_9) 
             ∏_s=4^5 p_ X,  Y,  Z( x_s,  Y_1 + h_1  u_s,  Z_1 + h_1  v_s) ∏_s=4^5 p_ X,  Y,  Z(x̃_s, Ỹ_1 + h_1 ũ_s, Z̃_1 + h_1 ṽ_s) 
             ∏_s=6^9 p_ X,  Z(  x_s,  Z_1 + h _2  u_s)  ∏_s=6^9 p_ X,  Z(x̃_s, Z̃_1 + h _2 ũ_s) }d𝒱 ,
 
 where d𝒱 := ∏_s=4^9 d x_s ∏_s=4^9 dx̃_s ∏_s=4^5 d u_s ∏_s=4^5 dũ_s ∏_s=4^9 d v_s ∏_s=4^9 dṽ_s.      
Denote the integral (<ref>) by γ( x, x^';  y_1, z_1,ỹ_1,z̃_1, h_1,h_2). Then taking expectation on both sides of (<ref>) gives, 
        
    [ϑ_1, 4^(2)( U,  U')^2] 
        = [1/h_1^4(d_Y+d_Z) w_ Y, Z( Y_1, Z_1) w_ Y', Z'( Y_1, Z_1) w_ Y, Z(Ỹ_1,Z̃_1) w_ Y', Z'(Ỹ_1,Z̃_1)γ( X, X^';  Y_1, Z_1,Ỹ_1,Z̃_1,h_1,h_2)]
        = 1/h_1^4(d_Y+d_Z)∫{ K(( y, z)-( y_1, z_1)/h_1) K(( y', z')-( y_1, z_1)/h_1) 
                                K(( y, z)-(ỹ_1,z̃_1)/h_1) K(( y, z)-(ỹ_1,z̃_1)/h_1) γ( x, x^';  y_1, z_1,ỹ_1,z̃_1 h_1,h_2)
                                p_ X,  Y,  Z( x, y, z)p_ X,  Y,  Z( x', y', z')p_ X,  Y,  Z( x_1, y_1, z_1)p_ X,  Y,  Z(x̃_1,ỹ_1,z̃_1) }d𝒯 ,
 
      where d𝒯 = d xd yd zd x'd y'd z'd x_1d y_1d z_1dx̃_1dỹ_1dz̃_1. By the change of variables: 
      ( y_1, z_1) = ( y,  z) + h_1 ( u_1,  v_1) and (ỹ_1, z̃_1) = (ỹ, z̃) + h_1 (ũ_1, ṽ_1) we get, 
      
    [ϑ_1, 4^(2)( U,  U')^2] 
        = 1/h_1^2(d_Y+d_Z)∫{ K(( u_1, v_1)) K(( y', z')-( y, z) - h_1( u_1, v_1)/h_1) 
                K((ṽ_1,ũ_1)) K(( y', z')-( y, z)-h_1(ṽ_1,ũ_1)/h_1) 
               γ( x, x^'; y + h_1 u_1, z+ h_1 v_1, y+h_1ũ_1, z + h_1ṽ_1, h_1,h_2) 
                p_ X,  Y,  Z( x, y, z) p_ X,  Y,  Z( x', y', z') p_ X,  Y,  Z( x_1, y + h_1  v_1, z+h_1 u_1) p_ X,  Y,  Z(x̃_1, y + h_1ṽ_1,  z + h_1ũ_1) }d𝒯̃ ,

      where d𝒯̃ = d xd yd zd x'd y'd z'd x_1d u_1d v_1dx̃_1dũ_1dṽ_1. Then by the change of variables: ( y',  z') = ( y,  z) + h_1 ( u' ,  v') we get, 
     
    [ϑ_1, 4^(2)( U,  U')^2] 
        = 1/h_1^(d_Y+d_Z)∫{ K(( u_1, v_1)) K(( u', v') - ( u_1, v_1))K((ũ_1,ṽ_1)) K(( u', v')-(ũ_1,ṽ_1)) 
               γ( x, x^';  y + h_1 u_1, z+ h_1 v_1, y+h_1ũ_1, z + h_1ṽ_1, h_1,h_2) p_ X,  Y,  Z( x, y, z) 
                p_ X,  Y,  Z( x', y + h_1  v',  z + h_1 u') p_ X,  Y,  Z( x_1, y + h_1  v_1, z+h_1 u_1) p_ X,  Y,  Z(x̃_1, y + h_1ṽ_1,  z + h_1ũ_1) }d𝒯' ,
 
       where d𝒯' = d xd yd zd x'd u'd v'd x_1d u_1d v_1dx̃_1dũ_1dṽ_1. 
                 Now, under Assumption <ref>, it can be shown that as n →∞, 
        
    h_1^d_Y+d_Z[ϑ_1, 4^(2)( U,  U')^2] 
       →𝒞_2(K) ∫[Q_0^2( x, x^') p( z)^4 p( y, z)^2]^2 p( x,  y, z) p( x', y, z) d x d x' d y d z = σ_2^2 𝒞_2(K) ,

        where Q_0 is as defined in Theorem <ref>, and
        
    𝒞_2(K) = ∫[∫ K(( u_1, v_1)) K(( u,  v) - ( u_1, v_1))d u_1d v_1]^2 d u d v.

    
  Now, define     
   
    Γ_n     := 1/n(n-1)∑_1≤ i  j≤ nϑ_1, 4^(2) ( U_i, U_j),

    where 
    
    ϑ_1, 4^(2) ( U_i, U_j) = {ϑ_1, 4^(2)( U_i, U_j)-_ U[ ϑ_1, 4^(2) ( U_i, U)]-_ U[ ϑ_1, 4^(2) ( U, U_j) ]+_ U, U'[ ϑ_1, 4^(2) ( U, U')]} ,

 with U,  U' i.i.d. with density p_ X,  Y,  Z. Note that Γ_n in (<ref>) is a degenerate U-statistics of degree 2. Hence, we can apply Theorem 1 from <cit.> to establish the convergence of Γ_n. 
 
   For this, note from (<ref>) and (<ref>) that 
    
    n^2 h_1^d_Y+d_Z[ Γ_n] = (1+ o(1)) h_1^d_Y+d_Z[ϑ_1, 4^(2) ( U_1, U_2)] = (1+ o(1)) h_1^d_Y+d_Z[ϑ_1, 4^(2)( U_1, U_2)^2]→σ_2^2  𝒞_2(K) .
  
Now, define 
    
    G_n( u, u') := [ϑ_1, 4^(2)( U_1,  u) ϑ_1, 4^(2)( U_1,  u')].

Using similar calculations we can show that h_1^(d_Y+d_Z)[G_n^2( U_1, U_2)] and h_1^3(d_Y+d_Z)[ϑ_1, 4^(2)( U_1,  U_2)^4] converges to a_1 and a_2, respectively, where a_1, a_2>0 are constants.  Therefore,
    
    [G_n^2( U_1, U_2)]/[[ϑ_1, 4^(2)( U_1, U_2)^2]]^2 = h_1^2(d_Y+d_Z)[G_n^2( U_1, U_2)]/[h_1^(d_Y+d_Z)[ϑ_1, 4^(2)( U_1, U_2)^2]]^2 = O(h_1^(d_Y+d_Z))→ 0,

    and 
    
    [ϑ_1, 4^(2)( U_1, U_2)^4]/n[[ϑ_1, 4^(2)( U_1, U_2)^2]]^2 = h_1^2(d_Y+d_Z)[ϑ_1, 4^(2)( U_1, U_2)^4]/n[h_1^(d_Y+d_Z)[ϑ_1, 4^(2)( U_1, U_2)^2]]^2    = h_1^3(d_Y+d_Z)[ϑ_1, 4^(2)( U_1, U_2)^4]/(nh_1^(d_Y+d_Z))[h_1^(d_Y+d_Z)[ϑ_1, 4^(2)( U_1, U_2)^2]]^2
        = O(1/nh_1^(d_Y+d_Z)) → 0,

    as n→∞. Hence, from Theorem 1 from <cit.> we have,
    
    nh_1^(d_Y+d_Z)/2Γ_n D→ N(0, σ_2^2 𝒞_2(K)) .
 
Also, by arguments as in (<ref>), 
    nh_1^(d_Y+d_Z)/21/n∑_i=1^n _ U[ϑ_1, 4^(2) (  U_i, U)] P→ 0 and nh_1^(d_Y+d_Z)/2_ U_1, U[ϑ_1, 4^(2) (  U_1, U)] → 0.
 Hence, the result in (<ref>) follows. 
 

With (<ref>), (<ref>), and (<ref>) proved, the proof of Lemma <ref> is complete. 
    


To complete the proof of Theorem <ref> (3), we need to show that 

    nh_1^(d_Y+d_Z)/2Δ^(c)P→ 0 ,
 
for c ≥ 3. Towards this, as in the proof Lemma <ref>, it can shown that, under H_0, [φ_n^(c)( U_1,…,  U_c)] = O(h_1^4), for c ≥ 1. 
 Hence, recalling (<ref>), 

    n h_1^(d_Y+d_Z)/2[Δ_n^(c)] =  O(n h_1^(d_Y+d_Z)/2 + 4) → 0.
  
  Moreover, by (<ref>) we have 

    [Δ_n^(c)] = 1/n c∑_a=1^c (-1)^c-ac a[φ_n^(a)( U_1,…,  U_a)] = O(1/n^c h_1^(c-1)(d_Y+d_Z)) .
 
Hence, 

    n^2 h_1^(d_Y+d_Z)[Δ_n^(c)] = O(1/n^c-2 h_1^(c-2)(d_Y+d_Z)) → 0 ,

for c ≥ 3. The result in (<ref>) follows from (<ref>) and (<ref>). Combining Lemma <ref>, Lemma <ref>, and (<ref>) completes the proof of Theorem <ref> (3).   




§ PROOFS FROM SECTION <REF>

 


In this section we present the proofs of the results from Section <ref>. The proof of Proposition <ref> 
is given in Section <ref> and the proof of Proposition <ref> is given in Section <ref>. In Section <ref> we prove Theorem <ref>.  





 §.§  Proof of Proposition <ref> 



    Note that by construction the observed dataset 𝒟_0 and the resampled datasets 𝒟_1',𝒟_2',…, 𝒟_M' are exchangeable under H_0. Therefore, ζ̂_n(𝒟_0) and {ζ̂_n(𝒟_i')}_1 ≤ i ≤ M are exchangeable under H_0. Using this exchangeability property we have the following results: 
    
        
  (i) Note that (M+1)p_CRT = ∑_i=1^M 1{ζ̂_n(𝒟_0)> ζ̂_n(𝒟_i')}+1 is the rank of ζ̂_n(𝒟_0) among the observations {ζ̂_n(𝒟_0),ζ̂_n(𝒟_1'),…, ζ̂_n(𝒟_M')}. Due to exchangeability (M+1)p_CRT is uniformly distributed over {1,2,…, M+1}. Therefore,
        
    (p_CRT≤α)     = ((M+1)p_CRT≤ (M+1)α) = ⌊ (M+1)α⌋/M+1.
 
        Hence, our test controls the Type I error uniformly under the CRT framework. 

        
  (ii)  Recall that, by construction, X_i'  Y_i| Z_i, for all 1 ≤ i ≤ n. Hence, by Theorem <ref>, ζ̂_n(𝒟_1') P→ 0, as n →∞. Theorem <ref> also tells us that under H_1, ζ̂_n(𝒟_0) P→ζ( X,  Y |  Z)>0. Therefore, 1{ζ̂_n(𝒟_i')>ζ̂_n(𝒟_0)}→ 0, as n →∞, for each 1 ≤ i ≤ M. Hence, for any M ≥ 1 fixed, 
           
    p_CRT = 1+∑_i=1^M 1{ζ̂_n(𝒟_0)>ζ̂_n(𝒟_i')}/1+MP→1/1+M,

        as n →∞. Therefore, if α>1/(M+1),  (p_CRT≤α) → 1 as n →∞. 
        
    




 §.§  Proof of Proposition <ref> 
 



    Here also under H_0, the observed data 𝒟_0 and 𝒟^π_1,𝒟^π_2,…, 𝒟^π_M are exchangeable (see <cit.>). Hence, ζ̂_n(𝒟_0) and {ζ̂_n(𝒟^π_j)}_1 ≤ j ≤ M are exchangeable. Now, using similar arguments as in the proof of Proposition <ref> we have our desired result.    





 §.§  Proof of Theorem <ref> 



    Let us define 𝒮 := {( Y_1, Z_1),( Y_2, Z_2),…, ( Y_n, Z_n)}, 𝒮_ Z := { Z_1, Z_2,…, Z_n}, 𝒮_ X := { X_1, X_2,…, X_n}, and 𝒮_ X' := { X_1', X_2',…, X_n'}. 
    
     Suppose the assumptions of Theorem <ref> hold. Then under H_0, 
        
    TV(ℒ(ζ̂_n(𝒟̂_1)|𝒮),ℒ(ζ̂_n(𝒟_0)|𝒮)) P→ 0 .

    
    
    
    Then by the definition of the total variation distance (denoted as TV) for any measurable function f we have,
    
    TV(ℒ(f( X_1,…, X_n)),ℒ(f( X_1',…,  X_n')))
        = sup_A∈ℬ()|(f( X_1,…, X_n) ∈ A)-(f( X_1',…,  X_n')∈ A)|
        = sup_A∈ℬ()|(( X_1,…, X_n) ∈ f^-1 (A))-(( X_1',…,  X_n')∈ f^-1(A))|
       ≤sup_A∈ℬ(^n)|(( X_1,…, X_n) ∈ A)-(( X_1',…,  X_n')∈ A)|
        = TV(( X_1,…,  X_n),( X_n',…,  X_n' )),

    where ℒ( V), for any random variable V, denotes the distribution of V.   
    Using (<ref>) we have
    
    TV(ℒ(ζ̂_n(𝒟̂_1)|𝒮),ℒ(ζ̂_n(𝒟_0)|𝒮))≤TV(ℒ(𝒮_ X'|𝒮),ℒ(𝒮_ X|𝒮)).

    However, under H_0:  X Y |  Z we have, ℒ(𝒮_ X|𝒮) = ℒ(𝒮_ X|𝒮_ Z) with density ∏_i=1^n p( x_i|  Z_i). Also, by the construction of the resampled data ℒ(𝒮_ X'|𝒮) = ℒ(𝒮_ X'|𝒮_ Z) with density [∏_i=1^n p̂( x_i|  Z_i)|  Z_1, …, Z_n] where p̂( x| z) is as in (<ref>). Therefore, under H_0, 
    
    TV(ℒ(ζ̂_n(𝒟̂_1)|𝒮),ℒ(ζ̂_n(𝒟_0)|𝒮)) 
       ≤TV(ℒ(𝒮_ X'|𝒮),ℒ(𝒮_ X|𝒮))
        = TV(ℒ(𝒮_ X'|𝒮_ Z),ℒ(𝒮_ X|𝒮_ Z))
        = 1/2∫|[∏_i=1^n p̂( x_i|  Z_i)|  Z_1, Z_2,…, Z_n] - ∏_i=1^n p( x_i|  Z_i)| ∏_i=1^n d x_i.

    Note that conditioned on Z_1, Z_2,…, Z_n we have,
    
    [∏_i=1^n p̂( x_i|  Z_i) |  Z_1,…,  Z_n] = ∑_ i∈ [n]^n∏_j=1^n γ_i_j(  Z_j )  [ ∏_j=1^n 1/h_0^d_Xϕ( x_j,  X_i_j, h_0  I_d_X)|  Z_1,…,  Z_n] .

    Observe that 
    [ ∏_j=1^n 1/h_0^d_Xϕ( x_j,  X_i_j, h_0  I_d_X)|  Z_1, Z_2,…, Z_n]
 is itself a valid density function, for any i = (i_1,…, i_n) ∈ [n]^n. 
   Using ∑_ i∈ [n]^n∏_j=1^n γ_i_j(  Z_j ) = ∏_j=1^n ∑_i=1^n γ_i( Z_j) = 1 gives, 
    
    ∫|[∏_i=1^n p̂( x_i|  Z_i)|  Z_1,…, Z_n ] - ∏_i=1^n p( x_i|  Z_i) | ∏_i=1^n d x_i
        = ∫|∑_ i ∈ [n]^n ∏_j=1^n γ_i_j(  Z_j )  [ ∏_j=1^n 1/h_0^d_Xϕ( x_j, X_i_j, h_0  I_d_X)|  Z_1,…,  Z_n]- ∏_i=1^n p( x_i|  Z_i) | ∏_i=1^n d x_i
       ≤∏_i=1^n γ_i( Z_i) ∫| [ ∏_i=1^n 1/h_0^d_Xϕ( x_i,  X_i, h_0  I_d_X)|  Z_1,…, Z_n]-∏_i=1^n p( x_i|  Z_i) |∏_i=1^n d x_i + U_n,

     where 
    
    U_n = ∑_π: [n] → [n] 
    π≠id∏_i=1^n γ_π(i)(  Z_i ) ∫| [ ∏_i=1^n 1/h_0^d_Xϕ( x_i,  X_π(i), h_0  I_d_X)|  Z_1,…, Z_n]-∏_i=1^n p( x_i|  Z_i)|∏_i=1^n d x_i ,
 
     with id: [n] → [n] denoting the identity map, that is, id(i) = i, for 1 ≤ i ≤ n.   

      For any π: [n] → [n] with π≠id note that,
     
    ∫| [ ∏_i=1^n 1/h_0^d_Xϕ( x_i,  X_π(i), h_0  I_d_X)|  Z_1,…, Z_n]-∏_i=1^n p( x_i|  Z_i) | ∏_i=1^n d x_i 
        = 2TV(ℒ( V_1,…, V_n), ℒ( X_1,…,  X_n)),

     for some random vector ( V_1,…,  V_n) having joint density function [ ∏_i=1^n 1/h_0^d_Xϕ( x_i,  X_π(i), h_0  I_d_X)|  Z_1,…, Z_n]. Bounding the TV distance by 1, we then have     
     
    U_n = O_P (  ∑_π: [n] → [n] 
    π≠id∏_i=1^n γ_π(i)(  Z_i )  ) = o_P(1) ,

         as n →∞, since ∏_s=1^n γ_s( Z_s) P→ 1.  
     Also, note that
     
    [ ∏_i=1^n 1/h_0^d_Xϕ( x_i,  X_i, h_0  I_d_X)|  Z_1,…, Z_n]     = ∫∏_i=1^n 1/h_0^d_Xϕ( x_i- w_i/h_0) ∏_i=1^n p( w_i|  Z_i) ∏_i=1^n d w_i
        = ∫∏_i=1^n ϕ( a_i) ∏_i=1^n p( x_i + h_0 a_i|  Z_i) ∏_i=1^n d a_i 
       = ∏_i=1^n ( ∫ϕ( a) p( x_i + h_0 a|  Z_i)  d a ) .

     Hence, by a telescoping argument, 
     
    A_n :=    ∫|[ ∏_i=1^n 1/h_0^d_Xϕ( x_i,  X_i, h_0 I_d_X)|  Z_1,…, Z_n] - ∏_i=1^n p( x_i|  Z_i)| ∏_i=1^n d x_i 
        = ∫| ∏_i=1^n ( ∫ϕ( a) p( x_i + h_0 a|  Z_i)  d a ) - ∏_i=1^n p( x_i|  Z_i)| ∏_i=1^n d x_i  
       ≤∑_i=1^n |∫ϕ( a) p( x_i + h_0 a |  Z_i) d a - p( x_i |  Z_i)| d x_i .
 
            Taking expectation with respect to Z_1,…, Z_n gives,          
            
    [A_n]      = n  [ |∫ϕ( a) p( x + h_0 a |  Z_1) d a - p( x |  Z_1)| d x ] 
        = n ∫|∫ϕ( a) p_ X, Z( x + h_0 a, z)/p_ Z( z)d a - p_ X, Z( x,  z)/p_ Z( z )| p_ Z( z)d x d z 
        = n ∫|∫ϕ( a) p_ X, Z( x + h_0 a, z) d a - p_ X, Z( x,  z)| d x d z 
        = O(nh_0^2) ,
 
            since p_ X, Z is nice in X. This implies, A_n P→ 0.  
     Now combining (<ref>),(<ref>),(<ref>), and (<ref>) the result in Lemma <ref> follows. 
         
    
    
     
     Now we prove the desired properties of the Type I error and power of the local wild bootstrap test ϕ_LWB = 1{ p_LWB<α}. 

    
        
  (a) Before proving the result in Theorem <ref> (a) let us first note that for two random variables T and W, 
        
    (T>x) = ∫_{ t>x}dℒ(T)(t)     = ∫_{t>x}d(ℒ(W) + ℒ(T)-ℒ(W))(t)
        = ∫_{t>x}dℒ(W)(t) + ∫_{t>x}d(ℒ(T)-ℒ(W))(t)
       ≤(W>x) + sup_A | ℒ(T)(A)-ℒ(W)(A)|
        = (W>x) + TV(ℒ(T), ℒ(W)),

        for any x∈. Using (<ref>) for ζ_n(𝒟̂_1) and ζ_n(𝒟_0) conditioning on 𝒮 we have,
        
    (ζ̂_n(𝒟_0)>x|𝒮)    ≤(ζ̂_n(𝒟̂_1)>x |𝒮) + TV(ℒ(ζ̂_n(𝒟̂_1)|𝒮),ℒ(ζ̂_n(𝒟_0)|𝒮)) .

       Taking expectation on both sides, applying Lemma <ref> and the Dominated Convergence Theorem gives       
       
    lim sup_n→∞(ζ̂_n(𝒟_0)>x)    ≤lim sup_n→∞(ζ̂_n(𝒟̂_1)>x) ,
 
   for any x∈. Now, define c_n^*(α) as the upper α-th quantile of ζ̂_n(𝒟̂_1) conditioned on 𝒟_0, that is, (ζ̂_n(𝒟̂_1) >c_n^*(α)|𝒟_0) ≤α. Also, define p^*:=(ζ̂_n(𝒟̂_1)≥ζ̂_n(𝒟_0) |𝒟_0). Let
        
    F(t) = {ζ̂_n(𝒟̂_1)≤ t|𝒟_0}   and    F_M(t) = 1/M{∑_i=1^M 1{ζ̂_n(𝒟̂_i)≤ t }}.

        Note that F and F_M are distribution functions conditioned on the observed data 𝒟_0. Then,
        
    |p^*-p_LWB|     = |(ζ̂_n(𝒟̂_1)≥ζ̂_n(𝒟_0)|𝒟_0) -1/M+1{∑_i=1^M 1{ζ̂_n(𝒟̂_i)≥ζ̂_n(𝒟_0) } + 1}|
        = |(ζ̂_n(𝒟̂)< ζ̂_n(𝒟_0)|𝒟_0) - 1/M+1{∑_i=1^M 1{ζ̂_n(𝒟̂_i)< ζ̂_n(𝒟_0)}}|
         = |F(ζ̂_n(𝒟_0)-)-M/M+1F_M(ζ̂_n(𝒟_0)-)|
       ≤|F(ζ̂_n(𝒟_0)-)-F_M(ζ̂_n(𝒟_0)-)| + |F_M(ζ̂_n(𝒟_0)-)/M+1|
       ≤sup_t∈|F(t)-F_M(t)|+1/M+1.
        
        Using the Dvoretzky-Keifer-Wolfwitz inequality now gives, 
        
    (|p_LWB-p^*|>ε|𝒟_0) ≤ 2exp{-2M(ε-1/M+1)^2 },

        for any ε>1/(M+1). Now, fix a large M and take any ε>1/(M+1). Then,
        
    (p_LWB≤α|𝒟_0)     = (p_LWB≤α, |p_LWB-p^*|>ε|𝒟_0) + (p_LWB≤α, |p_LWB-p^*|≤ε|𝒟_0) 
       ≤(|p_LWB-p^*|>ε|𝒟_0) +  1{p^*≤α+ε}
       ≤ 2exp{-2M(ε-1/M+1)^2} +  1{ζ_n(𝒟_0)>c_n^*(α+ε)} .

        Taking expectations on both sides gives, 
        
    (p_LWB≤α) ≤ 2exp{-2M(ε-1/M+1)^2} + (ζ_n(𝒟_0)>c_n^*(α+ε)).
  
        Now, taking limit as n→∞ and M→∞ we get,
        
    lim sup_n,M→∞(p_LWB≤α)    ≤lim sup_n →∞(ζ̂_n(𝒟_0)>c_n^*(α+ε) ) 
       ≤lim sup_n →∞(ζ̂_n(𝒟̂_1) >c_n^*(α+ε)) *(using (<ref>))
        = lim sup_n →∞[(ζ̂_n(𝒟̂_1) >c_n^*(α+ε)|𝒟_0) ]
       ≤lim sup_n →∞[α + ε]*(by the definition of c_n^*(α+ε))
         = α +  ε.

        Hence, since ε is arbitrary, under H_0, lim sup_n,M→∞_H_0(p_LWB≤α ) ≤α. This completes the proof of the first part of Theorem <ref>.   
        
        
  (b) Under H_1: X Y |  Z, using similar arguments as in the proof of  Lemma <ref> we can show that 
      
    TV(ℒ(ζ̂_n(𝒟̂_1)|𝒮),ℒ(ζ̂_n(𝒟_1')|𝒮))P→ 0,

     where 𝒟_1' is the resampled data obtained in the conditional randomization test. Therefore, even under H_1, ζ̂_n(𝒟̂_1) P→ 0, as n→∞, by Theorem <ref>. Moreover, ζ̂_n(𝒟_0) converges to ζ( X, Y |  Z) which is positive under H_1. Therefore, under H_1, p_LWB converges in probability to 1/(M+1), as n →∞. Hence, if α>1/(M+1) or M>(1-α)/α, lim sup_n→∞_H_1(p_LWB≤α ) =1, that is, the local wild bootstrap test is consistent. 
    

    



 §.§ Verifying the Weight Condition
 


Let γ_s( z) = κ_s( z) as defined in (<ref>). In the following lemma shows how to choose the bandwidth h_2' such that the condition ∏_s=1^n γ_s( Z_s)P→ 1 holds. 

 
Let γ_s( z) = κ_s( z) as defined in (<ref>). Also, suppose that the Assumptions <ref>, <ref>, <ref> hold. If h_2^'  = o(n^-2/d_Z) we have,
    
    ∏_s=1^n γ_s( Z_s)P→ 1 ,

    as n→∞. 
    


 First, recalling (<ref>), note that 
{γ_1( Z_1)<1 } implies that there exists Z_j, with j ≠1, such that K( Z_1- Z_j/h_2^')>0. Now, since the kernel K has compact support (by Assumption <ref>), this implies Z_j- Z_1< Ch_2^' for some constant  C>0. Therefore, by the union bound,
    
    (γ_1( Z_1)<1)    ≤∑_j=2^n ( Z_j- Z_1< Ch_2^')
        = (n-1) ( Z_2- Z_1< Ch_2^')  *(since Z_2, …,  Z_n are i.i.d.)
        = (n-1) ∫_ z_2- z_1< Ch_2^' p_ Z( z_1) p_ Z( z_2) d z_1 d z_2
       ≲  (n-1) ∫Vol(B( z_1, Ch_2^')) p_ Z( z_1) d z_1 *(since p_ Z is bounded by Assumption <ref>)
       ≲ (n-1)(h_2^')^d_Z.

    Here, Vol(B( z_1, Ch_2^')) denotes the volume of the ball of radius Ch_2^' which is centered at z_1 and the constants in the ≲ bounds depend on d_Z. Now, fix any arbitrary ϵ>0 and note that,
    
    (∏_s=1^n γ_s( Z_s)<1-ϵ)    ≤∑_s=1^n (γ_s( Z_s)<(1-ϵ)^1/n)
        = n (γ_1( Z_1)<(1-ϵ)^1/n) 
    
               ≤ n (γ_1( Z_1)<1) 
       ≲  n(n-1)(h_2^')^d_Z
       → 0 ,
 
    since h_2^' = o(n^-2/d_Z). This completes the proof. 






§ PROOFS FROM SECTION <REF> 
 


This section is organized as follows: In Appendix <ref> we collect the proofs of some technical lemmas which will be used in the analysis of the conditional empirical process. The proof of Proposition <ref> is given in Section <ref>. Proposition <ref> is proved in Section <ref>. 




 §.§ Auxiliary Lemmas






We begin with the following result: 

 Suppose ( X_1,  Z_1), ( X_2,  Z_2), …, ( X_r,  Z_r) are i.i.d. samples from p_ X,  Z. Further, suppose the following conditions hold:   


  * p_ X, Z satisfies assumption <ref>; 


  * the kernel K satisfies assumption <ref>; 


  * h_2 is a bandwidth satisfying the conditions in assumption <ref>. 


Fix an integer r ≥ 1. Then for a measurable, symmetric function ψ: (ℝ^d_X)^r →ℝ such that 
    𝔼[ψ^2(  X_1,  X_2, …,  X_r ) | { Z_s =  z}_1 ≤ s ≤ r ] < ∞,
 we have 

    lim_h_2→ 01/h_2^ r d_Z 𝔼[  ∏_s=1^r  K ( z -  Z_s /h_2) ψ( X_1, …,  X_r)  ] = p_ Z ( z)^r [ψ( X_1, …,  X_r)| { Z_s =  z}_1 ≤ s ≤ r].

Further, if

    sup_ z∈^d_Z𝔼[ψ^2(  X_1,  X_2, …,  X_r ) | { Z_s =  z}_1 ≤ s ≤ r ] < ∞,
 holds, we have,   

    lim_h_2→ 01/h_2^ r d_Z ∫𝔼[  ∏_s=1^r  K( z -  Z_s /h_2) ψ( X_1, …,  X_r)  ] p_ Z( z)d z 
        = ∫ p_ Z ( z)^r+1[ψ( X_1, …,  X_r)| { Z_s =  z}_1 ≤ s ≤ r] d z.
 

 

 
Denote X = ( X_1, …,  X_r) and x = ( x_1, …,  x_r). Then observe that  

    1/h_2^ r d_Z 𝔼[ ∏_s=1^r K( z -  Z_s /h_2) ψ(  X ) ]     = 1/h_2^ r d_Z ∫∫∏_s=1^r K( z - z̃_s /h_2) ψ( x) ∏_s=1^r p_ X,  Z ( x_s, z̃_s) d x_s dz̃_s 
        = ∫∫∏_s=1^r  K(  z_s ) ψ(  x) ∏_s=1^r p_ X,  Z (  x_s,  z + h_2 z_s) d x_s d z_s ,
 
with the change of variable z̃_s =  z+ h_2 z_s, for 1 ≤ s ≤ r. Denote 
    p^(r)(  x,  z )  := ∏_s=1^r p_ X,  Z ( x_s,  z)   and  p^(r)(   x,  z + h  z  )   := ∏_s=1^r p_ X,  Z ( x_s,  z + h_2 z_s) ,
 
where z = ( z_1, …,  z_r). Also, define Q( z) := ∏_s=1^r K( z_s) and 

    R( x_s, z; z_s, h_2) = 1/h_2^2{p_ X, Z( x_s, z + h_2 z_s)/p_ X, Z( x_s, z) -1 - h_2  z_s  ∂/∂ zp_ X, Z( x_s, z)/p_ X, Z( x_s, z)}.

Then, using the identity, 

    p_ X, Z( x_s,  z + h_2 z_s) = p_ X, Z( x_s,  z ) + h_2 z_s  ∂/∂ zp_ X, Z( x_s, z) + h_2^2 p_ X, Z( x_s, z) R( x_s, z;  z_s, h_2) ,
 
and noting that 
∫∫ K ( z_s) 
 z_s  ∂/∂ zp_ X, Z( x_s, z) d z_s = 0, 
since the kernel is symmetric about zero by Assumption <ref>, we have  

    A_n^ z    := ∫∫ Q ( z) ψ( x) p^(r)(   x,  z + h  z  )   ∏_s=1^r d x_s d z_s - ∫∫ Q( z) ψ(  x) p^(r)(  x,  z )  ∏_s=1^r d x_s d z_s  
        = ∑_k=1^r h_2^2 k∑_S⊆ [r]:
     |S| = k A_n^ z(S),
 
        where 
        
    |A_n^ z(S)|     := |∫∫ Q ( z) ψ( x) ∏_s∈ S^c p_ X,  Z ( x_s,  z) ∏_s∈ S( R( x_s, z; z_s, h_2) p_ X, Z( x_s, z)) ∏_s=1^r d x_s d z_s| 
       ≤∫ Q ( z) ( ∫|ψ( x)| ∏_s∈ S^c p_ X,  Z ( x_s,  z) ∏_s∈ S( |R( x_s, z; z_s, h_2)| p_ X, Z( x_s, z)) ∏_s=1^r d x_s ) ∏_s=1^r  d z_s 
       ≤∫ Q ( z)  ( ∫ψ( x)^2 p^(r)(  x,  z ) ∏_s=1^r d x_s )^1/2( ∫∏_s∈ S R( x_s, z; z_s, h_2)^2 p^(r)(  x,  z )  ∏_s=1^r d x_s )^1/2∏_s=1^r d z_s .
 
        Now, note that 
        
    lim sup_n→∞∫∏_s∈ S R( x_s, z; z_s, h_2)^2 p^(r)(  x,  z )  ∏_s=1^r d x_s
         = lim sup_n→∞∫∏_s∈ S( z_s^2 R( x_s, z; z_s/ z_s, h_2 z_s)^2 p^(r)(  x,  z) )  ∏_s=1^r d x_s*(by (<ref>))
       ≤(p( z))^r-|S|∏_s∈ S z_s^4  ( sup_ u: u=1lim sup_n→∞∫ R( x, z; u, h_2 z_s)^2 p( x,  z )  d x)^|S|≲∏_s∈ S z_s^4.

        Also, by Assumption <ref>, 
    ∫ Q( z) ∏_s∈ S z_s^2 ∏_s=1^r d z_s = (∏_s∈ S^c∫ K(  z_s  ) d z_s)( ∏_s∈ S∫ z_s^2 K(  z_s  ) d z_s) = (d_Zτ)^|S|.
 
                Hence, by Assumption <ref>, the finiteness of [ψ^2( X) | { Z_s =  z}_1 ≤ s ≤ r ]<∞, we have, 

    |A_n^ z(S)|≲ p_ Z( z)^r/2∫ Q ( z) ∏_s∈ S z_s^2 ∏_s=1^r  d z_s ≲ 1.
 
Hence, from (<ref>), we get A_n=O(h_2^2). Therefore, from (<ref>),  

    1/h_2^ r d_Z 𝔼[ ∏_s=1^r K( z -  Z_s /h_2) ψ(  X ) ]     = ∫∫∏_s=1^r  K(  z_s ) ψ(  x) ∏_s=1^r p_ X,  Z (  x_s,  z) d x_s d z_s + O(h_2^2) 
        =  ∫ψ(  x) ∏_s=1^r p_ X,  Z (  x_s,  z ) d x_s  + O(h_2^2)  
        = p_ Z ( z)^r [ψ( X_1, …,  X_r)| { Z_s =  z}_1 ≤ s ≤ r] + O(h_2^2) .
 
This completes the proof of (<ref>).  



To show (<ref>), note from the arguments in (<ref>) and (<ref>) and the assumption (<ref>) we get ∫ A_n^ z p_ Z( z) d z = O(h_2^2). The result in (<ref>) then follows by taking expectation with respect to Z on both sides of (<ref>). 



    Note that for Lemma <ref> to hold it is sufficient that 𝒞_K:= ∫ K( u)d u <∞. In this case, from the proof of Lemma <ref> it follows that
       
    lim_h_2→ 01/h_2^ r d_Z 𝔼[  ∏_s=1^r  K ( z -  Z_s /h_2) ψ( X_1, …,  X_r)  ] = 𝒞_K^r p_ Z ( z)^r [ψ( X_1, …,  X_r) | { Z_s =  z}_1 ≤ s ≤ r].
 


Using the above lemma we can derive the limit of the expectation of a homogeneous sum of r-fold kernel products. 




  
Suppose the assumptions the Lemma <ref> hold. Then 

    lim_n →∞1/(nh_2^ d_Z )^r 𝔼[ ∑_ i∈ [n]^r∏_s=1^r  K( z -  Z_i_s/h_2) ψ( X_ i)  ] = p_ Z ( z)^r [ψ( X_1, …,  X_r) | { Z_s =  z}_1 ≤ s ≤ r],

where i = (i_1,i_2,…, i_r) and X_ i = ( X_i_1, X_i_2,…,  X_i_r).  
Moreover,  

    lim_n →∞1/(nh_2^ d_Z )^r ∫𝔼[ ∑_i_1,…, i_r =1^n  ∏_s=1^r  K( z -  Z_i_s/h_2) ψ( X_i_1, …,  X_i_r)  ] p_ Z( z) d z 
        = ∫ p_ Z ( z)^r+1[ψ( X_1, …,  X_r) | { Z_s =  z}_1 ≤ s ≤ r] d z.
 
 

 Denote by 
i the distinct elements of i ∈ [n]^r. Now,
    
    1/(nh_2^ d_Z )^r 𝔼[ ∑_ i∈ [n]^r∏_s=1^r  K( z -  Z_i_s/h_2) ψ( X_ i)  ] 
        = 1/(nh_2^ d_Z )^r 𝔼[ ∑_k=1^r ∑_ i ∈ [n]^r: | i| =  k∏_s=1^r  K( z -  Z_i_s/h_2) ψ( X_ i)  ]
        = 1/(nh_2^ d_Z )^r ∑_k=1^r ∑_ i ∈ [n]^r: | i| = k𝔼[ ∏_s=1^r  K( z -  Z_i_s/h_2) ψ( X_ i)  ]
        = 1/(nh_2^ d_Z )^r ∑_k=1^r ∑_ i ∈ [n]^r: | i| = k𝔼[ ∏_ j ∈ iK_j ( z -  Z_j/h_2) ψ( X_ℓ_ i)  ] ,
   
       where for a given i∈ [n]^r, 
       
    K_j( z -  Z_j/h_2) := ∏_s: i_s = j K( z -  Z_j/h_2)
 
       and ℓ_ i=⋃_j=1^n{s: i_s = j}∈ [n]^r is the vector obtained after sorting i. Now, fix 1 ≤ k ≤ r and consider, 
      
    T_k, n    := ∑_ i ∈ [n]^r: | i| = k𝔼[ ∏_ j ∈ iK_j ( z -  Z_j/h_2) ψ( X_ℓ_ i)  ]    
        = h_2^k d_Z∑_ i ∈ [n]^r: | i| = k∫∏_j=1^kK_j (  u_j ) ψ(  x_ℓ_ i) ∏_j=1^k p_ X, Z( x_j,  z + h_2 u_j) ∏_j=1^kd x_j d u_j 
       * (after the substitution z_j =  z + h_2 u_j, for 1 ≤ j ≤ k) 
         = O(n^k h_2^k d_Z) ,

 by arguments as in Lemma <ref>. This implies, 
   
    1/(nh_2^ d_Z )^r 𝔼[ ∑_ i∈ [n]^r∏_s=1^r  K( z -  Z_i_s/h_2) ψ( X_ i)  ]  
        =  T_r, n/(nh_2^ d_Z )^r  +  O(1/nh_2^ d_Z ) * (by (<ref>) and (<ref>)) 
        = 1/(nh_2^ d_Z )^r   (n)_r h_2^r d_Z∫∏_j=1^r  K(  u_j ) ψ( x_1, …,  x_r) ∏_j=1^r p_ X, Z( x_j, z + h_2 u_j) ∏_j=1^r d x_j d u_j + O(1/nh_2^ d_Z ) * (where (n)_r := n(n-1) ⋯ (n-r+1)) 
       →∫∏_j=1^r  K( u_j) ψ( x_1, …,  x_r) ∏_j=1^r p_ X, Z( x_j,  z) ∏_j=1^r d x_j d u_j
           
           *(by (<ref>) and (<ref>))
        = p_ Z( z)^r [ψ( X_1, …,  X_r)| { Z_s =  z }_1 ≤ s ≤ r ] .

    This proves (<ref>). 


The result in (<ref>) can be proved similarly, by repeating the previous arguments and using (<ref>). 





Note that the limits obtained in Lemma <ref> is zero, when the function ψ:(^d_X)^r→ is completely degenerate, that is, 

    [ψ( x_1,…, x_r-1, X_r)|  Z_r =  z] = 0,
 
almost surely for x_1,…, x_r-1∈^d_X and z ∈^d_Z. In this case one has to normalize the r-fold kernel product differently to obtain a non-trivial limit. 




 
Suppose the assumptions of Lemma <ref> hold. Further, assume that the function ψ:(^d_X)^r→ is completely degenerate as defined in (<ref>). Then the following hold: 
    
    lim_n →∞1/(nh_2^ d_Z )^r/2𝔼[ ∑_ i ∈ [n]^r∏_s=1^r  K( z -  Z_i_s/h_2) ψ( X_ i)  ]
        = 
    {[ (∫ K^2( u)d u)^r/2(⌊r/2⌋) ! ∫ψ( x_1, x_1, …,  x_⌊ r/2⌋, x_⌊ r/2⌋) ∏_s=1^⌊ r/2⌋ p_ X, Z( x_s,  z) ∏_s=1^⌊ r/2 ⌋d x_s,                                                                                                      if  r  is even,;                                                                                                                    0                                                                                                       if  r  is odd. ].

        



As in the proof of Lemma <ref>, denote by 
i the distinct elements of i ∈ [n]^r. Then from (<ref>) we have, 
    
    𝔼[ ∑_ i∈ [n]^r∏_s=1^r  K( z -  Z_i_s/h_2) ψ( X_ i)  ]      =   ∑_k=1^r T_k, n,
  
   with T_k, n as defined in (<ref>). 
For any k < ⌊ r/2 ⌋, by (<ref>), 
    
    lim_n→∞T_k, n/(nh_2^ d_Z )^r/2≲lim_n→∞(nh_2^ d_Z )^k/(nh_2^ d_Z )^r/2 = 0.

Therefore, we only need to consider the case k≥⌊ r/2 ⌋. Towards this, suppose i ∈ [n]^r such that | i| = k >⌊ r/2 ⌋. Then by the pigeonhole principle, there exists an index  which occurs exactly once in i. Let ν_1( i) be the number of indices in i that repeats exactly once and ν_2( i) are those that repeats multiple times in i. Clearly, k = ν_1( i) + ν_2( i), for i ∈ [n]^r such that | i| = k. Now, recalling (<ref>), let  

    T_k, n = ∑_ i ∈ [n]^r: | i| = k𝔼[ ∏_ j ∈ iK_j ( z -  Z_j/h_2) ψ( X_ℓ_ i)  ] = ∑_0 ≤ a , b ≤ k 
      a + b = k T_a, b, n,
 
   where 
   
    T_a, b, n := ∑_ i ∈ [n]^r 
    ν_1( i) = a, ν_2( i) = b  𝔼[ ∏_ j ∈ iK_j ( z -  Z_j/h_2) ψ( X_ℓ_ i)  ] .
 Then, by the degeneracy of ψ all terms in the expansion (<ref>) with |S| ≤ a-1 are zero. Hence, T_a, b, n = O((nh_2^ d_Z )^a + b h_2^2 a). Also, note that r ≥ a+2b, hence a +b -r/2 ≤ a/2. 
   
  This implies, for k >⌊ r/2 ⌋, 
   
    T_a, b, n/(nh_2^ d_Z  )^r/2 = O((nh_2^ d_Z )^a +b -r/2h_2^2 a) = O((nh_2^d_Z+4)^a/2) → 0  ,
 
    since a ≥ 1 when k >⌊ r/2 ⌋ and nh_2^d_Z+4→ 0 (by Assumption <ref>). Hence, for k > ⌊ r/2 ⌋, by (<ref>), 
    
    lim_n→∞T_k, n/(nh_2^ d_Z )^r/2  = 0.
 

   
    Now, let us consider the case k=⌊ r/2 ⌋. If r is odd, then by arguments as in (<ref>), 
    
    lim_n→∞T_⌊ r/2 ⌋, n/(nh_2^ d_Z )^r/2 = lim_n →∞ O((nh_2^d_Z)^⌊ r/2⌋-r/2) = 0.

    Therefore, suppose r is even and k=⌊ r/2 ⌋. In this case, if a ≥ 1, then from (<ref>) we get, 
            
    lim_n →∞T_a, b, n/(nh_2^ d_Z )^r/2  =  lim_n →∞ O((nh_2^d_Z + 4)^a/2) = 0 ,

     Hence, recalling (<ref>)
     
    T_⌊ r/2 ⌋, n/(nh_2^ d_Z )^r/2 = T_0, ⌊ r/2 ⌋, n/(nh_2^ d_Z )^r/2 + o(1).

     Also, by the change of variable z_s =  z + h_2  u_s, 
            
    T_0, ⌊ r/2 ⌋, n/(nh_2^ d_Z )^r/2
        = n⌊ r/2⌋/n^⌊ r/2⌋( ⌊r/2⌋)! ∫∏_s=1^r/2 K ( u_s )^2 ψ( x_1, x_1, …,  x_⌊ r/2⌋, x_⌊ r/2⌋) ∏_s = 1^⌊ r/2 ⌋ p_ X, Z( x_s,  z + h_2 u_s) ∏_s = 1^⌊ r/2 ⌋d x_s d u_s 
       →( ⌊r/2⌋)! ( ∫ K ( u )^2 d u )^⌊ r/2 ⌋∫∏_s = 1^r/2ψ( x_1, x_1, …,  x_⌊ r/2⌋, x_⌊ r/2⌋) ∏_s = 1^⌊ r/2 ⌋ p_ X, Z( x_s,  z ) ∏_s = 1^⌊ r/2 ⌋d x_s  .
  
    Combining the above with (<ref>) and (<ref>), the proof of Lemma <ref> is completed. 
     


 
Note that if ψ is not symmetric, then Lemma <ref> holds with ψ' instead of ψ, where ψ' is defined as
    
    ψ'( x_1,…,  x_r) = 1/r!∑_τ∈ S_rψ( x_τ ) ,

        where S_r is the set of permutations of {1, 2, …, r} and x_τ = (  x_τ(1),  x_τ(2), … x_τ(r) ), for τ∈ S_r. 



To derive the asymptotic properties of the conditional 2-sample V-statistic (<ref>), we need the 2-sample analogous of the previous lemmas. We begin with the 2-sample version of Lemma <ref>. 
 

 Fix r_1, r_2≥ 1 and suppose ( X,  Y,  Z), ( X_1,  Y_1,  Z_1), …, ( X_r_1+r_2,  Y_r_1+r_2,  Z_r_1+r_2) are i.i.d. samples from p_ X, Y,  Z. Further, suppose the following conditions hold:   


  * p_ X,  Y,  Z satisfies assumption <ref>; 


  * the kernel K satisfies assumption <ref>; 


  * h_1 and h_2 are bandwidths satisfying the conditions in assumption <ref>. 

 
Consider a measurable function ψ: (ℝ^d_X)^r_1× (ℝ^d_X)^r_2→ℝ which is symmetric in its first r_1 and last r_2 coordinates and 

    sup_ y ∈ℝ^d_Y,  z ∈ℝ^d_Z𝔼[ψ^2(  X_[r_1+r_2]) |{ Y_s =  y }_ 1  ≤ s ≤ r_1, { Z_s =  z }_1 ≤ s ≤ r_1+r_2  ] < ∞,

where X_[r_1+r_2] = (  X_1, …,  X_r_1+r_2). Then the following holds, as h_1, h_2→ 0, 
   
    lim_h_1, h_2 → 01/h_1^r_1 (d_Y+d_Z)  h_2^ r_2 d_Z 𝔼[  ∏_s=1^r_1  K( ( y, z)  - ( Y_s, Z_s)  /h_1) ∏_s=r_1+1^r_1+r_2  K( z -  Z_s /h_2)    ψ(  X_[r_1+r_2] )  ] 
        =  p_ Y, Z ( y, z)^r_1 p_ Z ( z)^r_2[ψ(  X_[r_1+r_2] )  |{ Y_s =  y_s }_ 1  ≤ s ≤ r_1, { Z_s =  z_s }_1 ≤ s ≤ r_1+r_2] .
 
Further,  
   
    lim_h_1, h_2 → 0 1 /h_1^r_1 (d_Y+d_Z)  h_2^ r_2 d_Z [ ∏_s=1^r_1  K(  ( Y,  Z)   - ( Y_s,  Z_s)  /h_1)  ∏_s=r_1+1^r_1+r_2  K( Z -  Z_s /h_2) ψ(  X_[r_1+r_2] )  ] 
         
        = ∫  p_ Y, Z ( y, z)^r_1+1 p_ Z ( z)^r_2𝔼[ψ^2(  X_[r_1+r_2]) |{ Y_s =  y }_ 1  ≤ s ≤ r_1, { Z_s =  z }_1 ≤ s ≤ r_1+r_2  ] d y d z  .
 
    



    The proof follows similar to Lemma <ref>. The details are omitted. 
      



  Note that if the function ψ is conditionally degenerate of order 1 (as in Definition <ref>), then the limits in (<ref>) and (<ref>) are zero. In this case, by an expansion similar to (<ref>) one can show  that 

    1/h_1^r_1 (d_Y+d_Z)  h_2^ r_2 d_Z 𝔼[  ∏_s=1^r_1  K( ( y, z)  - ( Y_s, Z_s)  /h_1) ∏_s=r_1+1^r_1+r_2  K( z -  Z_s /h_2)    ψ(  X_[r_1+r_2] )  ]  
        = O(h_1^2 h_2^2) = O(h_1^4) ,

since h_2/h_1 → 0. Similarly, 

    1/h_1^r_1 (d_Y+d_Z)  h_2^ r_2 d_Z 𝔼[  ∏_s=1^r_1  K( ( Y,  Z)  - ( Y_s, Z_s)  /h_1) ∏_s=r_1+1^r_1+r_2  K( Z -  Z_s /h_2)    ψ(  X_[r_1+r_2] )  ] 
        = O(h_1^2 h_2^2) = O(h_1^4) .
 




Now, suppose the function ψ:(^d_X)^r_1× (^d_X)^r_2→, which is symmetric in its first r_1 and last r_2 arguments, is completely degenerate, as in Definition <ref>. Then we have following  2-sample version of Lemma <ref>. 


 
Suppose the assumptions of Lemma <ref> hold. Further, assume  that the function ψ:(^d_X)^r_1× (^d_X)^r_2→ is completely degenerate. Define 

    T_r_1, r_2, n^ y,  z : = 𝔼[ ∑_ i ∈ [n]^r_1∑_ j∈ [n]^r_2∏_s=1^r_1K( ( y, z)  - ( Y_i_s,  Z_i_s)  /h_1) ∏_s=r_1+1^r_1+r_2 K( z -  Z_j_s/h_2)  ψ(  X_( i, j))  ],

where  i = (i_1,i_2,…, i_r_1), j = (j_1, j_2,…, j_r_2), and 

    X_( i,  j) = ( X_i_1, X_i_2,…,  X_i_r_1;  X_j_1, X_j_2,…,  X_j_r_2).

Then the following hold:  



  * If r_1 = 2 ℓ_1 and r_2 = 2 ℓ_2 are both even,  
    
    lim_n →∞T_r_1, r_2, n^ y,  z/ (nh_1^ d_Y+d_Z )^r_1/2 (nh_2^ d_Z )^r_2/2
        =C_1^ℓ_1 C_2^ℓ_2ℓ_1 ! ℓ_2  ! ∫ψ^*( x_1,…, x_ℓ_1 + ℓ_2 ) ∏_i=1^ℓ_1 p_ X, Y, Z( x_i, y,  z)  ∏_i= ℓ_1+ 1^ℓ_1 + ℓ_2  p_ X, Z( x_i,  z)   ∏_i=1^ℓ_1 + ℓ_2 d x_i ,

    where ψ^*( x_1,…,  x_ℓ_1+ ℓ_2) := ψ( x_1, x_1, …, x_ℓ_1, x_ℓ_1;  x_ℓ_1+1, x_ℓ_1+1, …, x_ℓ_1+ℓ_2, x_ℓ_1+ℓ_2 ),  
    C_1 := ∫_^d_Y+d_Z K^2( ud u ), and C_2 := ∫_^d_Z K^2( v)d v. 
 
  * If either one of r_2 or r_1 is odd, 
 
    lim_n →∞T_r_1, r_2, n^ y,  z/ (nh_1^ d_Y+d_Z )^r_1/2 (nh_2^ d_Z )^r_2/2 = 0.








    




   The proof is similar to Lemma <ref>. We provide a sketch, omitting some details: Denote by 
i and j the distinct elements of i ∈ [n]^r_1 and j ∈ [n]^r_2. Then 
    
    T_r_1, r_2, n^ y,  z    =  ∑_0 ≤ a_1, b_1 ≤ r_1
     a_1 + b_1 ≤ r_1∑_0 ≤ a_2, b_2 ≤ r_2
     a_2 + b_2 ≤ r_2 T_a_1, b_1, a_2, b_2, n ,
 
       where 
       
    T_a_1, b_1, a_2, b_2, n
        := ∑_ i ∈ [n]^r_1
    ν_1( i) = a_1, ν_2( i) = b_1 ∑_ j ∈ [n]^r_2
    ν_1( j) = a_2, ν_2( j) = b_2 𝔼[ ∏_ s ∈ i K_s( ( y,  z)  - (  Y_s,  Z_s)  /h_1) ∏_ t ∈ j K_t( z -  Z_t/h_2)   ψ( X_(ℓ_ i, ℓ_ j))  ] ,

       with notation as in the proof of Lemma <ref>. 
       First note that when a_1+b_1  r_1/2 or a_2 + b_2  r_2/2, then as in the proof of Lemma <ref> we can show that 
        
    lim_n →∞ T_a_1, b_1, a_2, b_2, n/ (nh_1^ d_Y+d_Z )^r_1/2 (nh_2^ d_Z )^r_2/2 = 0 .
 
       Hence, suppose r_1 = 2 ℓ_1 and r_2 = 2 ℓ_2 are even and a_1+b_1 = ℓ_1 and a_2 + b_2 = ℓ_2. Then, by arguments similar to (<ref>) it can be shown that, when a_1 ≥ 1 or a_2 ≥ 1, 
           
    lim_n →∞ T_a_1, b_1, a_2, b_2, n/ (nh_1^ d_Y+d_Z )^r_1/2 (nh_2^ d_Z )^r_2/2 = 0 ,
 
      This implies, from (<ref>),            
      
    T_r_1, r_2, n^ y,  z/ (nh_1^ d_Y+d_Z )^r_1/2 (nh_2^ d_Z )^r_2/2  =  T_0, r_1/2, 0, r_2/2, n/ (nh_1^ d_Y+d_Z )^r_1/2 (nh_2^ d_Z )^r_2/2 + o(1) .
 
            The result in Lemma <ref> then now follows from Lemma <ref>. 
   


For a function ψ that is not necessarily symmetric, the result in Lemma <ref> holds with ψ' instead of ψ, where ψ' is the symmetrization of ψ as defined in (<ref>).   





 §.§ Proof of Proposition <ref>



We begin the following simple fact about kernel density estimation. 



    Suppose the assumptions of Proposition <ref> hold. If p_ Z( z)>0, then (recalling (<ref>))
    
    1/nh_2^ d_Z  w( z) = 
       1/nh_2^ d_Z ∑_i=1^n K( z- Z_i/h_2) L_2→ p_ Z( z) ,

    as n→∞. Similarly, if p_ Y,  Z( y,  z)>0, then 
    
    1/nh_1^ (d_Y+d_Z)  w(  y,  z) = 1/nh_1^ (d_Y+d_Z) ∑_i=1^n K(( y,  z) -( Y_i,  Z_i) /h_1) L_2→ p_ Y,  Z( y,  z).

    



    Note that by Lemma <ref>,
    
    [1/nh_2^ d_Z ∑_i=1^n K( z- Z_i/h_2)] = 1/h_2^d_Z[K( z- Z_1/h_2)] → p_ Z( z).
 
    Also, by Lemma <ref>,
    
    [1/nh_2^ d_Z ∑_i=1^n K( z- Z_i/h_2)]^2 =   n(n-1)/n^2h_2^2d_Z[K( z- Z_1/h_2) K( z- Z_2/h_2)] + n/n^2h^2d_Z[K^2( z- Z_1/h_2)]
       → p_ Z^2( z).

    Combining these we get [1/nh_2^ d_Z ∑_i=1^n K( z- Z_i/h_2)] → 0, as n→∞. This proves (<ref>). The result in (<ref>) can be proved similarly.  
    


Now, we proceed with the proof of Proposition <ref>. For any f∈ℱ_ z, we can write 
 
    S_n^ z(f)     :=  √(nh_2^ d_Z )(∫ f( x) dℙ_ X|  Z =  z( d x)-_ℙ_ X|  Z =  z [ f( X) ] ) 
        = √(nh_2^ d_Z )(∑_i=1^n w_ Z_i(  z ) f( X_i)/∑_i=1^n w_ Z_i(  z ) -_ℙ_ X|  Z = z [f( X)] ) 
        = √(nh_2^ d_Z )(1/nh_2^ d_Z ∑_i=1^n w_ Z_i(  z ) f( X_i)/1/nh_2^ d_Z ∑_i=1^n w_ Z_i(  z ) -_ℙ_ X|  Z = z [f( X)])
        = (1/√(nh_2^ d_Z )∑_i=1^n w_ Z_i(  z ) f( X_i) - 1/√(nh_2^ d_Z )∑_i=1^n w_ Z_i(  z ) _ℙ_ X|  Z = z [f( X)])/(1/nh_2^ d_Z ∑_i=1^n w_ Z_i(  z )  )  .

Similarly, for g ∈ℱ_ y,  z we have, 

    S_n^ y,  z(g)     := √(nh_1^ d_Y + d_Z )(∫ f( x) dℙ_ X|  Y= y,  Z =  z( d x)-_ℙ_ X|  Y= y,  Z =  z [ f( X) ] ) 
        = (1/√(nh_1^ d_Y + d_Z )∑_i=1^n w_ Y_i,  Z_i(  y,  z ) g( X_i) - 1/√(nh_1^ d_Y + d_Z )∑_i=1^n w_ Y_i,  Z_i(  y,  z ) _ℙ_ X|  Y =  y,  Z = z [g( X)])/(1/nh_1^ d_Y+d_Z ∑_i=1^n w_ Y_i,  Z_i(  y,  z )  )  .
 

Hence, given a finite collection of functions f_1, f_2, …, f_K_1∈ℱ_ z and a finite collection of functions g_1, g_2, …, g_K_2∈ℱ_ y,  z, 
to obtain the distribution of {{S_n^ z(f_s): 1 ≤ s ≤ K_1 }, {S_n^ y,  z(g_t): 1 ≤ t ≤ K_2 }}, 
we need to study the joint asymptotic behavior of 

    {(1/√(nh_2^ d_Z )∑_i=1^n w_ Z_i(  z ) , 1/√(nh_2^ d_Z )∑_i=1^n w_ Z_i(  z ) f_s( X_i) , 1 ≤ s ≤ K_1 ), 
               ( 1/√(nh_1^ d_Y+ d_Z )∑_i=1^n w_ Y_i,  Z_i(  y,  z ), 
    1/√(nh_1^ d_Y + d_Z )∑_i=1^n w_ Y_i,  Z_i(  y,  z ) g_t( X_i): 1 ≤ t ≤ K_2 ) } .
 
Due to the Cramer-Wold device, it is enough to derive the limiting distribution of 

    1/√(nh_2^ d_Z )∑_i=1^n w_ Z_i(  z ) ( ∑_s=1^K_1 a_s+ ∑_s=1^K_1 a_s' f_s( X_i) ) + 
    1/√(nh_1^ d_Y+ d_Z )∑_i=1^n w_ Y_i,  Z_i(  y,  z ) ( ∑_t=1^K_2 b_t  + ∑_t=1^K_2 b_t' g_t ( X_i) ),

for every a_1, a_2, …, a_K_1, a_1', a_2' …, a_K_1' ∈ and b_1, b_2, …, b_K_2, b_1', b_2' …, b_K_2' ∈. This means that it suffices to obtain the limiting distribution of 

    (1/√(nh_2^ d_Z )∑_i=1^n w_ Z_i(  z )  f( X_i), 1/√(nh_1^ d_Y+ d_Z )∑_i=1^n w_ Y_i,  Z_i(  y,  z ) g( X_i)) ,
 
for any f ∈^ z and g ∈ℱ^ y,  z. For this, consider, for α_1, α_2 ∈, 
    
    1/√(n)∑_i=1^n (α_1 1/√(h_2^d_Z) w_ Z_i( z) f( X_i) + α_2 1/√(h_1^d_Y+d_Z) w_ Y_i, Z_i( y, z) g( X_i))  = 1/√(n)∑_i=1^n V_n, i .

     Note that { V_n, i - [V_n, i]: 1≤ i≤ n} is a triangular array of random variables which are row-wise i.i.d. with variance σ_n^2 = [V_n, 1]. To compute the limit of σ_n^2, first note by that Lemma <ref>, as n →∞, 
     
    [ w_ Z_1( z) f( X_1)/√(h_2^d_Z)] → 0  and [w_ Y_1, Z_1( y, z) g( X_1)/√(h_1^d_Y+d_Z)] → 0.

     Also, by that Lemma <ref>, as n →∞, 
     
    [ w_ Z_1( z)^2 f( X_1)^2/h_2^d_Z] →(∫ K^2( u )d u) p_ Z( z) __ X|  Z = z[f( X)^2]
 and 
      
    [ w_ Y_1,  Z_1( y,  z)^2 g( X_1)^2/h_1^d_Y + d_Z] →(∫ K^2(( u,  v) )d u d v ) p_ Y,  Z( y,  z) __ X|  Y =  y,  Z = z[g( X)^2]  .
 
        Further, note that 
     
    [w_ Z_1( z) f( X_1)/√(h_2^d_Z)·w_ Y_1, Z_1( y, z) g( X_1)/√(h_1^d_Y+d_Z)] 
        = √(1/h_1^d_Y+d_Z h_2^d_Z)∫ K( z-  z_1/h_2) K ( ( y,  z) - ( y_1,  z_1) /h_1) f( x)g( x) p_ X, Y, Z( x, y_1,  z_1) d xd y_1 d z_1 
        = √(h_1^d_Y+d_Z h_2^d_Z)∫ K( u) K(( u,  v)) f( x)g( x) p_ X, Y, Z( x, y+h_1  u,  z + h_2  v) d xd u d v * (by the change of variables y_1 =  y +  h_1  u and z_1 =  z +  h_2  v)
       → 0 ,
 
    as n→∞, under Assumption <ref> and <ref>.  
    Hence, by Lemma <ref> we have,
     
    σ_n^2    →σ^2 
        := (∫ K^2( u)d u) α_1^2 p_ Z( z) __ X|  Z = z[f( X)^2] + (∫ K^2( (  u ,  v) )d u d v) α_2^2 p_ Y,  Z( y, z) _ℙ_ X |  Y= y,  Z =  z[g( X)^2] .

          Then by Lindeberg's central limit theorem, 
     
    1/√(n)∑_i=1^n ( V_n, i - [V_n, i]) D→ N(0,σ^2).

     Also, under assumption <ref> we have nh_1^d_Y+d_Z+4→ 0 and nh_2^d_Z+4→ 0, as n→∞. Therefore, by equation (<ref>)
     
    √(nh_2^d_Z)([1/h_2^d_Z w_ Z_1( z) f( X)] - ∫ f( x) p_ X, Z( x,  z)d x) → 0 ,  
       √(nh_1^d_Y+d_Z)([1/h_1^d_Y+d_Z w_ Y_1, Z_1( y,  z) g( X)] - ∫ g( x) p_ X, Y, Z( x, y,  z)d x) → 0.
 
Moreover, by Lemma <ref>, 
    1/nh_2^ d_Z ∑_i=1^n w_ Z_i(  z ) P→ p_ Z( z)  and 1/nh_1^ d_Y+d_Z ∑_i=1^n w_ Y_i,  Z_i(  y,  z ) P→ p_ Y,  Z(  y,  z).
  Hence, using the Cramer-Wold device and Slutsky's theorem we get the result in Proposition <ref>. 







 §.§ Proof of Proposition <ref>



We begin with the following result. 



 
Suppose the assumptions of Lemma <ref> hold. Further, assume  that the function ψ:(^d_X)^r_1× (^d_X)^r_2→ is completely degenerate as in Definition <ref>. If p_ Z( z)>0 and p_ Y, Z( y, z)>0, then  
    
    ∫ϕ( x_1, …,  x_r_1; x'_1,…, x'_r_2)∏_i=1^r_1d𝔾̂( x_i) ∏_j=1^r_2d𝔾̂'(x'_j) 
       D→∫ϕ( x_1, …,  x_r_1; x'_1…, x'_r_2)∏_i=1^r_1d𝔾( x_i) ∏_j=1^r_1d𝔾'(x'_j),

    as n →∞, where 
    
    𝔾̂ = √(nh_1^d_Y+d_Z)(ℙ_ X |  Y= y,  Z =  z - ℙ_ X|  Y=  y,  Z =  z)   and 𝔾̂'= √(nh_2^ d_Z )(ℙ_ X|  Z =  z - ℙ_ X|  Z =  z)  ,

    are the conditional empirical processes as defined in (<ref>) and (<ref>), respectively; and 
    
    𝔾 = 𝔾_ℙ_ X|  Y =  y,  Z =  z and 𝔾' =  𝔾_ℙ_ X|  Z =  z ,
 are as defined in Proposition <ref>.        



The proof of Lemma <ref> is given in Appendix <ref>. We will first use this result to prove Proposition <ref>. For this, let us define,

    V_n(ϕ)     = ∫ϕ( x_1, …,  x_r_1; x'_1,…, x'_r_2)  ∏_s=1^r_1d_ X| ( Y, Z)  = ( y, z) ( x _s) ∏_s=r_1+1^r_1+r_2d_ X|  Z =  z(x'_s)  ,
 
Now let us define,

    ψ(t,s) = ∫ϕ( x_1, …,  x_r_1; x'_1,…, x'_r_2) ∏_s=1^r_1dΔ_1( x_s) ∏_s=r_1+1^r_1+r_2Δ_2(x'_s),

where 
    Δ_1 = ℙ_ X |  Y= y,  Z =  z+s√(nh_1^d_Y+d_Z)(_ X|  Y =  y,  Z= z - ℙ_ X |  Y= y,  Z =  z)
 
and Δ_2 = _ X|  Z = z+t√(nh_2^d_Z)(_ X| Z= z-_ X|  Z = z). 
Note that ψ(t,s) is a bivariate polynomial in (t,s) with coefficient of t^is^j being 

    r_1 ir_2 j∫ϕ_i,j( x_1,…, x_i; x'_1,…, x'_j) ∏_s=1^id𝔾̂( x_s) ∏_s=1^jd𝔾̂'(x'_s) ,

where ϕ_i,j is the (i, j)-th Hoeffding's projection of the symmetrized function ϕ' (recall (<ref>)). 
Hence, by Lemma <ref>, 

    ∫ϕ_i,j( x_1,…, x_i; x'_1,…, x'_j) ∏_s=1^id𝔾̂( x_s) ∏_s=1^jd𝔾̂'(x'_s) →∫ϕ_i,j( x_1,…, x_i; x'_1,…, x'_j) ∏_s=1^id𝔾( x_s) ∏_s=1^jd𝔾'(x'_s) .

In particular, this implies the coefficients of t^is^j in ψ(s,t) are tight, for all i+j≥ 1. Hence, noting that ψ(0,0) = θ,  

    V_n(ϕ) = ψ( 1/(nh_1)^(d_Y+d_Z)/2 , 1/(nh_2)^d_Z/2) = ψ(0,0) + R_n = θ + R_n,
 
where R_n = O_P((nh_1^d_Y+d_Z)^-1/2). This completes the proof of Proposition <ref> (1). 


Next, suppose ϕ is conditionally degenerate of order k ≥ 0. (Note that k=0 means ϕ is conditionally non-degenerate). Then from Definition <ref>, the (i, j)-th Hoeffding's projection ϕ_i,j= 0, for all i+j≤ k. Then by a Taylor expansion,  

    V_n(ϕ) 
       = θ + ∑_(i, j) 
     i+j=k+1r_1 ir_2 j/n^(k+1)/2 h_1^ i (d_Y+d_Z)/2 h_2^ j d_Z/2∫ϕ_i,j ( x_1,…, x_i; x'_1,…, x'_j) ∏_s=1^id𝔾̂( x_s) ∏_s=1^jd𝔾̂'(x'_s) + R_n,

where 
    R_n=O_P( 1/(nh_1^d_Y+d_Z)^(k+2)/2 + 1/(nh_2^d_Z)^(k+2)/2),
 by Lemma <ref>. Since by Assumption <ref>, h_1^d_Y+d_Z/h_2^d_Z→0, we have,

    (nh_1^(d_Y+d_Z))^(k+1)/2 (V_n(ϕ) -θ) 
        =  ∑_i+j=k+1r_1 ir_2 j h_1^(k+1-i) (d_Y+d_Z)/2/ h_2^ (k+1-i) d_Z/2∫ϕ_i,j ( x_1,…, x_i; x'_1,…, x'_j) ∏_s=1^id𝔾̂( x_s) ∏_s=1^jd𝔾̂'(x'_s) + o_P(1) 
       D→r_1 k+1∫ϕ_k+1, 0 (x_1,…, x_k+1) ∏_i=1^k+1d𝔾(x_i).

The result in Proposition <ref> (2) follows by substituting k=0 in (<ref>). Also, Proposition <ref> (3) follows from (<ref>) by noting that θ=0, for k ≥ 1. 









  §.§.§ Proof of Lemma <ref> 



Let f_0=1, f_1,f_2,… be an orthogonal basis of L_2(ℝ^d_X, ℙ_ X|  Y= y,  Z= z) and g_0=1,g_1,g_2,… be the same for L_2(ℝ^d_X, ℙ_ X|  Z =  z). Since ϕ is completely degenerate, it can represented as 

    ϕ = ∑_i_1,…,i_r_1, j_1,…, j_r_2≥ 1⟨ϕ,f_i_1×⋯× f_i_r_1× g_j_1×⋯× g_j_r_2⟩ f_i_1×⋯× f_i_r_1× g_j_1×⋯× g_j_r_2,

where the equality holds in the L_2 sense. We define the truncated version of ϕ as follows: For L ≥ 1, 

    ϕ_L = ∑_1≤ i_1,…,i_r_1, j_1,…,j_r_2≤ L⟨ϕ,f_i_1×⋯× f_i_r_1× g_j_1×⋯× g_j_r_2⟩ f_i_1×⋯× f_i_r_1× g_j_1×⋯× g_j_r_2.
 
Note that by the orthogonality of {f_s}_s≥ 1 and {g_s}_s≥ 1 we have,
    
    ϕ-ϕ_L _2^2 
        = ∫ [ ϕ( x_1,…, x_r_1;x'_1,…,x'_r_2)-ϕ_L( x_1,…, x_r_1;x'_1,…, x'_r_2) ]^2 ∏_i=1^r_1dℙ_ X |  Y= y,  Z =  z(x_i) ∏_i=1^r_2d_ X|  Z = z(x'_i)  
       → 0,

    as L→∞. 
    
    The proof of Lemma <ref> proceeds in 3 steps: First we show that the integral of ϕ_L with respect to the conditional empirical processes 𝔾̂ and 𝔾̂' converges to its limiting counterpart with 𝔾 and 𝔾', as n →∞. To this end, using Proposition <ref> and the  continuous mapping theorem note that 
    
    {∏_s=1^r_1∫ f_i_s( u) d𝔾̂ ( u) ∏_s=1^r_2∫ g_j_s( u) d𝔾̂' ( u) }D→{∏_s=1^r_1∫ f_i_s( u) d𝔾 ( u) ∏_s=1^r_2∫ g_j_s( u) d𝔾' ( u) },

    where 𝔾̂ and 𝔾̂' are the conditional empirical processes and 𝔾 and 𝔾' the corresponding Gaussian processes as defined in the statement Lemma <ref>. Now, recalling (<ref>) and applying the continuous mapping theorem again gives, for every L ≥ 1, 
    
    ∫ϕ_L( x_1,…, x_r_1;x'_1,…, x'_r_2) ∏_i=1^r_1d𝔾̂ ( x_i) ∏_i=1^r_2d𝔾̂' (x'_i) 
        = ∑_1≤ i_1,…,i_r_1, j_1,…,j_r_2≤ L⟨ϕ,f_i_1×⋯× f_i_r_1× g_j_1×⋯× g_j_r_2⟩∏_s=1^r_1∫ f_i_s( u) dĜ( u)×∏_s=1^r_2∫ g_j_s( u)dĜ' ( u) 
       D→∑_1≤ i_1,…,i_r_1, j_1,…,j_r_2≤ L⟨ϕ,f_i_1×⋯× f_i_r_1× g_j_1×⋯× g_j_r_2⟩∏_s=1^r_1∫ f_i_s( u) d𝔾( u)×∏_s=1^r_2∫ g_j_s( u)d 𝔾' ( u) 
        = ∫ϕ_L( x_1,…, x_r_1;x'_1,…, x'_r_2) ∏_i=1^r_1d𝔾 ( x_i) ∏_i=1^r_2d𝔾' (x'_i),

    as n→∞. 
    
   The second step is to show that the difference ψ_L := ϕ-ϕ_L under 𝔾 and 𝔾' is negligible in L_2, as L →∞. Towards this, for a L_2 function f define,   
    
    V(f) := ∫ f( x_1,…, x_r_1;x'_1,…, x'_r_2) ∏_i=1^r_1d𝔾 ( x_i) ∏_i=1^r_2d𝔾' (x'_i).

    Since {∫ f_s ( u) d𝔾( u) }_1 ≤ s ≤ r_1 and {∫ g_s ( u) d𝔾'( u) }_1 ≤ s ≤ r_2 are two independent sequence of mean zero Gaussian random variables with variance c_K( z) and c_K( y, z), respectively, for any L ≥ 1, 
    
    [V(ψ_L)]^2    ≤ C_r_1, r_2, K∑_i_1,…,i_r_1, j_1,…,j_r_2≥ 1|⟨ψ_L, f_i_1×⋯× f_i_r_1× g_j_1×⋯× g_j_r_2⟩|^2 
       ≤ C_r_1, r_2, Kψ_L _2^2,

    for some constant C_r_1, r_2, K > 0 depending on the kernel K and r_1, r_2. Therefore, using (<ref>),  
    
    lim_L→∞[V(ψ_L)^2 ]=0.

    
 
    Finally, we show that ψ_L := ϕ-ϕ_L is negligible in L_2 under 𝔾̂ and 𝔾̂'. For this, similar to (<ref>),  define 
        
    V_n(f) := ∫ f( x_1,…, x_r_1;x'_1,…, x'_r_2) ∏_i=1^r_1d𝔾̂ ( x_i) ∏_i=1^r_2d𝔾̂' (x'_i).
 
       Since ϕ is completely conditionally degenerate, ψ_L is also  completely conditionally degenerate. Therefore, 
       recalling the definition of 𝔾̂ and 𝔾̂' from (<ref>) gives,     
       
    V_n( ψ_L ) 
        =  (nh_1)^r_1(d_Y+d_Z)/2 (nh_2)^ r_2 d_Z/2∫ψ_L( x_1,…, x_r_1;x'_1,…, x'_r_2) ∏_s=1^r_1dℙ_ X |  Y= y,  Z =  z (x_i)  ∏_s=r_1+1^r_1+r_2dℙ_ X|  Z =  z (x'_i) 
        = (nh_1)^r_1(d_Y+d_Z)/2 (nh_2)^ r_2 d_Z/2[   ∑_ i ∈ [n]^r_1∑_ j ∈ [n]^r_2∏_s=1^r_1 w_( Y_i_s,  Z_i_s)( y,  z) ∏_s=r_1+1^r_1+r_2 w_ Z_j_s( z)  ψ_L( X_ i ;  X_ j)   / w(  y,  z)^ r_1 w( z)^ r_2] .
  
By Lemma <ref>,  1/n h_2^d_Zw( z)  and  1/n h_1^d_Y+d_Zw( y,  z)  converges in L_2 to p_ Z( z) and p_ Y,  Z( y,  z). This implies, defining 

    T_n, L := 1/ (nh_1)^r_1(d_Y+d_Z)/2 (nh_2)^ r_2 d_Z/2∑_ i ∈ [n]^r_1∑_ j ∈ [n]^r_2∏_s=1^r_1 w_( Y_i_s,  Z_i_s)( y,  z) ∏_s=r_1+1^r_1+r_2 w_ Z_j_s( z)  ψ_L( X_ i ;  X_ j) ,

we have T_n, L/V_n(ψ_L) converges in L_2 to p_ Y,  Z( y,  z)^r_1 p_ Z( z)^r_2. Therefore, it suffices to shows that [T_n, L^2] → 0. Note that

    T_n, L^2     = ∑_ i ∈ [n]^2r_1∑_ j ∈ [n]^2r_2∏_s=1^2r_1 w_( Y_i_s,  Z_i_s)( y,  z)  ∏_s=1^2r_2 w_ Z_j_s( z)  ψ̃_L( X_ i;  X_ j) / (nh_1)^r_1(d_Y+d_Z) (nh_2)^ r_2 d_Z ,

where ψ̃_L is a function in 2 r_1 + 2 r_2 variables defined below: 
    
    ψ̃_L( x_1,…, x_2r_2,  x_2r_1+1',…, x_2(r_1+r_2)')
        = (ϕ-ϕ_L)( x_1,…,  x_r_2;  x_2r_1+1',…,  x_2r_1+r_2' )(ϕ-ϕ_L)( x_r_1+1,…,  x_2r_2;  x_2r_1+r_1+1',…,  x_2r_2+2r_1').
 
 Hence, using Lemma <ref>, 
    
    lim_n→∞[ T_n, L^2 ] 
        = (r_1)!(r_2) ! ∫ψ̃_L'( x_1, x_1,…, x_r_1+r_2, x_r_1+r_2) ∏_s= 1 ^r_1 p_ X, Y, Z( x_s, y,  z) ∏_s=r_1+1^r_1+r_2 p_ X, Z( x_s,  z)  ∏_s=1^ (r_1+r_2) d x_s 
        : = T_L ,

    where ψ_L' is the symmetrized version (as in (<ref>)). 
    Now, using the basis expansions (<ref>) and (<ref>) and the Cauchy-Schwarz inequality it follows that 
    
    T_L    ≤ (r_1)!(r_2)! p_ Y, Z( y, z)^r_1 p_ Z( z)^r_2𝔼[ ψ_L^2(  X_[r_1+r_2]) |{ Y_s =  y }_ 1  ≤ s ≤ r_1, { Z_s =  z }_1 ≤ s ≤ r_1+r_2  ] 
       → 0 ,

as L →∞, by (<ref>). Hence, from (<ref>) and recalling that T_n, L/V_n(ψ_L) converges in L_2 to p_ Y,  Z( y,  z)^r_1 p_ Z( z)^r_2, we have 

    lim_L→∞lim_n→∞[V_n(ψ_L)^2]→ 0.

Combining (<ref>), (<ref>), and (<ref>), the result in Lemma <ref> follows.   





§ ADDITIONAL RESULTS
 





 §.§ A Sufficient Condition for (<ref>)
 


Here, we derive a sufficient condition for a  density f( x,  y) to be nice (recall Definition <ref>). 


Suppose ( X,  Y) is a random variable in ^d_X+d_Y with joint density f( x,  y) and marginal densities f_ X( x) and f_ Y( y). Moreover, suppose an appropriate interchange of integrals and derivatives is allowed (see (<ref>)) and for every y, u ∈^d_Y, |R_n( x, y; u)|≤ M( x, y; u) where ∫ M^2( x, y;  u) f( x, y)d x is finite. Then (<ref>) holds if f_ Y( y), 

    sup_ u:  u=1[( u ∂/∂ ylog f_ Y| X( Y| X)^⊤)^2| Y= y],  and sup_ u: u=1[ u ∂^2/∂ y^2log f_ Y| X( Y| X) u^⊤| Y= y]
 are all bounded in y. 


 
By Taylor's theorem, 

    f( x, y+δ_n u) = f( x, y) + δ_n  u ∂/∂ y f( x, y)^⊤ + δ_n^2/2 u ∂^2/∂ y^2 f( x,  y_n^')  u^⊤,

for some intermediate y_n^' such that y -  y_n^'≤δ_n. Then, as n→∞, we have,

    R_n( x, y; u)  = 1/2 u ∂^2/∂ y^2 f( x,  y_n^')  u^⊤/f( x, y)   →1/2 u ∂^2/∂ y^2 f( x,  y)  u^⊤/f( x, y)
        = 1/2 u ∂^2/∂ y^2 f_ Y| X( y| x)  u^⊤/f_ Y| X( y| x)
        = 1/2{( u ∂/∂ ylog f_ Y| X( y| x)^⊤)^2 +  u ∂^2/∂ y^2log f_ Y| X( y| x) u^⊤}.

If the interchange of integrals and derivatives are allowed, it is easy to see that,

    ∫( u ∂/∂ ylog f_ Y| X( y| x)^⊤)^2 f( x, y)d x = -∫ u ∂^2/∂ y^2log f_ Y| X( y| x) u^⊤ f( x, y) d x.

This implies, 

    ∫{( u ∂/∂ ylog f_ Y| X( y| x)^⊤)^2 +  u ∂^2/∂ y^2log f_ Y| X( y| x) u^⊤} f( x, y) d x  = 0.

Now, using |R_n( x,  y; u)| ≤ M( x, y; u) where ∫ M^2( x,  y; u) f( x, y)d x is finite and the Reverse Fatou's Lemma we get 

    lim sup_n→∞∫(R_n( x, y; u))^2 f( x, y)d x 
       ≤∫lim sup_n→∞(R_n( x, y; u))^2 f( x, y)d x 
        = ∫1/4{( u ∂/∂ ylog f_ Y| X( y| x)^⊤)^2 +  u ∂^2/∂ y^2log f_ Y| X( y| x) u^⊤}^2 f( x, y)d x 
        = 1/4[( u ∂/∂ ylog f_ Y| X( Y| X)^⊤)^2 +  u ∂^2/∂ y^2log f_ Y| X( Y| X) u^⊤|  Y= y] f_ Y( y) .

Hence, (<ref>) holds if f_ Y( y), 

    sup_ u:  u=1[( u ∂/∂ ylog f_ Y| X( Y| X)^⊤)^2| Y= y],  and sup_ u: u=1[ u ∂^2/∂ y^2log f_ Y| X( Y| X) u^⊤| Y= y]
 are bounded in y. 






 §.§ Normalized Conditional Ball Divergence
 



In this section we prove the following result about the normalized conditional Ball Divergence (recall (<ref>)): 
    
    ℛ( X,  Y|  Z) :=   [Θ^2(_ X|  Y,  Z, _ X|  Z) ) ]/𝔼 [∫Θ^2(_δ_ x, _ X|  Z) d_ X| Y, Z ( x)   ]  .






   The normalized conditional Ball Divergence has the following properties: 
     
         
  (1) ℛ( X,  Y|  Z) ∈ [0, 1], 
         
  (2) ℛ( X,  Y|  Z) = 0 if and only if X Y |  Z, and
         
  (3) ℛ( X,  Y|  Z) = 1 if and only if X is a measurable function of Y and Z almost surely. 
     


 
    Recalling (<ref>) note that 
    
    Θ^2(F,G)     = A + B ,
 
    where 

    A     := ∫ (F-G)^2(B̅( u,ρ( v, u)))d F( u)d F( v) , 
     
        B     := ∫ (F-G)^2(B̅( u,ρ( v, u)))d G( u)d G( v) .
 
    Now, for any fixed u, v∈^d, define Q( w) =  1{ w ∈B̅( u,ρ( v, u)) } - G(B̅( u,ρ( v, u))). Note that, for W ∼ F, 
    
    𝔼[Q( W)] = 𝔼[  1{ W ∈B̅( u,ρ( v, u)) } - G(B̅( u,ρ( v, u))) ]  = (F-G)(B̅( u,ρ( v, u)).
 
    Hence, by Jensen's inequality, 
    
    (F-G)^2(B̅( u,ρ( v, u)) =  𝔼[Q( W)]^2    ≤𝔼 [Q( W)^2] 
        = 𝔼[ (  1{ W ∈B̅( u,ρ( v, u)) } - G(B̅( u,ρ( v, u))) )^2 ] 
        = ∫(  1{ w ∈B̅( u,ρ( v, u)) } - G(B̅( u,ρ( v, u))) )^2 d F( w) 
        = ∫ (ℙ_δ_ w - G)^2(B̅( u,ρ( v, u))) d F( w) .
     
  Now, combining (<ref>) and (<ref>) and applying Fubini's theorem gives, 
    
    A    ≤∫(∫ (_δ_ w - G)^2 (B̅( u,ρ( v, u))) d F( u)d F( v) ) d F( w) .

        Similarly, we get 
    
    B    ≤∫(∫ (_δ_ w - G)^2 (B̅( u,ρ( v, u))) d G( u)d G( v) ) d F( w).

   Combining (<ref>), (<ref>), and recalling the definition of Θ^2(F,G) from (<ref>) then gives, 
    
    Θ^2(F,G) ≤∫Θ^2(_δ_ w, G) d F( w).

    Setting F = _ X| Y, Z and G = _ X| Z in (<ref>) gives, 
    
    Θ^2(_ X| Y, Z,_ X| Z) ≤∫Θ^2(_δ_ x, _ X| Z) d_ X| Y, Z( x).

    Therefore, taking expectations on both sides with respect to ( Y,  Z) shows ℛ( X, Y| Z) ∈ [0,1]. This proves Proposition <ref> (1). 
    
    It is easy to see that ℛ( X, Y| Z) = 0 if and only if Θ^2(_ X| Y, Z, _ X| Z) = 0 almost everywhere, that is, if and only if X Y| Z (by Proposition <ref>). This shows Proposition <ref> (2). 
    
    Next, note that ℛ( X, Y| Z) = 1 if and only if  (<ref>) holds with an equality. This implies, by the equality condition of Jensen's inequality, for almost every u,  v, 
    
    Q( x) =  1{ x ∈B̅( u,ρ( v, u)) } - _ X |  Z(B̅( u,ρ( v, u)))  is constant almost everywhere _ X| Y, Z.
 
  This means _ X| Y, Z is a degenerate distribution or, in other words, X is a measurable function of ( Y, Z). This proves Proposition <ref> (3).  






 Paper Title: Generalized Multivariate Functional Additive Mixed Models for Location, Scale, and Shape.


GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Models
    [
    
============================================================================



§ ABSTRACT 
We propose a flexible regression framework to model the conditional distribution of multilevel generalized multivariate functional data of potentially mixed type, e.g. binary and continuous data. We make pointwise parametric distributional assumptions for each dimension of the multivariate functional data and model each distributional parameter as an additive function of covariates. The dependency between the different outcomes and, for multilevel functional data, also between different functions within a level is modelled by shared latent multivariate Gaussian processes. For a parsimonious representation of the latent processes, (generalized) multivariate functional principal components are estimated from the data and used as an empirical basis for these latent processes in the regression framework. Our modular two-step approach is very general and can easily incorporate new developments in the estimation of functional principal components for all types of (generalized) functional data. Flexible additive covariate effects for scalar or even functional covariates are available and are estimated in a Bayesian framework. We provide an easy-to-use implementation in the accompanying  package gmfamm on CRAN and conduct a simulation study to confirm the validity of our regression framework and estimation strategy. The proposed multivariate functional model is applied to four dimensional traffic data in Berlin, which consists of the hourly numbers and mean speed of cars and trucks at different locations.




§ INTRODUCTION


Repeated measurements of variables of interest are becoming more easily available to researchers, with one observation often combining several variables of different characteristics such as binary, count, or continuous data. A range of different approaches are available to model such multivariate longitudinal data of mixed types with <cit.> giving an excellent overview and current literature working on improving or extending these approaches, e.g. <cit.>. Given that the data can often be measured on a nearly continuous, fine grid, such as for data obtained from wearable devices <cit.>, or the repeated measurements can represent non-temporal dependencies, such as tract distance for diffusion tensor imaging <cit.>, understanding these data as generated from an underlying smooth multivariate process can often be insightful. Viewing the repeated scalar measurements as realizations of a multivariate functional observation over some domain ℐ then allows to compare functions and identify differences in their functional forms,  or analyze variation in the functional data sample, see e.g. <cit.> for an introduction to functional data analysis. We develop a flexible additive regression approach for such (generalized) multivariate functional data, which can be of mixed type, exhibit a multilevel structure, and are potentially only sparsely observed.

When the underlying continuous process is not Gaussian for each t ∈ℐ, however, the analysis of functional data becomes more intricate. Previous approaches in univariate non-Gaussian functional data analysis for fpca <cit.>, registration <cit.>, or functional regression <cit.> typically assume a latent continuous (often Gaussian) process related to the smooth mean of the non-Gaussian functional data via a known link function. Other authors have proposed to use copulas to model the temporal dependence <cit.> or approaches more robust against distributional assumptions building on Kendall's τ <cit.>. For generalized multivariate functional data of mixed type, however, few approaches exist. <cit.> and <cit.> model bivariate functional data with exponential family distributions using known link functions and correlated random effects. Such an approach can become computationally quite expensive when a higher number of functional outcomes are jointly analyzed. <cit.> extend this approach by including an ordinal component to model different distributional assumptions for the bivariate functional outcome depending on the individual's activity pattern. <cit.> apply a Gaussian copula model to skewed continuous functional data but do not provide an extension to discrete functional responses. <cit.> propose to model multivariate non-Gaussian continuous functional data through a shared univariate Gaussian process in a normal transformation linear model. While their semiparametric approach does not assume given link functions or covariance structures, the dependency between the multivariate functional data is restricted to one shared latent factor and the approach does not extend to discrete, e.g. binary, functional data.

We propose to use a multivariate latent Gaussian process to model the dependency within and between the functional dimensions of a generalized multivariate functional observation of possibly mixed type. Our approach is similar to <cit.>, who suggest to model (bivariate) binary and continuous multivariate functional data by either using a prespecified basis such as B-splines for the latent process or to estimate multivariate eigenfunctions from the data in a preliminary step. Their approach, however, in addition to only considering binary and continuous data, requires the estimation of all auto- and cross-covariance functions of the multivariate latent Gaussian process. This can become computationally intensive quite quickly when the multivariate functional data contain several dimensions, and cannot be applied to functional data that feature additional dependencies such as a multilevel structure. Instead, we show that the analysis can benefit from recent developments in mfpca. While most work has focused on advancing mfpca for Gaussian or continuous functional data, e.g. <cit.>, we are not aware of an mfpca approach dedicated to generalized multivariate functional data of mixed type. Notably, however, <cit.> provide a flexible framework allowing the estimation of the mfpc from separate univariate fpca.  This avoids the computationally intensive and potentially difficult explicit estimation of cross-covariance functions. <cit.> have shown that the mfpca approach proposed by <cit.> can be successfully applied to multivariate count functional data by assuming a latent Gaussian process. We extend this idea to multivariate functional data of mixed type, allowing us to build on and combine the wide range of fpca methods for univariate non-Gaussian functional data.

Combining the mfpca approach of <cit.> with the latent multivariate Gaussian process assumption allows us to establish a general and flexible approach to analyzing generalized multivariate functional data of mixed type with a potential multilevel structure. For the pointwise distributional assumptions of the generalized functional outcomes, we build on bamlss <cit.>, which opens up the analysis to a wide range of parametric distributions <cit.>, suitable for continuous, binary, count, and discrete functional data. This also allows to flexibly model all distributional parameters such as mean, variance, or skewness parameters as functions of covariates. Building on the famm established by <cit.>, our generalized multivariate functional regression framework  comprises possibly nonlinear effects or smooth interaction terms of scalar or functional covariates – possibly differing by dimension – as implemented in the well developed additive model framework <cit.>. As a result, the model proposed here extends the multivariate functional additive mixed model for multivariate Gaussian functional data introduced by <cit.> in two ways: the Gaussian assumption can be replaced by a more suitable distribution if needed and all distributional parameters instead of only the mean can be modeled by additive predictors. We employ a Bayesian estimation strategy to efficiently handle the (potentially) large number of parameters to estimate. Compared to existing approaches, our proposed model is thus more general  with respect to 

    
  a)the possible distributions and their combination for the different dimensions of generalized multivariate functional outcomes,
    
  b) the possible dependence structure between the dimensions as well as between potentially multilevel observations,

and more flexible with respect to 

    
  c) the modeling options of covariate information, including functional random effects, nonlinear additive, or smooth interaction effects, as well as
    
  d) flexible effects on other distributional parameters than the mean.


We showcase our approach in an analysis of data on traffic flows in the German capital. Scattered across the municipal area of Berlin, there is a large number of traffic detectors measuring the hourly number and mean speed of vehicles passing the detectors, giving a rich data base on traffic movements and changes since 2015. In our analysis, we focus on commuter traffic into the city center of Berlin and investigate seasonality and changes over the recorded years. The outcome of interest is a four dimensional generalized functional observation corresponding to the number and mean speed of cars and trucks measured over the last hour. Modeling these count and (potentially skewed) positive continuous outcomes together gives a more holistic understanding of the processes involved than would be obtained with univariate analyses. For example, the mean speed of cars might be closely connected to the number of cars passing the detectors as well as the number of (often slower) trucks on the road <cit.>.

In the following, Section <ref> introduces the proposed generalization of multivariate functional additive mixed models, while Section <ref> covers the estimation approach. Section <ref> then illustrates the suitability of the approach in a simulation study and Section <ref> presents the analysis of the Berlin traffic data. Section <ref> closes with a discussion and outlook.





§ MODEL



Let Y_i(t) = (Y_i^(1)(t), ..., Y_i^(K)(t))^⊤, i = 1,..., n be a (generalized) multivariate functional observation of K dimensions gathering K univariate functions Y_i^(k)(t), k = 1,.., K, t ∈ℐ on the common bounded interval ℐ. At each point t, we assume that given some covariate set 𝒳_i and some set of latent processes _i, the observed Y_it^(k) = Y_i^(k)(t) are conditionally independent and follow a parametric distribution 𝒟^(k) with R^(k) distributional parameters θ_r^(k)(t), r = 1,..., R^(k) depending on 𝒳_i and _i, so that 

    Y_it^(k)|𝒳_i, _i ∼𝒟^(k)(θ_i1^(k)(t), ..., θ_iR^(k)^(k)(t)),
    θ_ir^(k)(t) = h_r^(k)(η_ir^(k)(t)).

The distributional parameters are linked to additive predictors η_ir^(k)(t) containing the information on 𝒳_i and _i using known monotonic and twice differentiable functions h_r^(k). For notational convenience, we drop the conditional dependency on 𝒳_i and _i. We then assume that the θ_ir^(k)(t) ∈ L^2(ℐ) are smooth functions over t and that the first distributional parameter θ_i1^(k)(t) is related to the location of the distribution. Note that most common distributions such as the normal, Bernoulli, or Poisson distributions can be parameterized so that  θ_i1^(k)(t) corresponds to the conditional mean 𝔼(Y_i^(k)(t) |𝒳_i, _i). The introduced notation thus allows us to gather K univariate generalized functions of (potentially) mixed type, that is with different pointwise distributional assumptions, in Y_i(t). 

We assume that the dependency (conditional on 𝒳_i) between the univariate functions is fully captured by the set of latent processes _i and is limited to the location parameters θ_i1^(1),..., θ_i1^(K). In other words, distributional parameters θ_i2^(k), ..., θ_iR^(k)^(k), k = 1,...,K such as scale parameters related to the spread of the pointwise distributions are not connected to distributional parameters of other dimensions, except via shared covariates in 𝒳_i. For the location parameters, however, we allow dependence via independent replicates of a latent centered multivariate Gaussian process Λ_0i(t) = (Λ_0i^(1)(t),..., Λ_0i^(K)(t))^⊤∼𝒢𝒫(0(t), 𝒦_0(t, ·)) for each i = 1,..., n with zero-mean function 0(t) and covariance kernel 𝒦_0(t, ·). The univariate functions Λ_0i^(k)(t)∈ L^2(ℐ) are square-integrable functions  with inner product ⟨ f^(k), g^(k)⟩ = ∫_ℐf^(k)(t) g^(k)(t)dt for f^(k), g^(k)∈ L^2(ℐ), and Λ_0i(t) ∈ L_K^2(ℐ) = L^2(ℐ) × ... × L^2(ℐ), which is a Hilbert space with scalar product ⟨⟨f, g⟩⟩ = ∑_k = 1^K w_k ⟨ f^(k), g^(k)⟩ for f,g∈ L_K^2(ℐ) with positive weights w_k typically being 1. The associated covariance operator Γ_0 allows the decomposition of the covariance functions via Mercer's theorem as Cov(Λ_0i^(k)(s), Λ_0i^(k')(t)) = ∑_m=1^∞ν_0mψ_0m^(k)(s)ψ_0m^(k')(t), s,t ∈ℐ, k,k' = 1,..., K with ν_0m the eigenvalues and ψ_0m(t) = (ψ_0m^(1)(t),..., ψ_0m^(K)(t))^⊤ the eigenfunctions of Γ_0. We denote the covariance functions as cross-covariances for k ≠ k' and auto-covariances for k = k'. 

For data with more complex sampling regimes such as nested or crossed study designs, the set of latent processes _i and thus the model for θ_i1^(k) can be extended by additionally including mutually independent multivariate Gaussian processes Λ_u(t) ∼𝒢𝒫(0(t), 𝒦_u(t, ·)), Λ_u(t)∈ L_K^2(ℐ), u = 1,..., U as in <cit.>. Each Λ_u(t) for a given u then represents a grouping layer such as a hospital in a multicenter study with patients in different hospitals, capturing dependency between multivariate functional observations of patients in the same hospital and allowing a multilevel structure. For each level, e.g. hospital j = 1,..., J_u in the grouping layer u, we assume an iid copy of the multivariate Gaussian process Λ_u(t) denoted as Λ_uj(t) and introduce an indicator z_uij which is 1 if observation i belongs to level j of grouping layer u and is 0 otherwise. This allows us to account for similarities within the grouping levels and capture variation across groups.

We set up the following additive model structure

    η_i1^(k)(t) = ∑_l = 1^L_1^(k) f_1l^(k)(𝒳_i, t) + Λ_0i^(k)(t) + ∑_u = 1^U∑_j=1^J_uΛ_uj^(k)(t)z_uij, 
    η_ir^(k)(t) = ∑_l = 1^L_r^(k) f_rl^(k)(𝒳_i, t),    r =2,..., R^(k)

with additive functions f_rl^(k)(𝒳_i, t), l = 1,..., L_r^(k), r = 1,...,R^(k) flexibly modeling the additive predictor η_ir^(k)(t) depending on the covariate set 𝒳_i and _i. The number of additive functions L_r^(k) can differ for each distributional parameter r of each univariate functional outcome k and is chosen by the analyst. We use the general notation f_rl^(k)(𝒳_i, t) to indicate that each additive term can depend on single covariates in 𝒳_i as well as consist of interactions between subsets of 𝒳_i and can represent e.g. (functional) intercepts, linear effect terms, smooth (interaction) effects, or even more complex model terms such as spatio-temporal effects, see <cit.>. This flexibility is achieved by using basis function representations

    f_rl^(k)(𝒳_i, t) = b_rl^(k)(𝒳_i, t)^⊤β_rl^(k)

to model potentially nonlinear covariate effect terms. Here, b_rl^(k)(𝒳_i, t) 
denotes a vector of d_rl^(k) known basis functions evaluated at the included subset of 
covariates of 𝒳_i and t, and β_rl^(k) is the corresponding coefficient 
vector (see Section <ref>). The latent multivariate Gaussian process
Λ_0i(t) is represented using a truncated multivariate kl expansion with
k-th dimension

    Λ_0i^(k)(t) ≈∑_m = 1^M_0ρ_0imψ_0m^(k)(t),

where M_0 denotes the number of eigenfunctions ψ_0m(t) = (ψ_0m^(1)(t),..., ψ_0m^(K)(t))^⊤ in the approximation and ρ_0im is the m-th random score for subject i shared across the K dimensions. Note that for M_0 →∞, this approximation would become exact. As Λ_0i(t) is a Gaussian process, the random scores are iid ρ_0i = (ρ_0i1,...ρ_0iM)^⊤∼ N(0, diag(ν_01, ..., ν_0M)) with 0 a vector of zeros and diag(·) a diagonal matrix. Like the number of basis functions d_rl^(k), the truncation order M_0 is chosen by the analyst, depending on the data problem at hand. We will provide a guideline for setting M_0 in Section <ref>. Similarly, the additional U random grouping effects 

    Λ_uj^(k)(t) ≈∑_m = 1^M_uρ_ujmψ_um^(k)(t),    k=1,...,K,

are also represented using finite kl expansions with individual truncation orders M_u.

While the latent processes in _i could also be represented using rich basis function representations giving functional random effects similar in structure to (<ref>), the finite kl expansion has several advantages. The leading eigenfunctions ψ_0m(t) capture the directions of most variation in the random process, which not only generates insight into the variation of the functional data and provides interpretable basis functions, but also allows to intuitively quantify the goodness of fit of the kl approximation via the ratio of explained variation. In fact, the finite kl expansion can be seen as an optimal rank M_0 approximation with respect to the integrated squared error <cit.>. The finite kl expansion also constitutes a parsimonious basis representation as the eigenfunctions intrinsically model the dependency between the K different outcomes. This drastically reduces the number of parameters to estimate given that the scalar scores ρ_0im serve as weights for multivariate basis functions and are therefore not dimension-specific. Furthermore, the kl theorem <cit.> allows the assumption of independent, or for non-Gaussian random processes at least uncorrelated, random scores. Using an alternative basis representation (<ref>) would require the estimation of ∑_k=1^Kd_1l^(k) correlated random basis coefficients and their (co)variances, which is clearly more computationally expensive than estimating the (typically much) lower number M of independent random scores, i.e. with diagonal covariance matrix. Note that in practice, however, the eigenfunctions are generally unknown and have to be estimated as well.






§ ESTIMATION APPROACH



We simplify the estimation of covariate and latent effects in model (<ref>) and (<ref>) by applying a two-step approach, where the first step estimates the eigenfunctions using mfpca and with these as a basis, the second step estimates the covariate effects and random scores. Few approaches exist which use a kl expansion for the dependency structure and simultaneously estimate both eigenfunctions and scores in a reduced rank model, e.g. <cit.>. However, they do not allow multivariate non-Gaussian functional data with potentially mixed types and are computationally intensive. On the other hand, previous work such as <cit.>, and <cit.> has shown that estimating the multivariate eigenfunctions in a preliminary step is a practicable way to obtain a parsimonious basis representation replacing other common basis choices such as splines, yielding good results in fitting even non-standard multivariate functional data.

We emphasize the modular nature of our approach: Estimating the mfpc basis for the latent processes is detached from the second step of estimating the generalized functional additive model. This allows the user to select and combine suitable fpca methods from a wide toolbox, which can be subsequently updated as new methods are proposed in the literature. In particular, while direct estimation of mfpc has been proposed <cit.> and could be used in the first step, we exploit the flexibility of the mfpca approach proposed by <cit.>. They estimate the mfpc based on univariate scores and eigenfunctions obtained via univariate fpca. The specific estimation methods used for the univariate fpca, however, are a matter of choice and can differ between the different dimensions with different distributions, thus further modularizing the estimation process. This has the advantage that advances made for univariate (generalized) fpca can be directly transferred to the estimation of mfpc such as improvements of computational efficiency or accounting for multilevel structure, e.g. <cit.>. Consequently, we present a schematic outline of the estimation of the eigenfunctions in the following, while Sections <ref> and <ref> will offer more detailed accounts of the methods applied in the simulations and the application.



 §.§ Estimation of Eigenfunctions


<cit.> establish a relationship between the finite multivariate kl representation of Λ_0i(t) given by (<ref>) and the finite univariate kl representations Λ_0i^(k)(t) = ∑_m' = 1^M_0^kξ_0im'^(k)ϕ_0m'^(k)(t) for all k = 1,...,K, where ξ_0im'^(k) are the univariate scores and ϕ_0m'^(k)(t) are the univariate eigenfunctions of the auto-covariance operator Γ_0^(k) with auto-covariance kernel 𝒦_0^(k,k)(t, ·), allowing the representation Cov(Λ_0i^(k)(s), Λ_0i^(k)(t)) = ∑_m'=1^∞υ_0m'^(k)ϕ_0m'^(k)(s)ϕ_0m'^(k)(t), s,t ∈ℐ (Mercer's theorem) with υ_0m'^(k) the corresponding univariate eigenvalues. Note that the univariate eigenfunctions of dimension k are orthonormal with respect to the univariate scalar product ⟨·,·⟩ and the multivariate eigenfunctions are orthonormal with respect to ⟨⟨·,·⟩⟩. This relationship between univariate and multivariate kl representation then allows to estimate the multivariate eigenfunctions ψ_0m(t), m = 1,...,M based on the K sets of univariate eigenfunctions ϕ_0m'^(k)(t), m' = 1,...,M_0^k, using an explicit formula in <cit.> and the covariance between the univariate scores ξ_0im'^(k). <cit.> have shown that this approach can also be applied to each mutually independent latent process in the set _i.



  §.§.§ Estimation of Univariate Eigenfunctions

Separate univariate (generalized) fpca on each dimension offer estimates of the univariate eigenfunctions, which we denote as univariate fpc ϕ̂_m'^(k)(t). A wide range of different approaches for univariate (generalized) fpca exist, which are applicable for different distributional assumptions and study designs. We focus here on a branch of research that provides some of the most general, computationally efficient, and well implemented fpca approaches currently available. In particular,
<cit.> provide a computationally efficient eigendecomposition of the covariance matrix of Gaussian functional data observed on a regular grid, with <cit.> extending the approach to multilevel data for a nested study design (functional observations as repeated visits of independent subjects). For this special case of sampling regimes contained in (<ref>), <cit.> derive estimates for the between-subject covariance 𝒦_1^(k,k)(t, ·) and the within-subject covariance 𝒦_0^(k,k)(t, ·), respectively, and apply the fast covariance estimation approach of <cit.> for computationally efficient estimates of the corresponding univariate eigenfunctions. <cit.> and <cit.> generalize these fast fpca approaches to non-Gaussian functional data by first binning the data and estimating scalar generalized mixed models for each bin. The estimated random effects are then treated as noisy and discretized realizations of the latent processes and serve as starting points for the fast fpca approaches by <cit.> and <cit.>. Binning the data has the additional advantage that a common grid of scalar observation points is created, which allows the handling of sparse and irregularly sampled functional data. On the other hand, <cit.> and <cit.> point out that the resulting score estimates are biased due to the binning process. As a result, they propose to re-estimate the scores in a global generalized functional mixed model over the entire functional domain using the estimated fpc as basis functions. Adjusting for covariate information in this estimation approach of the univariate eigenfunctions is straightforward, as additive effects analogous to the multivariate functional regression models can be included in both the scalar generalized mixed models for each bin, dropping potential dependency on t in these pointwise models, as well as the global generalized functional mixed models.




  §.§.§ Estimation of Multivariate Eigenfunctions


With estimates of the univariate fpc and scores given, the multivariate eigenfunctions and eigenvalues can be estimated using the (weighted) covariance between the univariate scores. Let Ξ_0 be an n × M_0^+ matrix with rows (ξ_0i1^(1), …, ξ_0iM_0^(1)^(1), …, ξ_0i1^(K), …, ξ_0iM_0^(K)^(K)) containing the univariate scores of observation i, where M_0^+ = ∑_k=1^KM_0^(k) is the total number of estimated univariate fpc. Note that the number of rows n of Ξ_0 is replaced by the number of grouping levels for the mfpca of other latent processes in _i. We perform a matrix eigenanalysis for the matrix (n-1)^-1DΞ_0^⊤Ξ_0D with multivariate scalar product weights contained in the diagonal matrix D = diag(w_1^1/2,...,w_K^1/2) and w_k = (w_k, ... ,w_k) ∈ℝ^M_0^(k) to obtain eigenvectors ĉ_0m with corresponding eigenvalues ν̂_0m, m = 1,..., M_0^+. The multivariate eigenfunctions are then estimated as linear combination ψ̂_0m^(k)(t) = ∑_m' = 1^M_0^(k)[ĉ_0m]_m'ϕ̂_0m'^(k)(t), where [ĉ_0m]_m' denotes the m'-th element of eigenvector ĉ_0m <cit.>.

Given that the number M_0^+ of mfpc that can be estimated using the approach by <cit.> directly depends on the number of univariate fpc, we suggest to preserve as much information as possible in the univariate fpca and choose high univariate truncation orders (if necessary) accordingly <cit.>. Furthermore, we recommend to consider the estimated univariate eigenvalues before perfoming the mfpca. The sum ∑_m = 1^M^(k)υ_0m^(k) provides an estimate of the variation of the respective process on each dimension and thus reflects the contribution of the dimension to the overall variation in the data. When these contributions differ greatly between the dimensions, the leading mfpc can be dominated by single dimensions, capturing little of the covariance between the dimensions. If the full set of estimated mfpc is used as a basis, this is inconsequential as we are not primarily concerned with the interpretation of the mfpc. However, if the basis is truncated for a more parsimonious model, such imbalances might lead to considerably poorer model fits for some dimensions. An alternative to standardizing the data before the analysis is to use unequal weights in the definition of the multivariate scalar product <cit.>. We propose to use the inverse sum of univariate eigenvalues as weights to increase the contributions of dimensions with lower overall variation in the estimation of the mfpc. This weighting scheme can also be applied to each random process separately, giving U + 1 differently defined weighted scalar products and process-specific diagonal weight matrices D_u, u = 0,..., U. We argue that, although this complicates the comparison of eigenfunctions across processes, the main focus is to construct an empirical, parsimonious basis which represents the variation in different dimensions equally well.

As the resulting mfpc are ordered with respect to the (weighted) covariance they explain, we can additionally introduce a truncation order M < M^+ to decrease the number of scores to be estimated in the full model. This ensures that the main modes of variation are accounted for in the model but noisier, higher order modes of variation are excluded, offering an avenue for regularization of the model fit as well as increased computational efficiency. Different approaches are possible such as defining an elbow-criterion for the eigenvalues or using the amount of explained total variation in the data to choose the truncation order. For example, <cit.> propose to use a truncation order that provides a prespecified amount of univariate variation explained on each dimension or, alternatively, truncating based on the amount of total multivariate variation explained, accumulating the leading mfpc across all U + 1 processes. In extreme cases, this can lead to considerable model simplifications if no mfpc is included for a process, dropping the entire process from model. To avoid such a behavior and assure that all random processes are adequately represented in the model, we instead propose to keep a prespecified amount of total multivariate variation explained based on the estimated ν̂_um for each process u separately, leading to process specific truncation orders M_u, u = 0,...,U.




 §.§ Estimation of the Generalized Functional Additive Model

 
For the multivariate functional regression model, we apply a Bayesian estimation framework, where the estimated mfpc are treated as empirical basis functions to parsimoniously represent the functional random processes. This can be understood as an empirical Bayes approach with prior information about the covariance between different dimensions and the functional form of the random processes estimated from the data. We follow the famm framework <cit.> in using basis representations to model the observed functional data points in a scalar regression model. This allows to formalize a wide array of assumptions about functional form and smoothness of covariate effects specified via penalty matrices in normal priors. Our approach extends <cit.>, who have previously shown that the (generalized) famm framework can handle univariate generalized functional data to multivariate generalized functional data. In the following, we provide a short introduction to the famm framework for the basis representation of functional effects, as well as an introduction to the bamlss framework for the Bayesian model fitting approach used for estimation.



  §.§.§ Basis Representation of Effects



<cit.> have shown that it is possible to represent linear and smooth (functional) covariate effects of scalar and functional covariates using tensor products of appropriate bases. Their framework allows functional data to be observed at function- and here also dimension-specific time points t_i^(k) = 1,..., N_i^(k) so that irregular scalar observations of the functional data can be combined in matrix notation. Let y = (y^(1)⊤, ..., y^(K)⊤)^⊤ with y^(k) = (y_1^(k), ..., y_n^(k))^⊤ and y_i^(k) = (y_i1^(k), ..., y_iN_i^(k)^(k))^⊤ contain the scalar observations of y(t) at these time points, and denote t_i^(k) = (t_i1^(k), ..., t_iN_i^(k)^(k))^⊤, t^(k) = (t_1^(k)⊤, ..., t_n^(k)⊤)^⊤ of length N^(k) = ∑_i= 1^nN_i^(k), and t = (t^(1)⊤, ..., t^(K)⊤)^⊤ of length N = ∑_k= 1^KN^(k). Further define the row tensor product ⊙ of s × v matrix V and s × w matrix W as the s× vw matrix V⊙W = (V⊗1_w^⊤) · (1_v^⊤⊗W) with 1_c a c-vector of ones, the Kronecker product ⊗, and · element-wise multiplication. Then, all of the proposed flexible covariate effects can be represented in the matrix notation of (<ref>), now for all observations in y, as

    f_rl^(k)(𝒳, t^(k)) = Bβ_rl^(k) = (B_rlx⊙B_rlt)β_rl^(k),
 
where f_rl^(k)(𝒳, t^(k)) is the N^(k) vector of covariate effects of the respective covariate subset of 𝒳 = ⋃_i = 1^n𝒳_i at time points t^(k) for distributional parameter r = 1,..., R^(k) and covariate effect term l = 1,..., L_r^(k). The N^(k)× d_rlx^(k) matrix B_rlx contains evaluations of a suitable marginal basis for the respective covariate subset, while the N^(k)× d_rlt^(k) matrix B_rlt contains evaluations of a suitable marginal basis over the scalar observation times in t^(k). For a time-constant covariate effect, for example, B_rlt = 1_N^(k) or, for a smooth effect over time, it might contain an evaluated spline basis of d_rlt^(k) basis functions. Similarly, a linear covariate effect of a single covariate x ∈𝒳 can be represented using the corresponding covariate vector B_rlx = x, while a spline basis allows to model the covariate effect smoothly over different values of x. This basis approach supports a variety of effects such as smooth, spatial, time-varying or random effects terms. Note that tensor products can also be extended to higher dimensions with more than two margins as well as functional covariates, but we restrict this introduction to the more common two-dimensional and scalar cases. The tensor product representation (<ref>) is thus very flexible with <cit.> providing a more detailed overview of available modeling options.

Note that the flexibility of the estimated effects depends on the number of specified marginal basis functions d_rlx^(k) and d_rlt^(k). We, therefore, apply a Bayesian P-spline approach by using a large number of marginal B-spline basis functions and imposing random walk priors on β_rl^(k) <cit.>. This corresponds to a penalized estimation of smooth effects using difference penalties for adjacent B-spline coefficients in a likelihood framework. For the tensor product effects, the penalty term can be constructed from the marginal penalty matrices P̃_rlx and P̃_rlt associated with each marginal basis as

    β_rl^(k)⊤(1/τ_rlx^(k)2P̃_rlx⊗I_d_rlt^(k) + 1/τ_rlt^(k)2I_d_rlx^(k)⊗P̃_rlt)β_rl^(k) = β_rl^(k)⊤(1/τ_rlx^(k)2P_rlx + 1/τ_rlt^(k)2P_rlt)β_rl^(k)

with I_c the c× c identity matrix <cit.>.  The terms (τ_rlx^(k)2)^-1 and (τ_rlt^(k)2)^-1 can be interpreted as smoothing parameters and allow different degrees of smoothness in the marginal directions, that is, the covariate direction of 𝒳 and the functional direction of t. We point out that this construction can also be applied to linear covariate effects or time-constant effects by setting the respective penalty matrices P_rlx or P_rlt to a matrix of zeros and dropping the corresponding smoothing parameter from the model. As an analogy to the quadratic penalty (<ref>), we assume equivalent partially improper Gaussian priors for β_rl^(k) with variance parameters τ_lx^(k)2 and τ_rlt^(k)2 as described in more detail in Section <ref>.

In order to adequately represent the latent processes using tensor products, we slightly adapt (<ref>) to account for the inherently multidimensional nature of the mfpc, as the scores affect all K dimensions simultaneously. In particular, let ρ_0 = (ρ_0(1)^⊤, ..., ρ_0(n)^⊤)^⊤ contain all observation-specific scores ρ_0(i) = (ρ_0i1, ..., ρ_0iM_0)^⊤ of the finite kl expansion (<ref>) of Λ_0i(t), which accounts for the dependency within and between all functional observations i=1,...,n. The corresponding N-vector Λ_0(t) subsumes the evaluations of all n independent copies of Λ_0i(t) over all N = ∑_k= 1^KN^(k) scalar observations in t  as 

    Λ_0(t) = Ψ_0ρ_0 = (δ_0 ⊙Ψ̃_0)ρ_0,

where the i-th column of the N× n indicator matrix δ_0 contains a 1 if the corresponding element of t is from the i-th functional observation and 0 otherwise. The N× M_0 matrix Ψ̃_0 holds the evaluations of the M_0 mfpc ψ_0m^(k)(s) for time points t_is^(k) in t for the truncated kl expansion using M_0 basis functions. Other random processes in _i accounting for further grouping structure in the data can be represented analogously using the respective M_u mfpc in Ψ̃_u and scores in ρ_u = (ρ_1u^⊤, ..., ρ_J_uu^⊤)^⊤, ρ_ju = (ρ_j1u, ..., ρ_jM_uu)^⊤, together with adapting the indicator matrix to reflect the j=1,..., J_u grouping levels of Λ_u(t), u = 1,..., U. As the scores are independent across all grouping levels j (and for Λ_0(t) for all i) as well as for all mfpc, simple random effects prior assumptions can be used (see Section <ref>).





  §.§.§ Bayesian Model Fitting



We use Newton-Raphson and derivative-based mcmc algorithms to estimate the model parameters in a Bayesian framework. As previously stated, we condition on the estimated eigenfunction basis, i.e. the set  of all M_0 + ∑_u = 1^U M_u mfpc. Further, the model specification in (<ref>) and (<ref>) assumes conditional independence of all s = 1,..., n_i^(k) scalar observations y_is^(k) of the i = 1,..., n functional observations and the k = 1,..., K different dimensions given all latent processes. We can thus specify the likelihood as

    ℒ(β, ρ_0, ρ_1,..., ρ_U|y, 𝒳, )    = ∏_i=1^n∏_k=1^K∏_s=1^n_i^(k)ℒ_iks(β, ρ_0(i), ρ_1,..., ρ_U| y_is^(k), 𝒳_i, )
        = ∏_i=1^n∏_k=1^K∏_s=1^n_i^(k) d_k(y_is^(k)|θ_i1s^(k),..., θ_iR^(k)s^(k))

with d_k the probability density function or probability mass function of the parametric distribution 𝒟^(k) with distributional parameters θ_irs^(k), r = 1,..., R^(k).

We assign partially improper Gaussian priors for all covariate effects β_rl^(k), l = 1,..., L_r^(k) with multiple variance parameters τ_rl^(k)2 = (τ_rlx^(k)2, τ_rlt^(k)2), allowing anisotropic smoothing in the directions of the marginal basis functions for the covariates in 𝒳 and the functional domain t, respectively, given as

    p(β_rl^(k)|τ_rl^(k)2) ∝| 1/τ_rlx^(k)2P_rlx + 1/τ_rlt^(k)2P_rlt |^1/2exp(-1/2β_rl^(k)⊤[1/τ_rlx^(k)2P_rlx + 1/τ_rlt^(k)2P_rlt]β_rl^(k)).

For covariate terms which have a penalty term only in one marginal direction, such as linear functional effects or (scalar) random effects terms, this prior simplifies to a multivariate normal with one variance parameter. Vague normal priors β_rl^(k)∼ N(0, 1000^2I) are used for linear or parametric terms, which are not subject to smoothing and therefore are not modeled with an associated variance parameter. To improve readability for the prior specification of the random scores, let ρ_0^(m) = (ρ_01m,..., ρ_0nm)^⊤ with corresponding variance parameter denoted as ν_0m, m = 1,..., M_0, and ν_0 = (ν_01,...,ν_0M_0)^⊤. Then, the independence assumption between the levels of a grouping layer such as the functional observations i= 1,..., n for Λ_0(t) and between the respective mfpc leads to the product of multivariate normal priors

    p(ρ_0|ν_0) = ∏_m = 1^M_0p(ρ_0^(m)|ν_0m) ∝∏_m = 1^M_0( 1/ν_0m )^n/2exp(-1/2ν_0mρ_0^(m)⊤ρ_0^(m))

and analogously for ρ_u, u = 1, ..., U of the other U independent latent processes with variance parameters ν_u = (ν_u1, ..., ν_uM_u)^⊤, where n is replaced by the number of levels J_u in the grouping layer u. Independent inverse Gamma hyperpriors IG(0.001, 0.001) are used for all variance parameters so that we obtain inverse Gamma full conditionals for the variance parameters.

For a more concise notation, let ϑ denote the vector of all model parameters, containing all coefficient vectors, score vectors and corresponding variance parameters. Then, the posterior of the model is

    p(ϑ|y, 𝒳, ) =    ℒ(β, ρ_0, ρ_1,..., ρ_U|y, 𝒳, ) ·∏_k = 1^K∏_r=1^R^(k)∏_l = 1^L_r^(k)[p(β_rl^(k)|τ_rl^(k)2) p(τ_rl^(k)2)]
         · p(ρ_0|ν_0) p(ν_0)·∏_u = 1^U[p(ρ_u|ν_u)p(ν_u)],

where p(τ_rl^(k)2), p(ν_0), and p(ν_u) are the priors of the respective variance parameters. We evaluate the posterior using derivative-based mcmc sampling, Gibbs sampling, and slice sampling as described in detail in <cit.>. The derivative-based mcmc algorithm is used for the covariate effects and scores and constructs approximate full conditionals based on second-order Taylor series expansions of the log-posterior centered at the last state of the parameters. In the h-th iteration, the algorithm draws blocks of parameters β_rl^(k)[h] from a multivariate normal proposal density with mean β_rl^(k)[h-1] - H(β_rl^(k)[h-1])^-1s(β_rl^(k)[h-1]) and covariance matrix - H(β_rl^(k)[h-1])^-1 evaluated at the previous states in iteration h-1, where s(·) denotes the score vector and H(·) the Hessian of the log-posterior. <cit.> show that applying the chain rule allows to calculate these derivatives in modular blocks using e.g. the derivatives of the inverse link functions and derivatives of the additive predictors. Note that the computation of s(ρ), H(ρ), and the corresponding quantities for the other random score vectors ρ_u is similarly straightforward, as the factorization of the likelihood (<ref>) reduces the calculation to summation over all individual contributions. Variance parameters are sampled using Gibbs sampling when the full conditionals are inverse Gamma distributions, which is not the case for anisotropic model terms. Then, no closed-form full conditional can be obtained and slice sampling is employed. In order to find appropriate starting values for the sampling algorithm, a backfitting algorithm cycling through all model terms using a Newton-Raphson procedure to optimize the log-posterior, approximating the posterior mode, is used, where the variance parameters are chosen by minimizing the corrected AIC in each updating step. 




 §.§ Implementation Details


The estimation of the regression model is implemented in the R package bamlss <cit.>, which provides a very flexible infrastructure for estimating regression models with flexible covariate effects in a Bayesian framework. The wide range of modeling possibilities presented in Section <ref> are made available by leveraging the widely used mgcv package <cit.> for matrix and penalty set-up. The derivative-based mcmc algorithm described in Section <ref> is the default sampler used in bamlss. We provide the R package gmfamm <cit.> building on bamlss, which contains a convenience function constructing a bamlss-family for any number and combination of one- or two-parameter distributional assumptions 𝒟^(k) for the (generalized) multivariate functional data. This convenience function encompasses all most commonly used one- and two-parameter distributions with available implementations in the bamlss package but can be easily extended by the user. We provide further examples of families with distributional assumptions with more than two parameters as a blueprint to construct a user-specific model. 

In order to fit the regression model, the mfpc have to be first estimated in a preliminary step by the user. The mfpc basis is then to be evaluated at the scalar observation times in t and attached as covariates to the data set, for which we also provide a convenience function. Given the modular nature of our approach, this allows the user to compare different methods of estimating the mfpc or to combine different univariate fpca approaches. Notably, the R package MFPCA <cit.> provides an implementation of the mfpca approach proposed by <cit.>. To the best of our knowledge, the univariate fpca methods used in this manuscript do not have an R package implementation available but we provide the code to reproduce all simulations and application results in the gmfamm package.





§ SIMULATION



We conduct a simulation study to evaluate the impact of the sampling regime of the functional data, the quality of the mfpc estimation, and the truncation order on the model fit. Additionally, the multivariate modeling approach is compared to univariate (generalized) functional models using a similar principal component based modeling strategy. Note that while the proposed generalized multivariate functional regression models can include several different flexible covariate effect specifications, the focus of our simulation study is on the implications of the multivariate modeling approach rather than on the estimation of nonlinear coefficient effects, which have already been evaluated extensively for the famm framework in previous simulation studies <cit.>



 §.§ Simulation Design



We simulate 200 data sets containing n = 150 multivariate functions Y_i(t) with t∈ [0,1] and K = 3, which follow the binomial, Poisson, and Gaussian distributional assumptions

    Y_it^(1)|𝒳_i, Λ_0i∼Bin(1, θ_i1^(1)(t) = (1 + exp(-η_i1^(1)(t)))^-1),
       
    Y_it^(2)|𝒳_i, Λ_0i∼Poi(θ_i1^(2)(t) = exp(η_i1^(2)(t))),
    
    Y_it^(3)|𝒳_i, Λ_0i∼ N(θ_i1^(3)(t) = η_i1^(3)(t), (θ_i2^(3)(t))^2 = exp(η_i2^(3)(t))^2)

with iid 𝒳_i = {x_i, z_i}, x_i ∼ Unif(-1, 1) uniformly and z_i ∼ Bin(1, 0.5) binomially distributed. To facilitate direct comparison of estimation quality across the different dimensions, we include the same covariate effects in the additive predictors, which are specified as 

    η_i1^(k)(t) = β_0^(k)(t) + β_1^(k)(t)· x_i + Λ_0i^(k)(t) = cos(2π t) - cos(2π t)· x_i + ∑_m=1^6ρ_0imψ_0m^(k)(t),    k = 1,2,3
    η_i2^(3)(t) = γ_0 +  γ_1· z_i = -2 + 0.5 · z_i

with iid ρ_0im∼ N(0, ν_0m), linearly decreasing ν_0m = (M_0 +1 - m) / M_0, and multivariate eigenfunctions created by splitting each of the first M_0 = 6 Fourier basis functions into three parts and translating and randomly reflecting them as described in detail in <cit.>. Figure <ref> in Appendix <ref> shows the six multivariate eigenfunctions used in the simulation study.

Impact of sampling regime: We compare three different sampling regimes for the multivariate functional data to evaluate differences in accuracy of the model fits. For each simulated data set, we draw scalar observations of the functional data on a fine equidistant grid d of length 101 and create three versions of the functional data set by subsampling to mimic sparsely, regularly, and irregularly sampled functional data. For the sparse sampling regime, the numbers N_i^(k) of scalar observations on each dimension k is drawn independently from {1,2,...,10} with equal probability. Then, the vector of observation times t_i^(k) is subsampled similarly from the full grid d. The regularly sampled functional data set contains only scalar observations on the grid {0, 0.1,...,1}. The sampling for the irregular regime is analoguous to the sparse case but the number of scalar observations N_i^(k) lies between eleven and twenty. Due to this construction, the regularly sampled data sets have more scalar observations than the sparse but less than the irregular scenarios. The regularly sampled data, however, only provide information on eleven distinct time points, while both the sparsely and irregulary sampled data might better cover the entire range of the functional domain. In order to separate the model fitting process from the eigenfunction estimation in the evaluation of the impact of the sampling regime, we use the simulated eigenfunctions as mfpc in the regression models.

Impact of mfpc basis: In order to evaluate the impact of the quality of the mfpc basis, we estimate the mfpc separately for the three previously described sampling regimes and use these estimates for regression model fits on the sparse functional data setting. As mfpc estimation is typically more accurate for larger data sets, this allows to compare the regression fit for mfpc bases of different quality, while leaving the mfpc estimation approach unchanged. We apply the univariate fast generalized fpca proposed by <cit.> by first binning the data into eleven overlapping bins with equidistant centers and then fitting local scalar generalized mixed models. The local regression models use the data from the interval covering the bin center plus up to 0.3 on each side to get a local estimate of the latent process. Note that this specification of overlapping bins might induce spurious correlation in the estimated latent process but generally outperforms non-overlapping bins in simulations <cit.>. The authors do not provide a guideline for choosing e.g. the bin width, especially not for data with less than 1000 scalar observations per function. The parameters are thus specified so that a sufficient number of observations can contribute at each bin center to the univariate fpca. We apply the bamlss framework to fit the univariate binomial, Poisson, and Gaussian models, which include a random intercept as well as a fixed intercept and a linear covariate effect for x_i. The univariate Gaussian models also include a linear effect for z_i in the additive predictor for the scale parameter. For each of these 3·11 scalar models, we generate 1000 derivate-based mcmc samples as in Section <ref> and use the posterior means of the estimated random intercepts as input for the fast fpca of <cit.>. The proportion of variance explained, which is used to determine the number of univariate fpc to estimate, is set to 0.99 and the maximum possible number of third order B-splines (seven) is used for the efficient covariance smoothing. The resulting univariate fpc can then be directly passed on as input to the mfpca, but the univariate scores have to be re-estimated to avoid the bias introduced by the binning procedure. We thus refit the univariate models on the entire functional domain within bamlss using the estimated univariate fpc as basis functions, generating 1000 mcmc samples after a burn-in of 1000 samples and a thinning of five. The posterior score means are then used to estimate the mfpca, and the multivariate functional regression models in these scenarios include all available estimated mfpc. Note that parallelizing the estimation of the mfpc basis offers an avenue to reducing the computation times, in particular as the univariate fpca are calculated separately and, within one dimension, the discretized models are estimated independently. 

Impact of mfpc truncation: Truncating the mfpc basis results in a more parsimonious model but might affect the estimation accuracy of each dimension differently. We therefore compare the model containing all mfpc estimated from the sparse data scenario with a model truncated at 98% of total multivariate variance explained for the sparse data scenario. As a further alternative, we calculate the multivariate functional regression model using the estimates of a mfpca based on a weighted scalar product with an analogous truncation criterion. Here, the inverse of the sum of univariate eigenvalues ∑_m =1^M^(k)υ̂_0m^(k) is used as weight w_k. While the sum of true univariate eigenvalues in our simulation is similar across dimensions (≈ 1.16 for dimensions 1 and 3 and ≈ 1.18 for dimension 2), the univariate fpca can yield larger discrepancies in the estimated eigenvalues. Especially for the binomial functional data, we expect a less accurate estimation of the mfpc basis, as binary data inherently carries less information about the latent process than continuous outcomes. Consequently, we anticipate different accuracies when using different weighting schemes for the mfpca.

Univariate vs. multivariate: Finally, we compare univariate functional regression models in the bamlss framework to our proposed multivariate regression model on the sparsely sampled data. To isolate the effect of the multivariate modeling, both the univariate and the multivariate models estimate random scores for the true simulated eigenfunctions. For similar reasons to above, we expect the accuracy of the binomial model to benefit the most from the multivariate modeling approach.

Evaluation: For all functional regression models, we generate 1000 mcmc samples after a burn-in of 1000 and a thinning of five. To evaluate the model estimation of the generic model term f(t)∈ℒ_K^2(ℐ), we calculate the univariate rrmse over all true multivariate functions f_i(t)∈ℒ_K^2(ℐ) with levels i = 1,..., n and their estimates f̂_i(t), which we define as 

    rrMSE(f̂_i^(k)) = √(1/n∑_i=1^n||f_i^(k) - f̂_i^(k)||^2/1/n∑_i=1^n||f_i^(k)||^2)

with squared univariate norm ||·||^2. In particular, we calculate the rrmse in each simulated data set for the n = 150 fitted additive predictors η_i1^(k)(t), the latent processes Λ_0i^(k)(t), as well as the fitted covariate effect functions β_0^(k)(t) and β_1^(k)(t), for which n = 1 and the index i is dropped. Additionally, we evaluate the 95% frequentist coverage FC(f̂_i^(k), t) = 1/n∑_i= 1^n I(f̂_it^(k), 2.5≤ f_it^(k)≤f̂_it^(k), 97.5) with indicator function I, f_it^(k) the evaluation of f_i^(k)(t) at point t, and f̂_it^(k), α the α% quantile of mcmc samples at evaluation point t. This pointwise coverage criterion is then averaged over the 200 simulated data sets per t. The estimation of the coefficients γ_0 and γ_1 is evaluated using scalar definitions of bias, mse, and frequentist coverage given in Appendix <ref>. To isolate and quantify the differences in goodness of eigenfunction estimation, we evaluate the estimated mfpc bases by calculating the rrmse for the reconstructed latent processes Λ̂_0i^(k)(t) = ∑_m = 1^M_0^+ρ̂_0imψ̂_0m^(k) (t) outside of the functional regression framework, where we use the method of least-squares to estimate the ρ̂_0im from the true simulated Λ_0i^(k)(t)  using the previously estimated basis functions ψ̂_0m, m = 1,..., M_0^+.



 §.§ Results





Impact of sampling regime: When the true eigenfunctions are used as basisfunctions in the regression model, it is apparent that the multivariate models do not fit the different dimensions equally well. Table <ref> contains the mean rrmse values of the different evaluated model components, different dimensions, and different simulation scenarios with the first block <ref> comparing the sampling regimes. For the overall fitted additive predictors η_i1^(k)(t), the latent process Λ_0i^(k)(t), and both estimated coefficient effect functions β_0^(k)(t) and β_1^(k)(t), we find that the rrmse values decrease from the binomial (dimension one) to the Poisson (dimension two) to the Gaussian data (dimension three) in all sampling regimes. Given that, per construction, the dimensions exhibit similar amounts of variation and we find a coinciding trend for the estimated univariate models (last block <ref> of Table <ref>), we can conclude that this is due to continuous data apparently carrying more information about the latent trajectory than binary realizations. 

With more scalar observation points available when moving from the sparse to the regular and irregular settings, we see that generally, the estimation accuracy of all model components increases. Notably, the largest reduction in rrmse values can be observed between the sparse and regular settings, indicating that spreading the scalar observations over the entire functional domain for each observation might be the driving factor for this improvement, rather than the mere number of scalar observations. Only for β_1^(k)(t) there is a slight increase in rrmse values on the normal dimension between the regular and irregular settings; Figure <ref> in Appendix <ref> shows, however, that the overall distributions of rrmse values of this model component show little difference between these two settings and the improvement is likely due to sampling variation. 

Overall, the models fit the data well, see for example the exemplary fitted sparse trajectories in Figure <ref> in Appendix <ref>, and the functional shape of the coefficient effect functions is well estimated (compare Figure <ref> in Appendix <ref>). We find higher variance in the coefficient effect function estimates for the Poisson and even higher for the binomial dimensions, which is mirrored in the corresponding credible intervals, which are generally wide enough to achieve good pointwise frequentist coverage (see Figure <ref> in Appendix <ref>). Only for the functional intercept β_0^(k)(t), the 95% credible intervals can be too narrow (see Table <ref> in Appendix <ref>), especially for the normal dimension, and thus show a low pointwise frequentist coverage. Note that in general, Bayesian credible intervals do not necessarily have good frequentist coverage <cit.>. The computation of one regression model takes on average 80 minutes on a Linux system with 12 cores at 3.5 GHz and 32 GB memory for the sparse, 100 minutes for the regular, and 160 minutes for the irregular sampling regime.

Impact of mfpc basis: Estimating the mfpc basis for the different sampling regimes takes additional 60 minutes (on average, without parallelization) with most of the time spent on the univariate fpca, especially refitting the scores, and the mfpca calculated in under 10 seconds. The rrmse values for the reconstructed latent process confirm our assumption that more scalar observation points improve the estimation of the mfpc basis, as the mfpc basis estimated from the irregular sampling regime yields the best approximation to the true latent processes as measured by the rrmse values of the reconstructed latent process (compare Table <ref> and Figure <ref> in Appendix <ref>). In particular, in the irregular scenario, there is little difference between the quality of the mfpc basis between the different dimensions. Overall, we find clusters for the rrmse values of the reconstructed latent process corresponding to the number of univariate fpc chosen by the fast fpca approach by <cit.> based on the proportion of variance explained (compare Figure <ref> in Appendix <ref>). We see a tendency that more univariate fpc are associated with lower rrmse values but more research is needed to establish reliable guidelines on how to apply the binning approach of <cit.> to univariate functional data in different scenarios. 

The multivariate functional regression models then contain all estimated mfpc, resulting in, on average, nine included basis functions for the models using the estimates from the sparsely and regularly sampled data and ten for the irregularly sampled. This increases the computation times to e.g. on average 110 minutes for nine mfpc on the sparsely sampled data. Using an estimated eigenfunction basis in turn decreases the model accuracy as given by the rrmse values (compare the first row of the second block <ref> of Table <ref> with the bottom three rows). The models, however, still fit the data reasonably well (see Figure <ref> in Appendix <ref>) in this challenging setting with sometimes very few data points per (generalized) function and the increase in rrmse values for the coefficient effects is hardly noticeable in the coefficient effect estimates (compare Figure <ref> in Appendix <ref>). We find that a more accurate mfpc basis leads to considerably lower rrmse values of the additive predictors η_i1^(k)(t) and the latent process Λ_0i^(k)(t) as reported in Table <ref>, while the coefficient effect function estimates are much more robust: The distributions of rrmse values for β_0^(k)(t) and β_0^(k)(t) are very similar and differences between the true and differently estimated eigenfunction bases are small (compare Figure <ref> in Appendix <ref>). Given that the models do not explicitly account for the additional uncertainty in estimating the empirical basis and the estimation accuracy for the overall additive predictors and the latent process drops, it is not surprising that the frequentist coverage of η_i1^(k)(t) and Λ_0i^(k)(t) decreases, particularly for the binary data with mfpc estimated from the sparse sampling scenario, where the pointwise frequentist coverage can be as low as 60% (see Figure <ref> in Appendix <ref>). We again emphasize that this is a very challenging scenario and the frequentist coverage is ten to twenty percent higher for the mfpc bases from the other sampling regimes. The frequentist coverage of the coefficient effect functions, however, is similar to the coverage in the models using the true eigenfunction basis.

Impact of mfpc truncation: By excluding trailing mfpc from the regression model, we reduce the flexibility of the model and introduce a further level of regularization on the noisiest components of the covariance structure. This stabilizes the model estimation as fewer parameters have to be estimated and it has a negligible effect on the estimation accuracy of the examined model components as given in the third block <ref> of Table <ref>. Using a truncation order of 98% explained multivariate variance based on a scalar product with equal weights results in models containing between five and eight (median six) estimated mfpc. For all model components, this increases the mean rrmse values slightly but the overall distributions as well as the pointwise frequentist coverage are very similar to the models without truncation (see Figures <ref> and <ref> in Appendix <ref>).

On the other hand, using a truncated mfpc basis estimated from an mfpca with a weighted scalar product puts a high weight on the first dimension (mean 1.36) and low weights on the second (mean 0.15) and third (mean 0.14) dimensions in our simulation scenario. This focus on the binary dimension leads to lower rrmse values of the additive predictor η_i1^(k)(t) and the latent process Λ_0i^(k)(t) for the binary functional data compared to the count and Gaussian functional data as indicated by Table <ref>. Note that the weights are chosen to be the inverse of estimates of the total amount of variation on each dimension. As the true sums of eigenvalues are much more similar than the obtained estimates, such an extreme upweighting of dimension one is not beneficial for the analysis and constitutes a considerable overall deterioration of the estimation accuracy and frequentist coverage for the additive predictor and the latent process. The covariate effect functions, however, are again very robust and see little change in their rrmse values and frequentist coverage (see Figures <ref> and <ref> in Appendix <ref>). We thus find that using weights in the mfpca can have a considerable effect on the model estimates, improving the model fits for dimensions with high weights compared to those with lower weights, but we conclude that the binary fpca approach applied here seems to seriously underestimate the true variance of the latent process and weights should be carefully considered for plausibility before the analysis. In particular, the local scalar regression models feature a substantially lower posterior mean of the random effects variance for the binomial data (0.52) than for the Poisson (0.94) and Gaussian (0.98) data, which leads to less variation in the discretized realizations of the latent process on the binomial dimension and consequently a lower total sum of univariate eigenvalues in the approach proposed by <cit.>. This suggests that the generalized fpca approach by <cit.> is not ideal for estimating mfpc for data of mixed type where at least one dimension is binomially distributed, and substituting their method by a different approach or at least not upweighting the binomial data as strongly as the estimated variances seem to imply should be considered, when basis truncation is desired.

Multivariate vs. univariate: We note that the univariate modeling approach has a clear computational advantage: even without parallel computing of the univariate models, the total computation time is on average only 50 minutes compared to the 80 minutes for the multivariate approach. The last block <ref> in Table <ref> contains the mean rrmse values for a univariate functional regression approach on the sparse data using the true eigenfunctions. We find that the multivariate models clearly fit the additive predictor η_i1^(k)(t) and the latent process Λ_0i^(k)(t) better on all dimensions, as the covariance between the functional outcomes is incorporated into the analysis. The worse estimation of the latent process for the univariate models also negatively impacts the frequentist coverage of resulting credible intervals, leading to a considerably lower coverage than the nominal value for the univariate models, especially on the Gaussian dimension (compare Figure <ref> in Appendix <ref>). The estimated coefficient effect functions in the univariate models are overall similar to the estimates of the multivariate functions and we find only a slight tendency for better estimates of the coefficient effect functions for the multivariate approach on the binomial and Poisson dimensions: rrmse values are slightly smaller (see Fig. <ref>) and there is a slightly lager number of iterations, where the multivariate approach yields lower rrmse values for β_0^(k)(t) and β_1^(k)(t) (see Table <ref> in Appendix <ref>) for these two dimensions, with a reverse trend for the third Gaussian dimension. Only a small difference in these numbers suggests a trade-off for the multivariate approach between the first two dimensions, which are slightly better estimated compared to the univariate models, and the Gaussian dimension, which shows slightly higher rrmse values.

We thus conclude that the multivariate regression approach leads to overall better model fits and uncertainty quantification by combining the information across dimensions. This becomes especially apparent when comparing the univariate approach with true eigenfunctions to the more realistic scenario of using a multivariate regression with an estimated mfpc basis (without truncation) as even then, the multivariate analysis results in lower rrmse values for most components. The multivariate approach also leads to similar estimates of the additive predictor of the scale parameter η_i2^(3)(t), which is generally estimated well but is also slightly negatively affected by estimating the mfpc basis and potential basis truncation (see Table <ref> and Figure <ref> in Appendix <ref>). The coefficient effect function estimates for β_0^(k)(t) and β_1^(k)(t), however, are very robust over all examined scenarios, regardless of whether the true eigenfunctions are known or estimated (and truncated), or whether a multivariate or univariate analysis approach is followed. The main benefit of the multivariate models thus is the more accurate estimation of the latent process and we find that, for a reasonable number of scalar observations, our proposed mfpca approach  yields a good empirical eigenfunction basis, even for the (arguably more difficult) binomial data.




§ APPLICATION



In the following, we apply our proposed approach to modeling multivariate functional data of mixed type to new and publicly available data on traffic flows in the German capital. Over the last couple of years, the federal state of Berlin has taken several measures to reform its concept of urban mobility, such as the https://www.berlin.de/sen/uvk/en/mobility-and-transport/transport-policy/berlin-mobility-act/Berlin Mobility Act in 2018, in order to reduce emissions and traffic fatalities. As part of data-based policy making, Berlin uses infrared detectors to collect data on the number and mean speed of vehicles passing more than 240 sites at main roads spread out over the municipal area. The provided data collection (<https://daten.berlin.de/datensaetze/verkehrsdetektion-berlin>) contains hourly information for each location starting from January 2015. With four potential outcomes of interest given by the hourly number and mean speed of cars and trucks, this constitutes a rich and interesting data base, which, to the best of our knowledge, has seen little scientific attention: <cit.> restrict their analysis to one available site and follow a scalar time-series approach using lagged values and scalar copula regression. Instead, we view the outcomes as number and mean speed of vehicles over the last hour (t-1, t], allowing the interpretation as functional data over the course of one day with t ∈ [0, 24]. Adopting this view of the data, we have four dimensional multivariate functional data with two functional count outcomes (absolute number of cars, denoted as qCar, and number of trucks, qTruck) and two continuous and positive-valued functions (mean speed of cars in km/h, vCar, and trucks, vTruck).



 §.§ Data Set





In our analysis, we are interested in modeling the traffic flow of vehicles into the city center of Berlin given as the area within the Berlin S-Bahn (rapid transit system) circle line. We therefore manually select all available traffic detection sites outside of the Berlin S-Bahn circle line which are placed on streets leading towards the city center. A map of the 45 selected locations can be found in Figure <ref>. We include data collected from 2015 up to 2023 in our analysis. For this first analysis, in order to simplify the data structure, reduce the autocorrelation of the multivariate functional observations, and remove seasonality, we focus on the traffic on the first Monday of each September. This removes weekday and holiday effects as well as seasonal trends from the analysis and results in reasonable computing times. Figure <ref> shows the observed multivariate functional data for one randomly selected site over the different years.

We visually inspect the entire time-series of the sites to spot potential detector failures and consequently remove time intervals for selected sites where failure is apparent (see Figure <ref> in Appendix <ref> for an example). Such intervals, ranging from one multivariate observation to consecutive observations for multiple years, can be identified for 8 sites. The final data set then contains 354 multivariate functional observations with 8,024 scalar measurements. Note that missing values for vCar and vTruck are generated when no vehicles are measured so that only 8,022 and 7,793 scalar observations are available for these outcomes. The mean speed variables are provided as rounded integer values so we set 0 mean speed values to 0.5 (for 66 scalar observations of outcome vTruck).



 §.§ Model Specification


We assume negative binomial distributions for the functional counts, Gamma distributions for the positive-valued functional outcomes, and log-links to connect all additive predictors to the distributional parameters for all outcomes. The negative binomial distribution Y^(k')∼ NB(θ_1, θ_2), k'∈{qCar, qTruck} with 𝔼(Y^(k'))= θ_1 and Var(Y^(k')) = θ_1 + θ_1^2/θ_2 is often used to model count data as it allows to incorporate overdispersion in the analysis and has been shown to adequately fit the Berlin traffic data <cit.>. In contrast to <cit.>, who assume the symmetric Student t distribution for the marginal mean speed distributions and find “significant deviations from the distributional assumption”, we choose a Gamma distribution Y^(k”)∼ Ga(θ_1, θ_2),k”∈{vCar, vTruck} with 𝔼(Y^(k”)) = θ_1 and Var(Y^(k”)) = θ_1^2/θ_2 as this is an adequate assumption for skewed and positive data. We further assume constant scale parameters for the distributions and include covariate effects only in the location parameters. The dependency between the functional outcomes is modeled using a multilevel structure with a multivariate latent Gaussian process for the site and a nested year-specific multivariate functional random effect per site.

This gives the following additive predictors for the location parameters η_ij1^(k')(t), k' ∈{qCar, qTruck} and η_ij1^(k”)(t), k”∈{vCar, vTruck}, and the scale parameters η_ij2^(k)(t), k ∈{qCar, qTruck, vCar, vTruck}

    η_ij1^(k')(t) =     β_0^(k')(t) + f^(k')(year_j, t)+ lanes1_iβ_1^(k')(t) + lanes3_iβ_2^(k')(t) + Λ_1i^(k')(t) + Λ_0ij^(k')(t), 
    η_ij1^(k”)(t) =  
        β_0^(k”)(t) + f^(k”)(year_j, t)+ limit30_iβ_1^(k”)(t) + limit60_iβ_2^(k”)(t) +Λ_1i^(k”)(t) + Λ_0ij^(k”)(t), 
    η_ij2^(k)(t) = γ_0^(k),

with site i = 1,...,45, year j = 2015,...,2023, and continuous covariate year for the year of the observation. We also include the dummy covariates lanes1, lanes3, limit30, and limit60 indicating the number of lanes at site i, which are either one (seven sites), two (reference), or three (seven sites), and whether the speed limit of the given site, determined in February 2024, is at 30 km/h (seven sites), at 50 km/h (reference), or at 60 km/h or above (four sites). Slightly adapting the notation in (<ref>), Λ_1i^(k)(t) is the k-th dimension of the latent site-specific and Λ_0ij^(k)(t) the site-year-specific random process with iid Λ_1i(t) ∼𝒢𝒫(0(t), 𝒦_1(t, ·)) and iid Λ_0ij(t) ∼𝒢𝒫(0(t), 𝒦_0(t, ·)). Note that the specified model showcases different possible effect specifications: we include constant effects γ_0^(k), intercept functions β_0^(k)(t), categorical functional effects β_1^(k)(t) and β_2^(k)(t), as well as 
nonlinear functional effects f^(k)(year_j, t) in addition to nested multivariate functional random intercepts.



 §.§ Model Estimation


To account for the multilevel structure of the data set, we follow <cit.> to estimate the univariate generalized fpc. Their proposed method follows the binning and fast fpca approach presented in Section <ref> but allows to estimate fpc for a nested study design. A detailed description of how the approach by <cit.> is used to estimate the univariate eigenfunctions can be found in Appendix <ref>. We emphasize again, that while this approach represents a relatively fast generalized fpca for large data sets, other methods for the univariate fpc estimation might be suitable and could be easily substituted. The estimation of the univariate generalized fpca takes about 75 hours without parallelization on a Linux system with 12 cores at 3.5 GHz and 32 GB memory and results in three or six (for dimension qTruck) univariate fpc for the site-specific, and five (for dimension qTruck) or six univariate fpc for the site-year-specific random processes as given by Figure <ref> in Appendix <ref>.

We then use the posterior means of the scores in the estimation of the mfpca. In order to get holistic representations of the site-specific and site-year-specific processes, we choose unequal weights for the multivariate scalar product, which forms the basis of the mfpca. In fact, the estimated univariate eigenvalues given in Table <ref> in Appendix <ref> indicate that the amount of variation of both processes differs considerably across dimensions, with the dimension qTruck exhibiting most variation (sums of univariate eigenvalues 0.504 for the site-specific and 0.174 for the site-year-specific process) and vCar the least amount of variation (0.045 and 0.001, respectively). Using equal weights in the scalar product would ignore this and the corresponding mfpca would show leading eigenfunctions that mainly capture variation on the dimension qTruck. Given that we also find different relative contributions of the four dimensions to the total amount of variation for both processes (see Table <ref> in Appendix <ref>), we additionally decide to use different scalar products for the two multivariate Gaussian processes Λ_1i(t) and Λ_0ij(t). Consequently, we conduct the two separate mfpca with differing weights using the inverse sums of eigenvalues for each process respectively. As we use all available univariate fpc to avoid losing important information about the variation, we obtain 15 mfpc for the site-specific random process and 23 for the site-year-specific process.

When considering the eigenvalues of the estimated mfpca, it becomes apparent that for both processes, the trailing mfpc are associated with very small contributions to the variation of the random processes (see Table <ref> in Appendix <ref>). Given that for each mfpc included in the multivariate functional regression model a total of 45 site-specific and, respectively, 354 site-year-specific random scores plus variance parameter need to be estimated, we chose to introduce a truncation of the mfpc basis at 98% explained total variation of the random processes. This reduces the number of included mfpc to 9 and 14 basis functions, which implies a considerable reduction in model complexity and computation time compared to including all available mfpc.

The functional intercepts β_0^(k)(t) and the categorical functional effects β_1^(k)(t) and β_2^(k)(t)  are represented using 14 cubic P-splines while the bivariate interaction terms f^(k)(year_j, t) are modeled using anisotropic tensor product splines of 7 (for year) and 14 (for t) marginal cubic P-splines allowing different variance parameters for the different margins. These specifications lead to a model with around 5900 parameters, requiring about 16 GB of RAM for model fitting. After the 38-hour backfitting algorithm to find appropriate starting values as described in Section <ref>, we generate 2000 mcmc burn-in samples, which takes an additional 122 hours. We use the final burn-in sample as a starting point for ten separate mcmc chains, each generating 100 samples after a thinning of 10, resulting in a total of 1000 mcmc samples after generating 12000 draws from the posterior. This set-up allows to parallelize the model fitting process over several adequate platforms and reduces the computation time to 61 hours per parallel chain. 




 §.§ Results





In contrast to <cit.>, our analysis deliberately focuses on daily patterns in the data and takes the functional nature of the traffic outcomes into account. We are thus the first to describe a clear “rush hour” effect for traffic into Berlin as illustrated in Figure <ref> using predicted curves for sites with two lanes and speed limit 50, which manifests itself as a prominent peak around 7 a.m. for the number of vehicles qCar and qTruck and as a slightly shifted low point of the mean speed of vehicles vCar and vTruck around 8 a.m. Overall, we find that the estimated mean number can be more than ten times higher for cars than for trucks during this peak. Figure <ref> also shows that the amount of early morning traffic has decreased over the observed time period, i.e. the peak number of cars has declined from 2015 to 2023. For the mean number of trucks, the change over the years does not follow a similar linear trend but shows more variation, which reflects the raw data: the lowest estimated rush hour peak for outcome qTrucks is in 2018 and corresponds to the minimum when taking the simple (marginal) mean number of trucks at 7 a.m. in the data set. On the other hand, the estimated mean speed of trucks during the morning rush hour is relatively stable up until 2020 and has considerably increased since. The estimated mean speed of cars, on the other hand, exhibits smaller variation around 35 km/h over the years during the morning rush hour with the mean speed first decreasing until 2019 and increasing in later years. 

After this morning traffic, the number of vehicles slightly decreases and stabilizes around noon with a second, less pronounced peak for cars around 4 to 5 p.m.  after which the numbers for trucks and cars diminish. During the night, there is an overall low number of vehicles, which only starts to increase roughly around 3 a.m. for cars and 2 a.m. for trucks. Over the years, the functional form of the estimated mean qCar functions shows an overall downwards shift together with a flattening of the second rush hour peak. For qTruck, we see that the functional forms throughout the day vary more over the analyzed years. The nonlinear functional effect f^(k)(year_j, t) is estimated using the typical famm identifiability constraint <cit.>, which enforces a sum-to-zero constraint for all t. This allows to interpret the estimated effect as a local deviation from the global functional intercept and the credible interval of the estimated effect can be used for pointwise testing. We find that, for example at 4 p.m., the 95% credible intervals of the estimated effect for qTruck include zero over all years (without adjusting for multiple testing) and only differences in the morning hours of truck traffic would be considered statistically significant at the 5% level (see Figure <ref> in Appendix <ref>).

For the mean speed outcomes, there is an increase after the morning rush hour to a slight peak around noon for both cars and trucks. During the second peak in the number of cars at 4 p.m., we again see a slight drop in the mean speed outcomes for cars and trucks, followed by a steady increase throughout the night, reaching a high point around 4 a.m. for cars and 2 a.m. for trucks. We find that the overall change in mean speed over the course of the day is much more pronounced for cars than for trucks, as for example in 2015, the mean speed for cars ranges from about 35 km/h to 55 km/h, while the range for trucks is  roughly 28 km/h to 38 km/h. Over the years, this range has somewhat decreased for cars and increased for trucks. Additionally, for trucks, there is an almost constant shift (across t) of the estimated mean functions over the years: first slightly downwards from 2015 to 2019 followed by a stark upwards shift until 2022. For cars, the estimated mean functions for a location with two lanes and speed limit of 50 km/h show a relatively uniform downwards trend over the observed years except during the morning rush hour as described above. We thus conclude that over the last couple of years, especially car traffic has reduced and slowed down in Berlin while truck traffic might have only slightly decreased during the morning rush hour and even increased its mean speed over the entire day. For cars, at least, this development seems to be in line with the proclaimed efforts and goals of the Berlin government to reduce emissions. Note, however, that our analysis does not imply a causal effect of any particular policy measure.



The estimated coefficient effect functions for the considered covariates might also be interesting to policy makers as they can help to identify possible pathways for influencing traffic flows. Figure <ref> shows the estimated coefficient effect functions β̂_1^(k')(t) and β̂_2^(k')(t), k' ∈{qCar, qTruck} depicting the difference for sites with one or three lanes, respectively, compared to two lanes on the additive predictor scale. Keep in mind that e.g. a positive effect of β̂_1^(k')(t) on the additive predictor scale, using a log link, corresponds to a multiplicative effect on the scale of the distributional parameter by the factor expβ̂_1^(k')(t), which is then larger than one. We see that, all other things being equal, a site with one lane has considerably less traffic than a site with two lanes, corresponding to the estimated effects for both qCar and qTruck being significantly negative (compare the indicated pointwise 95% credible intervals) over the entire day. The estimated mean function for qCar is particularly decreased between 8 p.m. and 9 p.m., leading to even less cars in that hour compared to a site with two lanes. Note, however, that a credible region constructed from the pointwise credible intervals does not correctly account for the functional correlation and should not formally be used to draw conclusions about the functional form such as the linearity of the estimate. The effect estimate β̂_2^(k')(t) (bottom row in Figure <ref>) allows to assess the difference between a site with three and two lanes. We find a strong positive effect on the number of cars over the entire day, while the number of trucks sees a significantly positive effect only during daytime. For both estimated coefficient effect functions, the effect sizes are much more pronounced for cars than for trucks, suggesting that changing the number of lanes would have a much larger impact on the number of cars passing the respective site. 

Figure <ref> in Appendix <ref> shows the estimated coefficient effect functions for the covariate speed limit, which show few areas with significant effects on the estimated mean functions of vCar and vTruck, except for a significant increase of mean speed of trucks during the day for sites with a 60 km/h speed limit (or above) compared to 50 km/h. The multivariate nature of our regression approach allows to directly formulate and test joint hypotheses across dimensions, such as H_0: β̂_1^(vCar)(t) = β̂_1^(vTruck)(t) = 0 for a given t, which are helpful to assess whether a covariate might be dropped from the model altogether. Figure <ref> in Appendix <ref> demonstrates one such test for t = 12 using empirical two-dimensional credible regions based on a kernel density estimate of the mcmc samples, indicating that the null hypothesis could only be rejected at a significance level of 10% but not at the 5% level. This illustrates that the multivariate functional regression model provides a useful framework for efficiently combining information across dimensions and generating additional insights not available from univariate models.




Similarly, our multivariate approach provides empirical basis functions, which capture the multivariate functional dependency structure present in the data. Figure <ref> shows the first three estimated eigenfunctions for the site-specific process as the mean function for a site with two lanes and a speed limit of 50 km/h in 2020 (black line) plus (red +) or minus (blue -) 2√(ν̂_1m) times the estimated mfpc ψ̂_1m^(k)(t), where ν̂_1m, m = 1, 2,3 is the posterior variance of the random scores. The first leading mfpc ψ̂_11^(k)(t) accounts for about 3% of the total (unweighted) site-specific (posterior) variation and captures differences between light and heavy traffic sites, i.e. more or less vehicles together with higher or lower mean speeds, without really changing the functional form of the mean function. The second estimated mfpc ψ̂_12^(k)(t) shows a negative association between the number of vehicles and the mean speed so that e.g. a higher amount of traffic reduces the mean speed of the vehicles. This mode of site-specific variation is the second leading mode in the preliminary (weighted) mfpc estimation but shows the highest posterior variance of the site-specific mfpc basis functions, accounting for about 40% of the (posterior unweighted) variation. This illustrates that the mfpca here estimates multivariate eigenfunctions using weighted variances, upweighing dimensions with smaller variances, and ordering of mfpcs may flip when evaluating importance in terms of unweighted variances: The first mfpc mostly loads on the two speed dimensions with small absolute variation, which are upweighted in the mfpca, and thus corresponds to a relatively small amount of explained unweighted variance, while the second mfpc with large contributions to the number of vehicles dimensions is more important in these terms. The third leading mfpc again accounts for a large proportion, about 38%, of the total site-specific (posterior unweighted) variation and depicts a contrast between the types of vehicles: For example, a site with a large score on this component would feature an overall higher number of cars with lower mean speed but simultaneously fewer trucks with higher speed.

Generally speaking, the interpretation of the leading mfpc is easiest since they capture the main modes of variation in the data and all following mfpc must reflect modes orthogonal to them. For the site-year-specific random process, the mfpc accounting for the the most (posterior unweighted) variation (43%) is only the fifth, which somewhat complicates its interpretation (compare Figure <ref> in Appendix <ref>). One apparent feature of this mfpc, however, is its distinctive impact on the number of trucks late in the day, leading to a stark increase (or decrease) of traffic in the afternoon. The leading mfpc ψ̂_01^(k)(t) only accounts for 8% of the total site-year-specific (posterior unweighted) variation, which constitutes the second largest contribution to the total (posterior unweighted) variation (compare Table <ref> in Appendix <ref>). It captures a similar mode to ψ̂_11^(k)(t) in Figure <ref> with only slight differences (see Figure <ref> in Appendix <ref>). Note that a direct comparison between the processes is not straightforward as the scalar products used in the respective mfpca are different. Overall, we find that the qualitative examination of the empirical basis functions can provide first insights into the dependency structure in the data even though the proposed two-step approach produces “only” an empirical basis.

The presented model fits the data reasonably well (see Figure <ref> in Appendix <ref> for exemplary fitted curves of the location parameter for the detection site “Goerzallee”) but analyzing the normalized (randomized) quantile residuals <cit.> suggests that the model might be further improved (compare Figure <ref> in Appendix <ref> for outcome-specific worm plots <cit.>). While the presented plots seem to indicate a strong deviation from the distributional assumption, in particular for the mean speeds, we would like to point out that by estimating the mfpc, we might find the worm plots affected by e.g. residual auto-correlation. Increasing the flexibility of the model for the scale parameters is also straightforward in our modeling framework and might improve the residual diagnostics at the expense of additional computation time. Our analysis, however, has already shown that the proposed (generalized) multivariate functional regression models allows to identify daily characteristics of traffic flows and model their change over several years, as well as their dependence on different covariate effects, while incorporating information from various dimensions simultaneously. This provides policy makers with the means to better understand traffic flows in the Berlin capital, evaluate the effect of specific measures, or identify new avenues of potential traffic management.




§ DISCUSSION AND OUTLOOK



We propose a general flexible regression framework to handle generalized multivariate functional data of mixed type. In order to incorporate the covariance between and within the functional dimensions, we assume independent latent multivariate Gaussian processes and estimate their covariance functions from the data via mfpca. This allows a parsimonious representation of the latent processes and provides a direct tool for introducing further regularization in the model by truncating the multivariate kl representation of the latent processes. We further extend this approach to generalized multivariate functional data which exhibit a multilevel structure such as repeated observations for independent subjects. As the model builds on the famm framework, a wide range of covariate effects, e.g. linear, non-linear functional as well as interaction effects, can be included in the regression models. Our simulation study shows that a multivariate functional analysis approach is beneficial for fitting the functional data compared to separate univariate functional models, and provides robust estimation of covariate effects over several scenarios, including for challenging sparse functional data. Furthermore, we showcase our regression framework in an application to model traffic data in Berlin. To the best of our knowledge, this is the first time that an analysis of this complex data source has incorporated the (multivariate) functional nature of traffic flows. We are thus able to generate new insight into characteristic traffic patterns as well as changes over time or influences of additional characteristics. The analysis simultaneously models all available functional dimensions consisting of the number and mean speed of different types of vehicles and promotes the understanding of dependency structures within and between these dimensions.

While we present one avenue of estimating the mfpc basis, our two-step procedure can be easily adapted to account for new and improved mfpca strategies, as only the estimated mfpc are required for the model fit. Generally, we find that the estimation of the mfpc can have a substantial direct effect on the quality of the model fit and should thus be handled with care. In recent years, there have been some considerable developments for generalized univariate fpca, but well-thought-out large-scale comparison studies might be helpful to establish practicable guidelines and identify open issues. Simultaneous estimation of the mfpc, scores, and covariate effects in one multivariate functional regression model would arguably be more user-friendly and eliminate the implication that the resulting inference is conditional on the estimated mfpc basis. Such an approach, however, is challenging in our complex setting: it would require, for example, the extension of existing methods for univariate multilevel fpca for generalized functional data <cit.> to multivariate functional data while also supplying the powerful modeling possibilities of our flexible regression framework.

One important aspect to consider in such endeavors is providing efficient implementations of newly developed methods. Our analysis of the Berlin traffic data is restricted to a small subset of the original data base, in part due to the computational demands of the generalized multivariate functional regression model. It will therefore be of primary interest in the future to further improve the method's efficiency to handle large scale data sets by e.g. fully exploiting sparse structures in the design matrices and using more efficient programming. Improving the scalability of our regression framework is already a work in progress within a new R package. Alternatively, using approximate Bayesian methods such as Integrated Nested Laplace Approximation <cit.> or penalized frequentist likelihood approaches <cit.> present promising alternatives to our computationally quite intensive mcmc sampling algorithm. 

Improving our implementation will make the full range of our proposed framework accessible to the statistician and provide several opportunities to improve the model fit in our analysis. Including more mfpc in the application model can improve the residual diagnostics by supplying more flexibility to the location parameter. Introducing covariate effects in the additive predictors of the scale parameters further increases the flexibility of the model. Similarly, other pointwise distributional assumptions for the generalized multivariate functional data are generally available in our framework, yet our experience shows that distributions with more distributional parameters can lead to noticeably increased computation times. Further combining the available data with external data sources might also lead to a better model fit and provide deeper insight into the evolution of daily traffic patterns in Berlin. Including weather data or data about policy measures with respect to infrastructure or epidemiologically motivated sanctions might provide further relevant information to accurately model the data and interpret the estimated effects.

Throughout this paper, we assume that the latent processes are Gaussian – a common assumption, but the effects of a misspecification in this aspect are not yet fully understood. Further simulations might also focus on the impact of misspecifying the pointwise distributional assumptions of the (generalized) functional data. Despite these remaining questions and avenues for further improvements and research, we are able to considerably expand the previously available toolkit for and improve the modeling of generalized multivariate functional data of mixed types.










§ SIMULATION





 §.§ Simulation Design

Figure <ref> displays the six eigenfunctions used to simulate the generalized multivariate functional data with Figure <ref> showing nine exemplary simulated multivariate functions including scalar observations.








 §.§ Additional Results


Figure <ref> shows the rrmse values of the additive predictor η_i1^(k)(t) of the location parameter, the latent process Λ_0i^(k)(t), and the coefficient effect functions β_0^(k)(t) and β_1^(k)(t) for all examined scenarios and all 200 simulation iterations. Figure <ref> shows  exemplary model fits for a simulation iteration with rrmse values η_i1^(1)(t) = 0.556, η_i1^(2)(t) = 0.439, η_i1^(3)(t) = 0.211, which is close to the overall mean rrmse values, in the sparse sampling scenario using the true eigenfunctions. Figure <ref> shows the estimated coefficient effect functions of all multivariate regression models using the true eigenfunctions as an mfpc basis with data from the different sampling regimes. Both the overall model fits and the coefficient effect functions are well estimated. Figure <ref> shows the pointwise frequentist coverage of the Bayesian credible intervals for all model components over all examined simulation scenarios. Table <ref> shows the mean width of credible intervals for the coefficient effect functions β_0^(k)(t) and β_1^(k)(t). It is apparent that especially for β_0^(3)(t) on the Gaussian dimension, the posterior credible intervals are very narrow, leading to a low frequentist coverage of the estimates.

Table <ref> shows the mean rrmse values and their Monte Carlo error of the reconstructed latent processes for the mfpc bases estimated from different sampling scenarios. Note that this is different from the fitted latent process as for the reconstruction, the estimated basis functions are directly fit to the true latent process Λ_0i(t) using a least-squares approach. For some scenarios and dimensions, we find rather large Monte Carlo errors corresponding to wide boxplots of the rrmse values as shown in Figure <ref>. This can be attributed to the number of estimated univariate fpc, see Figure <ref>, which shows an overall tendency that a higher number of univariate fpc is associated with lower rrmse values for the reconstructed latent process Λ_0i^(k)(t). Figure <ref> shows exemplary model fits for a simulation iteration with estimated eigenfunction basis resulting in rrmse values η_i1^(1)(t) = 0.712, η_i1^(2)(t) = 0.511, η_i1^(3)(t) = 0.310, which is close to the overall mean rrmse values, in the sparsely sampled data. Figure <ref> shows the estimated coefficient effect functions of the models using the differently estimated mfpc on the sparsely sampled data. Comparing the estimates to the first row of Figure <ref> suggests that the coefficient function estimation is robust against the estimation of the mfpc basis.

Table <ref> illustrates that the univariate and multivariate modelling approach result in comparable rrmse values. The rows (columns) of the crosstables report the number of simulation iterations where the univariate models yield smaller rrmse values than the multivariate models, denoted as Uni, or vice versa, denoted as Multi, for β_0^(k)(t) (or β_1^(k)(t) for the columns). 

In order to evaluate the estimated scalar coefficient effects γ_l, l∈{0, 1} of the additive predictor η_i2^(3)(t) of the scale parameter on the Gaussian dimension, we use the bias Bias(γ̂_l) = 1/200∑_i= 1^200γ̂_l^[i] - γ_l with γ̂_l^[i] denoting the posterior mean estimate of the true value γ_l in iteration i = 1, ..., 200, the root mse rMSE(γ̂_l) = (1/200∑_i= 1^200(γ̂_l^[i] - γ_l)^2)^1/2, and the mean frequentist coverage FC(γ̂_l) = 1/200∑_i=1^200I(γ̂_l^[i], 2.5≤γ_l≤γ̂_l^[i], 97.5) with indicator function I and γ̂_l^[i], α the α% quantile of mcmc samples in iteration i. Figure <ref> shows the estimates of γ_0 and γ_1 together with the true values and Table <ref> contains the previously defined evaluation criteria for the examined simulation scenarios. We find that while the estimation of the scale parameter is good when the true eigenfunctions are used, estimating the mfpc basis introduces some bias and variance in the estimates, resulting in lower frequentist coverage.




































§ APPLICATION





 §.§ Data Preprocessing


Figure <ref> provides an example of a detector failure in the Berlin traffic data. The site “Müllerstraße” is not fully observed on the time interval and features an unrealistic jump in the outcome vCar between 2017 and 2021. As other outcomes are not affected, only this section is removed from the final data set. 



 §.§ Estimation of Univariate Generalized FPCs



We follow the outlined approach by <cit.> to estimate the univariate generalized fpc and adapt it to the application at hand. In the first step, the data is grouped into s = 1,..., 24 overlapping bins with one bin containing the measurement data t_ij∈ [s-2, s+2]. Note that we use data over the ends of the domains, e.g. the first bin contains all observations between 11 p.m. and 3 a.m., as the traffic data are of a cyclic nature. For each of these bins, four univariate bamlss regression models are estimated with the distributional assumption analoguous to the multivariate functional data. These models ignore the functional nature of the data and estimate site-specific and site-year-specific (scalar) random intercepts while accounting for the covariate effects. This results in additive predictors for the respective subset of data as

    η_ij1^s,k'(t) =  
        β_0^s,k' + f^s, k'(year_j) + lanes1_iβ_1^s,k' + lanes3_iβ_2^s,k' + b_1i^s, k' + b_0ij^s, k', 
    η_ij1^s,k”(t) =  β_0^s,k” + f^s, k'(year_j) + limit30_iβ_1^s, k” + limit60_iβ_1^s, k” + b_1i^s, k' + b_0ij^s, k'
    
         t = s-2,..., s+2,    k'∈{qCar, qTruck},   k”∈{vCar, vTruck},

where the random intercepts are denoted as b_1i^s,k and b_0ij^s,k. The models are estimated with 1000 mcmc samples after a burn-in of 200 using seven cubic P-spline basis functions with second order difference penalty for the nonlinear effect of year. Estimating one such univariate regression model takes five to ten minutes on a Linux system with 12 cores at 3.5 GHz and 32 GB memory, which amounts to 11 hours total without parallelization.

The estimated random intercepts b̂_1i^s,k and b̂_0ij^s,k, k ∈{qCar, qTruck, vCar, vTruck} are then used in the fast multilevel fpca method proposed by <cit.> to obtain smooth estimates of the univariate eigenfunctions ϕ_1m^(k)(t) and ϕ_0m^(k) of the site-specific and site-year-specific processes, taking only a few seconds. For the smooth estimation of eigenfunctions, we choose a cyclic cubic P-spline basis with second order difference penalty and ten knots, as we find a larger number of basis functions to yield undersmoothed results <cit.>. The number of eigenfunctions M_1^k and M_0^k estimated by the fast fpca approach is determined based on the cumulative percentage of variance explained with a threshold of 0.99. For the site-specific random process, (M_1^qCar, M_1^qTruck, M_1^vCar, M_1^vTruck) = (3, 6, 3, 3), and for the site-year-specific random process  (M_0^qCar, M_0^qTruck, M_0^vCar, M_0^vTruck) = (6, 5, 6, 6) fpc are estimated with the estimates given in Figure <ref>.

In order to avoid bias introduced by the binning step, we re-estimate the univariate scores in separate univariate generalized functional additive mixed models with additive predictors

    η_ij1^(k')(t |{ϕ̂_1m^(k')(t)}_m=1^M_1^k', {ϕ̂_0m^(k')(t)}_m=1^M_0^k') =    β_0^(k')(t) + f^(k')(year_j, t) 
        + lanes1_iβ_1^(k')(t) + lanes3_iβ_2^(k')(t) 
         + ∑_m=1^M_1^k'ξ_1im^(k')ϕ̂_1m^(k')(t) + ∑_m=1^M_0^k'ξ_0ijm^(k')ϕ̂_0m^(k')(t)
    η_ij1^(k”)(t |{ϕ̂_1m^(k”)(t)}_m=1^M_1^k”, {ϕ̂_0m^(k”)(t)}_m=1^M_0^k”) =    β_0^(k”)(t) + f^(k”)(year_j, t) + 
        + limit30_iβ_1^(k”)(t) 
         + limit60_iβ_2^(k”)(t)
         + ∑_m=1^M_1^k”ξ_im^(k”)ϕ̂_1m^(k”)(t) + ∑_m=1^M_0^k”ξ_ijm^(k”)ϕ̂_0m^(k”)(t),

for k'∈{qCar, qTruck} and k”∈{vCar, vTruck}, conditioning on the estimated univariate fpc ϕ̂_1m^(k)(t) and ϕ̂_0m^(k)(t), k ∈{qCar, qTruck, vCar, vTruck}. The functional intercepts β_0^(k)(t) and the categorical functional effects β_1^(k)(t) and β_2^(k)(t)  are represented using 14 cubic P-splines while the bivariate interaction terms f^(k)(year_j, t) are modeled using anisotropic tensor product splines of 7 (for year) and 14 (for t) marginal cubic P-splines allowing different variance parameters for the different margins.
These four univariate models are again estimated in the bamlss framework with 1,000 mcmc samples after a burn-in of 1,000 and a thinning of 5 in order to get more stable estimates of the univariate scores. The computation of the univariate generalized functional additive mixed models takes 12 to 18 hours per outcome. We then use the posterior mean of the scores as estimates of the univariate scores and Table <ref> reports the estimated univariate eigenvalues from this univariate generalized fpca approach.



 §.§ Additional Results


Table <ref> reports the estimated multivariate eigenvalues based on a weighted scalar product. Figure <ref> presents the areas where pointwise 95% credible intervals of the bivariate interaction effect f^(k)(year_j, t) do not include zero. Areas that do not include zero are highlighted by their effect (either positive or negative) compared to the global functional intercept. Figure <ref> shows the estimated coefficient effect functions β̂_1^(k”)(t) and β̂_2^(k”)(t), k”∈{vCar, vTruck} for sites with a 30 km/h or 60 km/h (or higher) speed limit, respectively, depicting the difference compared to a site with a 50 km/h speed limit on the linear predictor scale, all other things being equal. The dashed lines show the pointwise 95% credible intervals. Figure <ref> constructs a 90% and a 95% credible region for the joint estimate of (β̂_1^(vCar)(t), β̂_1^(vTruck)(t))^⊤ at time point t = 12 based on a two-dimensional kernel density estimate on the 1000 mcmc samples. This can be used to test the null hypothesis H_0: β̂_1^(vCar)(12) = β̂_1^(vTruck)(12) = 0 by checking whether the origin (0, 0)^⊤ (marked in red) is included in the 1-α credible region. Figure <ref> shows the impact of the five leading estimated mfpc on the mean for the site-year-specific (level 2) random process. The black line corresponds to the estimated mean of a site with two lanes and 50 km/h speed limit in 2020 with plus (red +) or minus (blue -) 2√(ν̂_2m), m = 1, 2, 3 times the estimated mfpc. Table <ref> reports the relative contributions of the estimated posterior score variances to the total (posterior unweighted) variation for the site-specific random process (level 1, ν_1,m, m = 1,...,9) and the site-year-specific random process (level 2, ν_0,m, m = 1,...,) based on an mfpca with scalar product with unequal weights. Figure <ref> shows the fitted location parameter together with the scalar observations for three different years at the traffic detector site “Goerzallee”. Figure <ref> presents outcome specific worm plots <cit.> of normalized (randomized) quantile residuals <cit.>.
 




























 Paper Title: A new unit-bimodal distribution based on correlated Birnbaum-Saunders random variables.


	Dynamical friction in rotating ultralight dark matter galactic cores
    V.M. Gorkavenko^1, O.V. Barabash^1,
  T.V. Gorkavenko^1,  
O.M. Teslyk^1,

 A.O. Zaporozhchenko^1, Junji Jia^3,  A.I. Yakimenko^1,4,5, E.V. Gorbar^1,2

^1 Faculty of Physics, Taras Shevchenko National University of Kyiv,

64, Volodymyrs'ka str., Kyiv 01601, Ukraine

^2 Bogolyubov Institute for Theoretical Physics, National Academy of Sciences of Ukraine,

14-b Metrolohichna str., Kyiv 03143, Ukraine

^3 School of Physics and Technology, Wuhan University, 299 Bayi Road, Wuhan, Hubei Prov., China

^4 Dipartimento di Fisica e Astronomia ’Galileo Galilei’, Universit'a di Padova,
 via Marzolo 8, 35131 Padova, Italy

^5 Istituto Nazionale di Fisica Nucleare, Sezione di Padova, via Marzolo 8, 35131 Padova, Italy

    
===============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
		
			In this paper, we propose a new distribution over the unit interval which can be characterized as a ratio of the type Zd=Y/(X+Y) where X and Y are two correlated Birnbaum-Saunders random variables. The stress-strength probability between X and Y is calculated explicitly when the respective scale parameters are equal. Two applications of the ratio distribution are discussed.
			
			
			
		
		
		
		
	
	Keywords. Unit distribution · Type I ratio ·  Maximum likelihood method · Monte Carlo simulation ·  software.
	

	Mathematics Subject Classification (2010). MSC 60E05 · MSC 62Exx · MSC 62Fxx.
	
	
		
		
		
	
	

§ INTRODUCTION

	
	Traditionally, the activities in statistical distribution theory have been concentrated on distributions with unbounded support. Only in recent years, attention has been devoted to the construction of distributions with bounded support. Bounded data have applications in economics, finance, biology and medicine among others, see e.g., the introductory remarks in  <cit.> and <cit.>. Such data arise naturally in the form of rates and proportions. Some of the most prominent bounded distributions are the beta distribution and their generalizations and the distributions of Kumaraswamy and Topp-Leone. 
	
	A natural way to create a model for bounded data is to transform one or two unbounded random variables in a bounded variable Z. For example, Z=X/(X+1), Z=exp(-X) and Z=X/(X+Y) have the support (0, 1), when X and Y are positive variables. In applications of the last model it is usually assumed that X and Y are independent. We consider some specific approaches using these transformations or more complicated ones. The first four techniques transform a single variable: 
	<cit.> derive a bimodal beta distribution, applying a quadratic transformation technique due to Elal-Olivero to the beta distribution.
	<cit.> apply an approach of Marshall and Olkin to the reduced Kies distribution.
	<cit.> propose a general framework in which a monotone function is applied to a single variable. 
	<cit.> applies a procedure due to  <cit.> to a uniformly distributed variable, resulting in a three-parameter family of distributions over the unit interval.
	<cit.> (unit log) use the approach Z=X/(X+Y) where X and Y are the components of a two-dimensional random vector, following a bivariate log-symmetric distribution. In this model, X and Y may be correlated.
	
	The distribution of ratios of the latter type is of great importance in practice, for example in modeling COVID-19-related death rates, see <cit.>. 
	






















	
	
		On the other hand, the Birnbaum–Saunders (BS) fatigue-life model, introduced by <cit.>, is based on renewal theory principles. This model focuses on the number of cycles needed to exceed a critical threshold and cause fatigue crack propagation. In the same year, <cit.> provided maximum likelihood estimates for the BS model parameters. Later, <cit.>, in his work on stochastic models of failure in random environments, offered an alternative derivation of the distribution using a biological model and relaxed several of the assumptions made by <cit.>. The BS distribution has been extensively studied and applied across various fields, including business, industry, economics, finance, management, engineering, and medical sciences. For a review of recent work on the BS distribution, see <cit.> and the references therein.
		
		Bivariate (and multivariate) versions of the BS distribution have also been explored. <cit.> introduced the bivariate BS distribution with five parameters using the bivariate normal distribution function. <cit.> revisited the bivariate BS distribution and discussed several new properties. <cit.> introduced a mean-based bivariate BS distribution, demonstrating that the mean is one of its parameters.
	
	
	
	
	In the present paper we propose the distribution of a random variable of the type Z=Y/(X+Y), where X and Y are two (possibly correlated) Birnbaum-Saunders random variables. The article is structured as follows. 
	In Section <ref>, we introduce the unit Birnbaum-Saunders (UBS) model and some of its forms are outlined in graphs. Note that bimodality in the model is observed as a function of its shape and correlation parameters.
	In Section <ref>, we show one of the main properties that the UBS model has, that is, the corresponding random variable to the UBS distribution originally comes from a type I ratio <cit.> of two variables following a bivariate BS distribution.
	Also in this section, we derive the cumulative distribution function (CDF), which is used to explicitly calculate the stress-strength probability in the case of the presence of correlation and equality of scale parameters. Finally, the maximum product of spacings method for parameter estimation is presented. In Section <ref>, we carry out a Monte Carlo simulation study to evaluate the performance of the maximum product of spacings estimators by means of their bias and root mean square error. In Section <ref>, we present two applications to income-consumption and body mass data. We demonstrate that these data can also be fitted well by the UBS distribution. Finally, in Section <ref>, some concluding remarks are made.




	
	
	
	
	
	

§ THE UNIT BIRNBAUM-SAUNDERS DISTRIBUTION

	
	
	We say that a continuous random variable Z, with support (0, 1), follows a unit BS (UBS) distribution with parameter vector θ_ρ=(α_1,α_2,β_1,β_2,ρ)^⊤, α_1> 0,α_2> 0 ,β_1> 0, β_2> 0 and ρ∈(-1,1),
	denoted by Z∼ UBS(θ_ρ), if its probability density function (PDF) is given by (for 0<z<1 and s=1/z-1)
	
		
    f_Z(z;θ_ρ)
    			   =
    			exp{1 1-ρ^2(
    				1α_1^2+1α_2^2)} 4πα_1α_2 √(1-ρ^2) (s+1)^2 s exp{
    			-
    			ρα_1α_2(1-ρ^2)(√(β_2 sβ_1)
    			+
    			√(β_1β_2 s))
    			}
       ×{(
    			√(β_2 sβ_1)
    			+
    			√(β_1β_2 s)) 
    			K_0(√(u_ρ v_ρ) 1-ρ^2)
    			+
    			[
    			√((v_ρ/u_ρ)sβ_1β_2)
    			+
    			√(β_1β_2 (v_ρ/u_ρ)s) ] 
    			K_1(√(u_ρ v_ρ) 1-ρ^2)
    			},

	where 
	
    u_ρ≡sα_1^2β_1
    		+
    		1α_2^2β_2
    		-
    		2ρα_1α_2√(sβ_1β_2),
    		   
    		v_ρ≡β_1α_1^2 s
    		+
    		β_2α_2^2
    		-
    		2ρα_1α_2√(β_1β_2 s).

	Furthermore, in formula (<ref>),
	
    K_λ(x)
    	   =
    	1 2∫_0^∞ w^λ-1exp{-1 2  x(w+1 w)} dw,    x>0, λ∈ℝ,
    	
       =
    	π 2sin(λπ)[I_-λ(x)-I_λ(x)]

	is the modified Bessel function of the third kind (also known as modified Bessel function of the second kind) <cit.>,
	with
	
    I_λ(x)
    	=
    	(x 2)^λ∑_k=0^∞1 k!Γ(λ+k+1)(x 2)^2k

	being the modified Bessel function of the first kind. When λ∈ℤ the following integral representation of I_λ is valid <cit.>:
I_λ(x)
	=
	(1/π) ∫_0^πexp{xcos(θ)}cos(λθ) dθ, 
	 λ∈ℤ.
	
	When ρ=0, that is, X and Y are independent, the PDF (<ref>) of Z is simplified as (for 0<z<1 and s=1/z-1)
		
    f_Z(z;θ_0)
    			   =
    			exp(
    				1α_1^2+1α_2^2) 4πα_1α_2 (s+1)^2 s{(
    			√(β_2 sβ_1)
    			+
    			√(β_1β_2 s)) 
    			K_0(√(uv))
    			+
    			[
    			√((v/u)sβ_1β_2)
    			+
    			√(β_1β_2 (v/u)s) ] 
    			K_1(√(uv))
    			},

	where u=u_0 and v=v_0, and θ_0
	=θ_ρ=0=(α_1,α_2,β_1,β_2)^⊤.
	
		Let S=X/Y be the type II ratio, where X∼ BS(α_1,β_1) and Y∼ BS(α_2,β_2) are independent random variables (that is, ρ=0), and let f_S(s;θ_0) denote its respective PDF.
	By using 
	the identity f_S(s;θ_0)=∫_0^∞ y f_X(sy)f_Y(y)  dy it is simple to check that the PDF of S is (for s>0)
	
    f_S(s;θ_0)
    	   =
    				exp(
    		1α_1^2+1α_2^2) 4πα_1α_2 1 s{(
    	√(β_2 sβ_1)
    	+
    	√(β_1β_2 s)) 
    	K_0(√(uv))
    	+
    	[
    	√((v/u)sβ_1β_2)
    	+
    	√(β_1β_2 (v/u)s) ] 
    	K_1(√(uv))
    	}
       =
    	1 (s+1)^2  f_Z(1 s+1;θ_0),

	where f_Z(z;θ_0) is as given in (<ref>). The PDF (<ref>) of S with β_1=β_2=1 has appeared in reference  <cit.>.
	
	
	Figures <ref>, <ref> and <ref> show the behavior of f_Z for some parameter choices.
	Note that the density modality changes according to the value of α_1 and α_2 parameters.
	
	
	
	
	
	


§ CHARACTERIZATION AND ESTIMATION


In this section, we establish some mathematical results of the UBS distribution. In particular, we define the UBS model as a ratio, present the cumulative distribution function, and calculate the stress-strength probability. We also derive parameter estimators for the UBS model based on the maximum product of spacings method.





 §.§ UBS model arising as a ratio

	In this section, we establish one of the most important properties of the UBS model. This property characterizes a UBS random variable Z in (<ref>) as a ratio of the form Y/(X+Y), where the random vector (X,Y)^⊤ follows a bivariate Birnbaum-Saunders distribution as discussed in <cit.>.
	
	Indeed, let X, Y and Z be continuous and positive random variables
	such that
	
    Zd=YX+Y,

	being d= equality in distribution, and the ratio in (<ref>) has support in the unit interval (0, 1). The random variable Z = Y/(X+Y) is known in the literature as type I ratio <cit.>.
	
	From (<ref>), it is simple to verify that the cumulative distribution function (CDF) of Z is given by
	
    F_Z(z)
    		=
    		
    		
    		
    		
    		
    		
    		
    		ℙ(X Y⩾s)
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		=
    		1-
    		∫_0^∞ℙ(X⩽ sy| Y=y) f_Y(y)  dy,
    		   s=1 z-1.

	
	
	
	
	
	
	
	
	
	
	
	
	Hence, the	 PDF of Z can be expressed as
	
    f_Z(z)
    		=
    		(s+1)^2
    		∫_0^∞ y f_X,Y(sy,y)  dy.

	
	
		Equation in (<ref>) can also be obtained using the Jacobian method by determining the marginal with respect to Z of the random vector (Z,U)^⊤, where Z=Y/(X+Y) and U=Y.
	
	
	Now, let's suppose that (X,Y)^⊤ has a bivariate Birnbaum-Saunders distribution with parameter vector θ_ρ=(α_1,α_2,β_1,β_2,ρ)^⊤, denoted by (X,Y)^⊤∼ BS(θ), that is, its joint density is given by
	
    f_X,Y(x,y)
    		=
    		ϕ_2(a(x;α_1,β_1),a(y;α_2,β_2);ρ)
    		a'(x;α_1,β_1)a'(y;α_2,β_2),

	where ϕ_2(u, v;ρ) denotes the standard bivariate normal density function;
	
    ϕ_2(u, v;ρ)
    		=
    		1 2π√(1-ρ^2) exp{-1 2(1-ρ^2)  (u^2+v^2-2ρ uv)},
    		   (u,v)^⊤∈ℝ^2

	and
	
    a(t;α,β)
    		=
    		1α(√(tβ)-√(β t)),
    		   
    		a'(t;α,β)
    		=
    		1 2α t(√(tβ)+√(β t)),
    		   t>0.

	
	Hence, the density (<ref>) of Z is written as
	
    f_Z(z)
    		=
    		(s+1)^2
    		∫_0^∞ y 
    		ϕ_2(a(sy;α_1,β_1),a(y;α_2,β_2);ρ)
    		a'(sy,α_1,β_1)a'(y,α_2,β_2)
    		 dy.

	
	Since
	
    a^2(sy;α_1,β_1)+a^2(y;α_2,β_2)-2ρ a(sy;α_1,β_1)a(y;α_2,β_2)
    		
       =
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		
    		(
    		u_ρ y
    		+
    		v_ρ y)
    		-
    		2
    		[
    		1α_1^2
    		+
    		1α_2^2
    		-ρα_1α_2(√(β_2 sβ_1)
    		+
    		√(β_1β_2 s))
    		],

	with u_ρ and v_ρ as given in (<ref>), note that the function ϕ_2(a(sy;α_1,β_1),a(y;α_2,β_2);ρ) in (<ref>) can be expressed as
	
    ϕ_2(a(sy;α_1,β_1),     a(y;α_2,β_2);ρ)
    		=
    		1 2 exp{-1 2(1-ρ^2)(
    		u_ρ y
    		+
    		v_ρ y)}
       ×1π√(1-ρ^2) exp{1 1-ρ^2[
    		1α_1^2
    		+
    		1α_2^2
    		-ρα_1α_2(√(β_2 sβ_1)
    		+
    		√(β_1β_2 s))
    		]}.

	
	Furthermore, notice that
	
    a'(sy;α_1,β_1)a'(y;α_2,β_2)
    		   =
    		
    		
    		
    		
    		
    		1 4α_1α_2 sy[
    		1 y^2√(β_1β_2 s)
    		+
    		1 y(
    		√(β_2 sβ_1)
    		+
    		√(β_1β_2 s))
    		+
    		√(sβ_1β_2)].

	
	By replacing (<ref>) and (<ref>) in formula (<ref>), we have
	
    f_Z(z)
    		   =
    		1π√(1-ρ^2) (s+1)^2 4α_1α_2 s exp{1 1-ρ^2[
    		1α_1^2
    		+
    		1α_2^2
    		-ρα_1α_2(√(β_2 sβ_1)
    		+
    		√(β_1β_2 s))
    		]}
       ×[
    		√(β_1β_2 s) 𝕀_1
    		+
    		(
    		√(β_2 sβ_1)
    		+
    		√(β_1β_2 s)) 𝕀_2
    		+
    		√(sβ_1β_2) 𝕀_3
    		],

	where
	
    𝕀_1
    		=
    		1 2 ∫_0^∞1 y^2 exp{-1 2(1-ρ^2)(
    		u_ρ y
    		+
    		v_ρ y)} dy
    		=
    				√(u_ρ v_ρ)  
    		K_-1(√(u_ρ v_ρ) 1-ρ^2),
    		
       𝕀_2
    		=
    		1 2 ∫_0^∞1 y exp{-1 2(1-ρ^2)(
    		u_ρ y
    		+
    		v_ρ y)} dy
    		=
    		K_0(√(u_ρ v_ρ) 1-ρ^2)

	and
	
    𝕀_3
    		=
    		1 2 ∫_0^∞exp{-1 2(1-ρ^2)(
    		u_ρ y
    		+
    		v_ρ y)} dy
    		=
    		√(v_ρ u_ρ)  
    		K_1(√(u_ρ v_ρ) 1-ρ^2).

	In the above, K_λ(x), x>0, denotes the modified Bessel function of the third kind defined in (<ref>).
	
	Finally, by using that the Bessel function is symmetric with respect to the index λ: K_λ(x)=K_-λ(x), and by employing the above formulas of 𝕀_1, 𝕀_2 and 𝕀_3, from (<ref>) the PDF of Z can be expressed as in (<ref>).
	
	

 §.§ The cumulative distribution function

	
	For (X,Y)^⊤∼ BS(θ_ρ), it holds that X∼ BS(α_1,β_1) and Y∼ BS(α_2,β_2). Furthermore, the  conditional CDF of X, given Y=y, is given by <cit.>
	
    ℙ(X⩽ x| Y=y) 
    		=
    		Φ{
    			a(x;α_1,β_1)
    			-
    			ρ a(y;α_2,β_2)	
    		√(1-ρ^2)},

	where a(t;α,β) is as given in (<ref>) and Φ is the CDF of the standard normal distribution.
	Hence, from (<ref>) it follows that
	
    F_Z(z;θ_ρ)
    		   =
    		1-
    		∫_0^∞ℙ(X⩽ sy| Y=y) 
    		f_Y(y)  dy,
    		   s=1 z-1,
    		
       =
    		1-
    		∫_0^∞Φ{
    			a(sy;α_1,β_1)
    			-
    			ρa(y;α_2,β_2)
    		√(1-ρ^2)}ϕ(a(y;α_2,β_2))a'(y;α_2,β_2)
    		 dy,

	where ϕ denotes the PDF of the standard normal distribution.
	By using the change of variable w=a(y;α_2,β_2), we write the above expression as
	
	
	
    1-
    		∫_-∞^∞Φ{
    			a(sa^-1(w;α_2,β_2);α_1,β_1)
    			-
    			ρw
    		√(1-ρ^2)}ϕ(w)
    		 dw
    		
       =
    		1 2
    		-
    		1 2∫_-∞^∞ erf{
    			a(sa^-1(w;α_2,β_2);α_1,β_1)
    			-
    			ρw
    		√(2(1-ρ^2))}ϕ(w)
    		 dw,

	where we have used the well-known identity 
	Φ(x)=(1/2)[1+(x/√(2))]. In the above, a^-1(t;α,β) denotes the inverse function of a(t;α,β).
	As 
	a^-1(t;α,β)
	=
	(β/4)[α t+√((α t)^2+4)  ]^2,
	the integral above can be written as
	
    1 2
    		-
    		1 2∫_-∞^∞ erf{(α_2α_1√(sβ_2β_1)-ρ)√(2(1-ρ^2)) 
    			w
    			+
    			√(2)(sβ_2-β_1)α_1√(sβ_1β_2(1-ρ^2)) 1α_2w+√((α_2 w)^2+4)}ϕ(w)
    		 dw.

	
	Therefore, the CDF of the UBS random variable Z is given by (for s=1/z-1 and 0<z<1)
	
    F_Z(z;θ_ρ)
    		=
    		1 2
    		-
    		1 2∫_-∞^∞ erf{(α_2α_1√(sβ_2β_1)-ρ)√(2(1-ρ^2)) 
    			w
    			+
    			√(2)(sβ_2-β_1)α_1√(sβ_1β_2(1-ρ^2)) 1α_2w+√((α_2 w)^2+4)}ϕ(w)
    		 dw.

	
	
	
	
	
	
	
	

 §.§ The stress-strength probability

	
	
	In what follows we explicitly calculate the stress-strength probability R≡ℙ(X<Y)  in the case where (X,Y)^⊤∼ BS(θ_ρ), that is,
	X and Y can admit correlation. 
	
	Indeed, by (<ref>), we have
	
    R
    		=
    		ℙ(X Y< 1)
    		=
    		1-ℙ(X Y⩾ 1)
    		=
    		1-F_Z(z;θ_ρ)|_z=1/2.

	Hence, as z=1/2 if and only if s=1, by using (<ref>),
	
	we get
	
    R
    		=
    		1 2
    		+
    		1 2∫_-∞^∞ erf{(α_2α_1√(β_2β_1)-ρ)√(2(1-ρ^2)) 
    			w
    			+
    			√(2)(β_2-β_1)α_1√(β_1β_2(1-ρ^2)) 1α_2w+√((α_2 w)^2+4)}ϕ(w)
    		 dw.

	
	In the particular case β_1=β_2, from (<ref>), the stress-strength probability R is simply written as
	
    R
    		=
    		1 2
    		+
    		1 2∫_-∞^∞ erf{(
    			α_2α_1
    			-
    			ρ)
    		√(2(1-ρ^2)) 
    		w
    		}ϕ(w)  dw
    		=
    		1 2,
    		  ∀ρ∈(-1,1),

	where in the last equality we have used the fact that erf(x) and ϕ are odd and even functions, respectively.
	
	To our knowledge, in this setting (β_1=β_2), identity R=1/2 was known only for the special case in which X and Y are independent (that is, ρ=0); see, e.g. <cit.> and <cit.>.
	
	



































































































































































































































	
	





 §.§ Maximum product of spacings method


The maximum product of spacings (MPS) method was introduced by <cit.> and <cit.> as a different approach to estimating parameters of continuous univariate distributions compared to maximum likelihood estimation. Let {Z_i:i = 1,…,n} be a univariate random sample of size n from the UBS(θ_ρ)
distribution with PDF as in (<ref>), and let z_i be the sample observation of Z_i. Then, the uniform spacings of  {z_i:i = 1,…,n} is given by

    Δ_i(θ_ρ)=F_Z(z_i:n;θ_ρ)-
     F_Z(z_i-1:n;θ_ρ),     1,…,n,

with F_Z(z_0:n;θ_ρ)=0, F_Z(z_n+1:n;θ_ρ)=1,
and Δ_0(θ_ρ)+Δ_1(θ_ρ)+⋯+Δ_n+1(θ_ρ)=1.

The maximum product of spacings estimator, θ_ρ, is obtained by maximizing the geometric mean of the spacings

    G(θ_ρ)=( ∏_i=1^n+1Δ_i(θ_ρ) )^1/n+1,

or, equivalently, maximizing the function

    H(θ_ρ)= 1/1+n∑_i=1^n+1log( Δ_i(θ_ρ)).











§ SIMULATION STUDY



In this section, we carry out a Monte Carlo simulation study for evaluating the performance of the above described maximum product of spacings method for the UBS distribution (<ref>). The simulation scenario considers the following setup: 300 Monte Carlo replications, sample size n ∈{100,200,400,800}, and vector of true parameters (α_1,α_2,β_1,β_2)^⊤= (0.5,0,5,1.0,1.0)^⊤, ρ∈{0.10,0.25,0.5,0.75}.


The performance and recovery of the maximum product of spacings estimators are evaluated by means of relative bias (RB), and root mean square error (RMSE), which are calculated from the Monte Carlo replicates, as

    RE(γ)    =   1/N∑_i = 1^N| (γ^(i) - γ)/γ| ,  RMSE(γ) = √(1/N∑_i = 1^N (γ^(i) - γ)^2),

where γ and γ^(i) are the true parameter value and its i-th estimate, and N is the number of Monte Carlo replications. The steps for the Monte Carlo simulation study are as:
i. set the values of the parameters of the UBS distribution; ii. generate 300 samples of size n from the chosen model; iii. estimate the model parameters using the maximum product of spacings method method for each sample; and iv. compute the empirical RB and RMSE.

The maximum product of spacings estimation results so obtained are shown in Figures <ref> and <ref> wherein the empirical bias and RMSE are both plotted. We observe that the results obtained for the UBS distribution are as expected in that as the sample size increases, both RE and RMSE decrease. In terms of the impact of the parameter ρ, we observe that both RB and RMSE of α_1, α_2, β_1 tend to increase as ρ increases. In the case of ρ, we observe the opposite behavior.


















§ APPLICATION

	

    We now give two applications using real data previously analyzed in the literature. We study the validity of the new model for the data sets and show that UBS distribution presents good fits on both cases.




 §.§ Income-consumption data



In order to evaluate the new probabilistic model, a public data set of income and consumption data was used.
The data (available at <https://www.bancaditalia.it/statistiche/tematiche/indagini-famiglie-imprese/bilanci-famiglie/documentazione/ricerca/ricerca.html?min_anno_pubblicazione=2008   max_anno_pubblicazione=2008>, accessed on July 22, 2024) comes from the Bank of Italy's Survey on Household Income and Wealth from the year 2008.


The income comprises payroll income, pensions and net transfers, net self-employment income, and property income. 






Consumption consists of expenditures on both durable and non-durable goods. Durable expenditure represents the net balance of purchased and sold goods. Non-durable expenditures include monthly expenses such as rent, food, and other essentials, as well as annual expenses.

We used the data from the survey, available as the data sets RFAM08 (income as Y variable) and RISFAM08 (expenditures as C variable), according to the Survey on Household Income and Wealth 2008 data description file.
We removed from the data set lines whose income or consumption was negative or unavailable.
Thus, the analyzed data set included information from 7,957 families, after removing 20 lines of the data.

The same data set was also used in <cit.> and <cit.>, however, they used a copula approach to model the data and compared income and consumption through stress-strength measures of the type ℙ(X<Y).
Our framework involves constructing a new variable U=X/(X+Y).
In summary, when X and Y represent expenditures and income, respectively,  U < 1/2 indicates that the family ended the year with more expenditures than income. If U > 1/2, the opposite is true. The case U = 1/2 means expenditures and income are equal.


Table <ref> presents descriptive statistics for U=X/(X+Y).
The estimated parameters of UBS model was θ_ρ = (0.275, 0.274, 1.041, 1.331, 0.149).
Figure <ref> shows the fit of the UBS distribution to the density and ECDF of the data. The histogram of U shows that with high (81.5%) probability U>1/2. That means, in general, the families have income greater than expenditures.

















 
 
 
 
 
 



Given that the U data has unit support, we can make a comparison between the utilization of the UBS model and other families of unit distributions, such as Beta and the unit bimodal BS (UBBS[The UBBS model was proposed in <cit.>.]) models, as illustrated in Table <ref>.
To obtain the estimation results for Beta and UBBS models in Table <ref>, we used the package AdequacyModel in the R software.
Deciding the better distribution to model the data was oriented by AIC (Akaike information criterion) and BIC (Bayesian information criterion) criteria.
In Table <ref>, these criteria suggest UBS fits the data better than Beta and UBBS models.









 §.§ Body mass data



Biomedical measurements of 202 athletes from different sports are presented in the Australian Institute of Sport (AIS) data set. <cit.> proposed the UBBS model and showed its fit on the distribution of the percentage of the body fat complement of the athletes. In this subsection, we show that the new UBS model is an alternative distribution to fit that data.
The data were imported directly through the software R by the command:


Descriptive statistics for 𝐮 (the athlete's body mass in the data set) are presented in Table <ref>.
As the data set has unit support, the UBS distribution is a candidate to model such data. 
Figure <ref> shows the fit of UBS to the data.
Maximum likelihood estimates and maximum product of spacings estimates can be compared graphically. 







In Table <ref>, when comparing the UBS model (with parameter estimates via product of spacings) with the Beta and UBBS models using information criteria, the BIC indicates the new model is better to fit the athlete’s body mass data. However, when considering the AIC, the UBBS model should be chosen.
	
	
	



	


§ CONCLUDING REMARKS

	

	
	

In this paper, we have introduced a new unit distribution. Its generation as a ratio of possibly dependent Birnbaum-Saunders random variables has been discussed in detail. As a more theoretical part, we have derived the cumulative distribution function and the stress-strength probability. As practical issues, we presented maximum product of spacings estimators and applications to income-consumption and body mass data sets.

The research could be continued in several directions. One could investigate the moments and the moment generating function and try to express shape characteristics like skewness and kurtosis. The analysis of the entropy of the distribution could also be of interest. Instead of using the ratio Y/(X+Y) one could explore other construction principles like those mentioned in the introduction to obtain a unit distribution. Finally, one could use a bivariate distribution other than the Birnbaum-Saunders to construct a unit distribution.
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

  
Acknowledgements
	This study was financed in part by the
	Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.
	
	

  
Disclosure statement
	There are no conflicts of interest to disclose.
	
	
	
	
	
	
	
	
	
	
	
	
	10
		
		[Abramowitz and Stegun, 1972]Abramowitz1972	
		M. Abramowitz and I. A. Stegun, 
		(eds.): Handbook of Mathematical Functions,
		9th edn. Dover, New York (1972).
		
		[Afifi et al., 2022]Afifi2022		
		A. Z. Afifi, M. Nassar, D. Kumar, and G.M. Cordeiro, 
		A New Unit Distribution: Properties, Inference, and Applications,
		Electronic Journal of Applied Statistical 
		Analysis (2022), 
		15(02), pp. 460–484, 
		
		
		[Bekker et al., 2009]Bekker2009
		 A. Bekker, J. Roux, and T. Pham-Gia,
		The type I distribution of the ratio of independent “Weibullized” generalized beta-prime variables,
		Statistical Papers (2009), 50, pp. 323–338.	
		
		[Birnbaum and Saunders, 1969a]Birnbaum1969a	
		Z. W. Birnbaum and S. C. Saunders,  
		A new family of life distributions,
		Journal of Applied Probability (1969), 
		6(2), pp. 319-327.
		
		[Birnbaum and Saunders, 1969b]Birnbaum1969b
		Z. W. Birnbaum and S. C. Saunders, 
		Estimation for a family of life distributions with applications to fatigue,
		Journal of Applied Probability (1969), 
		6(2), pp. 328–347.
		
		[Bourguignon et al., 2024]Bourguignon2024	
		M. Bourguignon, H. Saulo, and D. Gallardo. 
		Parametric quantile beta regression model, 
		International Statistical Review (2024), 92, pp. 106–129.
		
		[Cheng and Amin, 1979]ChengAmin1979
		R. C. H. Cheng and N. A. K. Amin,
		Maximum product-of-spacings estimation with applications to the lognormal distribution, Technical Report, Department of Mathematics, University of Wales (1979).

		[Cheng and Amin, 1983]ChengAmin1983
		R. C. H. Cheng and N. A. K. Amin,
		Estimating parameters in continuous univariate distributions with a shifted origin, Journal of the Royal Statistical Society. Series B (1983), 3, pp. 394–403.

		[Condino and Domma, 2023]Condino2023	
		F. Condino and F. Domma,
		Unit Distributions: A General Framework, Some Special 
			Cases, and the Regression Unit-Dagum Models, 
		Mathematics (2023), 11, 2888. 
		
		
		[Cortés et al., 2018]Cortes2018	
		M. A. Cortés, D.  Elal-Olivero, and J.,F. Olivares-Pacheco, 
		A New Class of Distributions Generated by the Extended Bimodal-Normal Distribution, 
		Journal of Probability and Statistics, (2018), 1-10.

	[Desmond, 1985]Desmond1985		
		A. Desmond,  
		Stochastic models of failure in random environments,
		Canadian Journal of Statistics (1985), 
		13(3), pp. 171–183. 
		


	[Domma and Giordano, 2012]domma2012stress	
		F. Domma and S. Giordano,
		A stress–strength model with dependent variables to measure household financial fragility, 
		Statistical Methods & Applications (2012), 21, pp. 375–389. 
  
		[Johnson  et al.(1995)]Johnson95
		N. L. Johnson, S. Kotz, and N. Balakrishnan,   
		Continuous Univariate Distributions - Vol. 2. 
		John Wiley & Sons, New York (1995).	
		
		[Jørgensen, 1982]Jorgensen1982	
		B. Jørgensen, 
		Statistical Properties of the Generalized Inverse Gaussian Distribution,
		Lecture Notes in Statistics, vol. 9. 
		Springer, New York (1982).
		
		[Kundu and Gupta, 2017]Kundu2017
		D. Kundu and R. C. Gupta,  
		On Bivariate Birnbaum-Saunders Distribution,
		American Journal of Mathematical and Management Sciences (2017),  36(1), pp. 21–33.
		
		[Kundu et. al, 2010]Kundu2010
		D. Kundu, N. Balakrishnan and A. Jamalizadeh,
		Bivariate Birnbaum-Saunders distribution and associated inference,
		Journal of Multivariate Analysis (2010), 101, pp. 113–125.

	[Lima et al., 2024]Lima2024assessing
		R. Lima,  F. Quintino, T. da Fonseca, L. Ozelim, P. Rathie and H. Saulo,
		Assessing the Impact of Copula Selection on Reliability Measures of Type P(X<Y) with Generalized Extreme Value Marginals,
		Modelling (2024), 5(1), pp. 180-200.

[Martínez-Flórez et al., 2024]Martinez2024unit
G. Martínez-Flórez, N. Olmos and O. Venegas,  Unit-bimodal Birnbaum-Saunders distribution with applications, Communications in Statistics-Simulation and Computation (2024), 53(5), pp. 2173-2192.
  
		[Teimouri et. al, 2013]Teimouri2013
		M. Teimouri, S. M. Hosseini and S. Nadarajah,
		Ratios of Birnbaum-Saunders Random Variables,
		Quality Technology & Quantitative Management (2013),
		10, pp. 457–481.
		
		[Quintino et. al, 2024]Quintino2024
		F. S. Quintino, L. C. d. S. M. Ozelim, T. A. d. Fonseca and P. N. Rathie,  
		Stress-Strength Reliability of the Type ℙ(X<Y) for Birnbaum-Saunders Components: A General Result, Simulations and Real Data Set Applications,
		Modelling (2024), 5, pp. 223–237.
		
		[Ranneby, 1984]Ranneby1984
		B. Ranneby,
		The maximum spacing method. An estimation method related to the maximum likelihood method,
		Scandinavian Journal of Statistics (1984),
		11, pp. 93–112.

		[Saulo et. al, 2019]Saulo2019
		H. Saulo, J. Leão, R. Vila, V. Leiva and V. Tomazella,
		On mean-based bivariate Birnbaum-Saunders distributions: Properties, inference and
			application, 
		Communications in Statistics - Theory and Methods (2019), 49(24), pp. 6032–6056.
		
		[Vila et al., 2024a]Vila2024a	
		R. Vila,  L. Alfaia, A.F.B. Menezes, M.N. Çankaya, and  M. Bourguignon,  
		A Model for Bimodal Rates and Proportions, 
		Journal of Applied Statistics (2024), 51(4), pp. 664–681.
		
		[Vila et al., 2024b]Vila2024b	
		R. Vila, N. Balakrishnan, and Bourguignon, M,
		On the distribution of a random variable               
			involved in an independent ratio, 
		Communications in Statistics-Theory and 
		Methods (2024), 
		<https://doi.org/10.1080/03610926.2024.2340608>.
		
		[Vila et al., 2024c]Vila2024c	
		R. Vila, N. Balakrishnan, H. Saulo, and P. Zörnig, Unit-log-symmetric models: characterization, statistical properties and their applications to analyzing an internet access data, 
		Quality and Quantity (2024),
		<https://doi.org/10.1007/s11135-024-01879-w>.
		
		[Zörnig, 2023]Zornig2023	
		P. Zörnig, 
		A new type of skew uniform distribution, 
		Asian Journal of Statistical Sciences (2023), 
		3, pp. 27–39.	
		
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	


