Can Your Generative Model Detect Out-of-Distribution Covariate Shift?
Christiaan Viviers Amaan Valiuddin Francisco Caetano Lemar Abdi Lena Filatova Peter de With Fons van der Sommen




 



CovariateFlow






C. Viviers 




Eindhoven University of Technology, Eindhoven, The Netherlands
c.g.a.viviers@tue.nl
Philips IGTs, Best, The Netherlands


Can Your Generative Model Detect Out-of-Distribution Covariate Shift?
    Christiaan Viviers10000-0001-6455-0288 Amaan Valiuddin1Equal contribution0009-0005-2856-5841 Francisco Caetano110000-0002-6069-6084 Lemar Abdi1Lena Filatova2Peter de With10000-0002-7639-7716 Fons van der Sommen10000-0002-3593-2356
    
==========================================================================================================================================================================================================================================




§ ABSTRACT 



Detecting Out-of-Distribution (OOD) sensory data and covariate distribution shift aims to identify new test examples with different high-level image statistics to the captured, normal and In-Distribution (ID) set. Existing OOD detection literature largely focuses on semantic shift with little-to-no consensus over covariate shift. Generative models capture the ID data in an unsupervised manner, enabling them to effectively identify samples that deviate significantly from this learned distribution, irrespective of the downstream task. In this work, we elucidate the ability of generative models to detect and quantify domain-specific covariate shift through extensive analyses that involves a variety of models. To this end, we conjecture that it is sufficient to detect most occurring sensory faults (anomalies and deviations in global signals statistics) by solely modeling high-frequency signal-dependent and independent details. We propose a novel method, CovariateFlow, for OOD detection, specifically tailored to covariate heteroscedastic high-frequency image-components using conditional Normalizing Flows (cNFs). Our results on CIFAR10 vs. CIFAR10-C and ImageNet200 vs. ImageNet200-C demonstrate the effectiveness of the method by accurately detecting OOD covariate shift. This work contributes to enhancing the fidelity of imaging systems and aiding machine learning models in OOD detection in the presence of covariate shift. The code for CovariateFlow is available at https://github.com/cviviers/CovariateFlowhttps://github.com/cviviers/CovariateFlow.





§ INTRODUCTION




Identifying abnormal image statistics is critical for deploying precise sensing technology and reliable machine learning. Out-of-Distribution (OOD) detection methods model the available data or a set of In-Distribution (ID) features, to identify test examples drawn from a different distribution. Notably, generative models offer an unsupervised paradigm to model without making explicit assumptions on the form of the OOD data. With a plethora of possible covariates (abnormal variations in high-level image statistics) and potential downstream machine learning image applications, unsupervised generative modelling is a promising approach for general OOD detection. The prevailing approaches for OOD detection predominantly focus on the semantic contents of the image data. Therefore, this study elucidates covariate shifts, i.e. the change in distribution of high-level image statistics (covariates) subject to consistent low-level semantics.




Likelihood-based methods, such as Normalizing Flows (NFs), offer an intuitive way of OOD detection by evaluating the likelihood of test samples. However, as evidenced in previous research <cit.>, NFs have exhibited limitations in effective OOD detection, often assigning higher likelihoods to OOD samples. Various works have explored this phenomenon and proposed alternative methods to direct likelihood estimation <cit.>. Recent theoretical investigations <cit.> indicate that these methodologies are inherently susceptible to certain types of OOD data. Moreover, the metrics employed for evaluation exhibit a predisposition towards specific categories of OOD data, suggesting an intrinsic limitation in the current approach to OOD detection. In this study we explore this phenomena while improving the covariate OOD detection capabilities of NFs. Additionally, we address this shortcoming by proposing to unify the log-likelihood (LL)-based metric with the typicality score <cit.> in a simple Normalized Score Distance (NSD). Other generative models have in various contexts been applied to the task of semantic OOD detection, ranging from density-based methods <cit.> to different reconstruction-based models <cit.>. However, OOD covariate shift within the context of generative models remains largely unexplored.

We indicate two branches of covariate shift: (1) domain covariate shift, such as images in different styles (e.g. natural vs. sketch) and (2) domain-specific covariate shift (also known sensory anomalies <cit.>), images under different sensory conditions (e.g different lighting, cameras or sensory level degradation (Figure <ref>). Covariate shifts are recognized for their potential to significantly degrade the predictive performance of the model, where in some specialized imaging applications it can indicate system failure. Detecting these covariate factors and the distribution shifts under consistent semantic content <cit.>, will enhance the safety and reliability of imaging systems in diverse fields and the machine learning systems built on-top of these images <cit.>. This necessitates their detection, and if possible, the quantification of its severity. To this end, we are, to the best of our knowledge, the first to implement unsupervised, domain-specific OOD covariate shift detection.

Images across different applications can demonstrate complex noise patterns and variability due to factors such as equipment variations, environmental conditions, and the specific nature of the imaged objects or scenes <cit.>. A novel and effective strategy for improving OOD detection should utilize the data-dependent (heteroscedastic) noise that is present in the signal. This inherent noise serves as a rich source of information that can be exploited to differentiate between ID and OOD samples. In fact, the noise patterns in images can encode subtle differences that may not be apparent from the semantic content in the image alone.

To address these challenges and leverage the nuanced information encoded in noise patterns across various imaging applications, we propose a streamlined approach that models the conditional distribution between low-frequency and high-frequency signal components. This method contrasts with conventional techniques that attempt to model the entire signal distribution, which may inadvertently obscure critical covariate details. We employ a simple filtering approach that segregates the image into distinct low-frequency and high-frequency components. By focusing on the interaction between these frequency components, the proposed approach effectively detect covariate shifts. The research contributions of this work are as follows.


   
   
  * Unsupervised OOD covariate shift detection with comprehensive evaluation of generative model using CIFAR10(-C) and ImageNet200(-C) 
   
  * CovariateFlow: A novel application of conditional Normalizing Flows for high-frequency heteroscedastic image-component density modelling.
   
  * Normalized Score Distance: which unifies Typicality and Log-likelihood as a general metric for OOD detection in normalizing flows.
   
  * Accurate detection of distribution shift and sensory-level changes indicating high sensitivity for potential fault detection. 





§  BACKGROUND AND RELATED WORK






 §.§ Semantic Out-of-Distribution Detection

Approaches to OOD detection are generally divided into two categories: supervised, which necessitates labels or OOD data, and unsupervised, which relies solely on in-distribution data. Although semantic OOD detection does not constitute the core focus of this study, we nevertheless provide a concise overview of the recent developments, since these methodologies hold the potential to translate to covariate OOD detection. For an in-depth exploration of OOD detection methodologies, we refer to the comprehensive review by Yang  <cit.>.



  §.§.§ Explicit Density Methods:
 A straightforward method for OOD detection involves the use of a generative model, p(x; θ) parameterized by θ and trained to fit a given distribution over data x. The process evaluates the likelihood of new, unseen samples under this model with the underlying assumption that OOD samples will exhibit lower likelihoods compared to those that are ID. The Evidence Lower Bound (ELBO) employed in Variational Auto Encoders (VAEs) <cit.> can be used for OOD detection by evaluating a lower bound on the likelihood of a test sample. Plumerault  <cit.> introduced the Adversarial VAE – a novel approach that marries the properties of VAEs with the image generation quality of Generative Adversarial Networks (GANs), thereby offering a robust auto-encoding model that synthesizes images of comparable quality to GANs while retaining the advantageous characteristics of VAEs. 

Unlike VAEs, Normalizing Flows (NFs) <cit.> offer exact and fully tractable likelihood computations. With the introduction of coupling layers <cit.>, NFs can be arbitrarily conditioned and seem to be excellent contenders for conditional OOD detection. However, as evidenced in previous research <cit.>, NFs have exhibited limitations in effective OOD detection, often assigning higher likelihoods to OOD samples. This limitation has been associated with an inherent bias in the flow model architectures, which tends to prioritize modeling local pixel correlations over the semantic content of the data <cit.>. Exploration by Gratwohl et al. <cit.> and Nalisnick et al. <cit.> posits that this phenomenon can be attributed to the fact that ID images are not high-likelihood samples but, rather, constituents of the typical set of the data distribution. Consequently, the investigation into methods that assess the typicality <cit.> of data instances, as an alternative to direct likelihood estimation, has gained traction. Despite empirical evidence demonstrating the efficacy of typicality in OOD benchmarks <cit.>, recent theoretical investigations <cit.> indicate that these methodologies have inherent susceptibilities to specific OOD types and an evaluative bias towards particular OOD categories, thereby underscoring the complexity of OOD detection.



  §.§.§ Image Reconstruction-based Methods:
 These OOD detection methods are based on the principle that models are less effective at accurately reconstructing data that significantly deviates from the training distribution. Graham  <cit.> improves on an innovative approach to OOD detection that leverages the potent generative prowess of recent denoising diffusion probabilistic models (DDPMs) <cit.>. Unlike prior reconstruction-based OOD detection techniques that necessitated meticulous calibration of the model's information bottleneck <cit.>, their method utilizes DDPMs to reconstruct inputs subjected to varying degrees of noise. In this work, we implement this DDPM method as baseline for OOD covariate shift detection.




 §.§ Covariate shift


In essence, covariate shift refers to the phenomenon where images share consistent semantic content (i.e. similar subjects), and yet, are captured under varying imaging conditions. The degree of variation in these conditions signifies the magnitude of the shift. For example, a minor shift might involve images of a subject under varying lighting conditions, while a more substantial shift, such as transitioning from natural images to graphical sketches of the same subject, exemplifies a transition towards domain shift (Figure <ref>). This paper concentrates on in-domain covariate shifts, as these scenarios represent instances where machine learning silently fails <cit.>. 



In related work on covariate shift detection, Averly . <cit.> adopt a model-centric approach to address both covariate and semantic shifts, suggesting a methodology for identifying instances that a deployed machine learning model, such as an image classifier, fails to accurately predict. This strategy implies that the decision to detect, and potentially exclude, a test example, is dependent on the specific model in question. While being effective for well-established machine learning models, this method inherently links the detection of shifts to the peculiarities of the individual model, resulting in each model having its unique set of criteria for rejecting data, which may vary broadly, even when applied to the same dataset. A significant drawback of this approach is its reliance on a robust pre-trained model, which poses a challenge for scenarios where identifying covariate shift is the primary objective, leaving such cases without a viable solution.Generalized ODIN <cit.> is another direction of work that adopts the model-centric approach. This method replaces the standard classification head and, instead, decomposes the output into scores to behave like the conditional probabilities for the semantic shift distribution and the covariate shift distribution. This approach is then only evaluated on out-of-domain covariate shift such as the DomainNet <cit.> benchmark. Follow-up work by Tian  <cit.> further explores calibrating the confidence functions proposed in <cit.>, which realizes improvements on both semantic and covariate OOD detection. They additionally apply their refinement on in-domain covariate shift, such as CIFAR10 vs. CIFAR10-C.Besides the above-mentioned work, covariate shift has been studied predominantly from a robustness perspective <cit.> or in a domain adaptation setting <cit.>. The defense against adversarial attacks <cit.> is another research direction that falls in the domain of covariate shift. This perspective stems from the recognition that adversarial examples, by nature, often represent data points that deviate significantly from the distribution observed during model training, thereby inducing a form of covariate shift. Researchers have leveraged insights from adversarial robustness <cit.> to devise methods that can identify and mitigate the effects of such shifts, focusing on enhancing model reliability and security against deliberately crafted inputs designed to deceive. Fortunately, the shift introduced is completely artificial and typically a shift targeted at a specific model.




 §.§ Normalizing Flows (NFs)


Consider an image sampled from its intractable distribution as 𝐱∼P_𝐗. Additionally, let us introduce a simple, tractable distribution p_𝐙 of latent variable 𝐳 (which is usually Gaussian). Normalizing Flows utilize k consecutive bijective transformations f_k:ℝ^D→ℝ^D as f = f_K ∘…∘ f_k∘…∘ f_1, to express exact log-likelihoods 


    log p(𝐱) = log p_Z(𝐳_0) - ∑_k=1^K log| d f_k(𝐳_k-1)/d𝐳_k-1|,

where z_k and z_k-1 are intermediate variables, and 𝐳_0 = f^-1(x).

Numerous bijections have been introduced which balance expressivity and have a simple evaluation of the Jacobian determinant in Equation (<ref>). Specifically, coupling flows have seen much success <cit.>. Since they can be parameterized through arbitrary complex functions, we explore conditioning the flow on the frequency components of an image.





 §.§ Typicality
 

Examining sequences of N independent and identically distributed (i.i.d.) datapoints 𝐱_n, the typical set comprises all 𝐱_n that satisfy


    H(𝐗)-ϵ≤ -1/N∑_n=1^N log_2 p(𝐱_n) ≤ H(𝐗)+ϵ,


where ϵ represents an arbitrarily small value and H(𝐗) denotes the Shannon entropy of the dataset. In other words, the empirical entropy of the set approaches to the entropy rate of the source distribution. Leveraging the Asymptotic Equipartition Property (AEP), it is deduced that

    1/N∑_n=1^N log_2 p(𝐱_n)→  H(𝐗)   s.t.  N→∞,


leading to the conclusion that the probability of any sequence of i.i.d. samples of sufficient length approaches unity. Thus, despite the typical set representing merely a small subset of all potential sequences, a sequence drawn from i.i.d. samples of adequate length will almost certainly be considered typical <cit.>. 

In various studies, indications have emerged that NFs perform poorly when the likelihood is utilized as a metric for detecting OOD samples <cit.>. It can be argued that datasets are a typical sequence of samples, rather than high in likelihood, also known as the Typical Set Hypothesis (TSH). Therefore, in the recent work by Nalisnick  <cit.>, an innovative approach is proposed for OOD detection that leverages typicality as an evaluation metric in lieu of likelihood. This methodology was further refined in subsequent studies <cit.>, introducing approximate mass. Motivated by the fact that typical samples are localized in high-mass areas on the PDF, the metric evaluates the gradient of the LL w.r.t the input data, also known as the score. It can be expressed mathematically as ∂ L(x;θ)/∂x, where x denotes the input, L the evaluated LL by the model parameterized by θ, and . represents the Euclidean norm. Despite some criticism on TSH <cit.>, this metric demonstrates superior performance in OOD detection across various benchmarks <cit.>.



§ APPROACH




 §.§ Definition of Covariate Shift

Formally, semantic- and in-domain covariate shifts can be delineated as follows. Consider samples from the training distribution, x ∼ P_X, and anomalous data from an OOD source x̂∼ P_X̂, subject to a low-pass filter (l, l:ℝ→ℝ) to obtain the low-frequency components, x_L = l(x) and the high-frequency components x_H = x - l(x). Semantic shift is characterized by a discrepancy in the marginal probability distributions, P_X_L≠ P_X̂_L, when the conditional probability distributions of high-frequency components remain consistent, P_X_H|X_L≈ P_X̂_H| X̂_L. Conversely, covariate shift is identified when the conditional probability distributions diverge, P_X_H|X_L≠ P_X̂_H|X̂_L, but the marginal probability distributions of the low-frequency components remain the same P_X_L≈ P_X̂_L. Furthermore, these definitions hold with in the supervised setting with predefined targets (Y).



 §.§ CovariateFlow

In the development of methodologies for detecting covariate shift within datasets, several critical factors must be meticulously considered to ensure efficacy and accuracy. Firstly, (1) the process of resizing images can significantly alter the distribution of high-frequency statistics, potentially obscuring key data characteristics. Secondly, (2) the inherent nature of encoding architectures, which essentially function as low-pass filters <cit.>, may limit their capacity to fully capture the complex distribution of noise present within the data. This limitation is particularly relevant as covariate shifts often manifest through alterations in the general image statistics, thereby necessitating a method capable of discerning such nuances. Thirdly, (3) the utilization of only log-likelihood-based evaluation in NFs, has proven to have a predisposition towards low-level semantics and is more sensitive to high-frequency statistics. An effective method should be sensitive to covariate shifts affecting all frequency bands, from noise degradations to contrast adjustments.



In light of the above considerations, Normalizing Flows (NFs) emerge as a particularly suitable candidate for modeling the imaging features essential for detecting covariate shift. NFs are distinct in that they abstain from any form of down-sampling or encoding processes to preserve their bijective property. It is also recognized that NFs prioritize pixel correlations over semantic content <cit.>. However, given the expectation that covariate shift involves changes in high-frequency image statistics, accurately modeling the complete image distribution—including both low-frequency semantics and high-frequency components—presents significant challenges, particularly given the relatively limited capacity of NFs compared to more recent generative models <cit.>.To address these challenges, we introduce a novel method that simplifies the modeling of components critical for covariate shift detection. Our approach involves a filtering strategy that divides the image into separate low-frequency and high-frequency components, thereby allowing the detection system to concentrate specifically on the high-frequency elements to improve detection capabilities. Consider an input signal x and the low-pass filter l, the conditional distribution of high-frequency components (𝐱_H  = x - l(x)) given the low-frequency components (𝐱_L = l(x)) . By recognizing that certain high-frequency components are correlated with low-frequency signals, we can model this relationship conditionally. On this premise, we develop CovariateFlow (Figure <ref>), a novel approach of modeling the conditional distribution between high-frequency and low-frequency components using conditional NFs as

    log p(𝐱_H|𝐱_L ) = log p_Z(𝐳_0) - ∑_k=1^K log| d f_k(𝐳_k-1, x_L)/d𝐳_k-1|.

This formulation sets the foundation for a detection system that is finely attuned to the nuances of covariate shift, enhancing its ability to identify and respond to shifts in high-frequency image statistics. The proposed model is predominantly defined by (1) a signal-dependent layer (SDL) <cit.>, (2) conditional coupling flow <cit.>, (3) an unconditional 1×1 convolutional (conv.) layer <cit.> and (4) uniform dequantization,. The SDL layer and conditional coupling layer are additionally conditioned on x_L. The 1×1-convolution and conditional coupling flow is repeated K times depending on the dataset at hand. We employ a Gated ResNet <cit.> as f_Θ and a checkerboard masking strategy <cit.> in our coupling layers. Figure <ref> depicts a high-level overview of the model architecture. We employ a simple Gaussian filter for l, to decompose the signal into low-frequency and high-frequency components. To minimize any assumptions about the high-frequency components, we use a conventional Gaussian kernel. A kernel with a standard deviation (σ) of one proved to yield the best performance. The coupling layers are depicted in Figure <ref> and the model totals a mere 945,882 trainable parameters with 16 coupling layers. For a detailed description on training details, ablation experiments and inference time comparisons, we refer to Section <ref> in the supplementary material. The code for the model will be made publicly available.





 §.§ Unifying Log-likelihood and Typicality


The inductive bias of NFs towards structural complexity when evaluating with LL has been discussed in Section <ref>. As an alternative, evaluation on typicality using the gradient of the LL w.r.t. the input data, has shown improvements in semantic OOD detection over LL <cit.>. However, it is understood that the metric and model are similarly biased towards certain categories of data <cit.>. As such, we propose to combine LL evaluation with the Typicality score to overcome the limitations of each approach individually. Our approach standardizes both the LL and the typicality scores in terms of their respective training statistics. After standardization, we can transform each metric into an absolute distance from the expected mean. The LL distance and gradient score distance can then simply be added to obtain a unified distance. In this manner, the evaluation is sensitive to all deviations, rather than only being lower in score, thereby reducing the effect of the biases of the respective metrics. The following paragraph gives the details of the mentioned approach.

Consider a sample x∼ P_X with log-likelihood log p(x). Furthermore, we denote the magnitude of the gradients as ||∇_x log p(x)||, i.e. the approximate score. The means for the empirical likelihoods are determined through μ_L=𝔼_P_X[log p(x)], and of the approximate scores with μ_T=𝔼_P_X[ ||∇_x log p(x)|| ]. Similarly, we can denote σ^2_L=𝔼_P_X[(x-μ_L)^2] and σ^2_T=𝔼_P_X[(x-μ_T)^2] for their respective variances. We can then obtain the Normalized Score Distance (NSD) for a new sample x^* as the summation of the standardized L1-norms through








    NSD(x^*) = | log p(x^*) - μ_T/σ_T| + | ||∇_x log p(x^*)|| - μ_L/σ_L|.

Figure <ref> depicts this procedure for two degradation types on CIFAR10. 




 §.§ Datasets


CIFAR10(-C) & ImageNet200(-C): CIFAR10 <cit.> and ImageNet200 with their respective corrupted counterparts, CIFAR10-C <cit.> and ImageNet200-C <cit.>, serve as exemplary datasets for developing and evaluating unsupervised covariate shift detection algorithms. CIFAR10 and ImageNet200 provide a collection of images that encompass a broad range of in-distribution covariate shifts, ensuring a suitable level of diversity. On the other hand, the corrupted versions introduce real-world-like (undesired) degradations, such as noise, blur, weather, and digital effects. Figure <ref> depicts 3 of the 15 effects employed in the ImageNet200(-C) dataset. Images are utilized in their original resolution at 64 × 64 pixels.  CIFAR10-C consists of 19 corruptions in total with images at 32 × 32 pixels. This setup enables testing the covariate shift detecting performance across multiple distortion types and severity levels. In all our experiments we train the models only on the original dataset's training set and then test it against all of the corruptions at every severity level. For CIFAR10 this is the original dataset's test set (ID test) and CIFAR10-C's 19 corruptions at 5 severity levels (95 OOD test sets). Similarly, we treat the ImageNet200 test set as ID test and the 15 corruptions at 5 severity levels from ImageNet200-C as 75 OOD datasets. The datasets follow the OpenOOD <cit.> benchmarks[https://github.com/Jingkang50/OpenOODhttps://github.com/Jingkang50/OpenOOD].




















































§ EXPERIMENTS 


The following section describes the conducted experiments and presents the key results obtained in our investigation. Further detailed experimental results can be found in the supplementary materials. Specifically, we present results on CIFAR10 (Section <ref>), ImageNet200 (Section <ref>) and extensive ablation experiments with the proposed CovariateFlow (Section <ref>).



 §.§ Evaluation Metrics & Models


To evaluate the model's ability to detect OOD covariate shifts, we utilize metrics commonly found in related work: the Area Under the Receiver Operating Characteristic (AUROC) curve and the False Positive Rate (FPR) at a 95% True Positive Rate (TPR). In all our experiments with CIFAR10(-C) and ImageNet200(-C), we use the designated test set (10k samples) to compute each metric. Our contributions include contextualizing the VAE, AVAE, GLOW evaluated with log-likelihood and the DDPM with the reconstruction loss, within OOD covariate shift as baseline models. Furthermore, we evaluate GLOW using typicality and the proposed NSD and CovariateFlow with all the aforementioned metrics. Most models are trained from scratch on the ID data. For the VAE-FRL <cit.>, a method leading in semantic OOD detection, the available pretrained CIFAR10 weights[https://github.com/mu-cai/FRLhttps://github.com/mu-cai/FRL] are employed. A detailed description of the implemented models can be found in Section <ref> of the supplementary materials. 




 §.§ Covariate Shift in CIFAR10 and ImageNet200


Table <ref> showcases various models and their averaged AUROC across all the degradations per CIFAR10-C/ImageNet-C severity level. While some models excel in handling specific types of degradation, only the overall performance is truly relevant, as one typically cannot predict the type of perturbation that will occur in real-world settings. A detailed breakdown of the results per perturbation is shown in Section <ref> of the supplementary materials. 





In Table <ref> it can be seen that models preserving the data dimension and maintaining the high-frequency signal components, such as the DDPM and NF-based approaches, perform best. ImageNet200-C contains fewer noise-based degradations than CIFAR10-C. The NF models evaluated with LL generally perform well on noise perturbations (Table <ref> and Table <ref>) and because of this disparity in the types of degradations present in the datasets, LL evaluation exhibits a drop in average performance from CIFAR10 to ImageNet200. The VAE-FRL is designed to focus on semantic content and thus fails to accurately detect a change in general image statistics. It can be observed that CovariateFlow with NSD consistently outperforms the other methods at every severity level, realizing an average improvement of 5.6% over GLOW on CIFAR10 and 7.8% over GLOW on ImageNet200 when evaluated with NSD. This shows the strength of the proposed NSD metric, consistently improving over just LL or Typicality on both the GLOW model and CovariateFlow. Figure <ref> showcases an explicit example of how NSD consistently performs well under different degradations. 

Section <ref> and Section <ref> presents a comprehensive evaluation of various methods for every OOD covariate shift type between the CIFAR10(-C) and ImageNet200(-C) datasets. Table <ref> focuses on the models' performances across three specific degradations (Gaussian Noise, Gaussian Blur, and Contrast) at five severity levels, that summarize the general results seen across all degradations. ImageNet200-C does not contain Gaussian Blur, but in general, the same trend can be observed between the two datasets for all the employed models. A complete comparison between all the models and their average performance per degradation type (averaged over severity levels) can be seen in Table <ref>.

To evaluate the impact of filter kernel size on performance, we conducted an experiment using CovariateFlow. Figure <ref> illustrates the average AUROC achieved with varying Gaussian filter sizes. The results indicate that a smaller filter yields the highest average performance. Example evaluations from the CovariateFlow (NSD) model are presented in Figure <ref>. Notably, the evaluated scores increase with each severity level, although the rate of increase is not linear or consistently increasing between the different degradation types. The CovariateFlow model is fully invertible and, as such, can generate heteroscedastic high-frequency components. Figure <ref> depicts an example with sampled high-frequency components, the reconstructed image and a comparison between the reconstructed image and the original image. Importantly, the sampling process is stochastic and the sampling range is not limited in the example.












§ DISCUSSION

The findings from our analyses validate our hypothesis that OOD covariate shifts can be effectively identified by explicitly modeling the conditional distribution between low-frequency and high-frequency components. The proposed CovariateFlow, designed to specifically capture this distribution, surpasses other methodologies in detecting covariate shifts in CIFAR10 and ImageNet200. Given the diverse array of subjects and covariate conditions within the corrupted datasets, focusing on this conditional distribution streamlines the model's task, allowing it to concentrate on the most relevant distribution for the detection process.
r0.45


  

    
    < g r a p h i c s >

    The average AUROC obtained with CovariateFlow on ImageNet200 vs. ImageNet200-C (all corruptions) at different filter sizes. The figure depicts the score obtained with evaluating using log-likelihood, typicality and the proposed NSD.
    
  

  


Extending on the analysis with Table <ref>, the VAE-based models show adequate performance in detecting noisy degradations due to their inductive bias towards modeling low-frequency image components. On the other hand, the model falls short for this exact reason when exposed to any blurring or color degradations in the images. The DDPM with the LPIPS + MSE metric, present strong performance on noise and blurring-based covariate shift, but struggles when exposed to color shift. This is likely due to color reconstructions happening earlier in the reconstruction schedule. Consistent with existing literature <cit.>, the NF-based methods evaluated using LL are extremely sensitive to noisy degradations. However, any blurring or color shift is evaluated as being highly probable under the modelled distribution, highlighting the bias of LL-based evaluation towards lower textural content. Employing the newly proposed typicality metric shows the exact opposite behaviour. Both GLOW and the proposed CovariateFlow, fail at detecting noise-based covariate shift, but show remarkable improvements on both blurring and color-based covariate shifts when evaluated with typicality. Combining typicality and LL in the newly proposed NSD metric, accentuates the strengths of each, enabling strong detecting performance across most of the covariates with CovariateFlow. NSD enhances the OOD detection capabilities of both the standard GLOW model and the proposed CovariateFlow, establishing it as a general and robust metric for OOD detection in NF-based models. On the higher resolution images from ImageNet200, the model also shows some effectiveness in distinguishing JPEG compression as OOD, a notoriously difficult perturbation to detect.

When to use CovariateFlow: Despite GLOW (LL)’s slightly superior performance in general noise detection, CovariateFlow, leveraging NSD, proves to be better overall. This provides a clear and general recommendation to the reader: LL is preferred in case strictly noise-based shifts are expected. Without a priori knowledge on the OOD shift type (which is usually the case), CovariateFlow with NSD is optimal.
This work demonstrates that it is possible to detect (even slight) perturbations in a target domain without introducing biases or prior knowledge of these perturbations into the model, unlike some contrastive learning approaches <cit.>. It only assumes access to a sufficiently large dataset that captures the in-distribution covariate shifts and aims to detect any covariate shift outside of this distribution.
 


§ FUTURE WORK & LIMITATIONS

Some concerns can be raised about the complexity of the typicality computation, since test time inference requires a forward pass to compute the LL followed by a backpropagation computation per sample. This increases the memory requirements when deploying the model and decreases the overall inference speed. However, in scenarios where accurate OOD covariate shift is essential, CovariateFlow provides the best accuracy vs. speed trade-offs (see Section <ref>). 

This work primarily focuses on detecting covariate shift, with explicit covariate shifts introduced to assess performance. Many publicly available datasets exhibit both semantic and potential covariate shifts. Although the proposed approach demonstrates effectiveness in CIFAR10 vs. SVHN (Table <ref>), future work should explore domain-specific datasets with limited ID covariate conditions to test the sensitivity of the proposed approach. As depicted in Figure <ref>, the scores acquired through evaluation with CovariateFlow (NSD) correctly increase with each severity level, however not at the same rate for each degradation type. Future work should explore the latent representations of each degradation to potentially aligning these scores with image quality metrics <cit.> for blind image quality assessment applications. 



§ CONCLUSION

This paper explores Out-of-Distribution (OOD) detection, specifically targeting covariate shifts caused by changes in general image statistics. This work introduces CovariateFlow, a novel approach utilizing conditional Normalizing Flows (cNFs) for effectively targeting heteroscedastic high-frequency image components, demonstrating its superior efficacy in detecting OOD shifts across diverse datasets such as CIFAR10(-C) (74.9 % AUROC) and ImageNet200(-C) (72.2 % AUROC). Our analysis reveals that by meticulously modeling the conditional distribution between low-frequency and high-frequency components, CovariateFlow outperforms existing models, particularly when employing the Normalized Score Distances (NSD) metric, which is a synthesis of log-likelihood and typicality evaluations. This approach not only highlights the importance of addressing covariate shifts for enhancing the fidelity of imaging systems, but also underscores the potential of unsupervised generative models in improving machine learning models' robustness against OOD data. 

Acknowledgments: This research was funded by the Philips IGTs and the Xecs Eureka TASTI Project.





splncs04



§ SUPPLEMENTARY 



The supplementary material is organized as follows: Section <ref> describes the implementation details of all the models employed in this paper. Section <ref> has a step-by-step rundown on how we obtain the Normalized Score Distance. Section <ref> provides detailed results on CIFAR10 and CIFAR10-C of the experiments and Section <ref> results on ImageNet200 and ImageNet200-C as described in the Experiments section of the main paper. Finally, we provide a series of additional ablation experiments in Section <ref>.



 §.§ Implementation Details

In this section, we detail the unsupervised training methodologies employed for five distinct baseline models and CovariateFlow aimed at OOD detection. 

VAE and Adversersial VAE: The VAE is trained to minimize the standard ELBO <cit.> loss. Model evaluations using SSIM and KL-divergence presented the best AUROC results. The AVAE model integrates adversarial training <cit.> into the variational autoencoder framework to enhance its capability in generating realistic samples. For OOD detection, one can leverage the reconstruction loss (Mean Squared Error (MSE)), the KL-divergence and the discriminative loss to compute a OOD score. We adopt the implementation described in <cit.>. In both the VAE and AVAE we employ a 4 layer deep network with a latent dim = 1024. The models were trained for 200 epochs following a cosine annealing learning rate scheduler.


VAE-FRL: The VAE with frequency-regularized learning (FRL) <cit.> introduces decomposition and training mechanism which incorporates high-frequency information into training and guides the model to focus on semantically relevant features. This proves effective in semantic OOD detection. We employ the pretrained model as publicly available[https://github.com/mu-cai/FRL/tree/mainhttps://github.com/mu-cai/FRL/tree/main]. For the CIFAR10 experiments, the model consists of a standard 3 layer deep VAE with strided convolutional down-sampling layer, transposed convolutional up-sampling and ReLu non-linear functions. The model has a latent dimension of 200. The OOD score is obtained by the log-likelihood (lower bound in the case of the VAE) minus the image complexity. The formulation is given as

    S(x) = -log p_θ(x) - L(x),

where L(x) is the complexity score derived from data compressors <cit.>, such as PNG.


Denoising Diffusion Probabilistic Model: We implemented the Denoising Diffusion Probabilistic Model (DDPM) following the specifications outlined in <cit.> and as publicly available [https://github.com/marksgraham/ddpm-oodhttps://github.com/marksgraham/ddpm-ood]. The method employs a time-conditioned UNet <cit.> architecture with a simplified training objective where the variance is set to time-dependent constants and the model is trained to directly predict the noise ϵ at each timestep t:

    L(θ) = 𝔼_t,x_0,ϵ[ ϵ - ϵ_θ(x_t)^2 ].

We aim to reconstruct an input x_t across multiple time steps (t), utilizing the DDPM sampling strategy which necessitates t steps for each reconstruction x̂_0,t, with each step involving a model evaluation. To enhance efficiency, we leverage the PLMS sampler <cit.>, a recent advancement in fast sampling for diffusion models, which significantly decreases the number of required sampling steps while preserving or enhancing the quality of samples. For evaluating the reconstructions, we employ both the mean-squared error (MSE) between the reconstructed and the input image, and the Learned Perceptual Image Patch Similarity (LPIPS) metric <cit.>, the latter of which assesses perceptual similarity through deep feature distances. For each of the N reconstructions we compute these 2 similarity measurements. Finally we average these scores (over the two metrics and all the reconstructions) to derive an OOD score for each input, integrating both quantitative and perceptual accuracy assessments.

The model architecture is implemented exactly as described in <cit.>. For training, we set T=1000 and employed a linear noise schedule, with β_t ranging from 0.0015 to 0.0195. The training process spanned 300 epochs, utilizing the Adam optimizer with a learning rate of 2.5e^-5. During the testing, we utilized the PLMS sampler configured to 100 timesteps and, in line with AnoDDPM <cit.>, we only test reconstructions from T = 250. Since we do not intend to detect semantic anomalies in this work and are more interested in high frequency image components, we focus on reconstructions later in the schedule. 

Finally, we experiment with the DDPM model trained on CIFAR10 and evaluated at different reconstruction starting points. Figure <ref> depicts our results obtained with different reconstruction starting points and the average AUROC across all the degradations in CIFAR10-C.
 

GLOW: Normalizing Flows enable OOD detection by modeling the ID data distributions with invertible transformations through a maximize the log-likelihood training objective. We employ the GLOW <cit.> architecture, as publicly available [https://github.com/y0ast/Glow-PyTorchhttps://github.com/y0ast/Glow-PyTorch], in this study. Additionally, following the recent work in typicality (Section <ref>), we train our model with the approximate mass augmented log-likelihood objective as described in <cit.>. We incorporate the approximate mass as a component in the loss function formulation. Let L(x; θ) = log(p(x; θ)) denote the average log-likelihood (LL) of the model, parameterized by θ, evaluated over a batch of input data x. Our revised training objective is expressed as:

    min_θ( -L(x; θ) + α∂ L(x; θ)/∂ x)

where α > 0 signifies a hyperparameter that balances the trade-off between local enhancement of the likelihood and reduction of the gradient magnitude. We employ α = 2 in the GLOW implementation. At test time, we compute the per sample LL and gradient score. These components are used to compute the NSD as described in Section <ref>.


CovariateFlow:
Section <ref> describes the CovariateFlow model proposed in this work. Figure <ref> depicts the architecture and general flow of information during training and when computing the OOD scores. Figure <ref> illustrates a detailed diagram of the low-frequency conditioned coupling steps employed in the model. Additionally, following the image decomposition through the Gaussian filtering, we encode the individual components as 16-bit depth data to avoid information loss. Our model is completely invertible and can thus also generate signal-dependent high-frequency components. The models are prepared following the typicality augmented training objective (Equation <ref>). We use an Adam optimizer (starting lr = 5e^-4) with a one-cycle annealing learning rate scheduler for 300 epochs across all our experiments. The code for the model is available at https://github.com/cviviers/CovariateFlowhttps://github.com/cviviers/CovariateFlow.



 §.§ Detailed analysis of the normalized score distance (NSD)


This section details the computation of the NSD from the LL and typicality score. Figure <ref> depicts this process through the evaluation of the GLOW model applied to three different OOD covariate shifts. In Figure <ref> the LL and typicality (gradient score) of the model subject to Gaussian Noise can be seen. Following the process described in Section <ref>, column 2 depicts the standardization of both scores using validation statistics. This is followed by converting the scores to absolute distance from the expected mean in column 3. The LL distance and gradient score distance can then simply be added to obtain a unified distance (Figure <ref>).
The same flow is depicted in Figure <ref> and Figure <ref> for the model subject to Gaussian Blur and Figure <ref> and Figure <ref> for Contrast change. Following this standardized approach, the change in each measure (LL and gradient score) w.r.t. the validation statistics are utilized and combined to provide a single and effective OOD score. All the results depicted in The Figure <ref> depicts the ID CIFAR10 test scores vs. the OOD CIFAR10-C scores.








 §.§ Detailed Results on CIFAR10 vs. CIFAR10-C

The following section presents detailed results obtained with various models on our experiments with ID CIFAR10 and CIFAR10-C as OOD.

Our analysis examines the reconstruction capabilities of the DDPM across various initial time steps, T. Figure <ref> presents the mean AUROC curve calculated for reconstructions assessed using the LPIPS, MSE, or a combination of LPIPS and MSE metrics at each time step. Notably, at larger time steps (e.g., T=250), the distinction in average reconstruction error between the ID CIFAR10 test set and the OOD CIFAR10-C dataset becomes less pronounced, leading to inferior OOD detection performance. This phenomenon is attributable to the high-level image perturbations characteristic of OOD data, which are predominantly addressed in the final stages of the diffusion process. In Contrast, initial diffusion stages focus on generating lower-level image semantics, resulting in reconstructions that significantly diverge from the test image, particularly in terms of low-frequency components.



Figures <ref>, <ref>, and Table <ref>, highlight the distinct sensitivities of log-likelihood (LL) and gradient scores when applied to GLOW under severe Gaussian Noise conditions, as depicted in Figure <ref>. These metrics diverge in their assessment, with LL clearly identifying distorted images as OOD, whereas gradient scores suggest such images are more typical than even the ID data. Conversely, Figure <ref> demonstrates the opposite trend for blurred images, where LL overestimates their likelihood relative to ID data, but gradient scores accurately classify them as OOD. These observations corroborate Zhang 's theoretical insights <cit.> about the propensity of certain model-metric combinations to misjudge the probability of natural images. To address these discrepancies, we introduce the NSD metric, which synthesizes LL and gradient movements into a unified OOD detection metric. Figures <ref>, <ref> and <ref> validate the NSD metric's effectiveness in discerning OOD samples across both conditions, with extended results available in the supplementary material.



Table <ref> depicts the AUROC for 3 degradations (each severity) from CIFAR-10C that summarizes the performance of all the models employed in this work. Figure <ref> additionally depicts the average AUROC of all the models at each severity. We also present the complete performance evaluation of all the models on CIFAR10-C on all the degradtions and at every severity level. The results are depicted in order of presentation: DDPM T150-LPIPS (<ref>), DDPM T20-LPIPS+MSE (<ref>), VAE (Table <ref>), AVAE (Table <ref>), GLOW-LL (Table <ref>), GLOW-Typicality (Table <ref>), GLOW-NSD (Table <ref>), CovariateFlow-LL (Table <ref>), CovariateFlow-Typicality (Table <ref>) and CovariateFlow-NSD (Table <ref>).





















































 §.§ Detailed Results on ImageNet200 vs. ImageNet200-C

The following section depicts detailed results obtained with various models on our experiments with ID ImageNet200 and ImageNet200-C as OOD. The results are depicted in order of presentation: DDPM T20-LPIPS+MSE (<ref>), GLOW-LL (Table <ref>), GLOW-Typicality (Table <ref>), GLOW-NSD (Table <ref>), CovariateFlow-LL (Table <ref>), CovariateFlow-Typicality (Table <ref>) and CovariateFlow-NSD (Table <ref>).





















 §.§ Ablation Study

This section details a series of ablation experiments conducted, including an analysis of the effect of the individual components in CovariateFlow on the detection performance (Table <ref>), mean scores per severity of the models and resource aspects are depicted in Table <ref>, model performance on a typically semantic OOD detection problem in Table <ref> and finally an example (Figure <ref>) of heteroscedastic high-frequency components sampled from the fully invertible CovariateFlow.

In our ablation experiments, we test the effect of explicitly modelling the conditional distribution between the low-frequency and high-frequency signal components as described in Section <ref>. This is achieved by training and evaluating the CovariateFlow model in four different settings: (1) unconditional coupling flows with the full input image, (2) unconditional coupling flows subject to only the high-frequency components of the image, (3) unconditional coupling flows subject to the high-frequency components and a conditional signal-dependent layer additionally subject to the low-frequency image components and finally, (4) the high-frequency image components applied to the conditional coupling flows and a signal dependent layer subject to the low-frequency components. For each of these implementations we follow the exact same training methodologies as described in Section <ref>. All the images are encoded at 16 bit depth during dequantization to ensure comparability. 



From Table <ref> it can be seen that while model 1 is limited in modelling the complete data distribution (11.32 Bits per dimension (BPD)), it performs well on detecting OOD covariate shift with NSD (AUROC 67.8%), comparable to the performance obtained with GLOW. Only using the high-frequency image components in an unconditional setting (model 2) yields a somewhat lower OOD detection performance of 65.5% AUROC. Introducing the SDL (model 3), lowers the mean BPD and improves on LL-based OOD detection (58.4%), but adversely effects the NSD evaluation (62.6%). While the SDL layer does not show improvement in the detection performance, it significantly aided in satabalizing model training. Finally (model 4), conditioning every coupling flow in the network on the low-frequency content significantly improves modelling the high-frequency components (9.77 BPD → 5.48 BPD), indicating the value in the additional information. Modelling this conditional relation between the low-frequency and high-frequency components also proves very effective in detecting OOD covariate shift. The model achieves a mean AUROC of 74.9% at detecting covariate factors across all variations and degredations when evaluated with NSD. 



Table <ref> presents additional information about each of the models employed in this research. This table showcases the mean distance measurements (CIFAR10-C), taken under different evaluation criteria, across increasing severity levels of covariate shifts within the dataset. Such a detailed breakdown allows for a nuanced understanding of each model's resilience and adaptability to changes in input data distribution. Notably, the LL evaluations of GLOW at the highest severity level encountered numerical stability issues, leading to the substitution of some results with the maximum representable floating-point number. This adjustment, while necessary, underscores the challenges in maintaining computational integrity under extreme conditions and the importance of implementing robust handling mechanisms for such anomalies. It is evident from the data that there is a consistent trend of increasing mean distance scores across all models as the severity level escalates, highlighting the impact of covariate shift on model performance. This trend underscores the ability to quantify covariate shift, although only briefly evaluated here. Furthermore, the table delineates the model size, quantified by the number of trainable parameters, and the inference speed, measured in milliseconds. These metrics are critical for understanding the trade-offs between model complexity, computational efficiency, and performance.The data presented in Table <ref> not only elucidates the ability to quantify covariate shifts, but also emphasizes the importance of balancing model complexity and computational efficiency when considering the model deployment conditions. 

r0.45


  1.0!
  
  
          OOD SVHN <cit.>

      Models     AUROC ↑ 

  1-3
  [t]5mm12*[origin=c]90CIFAR10 ID     Reconstruction

  2-3
      DDPM <cit.>     97.9*/ 95.8 

   2-3
      Explicit Density

   2-3
      Vanilla VAE (SSIM + KL Div)     24.4

      AVAE (MSE + KL Div + Adv Loss)      32.0

      VAE-FRL <cit.>     85.4* 

      GLOW-FRL <cit.>     91.5* 

      GLOW (LL)     0.7

      GLOW (Typicality)     91.3

      GLOW (NSD)     89.9

      CovariateFlow (LL)     0.3

      CovariateFlow (Typicality)     89.9

      CovariateFlow (NSD)     90.0

   
  The performance of various models on detecting SVHN as OOD when trained on CIFAR10 as ID. * indicates values taken from the published paper.
  
  



Modeling the conditional distribution between the low-frequency and high-frequency components using CovariateFlow is highly effective in detecting out-of-distribution (OOD) covariate shifts. The CIFAR10 dataset, known for its diversity, encompasses a range of in-distribution (ID) covariate conditions. When assessing CovariateFlow in the context of a semantic OOD detection problem, such as distinguishing between CIFAR10 and SVHN datasets, it is plausible that some covariate conditions in CIFAR10 overlap with those in the SVHN dataset. Despite this potential overlap, CovariateFlow demonstrates robust performance in identifying the OOD covariate conditions present in the SVHN dataset, as evidenced by the results shown in Table <ref>.  Although the DDPM (utilizing all 1000 starting points) achieves the best performance, CovariateFlow offers competitive results. This is notable given its significantly smaller size and its specific design focus on covariate conditions rather than semantic content. 


