Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning
Raphael Lafargue Luke Smith Franck Vermet Mathias Löwe Ian Reid Vincent Gripon Jack Valmadre

 Vacuum Radiation Pressure Fluctuations on Electrons
    L. H. Ford
    September 9, 2024
====================================================




§ ABSTRACT 
The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e. allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs computed with and without replacement. These reveal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FSL comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, we explore methods to further reduce the (size of the) CI by strategically sampling tasks of a specific size. We also introduce a new optimized benchmark, which can be accessed at <https://github.com/RafLaf/FSL-benchmark-again>. 




§ INTRODUCTION





The recent surge of interest in few-shot learning (FSL), driven by its potential applications in many real-world scenarios, has led to a proliferation of new methods and novel experimental protocols <cit.>. If in conventional machine learning it is common to benchmark methods using a fixed split into training and validation sets, FSL presents unique challenges due to its reliance on extremely small and, consequently, biased training datasets. In fact, the performance of FSL can  dramatically depend on the choice of the given labeled training samples <cit.>.

One question that FSL shares with conventional machine learning is that of the best performing methods. 
Especially relevant to FSL, the high variance of measured performance based on the choice of labeled data has led practitioners to quickly adopt the standard of aggregating statistics over a large number of artificially generated tasks, stemming from a single (or a few distinct) dataset(s). The predominant approach is to generate artificial few-shot tasks by randomly sampling the same dataset with replacement, i.e. permitting the same samples to appear across multiple tasks. The outcome of these numerous tasks is the calculation of an average accuracy and its associated confidence interval (CI) for each method, thereby providing researchers with a statistically relevant basis for comparing the efficacy of different methods.

By allowing the same samples to appear in multiple tasks, the computed CIs account for the randomness of the sampler but not the data itself. In fact, 
the computed CIs use the usual
Lindeberg-Lévy Central Limit Theorem (CLT). Hence, for 
these CIs to be statistically valid, the underlying random variables must be independent and identically distributed (IID). This means that the currently reported CIs should be understood as a likely range of outcomes if the experiment were reproduced using exactly the same data, which we will refer to in the remainder of this paper as Closed CIs (CCIs). This contrasts with what is often of interest in many areas of machine learning: the range of outcomes if the experiment were repeated with data from the same underlying distribution, termed Open CIs (OCIs). Note that the latter could be obtained simply by sampling tasks without replacement, but at the cost of considerably restricting the number of different tasks one can generate for a given dataset. This limitation is even more severe if the dataset is small, resulting in potentially larger CIs and inconclusive comparisons between methods.


r0.55 
0.55! 
  
         2cCLIP     2cDINO 

          NCC     FT     NCC     FT 

2*CLIP     NCC                                                                                 

         FT                                                                               

     2*DINO     NCC                                                                           

         FT                                                                        





Comparison between different methods for few-shot classification. Each entry consists of three elements: [With Replacement (Closed), Without Replacement (Open), Paired Tests (PT)]. Symbols  and  respectively indicate significant differences (positive and negative) between the row method and the column method while  is non-conclusive. Results derive from the DTD test split (bottom left triangle) and Traffic Signs (top-right) split of MetaDataset, with task sampling at 5 shots, 5 ways, and 15 queries. Note the inversion in bold. NCC (Nearest Class Centroid), FT (Fine-tune) with CLIP and DINO as feature extractors.



The purpose of this paper is  to highlight this crucial consideration when computing CIs.  We propose strategies to address this issue and obtain  meaningful comparisons while still accounting for the randomness of the data. These strategies rely on a) Paired Tests (PT), where methods are evaluated on the same set of generated tasks, and b) adequately sizing tasks. Throughout the paper, we focus on the specific case of few-shot classification in vision, the most popular area of research in the field of FSL.

Our investigation using Open Confidence Intervals (OCIs) can lead to conclusions that are inconsistent with those obtained using the classical approach in the field of few-shot learning. In particular, we find that some methods previously reported as statistically significantly outperforming others are actually indistinguishable when using OCIs, and vice versa. In addition, we show cases where the use of CCIs lead to (statistically significant) conclusions that are diametrically opposed to those obtained using PT. One such example is given in Table <ref>, where we compare different methods for few-shot classification depending on their used feature extractor (here CLIP <cit.> or DINO <cit.>) and their adapting methods (here Logistic Regression (LR), Nearest Class Centroid (NCC) or Fine-Tuning (FT)). In the table, we report three conclusions for each pair of model and method combinations: the first is obtained from the methodology described in <cit.> where the authors used the predominant way to compute CIs, that is CCIs; the second OCIs, and the third is based on PT. A conclusion that the row method is better than the column one is denoted with ,  means there is no conclusive statement, and  that the column method is the most performing of the two. The upper triangular values in blue correspond to the Traffic Sign test split while the rest of the table (in red) refers to the DTD test split in the Metadataset benchmark by <cit.>. Particularly striking results are found on the Traffic Signs dataset. Indeed, when studied with replacement tests, CLIP with the NCC adapter underperforms DINO with Fine-tuning, yet this outcome is reversed in paired test assessments. This clearly demonstrates the importance of distinguishing between measurement methods, as failing to do so can lead to significant misinterpretations of results.

The main contributions of this work are:

    
  * We highlight the importance of considering data replacement when computing CIs for FSL methods comparison. Our study illustrates the impact on CI ranges when transitioning from closed to open CIs on standard off-the-shelf vision datasets.
    
  * By implementing paired evaluations, wherein multiple methods are compared on identically generated task sets, we demonstrate the ability to reach conclusive comparisons more frequently than when relying on simple performances with OCIs. 
    
  * We investigate how to optimize task generation from a given dataset, taking into account its size and number of classes, to reach small ranges of CIs and lead to more conclusive comparisons between methods. The result is a benchmark that can be used for few-shot classification of images.






























§ CLOSED CIS VS. OPEN CIS

In this section, we are interested in better quantifying the difference between CCIs and OCIs. For this purpose, we define notations and outline algorithms for task sampling. We present a theoretical analysis of the differences between CCIs and OCIs. Finally, we empirically compare their ranges on real datasets. 




 §.§ A mathematical description of the problem






  §.§.§ Standard Evaluation and Notations


The predominant method of evaluation in the field of few-shot classification is described in Algorithm <ref>. A few-shot classification task 𝒯 = (𝒦, 𝒮, 𝒬) comprises a set of classes 𝒦, a support set 𝒮 = {𝒮_c}_c ∈𝒦 and a query set 𝒬 = {𝒬_c}_c ∈𝒦 where 𝒮_c, 𝒬_c denote the sets of support and query examples for each class c ∈𝒦.
Let K = |𝒦| denote the number of ways (i.e. classes in a few-shot task), S = |𝒮_c| the number of shots per class, and Q = |𝒬_c| the number of queries per class (for simplicity, we assume the classes to be balanced).
Few-shot evaluation is typically performed by constructing many tasks from a larger evaluation dataset.
An evaluation dataset 𝒟 = (𝒞, 𝒳) comprises a set of classes 𝒞 and examples for all classes 𝒳 = {𝒳_c}_c ∈𝒞.
Let C = |𝒞| ≥ K denote the number of classes, and N = |𝒳_c| ≫ S + Q the number of examples per class.
As highlighted in the introduction, the rationale followed by the CI computation predominant method is to consider 𝒟 to be fixed and non-probabilistic. 


0.9 






The standard few-shot task sampler constructs T random tasks with K ways, S shots and Q queries from a dataset 𝒟 as outlined in Algorithm <ref>.
This procedure introduces additional random variables (besides the dataset itself) in the selection of classes and examples.
Let 𝒯_t = (𝒦_t, 𝒮_t, 𝒬_t) for t = 1, …, T denote the sampling of tasks.
The average accuracy on each task is obtained as:

    A_t = 1/K Q∑_c ∈𝒦_t∑_x ∈𝒬_t c1[f_𝒮_t(x) = c],
 
where f is the evaluated model, conditioned by the support set. 

Next, we turn to the actually reported metric which is the average accuracy over several tasks. 



  §.§.§ Computing Confidence Intervals


We compute the accuracy A_t of the method on each task, and then take the mean across tasks A̅ = 1/T (A_1 + ⋯ + A_T). As such, we obtain the following formula for the variance:

    [A̅] = [A_1] + … + [A_T]/T^2 = [A]/T  .


Assuming a sufficient sample size and that the mean
A̅ is normally distributed according to the Central Limit Theorem, the 95% confidence interval is obtained as A̅± 1.96 σ_A̅ using the formula for standard error (standard deviation of the sample mean):

    CI  = 1.96 σ_A̅ = 1.96 σ_A/√(T)  .

Note that in the case of a very small number of tasks, Student's distributions can be used instead. Also, we took the example of 95% CIs, which is arbitrary but very common in the literature. For more generality, we consider a probability p_limit in the following for all theoretical considerations, and stick with the 95% value for experiments.







Note that, as the number of tasks becomes larger, it is inevitable that many tasks will re-use examples, since tasks are constructed independently with replacement.
Therefore, as T becomes large, the variance of the sample mean will approach the conditional variance [A̅|𝒟], and the confidence interval will represent the likely range of outcomes if we were to repeat the experiment with a random set of tasks on the same dataset. In other words, it provides no insight into how well a method would generalize on a distribution.
On the other hand, if T is chosen small enough, there will be minimal re-use of examples and the assumption of independence may approximately hold, although the confidence interval may be significantly larger.

We will now conduct an empirical evaluation of the disparity between closed (tasks sampled with replacement) and open (tasks sampled without replacement) CIs on real datasets.




 §.§ Are OCIs larger than CCIs? An empirical study



In contrast to Algorithm <ref>, the task sampling without replacement is presented in Algorithm <ref> (see Appendix). Note that we make an explicit use of a Student's law estimator in this algorithm as it is expected that for some small datasets it can only generate but a few independent tasks.
In the latter, the total number of tasks T is determined directly by the sampling process, thanks to a specific stopping condition, based on the exhaustion of the dataset. This is done in an effort to minimize the obtained CIs' ranges. Since samples cannot be sampled twice, we can consider that the classes and examples are drawn IID from an underlying data distributions p(C) and p(X | C). This is why OCIs also account for the randomness of the data.

In our experiments, we utilize datasets from the Metadataset Benchmark as referenced in <cit.>. This benchmark comprises 10 datasets, out of which we employ 9, excluding Imagenet, to focus on cross-domain results in line with the recent trend in the literature <cit.>.  These include Omniglot (handwritten characters), Aircraft, CUB (birds), DTD (textures), Fungi, VGG Flowers, Traffic Signs, Quickdraw (crowd-sourced drawings) and MSCOCO (common objects)  <cit.>.

<cit.> details few-shot accuracies for 2000 tasks with 5-shots, 5 ways, and 15 queries in a comprehensive table covering various works on the Metadataset datasets. Our study's only difference lies in the adoption of the T=600 setting, a more prevalent choice in existing literature. If CCIs are found to be narrower than OCIs with this smaller T, it will be even starker with T=2000 tasks as shown in Equation <ref>. Our primary reference for methods and models is the comprehensive compilation provided by <cit.>, a foundational starting point for our experiments.


Our findings are detailed in Table <ref>, showcasing results across different few-shot methods and datasets. Firstly, there is a noticeable homogeneity in CCIs, arising from the fixed number of tasks set at  T=600, which contrasts with the variability observed in OCIs. Interestingly, CCIs are substantialy narrower than OCIs for small datasets such as Aircraft and DTD. Conversely, in the case of larger datasets like Quickdraw, CCIs become larger than OCIs due to T=600 being insufficient to deplete the dataset. Indeed, Aircraft and DTD's test splits contain 1,500 and 840 samples respectively, whereas the test splits for MSCOCO and Quickdraw have much larger sizes of 152,000 and 7.7 million samples respectively. Across various datasets, models and methods, CCIs are on average 3.8 times larger than OCIs. These results highlight the imperative need for accurate interpretation of Confidence Intervals, given the dramatic differences between OCIs and CCIs ranges, that undoubtedly lead to disagreeing conclusions if misinterpreted.

We also notice that for cases where methods reach accuracies near 100%, like adaptation methods using CLIP (unlike those using DINO) on the CUB dataset, both types of CIs become narrower. This is due to accuracy saturation at 100%, which reduces the standard deviation of accuracies.











In the following, we delve into the conclusiveness of comparative studies using CCIs or OCIs.




 §.§ Impact on Conclusiveness


First let us recall how confidence intervals are used to draw conclusions when comparing methods. Suppose we have two variables of interest x_1 and x_2, with their corresponding p_limit - confidence intervals (a generalized version of 95%-CIs) [x̅_1 - δ_1, x̅_1 + δ_1] and [x̅_2 - δ_2, x̅_2 + δ_2]. To draw conclusions about the fact x_1 is smaller than x_2, we proceed as follows: if the two intervals do not intersect, and x_1 + δ_1 < x̅_2 - δ_2, then:













    P(x_1<x_2)    > P(x_1 < x̅_1 + δ_1  x_2 > x̅_2 - δ_2 ) = (1-1 - p_limit/2)^2 > p_limit ,

where the (1-p_limit)/2 part comes from the symmetry of the Gaussian distribution.






    
    
    
    



r0.50
    

    
    < g r a p h i c s >

    Scatter plot of task accuracies using two different combinations of feature extractor/adaptation methods on the Traffic Signs benchmark.
    

    

With this in mind, the results listed in Table <ref> can lead to different conclusions depending on whether CCIs or OCIs are used.
For example, using the DTD dataset, CCIs lead to conclusive comparisons between both backbones and methods, whereas OCIs are inconclusive as they all intersect. Again, this should not be seen as a contradiction, but rather as having different paradigms about the comparisons. CCIs compare methods if they were reused on the same data, whereas OCIs focus on the underlying distribution. Next, we propose two methods to improve conclusiveness when comparing methods, namely paired tests in Section <ref> and task sizing in Section <ref>.





§ PAIRED TESTS




 §.§ Definitions


As an effort to reduce the ranges of Open Confidence Intervals (OCIs), we propose to make use of paired tests.
Indeed, as we pointed out in the introduction, FSL tasks have a vast diversity in difficulty, leading to a high variance in accuracy across tasks. It is noteworthy that a task deemed hard for method A often aligns in difficulty for method B. This parallel in task difficulty across different methods was previously identified by <cit.>, and our findings are in agreement, as illustrated in Figure <ref>, where we plot the accuracies on tasks generated from the Traffic Sign dataset and using two different combinations of feature extractors and adaptation method. In the provided figure, a strong correlation, quantified at 0.675, is evident between two distinct methods. This highlights the potential for reducing the variance in accuracies resulting from task sampling by employing paired testing.

Formally, let us denote Δ_t = A_t - B_t the accuracy of the method on task t relative to a method with accuracy B_t, with Δ̅ = 1/T∑_t = 1^TΔ_t the mean difference across tasks.
While the mean of differences is simply the difference of the means, 𝔼[Δ̅] = 𝔼[A̅] - 𝔼[B̅], the variance of the difference may be significantly reduced when the accuracies are positively correlated:

    [Δ̅]
    = [ 1/T∑_t = 1^TΔ_t]
    
    = 1/T[ A_t - B_t ]

since [X - Y] = [X] + [Y] - 2  (X, Y).

The lower variance of Δ_t compared to A_t results in a correspondingly smaller confidence interval, as detailed in Equation <ref>. Consequently, this leads to scenarios where two methods can exhibit significant differences when analyzed using paired testing despite there being no significant differences when directly comparing accuracies.




We performed experiments comparing various methods to fine-tuning (FT). Results are shown in Table <ref>, where each line corresponds to a specific dataset and feature extractor and each column to a combination of an adaptation method and a feature extractor. Two conclusions are depicted, the first being based on directly comparing accuracies, and the second using paired tests instead. Note that in all cases, we rely on OCIs, that is to say sampling without replacements. Let us also notice that paired tests never lead to contradictory conclusions with direct comparisons of accuracies, a direct consequence of previous remarks about the mean of differences. Their difference lies in the ability to conclude or not.








 The fine-tuning adaptation method is selected as the baseline in Table <ref>, primarily due to its substantial computational cost. The cost of fine-tuning stems from the necessity to update the weights of the entire feature extractor for each considered task. The objective is to determine if it surpasses more cost-effective methods in performance. In comparative analyses excluding the comparison between CLIP and DINO and focusing only on the same feature extractor, it is frequently noted that fine-tuning either underperforms relative to other methods or the results are inconclusive. Indeed, fine-tuning significantly outperforms other methods in only 4 out of 36 cases, while underperforming in 14 cases. The outcomes in the remaining instances remain inconclusive. Consequently, we conclude that fine-tuning, in light of its considerable computational overhead, is not particularly advantageous to consider. These results should be nuanced by the dependency of fine-tuning on hyperparameters which makes any assertions about this method contingent upon a specific set of hyperparameters <cit.>. 

For each of the nine considered datasets, we have a total of 135 unique comparisons, taking into account only distinct pairs of (model, methods) across two models and three methods. We found that 57 comparisons were conclusive using direct comparison with OCI while 94 were conclusive using paired tests. Figure <ref> in the Appendix illustrates this.


Table <ref> illustrates, on the DTD and Traffic signs dataset, that the three different approaches for computing CIs discussed so far result in varied assessments of the significance of differences between two methods. On all datasets, in 27 instances out of the 114 where the comparison with replacement was conclusive, that is ∼ 23% of such cases, a pattern emerges: a comparison initially classified as significantly different becomes non-conclusive under sampling without replacement, and then conclusive again when a paired test is applied. On 12 instances, ∼ 11% of previously considered cases, conclusiveness is not confirmed by the paired test. 


Particularly striking is one instance of inversion, where a method previously deemed significantly more accurate than another was found to be significantly less accurate using a paired test. This reversal was observed in the comparison of Fine-tuning on DINO versus NCC with CLIP features on the Traffic Signs dataset. This implies that a method can significantly outperform another on a specific dataset, yet significantly underperform when evaluated across the entire distribution. The dataset is thus a particular instance of data that favors one method. This example powerfully exemplifies the need for clarity when interpreting CIs.











A natural question that arises from previous considerations is that of how to size tasks when performing sampling without replacement, aiming to reduce the range of obtained CIs. In that matter, we consider the size of the support set to be fixed at K S, leaving as the only free variable the number of queries per task and per class Q. Indeed, increasing the number of queries will inevitably reduce the total number of tasks we can construct, as shown in the following equation. Assuming a balanced dataset, we can estimate the number of tasks T that can be sampled by exhausting the full dataset:

    T ≈⌊|𝒟|/|𝒯|⌋≈⌊CN/K(Q+S)⌋ ,

with |𝒯| is the number of samples, accounting for both the support and query sets, in each task.




§ SIZING TASKS TO NARROW OCIS


r0.45 
  
    < g r a p h i c s >

  Variance of the average accuracy vs. the number of queries with synthetic data. The two classes are represented as 1D Gaussians 𝒩(-1, 1) and 𝒩(1, 1). The size of the dataset is N=1000 (500 samples per class). Tasks are sampled according to Algorithm <ref>. The number of shots is set to 5. We fit this with the model described in Equation <ref> and observe a strong fit of the model with our experiment.
  



If increasing Q reduces the number of tasks, it also changes √((A_t)) which is proportional to the CI. As such, there exists a trade-off between the number of queries and the feasible number of tasks that can be generated to minimize OCIs for any given dataset.

Intuitively, measuring A̅ with a small T (and consequently a high Q) results in extensive CI ranges, a phenomenon depicted in Equation <ref>. Conversely, measuring with Q=1 may generate many tasks (large T) with an extremely high variance because the accuracy per class becomes either 0% or 100%. In the following, we aim to identify the optimal number of queries, denoted Q^*, that effectively minimizes the variance of the average accuracy (A̅) and thus the obtained CIs. We first demonstrate mathematically the existence of such minimum by deriving (A̅).



We show in the Appendix that (A̅) can be written as: 


    ( A̅) = K/NC (α Q+ β/Q + γ),

with α, β and γ also defined in the Appendix, and β >0.


These parameters are difficult to estimate in particular when dealing with real datasets and methods. If α≤ 0, then ( A̅) is decreasing as a function of Q since Q∈ℕ. In the following, we focus only on cases where α>0. This choice is supported by empirical evidence, which we will present later, indicating a U-shaped relationship between the variance of A̅ and Q for a certain range of S.  Assuming this, ( A̅) reaches its minimum at Q^*=√(β/α). Next, we study what this entails as S and N vary. 








 §.§ Effect of S and N on Q^*



 Given the definition of α and β obtained in Equation <ref>, we find that Q^* is an increasing function of S and a constant function of N. In this section, we show that these results are confirmed  empirically on real datasets.


 
We propose to study the aforementioned variance model of A̅ with respect to Q in a simplistic 1D representation of samples. In our model, two class' distribution are represented as two Gaussians (𝒩_i = 𝒩(μ_i, σ_i) with i∈{1,2}). We then sample an artificial balanced dataset of fixed size N. Next, we sample tasks within this artificial dataset until exhaustion with the procedure described in Algorithm <ref> setting K=C=2. Using the NCC classifier we obtain a set of accuracies on which we can compute the average accuracy A̅. This procedure consisting of instantiating the synthetic dataset from Gaussians and measuring A̅ is iterated. This yields a set of {A̅_j }_j for a given set of parameters {S,Q,N,𝒩_1,𝒩_2} from which we compute an empirical variance (A̅). 



In Figure <ref>, we show the measured (A̅) vs Q. We use 1000 samples in the datasets, split between the two classes. We set the number of shots to S=5. The model in Equation <ref> is fitted very precisely. The discretisation effect seen at high Q is due to the low number of tasks. Next, we study the effect of S and N and compare it to our experiments with synthetic data.



Increasing S shifts the curve's minimum from Q^* =1 towards Q^* → +∞ as depicted in Figure <ref>. This aligns with our model's predictions. At S=1, opting for Q^*=1 effectively has two effects: (a) the high variance of A_t due to small support and query sets increases the variance of A̅ and (b) low Q allows a significantly larger T, thus reducing the overall variance of A̅ as shown in Equation <ref>. Conversely, for S≥ 20, the setting boils down to classical transfer learning. Indeed, the narrowest CI is attained with one task with a large support and query set task.
Finally, we find a third regime where, for a range of values of S, Q^* is nontrivial. It corresponds to what is shown in the S=5 regime in Figure <ref>.


When increasing N, Q^* should not be affected according to Equation <ref>. However, we observe a hardly perceptible shift of Q^* in Figure <ref> when N is increased. We consider this effect to be sufficiently small and therefore negligible. Next, we explore how these results extend to real-world datasets.















































  §.§.§ Real Dataset Experiments

We now shift our focus to the findings derived from real image datasets. These datasets are often unbalanced and exhibit a variety of class numbers. Our objective is to determine whether the earlier conclusions remain valid in this context.

First, we observe that Q^* does not depend on the size of datasets. However the size of the dataset scales T which in turn scales CI_95%. Similar to the results with synthetic data, we observe a discretisation of the confidence interval in the high Q (low T) regime.

We also observe a similar phenomenon with different regimes in 1, 5 and 10 shots in Figure <ref>. Our analysis suggests that for tasks in the 1 shot regime, the best number of queries, Q, should be set to 1. For tasks with S=5 and S=10, the best values for Q are approximately Q=5 and Q=7, respectively. Figure <ref> also clearly shows that the predominent 15 queries is not optimal to narrow the OCI. We also show similar values for Q^* using DINOv2 instead of CLIP in the Appendix (Figure <ref>).





§ BENCHMARK PROPOSAL

Building on previously obtained results, we propose a simple benchmark where Paired Tests are used and the value of Q is chosen as the minimum found in the previous paragraph. Implicitly, we are assuming that the minimum of Δ's OCI is also reached at Q^*. More precisely, we are assuming the independence of the covariance with respect to Q. These assumptions are backed by the improved number of conclusive comparisons in Figure <ref> when optimizing Q and using paired tests. Indeed, while PT yieled 94 conclusive comparisions, PT with optimized Q yields a little more with 97 conclusive comparisons.



We present our results with the baselines adaptation methods previously studied in Table <ref> using DINOv2 as our baseline model. Our experiments consistently show that, for a given model, fine-tuning tends to be less effective than both logistic regression and the nearest class centroid methods. We also observe the choice of model is primordial with a clear advantage of DINOv2 over CLIP and DINO on most datasets. Our benchmark, including the code, seed values, task descriptions, and accuracy results, is available for use.





§ RELATED WORK




  
Few-Shot Learning Since the seminal works of <cit.> and <cit.>, the field of few-shot learning has known many contributions. Most solutions rely on the use of a pretrained feature extractor, trained on a generic abundant dataset <cit.>. The feature extractor can then be used as is on the target problem <cit.>, or adapted before classifying <cit.>. Most proposed methods differ in the way they combine the pretrained feature extractor and their adaptation to the target problem <cit.>. Over the years, multiple benchmarks have been proposed, including MiniImageNet <cit.>, Omniglot <cit.> and TieredImegenet <cit.>. If initially, these benchmarks focused on the in-domain case, where the feature extractor is trained on disjoint classes from the target problem, but all drawn from the same initial dataset, the trend evolved with the introduction of Meta-dataset <cit.> (MD) and later COOP <cit.>, where the feature extractor is trained on a large generic dataset and applied to various other domains, including fine-grain problems, embodying a cross-domain evaluation.


A few papers focus on the sampling of tasks of targeted difficulty for few-shot learning <cit.>. In <cit.>, the authors claim that model performance can be improved by sampling meta-learning tasks of increasing difficulty. In other works, failed meta-training tasks (deemed hard) are sampled again <cit.> or previously misclassified samples/classes are more likely to be sampled in following tasks <cit.> to focus the model on difficult tasks. Estimating task difficulty can itself be a difficult task, and several solutions have been proposed <cit.>. 
The idea of difficulty-based sampling proposed in <cit.> is relevant to this paper since it enables the sampling of groups of tasks with homogeneous difficulty, effectively reducing the confidence interval ranges. In contrast, our research adopts paired tests, a method that obviates the need for such dependencies and provides a more universally applicable approach. Paired tests is not a novel contribution of our work. They were introduced over a century ago to study the evolution of small populations with time <cit.>. In these seminal studies, the authors shown that individual differences provide more statistical power and insights than changes in averages over the whole population.







  
Confidence Intervals Confidence intervals were established by Polish mathematician Jerzy Neyman <cit.> in the early 1930's coinciding with Fisher's ideas although supported by a different framework. They were more generally used and then required in medical research around the 1980's. They require assuming a specific model for the distribution of the considered data. In contrast, the bootstrap method, introduced in <cit.>, offers a distribution-agnostic approach for estimating ranges. We focused on traditional confidence intervals as they are better understood and more often implemented in the literature of few-shot learning, but similar conclusions could be drawn using Boostrap instead.

















  
Challenges in Statistical Interpretation and Methodological Biases This issue of misleading CIs is not isolated to our domain. <cit.> have similarly criticized the common misinterpretations surrounding confidence intervals in broader scientific research. Moreover, the propensity to overlook or underreport negative or null (non-conclusive) results further exacerbates the problem of biased interpretations. <cit.> argues for the importance of acknowledging and analyzing negative results in computer vision. Lastly, the impact of dataset biases on the evaluation metrics has been well-documented by <cit.>. In their work, “Unbiased Look at Dataset Biases”, they identify and measure several biases such as selection, capture and negative set. It serves as a critical reminder of how dataset-based results can differ from those obtained in real-world distributions.







§ LIMITATIONS



A first limitation of our study we would like to point out is that for large datasets such as MSCOCO or QuickDraw, the predominant method of CI calculation leads to intervals that are actually larger than our proposed OCI. As such, on such large datasets using CCI may not be an unfair approximation.

Furthermore, our mathematical modeling only explains the origin of the minimum of CI with respect to Q but does not provide a way to find it analytically since we cannot easily estimate α, β and γ in Equation <ref>.

As mentioned at the end of paragraph <ref>, saturation at 100% accuracy may negatively impact the computation of confidence intervals and particularly the value of paired tests CI in Equation <ref>.



Finally, we point out that paired tests introduce complexity as they require a fixed seed and necessitate saving and publishing individual task accuracies when using the benchmark and comparing methods.





§ CONCLUSION


In our study, we demonstrated the stark contrast between Open and Closed measurements of method accuracy in Few-Shot Learning. Notably, OCIs take into account data randomness but are far wider than CCIs. We identified two major approaches that contribute to narrowing the OCIs and subsequently introduced a benchmark which uses these approaches. Our findings underscore the importance of using confidence intervals that account for data randomness in evaluations, a practice we advocate extending beyond classification and vision to encompass all domains employing task-based few-shot learning assessments.








style






§ MATHEMATICAL DERIVATION OF (A̅)


Suppose tasks are drawn IID without replacement, we write the variance of A̅ as:

    ( A̅) = 1/T_t( A_t ),


with A_t the accuracy for an arbitrary task t. By definition of the variance, 

    ( A_t ) =𝔼[(A_t)^2]- (𝔼[A_t])^2.


For a given support set, the expected accuracy for some class c is denoted μ_t,c. 

    μ_t,c≜𝔼_𝒬_t(1[f_𝒮_t(x) = c] |𝒮_t ).

Then the expectation of A_t becomes

    𝔼[A_t]=𝔼_𝒮_t[μ_t,c],

  


Along the same lines, we can derive 𝔼[(A_t)^2],

    𝔼[(A_t)^2]=𝔼_𝒮_t𝔼_𝒬_t[(A_t)^2|𝒮_t].


For a fixed S_t,  1[f_𝒮_t(x) = c] and 1[f_𝒮_t(x') = c'] are not independent and their distribution will depend on the classes c and c'. Using  Equation <ref>, we obtain .

    𝔼[(A_t)^2]= 1/(KQ)^2∑_c ∑_c'∑_x ∑_x'𝔼_𝒮_t𝔼_𝒬_t[ 1[f_𝒮_t(x) = c] 1[f_𝒮_t(x') = c']].


We now separate cases where x=x' from cases where c=c' and finally cases where both are different. 

    𝔼[(A_t)^2]=1/KQ𝔼_𝒮_t[μ_t,c]+
    Q-1/KQ𝔼_𝒮_t[(μ_t,c)^2]
    +1/K^2∑_c ∑_c'≠ c𝔼_𝒮_t[μ_t,cμ_t,c'].

 
Using Equation <ref>, we find:

    ( A_t )=1/KQ𝔼_𝒮_t[μ_t,c]+Q-1/KQ𝔼_𝒮_t[(μ_t,c)^2]
    +1/K^2∑_c ∑_c'≠ c𝔼_𝒮_t[μ_t,cμ_t,c']- (𝔼_𝒮_t[μ_t,c])^2.



Let us define some parameters,

    m_1≜𝔼[A_t]=1/K∑_c𝔼_𝒮_t[μ_t,c],



    m_2≜1/K∑_c𝔼_𝒮_t[(μ_t,c)^2],

and 

    m_3≜1/K^2∑_c ∑_c'≠ c𝔼_𝒮_t[μ_t,cμ_t,c'].

We get that 

    𝔼[(A_t)^2]=1/KQm_1+Q-1/KQ m_2
    +m_3.

This gives 

    ( A_t )=1/KQm_1+Q-1/KQ m_2
    +m_3- (m_1)^2.

Then, using Equation <ref> (removing the rounding) and <ref>, we approximate:


    ( A̅) = K/NC (α Q+ β/Q + γ),

with α=m_2/K+m_3-(m_1)^2, β=S/K(m_1-m_2) and γ=m_1/K-m_2/K+S/K m_2+ S(m_3-(m_1)^2).

First, let us notice that β>0 since for μ∈ [0,1], μ^2<μ .




§ ALGORITHMS

Algorithm <ref> samples tasks with replacement and computes CCIs with Equation <ref> while Algorithm <ref> samples tasks without replacement and uses the student's distribution to compute OCIs.  











0.9 







§ ADDITIONAL RESULTS ON THE BENCHMARK

These results show the performance differences when taking DINO and CLIP with finetune as baselines. Again finetuning mostly underperforms other adaptation methods. 





§ IS Q^* DEPENDENT ON THE MODEL USED?

We show in Figure <ref> that the same values of Q^* are found for DINO v2 instead of CLIP.







§ STATISTICS ON CONCLUSIVENESS











This histogram illustrates that paired test and the optimization of the size of tasks yields the maximum number of conclusive comparisons. Optimizing Q slightly improves the number of conclusive comparisons compared to simple paired tests.




§ ACKNOWLEDGMENT


This work was performed using HPC resources from GENCI–IDRIS (Grant 202123656B).

The PhD study of Luke Smith is supported by GSK-plc and the Australian Government Research Training Program (RTP) scholarship.

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy EXC 2044 –390685587, Mathematics Münster: Dynamics–Geometry–Structure.

F. Vermet conducted this work within the France 2030 framework programme, the Centre Henri Lebesgue  ANR-11-LABX-0020-01



    < g r a p h i c s >

Raphael Lafargue was supported by Brittany Region, France. 



