+ The value of episodic training is increasingly being questioned, and the submission approaches the topic from a new and interesting perspective.
+ The connection between nearest-centroid few-shot learning approaches and NCA has not been made in the literature to my knowledge and has potential applications beyond the scope of this paper.
+ The paper is well-written, easy to follow, and well-connected to the existing literature.
- The extent to which the observations presented generalize to few-shot learners beyond Prototypical Networks is not evaluated, which may limit the scope of the submission’s contributions in terms of understanding the properties of episodic training.
- The Matching Networks / NCA connection makes more sense in my opinion than the Prototypical Networks / NCA connection.
- A single set of hyperparameters was used across learners for a given benchmark, which can bias the conclusions drawn from the experiments. Recommendation
I’m leaning towards acceptance. I have some issues with the submission that are detailed below, but overall the paper presents an interesting take on a topic that’s currently very relevant to the few-shot learning community, and I feel that the value it brings to the conversation is sufficient to overcome the concerns I have.
Detailed justification
The biggest concern I have with the submission is methodological. One one hand, the authors went beyond the usual practice of reporting accuracies on a single run and instead trained each method with five different random initializations, and this is a practice that I’m happy to see in a few-shot classification paper. On the other hand, the choice to share a single set of hyperparameters across learners for a given benchmark leaves a blind spot in the evaluation. What if Prototypical Networks are more sensitive to the choice of optimizer, learning rate schedule, and weight decay coefficient than NCA? Is it possible that the set of hyperparameters chosen for the experiments happens to work poorly for Prototypical Networks? Would we observe the same trends if we tuned hyperparameters independently for each experimental setting? In its current form the submission shows that Prototypical Networks are sensitive to the hyperparameters used to sample episodes while keeping other hyperparameters fixed, but showing the same trend while doing a reasonable effort at tuning other hyperparameters would make for a more convincing argument. This is why I take the claim made in Section 4.2 that "NCA performs better than all PN configurations, no matter the batch size" with a grain of salt, for instance.
I also feel that the submission misses out on an opportunity to support a more general statement about episodic training via observations on approaches such as Matching Networks, MAML, etc. I really like the way Figure 1 explains visually how Prototypical Networks miss out on useful relationships between examples in a batch and is therefore data-inefficient. To me, this is one of the submission’s most important contributions: the suggestion that a leave-one-out strategy could allow episodic approaches to achieve the same kind of data efficiency as non-episodic approaches, alleviating the need for a supervised pre-training / episodic fine-tuning strategy. To be clear, I don’t think the missed opportunity would be a reason to reject the paper, but I think that showing empirically that the leave-one-out strategy applies beyond Prototypical Networks would make me lean more strongly towards acceptance.
The connection drawn between Prototypical Networks and NCA feels forced at times. In the introduction the paper claims to "show that, without episodic learning, Prototypical Networks correspond to the classic Neighbourhood Component Analysis", but Section 3.3 lists the creation of prototypes as a key difference between the two which is not resolved by training non-episodically. From my perspective, NCA would be more akin to the non-episodic counterpart to Matching Networks without Full Contextual Embeddings – albeit with a Euclidean metric rather than a cosine similarity metric – since both perform comparisons on example pairs.
This relationship with Matching Networks could be exploited to improve clarity. For instance, row 6 of Figure 4 can be interpreted as a Matching Networks implementation with a Euclidean distance metric. With this in mind, could the difference in performance between "1-NN with class centroids" and k-NN / Soft Assignment noted in Section 4.1 – as well as the drop in performance observed in Figure 4’s row 6 – be explained by the fact that a (soft) nearest-neighbour approach is more sensitive to outliers?
Finally, I have some issues with how results are reported in Tables 1 and 2. Firstly, we don’t know how competing approaches would perform if we applied the paper’s proposed multi-layer concatenation trick, and the idea itself feels more like a way to give NCA’s performance a small boost and bring it into SOTA-like territory. Comparing NCA without multi-layer against other approaches is therefore more interesting to me. Secondly, 95% confidence intervals are provided, but the absence of identification of the best-performing approach(es) in each setting makes it hard to draw high-level conclusions at a glance. I would suggest bolding the best accuracy in each column along with all other entries for which a 95% confidence interval test on the difference between the means is inconclusive in determining that the difference is significant. Questions
In Equation 2, why is the sum normalized by the total number of examples in the episode rather than the number of query examples?
Can the authors comment on the extent to which Figure 2 supports the hypothesis that NCA is better for training because it learns from a larger number of positives and negatives? Assuming this is true, we should see that Prototypical Networks configurations that increase the number of positives and negatives should perform better for a given batch size. Does Figure 2 support this assertion?
Can the authors elaborate on the "no S/Q" ablation (Figure 4, row 7)? What is the point of reference when computing distances for support and query examples? Is the loss computed in the same way for support and query examples? The text in Section 4.3 makes it appear like the loss for query examples is the NCA loss, but the loss for support examples is the prototypical loss. Wouldn’t it be conceptually cleaner to compute leave-one-out prototypes, i.e. leave each example out of the computation of its own class’ prototype (resulting in slightly different prototypes for examples of the same class)? In my mind, this would be the best way to remove the support/query partition while maintaining prototype computation, thereby showing that the partition is detrimental to Prototypical Networks training.
Additional feedback
This is somewhat inconsequential, but across all implementations of episodic training that I have examined I haven’t encountered an implementation that uses a flag to differentiate between support and query examples. Instead, the implementations I have examined explicitly represent support and query examples as separate tensors. I was therefore surprised to read that "in most implementations [...] each image is characterised by a flag indicating whether it corresponds to the support or the query set [...]"; can the authors point to the implementations they have in mind when making that assertion?
I would be careful with the assertion that "during evaluation the triplet {w, n, m} [...] must stay unchanged across methods". While this is true for the benchmarks considered in this submission, benchmarks like Meta-Dataset evaluate on variable-ways and variable-shots episodes.
I’m not too concerned with the computational efficiency of NCA. The pairwise Euclidean distances can be computed efficiently using the inner- and outer-product of the batch of embeddings with itself.
- The extent to which the observations presented generalize to few-shot learners beyond Prototypical Networks is not evaluated, which may limit the scope of the submission’s contributions in terms of understanding the properties of episodic training.