weakness Originality
+ Interesting and novel insights into the task difficulty for meta-learners. Although the idea of controlling episode difficulty is not very novel [47, 28], the authors go about it in a novel way and propose to use importance sampling for a faster and more efficient way of sampling episodes that is model agnostic. Quality
+ Solid evaluation and analysis. Testing over multiple models, algorithms, and benchmarks makes for a convincing set of results.
+ Confirmed assumptions through experimentation. The paper uses Q-Q plots and Shapiro-Wilk to verify the assumption that episode difficulty follows a normal distribution.
- Advantage of UNIFORM over other procedures is not consistent. The tables show that UNIFORM does not always offer a clear advantage over the results, especially in the 1-shot setting. Do the authors have a theory for why the method is not as effective on the 1 shot setting? Clarity
+ Experiments are well designed, and the results are clear.
+ Paper is organized and well-written. Significance
+ Interesting solution to a relevant problem. Episode difficulty is understudied in few-shot learning, and uniform presents a model agnostic solution to the problem of how to sample tasks during meta-learning. Post-Rebuttal
I have read the rebuttal and other reviews, and I increase my original rating. This paper addresses an interesting and relevant issue of episode difficulty in meta-learning and will be a particularly valuable contribution at the conference. The paper presents a solid set of evaluations and analyses, verifying assumptions (e.g. via Q-Q plots and Shapiro-Wilk tests) and reporting results across multiple benchmarks. The paper presents a novel, simple but effective method that, on the most part, boosts algorithm performance by a couple of additional accuracy points on average. Therefore, I recommend this work for the conference with a score of 8.
- Advantage of UNIFORM over other procedures is not consistent. The tables show that UNIFORM does not always offer a clear advantage over the results, especially in the 1-shot setting. Do the authors have a theory for why the method is not as effective on the 1 shot setting? Clarity + Experiments are well designed, and the results are clear.