Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator

Large-scale <PRE_TAG>pre-trained language models (PLMs)</POST_TAG> are well-known for being capable of solving a task simply by conditioning a few input-label pairs dubbed <PRE_TAG>demonstrations</POST_TAG> on a prompt without being explicitly tuned for the desired downstream task. Such a process (i.e., in-context learning), however, naturally leads to high reliance on the <PRE_TAG>demonstrations</POST_TAG> which are usually selected from <PRE_TAG>external datasets</POST_TAG>. In this paper, we propose self-generated in-context learning (SG-ICL), which generates <PRE_TAG>demonstrations</POST_TAG> for in-context learning from PLM itself to minimize the reliance on the external demonstration. We conduct experiments on four different text classification tasks and show SG-ICL significantly outperforms zero-shot learning and is generally worth approximately 0.6 gold training samples. Moreover, our generated <PRE_TAG>demonstrations</POST_TAG> show more consistent performance with low variance compared to randomly selected <PRE_TAG>demonstrations</POST_TAG> from the training dataset.