<center class="half">
    <img src="assets/115_introducing_contrastive_search/formulation.png" width="750"/>
</center>

where \\(V^{(k)}\\) is the set of top-k predictions from the language model's probability distribution \\(p_{\theta}(v|x_{< t})\\). The first term, i.e. _model confidence_, is the probability of the candidate \\(v\\) predicted by the language model. The second term, _degeneration penalty_, measures how discriminative of \\(v\\) with respect to the previous context \\( x_{< t}\\) and the function \\(s(\cdot, \cdot)\\) computes the cosine similarity between the token representations. More specifically, the degeneration penalty is defined as the maximum cosine similarity between the token representation of \\(v\\), i.e. \\(h_{v}\\), and that of all tokens in the context \\(x_{< t}\\). Here, the candidate representation \\(h_{v}\\) is computed by the language model given the concatenation of \\(x_{< t}\\) and \\(v\\). Intuitively, a larger degeneration penalty of \\(v\\) means it is more similar (in the representation space) to the context, therefore more likely leading to the problem of model degeneration. The hyperparameter \\(\alpha\\) regulates the importance of these two components. When \\(\alpha=0\\), contrastive search degenerates to the vanilla greedy search.

**[Remark]** When generating output, contrastive search jointly considers (i) the probability predicted by the language model to maintain the semantic coherence between the generated text and the prefix text; and (ii) the similarity with respect to the previous context to avoid model degeneration.


<span id='contrastive_generation'/>

#### 5.2. Generating Text with Contrastive Search:
The context provides a detailed explanation of the role of the hyperparameter \(\alpha\) in contrastive search. It clearly states that \(\alpha\) regulates the importance of the two components: model confidence and degeneration penalty. The context also explains what happens when \(\alpha=0\), indicating that the search becomes a vanilla greedy search. This information directly answers the question about what \(\alpha\) regulates, making the answer unambiguous and clear.


The question about the hyperparameter \(\alpha\) in contrastive search is quite useful for machine learning developers working with NLP applications in the Hugging Face ecosystem. Contrastive search is a relatively advanced topic in NLP, and understanding the role of hyperparameters like \(\alpha\) is crucial for fine-tuning models to achieve better performance. This question can help developers optimize their models and improve the quality of text generation tasks, making it a valuable inquiry for those looking to deepen their understanding of model behavior and hyperparameter tuning.


The question "What does the hyperparameter \(\alpha\) regulate in contrastive search?" is context-independent to a large extent. It specifies a particular hyperparameter (\(\alpha\)) and a specific method (contrastive search), which are likely to be well-defined in relevant technical documentation or literature. An operator with access to such resources should be able to understand and answer the question without needing additional context.

