\begin{table}[htbp]
\centering
\begin{tabular}{c|c|c}
\hline
$\lambda$   & BLEU & NIST \\ \hline
1.0 & 7.71 & 25.67 \\ \hline
0.8 &  27.11    &   79.42   \\ \hline
0.6 &   27.23   &    79.86  \\ \hline
0.4 &   26.66   &    77.68  \\ \hline
0.2 &    26.54  &    78.60  \\ \hline
0.0 & 25.44 & 66.92 \\ \hline
\end{tabular}
\caption{Different $\lambda$ settings on Oxford test set.}\label{hyper-param}
\end{table}
\subsection{Task Formulation}
\label{Task Formulation}
Given a word or phrase $W = \{w_i,...,w_j\}$ and its surrounding context $C = \{w_0,...,W_k\}(0<i<j<k)$, the task of definition generation is to generate the definition $D = \{d_0,...d_T\}$ to explain the meaning of $W$ under $C$. This process can be formulated as:

Finally, a softmax function is added upon a linear head to transform $\mathbf{G}^{L_D}$ into a prediction distribution matrix $\mathbf{V}\in \mathbb{R}^{|V|\times |D|}$. Here $|V|$ and $|D|$ stand for the vocabulary size and the length of the ground-truth definition, respectively. To optimize, a cross-entropy loss is applied to measure the discrepancy between the generated distribution and the ground-truth distribution. 

Formally, we denote the {target word encoding} generated by the encoder in T5 as $\mathbf{H}_{target}$, and the {definition encoding} generated by the decoder as $\mathbf{G}^{L_D}$. In general, target word encoding $\mathbf{H}_{target}$ is obtained by extracting the encoding of the target word's position in $\mathbf{H}^{L_E}$, and
definition encoding $\mathbf{G}^{L_D}$ is generated by the decoder to decode and get the definition sequence later. 

where $N$ denotes a mini-batch of training samples. The $\tau$ is a temperature hyper-parameter and $\rm sim(,)$ stands for the cosine similarity function. During learning, the contrastive loss in Eq.~\ref{eq:contrastive} enforces the model to concentrate on the discrepancy between the two views of the same semantic unit, i.e., the target word. 

As depicted in Figure~\ref{overview}, our full training strategy follows a two-stage paradigm. At the first stage, we finetune our model only with the generation loss. In the second stage, we combine the contrastive loss in the training and optimize the model with mixed loss $L_{Final}$:

For evaluation, we follow previous works and acquire three popular datasets, which are ensembled by \citet{ishiwatari2019learning}\footnote{\url{http://www.tkl.iis.u-tokyo.ac.jp/~ishiwatari/naacl_data.zip}}.
Each entry in a dataset consists of three elements: (1) a target word or phrase, (2) the corresponding definition, and (3) one usage example of the target as a local context. If a target has multiple definitions and examples, we treat them as different entries. For fair comparison, each dataset is split into $\textit{train}$, $\textit{dev}$ and $\textit{test}$ sets according to \citet{ishiwatari2019learning}. The statistics of these datasets are shown in Table~\ref{Statistics of datasets}.

\subsection{Experimental Setups}
\label{Settings}
We train all models in PyTorch\footnote{\url{https://github.com/pytorch/pytorch}} \cite{paszke2019pytorch}, and use the HuggingFace\footnote{\url{https://github.com/huggingface/}} \cite{wolf2019huggingface} implementation of T5. We train each model on a V100 GPU. For compared models, we replicate experiments following the implementations details released by \citet{huang2021definition}. For training our model, we use the base version of T5 with the same size of \citet{huang2021definition}. 
For each dataset, we finetune it using Adam ~\cite{kingma2014adam} optimizer with an initial learning rate of 3e-4 and the batch size of 16. In all the experiments, we train our model with a two-stage strategy as described in the previous section. Please refer to Appendix~\ref{Detailed Training Setting} for the detailed training settings in each stage, like max-epoch and early-stop threshold.

\subsection{Analysis on Hyper-Parameter}
To explore how our method would be affected by the choice of the hyper-parameter $\lambda$ in Eq.~\ref{eq:Final_Loss}, we remain other settings the same as we mentioned in Section~\ref{Settings} and set different $\lambda$ for each model to observe the performance change. The results on the Oxford dataset are reported in Table~\ref{hyper-param}. As shown, when $\lambda$ is set to 0.0, the model is ``degraded'' to the compared T5-Base model. Considering T5-Base model is fine-tuned only using the generation loss in our setting, it is identical to a variant without contrastive loss in the second training stage. To this end, their performances are the same. Also, the performance of the model when $\lambda$ is set to 1.0 (without generation loss in the second training stage) is pretty bad. We attribute it to the fact that our task requires the ability of language generation and thus still need generation loss to guide contrastive learning in the right way. Besides the above extreme values of $\lambda$, we find the model achieves better performance when $\lambda$ is higher ($\lambda$=0.8 and $\lambda$=0.6). It further illustrates that after the first stage of generation-only training, the model will benefit more from our fine-grained contrastive learning.
{"value": "1.0", "type": "Hyper-parameter/Architecture", "model": ["T5-Contrast", "Ours"], "parameter/architecture name": "\\lambda", "dataset": "Oxford"} {"value": "7.71", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "BLEU", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "1.0"}} {"value": "25.67", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "NIST", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "1.0"}} {"value": "0.8", "type": "Hyper-parameter/Architecture", "model": ["T5-Contrast", "Ours"], "parameter/architecture name": "\\lambda", "dataset": "Oxford"} {"value": "27.11", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "BLEU", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "0.8"}} {"value": "79.42", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "NIST", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "0.8"}} {"value": "0.6", "type": "Hyper-parameter/Architecture", "model": ["T5-Contrast", "Ours"], "parameter/architecture name": "\\lambda", "dataset": "Oxford"} {"value": "27.23", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "BLEU", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "0.6"}} {"value": "79.86", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "NIST", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "0.6"}} {"value": "0.4", "type": "Hyper-parameter/Architecture", "model": ["T5-Contrast", "Ours"], "parameter/architecture name": "\\lambda", "dataset": "Oxford"} {"value": "26.66", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "BLEU", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "0.4"}} {"value": "77.68", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "NIST", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "0.4"}} {"value": "0.2", "type": "Hyper-parameter/Architecture", "model": ["T5-Contrast", "Ours"], "parameter/architecture name": "\\lambda", "dataset": "Oxford"} {"value": "26.54", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "BLEU", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "0.2"}} {"value": "78.60", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "NIST", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "0.2"}} {"value": "0.0", "type": "Hyper-parameter/Architecture", "model": ["T5-Contrast", "Ours"], "parameter/architecture name": "\\lambda", "dataset": "Oxford"} {"value": "25.44", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "BLEU", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "0.0"}} {"value": "66.92", "type": "Result", "training data/set": "Oxford", "test data/set": "Oxford", "task": "Definition Generation", "metric": "NIST", "experimental settings": {"xx": "yy"}, "model": ["T5-Contrast", "Ours"], "model settings": {"\\lambda": "0.0"}}