\begin{table}[t]
    \centering
    \caption{
        Recall for speech-text retrieval on Flickr8k and SpokenCOCO.
        `Sup.'' indicates the supervised version of parallel SpeechCLIP by replacing the image encoder with CLIP text encoder in parallel SpeechCLIP.
    }
    \label{tab:zeroshot_speech_text}
    \small
    \begin{tabular}{@{~~}l@{~~}c@{~~}c@{~~}c@{~}c@{~}c@{~~}c@{~~}c@{~~}}
        \toprule
        & \multicolumn{3}{c}{Speech $\rightarrow$ Text}  & & \multicolumn{3}{c}{Text $\rightarrow$ Speech} \\ 
        \cmidrule{2-4}
        \cmidrule{6-8}
        Method & R@1 & R@5 & R@10 & & R@1 & R@5 & R@10 \\ 
        \midrule
        & \multicolumn{7}{c}{Flickr8k} \\
        \cmidrule{2-8}
        Random & 0.10 &	0.50 &	0.99 & & 0.10 &	0.50 &	0.99 \\
        Parallel Large & 19.56 & 44.06 & 58.46 & & 22.50 & 44.14 & 54.54 \\
        Parallel Large (Sup.) & 97.06 & 99.24 & 99.46 & & 97.88 & 99.76 & 99.90 \\
        \midrule
        & \multicolumn{7}{c}{SpokenCOCO} \\
        \cmidrule{2-8}
        Random & 0.02 &	0.10 &	0.20 & & 0.02 &	0.10 &	0.20 \\
        Parallel Large & 60.32 & 81.81 & 88.18 & & 65.45 &	85.82 &	91.27 \\
        Parallel Large (Sup.) & 
        95.02 &	99.46 &	99.78 & & 95.35 &	99.68 &	99.93 \\
        \bottomrule
    \end{tabular}
\end{table}
The VQ process is described as follows.
We first compute the cosine similarity between the $k^{\text{th}}$ normalized CLS embedding ($\boldsymbol{z}_k$) and the $v^{\text{th}}$ subword embedding ($\boldsymbol{e}_v$) as
\vspace{-3pt}

\begin{equation}
    \vspace{-3pt}
    s_{kv} = \text{cos}\left( \boldsymbol{z}_k , \boldsymbol{e}_v \right).
    \label{eq:cos}
\end{equation}

\begin{equation}
    \vspace{-3pt}
    \boldsymbol{e}_{v^{\star}}, \text{~where~} v^{\star} = \underset{1 \leq v \leq V}{\text{argmax}} ~ s_{kv}.
    \label{eq:hardkw}
\end{equation}

\begin{equation}
    \vspace{-3pt}
    \overline{\boldsymbol{h}}_k = \left[ \boldsymbol{e}_1 \dots \boldsymbol{e}_V \right] \text{softmax}\left( \left[ s_{k1} \dots s_{kV} \right]^\top / \tau \right),
    \label{eq:softkw}
\end{equation}

\begin{equation}
    \vspace{-3pt}
    \boldsymbol{h}_k = \boldsymbol{e}_{v^{\star}} + \overline{\boldsymbol{h}}_k - \text{sg}\left( \overline{\boldsymbol{h}}_k \right),
\end{equation}

\noindent\textbf{Model.}
We implemented SpeechCLIP in two sizes: Base and Large, a detailed comparison is shown in Table \ref{tab:model_compare}.
Note that we omit the Base notation in the following sections.
The hidden dimension of the transformer encoder is the same as that of the audio encoder.
The feed-forward network in the cascaded model's transformer encoder is removed for better performance.
Parallel and cascaded models have respectively eight and one attention head.
We set $K$ to 8 in all experiments.
All models are trained with Adam optimizer with a weight decay of 10$^{-\text{6}}$, batch size of 256, and 50k steps in total.
The learning rate linearly increases to 10$^{-\text{4}}$ in the first 5k steps and linearly decreases to 10$^{-\text{8}}$ afterward.
All experiments are conducted on a 32GB V100 GPU except for pre-training on SpokenCOCO, which uses two.
The largest model's pre-training lasts approximately two days.

In this section, we evaluate SpeechCLIP on the image-speech retrieval task, showing how well models can align speech with CLIP image embeddings.
As shown in Table~\ref{tab:recall}, parallel SpeechCLIP models surpass almost all baseline methods, especially for the Large SpeechCLIP models.
The parallel Base model on Flickr8k also shows competitive performance with the FaST-VGS$_{\text{CO}}$ model, indicating that utilizing powerful pre-trained models and a small set of learnable parameters is sufficient.
Moreover, the cascaded models obtain the lowest recall scores because passing encoded speech through VQ and a CLIP text encoder loses information.
Overall, the results show the benefits of integrating CLIP in VGS models even with minimal fine-tuning.

According to Table~\ref{tab:zeroshot_speech_text}, proposed SpeechCLIP models yield considerably better performance than random retrieval, showing that speech and text embedding spaces are well aligned.
Specifically, parallel SpeechCLIP performs better on this task when trained on a larger dataset like SpokenCOCO.
Although the performance gap between the proposed methods and the supervised toplines remains, we show that bridging speech and text with image is possible and promising.
{"value": "0.10", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "0.50", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "0.99", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "0.10", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "0.50", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "0.99", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "19.56", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "44.06", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "58.46", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "22.50", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "44.14", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "54.54", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "97.06", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "99.24", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "99.46", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "97.88", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "99.76", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "99.90", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "0.02", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "0.10", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "0.20", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "0.02", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "0.10", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "0.20", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Random", "model settings": {"xx": "yy"}} {"value": "60.32", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "81.81", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "88.18", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "65.45", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "85.82", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "91.27", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large", "model settings": {"xx": "yy"}} {"value": "95.02", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "99.46", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "99.78", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Text": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "95.35", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "99.68", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}} {"value": "99.93", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "speech-text retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Text $\rightarrow$ Speech": "true"}, "model": "Parallel Large (Sup.)", "model settings": {"xx": "yy"}}