\begin{table}[t]
    \centering
    \caption{
        Recall scores on Flickr8k for ablation studies.
    }
    \label{tab:ablation}
    \small
    \vspace{-5pt}
    \begin{tabular}{@{~~}l@{~~}c@{~~}c@{~~}c@{~}c@{~}c@{~~}c@{~~}c@{~~}} 
        \toprule
        & \multicolumn{3}{c}{Speech $\rightarrow$ Image} & & \multicolumn{3}{c}{Image $\rightarrow$ Speech} \\ 
        \cmidrule{2-4}
        \cmidrule{6-8}
        Method & R@1 & R@5 & R@10 & & R@1 & R@5 & R@10 \\
        \midrule
        & \multicolumn{7}{c}{Batch Normalization} \\
        \cmidrule{2-8}
        Cascaded (w/ BN) & 8.2  & 25.7 & 37.2 & & 14.1 & 34.5 & 49.2 \\
        Cascaded (w/o BN) & 1.1 & 4.7 & 8.4 & & 1.4 & 5.7 & 9.4 \\
        \midrule
        & \multicolumn{7}{c}{Keyword Num} \\
        \cmidrule{2-8}
        Cascaded ($K$ $=$ 8) & 8.2  & 25.7 & 37.2 & & 14.1 & 34.5 & 49.2 \\
        Cascaded ($K$ $=$ 4) & 3.5 & 13.2  & 21.1 & & 5.2 & 17.5 & 27.4 \\
        Cascaded ($K$ $=$ 2) & 2.1 & 8.4   & 14.4 & & 2.7 & 10.6 & 17.6 \\
        
        
        
        
        \bottomrule
    \end{tabular}
    \vspace*{-10pt}
\end{table}
The VQ process is described as follows.
We first compute the cosine similarity between the $k^{\text{th}}$ normalized CLS embedding ($\boldsymbol{z}_k$) and the $v^{\text{th}}$ subword embedding ($\boldsymbol{e}_v$) as
\vspace{-3pt}

where each embedding $\boldsymbol{e}_v$ is a column vector and $\tau$ is a hyper-parameter ($\tau=0.1$).
Combining Eqs. \ref{eq:hardkw} and \ref{eq:softkw}, we apply straight-through gradient estimator~\cite{bengio2013estimating} to obtain quantized keywords
\vspace{-3pt}

where $\text{sg}(x) = x$ and $\frac{d}{dx}\text{sg}(x)=0$ is the stop gradient operator.
The $K$ keywords are then fed into the CLIP text encoder for computing the contrastive objective.

\noindent\textbf{Model.}
We implemented SpeechCLIP in two sizes: Base and Large, a detailed comparison is shown in Table \ref{tab:model_compare}.
Note that we omit the Base notation in the following sections.
The hidden dimension of the transformer encoder is the same as that of the audio encoder.
The feed-forward network in the cascaded model's transformer encoder is removed for better performance.
Parallel and cascaded models have respectively eight and one attention head.
We set $K$ to 8 in all experiments.
All models are trained with Adam optimizer with a weight decay of 10$^{-\text{6}}$, batch size of 256, and 50k steps in total.
The learning rate linearly increases to 10$^{-\text{4}}$ in the first 5k steps and linearly decreases to 10$^{-\text{8}}$ afterward.
All experiments are conducted on a 32GB V100 GPU except for pre-training on SpokenCOCO, which uses two.
The largest model's pre-training lasts approximately two days.

In this section, we evaluate SpeechCLIP on the image-speech retrieval task, showing how well models can align speech with CLIP image embeddings.
As shown in Table~\ref{tab:recall}, parallel SpeechCLIP models surpass almost all baseline methods, especially for the Large SpeechCLIP models.
The parallel Base model on Flickr8k also shows competitive performance with the FaST-VGS$_{\text{CO}}$ model, indicating that utilizing powerful pre-trained models and a small set of learnable parameters is sufficient.
Moreover, the cascaded models obtain the lowest recall scores because passing encoded speech through VQ and a CLIP text encoder loses information.
Overall, the results show the benefits of integrating CLIP in VGS models even with minimal fine-tuning.

In this section, we show which HuBERT hidden layers are crucial for SpeechCLIP to perform well in various tasks discussed earlier.
Hence, we visualize the learned weights in the weighted sum mechanism mentioned in Sec.~\ref{subsec:prelim} in Fig.~\ref{fig:speechCLIP_weight}.
Both parallel and cascaded SpeechCLIP utilize the roughly the 8$^{\text{th}}$ to the 10$^{\text{th}}$ layers in HuBERT, inferring that HuBERT's top layers capture rich content and semantic information.
This result is consistent with prior works investigating the importance of different hidden layers in speech SSL models~\cite{baevski2021wav2vec-u,pasad2021layer,chang2022distilhubert}, i.e., the top hidden layers contain word meaning and content information.
However, the cascaded model's weights distribute more evenly over the layers than parallel SpeechCLIP, showing that the model architecture design affects the utilization of HuBERT's layers.

Here, we demonstrate the importance of batch normalization in the cascaded SpeechCLIP.
We compare cascaded SpeechCLIP with its variant without using batch normalization, as shown in the first two rows of Table~\ref{tab:ablation}.
Removing batch normalization degrades retrieval performance significantly, showing the significance of mean and variance matching described in Sec.~\ref{subsec:cascaded}.

This section discusses the impact of the number of keywords in cascaded SpeechCLIP.
We report retrieval results on Flickr8k using different amounts of keywords in Table~\ref{tab:ablation}.
Results show that reducing keywords degrades retrieval performance, indicating that using fewer keywords is incapable of passing information from the speech encoder to the CLIP text encoder.
Furthermore, the number of subword tokens in a Flickr8k utterance is 11.3 $\pm$ 4.1, and some tokens carry less information like stopwords.
Therefore, we suggest 8 is a reasonable number for $K$ to obtain good performance with cascaded SpeechCLIP.
Although dynamically assigning $K$ for utterances of different lengths is more appropriate, we leave this approach for future investigation.
{"value": "8.2", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"w/ BN": "true"}} {"value": "25.7", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"w/ BN": "true"}} {"value": "37.2", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"w/ BN": "true"}} {"value": "14.1", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"w/ BN": "true"}} {"value": "34.5", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"w/ BN": "true"}} {"value": "49.2", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"w/ BN": "true"}} {"value": "1.1", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"w/o BN": "true"}} {"value": "4.7", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"w/o BN": "true"}} {"value": "8.4", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"w/o BN": "true"}} {"value": "1.4", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"w/o BN": "true"}} {"value": "5.7", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"w/o BN": "true"}} {"value": "9.4", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"w/o BN": "true"}} {"value": "8.2", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"K": "8"}} {"value": "25.7", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"K": "8"}} {"value": "37.2", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"K": "8"}} {"value": "14.1", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"K": "8"}} {"value": "34.5", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"K": "8"}} {"value": "49.2", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"K": "8"}} {"value": "3.5", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"K": "4"}} {"value": "13.2", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"K": "4"}} {"value": "21.1", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"K": "4"}} {"value": "5.2", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"K": "4"}} {"value": "17.5", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"K": "4"}} {"value": "27.4", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"K": "4"}} {"value": "2.1", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"K": "2"}} {"value": "8.4", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"K": "2"}} {"value": "14.4", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Speech $\rightarrow$ Image": "true"}, "model": "Cascaded", "model settings": {"K": "2"}} {"value": "2.7", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@1", "R@1"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"K": "2"}} {"value": "10.6", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@5", "R@5"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"K": "2"}} {"value": "17.6", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "speech-image retrieval", "metric": ["Recall@10", "R@10"], "experimental settings": {"Image $\rightarrow$ Speech": "true"}, "model": "Cascaded", "model settings": {"K": "2"}}