\begin{table}[t]
    \caption{
        Keyword hit rates for cascaded SpeechCLIP.
        Avg denotes averaged hit rate.
        $^\dagger$ and $^\ddagger$ respectively denote models trained on Flickr8k and SpokenCOCO.
    }
    \label{tab:kw_hit}
    \centering
    \small
    \begin{tabular}{@{~~}l@{~~}c@{~~}c@{~~}c@{~~}c@{~~}c@{~~}c@{~~}c@{~~}c@{~~}c@{~~}}
        \toprule
        Model & kw1 & kw2 & kw3 & kw4 & kw5 & kw6 & kw7 & kw8 & Avg \\
        \midrule
        Base$^{\dagger}$ & 57.0 & 25.6 & 20.2 & 5.0 & 20.0 & 26.5 & 10.5 & 16.6 & 22.7 \\
        Large$^{\dagger}$ & 56.5 & 19.6 & 20.5 & 37.5 & 21.7 & 34.6 & 26.4 & 44.7 & 32.7\\
        Large$^{\ddagger}$ & 27.5 & 22.4 & 35.8 & 61.0 & 21.6 & 54.2 & 60.1 & 22.9 & 38.2 \\
        

        \bottomrule
    \end{tabular}
    \vspace*{-12pt}
\end{table}
The VQ process is described as follows.
We first compute the cosine similarity between the $k^{\text{th}}$ normalized CLS embedding ($\boldsymbol{z}_k$) and the $v^{\text{th}}$ subword embedding ($\boldsymbol{e}_v$) as
\vspace{-3pt}

Since $\boldsymbol{e}_{v^{\star}}$ is not differentiable, we compute another embedding by weighted summing all $V$ subword embeddings as
\vspace{-3pt}

where each embedding $\boldsymbol{e}_v$ is a column vector and $\tau$ is a hyper-parameter ($\tau=0.1$).
Combining Eqs. \ref{eq:hardkw} and \ref{eq:softkw}, we apply straight-through gradient estimator~\cite{bengio2013estimating} to obtain quantized keywords
\vspace{-3pt}

where $\text{sg}(x) = x$ and $\frac{d}{dx}\text{sg}(x)=0$ is the stop gradient operator.
The $K$ keywords are then fed into the CLIP text encoder for computing the contrastive objective.

\noindent\textbf{Model.}
We implemented SpeechCLIP in two sizes: Base and Large, a detailed comparison is shown in Table \ref{tab:model_compare}.
Note that we omit the Base notation in the following sections.
The hidden dimension of the transformer encoder is the same as that of the audio encoder.
The feed-forward network in the cascaded model's transformer encoder is removed for better performance.
Parallel and cascaded models have respectively eight and one attention head.
We set $K$ to 8 in all experiments.
All models are trained with Adam optimizer with a weight decay of 10$^{-\text{6}}$, batch size of 256, and 50k steps in total.
The learning rate linearly increases to 10$^{-\text{4}}$ in the first 5k steps and linearly decreases to 10$^{-\text{8}}$ afterward.
All experiments are conducted on a 32GB V100 GPU except for pre-training on SpokenCOCO, which uses two.
The largest model's pre-training lasts approximately two days.

In this section, we evaluate SpeechCLIP on the image-speech retrieval task, showing how well models can align speech with CLIP image embeddings.
As shown in Table~\ref{tab:recall}, parallel SpeechCLIP models surpass almost all baseline methods, especially for the Large SpeechCLIP models.
The parallel Base model on Flickr8k also shows competitive performance with the FaST-VGS$_{\text{CO}}$ model, indicating that utilizing powerful pre-trained models and a small set of learnable parameters is sufficient.
Moreover, the cascaded models obtain the lowest recall scores because passing encoded speech through VQ and a CLIP text encoder loses information.
Overall, the results show the benefits of integrating CLIP in VGS models even with minimal fine-tuning.

Due to the unique design of cascaded SpeechCLIP, we investigate what and how well the speech encoder extracts keywords.
For each encoded and normalized CLS token $\boldsymbol{z}_k$, keywords are retrieved by finding subwords with the highest cosine similarities between $\boldsymbol{z}_k$ and the corresponding subword embeddings.
Notice that previous works~\cite{kamper2018semantic,pasad2019contributions} are also capable of retrieving semantically related keywords from speech. Nonetheless, they required pretrained image tagger and the size of keywords set is very limited. For SpeechCLIP, we can apply the same method to other pretrained langange models' vocabulary, technically.
Also, our setting is quite different from ~\cite{olaleye2022keyword}, where the 8 keywords are discovered from speech utterance without any text query in our work. Namely, SpeechCLIP can automatically summarize the speech by selecting 8 keywords.
We offer quantitative and qualitative analyses in the following paragraphs.

We inspect how well keywords are retrieved from speech signals for the quantitative analysis.
The evaluation metric is hit rate, which is the percentage of successful top-1 keyword retrieval of any word in the caption averaged over all testing samples.
In Table~\ref{tab:kw_hit}, some CLS tokens frequently retrieve words in the ground truth captions, showing that the cascaded architecture can directly capture words from speech.
Moreover, the first keyword's hit rate for models trained on Flickr8k is relatively high compared to other keywords.
Probably because the first word in a sentence has a higher chance to be ``a'', which is also the top-1 commonly retrieved subword from the first keyword in Flickr8k.
Another finding is that the Large model obtains a higher averaged keyword hit rate than the Base model on Flickr8k, which is consistent with the trend in Table~\ref{tab:recall}.
Hence, retrieving correct keywords is related to retrieving between speech and image samples.
Although some CLS tokens obtain reasonable hit rates, one might question whether the retrieved words are meaningful instead of stopwords.
Hence, we next analyze the results qualitatively to address this concern.
{"value": "57.0", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 1", "kw1"], "experimental settings": {"xx": "yy"}, "model": "Base", "model settings": {"xx": "yy"}} {"value": "25.6", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 2", "kw2"], "experimental settings": {"xx": "yy"}, "model": "Base", "model settings": {"xx": "yy"}} {"value": "20.2", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 3", "kw3"], "experimental settings": {"xx": "yy"}, "model": "Base", "model settings": {"xx": "yy"}} {"value": "5.0", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 4", "kw4"], "experimental settings": {"xx": "yy"}, "model": "Base", "model settings": {"xx": "yy"}} {"value": "20.0", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 5", "kw5"], "experimental settings": {"xx": "yy"}, "model": "Base", "model settings": {"xx": "yy"}} {"value": "26.5", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 6", "kw6"], "experimental settings": {"xx": "yy"}, "model": "Base", "model settings": {"xx": "yy"}} {"value": "10.5", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 7", "kw7"], "experimental settings": {"xx": "yy"}, "model": "Base", "model settings": {"xx": "yy"}} {"value": "16.6", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 8", "kw8"], "experimental settings": {"xx": "yy"}, "model": "Base", "model settings": {"xx": "yy"}} {"value": "22.7", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["average keyword hit rates", "avg"], "experimental settings": {"xx": "yy"}, "model": "Base", "model settings": {"xx": "yy"}} {"value": "56.5", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 1", "kw1"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "19.6", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 2", "kw2"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "20.5", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 3", "kw3"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "37.5", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 4", "kw4"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "21.7", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 5", "kw5"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "34.6", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 6", "kw6"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "26.4", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 7", "kw7"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "44.7", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["keyword hit rates 8", "kw8"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "32.7", "type": "Result", "training data/set": "Flickr8k", "test data/set": "Flickr8k", "task": "Keyword Retrieval", "metric": ["average keyword hit rates", "avg"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "27.5", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "Keyword Retrieval", "metric": ["keyword hit rates 1", "kw1"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "22.4", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "Keyword Retrieval", "metric": ["keyword hit rates 2", "kw2"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "35.8", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "Keyword Retrieval", "metric": ["keyword hit rates 3", "kw3"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "61.0", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "Keyword Retrieval", "metric": ["keyword hit rates 4", "kw4"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "21.6", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "Keyword Retrieval", "metric": ["keyword hit rates 5", "kw5"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "54.2", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "Keyword Retrieval", "metric": ["keyword hit rates 6", "kw6"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "60.1", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "Keyword Retrieval", "metric": ["keyword hit rates 7", "kw7"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "22.9", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "Keyword Retrieval", "metric": ["keyword hit rates 8", "kw8"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}} {"value": "38.2", "type": "Result", "training data/set": "SpokenCOCO", "test data/set": "SpokenCOCO", "task": "Keyword Retrieval", "metric": ["average keyword hit rates", "avg"], "experimental settings": {"xx": "yy"}, "model": "Large", "model settings": {"xx": "yy"}}