AIC-AB NET: A Neural Network for Image Captioning with Spatial Attention
  and Text Attributes
Image captioning is a significant field across computer vision and natural
language processing. We propose and present AIC-AB NET, a novel
Attribute-Information-Combined Attention-Based Network that combines spatial
attention architecture and text attributes in an encoder-decoder. For caption
generation, adaptive spatial attention determines which image region best
represents the image and whether to attend to the visual features or the visual
sentinel. Text attribute information is synchronously fed into the decoder to
help image recognition and reduce uncertainty. We have tested and evaluated our
AICAB NET on the MS COCO dataset and a new proposed Fashion dataset. The
Fashion dataset is employed as a benchmark of single-object images. The results
show the superior performance of the proposed model compared to the
state-of-the-art baseline and ablated models on both the images from MSCOCO and
our single-object images. Our AIC-AB NET outperforms the baseline adaptive
attention network by 0.017 (CIDEr score) on the MS COCO dataset and 0.095
(CIDEr score) on the Fashion dataset.
Guoyun Tu, Ying Liu, Vladimir Vlassov
# AIC-AB Net: A Neural Network for Image Captioning with Spatial Attention and Text Attributes

###### Abstract

Image captioning is a significant field across computer vision and natural language processing. We propose and present AIC-AB Net, a novel Attribute-Information-Combined Attention-Based Network that combines spatial attention architecture and text attributes in an encoder-decoder. For caption generation, adaptive spatial attention determines which image region best represents the image and whether to attend to the visual features or the visual sentinel. Text attribute information is synchronously fed into the decoder to help image recognition and reduce uncertainty. We have tested and evaluated our AIC-AB Net on the MS COCO dataset and a new proposed Fashion dataset. The Fashion dataset is employed as a benchmark of single-object images. The results show the superior performance of the proposed model compared to the state-of-the-art baseline and ablated models on both the images from MSCOCO and our single-object images. Our AIC-AB Net outperforms the baseline adaptive attention network by **0.017** (CIDEr score) on the MS COCO dataset and **0.095** (CIDEr score) on the Fashion dataset.

 image captioning, neural networks, spatial attention, text attributes

Regular Research Paper

## I Introduction

The significant growth in web images has brought plenty of opportunities for computational understanding of images. Automatic image captioning is crucial for many applications, including image searching, categorizing, and indexing, and it has attracted attention from academia and industry. One can split the image captioning task into two parts (1) image recognition to detect and recognize objects in an image and (2) caption generation to summarize the extracted information and put it into text that humans understand.

Significant successes have been achieved in the problem of image captioning using Deep Learning (DL). Many works on image captioning have applied DL methods to images containing multiple objects and rich contextual information. As a result, various image captioning methods have been proposed, such as Visual space-based model [1], multimodal space-based model [2], dense captioning [3], whole scene-based model, encoder-decoder architecture-based model [4], compositional architecture-based model [5], attention-based model [6], semantic concept-based model [7], stylized captions [8]. We address the following two problems (1) generating captioning on single-object images; (2) combining semantic text attributes and adaptive attention.

The first blind spot of the previous studies is that they focused on general multi-object images, whereas generating captioning on single-object images is barely studied. Two features distinguish single-object images from general images. First, single-object images contain more small details, thus requiring a higher recognition resolution. Second, the generated descriptions include more adjectives and nouns. In this work, we apply DL models on a fashion dataset of 144,422 images from 24,649 products. This dataset is used as a benchmark of single-object images. Each image has only one fashion item, and its caption describes that item, including its category, color, texture, and other details.

Secondly, previous DL approaches either boost image captioning with semantic concept [5, 9] or make use of attention encoder-decoder framework [6, 10], i.e., two frameworks are used separately and cannot inform each other. We propose a novel attribute-image-combined attention-based neural network architecture (AIC-AB Net1) based on the adaptive attention network [10]. It combines the semantic concept-based architecture with the spatial attention-based architecture. In AIC-AB Net, the text attributes are fed into each step of the LSTM decoder as an additional input when generating the captions. The attributes are obtained by an auxiliary CNN classifier.

Footnote 1: [https://github.com/guoyuntu/Image-Captioning-On-General-Data-And-Fashion-Data](https://github.com/guoyuntu/Image-Captioning-On-General-Data-And-Fashion-Data)

The major contributions of our work are as follows.

1. We propose an Attribute-Image-Combined Attention-Based Network (AIC-AB Net) that combines the adaptive attention architecture and text attributes in an encoder-decoder framework and, as a consequence, improves the accuracy of image captioning compared to state-of-the-art alternatives.
2. We evaluate AIC-AB Net and several other DL models on a single-object dataset, the Fashion dataset, containing 144,422 images from 24,649 products.

## II Related work

Prior works, e.g., [11, 12, 4, 13], use DL encoder-decoder methods, attention-based and semantic concept-based DL models for automated image captioning.

**Encoder-Decoder for Image Captioning**. The existing caption generation methods include template-based image captioning [14], retrieval-based image captioning [15], and language model caption generation [16, 6, 17]. Most methods [11, 12, 18, 19] use Deep Learning. An encoder-decoder, a popular approach to tackling language tasks, such as machine translation, can be used for image captioning to encode visual information and decode it in a natural language. A network of this category extracts global image features from the hidden activations of a CNN and feeds them into an LSTM to generate a caption as a sequence of words; one word at each step depends on a context vector, the previous hidden state, and the previously generated words [16].

**Attention-based Networks**. Following the trends to use the encoder-decoder architecture on image captioning, methods based on attention mechanisms [10, 6, 20] have been increasingly popular as they provide computer vision algorithms with the ability to know where to look. Instead of considering the image as a whole scene, an attention-based network dynamically focuses on various parts of the input image while generating captions.

An adaptive attention network is an encoder-decoder-based approach for image captioning. The decoding stage is split into two parts. First, the Spatial Attention Network outputs a context vector \(c_{t}\) that depends on the feature map \(V\) extracted from the encoder and the hidden state \(h_{t}\) of the LSTM decoder. It could be considered as the attention map. The second part is Visual Sentinel, which can fall back on when it chooses not to attend to the image. This visual sentinel \(s_{t}\) is dependent on the input \(x_{t}\), the hidden state \(h_{t-1}\), and the memory cell \(m_{t}\). Then, the new adaptive context vector is modeled as \(\mathbf{c}_{t}=\sum_{i=1}^{k}\alpha_{t}v_{ti}\). \(\mathbf{c}_{t}\) and \(h_{t}\) determine the conditional probability for each time step of LSTM.

**Semantic concept-based Models**. The idea of semantic concept-based models is to extract a set of semantic concept proposals. These concepts, combined with visual features and hidden states, are used to generate the captions in the decoding stage. Karpathy et al. [7] proposed a model, in which dependency tree relations are applied in training to map the sentence segments with the image regions with a fixed window context. Wu et al. [13] proposed a network, including high-level semantic concepts explicitly. It adds an intermediate attribute prediction layer in an encoder-decoder framework to extract from images attributes used to generate semantically rich image captions. The proposed technique in this paper is partially inspired by [9], where Ting et al. suggested that the high-level attributes are more semantically rich and easily translated into understandable human sentences. Moreover, the best method is to feed attribute representations and visual features as a joint input to LSTM at each time step. The prior works are limited to using only one architecture, either semantic concept-based or spatial attention-based. We propose AIC-AB Net that combines both structures so that they can communicate with each other and, as a consequence, improve the accuracy of image captioning.

## III AIC-AB Net: Attribute-Image-Combined Attention-Based Network

We present our Attribute-image-combined Attention-based Network (AIC-AB Net), a novel encoder-decoder neural framework for image captioning. AIC-AB Net is an end-to-end network that tackles image captioning in the meantime, generates the attention map of the image. Fig. 1 shows the network architecture. We extract image features using pre-trained ResNet-152 [21], which implements residual learning units to alleviate the degradation of deep neural networks. We freeze the first six layers and take the last convolutional layer as visual features. We believe the features extracted retain both object and interaction information from the images.

Formally, let us denote the whole dataset as \(\mathfrak{D}=\{(\mathbf{X}_{i},\mathbf{y}_{i})(i=1,2,...,N)\) where \(\mathbf{X}_{i}\) denotes the \(i\)-th image and \(\mathbf{y}_{i}=(y_{1},y_{2},...,y_{t})\) denotes its caption label as a sequence of words. In an encoder-decoder framework, LSTM network plays the role of decoder and each conditional probability is modeled as:

\[\sum_{t=1}^{L}\log p(y_{t}|y_{1},y_{2},...y_{t-1},\mathbf{X})=f(\mathbf{h}_{t},\mathbf{c}_{t}) \tag{1}\]

where \(f\) is a nonlinear function that outputs the probability of \(y_{t}\). \(\mathbf{c}_{t}\) is the visual context vector at time step \(t\) extracted from image \(\mathbf{X}\). \(\mathbf{h}_{t}\) denotes the hidden state at \(t\). For LSTM, \(\mathbf{h}_{t}\) could be modeled as:

\[\mathbf{h}_{t}=LSTM(\mathbf{x}_{t},\mathbf{h}_{t-1},\mathbf{m}_{t-1}) \tag{2}\]

where \(\mathbf{x}_{t}\) is the input feature map, \(\mathbf{m}_{t-1}\) is the memory cell vector at \(t-1\).

### _Text Attribute Extractor_

The first step in adding the text attributes into the LSTM decoder is to extract a set of words that are possible to attend in the image's description. These words may belong to the following parts of speech, including nouns and adjectives. As suggested by [5], we build the vocabulary \(V\) using the 1000 most common words of the training captions. Given a vocabulary of attributes, the next step is to detect these words from images. We train the text attribute extractors using a CNN-based model. An image passes the pre-trained VGG-16 model, and we express the Cov5 layer as the input feature map, which is fed into a 2-layer CNN following with one fully connected layer. The possibility \(p_{i}^{w}\) of image \(x_{i}\) containing word \(w\) is computed by a sigmoid layer:

\[p_{t}^{w}=\frac{1}{1+\exp(-(v_{t}^{w}\phi(b_{i})+u_{w}))} \tag{3}\]

here \(\phi(b_{i})\) is the fully connected representation for image \(b_{i}\), \(v_{t}^{w}\) and \(u_{w}\) are the associated weights and bias with word \(w\)
Due to the highly imbalanced ratio of the positive labels (5 words per image) vs. negative, The loss function used for training the detector is

\[\mathbf{\hat{c}}_{i}^{C}=-\beta_{p}(p(x_{i})\log(q_{i}))+\beta_{n}(1-p(x_{i}))( \log(1-q(x_{i}))) \tag{4}\]

where \(\beta_{p}\) and \(\beta_{n}\) are class weights assigned for giving higher penalty over false positive predictions. Due to the very unbalanced labeling strategy, \(\beta_{p}=100\hat{\beta}_{n}\).

### _Attribute-combined Model_

With injecting the high-level attributes into the adaptive attention framework, we propose our AIC-AB Net (Fig. 1). In our model, the decoder is modified by additionally integrating visual information and high-level attributes. As Fig. 2 shows, the encoded image features are fed from the start of the LSTM and text attributes are fed into each time step. Accordingly, given the attribute representation \(\mathbf{A}\) the calculation of the hidden state in each time step is converted from Eq. (2) to:

\[\mathbf{h}_{t}=LSTM(\mathbf{x}_{t},\mathbf{A},\mathbf{h}_{t-1},\mathbf{m}_{ t-1}) \tag{5}\]

Adding text attributes to the original adaptive attention framework. The architecture of AIC-AB Net at one time step is demonstrated in Fig. 3. The probability over the vocabulary at time step \(t\) can be computed as:

\[\mathbf{p}_{t}=\mathrm{softmax}(\mathbf{W}_{p}(\hat{\mathbf{c}}_{t}+\mathbf{ h}_{t})) \tag{6}\]

where \(\mathbf{W}_{p}\) is weight parameter to be learnt.

## IV Evaluation Setup

To evaluate the proposed AIC-AB Net, we have conducted evaluation experiments using multi-object and single-object image datasets and have compared the performance of our AIC-AB Net with competing baselines.

### _Datasets and Preprocessing_

In our evaluation, we have used two image datasets, the MS COCO dataset [22] and a fashion image dataset.

**MS COCO Dataset**[22] contains 328K images with a total of 2.5M object instances. For the image captioning task, five caption descriptions labeled for each image are used as ground truth. We use the MS COCO dataset to compare the performance of AIC-AB Net with a state-of-the-art work [10].

**Fashion Dataset** is our single-object fashion image dataset scraped in open websites from different fashion vendors, including Uniqlo, Toteme-Studio, Bestseller, Drykorn, Jindenberg, Joseph-fashion, Marc-o-polo, Rodebjer, Tigerofsweden, and Vince. The raw data contains 1,511,916 images from 194,453 fashion products. Each data includes a text label consisting of various amounts of sentences. Before the image captioning task, we have conducted a data cleaning task to remove images with invalid or unrelated captions. The cleaned dataset includes 144,422 images from 24,649 products. Each data is labeled with only one caption description sentence. Products are allowed to map to the same caption, and there are 10,091 unique captions in this dataset.

**Preprocessing**. We apply the same split for COCO and Fashion datasets: 70% of the data for training, 15% for validation, and 15% for testing. We resize all the images used in experiments to \(224\times 224\) with bilinear interpolation. We also create two variations for the Fashion dataset. The one-vendor condition focuses on the largest product vendor, Bestseller. Further called the **Fashion Bestseller dataset**, this subset contains 89,756 images from 19,385 products. The amount of unique captions is 8,448. The second condition employs

Fig. 1: Overview of AIC-AB Net. It extracts from the images the visual features and the attribute information. The former pass to the adaptive attention architecture [10], the latter are fed into the LSTM decoder at every time step.
all images in the dataset, which we further call the **Fashion 9 vendors dataset**. The reason behind it is that we found the same vendor usually describes their product in similar text form and style. For all three datasets, COCO, Fashion Bestseller, and Fashion 9 vendors, we automatically generate five attributes for each image with the following method to train the attribute extractor. First, we build an attribute vocabulary comprising 1000 most common words (nouns and adjectives) from the caption text. Then, we choose five words in the caption which occur in the vocabulary as attributes for each data.

### _Hyperparameters; Baseline and Ablated Models_

**Text Attribute Extractor**. The convolution layer of the text attribute extractor has kernel size \(5\times 5\) and stride \(1\times 1\). The max pooling layer has kernel size \(8\times 8\) and stride \(0\). We train the text attribute extractor for 10 epochs.

**AIC-AB Net network**. In the decoding stage, words in captions are embedded into 255-dimension vectors, and words of attributes are embedded into 51-dimension vectors using the default word embedding function provided by PyTorch. The hidden size is set as \(512\). Adam optimizer with learning rate decay is employed to train the model. The parameters is set as: \(\alpha=0.8\), \(\beta=0.999\), \(learning\_rate=4e-4\). The decay of learning rate is modeled as:

\[l_{r}^{E+1}=l_{r}^{E}*0.5^{\frac{E-20}{80}},E>20 \tag{7}\]

where \(l_{r}^{E}\) is the learning rate in epoch \(E\). We train the extractor for 50 epochs.

We compare AIC-AB Net to the following baseline and ablated models. 1.

1. **Adaptive**: the state-of-the-art method, Adaptive [10]. Note that our model without the text attributes (i.e., being ablated) is the same as Adaptive.
2. **The Vanilla Encoder-Decoder (Vanilla-ED)**: an ablated AIC-AB Net, where we remove the adaptive attention architecture and attribute information while keeping a CNN-based encoder and LSTM-based decoder.
3. **The Text Attributes Only (Attr-Only)**: a second ablated AIC-AB Net, where we feed the attribute information into the LSTM decoder but remove the attention architecture.

## V Results and Discussion

We evaluate image captioning on the MS COCO dataset, the Fashion Bestseller dataset, and the Fashion 9 vendors dataset. Table I reports the evaluation results for these three datasets, where B-\(n\) is BLEU score that uses up to \(n\)-grams. In each column, higher is better.

We observe that our AIC-AB Net achieves the best performance compared to all three ablated versions. The ablation study reveals the complementarity of all constituents of AIC-AB Net. In terms of CIDEr score, the Vanilla Encoder-Decoder network underperforms AIC-AB Net by 0.143, 0.319, and 0.148; the Adaptive attention network by 0.017, 0.095, and 0.095; the Attributes-combined model by 0.107, 0.201, and 0.142. Note that adaptive attention architecture improves the performance better than the attribute information. These results indicate that two components indeed complement each other, and their co-existence crucially benefits the caption generation.

The three experimental conditions establish a comprehensive spectrum. The general image dataset, MS COCO, is the most complex and contains multi objects in each image, for which the CIDEr score is the lowest across the datasets. We only compare the CIDEr score because it is the only metric that keeps a stable scale when the number of captions varies. The fashion dataset contains one single object per image. The Fashion Bestseller dataset is simpler than the Fashion 9 vendors dataset. Although the effectiveness of our network is still obvious, the performance gap widens as the task gets more complicated.

Although the attributes-combined model obtains a similar CIDEr score on the COCO dataset compared with the adaptive attention model, it observably underperforms on other scores. CIDEr focuses more on semantical correctness, while others

Fig. 3: An illustration of AIC-AB Net generating the \(t\)-th word \(y_{t}\). The input is the encoded image features \(V\) and attribute information \(A\).

Fig. 2: A simplified diagram of our attribute-image-combined network (AIC-AB Net).
reflect on grammaticality correctness [23]. These results indicate that attribute information provides significant semantic information. However, to demonstrate these attributes in the generated captions, the model achieves this at the expense of grammatical correctness as a result on MS COCO shows "attr-only" gains a similar CIDEr score as "Adaptive" but its BLEU scores are significantly poorer. Interestingly, it does not happen on the Fashion dataset. We argue that this is because of the small number of captions. The sentence pattern is easier to recognize on the Fashion dataset. However, this effect is not shown in our AIC-AB Net network. It reveals the attention architecture, especially the sentinel gate, corrects the bias brought by the attributes. The two components indeed complement each other.

On the fashion dataset, we observe that our model achieves better performance on the Fashion Bestseller dataset than the Fashion 9 vendors dataset, with an improvement of 0.892 (CIDEr score). This observation is the opposite of the regular pattern in which the increased data size improves the ML model's performance. The reason is the distinct styles and forms of captions from different vendors. The huge gaps in captions from one vendor to another are caused by the sub-standard labeling of the Fashion 9 vendors dataset.

### _Attention Distribution Analysis_

To better understand our model, we also visualize the image attention distributions \(\alpha\) for the generated caption. Using bilinear interpolation and pyramid expansion, we sample the attention map to the image size (\(224\times 224\)). Fig. 4 shows the generated captions and the image attention distribution for specific words in the caption. The first five cases are success cases, and the last case shows a failure example. We see that our model learns to pay attention to the specific region when generating different words in the caption, which corresponds strongly with human intuition. Note that on the failure case, although our model fails to focus on the region of the sleeves when generating "sleeves", it still successfully recognizes the position of the printed stripe.

Since the COCO dataset provides the ground truth of objects' bounding box, it can be used to evaluate the performance of attention map generation. The spatial intersection over union (sIOU) score is used to measure localization accuracy. Given the word \(w_{t}\) and its corresponding attention map \(\alpha_{t}\), we first segment the regions of the image with its attention value more extensive than a pre-class threshold \(th\) (after the map is standardly normalized to scale [0,1), where we set as \(0.6\). Then we take the bounding box covering the largest connected component in this segmentation map as the predicted attention region. We report the sIOU between the predicted bounding box and the ground truth for the top 20 most frequent COCO object categories, as Fig. 4(a) shows. The average localization accuracy for the "Adaptive" is 0.415, and 0.419 for our AIC-AB Net. This implies that the attribute information benefits the attention map generation as a combined model. We also observe that our AIC-AB Net and its attention-only version have a similar trend. They both perform well on informative visual objects and large objects such as "cat", "train", "bed", and "bus", while they have poor performance on small objects such as "sink" and "clock". We argue that it is because our attention map is extracted from \(7\times 7\) spatial map, which loses plenty of resolution and detail. This defect is remarkably exposed when detecting small objects. This reason can explain the wrong attention map on the Fashion dataset as well, where the majority of words are descriptions of details and refer to small regions on the image.

Since the bounding box ground truth is missing in the Fashion dataset, we apply statistic analysis for 5 typical words as quantitative analysis, _hood_, _cap_, _pants_, _dresses_, and _sleeves_. In common cases, _hood_ and _cap_ only show on the upper part of an image, _pants_ and _dresses_ only on the lower part, and _sleeves_ only on the left and right sides. We assume these regions are their ground truth respectively and apply the same approach as explained above to measure the localization accuracy. Fig. 4(b) reports the result of the Fashion dataset. We observe that AIC-AB Net outperforms on the first 4 words than the word _sleeves_ and shows a similar trend with the adaptive attention model.

## VI Conclusion

This work has been motivated by the task of generating captions for single-object fashion images and inspired by the adaptive attention architecture [10] and semantic

\begin{table}
\begin{tabular}{l c c c c c c} \hline Model & B-1 & B-2 & B-3 & B-4 & METEOR & ROUGE-L & CIDEr \\ \hline  & (a) Scores of image captioning on the MS COCO dataset & & & & \\ Adaptive & **0.743** & **0.572** & 0.424 & 0.313 & 0.265 & 0.546 & 1.088 \\ Vanilla-ED & 0.729 & 0.556 & 0.409 & 0.299 & 0.249 & 0.531 & 0.952 \\ Attr-Only & 0.671 & 0.495 & 0.366 & 0.276 & 0.255 & 0.535 & 1.059 \\ AIC-AB Net & 0.730 & 0.554 & **0.424** & **0.339** & **0.279** & **0.550** & **1.105** \\  & (b) Scores of image captioning on the Fashion Bestseller dataset & & & & & \\ Adaptive & 0.365 & 0.306 & 0.275 & 0.255 & 0.185 & 0.334 & 2.094 \\ Vanilla-ED & 0.345 & 0.284 & 0.251 & 0.231 & 0.173 & 0.316 & 1.870 \\ Attr-Only & 0.350 & 0.290 & 0.258 & 0.240 & 0.179 & 0.321 & 1.988 \\ AIC-AB Net & **0.385** & **0.316** & **0.289** & **0.280** & **0.190** & **0.349** & **2.189** \\  & (c) Scores of image captioning on the Fashion 9 vendors dataset & & & & & \\ Adaptive & 0.276 & 0.218 & 0.193 & 0.178 & 0.121 & 0.256 & 1.202 \\ Vanilla-ED & 0.264 & 0.216 & 0.179 & 0.165 & 0.115 & 0.238 & 1.149 \\ Attr-Only & 0.268 & 0.210 & 0.184 & 0.169 & 0.117 & 0.242 & 1.155 \\ AIC-AB Net & **0.290** & **0.231** & **0.202** & **0.191** & **0.125** & **0.268** & **1.297** \\ \hline \end{tabular}
\end{table} TABLE I: Evaluation Scores of Image Captioning
concept [9]. Toward this end, we present Attribute-Image-Combined Attention-Based Network (AIC-AB Net). We have evaluated AIC-AB Net on the MS COCO and the Fashion datasets. Our experiments indicate that the ability to locate the relevant region of an image when generating different words and the combination with the attribute information is crucial for accurate caption generation.

Further research could explore two directions. First, according to the results of transfer learning, we suggest creating an up-to-standard labeling system for the Fashion dataset, which will benefit the consistency of the data and the robustness of the models trained on it. Second, we argue that segmenting the images into more regions will improve the performance since, in some cases, our model cannot pay attention to the accurate region when generating words.

Image captioning is a challenging and promising task for the Internet industry and computer vision. We believe this work represents a significant step in improving image captioning and breeds useful applications in other domains.

Fig. 4: Visualization of generated captions and image attention maps on the Fashion Bestseller dataset. The text is the captions generated by AIC-AB Net. Different colors denote a correspondence between masked regions and underlined words. The first 5 cases are success cases; the last case is a failure.

Fig. 5: Localization accuracy over generated captions. Adaptive is the baseline model (an ablated version of AIC-AB Net); AIC-AB Net is our model.