Multi-task convolutional neural network for image aesthetic assessment
As people's aesthetic preferences for images are far from understood, image
aesthetic assessment is a challenging artificial intelligence task. The range
of factors underlying this task is almost unlimited, but we know that some
aesthetic attributes affect those preferences. In this study, we present a
multi-task convolutional neural network that takes into account these
attributes. The proposed neural network jointly learns the attributes along
with the overall aesthetic scores of images. This multi-task learning framework
allows for effective generalization through the utilization of shared
representations. Our experiments demonstrate that the proposed method
outperforms the state-of-the-art approaches in predicting overall aesthetic
scores for images in one benchmark of image aesthetics. We achieve near-human
performance in terms of overall aesthetic scores when considering the
Spearman's rank correlations. Moreover, our model pioneers the application of
multi-tasking in another benchmark, serving as a new baseline for future
research. Notably, our approach achieves this performance while using fewer
parameters compared to existing multi-task neural networks in the literature,
and consequently makes our method more efficient in terms of computational
complexity.
Derya Soydaner, Johan Wagemans
# Multi-task convolutional neural network for image aesthetic assessment

###### Abstract

As people's aesthetic preferences for images are far from understood, image aesthetic assessment is a challenging artificial intelligence task. The range of factors underlying this task is almost unlimited, but we know that some aesthetic attributes affect those preferences. In this study, we present a multi-task convolutional neural network that takes into account these attributes. The proposed neural network jointly learns the attributes along with the overall aesthetic scores of images. This multi-task learning framework allows for effective generalization through the utilization of shared representations. Our experiments demonstrate that the proposed method outperforms the state-of-the-art approaches in predicting overall aesthetic scores for images in one benchmark of image aesthetics. We achieve near-human performance in terms of overall aesthetic scores when considering the Spearman's rank correlations. Moreover, our model pioneers the application of multi-tasking in another benchmark, serving as a new baseline for future research. Notably, our approach achieves this performance while using fewer parameters compared to existing multi-task neural networks in the literature, and consequently makes our method more efficient in terms of computational complexity.

 Image aesthetics, convolutional neural network, deep learning, multi-task learning, regression, image aesthetic assessment.

## 1 Introduction

Image aesthetic assessment is a challenging task due to its subjective nature. Some people may find an image aesthetically pleasing, while others may disagree. Aesthetic preferences of individuals are diverse and they can depend on many factors. Because of the importance and complexity of the problem, the literature on automated image aesthetic assessment is extensive [4, 40]. In recent years, deep learning has become an important part of this literature based on its substantial impact in many areas. Given that deep neural networks can already perform tasks that were previously thought to be exclusive to humans, such as playing games [32], it is not unreasonable to expect them to be able to assess the aesthetic value of images as well. Currently, image aesthetic assessment has a significant impact on many application areas such as automatic photo editing and image retrieval.

In this context, neural networks have become a powerful tool in _computational aesthetics_. This interdisciplinary field of research is of great importance for the automatic assessment of image aesthetics, and has led to the development of several state-of-the-art models for aesthetics research. In this study, we aim to evaluate a computational approach for image aesthetics which considers the overall aesthetic score as well as individual attributes that can impact aesthetic preferences. Therefore, we focus on the multi-task setting to assess the model's performance across multiple tasks. We handle the image aesthetic assessment task as a regression problem, i.e., our aim is predicting the aesthetic ratings for images. However, predicting overall aesthetic scores using regression-based approaches is complicated because aesthetic liking is influenced by a multitude of interacting factors. Many of these factors are subjective, and their combined effect is notoriously difficult to predict. This difficulty is compounded further in a multi-task setting, making the task even more challenging.

To this end, we propose a multi-task convolutional neural network (CNN) that predicts an overall aesthetic score for a given image while also learning important attributes related to aesthetics. To demonstrate the effectiveness of our approach, we evaluate our multi-task CNN on two benchmark datasets in aesthetics research, namely the Aesthetics with Attributes Database (AADB) [17] and The Explainable Visual Aesthetics (EVA) dataset [13]. These datasets are unique in that they provide both overall aesthetic scores and attribute scores, making them valuable resources for evaluating image aesthetic assessment models.

Our proposed multi-task CNN performs well for image aesthetic assessment, while being efficient in terms of computational complexity. Our multi-task CNN is the first of its kind applied to the EVA dataset, making it a new baseline for the multi-task setting on this dataset. Moreover, it achieves near-human performance on the overall aesthetic scores of the AADB dataset while having fewer parameters than the previous studies in the literature, demonstrating the principle of Occam's razor in machine learning.

The rest of the paper is organized as follows. In Section 2, related work is presented. We introduce our multi-task CNN in Section 3. We describe our experimental setup in Section 4 and we discuss our results in Section 5. Finally, the conclusions and outlook are given in Section 6.
### _Contributions_

Our main contributions are summarized as follows.

* We propose an end-to-end multi-task CNN for image aesthetic assessment and conduct systematic evaluation of our model on two image aesthetic benchmarks.
* In the multi-task setting, our model achieves the state-of-the-art result on the overall aesthetic scores of the AADB dataset, while requiring fewer parameters than previous approaches.
* On the more recent EVA dataset, we conduct performance analysis and our model is the first multi-task CNN for this dataset, serving as the new baseline.
* Our evaluation shows that the multi-task setting consistently outperforms the single-task setting for the same neural network architecture across both datasets.
* As a result, we present a simple yet effective multi-task neural network architecture for image aesthetic assessment and provide a detailed evaluation of it on both image aesthetic datasets.

### _Problem formulation_

In this study, our aim is to develop a model that predicts aesthetic-related scores of images. We use aesthetic benchmarks that include images with overall aesthetic scores and scores for \(K\) aesthetic attributes. Our model learns from the training set \(\left\{(x^{(1)},x^{(2)},...,x^{(N)})\right\}_{i=1}^{N}\) with corresponding targets \(y^{(i)}\). Here, each training sample consists of an RGB image \(x^{(i)}\in\mathbb{R}^{d}\). Correspondingly, \(y^{(i)}\in\mathbb{R}^{K+1}\) is a concatenated vector of the overall aesthetic score \(y^{(i)}_{o}\in\mathbb{R}\) and scores for \(K\) aesthetic attributes \(y^{(i)}_{a}\in\mathbb{R}^{K}\). Our model learns from this training data to accurately predict the aesthetic-related scores of images.

Such problems, where the output is a numerical value, are known as the _regression_ problems. Here, the task is to learn the mapping from the input to the output. To this end, we assume a machine learning model of the form

\[y=f(x|\theta), \tag{1}\]

where \(f(.)\) denotes the model and \(\theta\) represents its parameters. Since we have images as input data, we choose the convolutional neural network (CNN) as the model \(f(.)\). We use a CNN in a _multi-task_ setting, as the target vector \(y^{(i)}\) includes overall aesthetic score and scores for \(K\) aesthetic attributes. Given the training set of \(N\) samples \(D=\left\{(x^{(i)},y^{(i)})\right\}_{i=1}^{N}\), our goal is to obtain a network \(f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{K+1}\), where \(f\) can simultaneously predict the overall aesthetic scores and attribute scores from input images.

## 2 Related Work

The task of image aesthetic assessment is typically approached as either a binary classification problem, where the aim is to classify an image as low or high aesthetics, or as a regression problem, where a model predicts an aesthetic score for a given image. Prior studies have investigated both classification and regression-based approaches to image aesthetic assessment. Deep learning techniques have achieved remarkable success in various fields, and image aesthetic assessment is no exception, as evidenced by the increasing number of studies exploring the use of deep neural networks in this area. It is clear that these techniques have played a critical role in contributing to notable advances in image aesthetics research. For example, Kang _et al._ (2014) [12] presented a CNN that predicts image quality, while Lu _et al._ (2014) [25] proposed an aesthetics classification network that learns several style attributes. Lee _et al._ (2019) [19] utilized a Siamese network-based approach in this area. In another study, Lu _et al._ (2015) [26] developed a deep neural network for image style recognition, aesthetic quality categorization, and image quality estimation. In some studies, models based on CNNs were used to predict a single aesthetic score for an image [14, 15]. Inspired by a visual neuroscience model, Wang _et al._ (2017) [38] introduced a model for image aesthetics assessment that predicts the distribution of human ratings.

Assessing the aesthetic quality of images involves numerous factors that contribute to preferences, and while many of these factors are difficult to quantify, there are some known aesthetic attributes that influence preferences. Previous studies have investigated the aesthetic value of images in conjunction with these attributes. Recently, deep neural networks based on multi-task learning have been employed to tackle this task, treating it as a multi-task problem that simultaneously predicts an overall aesthetic score and multiple attribute scores. For instance, Kong _et al._ (2016) [17] introduced the AADB dataset, which includes overall aesthetic scores and scores for eleven attributes of photos. In their study, Kong _et al._ (2016) [17] developed a multi-task neural network by fine-tuning AlexNet [18], and training a Siamese network [2] to predict aesthetic ratings. Subsequent studies have also utilized the AADB dataset for further research in this field. For example, Hou _et al._ (2017) [10] applied the squared earth mover's distance-based loss for training, and compared different deep networks including AlexNet, VGG16 [35], and a wide residual network [39], and found that fine-tuning a VGG-based model achieved the best performance.

Pan _et al._ (2019) [30] proposed a neural network architecture based on adversarial learning inspired by generative adversarial networks [8]. This is a multi-task deep CNN namely "rating network" which learns the aesthetic score and attributes simultaneously. While the rating network plays the role of "generator", a "discriminator" tries to distinguish the predictions of multi-task network from the real values. This model outperforms previous approaches and is currently considered as the state-of-the-art method for predicting the overall aesthetic scores on the AADB dataset in multi-task aesthetic prediction.

Since most of the images rated null (neutral) for three attributes (symmetry, repetition and motion blur) in the AADB dataset (Fig. 3), some studies [1, 21, 27, 31] have chosen to exclude these attributes from their multi-task models. For instance, Malu _et al._ (2017) [27] developed a multi-task CNN based on ResNet-50 [9] which simultaneously learns the eight aesthetic attributes along with
the overall aesthetic score. They also examined the salient regions for the corresponding attribute and applied the gradient based visualization technique [41]. Abdenebaoui _et al._ (2018) [1] used a deep CNN that predicts technical quality, high-level semantic quality, and a detailed description of photographic rules. Reddy _et al._ (2020) [31] proposed a multi-task network based on EfficientNet [37] for the same purpose, along with a visualization technique and activation maps generated using Gradient-weighted Class Activation Mapping (Grad-CAM) [33] to generate activation maps. Recently, Li _et al._ (2022) [21] presented a hierarchical image aesthetic attribute prediction model.

Besides, Li _et al._ (2020) [20] proposed a multi-task deep learning framework that takes into account an individual's personality in modeling their subjective preferences. Liu _et al._ (2020) [24] developed an aesthetics-based saliency network in a multi-tasking setting. The aesthetic evaluation system proposed by Jiang _et al._ (2021) [11] outputs the image style label and three forms of aesthetic evaluation results for an image.

More recently, another image dataset, namely The Explainable Visual Aesthetics (EVA) [13], has been released, which includes overall aesthetic scores and attribute scores. Although there are a few studies that have used this dataset for aesthetics research, their models only predict overall aesthetic scores [22, 23, 34]. Therefore, our study is the first multi-task neural network that can make predictions on the EVA dataset.

Current multi-task learning approaches have demonstrated the feasibility of predicting ratings in the AADB dataset by utilizing all the available attributes instead of excluding some. In line with this, we employ all the attributes in the AADB dataset and develop a neural network architecture that is both efficient and effective in predicting overall aesthetic scores of images. Moreover, we evaluate our multi-task CNN on the EVA dataset to further assess its performance. Our multi-task CNN provides predictions for the EVA dataset, serving as a baseline for the future research in this area.

## 3 Proposed Multi-Task Convolutional Neural Network

We propose a deep multi-task CNN that jointly learns the overall aesthetic score and the aesthetic-related attributes for images during training. This allows the resulting neural network to simultaneously predict multiple scores for an image. We train our deep neural network directly from RGB images, and it is based on the VGG16 pretrained network to extract features. The prior multi-task approaches mentioned in Section 2 have already proved that using a pretrained network is a better option than training a neural network from scratch, since both the AADB and EVA datasets do not include large amounts of images. We conducted experiments on several candidate per-trained CNNs to determine the optimal architecture for our task, and selected the model with the highest performance.

The proposed neural network takes images as input and uses VGG16 to extract feature representations, as shown in Fig. 1. We removed the fully-connected layers in VGG16 and used the five blocks of convolutional layers. We added a global average pooling layer to the output of the last convolutional block of VGG16. The resulting feature maps are fed into two fully-connected layers with ReLU activation function [7, 29], consisting of 128 and 64 hidden units, respectively. To prevent overfitting, we applied dropout [36] with a rate of 0.35 to the second fully-connected layer with 64 hidden units, which precedes the output layer. The architecture of our neural network consists of multiple units in the output layer, one for predicting the overall aesthetic score and additional units for predicting attribute scores. For the AADB dataset, which has 11 attributes, there are 12 output units in total. On the other hand, the EVA dataset has 4 attributes, so our model includes 5 output units. For more information about the datasets, please refer to Section 4.1. The output layer applies sigmoid activation function, and all the output units share the same hidden representation. Notably, we also designed our multi-task CNN with separate output layers: one for the overall aesthetic score and one

Fig. 1: The general architecture of our multi-task convolutional neural network.
for each attribute. Interestingly, both architectures perform similarly for predicting the overall aesthetic score. However, we found that the architecture with a single output layer outperforms the one with separate output layers when it comes to predicting the attribute scores.

## 4 Experimental Setup

### _Datasets_

We evaluate the performance of our proposed multi-task CNN on two publicly available datasets for image aesthetic assessment, namely the AADB and EVA. Both datasets provide aesthetic attribute scores that are suitable for regression modeling, in addition to the overall aesthetic scores. The Aesthetic Visual Analysis (AVA) dataset [28] is another widely used benchmark in aesthetics research. However, the AVA dataset only provides binary labels for attributes, which is not suitable for our proposed framework as it requires rating scores. Additionally, many images in the AVA dataset are either heavily edited or synthetic, which limits its applicability. In contrast, the AADB dataset provides a more balanced distribution of professional and consumer photos, as well as a more diverse range of photo qualities [17]. Consequently, we utilize the AADB and EVA datasets for training and evaluating our multi-task CNN, as they are the most appropriate datasets for this study. These datasets are described below.

**AADB.** We utilize the Aesthetics with Attributes Database (AADB) [17], an image aesthetic benchmark containing 10,000 RGB images of size 256 \(\times\) 256 collected from the Flickr website. Each image in the AADB dataset has overall aesthetic scores provided by 5 different raters. The scores are on a scale of 1 to 5, with 5 being the most aesthetically pleasing score. Additionally, there are eleven attributes that are known to impact aesthetic judgments according to professional photographers. In this dataset, every image has also scores for each attribute. These attributes are balancing element, interesting content, color harmony, shallow depth of field, good lighting, motion blur, object emphasis, rule of thirds, vivid color, repetition, and symmetry. The raters indicated whether each attribute has a positive, negative, or null (zero) effect on the aesthetics of an image, except for repetition and symmetry where only the presence or absence of the attribute is rated.

To obtain the ground-truth scores for each image in the AADB dataset, Kong _et al._ (2016) [17] calculated the average aesthetic scores provided by five different raters. Since only the average scores are reported, the individual rater scores are not available in the dataset. Then, the average scores are normalized to the range of [0,1], while all the attributes except for repetition and symmetry are normalized to the range of [-1,1]. Repetition and symmetry are normalized to the range of [0,1]. Two sample images from the AADB dataset, showcasing examples of both low and high aesthetics, are shown in Figure 2.

The AADB dataset includes eleven attributes in total, and the distribution of these attributes is presented in Fig. 3. Among them, the motion blur, repetition, and symmetry attributes are mostly rated neutral. Therefore, as mentioned in Section 2, some researchers excluded these three attributes from their multi-task neural networks. However, the motion blur attribute has both negative (700) and positive (397) scores, which may still provide useful information. Similarly, the repetition attribute has 1683 positive scores, while the symmetry attribute has 771 positive scores. It is worth noting that raters were not allowed to give negative scores for repetition and symmetry.

The AADB dataset has been split into three subsets: 500 images for validation, 1000 images for testing, and the remaining images for training, following the official partition [17]. For our experiments, we use this partition to train and test our multi-task CNN, allowing for direct comparison with other approaches.

**EVA.** The Explainable Visual Aesthetics (EVA) dataset [13] contains 4070 images, each rated by at least 30 participants. The EVA dataset overcomes the limitations of previous datasets by including images with 30 to 40 votes per image, collected using a disciplined approach to avoid noisy labels due to misinterpretations of the tasks or limited number of votes per image [13]. Each image has an aesthetic quality rating with an \(11\)-point discrete scale. The extremes

Fig. 3: Visualization of image attribute data in the training set of AADB dataset illustrating the distribution of negative, null, and positive levels for each attribute [17].

Fig. 2: Example images from the training set of the AADB dataset. Each image has overall aesthetic score and scores for 11 attributes. _(Left)_ High aesthetic: An image rated high on overall aesthetic score. _(Right)_ Low aesthetic: An image rated low on overall aesthetic score.
of the scale are labelled as "least beautiful" (corresponding to 0) and "most beautiful" (corresponding to 10). The EVA dataset contains four attributes: light and color, composition and depth, quality, and semantics of the image. For each attribute, the images were rated on a four-level Likert scale (very bad, bad, good, and very good). Two sample images from the EVA dataset, showcasing examples of both low and high aesthetics, are shown in Figure 4.

In contrast to the AADB dataset, Kang _et al._ (2020) [13] reported all ratings from the participants. So, we calculated the average scores for each image. Unlike the AADB dataset, which has predetermined train-validation-test splits, there is no official train-validation-test split for the EVA dataset, since Kang _et al._ (2020) did not use any neural network. However, studies focusing on predicting only the overall aesthetic scores (see Section 2) on the EVA dataset have utilized different training and testing splits. For example, Duan _et al._ (2022) [5] and Li _et al._ (2023a) [23] employed a split of 3,500 training images and 570 testing images, while Li _et al._ (2023) [22] used 4,500 training images and 601 testing images. Similarly, Shaham _et al._ (2021) [34] utilized a split of 2,940 training images and 611 testing images.

### _Implementation Details_

We initialize the fully-connected layer weights in our multi-task CNN with the Glorot uniform initializer [6]. We use _mean squared error_ as the loss function on the training set \(X\) to minimize the error between the predictions and the ground-truth values:

\[E(W|X)=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2} \tag{2}\]

where \(n\) is the number of samples in the training set, \(y_{i}\) are the ground-truth scores and \(\hat{y_{i}}\) are the predictions generated by Eq. 1.

Since both datasets we use in this study do not include large amounts of images, we apply horizontal flip as data augmentation. We train our multi-task CNN in two stages. In the first stage, we apply the Adam algorithm [16] with an initial learning rate of 0.001 and decay constants of 0.9 and 0.999, respectively. The VGG16 pretrained network is composed of five blocks, each of which includes convolutional and pooling layers. During the first stage, we freeze the weights for all five blocks and train the multi-task CNN for 5 epochs, with a minibatch size of 64. We closely monitor the training and validation loss during this stage and observe that the model is prone to overfitting if we train it for longer, without any notable improvement in its performance.

In the second stage, we fine-tune the multi-task CNN by unfreezing the last two convolutional layer in the fourth block of VGG16. We apply the Adam algorithm again, but we adjust its learning rate by using the exponential decay learning rate schedule. The initial learning rate is 0.0001, and it decays every 125 steps with a base of 0.50. We fine-tune the model with this setting for 3 epochs with a minibatch size of 64.

## 5 Results and Discussion

In this section, we provide a comprehensive evaluation of our proposed multi-task CNN on both datasets, with an emphasis on its efficiency in assessing image aesthetics. We explore the impact of fine-tuning and analyze the results for both overall and attribute scores predicted by our model. Additionally, we compare the performance of our multi-task approach with a single-task setting for predicting the overall aesthetic scores of images using the same neural network architecture.

### _Performance analysis on the AADB dataset_

#### 5.1.1 Model evaluation and comparison with the state-of-the-art

Table I provides an overview of the performances achieved by the studies in the literature which use the AADB dataset to develop multi-task deep neural networks. These neural networks learn the eleven attributes of the AADB dataset along with the overall aesthetic score of images. The reported results show the progress made by previous studies, with each subsequently becoming the state-of-the-art in this field. Since our aim is similar to these previous studies, we evaluate how well our multi-task CNN performs in comparison to these neural networks.

To make a comparison between the previous studies and our multi-task CNN, we use Spearman's rank correlation

\begin{table}
\begin{tabular}{l l}
**METHODOS** & \(\rho\) \\ \hline (Kong _et al._, 2016) & 0.6782 \\ (Hou _et al._, 2017) & 0.6889 \\ (Pan _et al._, 2019)\({}^{a}\) & 0.6927 \\ (Pan _et al._, 2019) & 0.7041 \\ Ours & **0.7067** \\ \end{tabular}
\end{table} TABLE I: Comparison of performances achieved by previous multi-task neural networks and our proposed multi-task CNN on the test set of the AADB dataset.

Fig. 4: Example images from the training set of the EVA dataset. Each image has overall aesthetic score and scores for 4 attributes. _(Left)_ High aesthetic: An image rated high on overall aesthetic score. _(Right)_ Low aesthetic: An image rated low on overall aesthetic score.
coefficient (\(\rho\)), which is a commonly used metric in this field. Table I summarizes the \(\rho\) values reported in each study, which represent the correlation between the estimated overall aesthetic scores by the multi-task neural network and the corresponding ground-truth scores in the test set. We calculate this correlation using the overall aesthetic scores predicted by our multi-task CNN and find it to be significant at p \(<\) 0.01. This allows us to compare the performance of our model to those in the literature.

As shown in Table I, there has been a slight improvement in the correlation between predicted overall scores and ground-truth scores over the years. The approach proposed by Pan _et al._ (2019) [30] has resulted in the highest correlation achieved thus far. Their study includes two methods, the first of which is a multi-task deep neural network that achieves a Spearman's rank correlation of 0.6927. The second one takes the first method one step forward by updating it with an adversarial setting, as described in Section 2. Compared to these methods, our multi-task CNN outperforms the first method (0.7067 \(>\) 0.6927). When we compare our model with the adversarial learning setting proposed by Pan _et al._ (2019) [30], we find that our neural network outperforms theirs again (0.7067 \(>\) 0.7041). Even though there isn't much of a difference in the results of the current approaches in terms of Spearman's rank correlations, our multi-task CNN currently performs slightly better than previous models.

In addition to achieving the highest Spearman's rank correlation, our multi-task CNN has other advantages over the state-of-the-art approach proposed by Pan et al. (2019) [30]. To compare the complexity of the two approaches, we examine the number of parameters in each model. Table II compares the neural network architectures of current state-of-the-art [30] and our multi-task CNN by taking into account the number of parameters. While Pan _et al._ (2019) [30] uses ResNet-50 for feature extraction, we utilize VGG16, which has fewer parameters for this particular problem. Furthermore, the fully-connected layers of our model have significantly fewer parameters compared to those in Pan _et al._ (2019) [30]. Specifically, our model's fully-connected layers have around 15 times fewer parameters in the neural network compared to their model. Similarly, the output layer of our multi-task CNN has fewer parameters, too. Moreover, the adversarial setting used in Pan _et al._ (2019) [30] makes the training of their neural network more complex.

We also evaluate the predictions made by our multi-task CNN and investigate the issue of overfitting. Fig. 5 shows that our model can predict overall aesthetic scores across a wide range. While the ground-truth overall aesthetic scores in the test data range from 0.05 to 1.0, our model's predictions range from 0.26 to 0.90. We also report the frequencies and percentages of ground-truth overall aesthetic scores in Table III for different intervals in the test data. Based on these data, our model cannot make predictions for 39 samples falling in intervals [0.05-0.10] and [0.10-0.20], and it fails for 15 samples falling in the interval [0.90-1.00]. In other words, our multi-task CNN can make successful predictions for approximately 95\(\%\) of test data. This indicates that our multi-task CNN is able to make predictions for the majority of the test data, with only a small percentage of samples falling outside of its prediction range. These results also indicate that there is no issue of overfitting.
present the most successful predictions in Fig. 6 and the least successful ones in Fig. 7, respectively. As shown in Fig. 6, our multi-task CNN makes accurate predictions for most images, with the exception of one low-aesthetic image with the ground-truth score of 0.15. On the other hand, in the least successful predictions shown in Fig. 7, our model tends to predict high scores to low aesthetic images, while giving lower scores to the high aesthetic images.

Overall, our multi-task CNN achieves the highest Spearman's rank correlation for overall aesthetic scores while

Fig. 6: Comparison of ground-truth overall aesthetic scores and corresponding predictions by our multi-task CNN on the test data of the AADB dataset. This figure shows the most successful predictions ranging from low aesthetic images to high aesthetic images.

Fig. 7: Comparison of ground-truth overall aesthetic scores and corresponding predictions by our multi-task CNN on the test data of the AADB dataset. This figure shows the least successful predictions ranging from low aesthetic images to high aesthetic images.
simultaneously predicting scores for 11 attributes. Notably, our approach accomplishes this with fewer parameters, making it more computationally efficient than the state-of-the-art method proposed by Pan _et al._ (2019) [30]. By combining a simplified neural network architecture with superior predictive performance, our approach represents a significant advancement in the field of image aesthetic assessment.

#### 5.1.2 Comparison with human performance

In addition to evaluating the performance of our multi-task CNN in rating image aesthetics, we compare its results with human performance on the AADB dataset. Kong _et al._ (2016) [17] previously reported the Spearman's rank correlation between each individual's ratings and the ground-truth average score on this dataset. A subset of raters was selected based on the number of images they have rated. In their study, they found that the more images an individual rated, the more stable their aesthetic score rankings became. We utilize this data and compare it to the performance of our model, as shown in Table IV.

Based on these correlations, we see that when the number of images rated by the same observer increases, human performance becomes better. On the other hand, it is also clear that our multi-task CNN performs above the level of human consistency averaged across all raters. Only when compared to the more experienced raters (i.e., the 42 raters who rated \(>\)200 images), our model performs slightly less. Our experiments demonstrate that our multi-task CNN achieves near-human performance in predicting the overall aesthetic scores on the AADB dataset. This narrows the performance gap between machines and humans in this domain.

#### 5.1.3 Attribute predictions and the fine-tuning effect

To further evaluate the performance of our multi-task CNN, we analyze its ability to predict image attributes in the AADB dataset and report the results in Table V. The table displays the Spearman's rank correlations between the ground-truth scores and the corresponding predictions made by our multi-task CNN for each attribute. Moreover, we investigate the effect of fine-tuning and include those results in Table V. As described in Section 4.2, we train our multi-task CNN, and then we fine-tune the model by unfreezing the last two convolutional layer in the fourth block of VGG16 (\(block4\_conv2\) and \(block4\_conv3\)). After fine-tuning our model, we observe an increase in the correlations for all attributes except symmetry. We also observe an increase in the correlation for the overall aesthetic score. Moreover, we aim to gain insight into the two convolutional layers that we fine-tune. To this end, we illustrate the activation maps generated using Grad-CAM [33] in Fig. 8 for two images from the test set of AADB dataset, one with a low aesthetic score and the other with a high aesthetic score.

We report the Spearman's rank correlations on the AADB dataset in Fig. 9. This figure shows the correlations for the ground-truth scores of all dataset on the left side, whereas on the right side, we present the correlations for the predictions on the test data made by our multi-task CNN. As it is known, the Spearman's rank correlation measures the strength and direction of the relationship between two variables, and it ranges from -1 to 1. A correlation of 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no correlation. We examine the correlations between the overall aesthetic score and the 11 attributes. Our model's highest correlations among all attributes are for the light (\(\rho\)=0.96) and content (\(\rho\)=0.95) attributes. This finding is consistent with the AADB dataset, where content has the highest correlation with overall aesthetic scores (\(\rho\)=0.70) and light follows in second place (\(\rho\)=0.58). Furthermore, when we compare the top-five correlations for our multi-task CNN (light, content, color harmony, vivid color, and rule of thirds attributes), we see the similar results in the AADB dataset indicating our model can capture the relationships between the overall aesthetic scores and the attributes. On the other hand, when we examine the lowest correlations in Fig. 9, we find that our model also exhibites lower correlations for the motion blur, symmetry, and repetition attributes, consistent with human data. Accordingly, we can conclude that the predictions made by our multi-task CNN closely match human interpretation.

#### 5.1.4 Single-task versus multi-task setting

We wondered what would happen if our proposed model were a single-task neural network instead of a multi-task one. In this case, the neural network just learns the overall aesthetic score, not the scores for attributes. We report our result in Table VI and compare it to those of Pan _et al._ (2019)

\begin{table}
\begin{tabular}{l l l}
**\# images rated** & **\# raters** & \(\rho\) \\ \hline (0,100) & 190 & 0.6738 \\ \([100,200)\) & 65 & 0.7013 \\ \([200,\infty)\) & **42** & 0.7112 \\ Our method & - & **0.7067** \\ \end{tabular}
\end{table} TABLE IV: The Comparison Between Human Performance and Our Multi-Task CNN on the AADB Database.

\begin{table}
\begin{tabular}{l l l}
**Attributes** & **Training** & **Fine-tuning** \\ \hline Content & 0.546 & 0.593 \\ Vivid Color & 0.617 & 0.669 \\ Object Emphasis & 0.600 & 0.639 \\ Color Harmony & 0.437 & 0.484 \\ Depth of Field & 0.466 & 0.497 \\ Lighting & 0.396 & 0.445 \\ Balancing Elements & 0.264 & 0.267 \\ Rule of Thirds & 0.216 & 0.235 \\ Motion Blur & 0.098 & 0.109 \\ Symmetry & 0.194 & 0.177 \\ Repetition & 0.322 & 0.355 \\
**Overall** & **0.650** & **0.707** \\ \end{tabular}
\end{table} TABLE V: Spearman’s rank correlations between the ground-truth scores for each attribute and the predictions by our multi-task CNN in the test data of the AADB Dataset. This table shows the correlations after training and after fine-tuning separately, in addition to the correlations for the overall aesthetic score.
[30]. In terms of single-task networks, the Spearman's rank correlation of our method is slightly higher than Pan _et al._ on the test set of AADB dataset. This indicates that our neural network performs slightly better in the single-task setting while utilizing fewer parameters, highlighting the effectiveness of our approach. Furthermore, we also add the multi-task setting results for both model to make a comparison with the single-task one. Both models show that multi-task learning improves the neural network performance, as the Spearman's rank correlations between the predicted aesthetic scores and ground-truth overall aesthetic scores are consistently higher for the multi-task neural networks than for the single-task ones.

### _Performance analysis on the EVA dataset_

#### 5.2.1 Model evaluation

The second benchmark we use in this study, the EVA dataset, provides access to all participant's ratings. To in

\begin{table}
\begin{tabular}{l c c}
**METHODOS** & **Single-task** & **Multi-task** \\ \hline Pan _et al._, 2019) & 0.6833 & 0.6927 \\ Ours & **0.6890** & **0.7067** \\ \end{tabular}
\end{table} TABLE VI: The performance comparison between single-task and multi-task neural networks in terms of Spearman’s rank correlations.

Fig. 8: The activation maps for two input images from the test set of AADB dataset. These maps highlight the regions of the input image that contributed the most to the neural network’s prediction. The heatmap is overlaid on top of the input image to provide a visualization of which areas of the image are most relevant for the task.

Fig. 9: Spearman’s rank correlations between the overall aesthetic scores and the attribute scores on the AADB dataset. _(a)_: The ground-truth scores on all dataset, _(b)_: The predictions by our multi-task CNN on the test data.
vestigate the performance of our multi-task CNN on this dataset, first, we calculated the average score for each image with respect to each attribute and overall aesthetic score. Table VII reports the minimum and maximum averages for each attribute and overall aesthetic score in the EVA dataset.

Since there are four attributes in the EVA dataset (light and color, composition and depth, quality, semantics), we modify the output layer of our multi-task CNN to include five units (one for the overall aesthetic score and one for each attribute). Consequently, the output layer of our multi-task CNN consists of 325 parameters for the EVA dataset. We applied dropout [36] with a rate of 0.25 to the second fully-connected layer with 64 hidden units, which precedes the output layer. Also, since there is no official train-test split for the EVA dataset, we follow the two studies [23, 5] which use 3,500 images for training and 570 for testing. Table VIII presents the performance of our multi-task CNN on this dataset, also highlighting the effect of fine-tuning. This table summarizes the Spearman's rank correlations between the estimated overall aesthetic scores by our multi-task CNN and the corresponding ground-truth scores in the test set. Here, we again wondered what would happen if our proposed model were a single-task neural network instead of a multi-task one. Therefore, we also evaluated the model's performance in the single-task setting and observed that the multi-task setting outperforms it. Consistent with the findings in Section 5.1.4, we note that predicting the attributes along with the overall aesthetic score has a positive effect on the overall score for the same neural network architecture.

Similar to the evaluation of the AADB dataset in the previous section, we evaluate the predictions made by our multi-task CNN and investigate the issue of overfitting. We compare the actual overall aesthetic scores of test images in the EVA dataset to the predicted scores generated by our model in Fig. 10. While the ground-truth overall aesthetic scores in the test data range from 2.46 to 9.0, our model's predictions range from 5.09 to 8.13. We also report the frequencies and percentages of ground-truth overall aesthetic scores in Table IX for different intervals in the test data. Based on these data, our model cannot make predictions for 83 samples falling in interval range [1.70-5.00], and it fails for some samples falling in the maximum intervals. In other words, our multi-task CNN can make successful predictions for approximately 85\(\%\) of test data. This indicates that our multi-task CNN is able to make predictions for a majority of the test data, with only a small percentage of samples falling outside of its prediction range. These results also indicate that there is no issue of overfitting.

In order to further evaluate the performance of our multi-task CNN, we visually examine its predictions and present the most successful predictions in Fig. 11 and the least successful ones in Fig. 12, respectively. As shown in Fig. 11, our multi-task CNN makes accurate predictions for most images in the test set of EVA dataset. However, it has more difficulty in predicting scores for the low aesthetic images compared to the AADB dataset. On the other hand, in the least successful predictions shown in Fig. 12, our model tends to predict high scores for the low aesthetic images, while giving lower scores to the high aesthetic ones. This behavior of the model is consistent with the results obtained from the AADB dataset.
the results in Table 10. The table displays the Spearman's rank correlations between the ground-truth scores and the corresponding predictions made by our multi-task CNN for each attribute. Moreover, we investigate the effect of fine-tuning and include those results in Table 10. This time, for all the attributes, the correlations increase after fine-tuning. Similar to the evaluation in the AADB dataset, we illustrate the activation maps generated using Grad-CAM [33] in Fig. 13 for two images from the test set of AADB dataset, one with a low aesthetic score and the other with a high aesthetic score.

Lastly, we report the Spearman's rank correlations on the EVA dataset in Fig. 14. This figure shows the correlations for the ground-truth scores of all dataset on the left side, whereas on the right side, we present the correlations for the predictions on the test data made by our multi-task CNN. Our model's highest correlations among all attributes are for the composition and depth (\(\rho\)=0.97) and semantics (\(\rho\)=0.95)

Fig. 11: Comparison of ground-truth overall aesthetic scores and corresponding predictions by our multi-task CNN on the test data of the EVA dataset. This figure shows the most successful predictions ranging from low aesthetic images to high aesthetic images.

Fig. 12: Comparison of ground-truth overall aesthetic scores and corresponding predictions by our multi-task CNN on the test data of the EVA dataset. This figure shows the least successful predictions ranging from low aesthetic images to high aesthetic images.
attributes. This finding is consistent with the EVA dataset, where composition and depth has the highest correlation with overall aesthetic scores (\(\rho\)=0.89) and semantics follows in second place (\(\rho\)=0.87). The lowest correlation belongs to the quality attribute, which is consistent with human data. Based on our evaluation, we can conclude that the predictions made by our multi-task CNN closely align with human interpretation in the EVA dataset as well.

### _Cross-dataset evaluation_

In the final part of our analysis, we investigate the generalization capability of our multi-task CNN by conducting a cross-dataset evaluation. Firstly, we examine the performance of our model trained on the AADB dataset when tested on the test set of the EVA dataset. Subsequently, we reverse the process and evaluate the performance of our model trained on the EVA dataset when tested on the test set of the AADB dataset. The results of these cross-dataset evaluations are summarized in Table XI. These results show the Spearman's rank correlations between the ground-truth overall aesthetic scores and the predictions made by our multi-task CNN.

Fig. 14: Spearman’s rank correlations between the overall aesthetic scores and the attribute scores on the EVA dataset. _(a)_: The ground-truth scores on all dataset, _(b)_: The predictions by our multi-task CNN on the test data.

\begin{table}
\begin{tabular}{l c c}
**Attributes** & **Training** & **Fine-tuning** \\ \hline Light and color & 0.610 & 0.709 \\ Composition and depth & 0.571 & 0.655 \\ Quality & 0.463 & 0.548 \\ Semantics & 0.586 & 0.659 \\ \end{tabular}
\end{table} TABLE XI: Spearman’s rank correlations between the ground-truth scores for each attribute and the predictions by our multi-task neural network on the test set of the EVA dataset. The table presents the results obtained after training and after fine-tuning.

Fig. 13: The activation maps for two input images from the test set of EVA dataset. These maps highlight the regions of the input image that contributed the most to the neural network’s prediction. The heatmap is overlaid on top of the input image to provide a visualization of which areas of the image are most relevant for the task.

\begin{table}
\begin{tabular}{|l|c|c|} \hline \multirow{2}{*}{**Train dataset**} & \multicolumn{2}{c|}{**Test dataset**} \\ \cline{2-3}  & **AADB**[17] & **EVA**[13] \\ \hline AADB & 0.707 & 0.321 \\ \hline EVA & 0.441 & 0.695 \\ \hline \end{tabular}
\end{table} TABLE XI: Spearman’s rank correlations between the ground-truth overall aesthetic scores and the predictions by our multi-task CNN for the cross dataset evaluation.