Optimising the Input Image to Improve Visual Relationship Detection
Noel Mizzi Adrian Muscat

	
	
	
	
	
	Optimising the Input Image to Improve Visual Relationship Detection
    
		Noel Mizzi

		Department of Computer Science 

		University of Malta, Malta 

		Adrian Muscat

		Department of Computer Science 

		University of Malta, Malta
	
    
=======================================================================================================================================================================


		Visual Relationship Detection is defined as, given an image composed of a subject and an object, the correct relation is predicted. To improve the visual part of this difficult problem, ten preprocessing methods were tested to determine whether the widely-used Union method yields the optimal results. Therefore, focusing solely on predicate prediction, no object detection and linguistic knowledge were used to prevent them from affecting the comparison results. Once fine-tuned, the Visual Geometry Group models were evaluated using Recall@1, per-predicate recall, activation maximisations, class activation maps, and error analysis. From this research it was found that using preprocessing methods such as the Union-Without-Background-and-with-Binary-mask (Union-WB-and-B) method yields significantly better results than the widely-used Union method since, as designed, it enables the Convolutional Neural Network to also identify the subject and object in the convolutional layers instead of solely in the fully-connected layers.
	
		
	

§ INTRODUCTION

	Understanding the image's content is one of the main goals of computer vision. This may be used[Information of how the fine-tuned models may be used to address these issues is provided in Appendix <ref>] for automatic image caption generation, image retrieval, visual question answering, and accessibility. To achieve this, machines must be capable of understanding the image contents – i.e. detecting objects (aka object detection) that are related to each other via a predicate (aka visual relationship detection). Taking advantage of cnn and large-scale datasets (such as ImageNet <cit.>), object detection has achieved great success but the detection of visual relationships is still in its infancy. This is mainly due to the long-tail distribution (where several predicates have very few instances), interpreting a 3D world from a 2D image, and intra-class diversity – thus, making it a very difficult task <cit.>.
	

	
		

	<cit.> addressed the vrd problem as a slc task where each triplet – composed of the subject, linked to an object via the predicate[In the Stanford vrd dataset, a predicate may be a verb (eg wear), spatial (eg next to), preposition (eg with), comparative (eg taller than), action (eg kick),	or preposition phrase (eg drive on). <cit.>] – as a unique class. This is referred to as a visual phrase (eg bicycle next to car). Later, <cit.> pointed out that it is not feasible due to the lack of instances for each visual phrase and the class exponential growth. Therefore, they predicted each one – the subject, predicate, and object – separately using three fine-tuned vgg models. Furthermore, to filter out irrelevant triplets, language priors obtained from the Stanford vrd training dataset were also used. Other research used a novel feature extraction layer <cit.>; a spatial module to extract spatial configurations and a drnet for statistical dependencies <cit.>; and knowledge distillation using both internal and external linguistic knowledge <cit.>.
	

		
	As <cit.> pointed out, mainly the Union method is used or the spatial configuration is used without any visual information. Additionally, <cit.> found that in vqa – where vision and language are intertwined – the answers were not grounded on the image content but based on superficial performance (eg predicting the answer based on the question only). This can also be applied for the vrd scenario, where both vision and language are being used, and models based on statistical dependencies are being employed <cit.>.
	To ensure that the results are based on the image content, this paper analyses the visual part of the vrd problem by focusing solely on predicate prediction[The ground truth bounding boxes and the respective object categories are provided; the model predicts only the predicate.] – ie the model is only given the image. 
	<cit.> found that the convolutional layers of a cnn act as object detectors but previous research mainly fed the Union into the cnn. Therefore, not taking full advantage of the convolutional layers as these layers are not capable of determining the location of the respective triplet subject and object. To address these shortcomings, ten preprocessing methods (depicted in Figure  <ref>) – Union-WB, Union-WB-SC, Union-WB-B, Union-WB-B-SC, Union-WB-and-B, Segment, Segment-B, Blur-Sigma3, Blur-Sigma5, Blur-Sigma7 – were used to determine if better results could be achieved when compared to the Union method.
	

	
	The main contributions[To the best of our knowledge they have never been used for the detection of visual relationships.] of this paper are: 
	(1) a novel analysis of the visual module using preprocessing methods;
	(2) the novel usage of segmentations for the detection of visual relationships;
	(3) the novel usage of activation maximisation to study the vrd problem; and
	(4) the per-predicate and error analysis.
	

	
	

	

§ RELATED WORK

	

 §.§ Object Detection

	Object detection – the kernel of vrd – is the process of detecting objects pertaining to a specific class  – slc – and their location <cit.>. Recently, cnn have been used for object detection to map the image pixels to one or more classes without the need for hand-designed features <cit.>. <cit.> developed a rcnn which replaces <cit.> sliding window approach. fastrcnn <cit.> used spp <cit.> which eliminates the need for feature caching and enables all parameters to be trainable <cit.>. Later, fasterrcnn <cit.> used a rpn <cit.>, while maskrcnn <cit.> also generated the object segmentations <cit.>.
	

	
	
	

 §.§ Visual Relationship Detection

	The concept of vrd is that given an image, the linguistically-correct predicate is predicted. Contrary to object detection, the images within each class are highly uncorrelated and therefore it is a challenge to learn from very few samples <cit.>. To address this problem, two main types of architectures have been developed – joint and separate. The former considers each triplet as a unique class (O(N^2R) where N = objects, and R = predicates) <cit.>, while the latter predicts the subject, predicate and objects separately (O(N+R)) <cit.>.
	

	
	<cit.> predicted each triplet as a unique class where the within-class variance is limited and therefore enables learning from few samples. As <cit.> noted, the issue is that: (1) it is difficult to collect enough samples for each visual phrase (eg "person riding horse"), and (2) the exponential growth of the number of visual phrases when new objects are added to the dataset – therefore, it is not capable of generalising. To overcome these issues, <cit.> predicted the subject, predicate, and object separately using three fine-tuned vgg models initialised using ImageNet <cit.> weights. Furthermore, they also used linguistic knowledge from the Stanford vrd training dataset to remove improbable predictions and enhance the zero-shot[Using a separate dataset, evaluate the model to predict unseen triplets] scenario. They achieved this by mapping similar relations (eg person–ride–elephant and person–ride–horse) close to each in an embedding space – thus, achieving an 11% improvement <cit.>. 
	

	
	<cit.> used a novel feature extraction layer (composed of class probabilities, bounding boxes location and scales) to simultaneously predict the objects and predicates using end-to-end training. Additionally, they also allowed knowledge transfer by replacing fasterrcnn's roi pooling layer with bilinear interpolation <cit.>. This resulted in better generalisation due to the class probabilities and improved object detection as a result of knowledge transfer <cit.>. On the other hand, <cit.> developed a model based on the spatial configuration and statistical dependencies. The former was composed of two binary masks, one for the subject and another for the object, resized to 32x32 pixels, convolved through three convolutional layers and a fully-connected one to generate a 64-D vector representing the spatial configuration (improving the Recall@50 results by approximately 8.5% – much higher than yu2017visual spatial features improvement) <cit.>. On the other hand, a novel drnet was used to exploit statistical dependencies. Analogous to lu2016visual method, better results were achieved when both visual and statistical modules were used – especially when the drnet was used (although no zero-shot results were provided) <cit.>. <cit.> used a probabilistic model where instead of increasing the dataset size (due to the long-tail distribution), a larger linguistic knowledge base was used. To achieve this internal knowledge from the Stanford vrd and the vg datasets and external knowledge from the Wikipedia dump[Used to overcome having triplets with a probability of zero.] were employed. From their research, <cit.> found that better results may be achieved by using a larger linguistic knowledge base. <cit.> noted that research mainly used the Union method which might contain noise or the object's positional information without any visual information. Therefore, they used <cit.> spatial module, a feature-level (learning discriminative features) and label-level (capture statistical dependencies) triplet prediction. <cit.> found that better discriminative features can be attained if the subject and object feature vectors were used. Additionally, analogous to previous methods the statistical dependencies yield better results <cit.>.
	

	
	
	

 §.§ Understanding cnn

	Due to the large number of interacting/non-linear parts, cnn are generally referred to as a black box since they automatically learn useful features from the input to generate the expected output <cit.>. To close this gap Deep Visualisation methods such as activation maximisation and cam have been used. The former numerically – through back-propagation – generates an image which depicts the neuron activations for a specific class <cit.>. On the other hand, – since the convolutional layers of a cnn act as object detectors – cam are used to visualise the activations (using a single forward pass) at a certain layer when a specific input (an image) is fed into the cnn <cit.>.
	

	
	
	

§ METHODOLOGY

	<cit.> found that in vqa, where both vision and language are used, the predictions were not grounded on the image content; while <cit.> found that the convolutional layers of a cnn act as object detectors. Therefore, since research carried out in the field of vrd mainly incorporates linguistic knowledge (statistical dependencies and probabilistic models) <cit.> or used spatial features as a vector <cit.> and mainly feed the Union to generate the predictions, this paper focuses solely on the visual module predicate prediction capability, with the aim of exploiting implicit language derived knowledge. As depicted in Fig. <ref>, the experimental setup is composed of a fine-tuned vgg model initialised using ImageNet weights. As input it takes a preprocessed image (containing only two annotated objects) and predicts the respective predicate.
	

	
	First, the Stanford vrd dataset images were preprocessed (Fig. <ref> depicts sample images) for each method – the Union, Union-WB, Union-WB-SC, Union-WB-B, Union-WB-B-SC, Union-WB-and-B, Segment, Segment-B, Blur-Sigma3, Blur-Sigma5, and Blur-Sigma7. Since the Stanford vrd dataset does not contain any segmentations, the intersection of the Stanford vrd and the mscoco datasets was used to extract the segmentations using maskrcnn <cit.>. To ensure that the correct object was segmented, only the respective bounding box content was fed into maskrcnn and the predicted class was compared with the ground truth one. Additionally, to ensure a fair setting, the bounding boxes provided by maskrcnn were used instead of those provided by the Stanford vrd dataset.
	

	
	The following is an explanation of each preprocessing method used for predicate predication:
	
		
  * Union — the smallest bounding box that encloses both the subject and the object bounding boxes;
		
  * Union-WB — the background (ie the pixels not within the subject or object bounding boxes) of the respective Union image is removed;
		
  * Union-WB-SC — the single channel version (ie grayscale image) of the Union-WB method;
		
  * Union-WB-B[Inspired by dai2017detecting spatial mask] — the object binary mask was set on the first channel, while the subject binary mask was set on the second channel;
		
  * Union-WB-B-SC — the Union-WB-B method on a single channel (ie grayscale image);
		
  * Union-WB-and-B — the Union-WB method but the single channel bounding boxes were used instead of the binary masks;
		
  * Segment — the Union was extracted and the pixels not within the subject or object segmentations were zeroed-out;
		
  * Segment-B — analogous to the Union-WB-B method but the respective segmentations were used instead of the bounding boxes;
		
  * Blurring —	<cit.> found that the lower layers of a cnn act as edge detectors. Therefore, instead of removing the background (eg Union-WB), three blurring variations were used to reduce attention in this area[Blurring the background pixels]. To achieve this, a Gaussian low-pass filter using standard deviations[For both the x and y directions and chosen through manual testing and as employed by <cit.>] 3 (Blur-Sigma3), 5 (Blur-Sigma5), and 7 (Blur-Sigma7)  was used to blur the background using the kernels 19×19, 31×31, and 43×43 respectively.
	
	
	
	<cit.> found that using a large dataset (such as ImageNet) to pre-train a cnn and then fine-tune it using the desired dataset was very effective. Consequently, a pre-trained vgg16/vgg19 model, initialised using ImageNet weights <cit.> was fine-tuned using mini-batch sgd for predicate prediction. This was achieved by first normalising the image pixels between zero and one, without any image augmentations[eg when flipped horizontally, on the left of becomes on the right of and vice versa]. Subsequently, the images were resized to 224 × 224 pixels, the respective vgg model was then initialised and all layers excluding the fully-connected layers were set to non-trainable – the weights were not fine-tuned. Using categorical cross-entropy, a batch size of 10, Nesterov momentum set to 0.9, and a learning rate of 0.001 the model was fine-tuned for five epochs. Once complete, the last convolutional block layers and the fully-connected layers were set to trainable and the model was fine-tuned for five epochs. Finally, using a learning rate of 0.00001, the model was fine-tuned for five more epochs. The fine-tuned model was then evaluated as explained in the following section.
	

	
	
	

§ EXPERIMENTS

	As already stated, the fine-tuned models were evaluated for predicate prediction – ie the model was provided the ground truth subject and object bounding boxes and categories (eg horse) – on the Stanford vrd dataset. Therefore, the model only had to predict the correct predicate – no object detection was used <cit.>.
		
		
	

	
	
	
	
	
	
	
	
	
	

	
		
		
	
	

	
	
	
	
	
	
	
	
	
	
	
	
	
	

 §.§ Evaluation Metrics

	Once trained, the models were evaluated on two datasets – the testing and zero-shot datasets – where, contrary to the testing dataset, the zero-shot dataset contains triplets that were never seen by the model. To evaluate these models Recall@1 (which compares the ground truth predicate with the predicted predicate having the maximum probability <cit.>), per-predicate recall, activation maximisation, cam, and error analysis were used. Since the dataset annotations are incomplete, precision and map were not used <cit.>. Moreover, each model was fine-tuned multiple times and then Tukey's HSD post-hoc test (using a p-value of 0.05) – which compares the groups' mean value while reducing Type I errors <cit.> – was used to check for statistical significance <cit.>.
	
	

 §.§ Dataset

	Two versions of the Stanford vrd dataset which is "the most widely benchmarked dataset for the relationship detection in real-world images" <cit.> were used. The first one is the full dataset (composed of 5,000 images, 100 object categories, and 70 predicate categories); while the second one, referred to as the Stanford vrd dataset subset, is a custom dataset – composed of the intersection of the Stanford vrd and the mscoco datasets[Resulting in 1,943 images, 38 object categories and 64 predicates categories.] – used to compensate for the lack of segmentations in the Stanford vrd dataset.
	
	

 §.§ Implementation Details

	Python and Keras api using a Tensorflow backend were used to implement and evaluate the cnn models, while the Python Keras-vis package was used to generate the activation maximisation images (using 1,024 backpropagation iterations) and the cam visualisations. 
	
	

 §.§ Results and Discussion

	<cit.> quote the predicate detection results obtained when using the vision module only (7.11%). However, this result is not applicable to our case since we train our model end-to-end. On the other hand, <cit.> made use of a softmax output trained end-to-end and obtained a Recall@1 of 34.82% when the input was the Union. The mean Recall@1 for the Union method[Presented in Table <ref>] (35.52%, vgg16) is very close to the result in <cit.> – (34.82%, k=1) – where a similar architecture was used. 
	

	
	

	The Recall@1 results for all methods (excluding the segmentation based methods), trained and tested on the full Stanford vrd dataset are given in  Tables <ref> and <ref> for both the testing and zero-shot datasets.  The results including the segmentations methods for the Stanford vrd dataset subset are given in Tables <ref> and <ref>, while Table <ref> gives Recall@k, (k=1, 2, ...,10)  results  for the Union-WB-and-B and the Union methods.	The following are some conclusions that can be drawn from these tables: (a) As expected, the full dataset Recall@1 results are higher than those of the subset as a result of more training data. (b) Results obtained using the vgg16 and vgg19 models are not significantly different. This suggests that less complex architectures may yield similar accuracies; (c) As the Union bounding box background is blurred, and eventually blacked out, the accuracies increase. This follows the intuition that extra objects can distract the attention on the two objects and confirms similar observations in <cit.>; (d) There is a significant difference in between the binary (spatial) methods and methods that depict the objects.  For example, there is an increase of 8.66% and 4.58% in recall from the Union-WB to the Union-WB-B for the (vgg16) testing and zero-shot datasets respectively. (e) Additionally, when both were merged – ie the Union-WB-and-B method, – even better results were achieved. This is mainly due to the use of different channels for the subject and object, while maintaining the respective subject/object visual features. Therefore, detecting object features (ie language features implicit in the image) help in modelling the dependencies. (f) The same pattern can be noted for the zero-shot dataset, indicating that object pairs not found in the training set (eg person and elephant) are probably closer to pairs found in the training dataset (eg person and horse). (g) Referring to Table <ref> the Segment-B and Union-WB-B are not statistically  different, while Table <ref> shows that they are statistically different. This could be due to maskrcnn splitting the respective objects into separate segments when occluded by other objects and using rectangular areas instead of segmentations.
	

			
	Table <ref> tabulates the per-predicate Recall@1 accuracies for all methods tested on the full Stanford vrd dataset. It can be noted that the models were mainly detecting spatial prepositions followed verbs, while there are no actions or comparatives. The highest recall rates are  shared between the Union-WB-B and Union-WB-and-B methods, with the exceptions of in and near, where the Union-WB and Blur-Sigma7 methods respectively achieved the highest recall. Additionally, the Union-WB-and-B method predicted the most diverse set at top rank. One would expect to have higher accuracies from the Union-WB-and-B method for the more functionally biased predicates, but this is not always the case; has and hold are two examples of functionally biased predicates that the Union-WB-and-B achieved better results as it includes both spatial and implicit language knowledge. However, below, which is known to be a geometrically biased predicate does not do well. This may be due to being predicated 77.98% of the time as under (a functionally biased predicate). Overall, the predicates on the left of, on the right of and near did not attain a high recall result but were predicted as next to – a topological predicate hence easier to comprehend. On the other hand, above which is a projective predicate (and requires more attention) achieved better results. The Union-WB-and-B does significantly better with behind and in front of, which could be that the implicit language knowledge is being used as a proxy to depth in a 3D world. 
	

	
	Error analysis was carried out on the outputs to better understand the sources of error.  Table <ref> summarizes the analysis results of a sample of one-hundred random predictions.  The main source of error (44%) is  alternative however plausible predicates, followed by different pov (16%). The former may be addressed by using the sigmoid loss function and binary cross-entropy enabling mlc, while the latter may be addressed by using a consistent pov – ie from the viewer or the respective object pov.
	

	
	The activation maximisation results for above, below, beneath and under are given in Fig. <ref>.  The subject and object channel images depict the respective activations for the respective channel – ie the green (subject) and blue (object) channels. Through the use of different channels, we note that the model identified the relationship direction correctly. Taking a closer look, below has a wider range of acceptance, whilst in beneath the subject is closer to the centre and under is somewhere in between.
	The direction can also be observed for the predicates on the left of and on the right of activation maximisation results presented in Fig. <ref>. Additionally, referring to Fig. <ref>, it can be noted that the cnn learnt similar features for predicates – drive, on, on the top of, ride, and sit on – where the subject is located on the object, or in general in the upper half of the union box. This shows that the cnn was not capable of identifying the difference between these predicates.
	

	
	When reviewing the cam results, they are inconclusive since there were no key differences between the methods and models apart from those presented in Fig. <ref> and <ref>. The former shows that for the Union-WB method, the vgg16 model paid attention to the distance between the buses, while the vgg19 did not. Conversely, the latter shows that for the Blur-Sigma3 method the cnn paid attention to the driver when the vgg19 model was used but not for the vgg16 model.
	

	

	

	
	

§ CONCLUSION

	The aim of this research was to analyse the visual part of the vrd problem by using various image preprocessing methods to enable detection in the convolutional layers of the cnn. To ensure that the results were based on the image content, no object detection and linguistic knowledge[In vqa it was found that the answers were not grounded based on the image content.] were used. To achieve this, the pre-trained vgg16 and vgg19 models were fine-tuned for predicate prediction. Overall, this resulted in the Union-WB-and-B method achieving the best results due to the use of different channels and maintaining the respective bounding box visual features. Furthermore, activation maximisation, cam, and error-analysis were used to determine what the cnn has learnt.
	
	

§ FUTURE WORK

	Analogous to how maskrcnn generated segmentations, we will train an object detector such as fasterrcnn or maskrcnn to additionally provide the respective object pose/posture. This is useful for both object detection and visual relationship detection. Furthermore, addressing the 44% alternative predicates[Error analysis results presented in Table <ref>], we will train the models using the sigmoid loss function and binary cross-entropy – mlc – instead of using the softmax loss function – slc – thus, predicting multiple valid predicates for each pair of objects. Moreover, we will also analyse how smaller cnn affect predicate predication.
	

	
	
	

§ USE CASES

	
	This section explains how the method proposed in this paper may be used for visual relationship detection, and other areas.
	

	
	Fig. <ref> depicts[The blue boxes depict the research carried out in this paper.] how an object detector such as fasterrcnn and the fine-tuned model (eg Union-WB-and-B) may be used to predict the triplets. This is achieved by first feeding the respective image into the object detector. The detected objects bounding boxes are then used to generate pairs of objects which are the used to generate the Union-WB-and-B pre-processed images. These are then fed into the respective fine-tuned model to predict the respective predicate. In the final stage, the triplets are generated using the objects bounding boxes, class names (provided by the object detector) and the respective predicate (provided by the fine-tuned model). Therefore, resulting in a triplet for each pair of detected objects.
	

	
	 The predicted triplets provide the basis for other computer vision problems such as automatic caption generation, image retrieval, vqa, and accessibility[To assist both blind and visual impaired persons to understand the world around them and carry out the desired task]. The following is a brief explanation of how the predicted triplets (as depicted in Fig. <ref>) may be used to achieve these tasks:
	 
	 	
  * Automatic Caption Generation — use the triplets to retrieve sentences from the internet that are relevant to the respective image (ie explain the image content);
	 	
  * Image Retrieval — index the triplets and use them when searching for an image based on the image content;
	 	
  * vqa — use an lstm neural network to understand the question and then use the image triplets to answer the question; and
	 	
  * Accessibility — for each video stream frame, generate the respective triplets, generate sentences and then use text-to-speech to inform the user accordingly.
	
	
	
	
