Evaluation and Comparison of Visual Language Models for Transportation Engineering Problems
Sanjita Prajapati Tanu Singh Chinmay Hegde Pranamesh Chakraborty

Types of Size-Dependent Melting in Fe Nanoclusters: 
a Molecular Dynamics Study
    David B. Graves
    September 9, 2024
===============================================================================





§ ABSTRACT

Recent developments in vision language models (VLM) have shown great potential for diverse applications related to image understanding. In this study, we have explored state-of-the-art VLM models for vision-based transportation engineering tasks such as image classification and object detection. The image classification task involves congestion detection and crack identification, whereas, for object detection, helmet violations were identified. We have applied open-source models such as CLIP, BLIP, OWL-ViT, Llava-Next, and closed-source GPT-4o to evaluate the performance of these state-of-the-art VLM models to harness the capabilities of language understanding for vision-based transportation tasks. These tasks were performed by applying zero-shot prompting to the VLM models, as zero-shot prompting involves performing tasks without any training on those tasks. It eliminates the need for annotated datasets or fine-tuning for specific tasks. Though these models gave comparative results with benchmark Convolutional Neural Networks (CNN) models in the image classification tasks, for object localization tasks, it still needs improvement. Therefore, this study provides a comprehensive evaluation of the state-of-the-art VLM models highlighting the advantages and limitations of the models, which can be taken as the baseline for future improvement and wide-scale implementation.


Keywords: visual-language models, crack detection, congestion detection, helmet violation detection, image understanding




§ INTRODUCTION


In recent years, there have been significant advancements in computer vision and language modeling for solving different tasks using deep learning. Inspired by the advancements in natural language processing using transformer based models, a new concept in computer vision called Vision Transformers (ViT) <cit.>, was introduced in 2021 for image understanding. In the field of language modeling, many Large Language Models (LLMs) such as Llama and ChatGPT have shown excellent capability to solve a large variety of tasks. These models, which were initially designed for text inputs, now also support visual inputs, connecting vision to language and enabling zero-shot or few-shot learning. This development has the potential to unlock several applications that will be key to the current AI-based technological revolution. 

On the other hand, Convolutional Neural Networks (CNNs) have been extensively utilized for the past decade for vision-based processing, demonstrating efficient real-time performance. Their simpler architecture renders them favorable for real-time deployment. However, CNNs have limitations, such as the requirement for extensive datasets and the need for fine-tuning for almost every use case to achieve better results. This process is labor-intensive requiring manual annotations, highlighting the need for pre-trained models. This has created a necessity for foundational models that can be applied to various tasks without the need for extensive fine-tuning. 

In this study, our primary focus is to understand the capabilities and limitations of the state-of-the-art Vision Language Models (VLMs) in the field of vision-based transportation engineering tasks. The study involved the careful selection of three specific transportation engineering tasks, each of which presents distinct challenges and complexities. The first task focuses on detecting congestion on highways from surveillance cameras, which is a critical issue in transportation management. The second task involves the identification of cracks in pavement surfaces, an essential aspect of infrastructure maintenance. While both these tasks fall under the domain of image classification, we go further to understand capabilities of VLMs in object detection tasks.  The third task addresses the vital issue of detecting helmet violation, specifically determining whether motorbike riders were wearing helmets or not, which is crucial for enhancing safety on roadways. 

These tasks have been chosen due to the fact that they require fine-tuning of pre-trained CNN models. Notably, the classes relevant to these tasks are not included in the COCO dataset <cit.>, necessitating specialized attention and refinement. The performance of the chosen tasks has been rigorously evaluated, providing valuable insights into the effectiveness of zero-shot VLMs in addressing transportation engineering tasks.  In this research, both open-source and closed-source foundation models of VLM have been considered. Within the open-source category, the study delved into the performance of models such as CLIP <cit.>, BLIP <cit.>, OWL-ViT <cit.>, and Llava-Next <cit.>. Furthermore, the study also included an analysis of the closed-source GPT-4o <cit.> model, presenting a thorough evaluation of a range of foundational VLMs for transportation engineering tasks. 

The following section discusses the past studies on the application of VLM models in transportation. This is followed by the methodology used in both  image classification tasks, congestion and crack detection, and then addresses the methodology used in object detection. After that, the paper discusses the datasets used in this study and the results of our research. Finally, the conclusions and the future scope are highlighted. 



§ LITERATURE REVIEW




Large Language Models (LLMs) have revolutionized natural language processing (NLP), allowing machines to understand and generate human-like language with unprecedented success. The performance of LLMs in textual understanding and their versatility in different domains of language tasks has led to the exploration of multi-modal LLMs <cit.>. Multimodal LLMs can process and generate information across various data types such as text, images, audio, and video. Vision Language Models (VLMs) blend computer vision and NLP capabilities. They are designed such that they can process and generate human-like text based on visual inputs, or the other way around <cit.>. By bridging the gap between visual and textual data understanding, VLMs have various applications such as image captioning, visual question answering, textual descriptions, and even image generation.

Recently, pre-trained VLM with zero-shot prediction has attracted significant attention, where VLM is pre-trained on a large-scale image-text dataset. The pre-trained VLM with a rich textual and image understanding can then be directly applied to any visual task without fine-tuning. Zero-shot prediction implies that the model can interpret and generate descriptions or answer questions, based on textual instructions it has never seen before.

 

 §.§ VLM in transportation


 The VLMs and LLMs have lately demonstrated strong zero-shot capabilities and human-like reasoning capabilties. Recently, few studies have integrated the VLMs and LLMs for traffic-related tasks such as understanding traffic scenes, autonomous driving, and anomaly detection for enhancing interpretability, safety, and generalization capabilities. 

Some studies have attempted to leverage VLMs in autonomous driving for various purposes such as navigation, forecasting, interpreting vehicle action, and planning. DriveVLM employs VLM to interpret complex traffic scenarios for understanding and analyzing the scene to plan the actions for autonomous driving <cit.>. Similarly, DriveGPT4, a multimodal LLM uses input multi-frame videos and textual queries to generate responses and predicts low-level control signals for vehicle action<cit.>. GPT in DriveGPT4 stands for Generative pre-trained transformer and The digit “4” represents multimodality. In Vision Language Planning (VLP), researchers also integrated language models with vision-based systems to enhance autonomous driving by improving their contextual understanding and generalization capabilities. It has two components, an agent-centric learning paradigm and a self-driving-car-centric learning paradigm that improves the local details in the BEV feature map and enhances the planning process respectively, by leveraging the knowledge encoded in the pre-trained language model <cit.>. While these works focused on improving autonomous driving systems, in the domain of Visual Language Navigation, VLN system was developed to navigate action for intelligent vehicles leveraging LLM and VLM, by extracting landmark names from user’s language instructions, matching landmark names with environmental objects, and finally reasoning navigation actions for the intelligent vehicle <cit.>. On the other hand, CityLLaVA, was developed to understand traffic scenarios in the city by fine-tuning VLMs by utilizing bounding-box guided view selection and prompt engineering modules <cit.>.

Apart from autonomous driving application and scene understanding, DriveCLIP <cit.> framework explores the application of vision-language models, particularly the CLIP model, to identify distracted driving activities from naturalistic driving videos and images. This system offers zero-shot transfer, fine-tuning, and video-based models for driver’s state prediction. All these studies have only performed research on the homogeneous driving environment, consisting of major four-wheelers. The heterogeneous driving environments have not been explored by any researcher being complex containing different types of vehicles with varying speeds. 

Video Anomaly Detection is another field where few works have applied VLMs and LLMs for improved performances.VAD-LLaMA <cit.> is such a framework where traffic anomaly is detected and localized in a long-range surveillance video. They have incorporated video-based large language models (VLLM) to make threshold-free detection and explain the reasons for the anomalies detected. They have also introduced a novel Long-Term Context module to alleviate the incapability of long-range context modeling in existing VLLMs. 

Except for anomaly detection work, the field explored by VLMs in transportation has used datasets primarily containing in-vehicle camera images or videos (except anomaly detection tasks). On the other hand, the surveillance camera based images/videos can also be analysed through VLMs for improved image/video understanding. Moreover, all these applications are focused on high-level tasks such as vehicle navigation, anomaly detection, etc. In addition to exploring VLMs in such high-level image and video understanding, there is also a need to understand and analyze the potential of VLMs in low-level image understanding tasks such as image classification and object detection. This will involve utilizing the vision and language modalities to significantly improve zero-shot or few-shot classification and detection tasks in the transportation domain.

We recognized these limitations in the application of VLMs in transportation engineering-related problems and applied different state-of-the-art vision-language models in basic image understanding tasks such as classification and object detection to understand the capabilities and limitations of the models.



§ METHODOLOGY

This study aims to leverage the capabilities of VLM for vision-based transportation engineering tasks which include; a) image classification and b) object detection. In this study, we tested the state-of-the-art VLMs for two image classification tasks a) congestion detection and b) crack detection. On the other hand, in the domain of object detection, we have evaluated the performance of VLM for detecting helmet violation cases i.e. motorbike riders wearing helmet or not.



These image classification and object detection tasks are selected to identify the potential of VLM in tasks that go beyond detecting regular traffic entities (such as cars, pedestrians, etc) and therefore can harness the capabilities of language understanding for vision-based transportation tasks. In this section, we discuss the details of the state-of-the-art VLM models that have been used for the selected image classification and object detection tasks






 §.§ Image classification task

In this study, the first image classification task involves congestion detection i.e., detecting the congestion in any of the highway lanes and classifying the image as congested or not. The second task of crack detection implies that, given the pavement surface images, the model needs to identify whether any cracks are present or not.

For these image classification tasks, the three models used are OpenAI Contrastive Language-Image Pre-Training (CLIP) <cit.>, Bootstrapping Language-Image Pre-training (BLIP) <cit.>, and Large Language and Vision Assistant - Next Generation (LLaVA-NeXT)  <cit.> and GPT-4o. As these models have strong zero-shot performance, this eliminates the need for annotated training data in the image classification task. Therefore these models were selected for classifying vision-based transportation-related tasks using zero-shot prompting. As explained earlier, zero-shot prompting is a technique where a model is given a task or instruction without any prior example or training on that specific task. 




  §.§.§ CLIP model


CLIP model <cit.> is trained on 400 million image-text pairs available on the internet, allowing it to learn a range of visual features along with their corresponding text description. During the training of CLIP, it employs contrastive learning where it learns to predict which text and image are paired together. An image and text encoder are trained to maximize the cosine similarity of the correct image and text pairs while minimizing the cosine similarity of the incorrect pairings.
The CLIP model has zero-shot learning capability, allowing it to classify images based on natural language prompts without requiring additional task-specific training. 

In this study, image classification has been performed for two vision-based transportation tasks, congestion and crack classification. For both tasks, we used the names of the binary classes as the probable text pairings and employed CLIP to predict the most likely (image, text) pair.

To classify congestion, we used five different class names for the CLIP model: 

A1: ["Congested", "Non-congested"], 

A2: ["Congested lanes", "Non-congested lanes"], 

A3: ["Lanes with congestion", "Lanes without congestion"], 

A4: ["Queued traffic", "Free-flow traffic"], 

A5: ["Congested lanes", "Free-lanes"]. 

Similarly, for classifying cracks in the pavement surface, the different class names used were: 

B1: [“Cracked”, “Non-Cracked”], 

B2: [“Cracks present”, “Cracks absent”], 

B3: [“Cracked surface”, “Non-Cracked surface”], 

B4: [“Cracked pavement”, “Crack-free pavement”], 

B5: [“Crack”, “No crack”].



  §.§.§ BLIP model

BLIP addresses both vision-language understanding and generation tasks by using a multimodal mixture of encoder-decoder architecture and a novel data bootstrapping technique called CapFilt. BLIP generates synthetic captions and filters out noisy ones to enhance training data quality, leading to state-of-the-art performance across various vision-language tasks, including image-text retrieval, image captioning, and visual question answering <cit.>. Therefore BLIP is more versatile for both generation-based and understanding tasks compared to CLIP which focuses on alignment and representation learning.

Similar to the CLIP model, probable class names were used by BLIP to classify images for both tasks. The different class names utilized in the BLIP model are the same as those mentioned for the CLIP model.




  §.§.§ LLaVA model



The Large Language and Visual Assistant (LLaVA)<cit.> is an open-source multimodal model that is designed to interpret and generate results based on both visual and textual input (10). It leverages the LLaMa <cit.> model and incorporates the pre-trained CLIP visual encoder for processing visual content. The encoder extracts visual features from input images and links them to language embedding through a trainable projection matrix, effectively translating visual features into language embedding tokens and bridging the gap between text and images. Although trained on smaller datasets than closed-source multimodal GPT models, Llava purports to demonstrate behavior analogous to the proprietary models.  

The LLaVA-NeXT (an updated version of LLaVa) <cit.> focuses on enhancing multimodal instruction following capabilities on data generated to follow detailed visual and textual instructions, for interactive and complex visual tasks. In contrast, CLIP learns generalizable visual representations from large-scale natural language supervision aligning image and text embedding to enable zero-shot learning across diverse vision. 


The LLaVA-NeXT model was also employed in this study for both classification tasks with different task-specific instructions. We first selected the initial prompt to query the model to generate the description of each image. To get the desired output as a discrete class name, we further queried the model with the generated description and follow-up prompt to get output as class names.

The five different initial prompts that were adopted to generate the description of congested/non-congested dataset are as follows:

P1: Classify whether highway lanes are congested or not in the image.

P2: Classify whether highway lanes are congested or not in the image.

P3: Classify whether in the image highway lanes are congested or not.

P4: Classify whether the highway have congested lane or free-lane in the image.

P5: Check whether the highway lanes are congested or not in the image.


The follow-up prompt selected corresponding to each of these initial prompts were

F1: Write Yes for congested, No for non-congested. 

F2: Write Congested lanes if lanes are congested, Free-lanes if lanes are not congested. 

F3: Write Congested lanes if lanes are congested, Free-lanes if lanes are not congested.

F4: Write  Congested lanes if lanes are congested, Free-lanes if free-lane. 

F5: Write Congested lanes if lanes are congested, Free-lanes if lanes are not congested. 


Similarly, for classifying cracked/non-cracked images, the initial prompt adopted were

P1: Classify whether the pavements have cracks or not in the image? 

P2: Classify whether the cracks are present or not in the pavement surface image?

P3: Classify whether the pavement surface is cracked or not in the image?

P4: Classify whether in the image, the pavement surface have cracks or not? 

P5: Check whether the pavement surface has any cracks or not?


The corresponding follow-up prompt selected for query were

F1: Write Cracked if cracks present, Non-cracked if cracks not present. 

F2: Write Cracked if cracks present, Non-cracked if cracks not present. 

F3: Write Cracked if surface is cracked, Non-cracked if surface is not-cracked.

F4: Write Cracked if surface has cracks, Non-cracked if surface do not have cracks.

F5: Write Cracked if cracks present, Non-cracked if cracks not present.



  §.§.§ GPT-4o

GPT-4-o, with the "o" for "omni", is an advanced iteration of the Generative Pre-trained Transformer series by OpenAI. GPT-4o is built to handle multimodal data including text, images, and audio, allowing it to process and generate not only natural language but also interpret and respond to visual and auditory data. Its ability to understand and generate human-like text makes it a valuable tool in diverse fields.

In this study, we have prompted GPT-4o for both image classification tasks for comparison and evaluation purposes. The prompt used for congestion classification was "Can you tell me whether the closer lane are free lanes or not. Only return non-Congested if there are all free lanes otherwise return congested", whereas for crack classification "Can you tell me whether the pavements have cracks or not in the image. Only return yes if crack is present and no if crack is not present."




  §.§.§ Benchmark CNN model

The task of congestion classification has been compared using a DCNN model present in Chakraborty et al.<cit.>. It took 25 minutes to train the model on an NVIDIA Tesla K20m GPU with 4 GB RAM. On the other hand, for crack classification, the VLM models were compared with CNN EfficientNet B1 model architecture, it took 903 secs to run on the test dataset.




 §.§ Object detection tasks


Our study focuses on the object detection task of identifying helmet violations. The task aims to detect whether motorcyclists were wearing helmets, which is a mandatory rule of road safety in many countries. This class is not present in the COCO dataset or any other pre-trained CNN model, necessitating fine-tuning of the model for our specific use case. Our objective is to explore how well zero-shot Vision Language Models (VLM) perform in these scenarios, to reduce the need for intensive datasets and fine-tune them, thereby streamlining the resource-intensive processes of dataset creation. 

We aim to detect two classes: “Helmet” – a motorcyclist wearing helmet, and “No-Helmet” – a motorcyclist without wearing helmet. While identifying the positive sentiment class is relatively straightforward, but the challenge lies in identifying the negative sentiment class, "No-Helmet," especially for Vision Language models. For any language model, it is easy to understand the positive sentiment class, but VLMs have been found to face difficulties identifying a negative sentiment class.  

For zero-shot object detection, we have been used the Vision Transformers for Open-World Localization (OWL-ViT) model <cit.> with basic classes and performing required post-processing to improve results. Additionally, we are utilizing textual class prompts to eliminate the need for post-processing. However, OWL-ViT does not perform well on textual classes since it is not trained on Large Language Models. As a result, we are considering open-source Large Language Vision models such as Llava-Next <cit.>, as well as close-source VLMs like GPT-4o <cit.>. 






  §.§.§ OWL-ViT model

OWL-ViT <cit.> is a state-of-the-art open vocabulary object detection model, which was launched by the Google research team in 2022. This model is designed to understand the relationship between images and text. Operating as a zero-shot object detection model, it leverages CLIP <cit.> as its multimodal backbone in conjunction with a ViT-like (Vision Transformer-like) model. To use CLIP for object detection, OWL-ViT removes the final token pooling layer of the vision model and adds a lightweight classification and box head to each transformer output token pool. Open-vocabulary classification is achieved by substituting the fixed classification layer weights with the embeddings obtained from the text model. 

In this part of our study, we are focused on Helmet violation detection using OWL-ViT <cit.>. For that, we have explored its performance in basic classes, i.e., one-word classes given to the prompts. This method requires post processing for better accuracy. Furthermore, we have extended it to beyond one-word classes by checking its performance on textual classes.
 



a) Detection of basic classes and post processing 

Initially, a single word class name is input via prompt, i.e., Motorbike, Person and Helmet. After detecting these classes, post processing is necessary to obtain the desired output class, i.e., (1) “Helmet” (a person who is sitting on a motorbike wearing helmet) and (2) “NoHelmet” (a person who is sitting on a motorbike without wearing helmet). 

Three steps are involved in the post-processing module, as shown in Fig 1. First, a non-maximum suppression method is used to remove the duplicated bounding boxes. Second, the person bounding boxes that are aligned with the motorbike are selected based on the calculation of the Intersection over Union (IoU) of the motorbike and person bounding boxes. Those with an IoU greater than 60% are retained, thereby excluding persons not seated on motorbikes such as pedestrians, bicyclists etc. In the third step, we identify person bounding boxes sharing an IoU of over 60% with the helmet bounding boxes and assign them to the "Helmet" class. Any remaining person bounding boxes, which are not aligned with the helmet bounding boxes, are categorized as members of the "NoHelmet" class. 






b) Detection using text classes directly so that post processing not required

As we had observed in the section above, OWL-ViT <cit.> with basic classes need some post processing. We aim to use VLMs so that no post-processing is needed. In this case, we provided prompt with textual classes, which consists of entire sentences instead of individual words, providing a complete explanation. OWL-ViT is an open vocabulary object detector, and it performs well at identifying basic classes. In our study we are interested in determining whether OWL-ViT could perform equally well with textual classes without requiring any post-processing. 

For our research we had selected different textual prompts 

Prompt 1: “A person on a motorbike wearing helmet” 

Prompt 2: “A person on a motorbike bareheaded” 

Prompt 3: “A person on a motorbike without wearing helmet” 



  §.§.§ LLava model

As discussed in the earlier section, OWL-ViT <cit.> needs a post processing module to understand negative sentiment and give better results. In this scope of study, we wanted to test the performance of large language models like Llava-Next <cit.> on our use case. As mentioned in the section above, Llava-Next holds very good image understanding capabilities.  

In this scope of our study, we undertook several experiments with Llava-Next <cit.> to evaluate its performance in the field of Object Detection. To assess its performance in Image Understanding, we provided the prompt “Describe the image”. The Llava-Next model is known for its exceptional image interpretation capabilities, yielding precise results. However, Llava-Next is unable to generate bounding boxes or provide updated images with bounding boxes. We attempted to obtain the coordinates using different prompts but found that it exhibits poor object localization capability and negative sentiment understanding. Therefore, to leverage Llava-Next image understanding capabilities, we combined OWL-ViT with Llava-Next. 

OWL-ViT (basic classes) with Llava-Next :
In this part of our study, we integrated OWL-ViT <cit.> with Llava-Next <cit.> to optimize the outcomes. Initially OWL-ViT was employed with basic classes in it’s prompt, specifically, Motorbike, Person and Helmet. Following this, we implemented non-maximum suppression to remove redundant bounding boxes. Subsequently, we utilized IoU selection to extract the images of individuals seated on motorbikes, as mentioned in the preceding section. These images were then processed as inputs for Llava-Next, using the prompt: "Identify whether all person sitting on motorbike is wearing helmet or not?". 

Additionally, we employed a follow-up prompt to assign discrete classes. Specifically, we assigned the class “Helmet” to all crop images in which all visible individuals were wearing helmets, and the class “NoHelmet” to the crop images in which any of the visible persons was not wearing a helmet. The follow-up prompt was: " Write no if any person is not wearing helmet and write yes if all person is wearing helmet."  



  §.§.§ GPT-4o

As mentioned above, GPT-4o <cit.> is OpenAI latest LLM model. Being a closed source model, it exhibits visual language understanding than any other available model. It is exceptionally good in visual understanding, but similar to Llava-Next <cit.>, GPT-4o <cit.> also lacks the capability of object localization and fails to return correct bounding box coordinates. Therefore, we combine OWL-ViT and GPT-4o. The crops from OWL-ViT (basic classes) are given to GPT-4o, with a prompt: “Can you tell me the if there is a person wearing helmet or not. Only return helmet if all person are wearing helmet otherwise result nohelmet”. We don’t require a follow-up while using GPT-4o, as its textual understanding is good and it returned the expected classes, i.e., Helmet and NoHelmet. 




  §.§.§ Benchmark CNN Model

For comparing the results of VLM with CNN models, we finetune a YOLOv8 model <cit.>. The model is trained using 2500 training images. The YOLOv8 is trained on Nvidia-RTX A4000 GPU, it takes around 6 hours for 250 epochs.   



§ DATASET

In this section, we discussed the dataset used in our study for the tasks discussed earlier, starting with image classification tasks i.e., congestion and crack and classification and then object detection.



 §.§ Classification tasks

The congestion dataset was taken from the work of Chakraborty et al <cit.>, where images were obtained from 121 cameras from the Iowa DOT CCTV camera database spread across the inter-states and highways. The dataset has 1010 images in total, having 516 congested images and 494 non-congested images of highways. 

The dataset used for crack classification is SDNET2018 <cit.>, a publicly available dataset containing more than 56,000
images of concrete walls, bridges, and pavements. The pavement images are labeled and categorized into cracked and non-cracked classes. In our study, all the cracked 2608 images were used and we randomly selected 2600 non-cracked pavement images to balance the dataset.



 §.§ Object detection tasks

The dataset used is sourced from AICity Challenge 2024, specifically Track 5 - Detecting Violation of Helmet Rule for Motorcyclists <cit.>. The dataset consists of 100 training and 100 testing videos, recorded at 10 fps and 1080p resolution from various locations in an Indian city. We extracted the dataset and selected 2500 training images and 200 test images. These images show a close-up view of traffic captured by cameras. The dataset contains 3 object classes: motorbike, Helmet (a person seated on a motorbike wearing a helmet), and NoHelmet (a person seated on a motorbike without wearing a helmet). The dataset includes images of individuals wearing helmets, scarves, and turbans, as well as those without any headgear. Additionally, it contains footage from congested lanes. The same test dataset, consisting of 200 images, has been utilized for the vision-language model OWL-ViT <cit.>, Llava-Next <cit.> and GPT-4o <cit.>.




§ RESULTS AND DISCUSSION

In this section, we discussed the results achieved in our study by applying different state-of-the-art VLM models discussed earlier, starting with image classification tasks i.e., congestion and crack and classification and then object detection.


 §.§ Classification tasks



  §.§.§ Congestion classification






The task of classifying congestion was accomplished by applying zero-shot prompting to the three models. As zero-shot prompting involves performing tasks without any training on those tasks, it has to rely entirely on the instruction provided in the prompt itself to perform the task. 

The performance of the CLIP model varied depending on the class name used as prompts. The accuracy achieved for different classnames A1, A2, A3, A4 and A5 were 76%, 76%, 66%, 77%, and 88% respectively. The same classname was used by the BLIP model too but it had different results. For BLIP model the accuracy received were 86%, 94%, 93%, 49%, and 87% for A1, A2, A3, A4, and A5 respectively. The LLaVA-NeXT model utilized the initial and follow-up prompts instead of class names to classify the image. The result achieved by different combinations of initial and follow-up prompts, P1-F1, P2-F2, P3-F3, P4-F4, P5-F5 was 86%, 87%, 82%, 64%, and 87% accuracy respectively. The best results of all the models are presented in Table 1 with the Precision, Recall, and F1-score. 













            

                     





                       





 The True Positive (TP), True Negative (TN), False Positive (FP), and  False Negative (FN) are shown in Figure 3. As demonstrated in Fig 2 (g-i), it was observed that the models gave false positive results for the night-time because of the lightning effect and hence assumed it to be congested. Moreover, it can be also inferred from Fig 2 (j-l) that if one of the lanes had free-flowing conditions, it was classified as non-congestion even though the other lanes had congestion. The models are still not capable of understanding language in terms of whether any of the lanes are congested or not.  
From Table 1, it can be inferred that all the models gave comparative results and BLIP even outperformed the benchmark DCNN model.



























  §.§.§ Crack classification



The zero-shot prompting techniques being highly flexible allows the model to be applied to a wide range of tasks. Therefore, we also applied it to the crack classification task, but its ability to adapt to the specific demands of a new task is limited by what it has learned during its training.

The prompts play a vital role in getting the result from models through zero-shot prompting. The different class names used in the CLIP model with their accuracy, B1, B2, B3, B4, and B5 with 79%, 68%, 79%, 74%, and 70% respectively. The same class name was used by the BLIP model and it has varying results to CLIP, the accuracy achieved was 50%, 71%, 57%, 61%, and  50%, for class names B1, B2, B3, B4, and B5 respectively. Though the class names used by both models were the same, the results attained were contrasting. For the first and the last prompt used in the BLIP, it gave 50% accuracy which represents that the model is not able to handle negation prompts. The initial and follow-up prompts were used by the LLaVA-NeXT model to classify the image. The result attained by different combinations of initial and follow-up prompts P1-F1, P2-F2, P3-F3, P4-F4, and P5-F5 was 72%, 67%, 53%, 58%, 76% accuracy respectively. 



















The best result of all the models is demonstrated with precision, recall, and F1-score of each class in Table 2. The TP, TN, FP, and  FN are shown in Figure 3. It was inferred that the models are not able to distinguish between rough surface and crack, all the models classified the rough surface as cracked as shown in Fig 3 (g-h). On the other hand, if the crack was present near the edge as shown in Fig 3 (j-l), the model was not able to identify the crack.

In the image classification task, VLMs performed well even with zero-shot prompting. As no specific prompt was used, further by applying prompt-based strategies, we can enhance the models’ performance, making them viable for high-level tasks and possibly reducing the reliance on extensively annotated datasets.






















 §.§ Object detection tasks






  §.§.§ OWL-ViT







As mentioned in the Table 3, OWL-ViT <cit.> shows significant good results when given basic classes in its prompts, such as Motorbike, Person, and Helmet. In order to achieve the Helmet and NoHelmet classes, some post-processing needs to be done. After post-processing, the Zero-Shot OWL-ViT model shows significantly better results compared to a trained YOLOv8 <cit.> model. It achieved a precision of 95% in the Helmet class and 74% precision in the NoHelmet class, as mention in Table 3. The inference results are shown in the Fig 4. One major advantage of the OWL-ViT is, with basic prompts, can also identify the differences between a cap, turban, scarf, and helmet, as shown in the Fig 4 (a,b,e). 


OWL-ViT <cit.> has not been trained on higher language models, resulting in lack of visual and textual understanding. With the help of different prompts, we have observed that OWL-ViT performs poorly in processing textual data that consist of complete sentences. Furthermore, it also lacks the understanding of negative sentiments.






  §.§.§ LlaVa-Next

Llava-Next <cit.> utilizes LlaMa models <cit.>, showing significant advancements in image understanding. Llava-Next cannot detect objects or provide images with bounding boxes, which can be achievable by OWL-ViT <cit.>. In our study, by tuning prompt, we get the bounding box coordinates in pascal voc format <cit.> (xmin, ymin, xmax, ymax). Llava-Next cannot accurately locate objects and gives incorrect coordinates, despite its excellent understanding of prompts and images. 

Our approach involved integrating the OWL-ViT <cit.> model with Llava-Next <cit.> to leverage its image understanding capabilities. The basic class prompts of OWL-ViT were utilized as inputs for the Llava-Next model, as shown in Fig 5(a). Furthermore, we employed follow-up prompt for summarizing the results. As mentioned in the Table3, with our experiment we achieved 88% precision in Helmet class and 90% precision in NoHelmet class. 



  §.§.§ GPT-4o

GPT-4o <cit.> has a great capability for understanding visual language, similar to Llava-Next <cit.>. However, it does not support object detection. It has excellent image understanding, even more so than Llava-Next. By combining it with OWL-ViT, and providing GPT-4o with crops of OWL-ViT <cit.> (basic class prompt), we achieved 99% precision in the Helmet class and 92% precision in the NoHelmet class, which is almost equal to the benchmark results of CNN, which is shown in the Fig 5(b). Additionally, GPT-4o shows a recall of 99% for the NoHelmet class, which is higher than the benchmark results of CNN as shown in Table 3.  





According to table 3, VLM demonstrates significantly good precision and recall in zero-shot performance. This capability can lead to cost reduction by eliminating the need to fine-tune models for every new use case and annotate millions of images. With the right methodology, engineering, and utilization, VLM has the potential to outperform traditional CNN models. Despite excelling in these areas, VLM has poor image localization capabilities. Models such as Llava-Next and GPT-4o understand images well but struggle to localize objects. Additionally, VLMs are not currently lightweight or fast compared to CNN models. For example, OWL-ViT with Llava-Next took 6.2 seconds to process one image, while OWL-ViT with post-processing took 0.68 seconds and even the closely related GPT-4o took 3.3 seconds. In contrast, a CNN model only took 0.14 seconds for the same task. While VLMs can provide better accuracy, further work is needed to make them suitable for real-time field use.












§ CONCLUSIONS

The objective of the study was to understand the performance of visual language models in vision-based transportation tasks. It was carried out by comparing different state-of-the-art VLM models by zero-shot prompting for two tasks i.e., image classification and object detection. The transportation-related vision task selected for image classification was congestion detection and crack identification whereas, for object detection, it was the identification of helmet violations. VLMs performed at par sometimes but performance needs to be improved in terms of prompt engineering, localization of object detected, and inference time. This paper gives a comprehensive understanding of VLM models' limitations which need to be worked upon for large-scale implementation. Future studies can also be done in other case studies related to vision-based transportation tasks. Further other VLM models can also be explored with different benchmark datasets.


 

§ AUTHOR CONTRIBUTIONS

The authors confirm their contribution to the paper as follows: study conception and design:  T. Singh, S. Prajapati, C. Hedge, P. Chakraborty; data collection: S. Prajapati, T. Singh; analysis and interpretation of results: S. Prajapati, T. Singh, C. Hedge, P. Chakraborty; draft manuscript preparation: S. Prajapati, T. Singh, C. Hedge, P. Chakraborty. All authors reviewed the results and approved the final version of the manuscript.




§ ACKNOWLEDGEMENTS

Our research results are based upon work supported by the IITK-NYU Joint Research Grant. Any opinions, findings, and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the IITK and NYU. 

trb



