Traffic Light Recognition using Convolutional Neural Networks: A Survey
Real-time traffic light recognition is essential for autonomous driving. Yet,
a cohesive overview of the underlying model architectures for this task is
currently missing. In this work, we conduct a comprehensive survey and analysis
of traffic light recognition methods that use convolutional neural networks
(CNNs). We focus on two essential aspects: datasets and CNN architectures.
Based on an underlying architecture, we cluster methods into three major
groups: (1) modifications of generic object detectors which compensate for
specific task characteristics, (2) multi-stage approaches involving both
rule-based and CNN components, and (3) task-specific single-stage methods. We
describe the most important works in each cluster, discuss the usage of the
datasets, and identify research gaps.
Svetlana Pavlitska, Nico Lambing, Ashok Kumar Bangaru, J. Marius Zöllner
# Traffic Light Recognition using Convolutional Neural Networks:

###### Abstract

Real-time traffic light recognition is essential for autonomous driving. Yet, a cohesive overview of the underlying model architectures for this task is currently missing. In this work, we conduct a comprehensive survey and analysis of traffic light recognition methods that use convolutional neural networks (CNNs). We focus on two essential aspects: datasets and CNN architectures. Based on an underlying architecture, we cluster methods into three major groups: (1) modifications of generic object detectors which compensate for specific task characteristics, (2) multi-stage approaches involving both rule-based and CNN components, and (3) task-specific single-stage methods. We describe the most important works in each cluster, discuss the usage of the datasets, and identify research gaps.

## I Introduction

Detection and classification of traffic lights (TL) from camera images, also called _traffic light recognition_ (TLR), plays a pivotal role in enabling automated driving. It helps to maintain efficient and safe traffic flow management, reduce traffic congestion and minimize the risk of accidents. TLR as a task comprises _traffic light detection_, which aims at localizing the traffic lights in the image, as well as _classification of TL states_ (colors) and _pictograms_ (arrows), as shown in Figure 1. The development of convolutional neural networks (CNNs) has dramatically improved the accuracy of traffic light detection due to their ability to learn complex features from images. The effectiveness of CNN-based approaches depends on the choice of architecture and training data.

In this work, we review and group existing CNN-based approaches for traffic light detection and classification. Unlike existing surveys on TLR, we focus on the choice of CNN architectures. Older surveys [1, 2] focused more on classic image processing approaches since neural networks were only sporadically used then. To the best of our knowledge, the only concurrent modern work is that by Gautam et al. [3], which presents an in-depth overview but focuses on the whole pipeline. For the three main steps in the proposed pipeline (segmentation, feature extraction, and classification), Gautam et al. consider both classical computer vision approaches, like histograms of oriented gradients, and those using neural networks. In contrast, we focus on CNN model architectures and particularly on the modifications made to generic object detectors.

## II Datasets for Traffic Light Recognition

Since the appearance of traffic light signalling devices varies over different countries, a number of TLR benchmarks has been released. We provide an overview of publicly available datasets in Table I. We also refer to the journal paper by Jensen et al. [2], which gives a comprehensive overview of datasets published before 2016.

**La Route Automatisce (LaRa) dataset**[6] was one of the first publicly available datasets published in 2015 by a French joint research unit La Route Automatisce. It contains over 11,000 images and 9,000 annotations recorded as a 25 Hz video during about a 9-minute long ride in Paris. The images have a relatively low resolution of \(640\times 480\) pixels. All labels were annotated manually as bounding boxes (BBoxes) with object IDs for tracking evaluation. The TL state was labeled as _green, orange, red_, or _ambiguous_. Furthermore, each

Fig. 1: Examples of subtasks within the TLR task.
image was annotated with a sequence ID and timestamp.

**LISA Traffic Light Dataset**[2] is a comprehensive dataset that contains over 40,000 images, originating from the Vision for Intelligent Vehicles and Application (VIVA) challenge, which included the traffic light detection benchmark. Therefore, the dataset itself is sometimes also referred to as the _VIVA dataset_. The data was captured as a 10 Hz video using a stereo camera with an image resolution of \(1280\times 960\) pixels and a horizontal field of view of approximately 43\({}^{\circ}\). Additionally, depth disparity maps for each image are provided. The dataset consists of a training subset and a test subset, the latter is kept private to serve as the basis for benchmarking. All labels were annotated manually and are provided as pixel-level binary masks and BBoxes. TL states are encoded using seven classes: _go, go forward, go left, warning, warning left, stop, stop left_. The LISA dataset covers several USA cities (San Francisco, Berkeley, and Chicago) under different lighting and weather conditions.

**Bosch Small Traffic Lights Dataset (BSTLD)**[8] was recorded along the El Camino Real in California's San Francisco Bay Area using a stereo camera. The dataset includes the corresponding disparity maps. All labels were annotated manually utilizing 15 classes to describe the color and pictogram of the TLs. The labels are provided as pixel-wise binary masks and BBoxes and are split into a training set and a test set of nearly equal size. Although the training data is labeled with a full set of 15 classes, the test data includes only four classes (_red, yellow, green, off_).

**DriveU Traffic Light Dataset (DTLD) v1.0**[9] was published in 2016 by the Intelligent User Interfaces (IUI) group at the University of Ulm in Germany. It has a number of images comparable to LISA but exceeds all other datasets in terms of the number of annotations (more than 230,000). Images with a resolution of \(2048\times 1024\) pixels were recorded by a stereo camera with a frame rate of 15 Hz. The dataset includes the corresponding disparity maps.

The DTLD v2.0 dataset was released in 2021 as an extension of the DTLD v1.0. It contains images of the same resolution and frame rate but covers a broader range of traffic scenarios, such as roundabouts and T-junctions. Both datasets were annotated with BBoxes using manual and semi-automatic methods. The manual annotation was performed by human annotators, who labeled the TLs with pixel-level accuracy. The semi-automatic annotation was performed using a deep neural network trained to detect and classify TLs in the images.

Both datasets provide a comprehensive set of labels, arranged into the following groups: the orientation (_front, back, left, right_), relevance/occlusion, orientation (_horizontal, vertical_), the number of lamps, state (_red, yellow, green, red-yellow, off_), and pictogram (_circle, arrow left, pedestrian_, etc.). Because of the large number of possible combinations of these tags, the resulting number of unique labels exceeds that of any other dataset. DTLD v1.0 and v2.0 were collected from eleven German cities, including urban and suburban environments, to provide diverse TL scenarios.

Furthermore, a number of **other public datasets** either include labeled TL states or have been extended to include them. However, they lack additional attributes such as orientation, pictogram, and relevance information, which are necessary to utilize the detected TLs for autonomous driving. Examples of the datasets extended with TL states include COCO Traffic [12], where TL states were annotated in the images from the COCO [13] dataset, as well as Cityscapes TL++ dataset [10] containing images with fine annotations from the Cityscapes [14] dataset with additional TL labels for four attributes: type (_car, pedestrian, bicycle, train, unknown_), relevant (_yes, no_), visible (_yes, no_), and state (_red, red-yellow, yellow, green, off, unknown_). Other datasets containing only TL state labels are the Roboflow Self-Driving Car dataset [15], a modified version of the Udacity Self-Driving Car Dataset [16], Waymo Open Dataset [17], WPI [7], BDD100K [18], and ApolloScape [19] datasets.

## III Overview of Architectures for Traffic Light Recognition

Compared to the generic object detection task, specific challenges in TLR include small object size, sparse structure, and high variability of the background. Various works have proposed different methods to approach these issues. We cluster them into three groups: (1) **modifications** of generic object detectors, (2) **multi-stage approaches**, which perform TL localization and TL state/pictogram classification in separate steps, and (3) **task-specific single-stage approaches**, which perform TLR within a single network.

Table II summarizes existing work on CNN-based TLR approaches. In the following, we give an overview of the most important works in each group. Please note that we have deliberately omitted approaches involving only TL classification, without previous detection step (e.g., Gautam and Kumar [20]), as they are unrealistic for the deployment.

### _Modifications of Generic Object Detectors_

The first group comprises approaches that use an existing CNN-based model for generic object detection with minor

\begin{table}
\begin{tabular}{|c c c c c c c c c c c c|} \hline
**Dataset** & **Year** & **Ref.** & 
\begin{tabular}{c} **Number of** \\ **Image** \\ \end{tabular} & **Resolution** & **Depth [bit]** & **Frame Rate** & **Number of** & **Impairly** & **Platforms** & **Clauses** & **Country** & **License** \\ \hline LaRa & 2015 & [6] & 11.179 & 640\(\times\)480 & 8 & 25 & 9,168 & & 4 & France & NA \\ LISA & 2016 & [2] & 43.007 & 1280\(\times\)960 & 8 & 16 & 119.231 & ✓ & ✓ & 7 & USA & CC & BY-NC-SA 4.0 \\ WPI & 2016 & [7] & 3.456 & 1920\(\times\)1080 & NA & NA & 6766 & & ✓ & 21, 2 & USA & NA \\ BSTLD & 2017 & [8] & 13.427 & 1280\(\times\)720 & 8.12 & 15 & 24,242 & ✓ & ✓ & 15,4\({}^{*}\) & USA & MIT \\ Direct v1.0 & 2018 & [9] & 40.979 & 2048\(\times\)1024 & 8.16 & 15 & 229.029 & ✓ & ✓ & 423 & Germany & Academic \\ Direct v2.0 & 2021 & [9] & 40.979 & 2048\(\times\)1024 & 8.16 & 15 & 292,245 & ✓ & ✓ & 620 & Germany & Academic \\ Cityscapes TL++ & 2022 & [10] & 5.000 & 2048\(\times\)1024 & 16 & 17 & N/A & ✓ & 6 & Germany & LGP-L2.1 \\ S\({}^{\prime}\)TLD & 2022 & [11] & 5.786 & 1080\(\times\)1920 & 720\(\times\)1280 & NA & NA & 14.130 & & 5 & China & MIT \\ \hline \end{tabular}
\end{table} TABLE I: Comparison of TLR datasets (\(*\) – the number of classes in the test subset)
\begin{table}
\begin{tabular}{|l l l|l l l l l l l|} \hline
**Author** & **Year** & **Ref.** & **Approach** & **Dataset** & **Inference** & **Accuracy** & **TL** & **TL** & **TL** & **platforms** & **Source** \\  & & & & & & & & **speed** & & **classified** & **classified** & **code** \\ \hline John et al. & 2014 & [21, 22] & CNN similar to LeNet & Private (USA, & & & & & **reduce**: 96.25-99.4\% & ✓ & & \\ \hline Weber et al. & 2016 & [23] & DeepTLR (single CNN) & Private (Germany) & 30-77 ms & F1: 93.5\% & ✓ & & & \\ \hline Behrend et al. & 2017 & [8] & Detection: VOLO/L & BSTLD & 67-100 ms & F1: \(-\)80\% & ✓ & & & \\ \hline Jensen et al. & 2017 & [24] & Modified VOLO/Q2 & LISSA/LRaRa & N/A & AUC: 90.499 (LSA) & & & \\ \hline Weber et al. & 2018 & [5] & BSTLD: H/s single CNN & Private (Germany) & 83 ms & F1: 88.8\% (BSTLD) & ✓ & ✓ & & \\ \hline Muller and Dietzayer & 2018 & [4] & (Interpolito-v3) & DTLD & 100 ms & Recall: 95\% & ✓ & & ✓ \\ \hline Fun et al. & 2018 & [5] & Faster R-CNN (NeNeNe-30) & BSTLD & 15 ms & mAP: 53\% & ✓ & & \\ \hline Bich et al. & 2018 & [25] & Modified Faster R-CNN (ResNet-50) & DTLD & N/A & mAP: 83\% & ✓ & & \\ \hline Kim et al. & 2018 & [25] & Color space transformation + an ensemble of 3 networks: & & & & & \\ \hline Kim et al. & 2018 & [25] & Faster R-CNN (inception-ResNet-v2) & SSTLD & N/A & mAP: 38.48\% & ✓ & & \\  & & or Resnet-101 & of RF-CNN (ResNet-101) & & & & & \\ \hline Lu et al. & 2018 & [25] & Visual attention proposal + detection, both based on Faster R-CNN & Private (China) & N/A & mAP: 91.1\% (LISA) & ✓ & ✓ & \\ \hline Wang et al. & 2018 & [25] & saliency map filtering, classification: AlexNet & Private (Singapore) & 35 ms & mAP: 98.9\% & ✓ & ✓ & \\ \hline Kim et al. & 2018 & [30] & SSD for coarse-grained detection + spatiotemporal refinement & Private (USA) & N/A & F1: 10.05\% - 69.68\% & ✓ & ✓ & \\ \hline Wang et al. & 2018 & [31] & Detection: VOLO/G3 & BDD110K & 35 ms & Accuracy: 98\% & ✓ & & \\ \hline Yudin et al. & 2018 & [32] & Detection: fully-connected network & Near TLR & 63 ms & Recall: 94.7\%, & & & \\ \hline Hin et al. & 2019 & [33] & Modified Faster R-CNN (VGG16) & Private (China) & N/A & mAP: 89.26\% & & & \\ \hline Possari et al. & 2019 & [34] & VOLO’s prior maps & Pritute (Brazil) & 48 ms & mAP: 50.5\% (LISA) & ✓ & & ✓ \\ \hline Eminhield et al. & 2019 & [35] & Faster R-CNN (ResNet-101), & BSTLD, LISA & 200.333 ms & mAP: 79.01\% & ✓ & & \\ \hline Gupta and Choudhary & 2019 & [36] & Detection: Faster R-CNN (VGG16), Classification: Grassmannian manifold & BSTLD, LaRa, LISA, WPI & 31 ms & Accuracy: 98.80\% & ✓ & ✓ & \\ \hline Du et al. & 2019 & [37] & YOLO3 & Private (China) & 106 ms & mAP: 96.18\% & & & \\ \hline Yoh et al. & 2019 & [38, 39] & DTL state classification: VOLO-3-airy & LISA, Firestone (China) & 31-52 ms & mAP: 66\% (LISA) & ✓ & ✓ & \\ \hline Kim et al. & 2019 & [40] & Detection: ENsek-based network, Classification: LeNet-based CNN & BSTLD & 34 ms & F1: 95.10\% & ✓ & & \\ \hline Amechs et al. & 2019 & [41] & Retain
modifications to compensate for smaller object sizes. The corresponding approaches are marked green in Table II. Generic object detectors are especially favorable due to their inference speed.

The earliest approach to modify YOLO [67] for the TLR task was presented by Jensen et al. [24]. Here, YOLOv2 [68] was modified by removing the last convolutional layer and adding three \(3\times 3\) convolutional layers with 1024 filters, followed by a \(1\times 1\) convolutional layer with the number of outputs needed for the specific detection. This model, however, only performed detection, not the classification of the TL states. Bali et al. [58] tried to replace the YOLOv2 backbone with different lightweight CNNs, whereas the best results were achieved with SqueezeNet [69].

Muller and Dietmayer [4] presented a modified version of the SSD [70] architecture for TLR with Inception-v3 [71] instead of VGG [72] backbone for a better accuracy-speed trade-off. The authors analyzed the layer and feature map sizes of Inception-v3 and showed that they cannot guarantee the detection of objects with a width of 5 pixels. Therefore, to increase the recall on small objects, they introduced modified priors placed not in the center of each feature cell but arbitrarily using the offset vectors. Furthermore, early and late feature layers were concatenated for the BBox and confidence prediction to use context information from the early layers better. As in SSD, the confidence loss was formulated as a two-class problem (TL vs. background). Also a further layer was added to detect the TL state (_red, yellow, green, off_).

Faster R-CNN [73] was first applied by Pon et al. [25] for TLR within the joint traffic light and traffic sign detection network. Bach et al. [26] suggested further modifications to Faster R-CNN for TLR. In particular, some layers of the feature extractor networks (ResNet-50) were modified. Furthermore, anchors were determined not arbitrarily but via k-means clustering of the training set BBoxes. Finally, the loss function was expanded to allow for TL classification. Han et al. [33] used the modified Faster R-CNN with VGG16 backbone for traffic sign and traffic light detection. To account for small object size, a small region proposal generator was used. For this, the _pool4_ layer of VGG16 was removed. Additionally, the online hard examples mining (OHEM) [74] approach was applied to locate small objects more robustly and helped to increase mAP by 2-3 pp. The best results, however, were achieved with ResNet-50 [75] with dilation.

Abraham et al. [52] used a modified YOLOv4 [76] with cross-stage partial connections (CSP). The feature extractor contained a Darknet53 [77] backbone, a path aggregation network, spatial pyramid pooling, and a spatial attention module, while the detector used the YOLOv4 head. A similar approach was followed by Wang et al. [56]. Here, YOLOv4 with CSPDarknet-53 feature extraction network was modified by fusing certain layers and enhancing the shallow features. Furthermore, the BBox uncertainty prediction was also added. Lastly, Zhao et al. [57] showed that ShuffleNet [78] leads to better results when used as a backbone in YOLOv4.

The work by Ennahhal et al. [35] is one of the few that compared several approaches. Their results show that Faster R-CNN outperformed R-FCN [79] and SSD in terms of mAP. Later, Gokul et al. [51] have also demonstrated that Faster-R-CNN has the best trade-off between accuracy and speed compared to YOLOv2 and YOLOv3.

Liu and Li [64] proposed to modify the backbone of the YOLOv51. The custom backbone architecture is inspired by the U2Net [80] and contains a series of residual U-blocks. Additionally, the authors replace the C3 modules in the neck part of YOLOv5 with ConvNextBlocks [81] to improve feature extraction. The resulting model has demonstrated better accuracy compared to the baseline YOLOv5. Models based on YOLOv5s have demonstrated a remarkable inference speed of 48 FPS.

Footnote 1: [https://github.com/ultralytics/yolov5](https://github.com/ultralytics/yolov5)

Finally, a single approach that goes beyond CNN-based object detectors is that by Greer et al. [65]. The authors used the Deformable DETR [82], a generic object detector with a transformer encoder-decoder architecture and features extracted using a CNN backbone (ResNet-50). The authors evaluated the impact of the salience-sensitive focal loss and showed better performance on salient traffic lights.

### _Multi-Stage Approaches_

The second group contains approaches where the TLR task is split into two subtasks: detection and classification, s.t. a separate model is used for each of them. The corresponding approaches are marked blue in Table II.

**Generic object detector + CNN for classification:** In the work by Behrendt et al. [8] introducing the BSTLD dataset, YOLO was modified to detect TL objects as small as \(3\times 10\) pixels. For this, the authors took random crops of size \(448\times 448\) from an image. Also, the number of grid cells was increased from \(7\times 7\) to \(11\times 11\). The classification part of the original YOLO loss was removed. Instead, a small classification network consisting of three convolutional and three fully-connected layers was used to detect TL states.

Lu et al. [28] proposed an approach consisting of two parts: the first one proposes attention regions that can contain traffic lights, and the second part performs localization and classifications on the cropped and resized attention regions found by the first model. Both blocks follow the Faster R-CNN architecture.

A similar approach was followed by Wang et al. [31], who used YOLOv3 [77] for the detection of regions of interest (ROI). The classification of a TL status was performed with a lightweight CNN consisting of two convolutional and two max-pooling layers. Similarly, as in the previous work, the lightweight CNN gets ROIs from the YOLOv3 as input and predicts one of the four states (_red, green, yellow, unknown_).

Cai et al. [43] proposed a two-stage approach, where the detection part consisted of the SSDLite with MobileNetv2 [83], whereas classification was performed by a small three-layer network.
In the work by Kim et al. [40], the detection stage is performed by a semantic segmentation network, which is then used to calculate BBoxes. This is motivated by its better performance on very small objects. In particular, a binary version of the ENet [84] is used. For the classification part, a LeNet-5-based [85] model is used. This model was shown to beat Faster R-CNN from the previous work by authors [27] both in terms of accuracy and speed.

Jayasinghe et al. [60] used a two-stage approach, where detection was performed either with Faster R-CNN with a ResNet-50 backbone or SSD with MobileNetv2 backbone, and the classification was done with ResNet-18.

**Generic object detector + non-deep learning approach for classification:** Kim et al. [30] used an unmodified SSD with a standard VGG16 backbone as a coarse-grained detector. The fine-grained detection is performed via spatiotemporal filtering and has the goal to compensate for the poor performance of SSD on small objects. The latter uses a point-based reward system; the points are rewarded for detections consistent in the spatial and temporal domains.

Yudin et al. [32] used a U-Net [86]-inspired fully-convolutional network to predict a grayscale map of TL locations, which is further binarized using thresholding. After that, the detected regions are clustered using DBSCAN and filtered, yielding the predicted location. The proposed approach is shown to lead to higher precision and recall compared to the SSD300.

Gupta and Choudhary [36] used Grassman manifold learning for TL and pictogram classification, while the detection step was performed with a Faster R-CNN. For the TL classification, features extracted from VGG16 were used to create subspaces on a Grassman manifold for each TL state. After that, discriminant analysis on the manifold was used to distinguish between TLs.

In the work of Tran et al. [46], the detections and classifications made by YOLOv4 are additionally processed by a color-based clustering method to remove irrelevant predictions. Moreover, a rule-based heuristic to identify the most important TL in an input image is applied as the last step. Similarly, Nguyen et al. [47] validate the predictions done by YOLOv3 via hand-crafted features and classification using HSV color space.

**Non-deep learning detector + CNN for classification:** Wang et al. [29] used a high dynamic range camera to get input images for different channels; this allowed them to detect TL ROIs from input images using a saliency map. Then, a customized AlexNet was used for the TL classification. Kim et al. [27] also used a color-based approach. They proposed transforming an input image to another color space before passing it to a generic object detector. Different models represented the latter, whereas Faster R-CNN with Inception-ResNet-v2 was shown to be the most suitable for the task. The HSV color space was used in work by Gao et al. [49] to generate the ROIs, whereas the classification was performed with AlexNet. Vitas et al. [50] applied adaptive thresholding to generate ROIs at the detection step, whereas the classification was done with a simple three-layer CNN.

**Further approaches:** Possatti et al. [34] incorporated the usage of prior maps containing coordinates of TLs, whereas YOLOv3 was used for TLR. YOLOv3 was not additionally modified and trained to distinguish between two classes: _red-yellow_ and _green_ TLs. The TL position is projected to the image plane using the data from the prior maps and the vehicle localization data. Finally, only those BBoxes predicted by the YOLOv3 corresponding to the projected map objects are used for final predictions.

Yeh et al. [38, 39] presented a three-stage approach, where YOLOv3 first localizes traffic lights. Next, YOLOv3-tiny detects the TL states. Finally, LeNet is applied to classify the arrows in different directions. HD maps and collected LiDAR data are used to find the TL position.

### _Task-specific Single-stage Approaches_

Finally, the third group comprises those approaches where TLR is performed within a single network deliberately designed for this task. The corresponding methods are marked yellow in Table II. Unlike most methods, which follow the two-step approach involving TL detection and subsequent classification, the DeepTLR by Weber et al. [23] is a pure CNN that directly classifies each fine-grained pixel region over the image, thus creating a probability map for each of three classes: _red, yellow,_ and _green_. For the pixels in probability maps, which surpass a certain threshold, BBox prediction is performed. The feature extraction part of DeepTLR uses the AlexNet architecture [87], whereas the BBox regression follows that of the OverFeat [88].

The HDTLR approach [5] by Weber et al. builds upon DeepTLR, extending and improving the detection part. Unlike DeepTLR, HDTLR can use any CNN for the feature extraction part. Experiments were performed with AlexNet, GoogLeNet, and VGG, while the latter performed the best.

Wang et al. [59] proposed a joint detection and tracking approach, whereas a CNN and integrated channel feature tracking are used to predict both TL coordinates and states.

## IV Conclusion

In this paper, we gave an overview of the existing works on traffic light recognition. Our analysis has revealed that the predominant approach in the literature is the modification of a generic object detector like YOLO, SSD, or Faster R-CNN. In particular, YOLO versions 1-5 were used especially often. A large group of multi-stage approaches uses an existing detector as an attention or region proposal module, which determines the positions of the traffic lights, whereas an additional CNN classifier distinguishes between traffic light states and pictograms. This classification network usually has a very simple architecture. Less popular is the usage of a rule-based ROI detector or of a non-CNN classification method. Finally, a separate cluster of approaches is formed by methods that perform traffic light recognition within a single model so that the task is learned end-to-end without intrinsic separation into detection and classification steps.

Furthermore, our overview has shown that a lot of works reach real-time performance, but perform evaluation on private datasets, which makes a fair comparison of different
methods difficult. We also have determined that, unlike most object detection tasks, open-sourcing the code of the TLR models is still rare. We hope our findings facilitate further research on traffic light recognition.

## Acknowledgment

The research leading to these results is funded by the German Federal Ministry for Economic Affairs and Climate Action within the project "Shuttle2X" (grant 19S22001B).
