# Beyond the Mud: Datasets and Benchmarks for Computer Vision in Off-Road
Racing

Jacob Tyo<EMAIL_ADDRESS>DEVCOM Army Research LaboratoryCarnegie Mellon
University5000 Forbes AvePittsburghPennsylvaniaUSA15239 ,  Motolani Olarinre
<EMAIL_ADDRESS>Carnegie Mellon University5000 Forbes
AvePittsburghPennsylvaniaUSA15239 ,  Youngseog Chung<EMAIL_ADDRESS>Carnegie Mellon University5000 Forbes AvePittsburghPennsylvaniaUSA15239 and
Zachary C. Lipton<EMAIL_ADDRESS>Carnegie Mellon University5000 Forbes
AvePittsburghPennsylvaniaUSA15239

(2024)

###### Abstract.

Despite significant progress in optical character recognition (OCR) and
computer vision systems, robustly recognizing text and identifying people in
images taken in unconstrained _in-the-wild_ environments remain an ongoing
challenge. However, such obstacles must be overcome in practical applications
of vision systems, such as identifying racers in photos taken during off-road
racing events. To this end, we introduce two new challenging real-world
datasets - the off-road motorcycle Racer Number Dataset (RND) and the Muddy
Racer re-iDentification Dataset (MUDD) - to highlight the shortcomings of
current methods and drive advances in OCR and person re-identification (ReID)
under extreme conditions. These two datasets feature over 6,300 images taken
during off-road competitions which exhibit a variety of factors that undermine
even modern vision systems, namely mud, complex poses, and motion blur. We
establish benchmark performance on both datasets using state-of-the-art
models. Off-the-shelf models transfer poorly, reaching only 15% end-to-end
(E2E) F1 score on text spotting, and 33% rank-1 accuracy on ReID. Fine-tuning
yields major improvements, bringing model performance to 53% F1 score for E2E
text spotting and 79% rank-1 accuracy on ReID, but still falls short of good
performance. Our analysis exposes open problems in real-world OCR and ReID
that necessitates domain-targeted techniques. With these datasets and analysis
of model limitations, we aim to foster innovations in handling real-world
conditions like mud and complex poses to drive progress in robust computer
vision. All data was sourced from PerformancePhoto.co, a website used by
professional motorsports photographers, racers, and fans. The top-performing
text spotting and ReID models are deployed on this platform to power real-time
race photo search. All code and data can be found at
https://drive.google.com/file/d/1YFlQLh3rorLLvG8reYxHiqs7HkWf1lJF/view?usp=sharing.

Motorcycle Dataset, Text Detection, Text Recognition, Person Re-
Identification, Text Spotting, Machine Learning

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
Preprint; February 12, 2024; ††isbn: XXX-X-XXXX-XXXX-X/XX/XX††ccs: Computing
methodologies Visual content-based indexing and retrieval

(a)

(b)

Figure 1. (a) Example of muddy images, grouped by the person re-identification
label, emphasizing both the text spotting and person re-identification
difficulty. (b) A clean (i.e. not muddy) example from the text spotting
dataset.

##  1\. Introduction

Robustly recognizing visual concepts like text and people in unconstrained
real-world environments remains an ongoing challenge in computer vision. While
the fields of document text recognition and detection (jointly referred to as
_text spotting_) as well as person re-identification (ReID) have made
significant advances in recent years owing to deep learning methods and large
annotated datasets, performance still tends to degrade under difficult
lighting, occlusion, unusual perspectives, motion blur, and other factors not
well-represented in training data (Appalaraju et al., 2021; Shashirangana et
al., 2020; Netzer et al., 2011; Ye et al., 2021; Gou et al., 2018; Chen et
al., 2021).

For example, state-of-the-art optical character recognition (OCR) models can
now transcribe scanned documents and signage with over 99% word accuracy
(Fujitake, 2024). However, accurately reading text on objects in cluttered
scenes or under heavy occlusion is still limited to around 80% (Zhao et al.,
2023; Ye et al., 2023) in benchmarks like Total-Text (Ch’ng and Chan, 2017)
and COCO-Text (Veit et al., 2016). Similarly, person re-identification models
can reliably match people across different camera views when imaging
conditions are consistent. But model accuracy drops substantially in the
presence of occlusions or rarely seen poses (Ye et al., 2021).

In this work, we focus on one particularly challenging real-world domain that
exposes the brittleness of current techniques - off-road motorcycle racing.
Correctly identifying riders and detecting racing numbers is not only
important for accurate photo searching, but also for sports analytics,
automatic timing and scoring, photo searching, and more. However, this setting
induces several factors that significantly undermine the capability of
existing vision models:

  1. (1)

Heavy mud spatters accumulate and obscure text and riders in irregular,
unprecedented ways. Mud caking exhibits occlusion patterns not seen in typical
datasets.

  2. (2)

Riders strike complex poses including jumps, crashes, and wheelies, among
others which are rarely depicted under standard walking scenarios. These
confuse models tuned to pedestrian datasets.

  3. (3)

Sizes of number tags vary from a few inches to over a foot tall, placed in
diverse layouts on bikes and rider gear, and in an ever-increasing number of
fonts.

  4. (4)

Racing conditions cause glare, blurring, awkward angles, and low-resolution
imagery not prevalent in controlled datasets. As a race progresses, the
appearance of a single racer can change dramatically.

To spur progress in this domain, we introduce two real-world datasets captured
from off-road competitions. The _off-road Racer Number Dataset_ (RND) contains
2,411 images exhibiting riders engaged in races. Each image is annotated with
bounding boxes around visible racer numbers on bikes, jerseys, or helmets,
along with the number text transcription (e.g. “15A”). In total, there are
over 5,500 labeled rider numbers spanning challenges like mud, awkward
perspectives, low resolution, and more.

The second dataset focuses specifically on the rider re-identification task.
The _MUddy Racer re-iD Dataset_ (MUDD) includes 3,906 images capturing 150
riders across 10 different off-road events. Each rider crop is annotated with
a unique identity label. The imagery is subject to real-world variations in
mud, lighting, pose, blurring, age, outfit changes, crashes, and more over
each race.

We benchmark state-of-the-art vision models on both datasets and find that
pretrained models transfer poorly, with accuracies under 30% in both text
spotting and ReID. Even after fine-tuning, the best models achieve only 53% F1
score for end-to-end racer number recognition, and just 79% rank-1 accuracy on
rider matching. Our analysis exposes open problems in handling mud occlusion,
appearance change, layouts, resolution, and other factors. Through domain-
targeted data, benchmarks, and analysis, we aim to spur innovation advancing
robust capabilities.

Our contributions are:

  * •

RnD: an off-road motorcycle Racer number Dataset containing 2,411 images with
5,578 labeled numbers sampled from professional photographers at 50 distinct
off-road races.

  * •

MUDD: a Muddy racer re-identification Dataset for person re-id, containing
3,906 images of 150 identities captured over 10 off-road events by 16
professional motorsports photographers.

  * •

Initial benchmarking of state-of-the-art models, which reveal limitations on
these datasets and show substantial room for further improvement.

  * •

Analysis of failure cases that provide insights to guide future research on
robust re-identification and text spotting for sports analytics and computer
vision broadly.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2. Beyond the Mud Dataset Examples: (a) Racers can have multiple, non-
matching, numbers, (b) glare renders some numbers impossible to read, (c) a
crashing racer, (d) occlusions from vegetation, (e) an extreme example of mud,
(f) a stereotypical amount of mud.

##  2\. The Datasets

We introduce two large-scale datasets constructed from real-world off-road
motorcycle competition imagery from PerformancePhoto.co. The dataset was
collected from over 50 different off-road motorcycle and ATV races, from 16
different professional photographers, using a wide range of high-end cameras.
This includes both professional and amateur races across a wide range of
geographical locations in the United States. All labeling was done manually by
three human annotators. Both datasets are separated into 72%/8%/20%
train/validation/test sets.

The data exhibits extreme diversity in uncontrolled conditions, including:

  * •

Heavy mud occlusion - Racers accumulate significant mud spatters and caking
throughout the duration of each race. This represents a unique occlusion
pattern not present in existing re-id datasets.

  * •

Complex poses—Racers exhibit varied poses including leaning, jumping, crashes,
and more that are uncommon during regular walking.

  * •

Distance and resolution—Images captured from a distance with small, low-
resolution racer crops.

  * •

Dynamic lighting—Outdoor conditions cause glare, shadows, and varying levels
of exposure.

  * •

Clothing—Jerseys and numbers that could ease re-id are often obscured by mud,
gear, and positioning.

  * •

Motion blur—Racers maneuver at high speeds causing motion-blurring effects.

Given the challenges associated with this dataset, one may perceive the proper
re-identification of racers to be nearly impossible even for humans, less for
any vision system. However, there are several consistent features even among
the most extreme mud. Some of those features include the helmet shape,
hydration system shape, the shape of the handguards, boot shape, and other
similar characteristics.

###  2.1. RnD Dataset

The off-road motorcycle Racer number Dataset (RnD) is comprised of 2,411
images. Each image depicts motorcycle racers engaged in competitive events,
with visible racer numbers on themselves and their motorcycles, labeled with
bounding box annotations and text transcriptions.

Racers can have anywhere from one to as many as 20 numbers located on their
body and motorcycle. The common locations for a number include the front and
sides of the motorcycle, on the cheeks of the racer’s helmet, and on the back
of the racers jersey. However, in rare cases, numbers can also be seen on the
wheels and handguards. The numbers on a single racer and vehicle do not need
to all be the same number. Commonly, the numbers on the helmet do not match
the numbers on the motorcycle, and the number on the front of the motorcycle
does not need to match the number on the side. It is also common for numbers
to only be present on the racer, but not on the motorcycle. Figure 2
highlights some of these examples.

In RnD, there is a total of 5,578 racer number annotations. The numbers can
span from 1 to 5 characters in length, optionally including alphabetical
characters (e.g., adding a letter to the end of a number is a common modifier
- for convenience, we still refer to all of these as _numbers_). 6% of the
dataset includes numbers that have alphabetical characters in addition to the
numerics. The train/validation/test split was done randomly. Only the racer
numbers were annotated (instead of all visible text) by one of the authors.
All visible racer numbers were tightly bound by a polygon (i.e. the bounding
box), and each polygon is tagged with the characters contained within (i.e.
the number). If a character was ambiguous or unclear, it was labeled with a
“#” symbol. Only the humanly identifiable text was transcribed. Racer numbers
that were fully occluded or too blurry to discern were not annotated.

The annotation task was restricted to only use the context of each individual
bounded region. The full image context could not be used to infer ambiguous
numbers based on other instances of that racer’s number elsewhere on the
motorcycle. This stimulates the local context available to optical character
recognition models. Figure 2 depicts a few characteristic images of this
dataset. Glare (8%), shadows (7%), blur (3%), dust (2%), and mud (44%) make
for a very challenging task. Several other factors even further complicate the
text-spotting task, such as numbers placed on top of other numbers and odd
orientations from crashes.

###  2.2. MUDD: Muddy Racer Re-Identification Dataset

MUDD contains 3906 images capturing 150 identities across 10 different off-
road events from the Grand National Cross Country (GNCC) racing series. We
used YOLOX (Ge et al., 2021) to detect the bounding boxes for people, which
were then manually labeled by our annotators. Importantly, there is no
identity overlap between the train/validation/test sets. Furthermore, both the
validation and test sets are further broken into a _query_ and _gallery_ set.
Based on the query set, the gallery images are ranked based on similarity for
evaluation. Data was only used from 10 events because we only matched the
identities of racers within events. It is more difficult to tell apart two
people from a single race than two people from a different race, solely
because of geographical hints in the background.

In labeling this re-identification dataset, we created an initial clustering
using off-the-shelf re-id (Zhou et al., 2019) and text-spotting models (Krylov
et al., 2021). First by detecting numbers and people, then for each
overlapping person and number, ranking every other person based on their
similarity. Our human annotators then manually review and create ground truth
clusters, aided by this initial ranking. This is discussed in more detail in
Appendix B. Just as with RnD, this dataset contains all of the difficulties of
off-road racing like mud, dust, glare, blur, etc.

##  3\. Experiments

We conducted experiments to benchmark the performance of both state-of-the-art
off-the-shelf methods, as well as their fine-tuned counterparts on both person
search (i.e. the re-id task) and text spotting in this motorcycle racing
setting. We also leverage the attributes of each image to analyze how mud,
glare, shadow, dust, and blur affect performance. All experimentation was done
on four NVIDIA Tesla V100 GPUs. Hyperparameter searching was performed for
each experiment using the training and validation sets. Then the best-
performing hyperparameters were used to train a final model on the combined
training and validation set, for evaluation on the test set. Further details
on the exact configuration of the hyperparameter search can be found in
Appendix LABEL:.

###  3.1. Spotting Racer Numbers

Our experiments leverage two state-of-the-art scene text spotting models:

  * •

YAMTS: Yet Another Mask Text Spotter (Krylov et al., 2021)

YAMTS is a Mask R-CNN-based model with an additional recognition head for end-
to-end scene text spotting. A ResNet-50 (He et al., 2016) is used for text
detection, with a convolutional text encoder and a GRU decoder.

  * •

SwinTS: Swin Text Spotter (Huang et al., 2022)

The Swin Text Spotter is an end-to-end Transformer-based model that improves
detection and recognition synergy through a recognition conversion module. A
feature pyramid network is used to decrease the sensitivity to text size, and
the recognition conversion model enables joint optimization of the detection
and recognition losses.

For both models, we first benchmark their performance on the RnD test set
using their published pre-trained weights. Afterwards, we fine-tune these
models further on the RnD training set and evaluate their performance again.
The hyperparameter search, detailed in Appendix A, revealed the best
parameters to be a cosine annealing learning rate schedule with a warm-up,
using a total batch size of 32, with the random scaling and rotation data
augmentations. The learning rate starts at 1e-6 and is then raised to 1e-3
after 1,000 iterations, and then annealed back down to 1e-6 over the remainder
of training. These hyperparameters were used to fine-tune the models over 150
epochs. The fine-tuned models are evaluated on the RnD test set.

####  3.1.1. Text Spotting Evaluation Metrics

Following the standard evaluation protocol (Huang et al., 2022; Ye et al.,
2023), we report results for both the text detection and end-to-end
recognition tasks. For detection, we compute precision, recall, and F1-score,
which we denote Det-P, Det-R, and Det-F1 respectively. A predicted box was
considered a true positive if it overlapped with a ground truth box by at
least 50% intersection over the union. For end-to-end recognition, we report
precision, recall, and F1-score at the sequence level, and we likewise denote
these metrics as E2E-P, E2E-R, and E2E-F1. A predicted text sequence was
considered correct only if it exactly matched the ground truth transcription
for the corresponding ground truth box.

###  3.2. Re-Identifying Racers

We evaluated the performance of two models on MUDD in three settings:

  * •

Off-the-shelf: Pre-trained state-of-the-art re-id models applied directly to
MUDD.

  * •

Random Initialization: Models trained from random initialization only on MUDD.

  * •

Transfer: Person re-identification pre-trained models fine-tuned on MUDD.

The models used were strong open-source implementations of CNN-based
architectures, specifically OSNet (Zhou et al., 2019) and ResNet50 (He et al.,
2016). The hyperparameter search detailed in Appendix A revealed the best
parameters to be the Adam optimizer using the triplet loss, with data
augmentation of random flips, color jitter, and random crops. A cosine
learning rate schedule was used with a maximum learning rate of $0.0003$, over
100 epochs. The mean and standard deviation for each experiment are reported
over three random seeds.

####  3.2.1. ReID Evaluation Metrics

We use the standard re-id metrics: cumulative matching characteristic (CMC)
rank-1, rank-5, rank-10 and mean Average Precision (mAP). CMC measures
rank-$k$ accuracy, the probability of the true match appearing in the top $k$.
The mAP metric computes mean average precision across all queries. Both
operate directly on the re-id model output.

##  4\. Results and Discussion

###  4.1. Text Spotting

Table 1. The RnD text detection and recognition performance on the RnD test set using off-the-shelf (OTS) and fine-tuned (FT) models.  Model | Det-P | Det-R | Det-F1 | E2E-P | E2E-R | E2E-F1
---|---|---|---|---|---|---
OTS | SwinTS | 0.195 | 0.287 | 0.232 | 0.101 | 0.148 | 0.120
YAMTS | 0.192 | 0.491 | 0.276 | 0.106 | 0.244 | 0.148
FT | SwinTS | 0.810 | 0.673 | 0.734 | 0.513 | 0.415 | 0.459
YAMTS | 0.847 | 0.715 | 0.775 | 0.758 | 0.404 | 0.527

Table 1 summarizes the quantitative results on the RnD test set. The off-the-
shelf SwinTS and YAMTS models, which were pretrained on large generic OCR
datasets, achieve poor accuracy. This is expected due to the substantial
domain gap between existing datasets and this new motorsports application.
Even state-of-the-art models fail without adaptation to racer numbers. Fine-
tuning the pretrained models on RnD led to major improvements. SwinTS achieved
0.734 detection F1 and 0.459 end-to-end recognition F1 after fine-tuning. For
YAMTS, fine-tuning improved to 0.775 detection and 0.527 recognition F1
scores. However, these fine-tuned results still leave substantial room for
improvement.

####  4.1.1. Text Spotting Performance Among Occlusion

Table 2. The RnD Performance broken down by occlusion type. Occlusion (%) | Det-P | Det-R | Det-F1 | E2E-P | E2E-R | E2E-F1
---|---|---|---|---|---|---
None (41%) | OTS | 0.196 | 0.568 | 0.291 | 0.124 | 0.330 | 0.180
FT | 0.880 | 0.726 | 0.795 | 0.826 | 0.470 | 0.599
Blur (3%) | OTS | 0.231 | 0.545 | 0.324 | 0.140 | 0.295 | 0.190
FT | 0.860 | 0.841 | 0.851 | 0.750 | 0.409 | 0.529
Shadow (7%) | OTS | 0.144 | 0.536 | 0.227 | 0.033 | 0.107 | 0.050
FT | 0.875 | 0.778 | 0.824 | 0.769 | 0.370 | 0.500
Mud (44%) | OTS | 0.194 | 0.389 | 0.259 | 0.086 | 0.152 | 0.110
FT | 0.811 | 0.718 | 0.761 | 0.681 | 0.359 | 0.470
Glare (8%) | OTS | 0.162 | 0.547 | 0.250 | 0.052 | 0.156 | 0.078
FT | 0.787 | 0.686 | 0.733 | 0.519 | 0.200 | 0.289
Dust (2%) | OTS | 0.173 | 0.310 | 0.222 | 0.113 | 0.190 | 0.142
FT | 0.925 | 0.638 | 0.755 | 0.833 | 0.259 | 0.395

(a)

(b)

(c)

(d)

(e)

Figure 3. Analysis of model performance on mud occluded numbers. (a) The model
correctly recognizes the front number by ignoring mud. (b) The quad number is
recognized but the muddy helmet number is missed. (c) The front number is read
but a very muddy helmet number is missed. (d) The number is detected but
misrecognized due to its odd position. (e) Two numbers are correctly read but
the muddy side number is missed.

We further analyzed model performance on the RnD test set when numbers were
occluded by different factors. Note that a single image can contain multiple
occlusions (i.e. it can be dusty and have glare, or it can be blurry and
muddy, etc.). Table 2 breaks down the detection and recognition results on
images with no occlusion, motion blur, shadows, mud, glare, and dust.

Mud occlusion was the most prevalent, accounting for 44% of the test data.
Both off-the-shelf and fine-tuned models struggled with heavy mud. The fine-
tuned model improved over the off-the-shelf version, achieving 0.761 detection
F1 and 0.470 recognition F1 on muddy images. But this remains far below the
0.795 detection and 0.599 recognition scores attained on non-occluded data.
There is substantial room to improve robustness to real-world mud and dirt
occlusion.

The fine-tuned model also struggled with glare occlusion, scoring just 0.733
detection F1 and 0.289 recognition F1 on such images. Glare creates low-
contrast regions that likely hurt feature extraction. Shadows likewise proved
challenging, with a 0.824 detection but only a 0.500 recognition F1 score
after fine-tuning. The changing lighting and hues may degrade recognition.

For motion blur, the fine-tuned model achieved 0.851 detection F1 but 0.529
recognition F1. Blurring degrades the crispness of text features needed for
accurate recognition. Surprisingly, the model performed worst on dust
occlusion, despite it being visually less severe than mud and glare. This
highlights the brittleness of vision models to unusual textures.

Overall, the breakdown reveals mud as the primary challenge, but substantial
room remains to improve OCR accuracy under real-world conditions like shadows,
dust, blur, and glare. Researchers should prioritize occlusions seen in
natural operating environments that undermine off-the-shelf models.

####  4.1.2. Text Spotting Qualitative Analysis

In this subsection, we provide a qualitative analysis of model performance on
RnD using the fine-tuned YAMTS model, which achieved the highest end-to-end F1
score. The detection confidence threshold was set to 0.65 and the recognition
threshold is set to 0.45. Figures 3 and 4 showcase successes and failures on
challenging examples. Photos from the beginning of a race are typically the
most complex, due to the number of motorcycles in a single image and
background clutter. Figure 4 looks at a photo from the start of a race, but
this time in rainy conditions. The top photo highlights the detection’s of the
off-the-shelf model before fine-tuning, where it can recognize only a single
number properly. However, after fine-tuning, the model can properly recognize
5 of the 6 visible numbers.

Figure 4. Example showcasing off-the-shelf (top image) vs fine-tuned (bottom
image) model predictions in rainy conditions.

Figure 4 compares the text spotting performance before and after fine-tuning
on a photo of the start of a muddy race. The fine-tuned model properly detects
all 8 visible numbers, demonstrating capabilities to handle partial mud
occlusion. However, it only correctly recognizes 3 of the 8 numbers,
highlighting limitations in recognizing degraded text. Without fine-tuning,
only 1 number is detected, and no numbers are properly recognized, showing the
benefits of fine-tuning. But substantial challenges remain in muddy
conditions.

Figure 3 showcases common mud-related successes and failures. In some cases,
the fine-tuned models can see through mud occlusions to properly recognize the
racer number, as shown in Figure 3(a). However, mud often prevents smaller
helmet numbers from being recognized (Fig 3(b), 3(c)). Odd orientations also
confuse models (Fig 3(d)). Overall, heavy mud occlusion remains the biggest
challenge. Some other failure types, such as overlapping numbers, are
discussed in Appendix C. These results show promising capabilities but also
expose key areas for improvement.

###  4.2. Re-ID Results

Table 3. MUDD re-id benchmark results comparing off-the-shelf, from scratch, and fine-tuning training strategies. Fine-tuning provides major accuracy gains indicating the importance of transfer learning. Training | Backbone | R1 | R5 | R10 | mAP
---|---|---|---|---|---
Best off-the-shelf | OSNet | 0.3252 | 0.5219 | 0.6327 | 0.3853
ResNet-50 | 0.3164 | 0.5099 | 0.6306 | 0.3634
Trained From Scratch | OSNet | 0.2146 (0.03108) | 0.4846 (0.04466) | 0.6755 (0.03606) | 0.2491 (0.0163)
ResNet-50 | 0.1591 (0.01154) | 0.4155 (0.01564) | 0.6194 (0.2826) | 0.1923 (0.01853)
Pretrained on Imagenet | OSNet | 0.7844 (0.01284) | 0.9416 (0.005594) | 0.9771 (0.004829) | 0.8215 (0.01258)
ResNet-50 | 0.762 (0.00817) | 0.9442 (0.004079) | 0.9787 (0.002729) | 0.8073 (0.006272)
Pretrained on MSMT17 | OSNet | 0.7924 (0.009929) | 0.9445 (0.001521) | 0.9779 (0.002051) | 0.8287 (0.005843)
ResNet-50 | 0.7596 (0.02279) | 0.9407 (0.0118) | 0.9767 (0.006813) | 0.8028 (0.02378)
Pretrained on DukeMTMC | OSNet | 0.7887 (0.01515) | 0.9388 (0.003319) | 97.57 (0.004367) | 0.826 (0.0117)
ResNet-50 | 0.7858 (0.01726) | 0.9562 (0.007937) | 0.9847 (0.002225) | 0.8277 (0.01079)
Pretrained on Market-1501 | OSNet | 0.7931 (0.01738) | 0.9442 (0.006051) | 0.9778 (0.00483) | 0.827 (0.01546)
ResNet-50 | 0.7812 (0.02546) | 0.9475 (0.01395) | 0.9807 (0.008821) | 0.8233 (0.02077)

Table 3 summarizes re-id performance on MUDD. Applying pre-trained re-id
models directly to MUDD leads to very poor accuracy. The highest Rank-1 is
only 32.52% using OSNet pre-trained on Market-1501. Again, this is expected as
there is a significant domain gap between existing re-id datasets and MUDD’s
challenging conditions. Yet training from scratch proves even more difficult,
yielding only 21.46% rank-1 accuracy. Fine-tuning pre-trained models on MUDD
proves the most effective strategy. Fine-tuned OSNet reaches 79.31% Rank-1,
over 2.5x higher than the best off-the-shelf model.

Interestingly, models pre-trained on generic ImageNet data perform nearly as
well as those pre-trained on re-id specific datasets, like Market-1501, after
fine-tuning. This is likely because MUDD represents a significant domain shift
even from existing re-id datasets, yet we are still able to glean benefits
from generic pretraining. In terms of architecture, we find that the OSNet-
based models achieve slightly higher rank-1 accuracy (79.31%) versus the more
traditional ResNet-50 (78.12%). While OSNet is a specialized architecture for
person re-identification, ResNet’s more general application still performs
competitively. Overall, both architectures can adapt to MUDD’s domain when
fine-tuned, with OSNet’s inductive biases providing a small boost.

We also were interested in determining the right pretraining corpora to get
the best-performing MUDD model. We compared the models pretrained on one of
the re-id datasets of MSMT17 (Wei et al., 2018), DukeMTMC (Ristani et al.,
2016), or Market-1501 (Zheng et al., 2015), which are all then further fine-
tuned on MUDD. The performance of these models is nearly the same across
different source datasets. They all substantially improve over off-the-shelf
and from scratch approaches, but don’t present any particular benefit over one
another.

####  4.2.1. Analysis

Table 4. The performance of the best ReID method when controlling the query and gallery set for muddy images. “No Mud ->Mud” corresponds to the query set containing only clean images, and the gallery set containing only muddy images. Query Set -> Gallery Set | Rank1 | Rank5 | Rank10 | mAP
---|---|---|---|---
No Mud -> No Mud | 0.8852 | 0.9727 | 0.9968 | 0.8809
Mud -> Mud | 0.5624 | 0.7162 | 0.7349 | 0.6002
No Mud -> Mud | 0.7342 | 0.8641 | 0.8543 | 0.6916
Mud -> No Mud | 0.7335 | 0.8267 | 0.8333 | 0.7690
Figure 5. Example of successful re-id by the fine-tuned model under light mud
occlusion. All top 10 ranked results correctly match the query rider despite
the mud, blurring, lighting, pose, and complex backgrounds. Green boundaries
signify correct matches and red incorrect. Figure 6. Failure case with heavy
mud occlusion on the query image. Only 1 out of the top 10 results is a
correct match, despite over 20 images of the same rider appearing in the
gallery set, most of which are clean. Green boundaries signify correct matches
and red incorrect.

Our fine-tuned models demonstrate significant improvements in re-identifying
riders compared to off-the-shelf and random initialization from scratch
approaches. As seen in Figure 5, the model can correctly match identities even
with significant mud occlusion. This indicates that fine-tuning successfully
incorporates invariances to mud while still distinguishing small inter-class
differences like the shape of a rider’s gear. However, challenges remain under
more extreme conditions. In the rest of this section, we analyze several key
factors that still cause fine-tuned model failures on MUDD.

As expected, heavy mud occlusion poses the most significant challenges. And
while we do see significant progress and capabilities on this front, there is
more room for improvement. Mud induces high intra-class variation as the
amount of mud covering a rider can vary drastically across images. It also
causes low inter-class variation since mud occludes distinguishing features
like jersey numbers and colors. As shown in Figure 6, querying with a muddy
image typically retrieves other muddy images rather than cleaner images of the
same identity.

Appendix D includes more failure cases and examples. In summary, we see that
changes in the natural appearance of a rider over a race also confuse models.
Riders may change gear like goggles or gloves multiple times. Crashes can rip
clothing and jerseys. The model must learn to link different levels of mud,
gear, and damage to a rider. Complex poses like jumps, crashes, and wheelies
are difficult to match, especially combined with mud and appearance variation.

Table 4 breaks down the performance of the best fine-tuned re-id model by
controlling the query and gallery set for muddy and clean images. We see that
by looking only at the clean imagery (i.e. no mud), we get a much better
performing model, with gains around 10%. On the other hand, when we evaluate
using only the muddy imagery, we see drops in performance of around 20% across
the board. Lastly, when a clean or muddy image is used as the query point, and
the opposite is used for the gallery set, performance falls between the
evaluation using only muddy or only clean images. The accuracy is a bit higher
for matching muddy query images to clean gallery images versus the reverse.

##  5\. Related Work

###  5.1. Person Re-Identification

Person re-identification (re-id) aims to match people across non-overlapping
camera views and time horizons (Ye et al., 2021; Zheng et al., 2016, 2015;
Farenzena et al., 2010). Early re-id methods relied on handcrafted features
like color histograms, textures, and local descriptors (Farenzena et al.,
2010). With the rise of deep learning, Convolutional Neural Network (CNN)
(Zhou et al., 2019) and Transformer (He et al., 2021) based approaches now
dominate re-id research, spurred by datasets like Market-1501 (Zheng et al.,
2015), DukeMTMC-ReID (Ristani et al., 2016), and MSMT17 (Wei et al., 2018).

A few datasets address environmental factors. For example, Xiao et al. (2016)
introduce a dataset with a low-resolution challenge set. Occlusions have also
been well studied, spearheaded by datasets with high levels of occlusion
(Schwartz and Davis, 2009; Wang et al., 2011b, 2016; Figueira et al., 2015;
Xiao et al., 2016). However, these occlusions are unrelated to the heavy mud
occlusion in our dataset. The addition of mud drastically complicates re-
identification. Furthermore, no prior datasets exhibit such a complex
combination of lighting, diversity, motion, and diverse cameras as our off-
road racing dataset.

Prior work has focused on the re-identification of motorcycles and bicycles
(Figueiredo et al., 2021; Li and Liu, 2022; Yuan et al., 2018), however, these
are restricted to street vehicles in urban settings. A highly related domain
is identifying athletes in sports imagery. Penate-Sanchez et al. (2020)
released a dataset of ultra-runners competing in a 128km race over the course
of a day and a night. While this is more similar to the off-road setting in
our dataset, they only have 416 different identities between 5 locations at a
single event. Furthermore, there is near zero mud in the dataset. Along
similar lines, but in even more controlled and limited settings, are the
SoccarNet-ReID (Giancola et al., 2022) and DeepSportRadar-ReID (Van Zandycke
et al., 2022) datasets, which contain images from broadcast video of soccer
and basketball games respectively.

These datasets have driven research to develop methods to deal with the
occlusions common in them. Approaches such as invariant representations (Chen
et al., 2019b), metric learning (Yi et al., 2014), semantic attributes (Shi et
al., 2015), part-based (Cheng et al., 2016) and pose-aware models (Cho and
Yoon, 2016), and adversarial learning (Huang et al., 2018) have been proposed
to alleviate occlusion problems. Other methods have been developed to handle
misalignment, utilizes temporal cues in video (Li et al., 2019), use domain
adaptation techniques (Deng et al., 2018), or unsupervised methods (Fan et
al., 2018) to reduce label dependencies. Unlike our dataset, these all operate
in controlled conditions. Existing models thus fail on our data.

In summary, re-id research has focused on controlled conditions and modest
variation. Our dataset introduces real-world challenges absent in existing
datasets. Our experiments expose clear gaps between current methods and this
application. MUDD provides diverse imagery to spur new techniques for robust
re-id under uncontrolled conditions.

###  5.2. Text Spotting

Text detection and recognition in images is a classic computer vision task.
Early traditional methods relied on sliding windows, connected components, and
handcrafted features like HOG (Wang and Belongie, 2010). With the advent of
deep learning, convolutional and recurrent neural networks now dominate scene
text recognition pipelines (Chen et al., 2021). Models leverage large
annotated datasets to learn powerful representations tuned for text detection
and recognition in a specific domain.

Many datasets and competitions have driven progress in general OCR. These
include ICDAR (Karatzas et al., 2013), COCO-Text (Lin et al., 2014), and
Street View Text (Wang et al., 2011a). Popular detection models build on
Region Proposal Networks and include CTPN (Tian et al., 2016), EAST (Zhou et
al., 2017), and Craft (Baek et al., 2019). Recognition is often achieved via
CNN + RNN architectures like CRNN (Wang et al., 2019) or transformer networks
like ASTER (Shi et al., 2019). More recent state-of-the-art methods utilize
pre-trained vision models like ViTSTR (Atienza, 2021), PARSeq (Bautista and
Atienza, 2022), CLIP4STR (Zhao et al., 2023), and DeepSolo (Ye et al., 2023).
However, most OCR research targets images of documents, signs, or web images.
While many of these works aim to go beyond structured settings (e.g.images of
documents, signs, or web images) and address the task of “robust reading”,
i.e. OCR in incidental or real scenes, recognizing text “in the wild” with few
assumptions remains an open challenge (Chen et al., 2021). Furthermore, domain
gaps exist where current methods fail on specialized applications. Our work
focuses on one such gap - recognizing racer numbers in motorsports.

A few prior works address detecting and recognizing the license plates on
vehicles (Ap et al., 2020; Laroca et al., 2018; Chen et al., 2019a; Lee et
al., 2019; Silva and Jung, 2018; Quang et al., 2022; Laroca et al., 2021).
Some have focused specifically on street motorcycle number plates (Kulkarni et
al., 2018; Sathe et al., 2022; Sanjana et al., 2021; Lee et al., 2004). All of
these efforts use data gathered from some form of street camera, which are
placed in strategic locations with recognizing license plates specifically in
mind. In contrast, our dataset is gathered from professional motorsport
photographers focused on capturing the most aesthetically pleasing photograph
of each racer. Furthermore, existing datasets have standardized plates which
differ greatly from the diverse layouts and occlusions of off-road motorcycle
numbers. Street motorcycle plates exhibit consistency in position and
appearance, unlike the numbers encountered during off-road competitions. The
conditions during races also introduce and exacerbate factors like motion
blur, mud occlusion, glare, and shaky cameras are not prevalent in street
imagery. RnD provides novel real-world imagery to push OCR capabilities.

The most relevant prior domain is recognizing runner bib numbers in marathon
images (Shivakumara et al., 2017; Ben-Ami et al., 2012; Boonsim, 2018; Kamlesh
et al., 2017). This shares similarities, but runner bibs provide more spatial
and appearance consistency than motorcycle racing numbers. Datasets like
TGCRBNW (Hernández-Carrascosa et al., 2021) exhibit some motion blur and night
racing, but do not contain the mud, vehicle occlusion and diversity of layouts
seen in motorsports.

Number recognition has also been studied in other sports - football (Yamamoto
et al., 2013; Bhargavi et al., 2022), soccer (Gerke et al., 2015, 2017; Šaric
et al., 2008; Diop et al., 2022; Alhejaily et al., 2023), basketball (Ahammed,
2018), track and field (Messelodi and Modena, 2013), and more (Liu and Bhanu,
2019; Nag et al., 2019; Vats et al., 2021; Wrońska et al., 2017). However,
most focus on jersey numbers in commercial broadcast footage rather than
track/field-side imagery. Existing sports datasets offer limited diversity and
size. To our knowledge, RnD represents the largest and most varied collection
of motorsports numbers in natural contexts.

In summary, prior work has made great progress in OCR for documents, signs,
and other domains, but real-world applications like recognizing racers in off-
road competitions remain extremely challenging due to domain gaps in current
data. RnD provides novel imagery to spur advances in OCR for motorsports. Our
benchmark experiments expose substantial room for improvement using this data.

##  6\. Conclusion

In this work, we introduce two datasets jointly referred to as Beyond the Mud.
These two datasets, RnD and MUDD, are the first datasets to benchmark the
performance of modern methods for text spotting and person re-identification
in off-road motorcycle races. These datasets capture challenging factors
including heavy mud occlusion, complex poses, variable lighting, occlusions,
complex backgrounds, monition blur, and more. All images were collected from
PerformancePhoto.co, and captured by 16 professional photographers across 50
distinct off-road competitions, distributed geographically throughout the USA.
To our knowledge, these datasets represent the largest, most varied collection
of annotated motorsports numbers and racers in unconstrained environments.

Benchmark results on both datasets highlight significant room for improvement.
This dataset is significantly different than typical, leading to extremely low
off-the-shelf (i.e. transfer learning) performance. The best off-the-shelf
text-spotting method achieved an end-to-end F1 score of only 15%, and the best
off-the-shelf person re-identification method reached only 33% rank-1
accuracy. When fine-tuning models to these datasets, the extremely high intra-
class variability, paired with the low inter-class variability makes training
particularly difficult. The best text-spotting model achieves an end-to-end F1
score of only 53%, while the best person re-identification model produces a
rank-1 accuracy of 79%.

Through both quantitative and qualitative analysis, we revealed some of the
primary factors degrading performance in both settings to be heavy mud
occlusion, glare, dust, and complex poses. Our work exposes the limitations of
current models and methods amongst extreme off-road conditions and provides
datasets of real-world imagery with baseline results. We hope the community
will build upon these initial experiments to make advances on the problem of
accurately reading text and matching racers in extreme environments.

## References

  * (1)
  * Ahammed (2018) Zubaer Ahammed. 2018\.  _Basketball player identification by jersey and number recognition_.  Ph. D. Dissertation. Brac University.
  * Alhejaily et al. (2023) Ragd Alhejaily, Rahaf Alhejaily, Mai Almdahrsh, Shareefah Alessa, and Saleh Albelwi. 2023\.  Automatic Team Assignment and Jersey Number Recognition in Football Videos.  _INTELLIGENT AUTOMATION AND SOFT COMPUTING_ 36, 3 (2023), 2669–2684.
  * Ap et al. (2020) N Palanivel Ap, T Vigneshwaran, M Sriv Arappradhan, and R Madhanraj. 2020.  Automatic number plate detection in vehicles using faster R-CNN. In _2020 International conference on system, computation, automation and networking (ICSCAN)_. IEEE, 1–6.
  * Appalaraju et al. (2021) Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. 2021.  DocFormer: End-to-End Transformer for Document Understanding. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 993–1003.
  * Atienza (2021) Rowel Atienza. 2021\.  Vision Transformer for Fast and Efficient Scene Text Recognition. In _Document Analysis and Recognition – ICDAR 2021_ , Josep Lladós, Daniel Lopresti, and Seiichi Uchida (Eds.). Springer International Publishing, Cham, 319–334.
  * Baek et al. (2019) Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. 2019.  Character region awareness for text detection. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 9365–9374.
  * Bautista and Atienza (2022) Darwin Bautista and Rowel Atienza. 2022.  Scene Text Recognition with Permuted Autoregressive Sequence Models. In _European Conference on Computer Vision_. Springer Nature Switzerland, Cham, 178–196.  https://doi.org/10.1007/978-3-031-19815-1_11
  * Ben-Ami et al. (2012) Idan Ben-Ami, Tali Basha, and Shai Avidan. 2012.  Racing Bib Numbers Recognition.. In _BMVC_. 1–10.
  * Bhargavi et al. (2022) Divya Bhargavi, Erika Pelaez Coyotl, and Sia Gholami. 2022\.  Knock, knock. Who’s there?–Identifying football player jersey numbers with synthetic data.  _arXiv preprint arXiv:2203.00734_ (2022).
  * Boonsim (2018) Noppakun Boonsim. 2018\.  Racing bib number localization on complex backgrounds.  _WSEAS Transactions on Systems and Control_ 13 (2018), 226–231.
  * Chen et al. (2019a) Rung-Ching Chen et al. 2019a.  Automatic License Plate Recognition via sliding-window darknet-YOLO deep learning.  _Image and Vision Computing_ 87 (2019), 47–56.
  * Chen et al. (2021) Xiaoxue Chen, Lianwen Jin, Yuanzhi Zhu, Canjie Luo, and Tianwei Wang. 2021.  Text Recognition in the Wild: A Survey.  _ACM Comput. Surv._ 54, 2, Article 42 (mar 2021), 35 pages.  https://doi.org/10.1145/3440756
  * Chen et al. (2019b) Yun-Chun Chen, Yu-Jhe Li, Xiaofei Du, and Yu-Chiang Frank Wang. 2019b.  Learning resolution-invariant deep representations for person re-identification. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 8215–8222.
  * Cheng et al. (2016) De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng. 2016.  Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In _Proceedings of the iEEE conference on computer vision and pattern recognition_. 1335–1344.
  * Ch’ng and Chan (2017) Chee Kheng Ch’ng and Chee Seng Chan. 2017.  Total-text: A comprehensive dataset for scene text detection and recognition. In _2017 14th IAPR international conference on document analysis and recognition (ICDAR)_ , Vol. 1. IEEE, 935–942.
  * Cho and Yoon (2016) Yeong-Jun Cho and Kuk-Jin Yoon. 2016.  Improving person re-identification via pose-aware multi-shot matching. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1354–1362.
  * Deng et al. (2018) Weijian Deng, Liang Zheng, Qixiang Ye, Guoliang Kang, Yi Yang, and Jianbin Jiao. 2018\.  Image-image domain adaptation with preserved self-similarity and domain-dissimilarity for person re-identification. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 994–1003.
  * Diop et al. (2022) Charles-Alexandre Diop, Baptiste Pelloux, Xinrui Yu, Won-Jae Yi, and Jafar Saniie. 2022\.  Soccer Player Recognition using Artificial Intelligence and Computer Vision. In _2022 IEEE International Conference on Electro Information Technology (eIT)_. IEEE, 477–481.
  * Fan et al. (2018) Hehe Fan, Liang Zheng, Chenggang Yan, and Yi Yang. 2018\.  Unsupervised person re-identification: Clustering and fine-tuning.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ 14, 4 (2018), 1–18.
  * Farenzena et al. (2010) Michela Farenzena, Loris Bazzani, Alessandro Perina, Vittorio Murino, and Marco Cristani. 2010\.  Person re-identification by symmetry-driven accumulation of local features. In _2010 IEEE computer society conference on computer vision and pattern recognition_. IEEE, 2360–2367.
  * Figueira et al. (2015) Dario Figueira, Matteo Taiana, Athira Nambiar, Jacinto Nascimento, and Alexandre Bernardino. 2015\.  The HDA+ data set for research on fully automated re-identification systems. In _Computer Vision-ECCV 2014 Workshops: Zurich, Switzerland, September 6-7 and 12, 2014, Proceedings, Part III 13_. Springer, 241–255.
  * Figueiredo et al. (2021) Augusto Figueiredo, Johnata Brayan, Renan Oliveira Reis, Raphael Prates, and William Robson Schwartz. 2021.  More: a large-scale motorcycle re-identification dataset. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 4034–4043.
  * Fujitake (2024) Masato Fujitake. 2024\.  Dtrocr: Decoder-only transformer for optical character recognition. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 8025–8035.
  * Ge et al. (2021) Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. 2021.  Yolox: Exceeding yolo series in 2021.  _arXiv preprint arXiv:2107.08430_ (2021).
  * Gerke et al. (2017) Sebastian Gerke, Antje Linnemann, and Karsten Müller. 2017\.  Soccer player recognition using spatial constellation features and jersey number recognition.  _Computer Vision and Image Understanding_ 159 (2017), 105–115.
  * Gerke et al. (2015) Sebastian Gerke, Karsten Muller, and Ralf Schafer. 2015\.  Soccer Jersey Number Recognition Using Convolutional Neural Networks. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops_.
  * Giancola et al. (2022) Silvio Giancola, Anthony Cioppa, Adrien Deliège, Floriane Magera, Vladimir Somers, Le Kang, Xin Zhou, Olivier Barnich, Christophe De Vleeschouwer, Alexandre Alahi, et al. 2022\.  SoccerNet 2022 challenges results. In _Proceedings of the 5th International ACM Workshop on Multimedia Content Analysis in Sports_. 75–86.
  * Gou et al. (2018) Mengran Gou, Ziyan Wu, Angels Rates-Borras, Octavia Camps, Richard J Radke, et al. 2018\.  A systematic evaluation and benchmark for person re-identification: Features, metrics, and datasets.  _IEEE transactions on pattern analysis and machine intelligence_ 41, 3 (2018), 523–536.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * He et al. (2021) Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. 2021\.  Transreid: Transformer-based object re-identification. In _Proceedings of the IEEE/CVF international conference on computer vision_. 15013–15022.
  * Hernández-Carrascosa et al. (2021) Pablo Hernández-Carrascosa, Adrian Penate-Sanchez, Javier Lorenzo-Navarro, David Freire-Obregón, and Modesto Castrillón-Santana. 2021\.  TGCRBNW: A Dataset for Runner Bib Number Detection (and Recognition) in the Wild. In _2020 25th International Conference on Pattern Recognition (ICPR)_. 9445–9451.  https://doi.org/10.1109/ICPR48806.2021.9412220
  * Huang et al. (2018) Houjing Huang, Dangwei Li, Zhang Zhang, Xiaotang Chen, and Kaiqi Huang. 2018.  Adversarially occluded samples for person re-identification. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 5098–5107.
  * Huang et al. (2022) Mingxin Huang, Yuliang Liu, Zhenghao Peng, Chongyu Liu, Dahua Lin, Shenggao Zhu, Nicholas Yuan, Kai Ding, and Lianwen Jin. 2022.  Swintextspotter: Scene text spotting via better synergy between text detection and text recognition. In _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 4593–4603.
  * Kamlesh et al. (2017) Kamlesh, Pei Xu, Yang Yang, and Yongchao Xu. 2017\.  Person re-identification with end-to-end scene text recognition. In _Computer Vision: Second CCF Chinese Conference, CCCV 2017, Tianjin, China, October 11–14, 2017, Proceedings, Part III_. Springer, 363–374.
  * Karatzas et al. (2013) Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. 2013\.  ICDAR 2013 robust reading competition. In _2013 12th international conference on document analysis and recognition_. IEEE, 1484–1493.
  * Krylov et al. (2021) Ilya Krylov, Sergei Nosov, and Vladislav Sovrasov. 2021\.  Open images v5 text annotation and yet another mask text spotter. In _Asian Conference on Machine Learning_. PMLR, 379–389.
  * Kulkarni et al. (2018) Yogiraj Kulkarni, Shubhangi Bodkhe, Amit Kamthe, and Archana Patil. 2018.  Automatic number plate recognition for motorcyclists riding without helmet. In _2018 International Conference on Current Trends towards Converging Technologies (ICCTCT)_. IEEE, 1–6.
  * Laroca et al. (2018) Rayson Laroca, Evair Severo, Luiz A Zanlorensi, Luiz S Oliveira, Gabriel Resende Gonçalves, William Robson Schwartz, and David Menotti. 2018.  A robust real-time automatic license plate recognition based on the YOLO detector. In _2018 international joint conference on neural networks (ijcnn)_. IEEE, 1–10.
  * Laroca et al. (2021) Rayson Laroca, Luiz A Zanlorensi, Gabriel R Gonçalves, Eduardo Todt, William Robson Schwartz, and David Menotti. 2021.  An efficient and layout-independent automatic license plate recognition system based on the YOLO detector.  _IET Intelligent Transport Systems_ 15, 4 (2021), 483–503.
  * Lee et al. (2004) Hsi-Jian Lee, Si-Yuan Chen, and Shen-Zheng Wang. 2004\.  Extraction and recognition of license plates of motorcycles and vehicles on highways. In _Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004._ , Vol. 4. IEEE, 356–359.
  * Lee et al. (2019) Younkwan Lee, Juhyun Lee, Hoyeon Ahn, and Moongu Jeon. 2019\.  SNIDER: Single noisy image denoising and rectification for improving license plate recognition. In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_. 0–0.
  * Li and Liu (2022) Jiaze Li and Bin Liu. 2022\.  Rider Re-identification Based on Pyramid Attention. In _Chinese Conference on Pattern Recognition and Computer Vision (PRCV)_. Springer, 81–93.
  * Li et al. (2019) Jianing Li, Jingdong Wang, Qi Tian, Wen Gao, and Shiliang Zhang. 2019.  Global-local temporal representations for video person re-identification. In _Proceedings of the IEEE/CVF international conference on computer vision_. 3958–3967.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 740–755.
  * Liu and Bhanu (2019) Hengyue Liu and Bir Bhanu. 2019.  Pose-guided R-CNN for jersey number recognition in sports. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 0–0.
  * Lyu et al. (2018) Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai. 2018.  Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes. In _Proceedings of the European conference on computer vision (ECCV)_. 67–83.
  * Messelodi and Modena (2013) Stefano Messelodi and Carla Maria Modena. 2013.  Scene text recognition and tracking to identify athletes in sport videos.  _Multimedia tools and applications_ 63, 2 (2013), 521–545.
  * Nag et al. (2019) Sauradip Nag, Raghavendra Ramachandra, Palaiahnakote Shivakumara, Umapada Pal, Tong Lu, and Mohan Kankanhalli. 2019.  CRNN based jersey-bib number/text recognition in sports and marathon images. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, 1149–1156.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. 2011\.  Reading digits in natural images with unsupervised feature learning.  (2011).
  * Penate-Sanchez et al. (2020) Adrian Penate-Sanchez, David Freire-Obregón, Adrián Lorenzo-Melián, Javier Lorenzo-Navarro, and Modesto Castrillón-Santana. 2020\.  TGC20ReId: A dataset for sport event re-identification in the wild.  _Pattern Recognition Letters_ 138 (2020), 355–361.  https://doi.org/10.1016/j.patrec.2020.08.003
  * Quang et al. (2022) Huy Che Quang, Tung Do Thanh, and Cuong Truong Van. 2022\.  Character Time-series Matching For Robust License Plate Recognition. In _2022 International Conference on Multimedia Analysis and Pattern Recognition (MAPR)_. IEEE, 1–6.
  * Ristani et al. (2016) Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. 2016.  Performance measures and a data set for multi-target, multi-camera tracking. In _European conference on computer vision_. Springer, 17–35.
  * Sanjana et al. (2021) S Sanjana, S Sanjana, VR Shriya, Gururaj Vaishnavi, and K Ashwini. 2021.  A review on various methodologies used for vehicle classification, helmet detection and number plate recognition.  _Evolutionary Intelligence_ 14, 2 (2021), 979–987.
  * Šaric et al. (2008) Matko Šaric, Hrvoje Dujmic, Vladan Papic, and Nikola Rožic. 2008.  Player number localization and recognition in soccer video using hsv color space and internal contours.  _International Journal of Electrical and Computer Engineering_ 2, 7 (2008), 1408–1412.
  * Sathe et al. (2022) Pushkar Sathe, Aditi Rao, Aditya Singh, Ritika Nair, and Abhilash Poojary. 2022.  Helmet Detection And Number Plate Recognition Using Deep Learning. In _2022 IEEE Region 10 Symposium (TENSYMP)_. IEEE, 1–6.
  * Schwartz and Davis (2009) William Robson Schwartz and Larry S Davis. 2009.  Learning discriminative appearance-based models using partial least squares. In _2009 XXII Brazilian symposium on computer graphics and image processing_. IEEE, 322–329.
  * Shashirangana et al. (2020) Jithmi Shashirangana, Heshan Padmasiri, Dulani Meedeniya, and Charith Perera. 2020.  Automated license plate recognition: a survey on methods and techniques.  _IEEE Access_ 9 (2020), 11203–11225.
  * Shi et al. (2019) Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. 2019\.  ASTER: An Attentional Scene Text Recognizer with Flexible Rectification.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 41, 9 (2019), 2035–2048.  https://doi.org/10.1109/TPAMI.2018.2848939
  * Shi et al. (2015) Zhiyuan Shi, Timothy M Hospedales, and Tao Xiang. 2015\.  Transferring a semantic representation for person re-identification and search. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4184–4193.
  * Shivakumara et al. (2017) Palaiahnakote Shivakumara, Ramachandra Raghavendra, Longfei Qin, Kiran B Raja, Tong Lu, and Umapada Pal. 2017.  A new multi-modal approach to bib number/text detection and recognition in Marathon images.  _pattern recognition_ 61 (2017), 479–491.
  * Silva and Jung (2018) Sergio Montazzolli Silva and Claudio Rosito Jung. 2018.  License plate detection and recognition in unconstrained scenarios. In _Proceedings of the European conference on computer vision (ECCV)_. 580–596.
  * Tian et al. (2016) Zhi Tian, Weilin Huang, Tong He, Pan He, and Yu Qiao. 2016.  Detecting text in natural image with connectionist text proposal network. In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14_. Springer, 56–72.
  * Van Zandycke et al. (2022) Gabriel Van Zandycke, Vladimir Somers, Maxime Istasse, Carlo Del Don, and Davide Zambrano. 2022\.  Deepsportradar-v1: Computer vision dataset for sports understanding with high quality annotations. In _Proceedings of the 5th International ACM Workshop on Multimedia Content Analysis in Sports_. 1–8.
  * Vats et al. (2021) Kanav Vats, Mehrnaz Fani, David A Clausi, and John Zelek. 2021\.  Multi-task learning for jersey number recognition in ice hockey. In _Proceedings of the 4th International Workshop on Multimedia Content Analysis in Sports_. 11–15.
  * Veit et al. (2016) Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. 2016.  Coco-text: Dataset and benchmark for text detection and recognition in natural images.  _arXiv preprint arXiv:1601.07140_ (2016).
  * Wang et al. (2011a) Kai Wang, Boris Babenko, and Serge Belongie. 2011a.  End-to-end scene text recognition. In _2011 International conference on computer vision_. IEEE, 1457–1464.
  * Wang and Belongie (2010) Kai Wang and Serge Belongie. 2010.  Word spotting in the wild. In _Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part I 11_. Springer, 591–604.
  * Wang et al. (2019) Ruishuang Wang, Zhao Li, Jian Cao, Tong Chen, and Lei Wang. 2019.  Convolutional Recurrent Neural Networks for Text Classification. In _2019 International Joint Conference on Neural Networks (IJCNN)_. 1–6.  https://doi.org/10.1109/IJCNN.2019.8852406
  * Wang et al. (2011b) Simi Wang, Michal Lewandowski, James Annesley, and James Orwell. 2011b.  Re-identification of pedestrians with variable occlusion and scale. In _2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops)_. IEEE, 1876–1882.
  * Wang et al. (2016) Taiqing Wang, Shaogang Gong, Xiatian Zhu, and Shengjin Wang. 2016\.  Person re-identification by discriminative selection in video ranking.  _IEEE transactions on pattern analysis and machine intelligence_ 38, 12 (2016), 2501–2514.
  * Wei et al. (2018) Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. 2018\.  Person transfer gan to bridge domain gap for person re-identification. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 79–88.
  * Wrońska et al. (2017) Ada Wrońska, Kacper Sarnacki, and Khalid Saeed. 2017\.  Athlete number detection on the basis of their face images. In _2017 International Conference on Biometrics and Kansei Engineering (ICBAKE)_. IEEE, 84–89.
  * Xiao et al. (2016) Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang. 2016.  End-to-end deep learning for person search.  _arXiv preprint arXiv:1604.01850_ 2, 2 (2016), 4\.
  * Yamamoto et al. (2013) Taiki Yamamoto, Hirokatsu Kataoka, Masaki Hayashi, Yoshimitsu Aoki, Kyoko Oshima, and Masamoto Tanabiki. 2013.  Multiple players tracking and identification using group detection and player number recognition in sports video. In _IECON 2013-39th Annual Conference of the IEEE Industrial Electronics Society_. IEEE, 2442–2446.
  * Ye et al. (2021) Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi. 2021\.  Deep learning for person re-identification: A survey and outlook.  _IEEE transactions on pattern analysis and machine intelligence_ 44, 6 (2021), 2872–2893.
  * Ye et al. (2023) Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Tongliang Liu, Bo Du, and Dacheng Tao. 2023.  DeepSolo: Let Transformer Decoder With Explicit Points Solo for Text Spotting. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 19348–19357.
  * Yi et al. (2014) Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. 2014\.  Deep metric learning for person re-identification. In _2014 22nd international conference on pattern recognition_. IEEE, 34–39.
  * Yuan et al. (2018) Yuan Yuan, Jian’an Zhang, and Qi Wang. 2018.  Bike-person re-identification: a benchmark and a comprehensive evaluation.  _IEEE Access_ 6 (2018), 56059–56068.
  * Zhao et al. (2023) Shuai Zhao, Xiaohan Wang, Linchao Zhu, and Yi Yang. 2023\.  CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model.  arXiv:2305.14014 [cs.CV]
  * Zheng et al. (2015) Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. 2015\.  Scalable person re-identification: A benchmark. In _Proceedings of the IEEE international conference on computer vision_. 1116–1124.
  * Zheng et al. (2016) Liang Zheng, Yi Yang, and Alexander G Hauptmann. 2016\.  Person re-identification: Past, present and future.  _arXiv preprint arXiv:1610.02984_ (2016).
  * Zhou et al. (2019) Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang. 2019.  Omni-scale feature learning for person re-identification. In _Proceedings of the IEEE/CVF international conference on computer vision_. 3702–3712.
  * Zhou et al. (2017) Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. 2017.  EAST: An Efficient and Accurate Scene Text Detector. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.

##  Appendix A Hyperparameter Searches

For all experiments in this paper, a grid search was used on the training and
validation set to select the best-performing hyperparameters after 50 epochs.
Then given the best hyperparameters, a final training run was performed on the
combined training and validation sets, and then the best reported metrics on
the test set were reported. The hyperparameter search ranges for the RnD
dataset are in Table 5 and in Table 6 for MUDD. The best-performing
hyperparameters are shown in bold.

Table 5. Hyperparmeter sweep ranges for the models trained on the RnD dataset. Parameter | Configuration
---|---
Learning Rate | [0.0001, 0.001 0.01]
Learning Rate Schedule | [linear, cosine, one-cycle]
Warm-Up Period | [0, 100, 1000, 2000]
Batch Size | [8, 16, 32]
Table 6. Hyperparmeter sweep ranges for the models trained on the MUDD dataset. Parameter | Configuration
---|---
Learning Rate | [0.0001, 0.0003, 0.001, 0.003, 0.01]
Learning Rate Schedule | [linear, cosine, one-cycle]
Data Augmentation |  | [random flip, color jitter,
---
random crop, resize,
random rotation]
Warm-Up Period | [0, 100, 1000, 2000]
Batch Size | [32, 64, 128, 256]

##  Appendix B Efficient ReID Labeling via Auxiliary Information

Figure 7. Leveraging detected jersey numbers as auxiliary information enables
generating higher quality identity clustering proposals for manual
verification. This proposed cluster contains both clean and muddy images of
the same rider, whereas proposing clusters with off-the-shelf re-id models
fail. Figure 8. Additional proposed results for the same identity cluster as
Figure 7. Our methodology provides high-quality recommendations to simplify
manual verification and labeling.

One key challenge in constructing re-id datasets is to efficiently group
images of the same identity during labeling. Exhaustively labeling identities
from scratch can become intractable for a large dataset of images with an
unknown number of identities. To assist in this labeling process, images can
be clustered into groups using pretrained models, and then manually verified
by annotators. However, in constructing MUDD, we found that annotators still
spent over 30 minutes on each identity, requiring a more efficient process.

Off-the-shelf re-id models focus on extracting features invariant to nuisance
factors like pose, lighting, and blurring while discriminating between
identities. However, these features are based on their pretraining dataset and
they cannot explicitly leverage domain-specific cues—especially if the domain-
specific cues are not available in the pretraining dataset, such as racer
numbers. The re-id model treats the image holistically without localizing and
recognizing semantic concepts like digits. Therefore, when the models are
applied on different image domains, any useful domain-specific cues are not
used.

In light of this challenge, we leverage the fact that each identity (i.e.
racer) in this dataset is assigned a visible number and we propose directly
utilizing this auxiliary information during the clustering and re-id process
via a pretrained text detection model (Lyu et al., 2018). This domain
knowledge provides strong localization cues to group images with the same
numbers. The re-id model alone struggles to consistently spot and match the
small-digit regions amidst mud, motion, and variations.

Explicitly guiding search and clustering with the auxiliary numbers, even if
noisy, complements the holistic re-id model. Our breadth-first attribute
search leverages the domain knowledge to effectively explore the data and
retrieve number matches. This creates high-quality initial clusters that seed
the depth-first re-id search.

In essence, we get the best of both worlds: domain-driven localization from
the auxiliary cues, combined with holistic identity discrimination from the
re-id model. The re-id model alone lacks the explicit semantic guidance,
resulting in poorer search and clustering. Our hybrid approach better utilizes
both domain knowledge and learned representations.

Specifically, to generate ground truth labels for specific racers, we first
extract all numbers using a pretrained text detection model (Lyu et al.,
2018), and also create a re-id embedding using a pretrained OSNet model. Then
we iterate over the following process:

  1. (1)

Pick a number that was detected more than 10 times and retrieve all images
containing it.

  2. (2)

For each result from Step 1, take the top $k$ nearest neighbors based on the
re-id embedding.

  3. (3)

Combine the results for each search by rank, and present to annotators for
manual refinement and verification.

This updated process reduced the average time to verify an identity cluster
from over 30 minutes to under 10.

Figure 7 shows a proposed cluster from our labeling system. The top section
contains all photos where the number 530 was detected. The bottom section
shows the most similar images according to the pertained OSNet re-id model.
Critically, leveraging the auxiliary number information provides an initial
cluster with clean and muddy images of the same racer that can be used as a
seed image for a search by the re-id model. Figure 8 shows additional results
deeper in the ranking.

##  Appendix C Additional Text Spotting Results

(a)

(b)

(c)

(d)

(e)

Figure 9. Analysis of common non-mud failures: (a) Incorrect side number
recognition. (b) Overlapping “stacked” numbers confuse the model. (c) A letter
is misrecognized as a number. (d) The letter portion of the racer number is
missed. (e) Complex graphics on quad confuse model.

Figure 10. Example showcasing the fine-tuned model learning to see through the
mud. The left image depicts the predictions from the off-the-shelf YAMTS model
before fine-tuning, which does not recognize any text. The right image
displays results from the fine-tuned YAMTS model, which can see through the
heavy mud occlusion and properly detect and recognize the racer number. This
demonstrates improved robustness to real-world mud occlusion after domain-
specific fine-tuning.

Figure 11. Example showcasing model successes and failures on a complex muddy
image. The top image shows detected text from the off-the-shelf YAMTS model
before fine-tuning, which recognizes only 1 number correctly (“251”). The
bottom image displays results from the fine-tuned YAMTS model, which detects
all 8 visible numbers but only correctly recognizes 3 of them. This highlights
the benefits of domain-specific fine-tuning, as the pre-trained model
struggles. However, even the fine-tuned model has difficulty accurately
recognizing highly degraded text, exposing substantial room for improvement.

Figure 3 showcases common mud-related successes and failures. In some cases,
the fine-tuned models can see through mud occlusions to properly recognize the
racer number, as shown in Figure 3(a). However, mud often prevents smaller
helmet numbers from being recognized (Fig 3(b), 3(c)). Odd orientations also
confuse models (Fig 3(d)). Overall, heavy mud occlusion remains the biggest
challenge. Figure 9 reveals other common failures like missing side numbers
(Fig 9(a)), overlapping numbers (Fig 9(b)), confusion between letters and
numbers (Fig 9(c)), missing letter portions (Fig 9(d)), and distractions from
graphics (Fig 9(e)). In summary, the analysis reveals promising capabilities
but also exposes key areas for improvement, particularly among extreme mud and
small text. Substantial opportunities remain to enhance OCR for this
challenging real-world application.

##  Appendix D Additional Reid results

Figure 12. Example of successful re-id by the fine-tuned model under moderate
mud occlusion. The 10 top retrievals correctly identify the query rider
despite the mud, pose, and other variations. Green boundaries signify correct
matches and red incorrect. Figure 13. Example of the model correctly matching
a clean image of a rider to a muddy image of the same rider when the pose is
similar between the query and gallery image. Green boundaries signify correct
matches and red incorrect. Figure 14. Example of a failure case due to extreme
pose variation in the query image. The rider is captured doing a wheelie,
leading to incorrect matches despite no mud occlusion. Green boundaries
signify correct matches and red incorrect. Figure 15. Failure case due to pose
variation between the query and gallery images. The backward-facing query
rider is not matched to forward-facing images of the same identity. Green
boundaries signify correct matches and red incorrect. Figure 16. Example
failure case due to two different riders having very similar jerseys and gear,
leading to confusion between their identities. Green boundaries signify
correct matches and red incorrect. Figure 17. Failure case due to low
resolution of the query image preventing distinguishing details from being
visible. The small, distant crop of the rider cannot be matched accurately.
Green boundaries signify correct matches and red incorrect.

As seen in Figure 14, a rider doing a wheelie is not matched to more standard
riding poses. Even common pose differences like front versus back views are
challenging (Figure 15). Images with small, distant crops of riders lack fine
details for discrimination. Figure 17 shows a failure case where the query is
low resolution. In some cases, different riders with very similar gear is
confused. This is common as racers supported by the same team will typically
purposefully coordinate their appearance. An example is shown in Figure 16.