Sketch2Diagram: Generating Vector Diagrams from Hand-Drawn Sketches
Under review as a conference paper at ICLR 2025

SKETCH2DIAGRAM: GENERATING VECTOR DIA-
GRAMS FROM HAND-DRAWN SKETCHES

Anonymous authors
Paper under double-blind review

ABSTRACT

We address the challenge of automatically generating high-quality vector dia-
grams from hand-drawn sketches. Vector diagrams are essential for communicat-
ing complex ideas across various fields, offering flexibility and scalability. While
recent research has progressed in generating diagrams from text descriptions, con-
verting hand-drawn sketches into vector diagrams remains largely unexplored, pri-
marily due to the lack of suitable datasets. To address this, we introduce SKETIkZ,
a dataset containing 3,231 pairs of hand-drawn sketches, reference diagrams,
and corresponding TikZ codes. Our evaluations highlight current limitations of
state-of-the-art vision and language models (VLMs), establishing SKETIkZ as a
key benchmark for future research in sketch-to-diagram conversion. Along with
SKETIkZ, we present IMGTIkZ, an image-to-TikZ model that integrates a 6.7B
parameter code-specialized open-source large language model (LLM) with a pre-
trained vision encoder. Despite its modest size, IMGTIkZ demonstrates perfor-
mance comparable to more extensive models such as GPT-4o. The model’s suc-
cess is largely driven by using our two data augmentation techniques and a multi-
candidate inference strategy. These findings provide promising avenues for future
research in sketch-to-diagram conversion and may have broader implications for
image-to-code generation tasks. SKETIkZ is publicly available.1

1

INTRODUCTION

Diagrams are powerful visual tools widely used in academia and various professional fields to con-
vey complex ideas. They are essential for clear communication and knowledge sharing by effectively
simplifying complex information. Vector graphics, in particular, are commonly used to create these
high-quality diagrams due to their scalability and precision. Their scalability and flexibility make
them particularly suitable for professional and academic contexts. These properties allow seamless
resizing and modification without losing quality, enabling efficient adaptation to various presenta-
tion formats and requirements. While established tools and languages such as TikZ and Graphviz
are popular for creating high-quality vector graphics, they often require significant manual effort
and specialized expertise. Recent advances in large language models (LLMs), such as GPT-4o, have
triggered a growing interest in automating the generation of vector graphic diagrams from textual
descriptions (Belouadi et al., 2023; Zala et al., 2023; Zou et al., 2024). This emerging research area
has significant potential to streamline the diagram creation process and make high-quality visualiza-
tions more accessible. Despite the significant advancements in text-to-code generation, generating
diagrams from sketches remains largely unexplored. Sketch-based input often provides a more intu-
itive and user-friendly way to express visual ideas (Figure 1). This approach leverages the inherent
human ability to quickly and effectively communicate complex visual information through simple
drawings. A primary reason for the limited research in this area is the lack of publicly available
datasets that pair hand-drawn sketches with their corresponding codes. Such datasets are essential
for training and evaluating models that translate sketch-based input into structured diagram code.
To address this gap, we introduce SKETIkZ, a new dataset designed for benchmarking sketch-to-
diagram generation. SKETIkZ comprises 3,231 pairs of hand-drawn sketches and corresponding
TikZ codes. The sketches were created using several tools commonly employed in real-world sce-
narios: paper, whiteboards, and tablets. This diverse collection provides a valuable resource for
advancing research in automated diagram generation from sketches. SKETIkZ aims to facilitate
the development of models capable of generating high-quality diagrams from hand-drawn inputs for
real-world applications. We also developed IMGTIkZ, a Vision-Language Model (VLM) specifically

1The dataset link is provided in the reproducibility statement section.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: Overview of sketch-to-diagram. We consider scenarios where users hand-drawn diagrams
that they want to create. Sketch-to-diagram models (e.g., VLM) take these sketches Is and pre-
defined instructions X, and generate code Yg for producing vector graphics. Yg is subsequently
rendered into generated image Ig. The process of text-to-diagram is also provided for comparison.

designed for this task. Our model combines three components: an open-source LLM specialized in
code generation, a vision encoder, and an adapter. This combination aims to create a model capable
of efficiently converting sketches into TikZ code. We confirmed the effectiveness of two strategies:
expanding our dataset using two data augmentation methods and employing an inference strategy
that generates multiple candidates and selects the best one. As a result, IMGTIkZ achieved perfor-
mance comparable to GPT-4o in subjective evaluations despite having a relatively small model size
of 6.7B parameters. However, both IMGTIkZ and the latest state-of-the-art models still struggle to
accurately generate code that captures all elements and layouts of sketches, indicating the potential
for further advances. We aim for our dataset and findings to drive future research and development
in this field. Our contributions are summarized as follows:

• We introduce SKETIkZ: A new dataset containing 3,231 pairs of hand-drawn sketches,
reference diagrams, and corresponding TikZ codes, addressing the lack of real-world data
for sketch-to-vector diagram conversion and serving as a benchmark for future research.

• We develop IMGTIkZ: A image-to-TikZ model that combines a 6.7B parameter code-
specialized LLM with a pre-trained vision encoder, achieving accuracy comparable to
larger models despite its modest size.

• We empirically demonstrate the effectiveness of two types of data augmentation and a

multi-candidate inference strategy.

2 RELATED WORK

Vision and language models Constructing VLMs that understand images and generate text has
become increasingly feasible with advancements in LLMs. A particularly effective approach in-
volves integrating vision encoders, such as CLIP (Radford et al., 2021), and LLMs using adapter
modules. This method has demonstrated promising results (Liu et al., 2023; Dai et al., 2023; Ye
et al., 2023; Zhu et al., 2023; Li et al., 2024; Wang et al., 2024), efficiently creating VLMs that
leverage the extensive knowledge base of pre-trained models. In this study, on the same line as these
approaches, we build a VLM to generate TikZ code from images.

Image to code generation While VLMs typically focus on generating natural language outputs,
such as answers to questions or image descriptions, research that produces code for rendering im-
ages, such as HTML, LaTeX, or SVG, has proven to be a valuable application. For instance, mod-
els have been developed to generate LaTeX code from screenshots of mathematical formulas or
handwritten images (Deng et al., 2016; Gervais et al., 2024), HTML code from web page screen-
shots (Soselia et al., 2023; Si et al., 2024; Laurenc¸on et al., 2024; Gui et al., 2024), and SVG code
from icon images (Rodriguez et al., 2023). While generating LaTeX and TikZ code are similar
in terms of code output, our research tackles significantly more complex problems than previous
formula-to-LaTeX conversion studies. It involves much longer output sequences (739 tokens on
average compared to 65 tokens in prior work) and requires an understanding of two-dimensional
layouts. We introduce three key advances to handle this increased complexity: code-specialized
VLM, two data augmentation strategies, and multi-candidate generation.

Diagram understanding Understanding diagrams has been an important and long-standing re-
search topic, including question answering (Kembhavi et al., 2016; Lu et al., 2023; Wang et al.),

2

figure.tex\begin{document}\begin{tikzpicrure}\node ......prompt.txtfigure.tex\begin{document}\begin{tikzpicrure}\node ......VLMLLMtext-to-diagramsketch-to-diagramGenerate TikZ code for a directed graph withthe following specifications: Five nodeslabeled 0, 1, 2, 3, and 4. Node 0 hasoutgoing red edges to ...... Nodes 0, 2, and4 have "text" written below them. ......Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Figure 3: Sketch Tool Usage Statistics.

Tool

Number

Proportion

Paper
Whiteboard
Tablet

All

2,545
346
340

3,231

78.8%
10.7%
10.5%

100%

Figure 2: Dataset construction process.

caption generation (Hsu et al., 2021; Singh et al., 2023; Huang et al., 2023), and gene rating de-
scriptions (Hu et al., 2023; Bhushan & Lee, 2022; Bhushan et al., 2024). Recent research proposed
benchmark datasets to assess not only the understanding of diagram images, but also the direct com-
prehension of vector graphics code (Zou et al., 2024; Qiu et al., 2024). This expanding research area
reflects the growing interest in understanding vector graphics diagrams.

Diagram generation Ellis et al. (2017) proposed a model generating TikZ code for primitive
geometric sketches, limiting their scope to three basic shapes (circles, rectangles, and lines) without
text. Our work differs significantly as we address real-world diagrams with unrestricted shapes and
text elements. Furthermore, our dataset reflects realistic environments by including sketch images
from various sources such as paper, whiteboards, and tablets. Regarding the model structure, while
they used a two-stage approach for code identification and generation, we developed an end-to-end
model to handle diverse, unrestricted shapes. Recently, there has been growing interest in generating
real-world diagrams from text inputs (Belouadi et al., 2023; Zala et al., 2023). For example, Belouadi
et al. (2023) proposed a method for generating TikZ code to render diagram images from caption
text. Generating diagrams through code synthesis provides better controllability and editability than
pixel-based image generation methods, while enabling LLM integration. Belouadi et al. (2023)
also highlights the challenge of image-to-diagram generation, which remains limited due to the
scarcity of paired image-code data. Concurrent work by (Belouadi et al., 2024) addresses the task
of generating diagrams from images, which is closely related to our task. However, their evaluation
of sketch-based generation is limited to a small dataset, which lacks corresponding TikZ code and
thus cannot be used for image-to-code training. Our dataset provides the largest and most diverse
sketch-to-diagram dataset with TikZ code, captured under real-world conditions. The dataset serves
as a valuable benchmark for evaluating model robustness on real sketches since hand-drawn data
cannot be collected in large quantities from the web. Beyond sketch-to-TikZ applications, it can
enable the development of more general models through multitask learning with other image-to-
code datasets. We also contribute novel data augmentation methods and multi-candidate generation
strategies, providing new insights for future research directions in this field. Another line of research
addresses the generation of CAD sketches (referring to parameters and constraints within CAD
systems rather than hand-drawn images) (Para et al., 2021; Seff et al., 2021; Ritchie et al., 2023).
These approaches are specifically designed for CAD generation, which differs from our focus.

3 DATASET AND TASK

3.1 TASK DEFINITION
We introduce a sketch-to-diagram task (Figure 1), where the input consists of a sketch image of
a diagram Is and a language instruction X, and the output is a sequence of TikZ code Yg. Then
generated TikZ code Yg are compiled to render the diagram image Ig.

3.2 DATASET CONSTRUCTION
We constructed our dataset in three steps: rendering, filtering, and sketch annotation (Figure 2).

Step1: Rendering diagrams from TikZ code We first rendered diagrams from TikZ code in the
DaTikZ (Belouadi et al., 2023) by using pdflatex. We then paired the rendered reference diagrams
Ir with the corresponding TikZ code Yr. We refer to the rendered diagrams as the reference images.

Step2: Diagram classification and filtering Diagrams can be classified into various categories,
as demonstrated by ACL-Fig (Karishma et al., 2023) with its 19-category dataset. For our sketch-to-
diagram task, we focused on diagrams composed of geometric shapes and arrows, excluding those
primarily based on numerical data. We specifically targeted diagrams categorized as Tree, Graph,

3

TikZcodeTikZcodeTikZcodeTikZcodeTikZcodeSketchImageSketchImageRenderedImageRenderedImageRenderedImageRenderedImageRenderedImageStep3. AnnotationStep2. FilteringPairing TikZ codes, sketch images, and rendered images.Step1. RenderingUnder review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Figure 4: Examples of sketch images. Left: paper, Center: whiteboard, Right: tablet.

Table 1: Datasets used for training IMGTIkZ.

No Name

Input

Output

Size

1
2
3
4
5
6
7
8

arXiv figure
arXiv figure
LLaVA-Pretrain2
SKETIkZ
RenderTikZ
AugTikZ
ImgAugTikZ
DaTikZ-v23

Figure or table image
Figure or table image
Multi-domain image

OCR text
Caption text
Caption text
Diagram sketch image TikZ source code
Diagram image TikZ source code
Diagram image TikZ source code
Noised Diagram image TikZ source code
Diagram image TikZ source code

1.2M
1.1M
558K
2.5K
155K
556K
714K
46K

Stage2

Stage1
✓
✓
✓

✓
✓
✓
✓
✓

Architecture Diagram, Neural Networks, and Venn Diagram according to ACL-Fig labels. We chose
these categories because sketch-to-diagram generation is particularly effective for visually oriented
diagrams. These diagrams often involve complex combinations of shapes and interconnections,
making manual creation time-consuming and precise linguistic instructions challenging. Using an
image classification model trained on the ACL-fig dataset (details in Appendix E), we extracted and
sampled 4,000 diagram images from our targeted categories for annotation. We present the detailed
breakdown of categories in Table 9 and Figure 9 in Appendix E.

Step3: Sketch data collection 28 annotators created sketch images Is based on filtered reference
images Ir. Annotators used black pens primarily, with red, blue, and green for colored elements,
excluding complex diagrams and ignoring color filling. They chose from paper, whiteboard, or tablet
tools. Table 3 shows the distribution of sketches by tool, with the paper being the most common.
Figure 4 illustrates examples from each tool. The dataset includes diverse sketches mimicking real-
world scenarios, with paper and whiteboard sketches showing varied lighting and backgrounds. We
aligned sketches Is with corresponding TikZ codes Yr and reference images Ir, creating a dataset of
2,585 training, 323 validation, and 323 test samples. More examples are shown in Appendix F

4

IMGTIkZ: VISION-LANGUAGE MODEL FOR IMAGE-TO-TIkZ GENERATION

4.1 MODEL STRUCTURE
We developed IMGTIkZ, a VLM specifically designed for this task using the model architecture of
LLaVA 1.5 (Liu et al., 2023). The model architecture comprises three key components: a code-
specialized LLM, a vision encoder, and an adapter, illustrated in Figure 5 (a). The model inputs a
diagram image and generates a corresponding TikZ code. While the original LLaVA1.5 employs a
language model for natural language generation, we replaced this component with a code-specific
LLM, a model specialized for code generation tasks. Specifically, we used the instruction-tuned
DeepSeek coder (Guo et al., 2024) of 6.7B size as the code-specific LLM and SigLIP model Zhai
et al. (2023) for our vision encoder. We employed the same architecture as LLaVA 1.5 for the
adapter module - a simple two-layer MLP. We trained our model in two stages: first updating
only the adapter parameters, then training both adapter and LoRA (Hu et al., 2021) parameters.
The language and vision model parameters remained frozen throughout training. For more detailed
information about the model hyperparameters, please refer to supplementary material B and Table 7.

4.2 TRAINING DATA

2https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain
3https://huggingface.co/datasets/nllg/datikz-v2

4

Under review as a conference paper at ICLR 2025

Figure 5: IMGTIkZ model structure (a) and multi-candidate generation process for inference (b).

Datasets used in stage 1 training In stage 1 training, we incorporated arXiv figure data (No.
1 and 2 in Table 1) in addition to LLaVA-pretrain data (No. 3). This arXiv figure dataset was
created by extracting figures, tables, and captions from arXiv paper PDFs in arXiv bulk dataset using
PDFFigure2.0.4 We also used Google Cloud Vision API5 to extract text from these images. The
arXiv data served two purposes: (1) generating OCR text from images to improve text recognition
and (2) generating captions from diagram images to enhance diagram image understanding.

Datasets used in stage 2 training
In the second stage of training, we focused on enhancing the
model’s ability to generate TikZ code. Given the limited size of the SKETIkZ dataset alone, we
supplemented our training data by creating paired rendered diagram images and TikZ code collected
from arXiv source file in bulk data, which is referred to as RenderTikZ (No. 5). We implemented
two data augmentation techniques to increase diagram and image variations further. The first method
involves generating TikZ code using GPT-3.5 to increase the variety of diagrams, referred to as
AugTikZ (No. 6). The second is an image augmentation technique designed to account for vari-
ous types of noise commonly found in sketches, such as background images, lighting conditions,
and rotation, which is referred to as ImgAugTikZ (No. 7). In addition to these, we also utilized
existing pairs of TikZ code and images (No. 8). The subsequent paragraphs will explain these two
augmentation methods in more detail.

Data augmentation for increasing diagram variations While we collected approximately 916K
original TikZ codes from arXiv sources, many failed to compile during RenderTikZ creation. We
used GPT-3.5 to fix these compilation errors with a prompt such as ’Please modify the code to make
it compilable.’ To increase diagram variety, we instructed GPT-3.5 to modify the original diagram
into a different diagram, producing altered versions of the original diagrams. These augmentation
techniques resulted in 556K AugTikZ data samples. Previous data augmentation for VLMs used
other VLMs to generate instruction-response pairs from images, which was costly due to image pro-
cessing. Instead, we generate data efficiently by modifying only TikZ code using text-based LLMs.
This approach could be applied to various image-to-code tasks. More details are in Appendix G.1.

Data augmentation for increasing image variations
Hand-drawn sketch diagrams inherently contain more image
noise compared to rendered images. This noise can appear as
background interference or lighting variations when captur-
ing sketches from paper or whiteboards. Furthermore, hand-
written text and lines often exhibit significant distortions,
and diagrams are frequently stored with angular rotations.
To address these issues, we applied multiple image aug-
mentation techniques to RenderTikZ and AugTikZ datasets,
such as synthesizing notebook backgrounds, adding Gaus-
sian noise, varying brightness and contrast, and introducing
distortion. Figure 6 illustrates an example of the augmented

4https://github.com/allenai/pdffigures2
5https://cloud.google.com/vision/docs?hl=en

5

Figure
6:
gAugTikZ
data.
image, bottom: augmented image.

Example
Top:

of
Im-
original

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

CodeLLM  Vision EncoderAdapterInstructionPlease generateTikZ code to drawthe diagram of thegiven image.ImageTikZ codeImgTikZTikZTikZTikZTikZDiagramDiagramDiagramDiagramDiagram1. Generate K candidates3. Select onecandidate2. RenderingSketchCalculate similarity using selector (a) ImgTikZ model structure(b) Multi-candidate generationLoRAfigure.tex\begin{document}Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

image. This augmentation approach is general-purpose and
can be applied to various sketch-to-diagram tasks. More details are in Appendix G.2.

INFERENCE

4.3
We implemented two inference methods: iterative generation and multi-candidate generation. In the
paper, we refer to them as IMGTIkZ-IG and IMGTIkZ-MCG, respectively.

Iterative genetaion Iterative generation produces one candidate per test sample, regenerating
upon compilation failure until success. We set a maximum number of generation attempts M to
limit this process. This method is straightforward and can be considered a baseline approach.

Multi-candidate generation Multi-candidate generation creates K candidates simultaneously, se-
lecting the best one (Figure. 5 (b)) using a selector model.
In our study, we generate multiple
TikZ codes and render them as images. The selector selects the best candidate by maximizing the
similarity between the input sketch image Is and the generated diagram image Ig. As general vision
encoders cannot accurately measure diagram similarity, we propose D-SigLIP (Diagram-Specialized
SigLIP) as our selector. D-SigLIP adds a trainable linear layer to a pre-trained SigLIP model, and
we fine-tune only this layer through contrastive learning with noise-augmented diagram pairs from
RenderTikZ and AugTikZ (Chen et al., 2020). More details are in Appendix C. To calculate the sim-
ilarity score, we utilized the vector obtained from the CLS token of the D-SigLIP and computed the
cosine similarity, as in CLIPScore (Radford et al., 2021). Our task requires generating lengthy code
sequences (averaging 739 tokens), making producing error-free code in a single-generation attempt
challenging. Furthermore, since the model training is based on next-token prediction loss for code
sequences, metrics related to image quality are not explicitly considered during code generation.
The multi-candidate generation and selection strategy allows us to evaluate these metrics after code
generation, which could not be considered during the training phase. While similar approaches have
been proposed for text inference and coding tasks (Brown et al., 2024), our work is the first to use
image similarity for candidate selection in image-to-diagram conversion.

5 EVALUATION METRICS

5.1 AUTOMATIC EVALUATION
We used four aspects of automatic evaluation: compilation success rate, image similarity, code
similarity, and character similarity.

Compilation success rate The compilation success rate (CSR) represents the percentage of gen-
erated TikZ codes that successfully compile into images. In this study, we employ two CSR metrics.
The first is the averaged CSR, which calculates the ratio of successful compilations Nsuccess to the
total number of generation attempts Ngen, expressed as CSRavg = Nsuccess
. This metric indicates how
Ngen
often a model succeeds in compilation on average. The second is the cumulative CSR, which repre-
sents the number of test samples that compiled successfully Ntest success out of the total number of test
samples Ntest, expressed as CSRcum = Ntest success
. This metric shows the proportion of test samples
that were correctly compiled. Detailed examples are provided in Appendix J.

Ntest

Image similarity We used cosine similarity between image embeddings to measure the similarity
between the generated image Ig and the reference diagram image Ir. We used our D-SigLIP (see
Sec. 4.3) for calculating image embeddings. This approach can be considered as a modification
of widely-used CLIPScore, where the CLIP model is replaced with D-SigLIP. We also calculated
CLIPScore using the original CLIP model; however, CLIPScore showed a lower correlation with
human evaluations compared to the similarity calculated using D-SigLIP. If the compilation failed,
we set the similarity score to 0.

Code similarity We used cosine similarity in the embedding space between Yg and Yr. We gen-
erated the code embeddings using OpenAI’s text embedding model.6

Character similarity The character similarity calculates the similarity between the text in the
generated image Ig and the text in the reference image Ir using Rouge-1 score (Lin, 2004). We used
the OCR included in the Google Cloud Vision API to extract text. This metric indicates how well
the model can read and generate text from the sketch.

6We used text-embedding-3-small version.

6

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

5.2 SUBJECTIVE EVALUATION

We conducted a subjective evaluation focusing on two key aspects: alignment and quality following
established practices in previous studies(Otani et al., 2023; Ku et al., 2023). In our study, alignment
measures the similarity between the generated and reference images, while quality assesses the
coherence and appropriate arrangement of elements within the generated diagram. We employed a
five-point scale for both metrics to ensure a nuanced evaluation.

Alignment Annotators assessed alignment by visually comparing the generated diagram image Ig
to the reference diagram image Ir. The sketch diagram image Is was also provided for evaluation.
A score of 1 indicated that the diagram’s elements were completely misaligned, while a score of
5 signified that they were almost perfectly aligned. To illustrate a score of 1, a randomly selected
rendered diagram image from the training dataset was displayed.

Quality Annotators assessed the quality of the generated diagram images independently of the
reference images, focusing on the structural clarity and arrangement of elements within the layout.
A score of 1 was assigned to diagrams with poorly arranged, overlapping elements that were nearly
unreadable. Conversely, a score of 5 was given to well-structured diagrams with logically arranged
shapes and text that closely resembled human-created diagrams. The scale reflects the overall layout
quality, ranging from incomprehensible to highly coherent visual representations.

Annotation We comprehensively evaluated each model’s outputs across the entire test set using
Amazon Mechanical Turk. The manual assessment involved 40 annotators, each evaluated by five
distinct annotators, ensuring coverage of all six models. Diagrams that failed to compile were au-
tomatically assigned the minimum score of 1 for alignment and quality metrics. We computed the
final score for each system and instance by averaging the three median evaluation scores, excluding
potential outliers. A detailed description is provided in Appendix H.

6 EXPERIMENTAL SETUP

Models for Comparison We evaluated several state-of-the-art models in our study.7 GPT-4o,
OpenAI’s most efficient multimodal model. We also included GPT-4o mini, their top small model.
From Anthropic, we employed Claude 3.5 Sonnet, the latest in their multimodal large language
model series. Lastly, we assessed LLaVA-Next, a popular open-source model.

Training parameters for IMGTIkZ We set the LoRA tuning parameters for training to r=128
and α=256. The stage 1 training was conducted with a batch size of 256 for 6,000 steps. Stage 2
training used a batch size of 128 for 1 epoch. We used 8 A100 GPUs for training IMGTIkZ, and 1
H100 GPU for inference. More details are in Appendix B.

Inference We applied iterative generation as the baseline for the four comparison models (see
Sec.6), while for IMGTIkZ, we implemented both iterative and multi-candidate generation. The
maximum number of attempts M for iterative sampling was set to 5, and the number of candidates
K for multi-candidate generation was set to 20. More details are in Appendix A.

7 RESULTS

7.1 MAIN RESULTS

Can models generate compilable TikZ code for diagrams? Table 2 presents the averaged CSR
IMGTIkZ significantly outperformed other models in averaged CSR. Other
results (CSR avg).
models showed relatively low CSR avg values (approximately 0.35-0.54), indicating insufficient
adaptation to TikZ data. Since averaged CSR directly impacts user convenience, achieving higher
scores is crucial. Figure 7 illustrates the progression of cumulative CSR across iterative generation
attempts. IMGTIkZ achieved nearly 100% success after five attempts for the test data, while other
methods leveled off at 0.8-0.9. This result indicates that 10-20% of samples remain uncompilable
even after five attempts with these models.

7We used the gpt-4o-2024-05-13 version for GPT-4o, the gpt-4o-mini-2024-07-18 ver-
the claude-3-5-sonnet-20240620 version for Claude 3.5, and the

sion for GPT-4o mini,
llama3-llava-next-8b version, which is trained on the 8B Llama 3 model, for LLaVA-Next.

7

Under review as a conference paper at ICLR 2025

Table 2: The results of the automatic (0-1) and subjective (1-5) evaluations. The best results are
highlighted in bold.

Model

ImageSim CodeSim CharSim CSR avg Alignment Quality

Automatic

Subjective

Closed models
GPT-4o
GPT-4o-mini
Claude 3.5 Sonnet

Open source models
LLaVA-Next
IMGTIkZ-IG (ours)
IMGTIkZ-MCG (ours)

0.695
0.595
0.753

0.315
0.734
0.821

0.821
0.814
0.813

0.727
0.815
0.822

0.611
0.514
0.671

0.206
0.503
0.594

0.479
0.376
0.544

0.350
0.767
0.799

3.00
2.39
3.32

1.43
2.78
3.13

3.20
2.71
3.54

1.93
2.92
3.30

Figure 7: Progression of cumulative compi-
lation success rate with varying number of
attempts in an iterative generation.

Figure 8: Progression of image similarity
with varying number of candidates in multi-
candidate generation

Can models generate diagram images close to the reference images?
ImageSim and Alignment
in Table 2 present the similarity between generated and reference images. Claude 3.5 achieved the
highest performance in Alignment, with IMGTIkZ-MCG following as the second-best. Conversely,
for ImageSim, IMGTIkZ-MCG performed best, while Claude 3.5 attained the second-highest score.
LLaVA-Next, with a comparable model size to IMGTIkZ but without TikZ data training, performed
poorly and rarely generated correct output. IMGTIkZ-MCG achieved comparable performance to
GPT-4o on the Alignment score despite being smaller, highlighting the effectiveness of our adap-
tation and multi-candidate generation inference process. Overall, even the best-performing Claude
3.5 model achieves an average Alignment score of only about 3.3. This indicates that the generated
diagrams, on average, only match about 50-60% of the correct diagrams based on the subjective
assessment criteria. This suggests that the task remains challenging even for state-of-the-art models.

Can models generate TikZ code close to the reference code? Table 2 indicates that IMGTIkZ-
MCG achieved the highest similarity scores for code similarity. However, code similarity scores are
generally high with minimal inter-model differences. This indicates that high code similarity does
not necessarily guarantee quality image generation. This discrepancy highlights a critical insight
for model training: generating code that closely resembles the ground truth is insufficient. Similar
to conventional VLMs, IMGTIkZ training relies on loss based on the next-word prediction of code.
However, our findings suggest the need to incorporate image similarity metrics in training or infer-
ence phrases. This result aligns with the significant performance improvements of IMGTIkZ-MCG.

Can models accurately render text in sketch images? The CharSim in Table 2 provides in-
sight into each model’s ability to recognize characters in sketch images and render them accurately
in TikZ diagram. Claude 3.5 achieved the highest CharSim score, followed by GPT-4o. While
IMGTIkZ achieved comparable performance to GPT-4o in Alignment, it significantly underperforms
in CharSim. This suggests that IMGTIkZ has enhanced diagram shape recognition but struggles with
detailed character recognition. This limitation may reflect the resolution constraints of the SigLIP
vision encoder. However, the substantial improvement in CharSim with multi-candidate generation
indicates the need to strengthen character recognition during training.

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

12345Number of Attempts0.40.50.60.70.80.91.0Cumulative CSRClaude 3.5 SonnetGPT-4oGPT-4o miniLLaVA-Next 8BImgTikZ-IG151020Number of Generated Candidates0.600.650.700.750.800.85Image Similarity (ImageSim)OracleImgTikZ with D-SigLIP selectorImgTikZ with CLIP selectorUnder review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Table 3: Evaluation of the effectiveness of
SKETIkZ as training data.

Table 4: Effectiveness of two data augmen-
tation: (a) ImgAugTikZ and (b) AugTikZ.

Model

ImageSim CharSim CSR avg

Model

ImageSim CharSim CSR avg

IMGTIkZ-IG
w/ SKETIkZ only

LLaVA-Next 8B

0.734
0.513

0.315

0.502
0.358

0.205

0.767
0.533

0.350

IMGTIkZ-IG

w/o (a)
w/o (a) and (b)

0.734

0.668
0.601

0.502

0.457
0.439

0.767

0.635
0.541

Can models generate high-quality diagrams? Table 2 presents quality scores from subjective
evaluations. Claude 3.5 achieved the highest average score of 3.54 out of 5, followed by IMGTIkZ-
MCG. Even the best-performing Claude model produces approximately 38% of samples with quality
scores below 3 (indicating significant overlap of shapes and text), demonstrating that current VLMs
still struggle with correct diagram layout rendering. This limitation in spatial reasoning is a common
challenge among current VLMs. Our task and dataset can be considered one of the benchmark
datasets for evaluating VLMs’ spatial reasoning capabilities.

How does the number of candidates in multi-candidate generation affect performance? Fig-
ure 8 illustrates the image similarity trends for ImgTikZ-MCG as the number of candidates K in
multi-candidate generation varies. The oracle represents the highest achievable performance by
selecting the best candidate based on image similarity to the reference diagram Ir. Results show
significant performance improvement when increasing candidates from 1 to 5. Both oracle and
IMGTIkZ demonstrate enhanced image similarity with more candidates. However, when replacing
the selection model from D-SigLIP to CLIP, performance doesn’t increase beyond 5 candidates.
This indicates the importance of selection model quality in multi-candidate generation.

Do subjective evaluations correlate with automated evaluations? We analyzed correlations be-
tween subjective alignment ratings and automatic evaluation metrics. Pearson’s correlation coef-
ficients were calculated between human-rated alignment and image similarity (0.759), code simi-
larity (0.365), and character similarity (0.592). Image similarity metric showed a high correlation
with subjective evaluation, while code similarity demonstrated a low correlation. Character simi-
larity exhibited moderate correlation, highlighting the importance of textual information in diagram
evaluation. Image similarity metrics often fail to capture this local textual similarity.

Are the subjective evaluations consistent? To assess inter-annotator agreement in subjective
evaluations, we employed Krippendorff’s α (Krippendorff, 1980), a measure commonly used in
related research (Otani et al., 2023; Ku et al., 2023). The analysis showed Krippendorff’s α of 0.761
for alignment and 0.662 for quality, indicating substantial to moderate agreement among annotators
in their subjective assessments.

7.2 DETAILED ANALYSIS
How effective is SKETIkZ alone as training data? We evaluated the effectiveness of our
SKETIkZ dataset, comprising only 2.5k hand-drawn sketch samples, as training data. We evalu-
ated the performance of a model trained solely on SKETIkZ in step 2. Results are presented in
Table 3. While the SKETIkZ-only model underperforms compared to the full-data model, it signifi-
cantly outperforms LLaVA-Next, indicating meaningful adaptation even with this limited dataset.

Is data augmentation effective? To assess the impact of our two data augmentation methods, we
trained models excluding ImgAugTikZ and both ImgAugTikZ and AugTikZ. Results are presented
in Table 4. The observed significant decrease in image similarity, character similarity, and CSR avg
when excluding these datasets confirms the effectiveness of both augmentation methods.

To what extent does image augmentation improve sketch recognition? While the ablation
study in Table 4 confirmed improved performance through image augmentation, we further inves-
tigated its impact on sketch recognition. Specifically, we compared the performance gap between
using rendered reference images Ir and sketch images Is as input. The closer the performance
of sketch input approaches that of rendered image input, the more robust the model’s understand-
ing of sketch noise can be considered. Results are shown in Table 5. Without ImgAugTikZ, im-
age similarity decreased by approximately 12.5% and character similarity by 22.7%. In contrast,
ImgTikZ limited these reductions to 6.97% for image similarity and 17.0% for character similarity.

9

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Table 5: Performance gap between ren-
dered and sketch image inputs: comparison
IMGTIkZ-IG and IMGTIkZ-IG without Im-
gAugTikZ data.

Table 6: Performance gap between ren-
dered and sketch image inputs across differ-
ent sketching tools. Evaluation conducted
using the IMGTIkZ-IG.

Model

Tool

Metric

IMGTIkZ-IG w/o ImgAugTikZ

Metric

Paper Whiteboard Tablet

ImageSim
Rendered Image
Sketch Image

0.789
0.734

Performance Gap

-6.97%

CharSim
Rendered Image
Sketch Image

0.605
0.503

Performance Gap

-16.9%

0.763
0.668

-12.5%

0.591
0.457

-22.7%

ImageSim
Rendered Image
Sketch Image

0.793
0.735

0.796
0.716

0.754
0.740

Performance Gap -7.31% -10.1% -1.90%

CharSim
Rendered Image
Sketch Image

0.587
0.502

0.627
0.451

0.581
0.570

Performance Gap -14.5% -28.1% -1.89%

However, ImgTikZ still doesn’t match rendered image input performance, suggesting potential for
further improvement through more noise-robust model construction.

Does image augmentation improve performance for non-sketch images? Comparing Ima-
geSim and CharSim results for Rendered Images in Table 5 reveals that ImgTikZ outperforms
the model without it. Image augmentation enhanced both ImageSim (0.763→0.789) and CharSim
(0.591→0.605) scores, showing improved recognition even for clean, computer-rendered images.

Does image recognition difficulty vary across sketch tools? Table 6 presents the performance
gap in image and character similarity when using rendered images versus sketches as inputs across
different sketching tools. Results show that tablet sketches maintain image and character similarity
close to rendered images. However, sketches from paper and whiteboard tools show significant
performance degradation. These exhibit a 7-10% drop in image similarity and a 14-28% decline
in character similarity. This performance drop suggests that paper and whiteboard sketches are
more challenging for the model to process, likely due to their increased noise variety compared to
tablet sketches. Whiteboard sketches showed the most significant decline in performance. While our
image augmentation techniques have relatively minimized the gap with rendered image input, further
performance improvements will require developing methods more robust to real-world noise.

8 CONCLUSION

We introduced SKETIkZ, a benchmark dataset with 3,231 pairs of hand-drawn sketches and their
corresponding TikZ codes for generating diagrams. Our experiments demonstrate that current VLMs
face considerable challenges in this task, highlighting the value of SKETIkZ as a benchmark for
future research. We also developed IMGTIkZ, an image-to-TikZ model. Despite being smaller,
this model performed as well as GPT-4o in subjective evaluations. This success came from using
two data augmentation techniques and generating multiple candidates during inference. SKETIkZ is
publicly available, and we expect these data resources and insights to drive the development of more
advanced and efficient methods for automating vector graphics creation from hand-drawn sketches.

9 LIMITATION

Currently, SKETIkZ is restricted to generating diagrams using TikZ. However, the methodology
could be extended to other formats such as SVG, HTML, Python, and JavaScript for diagram gen-
eration from code. Exploring these additional formats could enhance the dataset’s generality and
applicability. Transforming sketches into well-formed diagrams involves information completion,
which can potentially lead to hallucination. An important direction for future work is developing an
interactive system that allows users to modify generated diagrams through instructions. Further-
more, while our multi-candidate generation strategy considers code correctness and image quality
metrics after code generation, incorporating these metrics directly into the training phase could po-
tentially lead to better generation results, representing a promising direction for future work.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

ETHICS STATEMENT

Were annotators for sketch creation told what the dataset would be used for, and did they
consent? Yes. BAOBAB Inc. was fully responsible for managing the annotators. BAOBAB Inc.
provides task descriptions, training, and agreements for each project with the annotators https:
//baobab-trees.com/en/service.

Data License SKETIkZ is derived from a publicly available subset of DaTikZ (Belouadi et al.,
2023), which permits copying and redistributing content under a Creative Commons Attribution
License,8 the GNU Free Documentation License,9 or the MIT License.10

Potential ethical considerations we believe that there are minimal ethical considerations within
the scope of this current research. However, as more accurate automatic diagram generation be-
comes feasible in the future, several considerations may arise. These potential issues include the
misuse of highly accurate auto-generated diagrams to spread misinformation, the risk of AI models
perpetuating or amplifying biases from their training data, and the possibility of advanced systems
inadvertently reproducing copyrighted diagram designs, thereby raising intellectual property and
copyright infringement issues; all of these challenges necessitate the establishment of appropriate
guidelines to address them effectively.

REPRODUCIBILITY STATEMENT

Dataset Distribution All sketch image data in SKETIkZ is available at https://storage.
googleapis.com/sketikz/sketch_images.tar.gz. The TikZ code and other infor-
mation for train, val, and test can be downloaded from the following link https://storage.
googleapis.com/sketikz/sketikz_data.json. The metadata of SKETIkZ is also
available at this link https://storage.googleapis.com/sketikz/sketikz_meta_
data.json. This metadata provides information about the fields in the data.

Details of models, hyperparameters, and manual evaluation Appendices B, C, and E provide
detailed information about the models developed in this study. Appendix A describes the specifics
of our inference process. Appendix H presents details regarding the subjective evaluation. Addi-
tionally, Appendices D and G presents details of the data creation process.

REFERENCES

Jonas Belouadi, Anne Lauscher, and Steffen Eger. AutomaTikZ: Text-guided synthesis of scientific
vector graphics with TikZ. arXiv [cs.CL], 2023. URL http://arxiv.org/abs/2310.
00367.

Jonas Belouadi, Simone Paolo Ponzetto, and Steffen Eger. DeTikZify: Synthesizing graphics
programs for scientific figures and sketches with TikZ. arXiv [cs.CL], 2024. URL http:
//arxiv.org/abs/2405.15306.

Shreyanshu Bhushan and Minho Lee. Block diagram-to-text: Understanding block diagram im-
ages by generating natural language descriptors.
In Yulan He, Heng Ji, Sujian Li, Yang Liu,
and Chua-Hui Chang (eds.), Findings of the Association for Computational Linguistics: AACL-
IJCNLP 2022, pp. 153–168. Association for Computational Linguistics, 2022. URL https:
//aclanthology.org/2022.findings-aacl.15.

Shreyanshu Bhushan, Eun-Soo Jung, and Minho Lee. Unveiling the power of integration: Block
diagram summarization through local-global fusion. In Lun-Wei Ku, Andre Martins, and Vivek
Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 13837–
13856. Association for Computational Linguistics, 2024. URL https://aclanthology.
org/2024.findings-acl.822.

Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R´e, and
Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.
arXiv [cs.LG], 2024. URL http://arxiv.org/abs/2407.21787.

8https://creativecommons.org/licenses
9https://www.gnu.org/licenses/fdl-1.3.en.html
10https://opensource.org/license/mit

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. arXiv [cs.LG], 2020. URL http://arxiv.
org/abs/2002.05709.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang,
Boyang Li, Pascale Fung, and Steven Hoi.
InstructBLIP: Towards general-purpose vision-
language models with instruction tuning. arXiv [cs.CV], 2023. URL http://arxiv.org/
abs/2305.06500.

Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M Rush. Image-to-markup generation
with coarse-to-fine attention. arXiv [cs.CV], 2016. URL http://arxiv.org/abs/1609.
04938.

Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B Tenenbaum. Learning to infer
graphics programs from hand-drawn images. arXiv [cs.AI], 2017. URL http://arxiv.org/
abs/1707.09627.

Philippe Gervais, Asya Fadeeva, and Andrii Maksai. MathWriting: A dataset for handwritten
mathematical expression recognition. arXiv [cs.CV], 2024. URL http://arxiv.org/abs/
2404.10690.

Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Yi Su, Shaoling Dong, Xing Zhou, and
Wenbin Jiang. VISION2UI: A real-world dataset with layout for code generation from UI designs.
arXiv [cs.CV], 2024. URL http://arxiv.org/abs/2404.06369.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao
Bi, Y Wu, Y K Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. DeepSeek-coder: When the
large language model meets programming – the rise of code intelligence. arXiv [cs.SE], 2024.
URL http://arxiv.org/abs/2401.14196.

Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. SciCap: Generating captions for scientific fig-
ures. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-Tau Yih (eds.),
Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3258–3264. Asso-
ciation for Computational Linguistics, 2021. URL https://aclanthology.org/2021.
findings-emnlp.277.

Anwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye, Qinghao Ye, Ming Yan, Chenliang Li, Qi Qian,
Ji Zhang, and Fei Huang. mPLUG-PaperOwl: Scientific diagram analysis with the multi-
modal large language model. arXiv [cs.MM], 2023. URL http://arxiv.org/abs/2311.
18248.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang,
and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv [cs.CL], 2021.
URL http://arxiv.org/abs/2106.09685.

Chieh-Yang Huang, Ting-Yao Hsu, Ryan Rossi, Ani Nenkova, Sungchul Kim, Gromit Yeuk-Yin
Chan, Eunyee Koh, Clyde Lee Giles, and Ting-Hao ’kenneth Huang. Summaries as captions:
Generating figure captions for scientific documents with automated text summarization. arXiv
[cs.CL], 2023. URL http://arxiv.org/abs/2302.12324.

Zeba Karishma, Shaurya Rohatgi, Kavya Shrinivas Puranik, Jian Wu, and C Lee Giles. ACL-fig:
A dataset for scientific figure classification. arXiv [cs.AI], 2023. URL http://arxiv.org/
abs/2301.12293.

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali
Farhadi. A diagram is worth a dozen images. In Computer Vision – ECCV 2016, pp. 235–251.
Springer International Publishing, 2016. URL http://link.springer.com/10.1007/
978-3-319-46493-0_15.

Klaus Krippendorff. Content analysis: An introduction to its methodology. SAGE Publications,

1980. URL https://books.google.at/books?id=CyW7WBRzOqIC.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Imagen-
Hub: Standardizing the evaluation of conditional image generation models. arXiv [cs.CV], 2023.
URL http://arxiv.org/abs/2310.01596.

Hugo Laurenc¸on, L´eo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots
into HTML code with the WebSight dataset. arXiv [cs.HC], 2024. URL http://arxiv.
org/abs/2403.09029.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li,
Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy visual task transfer. arXiv [cs.CV], 2024.
URL http://arxiv.org/abs/2408.03326.

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization
Branches Out, pp. 74–81. Association for Computational Linguistics, 2004. URL https://
aclanthology.org/W04-1013.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction

tuning. arXiv [cs.CV], 2023. URL http://arxiv.org/abs/2310.03744.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-
Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of
foundation models in visual contexts. arXiv [cs.CV], 2023. URL http://arxiv.org/abs/
2310.02255.

Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, J Heikkila,
and Shin’ichi Satoh. Toward verifiable and reproducible human evaluation for text-to-image
Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 14277–
generation.
14286, 2023. URL http://openaccess.thecvf.com/content/CVPR2023/html/
Otani_Toward_Verifiable_and_Reproducible_Human_Evaluation_for_
Text-to-Image_Generation_CVPR_2023_paper.html.

Wamiq Reyaz Para, Shariq Farooq Bhat, Paul Guerrero, Tom Kelly, Niloy Mitra, Leonidas Guibas,
and Peter Wonka. SketchGen: Generating constrained CAD sketches. arXiv [cs.LG], 2021. URL
http://arxiv.org/abs/2106.02711.

Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z Xiao, Katherine M Collins, Joshua B
Tenenbaum, Adrian Weller, Michael J Black, and Bernhard Sch¨olkopf. Can large language models
understand symbolic graphics programs? arXiv [cs.LG], 2024. URL http://arxiv.org/
abs/2408.08313.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar-
wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya
In Marina
Sutskever. Learning transferable visual models from natural language supervision.
Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine
Learning, volume 139, pp. 8748–8763. PMLR, 2021. URL https://proceedings.mlr.
press/v139/radford21a.html.

Daniel Ritchie, Paul Guerrero, R Kenny Jones, Niloy J Mitra, Adriana Schulz, Karl D D Willis,
and Jiajun Wu. Neurosymbolic models for computer graphics. arXiv [cs.GR], 2023. URL
http://arxiv.org/abs/2304.10320.

Juan A Rodriguez, Shubham Agarwal, Issam H Laradji, Pau Rodriguez, David Vazquez, Christopher
Pal, and Marco Pedersoli. StarVector: Generating scalable vector graphics code from images.
arXiv [cs.CV], 2023. URL http://arxiv.org/abs/2312.11556.

Ari Seff, Wenda Zhou, Nick Richardson, and Ryan P Adams. Vitruvion: A generative model of
parametric CAD sketches. arXiv [cs.LG], 2021. URL http://arxiv.org/abs/2109.
14124.

Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2Code: How far
are we from automating front-end engineering? arXiv [cs.CL], 2024. URL http://arxiv.
org/abs/2403.03163.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Ashish Singh, Prateek Agarwal, Zixuan Huang, Arpita Singh, Tong Yu, Sungchul Kim, Vic-
tor Bursztyn, Nikos Vlassis, and Ryan A Rossi. FigCaps-HF: A figure-to-caption genera-
tive framework and benchmark with human feedback. arXiv [cs.CL], 2023. URL http:
//arxiv.org/abs/2307.10867.

Davit Soselia, Khalid Saifullah, and Tianyi Zhou. Learning UI-to-code reverse generator using
visual critic without rendering. arXiv [cs.CV], 2023. URL http://arxiv.org/abs/2305.
14637.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing
Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men,
Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhancing vision-
arXiv [cs.CV], 2024. URL
language model’s perception of the world at any resolution.
http://arxiv.org/abs/2409.12191.

Shaowei Wang, Lingling Zhang, Longji Zhu, Tao Qin, Kim-Hui Yap, Xinyu Zhang, and Jun Liu.
CoG-DQA: Chain-of-guiding learning with large language models for diagram question answer-
https://openaccess.thecvf.com/content/CVPR2024/papers/Wang_
ing.
CoG-DQA_Chain-of-Guiding_Learning_with_Large_Language_Models_
for_Diagram_Question_CVPR_2024_paper.pdf.

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen
Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian
Qi, Ji Zhang, and Fei Huang. mPLUG-owl: Modularization empowers large language models
with multimodality. arXiv [cs.CL], 2023. URL http://arxiv.org/abs/2304.14178.

Abhay Zala, Han Lin, Jaemin Cho, and Mohit Bansal. DiagrammerGPT: Generating open-domain,
open-platform diagrams via LLM planning. arXiv [cs.CV], 2023. URL http://arxiv.org/
abs/2310.12128.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language
image pre-training. arXiv [cs.CV], 2023. URL http://arxiv.org/abs/2303.15343.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing
vision-language understanding with advanced large language models. arXiv [cs.CV], 2023. URL
http://arxiv.org/abs/2304.10592.

Bocheng Zou, Mu Cai, Jianrui Zhang, and Yong Jae Lee. VGBench: Evaluating large language
models on vector graphics understanding and generation. arXiv [cs.CV], 2024. URL http:
//arxiv.org/abs/2407.10972.

A DETAILS OF INFERENCE

Inference Procedure We used pdflatex from TeX Live 202311 to compile generated TikZ code
into a diagram image. We first cropped the rendered image using pdfcrop and then converted it to a
PNG file to calculate image similarity.

Hyperparameters for closed models We used the API’s default parameters for the closed models
GPT-4o, GPT-4o mini, and Claude. The max token parameter was set to 2,048 for all models.

Hyperparameters for LLaVA1.6 and IMGTIkZ We set the maximum number of newly gener-
ated tokens to 2,048 and generated the code through sampling. The sampling temperature was set to
0.6, a value determined through evaluation using the validation set.

B HYPERPARAMETERS FOR TRAINING IMGTIkZ

We conducted the training using the official code of LLaVA.14 Table 7 details the hyperparameters
used for stage 2 training of IMGTIkZ. For stage 2 training, we used a total batch size of 128. The
stage 1 training employed similar hyperparameters, with a few exceptions: we set the batch size to 32

11https://tug.org/texlive/
14https://github.com/haotian-liu/LLaVA

14

Under review as a conference paper at ICLR 2025

Table 7: Configuration for the IMGTIkZ model training.

Option

Value

deepseek-ai/deepseek-coder-6.7b-instruct12

128
256
2e-5
mlp2x gelu
True
True
1
16

model name (LLM)
model name (Vision encoder) google/siglip-so400m-patch14-38413
lora r
lora alpha
mm projector lr
mm projector type
group by modality length
bf16
num train epochs
batch size
gradient accumulation steps 8
weight decay
0
0.03
warmup ratio
lr scheduler type
cosine
4096
model max length
True
gradient checkpointing

with gradient accumulation over 4 steps, resulting in a total batch size of 128, and we increased the
max length to 2048. These parameters were derived from the original implementation of LLaVA1.5.
The training process consisted of 6000 steps for stage 1 and a full epoch for stage 2. We conducted
the training using 8 A100 GPUs. The total training time was approximately 24 hours for stage 1 and
60 hours for stage 2.

C D-SIGLIP: AN SIGLIP MODEL ADAPTED FOR DIAGRAM

We trained D-SigLIP using a contrastive learning framework using the Hugging Face’s code.15 We
used the google/siglip-so400m-patch14-384 version of SigLIP as the vision encoder.
During training, we applied augmentation twice to each image, aiming to maximize the similarity
between augmented versions of the same image within the batch. Image augmentation was per-
formed on-the-fly using imgaug.16 The noise pipeline applied through imgaug is detailed below.

Listing 1: Image Augmentation Pipeline for D-SigLIP Training

pipeline = iaa.Sequential([

iaa.Affine(scale={"x": (0.7, 1.0), "y": (0.7, 1.0)}, cval=255),
iaa.Affine(rotate=(-5, 5), cval=255),
iaa.Affine(translate_percent={"x": (-0.1, 0.1), "y": (-0.1, 0.1)},

cval=255),

iaa.Sometimes(0.2, iaa.ChangeColorTemperature((1100, 3000))),
iaa.Sometimes(0.3, iaa.AdditiveGaussianNoise(scale=(10, 20))),
iaa.Sometimes(0.3, iaa.MultiplyAndAddToBrightness(mul=(0.8, 1.2), add

=(-5, 5))),

iaa.Sometimes(0.3, iaa.GammaContrast((0.8, 1.2))),
iaa.Sometimes(0.3,

iaa.BlendAlphaSimplexNoise(
iaa.Multiply((1.5, 2.5), per_channel=True),
upscale_method=’cubic’,
iterations=(1, 2)

)),
iaa.Sometimes(0.1, iaa.LinearContrast((0.8, 1.2))),
iaa.ElasticTransformation(alpha=(15.0, 40.0), sigma=(5.0, 10.0)),

])

15https://github.com/huggingface/transformers/tree/main/examples/

pytorch/contrastive-image-text

16https://imgaug.readthedocs.io/en/latest/

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

The training was conducted using four H100 80G GPUs. We set the batch size to 1024, the learning
rate to 5e-5, and the warmup steps to 0, with training carried out for 200 steps.

D DATASET COLLECTION PROCESS

First, we compiled the TikZ code from DaTikZ (Belouadi et al., 2023) to render the diagram images.
Then, we developed a diagram classification model (See Section E) using the ACL-fig (Karishma
et al., 2023) data, which was subsequently employed to classify the rendered diagrams from the
DaTikZ dataset.
We then extracted diagrams with the predicted labels Tree, Graph, Architecture diagram, Neural
networks, and Venn diagram and sampled 4,000 instances from them.
BAOBAB Inc. coordinated multiple annotators to create the corresponding sketches for sampled
instances. We excluded diagrams that were too complex to be sketched, diagrams of bar charts and
line graphs that require numerical data, overly simplistic diagrams comprising only straight lines or
dots, diagrams with illegible text, diagrams containing non-English text, and incomplete diagrams
that were unnaturally truncated from the tasks during this process. The annotators selected one
of the following tools to create the sketches: paper, whiteboard, or tablet. When using paper or
whiteboard, they captured photos of the hand-drawn images with a smartphone camera. They used
the drawing tool’s save function for tablets to save the images. All images were then converted to
PNG format. As a result of these processes, we ultimately created 3,231 instances.

E DIAGRAM IMAGE CLASSIFICATION MODEL FOR DATA CONSTRUCTION

We developed a model to classify diagram images into categories by fine-tuning a pre-trained
vision transformer on the ACL-fig dataset.17
For the pre-trained VIT, we used Google’s
vit-large-patch16-224-in21k.18 The training was conducted using Hugging Face’s
tools.19 The parameters used for the training are listed in Table 8. We trained the model using
a NVIDIA A100 GPU. The model achieved a classification accuracy of 0.886 on the evaluation
dataset.

Table 8: Configuration for the image classification model.

Option

Value

google/vit-large-patch16-224-in21k
2e-5

model name
learning rate
num train epochs 5
8
batch size
0
warmup ratio
0
weight decay

Table 9 presents the breakdown of estimated image labels within the sampled data. Furthermore,
Figure 9 illustrates example diagrams for each estimated label category. While these are estimated
labels and may potentially include diagrams that do not strictly conform to any specific category or
contain estimation errors, we confirmed that there are diverse types of diagrams in our dataset.

F SKETCH IMAGE EXAMPLES

Figure 10 shows a subset of the collected sketch images.

G DETAILS OF THE DATA AUGMENTATION

G.1 AUGTIkZ: THE AUGMENTATION FOR INCREASING DIAGRAM VARIATION
From the arXiv source files,20 we initially obtained 916,123 TikZ code samples. However, only
155K of these were successfully compiled. We utilized these compilable codes as RenderTikZ.

17https://huggingface.co/datasets/citeseerx/ACL-fig
18https://huggingface.co/google/vit-large-patch16-224-in21k
19https://github.com/huggingface/transformers/blob/main/examples/

pytorch/image-classification/run_image_classification.p

20https://info.arxiv.org/help/bulk_data_s3.html

16

Under review as a conference paper at ICLR 2025

Table 9: Proportion of estimated image labels in the sampled data.

Category

Number Proportion

1,799
Tree
Graph
1,046
Architecture diagram 646
459
Neural networks
50
Venn diagram

45.0 %
26.2 %
16.2 %
11.5 %
1.1 %

All

4,000

100%

Figure 9: Examples of estimated image labels and their diagrams.

While the remaining codes failed to compile, we recognized their potential to significantly increase
diagram variations if effectively utilized. To achieve this, we employed two types of augmentation
prompts. The first prompt focused on code revision and was applied to the initially failed compila-
tions. The second prompt, aimed at code modification, was applied to the entire dataset. The specific
instructions provided were as follows. We used the gpt-3.5-turbo-0125 version of GPT-3.5
to create the augmentation data.

Prompts for data augmentation

• Please modify the given LaTeX source code to make it compilable, including only
the required preamble statements. If any external files are referenced, please modify
the code to avoid referencing external files and include the content directly. The
output should consist solely of the code itself, without any supplementary text.
• Please generate TikZ source code that modifies parts of the following code to create
a different diagram. Ensure the code is compilable and includes only the required
preamble statements. If any external files are referenced, please modify the code to
avoid referencing external files and include the content directly. The output should
consist solely of the code itself, without any supplementary text.

We included only the code that successfully compiled and rendered images correctly in our dataset
AugTikZ. Furthermore, we excluded images that were rendered at extreme scales (either too large
or too small) from the training dataset.

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

Figure 10: Examples of collected sketch images.

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

G.2

IMGAUGTIkZ: THE AUGMENTATION FOR INCREASING IMAGE VARIATION

To simulate the noise typically present in sketches, we applied several augmentation techniques to
both RenderTikZ and AugTikZ. These included compositing with notebook background images,
augmentation using imgaug, and white balance augmentation.21 For notebook backgrounds, we
created eight unique images independently of the sketch annotation process. The imgaug library
was used to generate variations in rotation, distortion, Gaussian noise, brightness, and contrast. The
specific augmentation pipeline created with imgaug is detailed below.

Listing 2: Image Augmentation Pipeline for Image Augmentation

pipeline = iaa.Sequential([

iaa.Pad(percent=0.3, pad_mode="median"),
iaa.Sometimes(0.3, iaa.AdditiveGaussianNoise(scale=(10, 20))),
iaa.Sometimes(0.3, iaa.MultiplyAndAddToBrightness(mul=(0.8, 1.2), add

=(-5, 5))),

iaa.Sometimes(0.3, iaa.GammaContrast((0.8, 1.2))),
iaa.Sometimes(0.3,

iaa.BlendAlphaSimplexNoise(
iaa.Multiply((1.5, 2.5), per_channel=True),
upscale_method=’cubic’,
iterations=(1, 2)

)),
iaa.Sometimes(0.1, iaa.LinearContrast((0.8, 1.2))),
iaa.Affine(rotate=(-5, 5)),
iaa.ElasticTransformation(alpha=(15.0, 30.0), sigma=(5.0, 10.0)),
iaa.CropToFixedSize(width=int(width*0.8), height=int(height*0.8))

])

H SUBJECTIVE EVALUATION

For each test sample, annotators evaluated the alignment and quality of the six systems’ outputs,
GPT-4o, GPT-4o mini, Claude 3.5 Sonnet, LLaVA-Next, IMGTIkZ-IG, IMGTIkZ-MCG. We com-
pensated annotators at a rate of $1.5 per test sample.
We provided annotators with the following instructions for conducting their evaluations:

Instructions

For each image A-F, please assign a score from 1 to 5 based on the following two aspects.
You may also use 0.5 increments, such as 1.5 or 3.5.

• Alignment: The extent to which the generated diagram image matches the layout and

content of the hand-drawn image.

• Quality: The overall completeness of the generated diagram image, regardless of the

presence or absence of the hand-drawn and reference image.

The specific evaluation criteria for alignment that we instructed the annotators to follow are as fol-
lows:

21https://github.com/mahmoudnafifi/WB_color_augmenter

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

Evaluation Criteria for Alignment

1: The elements of the diagram in the generated image and the hand-drawn image do

not match at all.

2: The elements of the diagram in the generated image and the hand-drawn image

match approximately 25%.

3: The elements of the diagram in the generated image and the hand-drawn image

match approximately 50%.

4: The elements of the diagram in the generated image and the hand-drawn image

match approximately 75%.

5: The elements of the diagram in the generated image and the hand-drawn image

match almost perfectly.

The specific evaluation criteria for quality that we instructed the annotators to follow are as follows:

Evaluation Criteria for Quality

1: Almost complete overlap of text or shapes, making the diagram unreadable.
2: Significant overlap of text or shapes, and the arrangement of elements is unnatural.
3: Significant overlap of text or shapes, making some elements unreadable, or some

elements are arranged unnaturally.

4: Some overlap of text or shapes, but the arrangement of elements is neat.
5: No overlap of text or shapes, and the arrangement of elements is as neat as a human-

created diagram.

Figure 9 presents a partial screenshot of the annotation system interface. The complete template
file for the annotation system, which includes all instructions, can be accessed this link https:
//storage.googleapis.com/sketikz/template_202409_example.html.

Figure 11: Screenshot of the annotation interface: In the HTML, each image can be clicked to
enlarge, allowing annotators to view the details of the diagrams.

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

I GENERATED DIAGRAM EXAMPLES WITH EVALUATION SCORES

Tables 10 and 11 show some examples of generated diagrams. IMGTIkZ-MCG generally selects
better candidates compared to IMGTIkZ-IG.

Table 10: Examples of generated diagrams and their metric scores.⊠ indicates a compile error and,
therefore, has no score.

(a)

sketch diagram

reference diagram

Models

GPT-4o

GPT-4o mini Claude 3.5

LLaVA

IMGTIkZ-
IG

IMGTIkZ-
MCG

Diagram

Alignment
Quality
ImageSim

3.67
3.67
0.86

2.83
2.67
0.69

3.83
4.17
0.82

1.67
2.17
0.38

4.00
3.67
0.81

4.67
5.00
0.91

(b)

sketch diagram

reference diagram

Models

GPT-4o

GPT-4o mini Claude 3.5

LLaVA

IMGTIkZ-
IG

IMGTIkZ-
MCG

Diagram

Alignment
Quality
ImageSim

3.83
4.00
0.79

⊠

N/A
N/A
N/A

4.50
4.83
0.92

1.00
1.00
0.05

4.17
4.83
0.87

4.67
4.83
0.92

J DETAILED EXPLANATION OF COMPILATION SUCCESS RATE (CSR)

To better illustrate the difference between CSRavg and CSRcum, we provide examples below.
CSRavg represents the success rate across all generation attempts. For example, if a model attempts
N generation for each of the 100 test samples and succeeds in compilation K times, then

CSRavg =

Nsuccess
Ngen

=

K
(100 × N )

.

(1)

21

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

Table 11: Examples of generated diagrams and their metric scores.⊠ indicates a compile error and,
therefore, has no score.

(c)

sketch diagram

reference diagram

Models

GPT-4o

GPT-4o mini Claude 3.5

LLaVA

IMGTIkZ-
IG

IMGTIkZ-
MCG

Diagram

Alignment
Quality
ImageSim

3.83
4.17
0.70

3.17
3.50
0.76

4.67
4.83
0.88

2.33
3.83
0.27

3.00
3.33
0.77

3.83
4.17
0.81

(d)

sketch diagram

reference diagram

Models

GPT-4o

GPT-4o mini Claude 3.5

LLaVA

Diagram
Alignment
Quality
ImageSim

⊠

N/A
N/A
N/A

⊠

N/A
N/A
N/A

⊠

N/A
N/A
N/A

⊠

N/A
N/A
N/A

IMGTIkZ-
IG

IMGTIkZ-
MCG

2.83
4.33
0.75

4.33
3.50
0.86

To illustrate, if we make 10 generation attempts for each of the 100 test samples (totaling 1,000
generations) and achieve successful compilation in 400 cases, then

CSRavg =

400
1000

= 0.4.

(2)

CSRcum, which is exclusively used for iterative generation, measures the cumulative proportion of
test samples achieving successful compilation across multiple attempts. Consider the following
sequential process for 100 test samples:

• first generation: 50 of the 100 samples compile successfully

• Second generation: 20 of the remaining 50 (100 - 50) samples compile successfully

• Third generation: 10 of the remaining 30 (50 - 20) samples compile successfully

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Under review as a conference paper at ICLR 2025

In this scenario,

CSRcum =

Ntest success
Ntest

=

50 + 20 + 10
100

= 0.8.

(3)

This metric specifically quantifies the proportion of test samples that eventually achieve successful
compilation, independent of the total generation attempts.
The motivation for utilizing these two distinct evaluation metrics arises from their complementary
analytical perspectives: CSRavg represents the average compilation success rate, enabling fair model
comparison. CSRcum measures the proportion of successfully compiled test samples across multiple
attempts, analogous to a recall metric.

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

23

