CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair
Under review as a conference paper at ICLR 2025

CRAFTRTL: HIGH-QUALITY
SYNTHETIC DATA
GENERATION FOR VERILOG CODE MODELS WITH
CORRECT-BY-CONSTRUCTION NON-TEXTUAL REP-
RESENTATIONS AND TARGETED CODE REPAIR

Anonymous authors
Paper under double-blind review

ABSTRACT

Despite the significant progress made in code generation with large language mod-
els, challenges persist, especially with hardware description languages such as
Verilog. This paper first presents an analysis of fine-tuned LLMs on Verilog cod-
ing, with synthetic data from prior methods. We identify two main issues: dif-
ficulties in handling non-textual representations (Karnaugh maps, state-transition
diagrams and waveforms) and significant variability during training with models
randomly making “minor” mistakes. To address these limitations, we enhance
data curation by creating correct-by-construction data targeting non-textual rep-
resentations. Additionally, we introduce an automated framework that generates
error reports from various model checkpoints and injects these errors into open-
source code to create targeted code repair data. Our fine-tuned Starcoder2-15B
outperforms prior state-of-the-art results by 3.8%, 10.9%, 6.6% for pass@1 on
VerilogEval-Machine, VerilogEval-Human, and RTLLM.

1

INTRODUCTION

Large Language Models (LLMs) have achieved significant success across various natural language
processing tasks and have extended their capabilities to code generation, leading to the development
of specialized models targeting code generation. The effectiveness of these models is largely in-
fluenced by the size and quality of their training datasets, as highlighted by scaling laws (Achiam
et al., 2023; Zhang et al., 2024a). Prominent code LLMs have set new benchmarks records by uti-
lizing extensive, synthetically generated datasets through methods like Self-Instruct (Wang et al.,
2022; Chaudhary, 2023), Evol-Instruct (Xu et al., 2023), and OSS-Instruct (Wei et al., 2023). These
synthetic data generation techniques allow code LLMs to generate a wide range of complex code
examples, enhancing their training and performance in real-world coding scenarios.

While most code LLMs concentrate on software programming languages, there is increasing interest
in developing models for hardware description languages (HDLs), which are essential for chip de-
sign and hardware verification. Despite efforts to collect and synthesize more diverse Verilog code
to enhance specialized code LLMs (Liu et al., 2023c; Pei et al., 2024; Cui et al., 2024; Zhao et al.,
2024), HDLs still face challenges akin to those encountered in low-resource languages (Cassano
et al., 2022). These challenges are mainly due to the limited availability of high-quality instruction-
following data and the constrained capability of existing LLMs to generate RTL code, which affects
the models’ performance and their ability to generalize across programming languages.

Developing high-quality synthetic Verilog code for training code large language models (LLMs)
faces significant challenges due to two primary factors. Firstly, Verilog is considered a low-resource
language (Cassano et al., 2022), meaning there is a scarcity of available training data compared to
high-resource software programming languages like Python. This limited data availability restricts
the models’ ability to learn diverse and complex coding patterns effectively. Secondly, verifying
the correctness of hardware description language (HDL) code, such as Verilog, is inherently more
complex than verifying software code. While software code correctness can often be assessed using
random test cases and automated unit tests (Chen et al., 2022), hardware code requires comprehen-
sive testbenches and rigorous verification planning and methodologies. This additional complexity

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

makes it challenging to ensure that synthetic Verilog code is functionally accurate (Bhandari et al.,
2024; Qiu et al., 2024), posing a barrier to improving model performance.

In this paper, we start with a thorough analysis of fine-tuned large language models (LLMs) applied
to Verilog code, using synthetic data techniques from previous works. Our analysis reveals two key
issues: (1) models have difficulty handling non-textual elements in problem statements, indicating
challenges in interpreting complex or unconventional inputs; and (2) there is notable variability in
the models’ pass rates across different benchmark problems and training checkpoints, exposing in-
consistencies in learning outcomes, often due to the models making “minor” programming mistakes.

Given the limitations identified in our analysis of relying solely on LLMs for generating synthetic
data, we shift our focus to improving data curation to address these issues. Current LLMs fre-
quently struggle with interpreting and processing non-textual representations and are insufficient in
generating effective testbenches for evaluating solution quality. Therefore, instead of depending ex-
clusively on LLMs to address data quality concerns, we develop targeted fine-tuning data to better
mitigate these problems. Experimental results demonstrate that our models achieve state-of-the-art
(SOTA) results on VerilogEval (Liu et al., 2023b) and RTLLM v1.1 (Lu et al., 2024) benchmarks,
outperforming prior works by large margins on problems with human-level description. The major
contributions of this paper are as follows:

• We perform a thorough analysis of fine-tuned LLMs on Verilog code using previously
established synthetic data generation methods, uncovering challenges with non-textual el-
ements and notable variability in performance across benchmark problems during training.

• We create correct-by-construction data to ensure solution correctness, incorporating Kar-
naugh Maps, state-transition diagrams, and waveforms, which significantly enhance the
model’s ability to handle non-textual representations.

• We develop an automated framework that utilizes LLMs to generate error reports from
benchmark problems at various checkpoints, which are then injected into open-source code
to create a fine-tuning dataset targeted at correcting the model’s specific “minor” mistakes.

• We rigorously evaluate the latest foundational and frontier code models. We note that
recent advanced models like GPT-4o already reached competitive performance compared
to previous efforts targeting Verilog code generation.

• Experimental results demonstrate that models fine-tuned with our data achieve state-of-the-
art performance on Verilog coding. Specifically, our fine-tuned model based on Starcoder2-
15B (Lozhkov et al., 2024) outperforms prior SOTA results by 3.8%, 10.9%, 6.6% for
pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM, respectively.

2 EXAMINING FINE-TUNED LLMS USING SYNTHETIC GENERATED DATA

ON VERILOG CODING

In this section, we start with a thorough analysis of fine-tuned large language models (LLMs) applied
to Verilog code. We adapt previous approaches for generating synthetic data for general coding to
focus on Verilog code. For our pilot study, we only present results based on fine-tuning StarCoder2-
15B (Lozhkov et al., 2024). Details on experimental settings are the same as in Section 4. We
assess model performance in Verilog code completion and identify two main issues. First, the mod-
els demonstrate notably poor performance when dealing with non-textual elements in the problem
statements. Second, the variability in the models’ pass rates across different benchmark problems
and training checkpoints suggests inconsistencies in learning outcomes and model variability.

2.1 SYNTHETIC DATA GENERATION FOR VERILOG CODING

We build on previous methods for synthetic data generation by applying Self-Instruct (Wang et al.,
2022) and OSS-Instruct (Wei et al., 2023) with custom prompt templates tailored for Verilog coding.
To enhance data coverage and diversity, we supplement these techniques with additional context
from Wikipedia and textbooks. We also prompt models to generate problem descriptions to include
non-textual representations.

2

Under review as a conference paper at ICLR 2025

We use nemotron-4-340b-instruct (Nvidia et al., 2024) selected for
its open license that allows commercial use. Our process includes
deduplication and a decontamination procedure akin to that out-
lined by Li et al. (2023). Additionally, we conduct syntax checks
to eliminate coding problems containing docstrings or solutions
from Verilog benchmarks. To ensure further data quality, we dis-
card code solutions that fail these syntax checks and apply self-
verification (Weng et al., 2023) to remove entries where the LLM
identifies errors in the solution. Table 1 shows the quantity of our
synthetic data generation (denoted as SDG) after deduplication and
filtering, yielding a total of 80.1k fine-tuning examples.

Table 1: Data quantity SDG.

Method

Quantity

Self-Instruct

OSS-Instruct

Docu-Instruct

Non-textual

SDG Total

24.7k

28.4k

12.0k

15.0k

80.1k

Self-Instruct We follow the approach outlined in Wang et al. (2022) to generate synthetic Verilog
coding problems. Initially, we randomly generate from the LLM and curate 50 questions that request
Verilog coding problems without any in-context examples. From these, we then randomly choose 1
to 5 seed questions to use as in-context examples.

OSS-Instruct We begin by processing pretraining code data to extract our seed code from The
Stack v2 (Lozhkov et al., 2024), focusing on Verilog and SystemVerilog. Following the approach
in Liu et al. (2023b), we post-process this data by selecting self-contained Verilog code that passes
syntax checks using Pyverilog (Takamaeda-Yamazaki, 2015). With the refined seed code data, we
then prompt large language models (LLMs) to use this code as inspiration for generating Verilog
coding problems similar to Wei et al. (2023).

Docu-Instruct Drawing inspiration from Nvidia et al. (2024) and Sudalairaj et al. (2024), we
utilize document sources from Wikipedia and textbooks for instruction generation. We begin by
filtering Wikipedia entries, prompting the LLM to classify whether the content pertains to hardware
design or Verilog coding concepts. Additionally, we manually selected approximately relevant 100
textbooks. These textbooks are then segmented into chunks of paragraphs or sentences, ensuring
each chunk contains fewer than 2k tokens.

Non-textual Representations VerilogEval-Human (Liu et al., 2023b) includes benchmark prob-
lems involving non-textual representations. For example, Boolean logic tables and Karnaugh maps
are presented in tabular formats, state-transition diagrams for finite state machines are depicted as
edge lists and sequential waveforms are described in tables with signal outputs recorded at various
time steps. To incorporate such representations, we encouraged LLMs to generate problems from
open-source code, with instructions to utilize these tabular data structures.

2.2 CHALLENGES WITH NON-TEXTUAL REPRESENTATIONS

Model

Machine Human NonText

Table 2: pass@1 results on VerilogEval sam-
pled with temperature of 0.8.

We observe that models underperform on bench-
mark problems involving non-textual input formats,
such as Karnaugh Maps, state-transition diagrams,
and waveforms. Table 2 shows the pass@1 results
for the VerilogEval (Liu et al., 2023b). Addition-
ally, we have identified a subset of 45 questions
within VerilogEval-Human that include non-textual
representations, termed VerilogEval-NonText. It ap-
pears that models like GPT-4o and Starcoder2 strug-
gle with these non-textual formats, likely due to insufficient representation of such data during both
pretraining and fine-tuning. Despite our efforts to generate such questions during synthetic data cre-
ation, our fine-tuned models still lag in these areas. This outcome is not entirely surprising, given
that the LLMs used were also ineffective at generating problems with these representations, compli-
cating the validation of fine-tuning data. These results suggest that merely including non-textual data
is insufficient; ensuring the quality and correctness of the data, particularly that the code solutions
accurately align with these representations, is crucial.

Starcoder2-SDG

Starcoder2

GPT-4o

63.7

55.4

57.7

27.0

29.1

22.2

10.3

47.4

73.7

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

(a) Starcoder2-15B on SDG.

(b) Starcoder2-15B on SDG-CC-Repair.

Figure 1: Our methods reduce pass rate variability during training: SDG (left) shows high volatility
with significant degradation on many problems, while SDG-CC-Repair (right) stabilizes learning
outcomes on solvable problems (details in Appendix A.10).

2.3 VARIABILITY ON PASS RATES DURING TRAINING

During our training, we observed significant variability in the model’s pass rate on specific bench-
mark problems across different checkpoints. We note such variance is different from training insta-
bility (Wortsman et al., 2023) as we observe a stable decrease in the training loss. This variability
persists even in the later stages of training, despite using a low learning rate. We illustrate this vari-
ability in Figure 1a. The scatter plot tracks the pass rate for each problem in VerilogEval-Human,
with each point representing the pass rate for the same problem across two checkpoints. The size of
each point indicates the number of problems with the same pass rates for the two model checkpoints.
We further categorize the region into areas where the checkpoints agree on problem difficulty and
areas where they do not.

Alarmingly, we find that nearly 15% of the problems show significant discrepancies between these
two checkpoints, with an equal number of problems demonstrating improvement and degradation.
Our detailed analysis of the sampled code completions for such problems when pass rate degrades
suggests that the model is generally on the right track but makes “minor” errors that are small,
detailed, and seemingly trivial. While it is possible that LLMs experience catastrophic forgetting
during fine-tuning (Luo et al., 2024a), we do not anticipate this being a major factor due to the low
learning rate and the small number of gradient updates (64 steps with 16k data samples). Instead, we
believe the primary issue is our inability to ensure the quality of our data, particularly in verifying
whether the sampled code solutions correctly solve the code problems.

3

IMPROVING VERILOG CODING WITH CORRECT-BY-CONSTRUCTION
NON-TEXTUAL REPRESENTATIONS AND TARGETED CODE REPAIR

Based on our detailed analysis of the limitations of relying solely on LLMs for generating syn-
thetic data, we focus our data curation efforts to address these shortcomings. Our goal is to enhance
data quality and ensure the correctness of solutions for the generated problems. We have found
that current LLMs often lack the capability to understand and process non-textual representations
effectively and are unable to generate satisfactory testbenches for assessing solution quality. Con-
sequently, rather than depending entirely on LLMs to resolve data quality issues, we instead create
targeted fine-tuning data to mitigate these problems.

3.1 ENSURING QUALITY THROUGH CORRECT-BY-CONSTRUCTION

We generate Verilog code problems and solutions that are correct-by-construction. Our focus is
on creating problems and solutions for non-textual representations. Table 3 shows the quantity of
our correct-by-construction generation data (referred to as CC). To prevent data contamination, we
exclude entries that duplicate the data representations of benchmark problems.

4

UnsolvableSolved0.33UnsolvableSolved0.330.67Pearson Corr Coeff: 0.638UnsolvableSolved0.330.67Pearson Corr Coeff: 0.782Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Table 3: Data quantity CC.

Karnaugh Maps and Truth Tables (KMap) We start by sam-
pling random configurations, which include selecting the number
of variables and their names. After determining the number of vari-
ables, we randomly choose valid minterms and don’t-cares. For
n variables, there are 2n possible states, and each state can be as-
signed one of three values (0, 1, or x), leading to 32n
possible com-
binations of minterms and don’t-cares. From these minterms, we
derive the sum-of-products (SOP) form to represent the Boolean
logic. We then create Truth Tables and Karnaugh Maps based on
the chosen minterms and don’t-cares. In the KMap, Gray encoding is used as default for the row
and column sequences to ensure that only a single bit changes between adjacent cells. Additionally,
we apply modifications by transposing the map and randomly swapping adjacent rows or columns.
We randomly sample from n = {3, 4} variables.

Waveforms

CC Total

Quantity

Method

KMap

28.5k

12.5k

FSM

8.0k

8.0k

State Transition Graphs and Tables (FSM) We construct problems for finite-state machines
(FSMs) with state-transition representations with a similar approach to KMaps. We begin by sam-
pling random configurations, including the number of states (e.g., 4, 6, or 10) and the bit width
of the input (e.g., 1 or 2). We then create the transition graph, ensuring that it is both meaningful
and legally defined. We generate state-transition graphs for both Moore and Mealy state machines.
From these graphs, we produce edge-list and transition table representations. Finally, we construct
the Verilog code to implement the logic for state transitions and output assignments.

Algorithm 1 outlines the process
for generating a Moore FSM with
random transitions. State reacha-
bility is ensured by first construct-
ing a tree. Legality for state tran-
sition is ensured by ensuring each
node has an out-degree of 2w with
the input bit width of w. The re-
sult is an FSM where transitions
between states are randomly as-
signed but conform to the speci-
fied input bit width. The algorithm
can be easily modified for a Mealy
FSM by assigning the output to the
edges rather than nodes.

Algorithm 1 Generate transition graph for Moore FSM.

Input: Number of states n, bit width of input w
Output: FSM graph with transitions and states
Initialize the number of states n and bit width of input w
Randomly generate a tree with n nodes
Define the root of the tree as the reset state
for each node in the tree do

Assign a unique state to the node
Assign an output to the node

end for
for each node in the tree do

Add additional transition edges to form a graph
Ensure that each node has an out-degree of 2w

end for

Figure 2 illustrates our approach for generating state
transition logic in Verilog from a state-transition
graph. Our method predominantly employs an out-
edge focused strategy for state transitions. Addi-
tionally, we incorporate in-edge focused transition
logic to address specific challenges encountered in
benchmark problems. These benchmarks often in-
volve states represented using one-hot encoding and
require rigorous testing of non-default states.

Figure 2: State transition logic.

Waveforms We utilize correct-by-construction code solutions for both KMaps and FSMs. Be-
cause these codes are generated using similar templates, designing corresponding testbenches is
straightforward. We simulate the generated code to produce waveform Value Change Dump (VCD)
files. These VCD files are then parsed and converted into waveform representations. Our approach
covers KMaps as combinational circuits and FSMs as sequential circuit waveforms.

3.2 MITIGATING “MINOR” ERRORS WITH TARGETED CODE REPAIR

Our analysis revealed that the models were generally on the right track to correct solutions but
were making minor errors—small, detailed, and seemingly trivial. Unlike complex, unsolvable
problems, these minor errors could be easily corrected by language models. This insight led us

5

resetA/0C/1B/0D/110101001out-edge focused:case(state)      B: next_state = in ? C : D... endcasein-edge focused:next_state[B] = (state[A] &in) | (state[C] & ~in)...Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Figure 3: Overview of our approach for generating targeted code repair data: (1) prompting the
LLM to generate detailed error reports from correct and erroneous code, (2) validating error report
quality by ensuring the LLM can debug the errors based on the report, and (3) leveraging the LLM
to inject similar errors into open-source code, creating a diverse training dataset.

to develop a new strategy centered on targeted error code repair. Our approach includes creating
detailed error reports on benchmark problems, re-creating these errors on correct open-source code,
and conducting rigorous validation to ensure quality. We use nemotron-4-340b-instruct as the LLM
to construct our targeted Repair data. We generated 847 error reports across the three benchmarks
and produced 2,736 data samples. After filtering, this resulted in a final set of 1,406 targeted code
repair data points.

Error Report Construction To systematically address the issue, we first created a comprehen-
sive Error Report for benchmark problems using LLMs, targeting those with significant pass rate
fluctuations across training checkpoints for models on SDG data. We prompt the LLM to examine
the nature of the mistakes by comparing correct and erroneous code completions for each problem,
categorizing the errors into common error types (details in Appendix A.9). This detailed report not
only categorizes the errors but also highlights areas where the model consistently underperforms.

Targeted Code Repair Dataset Building on the error report, we further develop a targeted code
repair dataset to address these common errors. This dataset is constructed using two main sources:
the errors identified in the Error Report and correct code snippets gathered from open-source repos-
itories. We introduced the identified errors into correct code snippets to create repair problems,
which include a problem description, erroneous code implementation, and hints about the nature of
the error and how to fix it. This targeted strategy enables the model to learn how to avoid common
errors and generate improved code completions, thereby enhancing model accuracy.

Quality Assurance with LLM Validation To ensure the reliability of the error report and the code
repair dataset, we implemented a two-phase validation process with LLMs. In the first phase, we
conducted a self-consistency check of the Error Report by having the language model attempt to the
fix error code based on the report’s hints. This step verifies the accuracy of the report by confirming
that the model can resolve the errors using the provided guidance, whereas directly prompting the
LLM without detailed error reports could resolve only 13% of the errors.
In the second phase,
during the generation of the code repair dataset, we apply self-verification, including deduplication,
syntax filtering, and benchmark decontamination. These measures ensure the dataset’s quality and
uniqueness, preventing overlap with evaluation benchmarks.

4 EXPERIMENT

4.1

IMPLEMENTATION DETAILS

Training Data Our fine-tuning training data is comprised of 80.1k LLM synthetic generated data
using various prompting methods as described in Section 2.1, 28.5k data samples generated correct-
by-construction aimed at non-textual representations detailed in Section 3.1, and 1.4k carefully fil-

6

❌ Error Code ✅ Correct Codeassign output_vector ={input_vector, 1'b1, 1'b1};assign output_vector = {1'b1, 1'b1, input_vector};Volatile Training TrajectoryHere is an Verilog spec:[PROBLEM]Here is an erroneous implementation:[ERROR CODE]Here is an correct implementation:[CORRECT CODE]Now, generate a detail error report.  📄 Prompt for Error AnalysisLLM Generated Error ReportError Type: Incorrect vector concatenation and splittingCategory: Combinatorial: wiringDescription: the output vectors are assigned in the wrong order. Two 1bits should be in the LSB (least significant bit) positions not in the MSB(most significant bit) positions.Steps to Repair the Erroneous Implementation:1. Concatenating the two 1 bits at the beginning of the concatenatedvector.2. Assign the output vectors from the concatenated vector in the correctorder and bit ranges.Your goal is to create an error-fixing Verilogpractice problem for programmers. [ERROR REPORT][CODE SNIPPET]Inject the commonly made error into the abovemodule and create an error repair problem. 📄 Prompt for Error InjectionProblem DescriptionYou are given a Verilog module that demonstrates the use of bitslicing and concatenation in a loop....Erroneous Implementation...Hints for FixingEnsure explicit bit-slicing access the register correctly withoutreversing the bit order.Correct Code Solution...Targeted Code Repair Training DataRe-generate Error Report💡 Open-Source Code Snippetmodule block;    reg [31:0] data;    int i;    initial begin    data = 32'hFACE_CAFE;      ....Validated Error ReportFix error codewith error reportSelf-Consistency CheckUnder review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Table 4: We compare our models with various baseline models on VerilogEval (Liu et al., 2023b).
We update the results from Zhao et al. (2024) with the latest foundational and frontier code models.
The best results are highlighted in bold.

Type

Model

Foundational
Models

Code
Models

RTLCoder
(Liu et al., 2023c)

BetterV
(Pei et al., 2024)

CodeV
(Zhao et al., 2024)

OriGen (Cui et al., 2024)

Ours
SDG-CC-Repair

Llama-3.1
Llama-3.1
Nemotron-4
GPT-3.5-turbo
GPT-4o
CodeLlama
CodeQwen
Starcoder2
DeepSeek-Coder
DeepSeek-Coder-V2
DeepSeek-Coder-V2
Mistral
DeepSeek-Coder
CodeLlama
DeepSeek-Coder
CodeQwen
CodeLlama
DeepSeek-Coder
CodeQwen
DeepSeek-Coder
CodeLlama
DeepSeek-Coder
Starcoder2

VerilogEval (Liu et al., 2023b)

Size

8B
405B
340B
-
-
7B
7B
15B
6.7B
16B
236B
7B
7B
7B
6.7B
7B
7B
6.7B
7B
6.7B
7B
6.7B
15B

pass@1
48.7
67.3
53.0
58.0
65.9
43.1
46.5
68.7
52.2
67.4
68.2
62.5
61.2
64.2
67.8
68.1
78.1
77.9
77.6
74.1
78.1
77.8
81.9

Machine (%)
pass@5
67.3
75.1
60.3
74.0
71.4
47.1
54.9
82.3
55.4
78.3
74.1
72.2
76.5
75.4
79.1
79.4
86.0
88.6
88.2
82.4
85.5
85.5
86.9

pass@10
74.1
76.9
62.2
77.6
72.7
47.7
56.4
88.5
56.8
81.8
76.2
76.6
81.8
79.1
84.0
84.5
88.5
90.7
90.7
85.7
87.8
88.1
88.1

pass@1
26.9
53.8
43.1
31.2
57.1
18.2
22.5
37.7
30.2
46.9
56.4
36.7
41.6
40.9
45.9
46.1
45.2
52.7
53.2
54.4
63.1
65.4
68.0

Human (%)
pass@5
37.8
61.0
48.3
44.1
63.9
22.7
26.1
50.6
33.9
55.9
62.2
45.5
50.1
50.0
53.3
53.7
59.5
62.5
65.1
60.1
67.8
70.0
72.4

pass@10
44.2
62.8
50.0
47.4
66.7
24.3
28.0
57.2
34.9
58.9
66.0
49.2
53.4
53.3
57.6
58.2
63.8
67.3
68.5
64.2
69.7
72.1
74.6

tered data for targeted code repair as outlined in Section 3.2. We refer to each data set as SDG, CC,
and Repair, respectively.

Pretrained Models Following prior work, we use CodeLlama-7b-Instruct (Roziere et al., 2023)
and Deepseek-Coder-6.7b-Instruct (Guo et al., 2024) as the base model, formatting our data accord-
ing to their default chat prompt templates. Additionally, we explore the Starcoder2-15B (Lozhkov
et al., 2024) model in our experiments.

Model Training Training is conducted with 32 NVIDIA A100-80GB GPUs through the Dis-
tributed Data Parallel (DDP) module from PyTorch. We set the learning rate at 5e-5 for CodeLlama
and DeepSeek-Coder, and 1e-5 for Starcoder2. We use Adam (Kingma & Ba, 2017) as our opti-
mizer with full parameter updates and truncate sequence lengths longer than 4096 tokens. We used
a batch size of 256 samples. We fine-tune models for 1 epoch using a standard cross entropy loss on
the response tokens (while masking loss on prompt tokens).

Model Inference We use vLLM (Kwon et al., 2023) where the inference engine is set up with
bf16 dtype, tensor parallel size of 8, and a maximum token limit of 4096. We sample each problem
20 times. We report the best results from two different temperatures 0.2 and 0.8, as consistent with
prior work (Liu et al., 2023c; Zhao et al., 2024).

4.2 EVALUATION METRIC AND BENCHMARK

Evaluation Metric Following prior work (Chen et al., 2021; Liu et al., 2023a), for each experiment
we use the unbiased pass@k metric to measure the Verilog generation accuracy. The pass@k metric
estimates the proportion of problems that can be solved at least once in k attempts:

pass@k := EProblems

1 −

(cid:34)

(cid:35)

(cid:1)

,

(cid:0)n−c
k
(cid:1)
(cid:0)n
k

(1)

where n ≥ k represents the total number of trials for each problem, and c represents the number of
trials that pass the functional check.

VerilogEval (Liu et al., 2023b) contains two subsets of problems, where VerilogEval-Human con-
tains manually converted problem descriptions from the original HDLBits website, and VerilogEval-
Machine with GPT-3.5 generated problem descriptions.

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 5: Evaluations on RTLLM v1.1 (Lu et al., 2024) using unbiased pass@k metrics. The best
results are highlighted in bold. We re-evaluate all models (see Appendix A for details).

Type

Model

Foundational
Models

Code
Models

RTLCoder
(Liu et al., 2023c)

CodeV
(Zhao et al., 2024)

OriGen (Cui et al., 2024)

Ours
SDG-CC-Repair

Llama-3.1
Llama-3.1
Nemotron-4
GPT-3.5-turbo
GPT-4o
CodeLlama
CodeQwen
Starcoder2
DeepSeek-Coder
DeepSeek-Coder-V2
DeepSeek-Coder-V2
Mistral
DeepSeek-Coder
CodeLlama
DeepSeek-Coder
CodeQwen
DeepSeek-Coder
CodeLlama
DeepSeek-Coder
Starcoder2

RTLLM v1.1 (Lu et al., 2024)

Size

8B
405B
340B
-
-
7B
7B
15B
6.7B
16B
236B
7B
6.7B
7B
6.7B
7B
6.7B
7B
6.7B
15B

pass@1
40.7
56.5
41.7
50.3
50.3
46.6
45.8
38.3
51.4
51.4
63.4
64.6
73.4
79.0
78.3
78.8
-
85.7
84.3
79.8

Syntax (%)
pass@5
60.6
64.4
47.2
61.2
59.9
62.6
65.8
81.0
64.4
57.8
78.1
73.7
83.9
89.2
87.4
89.5
-
93.9
92.9
93.9

pass@10
65.5
72.4
48.3
65.5
62.1
68.9
72.4
94.7
68.9
58.6
79.3
78.3
86.2
89.9
89.1
92.4
-
94.8
95.4
96.2

pass@1
19.3
38.9
18.9
28.3
33.8
17.9
24.1
15.5
23.1
33.1
34.5
24.5
35.8
39.4
42.4
36.6
-
42.6
53.1
49.0

Func. (%)
pass@5
34.7
45.8
20.7
36.9
44.4
29.9
34.0
37.6
29.3
37.1
50.2
37.3
40.3
50.3
51.5
53.3
65.5
52.9
58.8
65.8

pass@10
37.9
51.7
20.7
41.4
48.3
34.5
37.9
45.7
34.5
37.9
55.1
42.3
43.1
53.1
53.2
61.3
-
58.2
62.6
74.5

RTLLM (Lu et al., 2024) is an open-source benchmark designed for generating Register Transfer
Level (RTL) code from natural language instructions. It evaluates models on syntax correctness,
functional correctness, and design quality, offering a thorough analysis of model outputs.

4.3 RESULTS

Main Results Table 4 and Table 5 compare our models with baselines on VerilogEval and
RTLLM. We mainly source baseline results from Zhao et al. (2024). For RTLLM we found a
large variance with biased pass@5, thus we re-evalaute all models and report unbiased pass@k
metric. We further rigorously evaluate the latest foundational and frontier code models, including
Llama-3.1 (Dubey et al., 2024), DeepSeek-Coder-V2 (DeepSeek-AI et al., 2024), and GPT-4o. Re-
cent foundational and frontier code models already reached competitive performance compared to
previous efforts targeting Verilog code generation.

Compared to previous approaches like CodeV (Zhao et al., 2024), our models achieve compara-
ble performance on VerilogEval-Machine and show significant improvements on benchmarks with
human-like descriptions. Machine descriptions often provide detailed, line-by-line coding instruc-
tions, whereas human descriptions are high-level, integrating problem-solving skills and a deeper
understanding of the hardware module’s functionality. Enhancing the model’s ability to handle
human-like descriptions is crucial, as these more accurately reflect how designers interact with the
models and set expectations for Verilog generation. Our fine-tuned Starcoder2-15B surpasses previ-
ous state-of-the-art results by 3.8%, 10.9%, and 6.6% in pass@1 metrics on VerilogEval-Machine,
VerilogEval-Human, and RTLLM, respectively.

Table 6 highlights the effectiveness of our generated
data fine-tuned on Starcoder2-15B. Our CC data en-
hances the model’s ability to handle non-textual repre-
sentations, leading to improved scores on VerilogEval-
Human. Our targeted code Repair data boosts per-
formance across all benchmarks, suggesting that the
model has learned to generalize from code repair tasks
and reduce similar errors during code completion.

Table 6: Ablation study on training data.
Data quantity indicated in parentheses.

Model

VerilogEval
Machine Human
pass@1 (%)

Starcoder2-15B
SDG (80.1k)
SDG-CC (108.6k)
SDG-CC-Repair (110.0k)

68.7
75.2
73.9
81.9

37.7
54.7
62.0
68.0

RTLLM v1.1
Func
pass@5 (%)
37.6
62.1
62.8
65.8

Improved Variability During Training Figure 1b displays the pass rates for two consecutive
checkpoints of Starcoder2-SDG-CC-Repair on VerilogEval-Human problems, sampled with a tem-
perature of 0.8. Compared to Figure 1a, the updated model shows significant improvements by (1)
moving previously unsolved problems into the solved category, including those with non-textual
representations addressed by our correct-by-construction CC data, and (2) reducing the number of
problems with large pass rate discrepancies, particularly where performance had degraded. The tar-

8

Under review as a conference paper at ICLR 2025

geted repair data has effectively mitigated the model’s tendency to repeat common mistakes found
in our Repair dataset, despite the noise inherent in synthetically generated SDG data.

Scaling Data
for Non-textual Representa-
tions Figure 4 illustrates the scaling of correct-
by-construction (CC) data and the fine-tuned
Starcoder2-15B pass rate on problems involving
non-textual representations. We expanded our test-
ing to include strictly in-distribution test set, with
each category containing around 50 problems. The
results show that the model can quickly learn and
comprehend these non-textual representations with
as few as 4k training data samples, with the pass
rate steadily improving as more data is provided.
Additionally,
the model demonstrates the ability
to generalize to VerilogEval-NonText benchmark
problems. While our models achieve near-perfect
scores on KMap and FSM problems, they perform
less effectively on Waveforms,
suggesting that
reverse engineering circuits from waveforms pose a
greater challenge.

1
@

s
s
a
p

1

0.8

0.6

0.4

0.2

0

0.956

0.965

0.970

0.986

1.00

1.00 1.00

0.994 1.00

0.962

0.846

0.931

0.710

0.722

0.731

0.7510.761

0.799

0.688

0.639

0.658

0.6760.664

0.579

0.593

KMap
FSM
Waveform
VerilogEval-NonText

0.760

0.645

0.551

0.103
0.050
0.000

5

0
15
Total Number of Targeted Data Samples (k)

20

25

10

30

Figure 4: pass@1 on non-textual problems
with total number of CC data with tempera-
ture 0.8.

Table 7: Ablation study on Repair data qual-
ity with Starcoder2-15B.

Ensuring Quality for Targeted Code Repair
The quality control mechanisms integrated into the
data generation pipeline are crucial for improving
model performance, particularly in correcting minor
errors through targeted code repair. To evaluate the
impact of these quality controls, we conducted an
ablation study in Table 7, where we systematically
removed each component of the targeted code repair
generation pipeline and assessed the resulting model
performance. Specifically, we eliminated the self-consistency checks that validate whether the gen-
erated error report effectively guides the LLMs in correcting mistakes. Additionally, we tested the
removal of the error report entirely, substituting it with random errors injected into the open-source
code by the LLMs. The benchmark results indicate a significant performance drop when these vali-
dation processes are excluded. These findings highlight the essential role of both the self-consistency
checks and the targeted error report in improving the model’s ability to correct errors.

RTLLM v1.1
Func
pass@5 (%)
62.8
65.8
63.7
59.4

SDG-CC
SDG-CC-Repair
w/o self-consistency
w/o error report

VerilogEval
Machine Human
pass@1 (%)

62.0
68.0
63.3
59.6

73.9
81.9
75.3
76.9

Model

5 RELATED WORK

Synthetic Data Generation for Model Fine-tuning. The performance of large language models
(LLMs) hinge on the quality and diversity of their training data. To address the limitations of manual
datasets, synthetic data generation methods (Wang et al., 2022; Xu et al., 2023) have been developed
to automatically create instruction-following examples from LLMs, reducing reliance on human an-
notations. Various techniques enhance data quality: Wang et al. (2022) generates multiple reasoning
traces and selects the most frequent output to improve robustness, while other approaches (Light-
man et al., 2023; Zhang et al., 2024b) assess response quality based on these traces. Self-training
methods utilize synthetic data for iterative fine-tuning, boosting reasoning capabilities (Singh et al.,
2023; Feng et al., 2023). These advancements show how synthetic data can effectively scale and
optimize models through iterative feedback.

Large Language Models for Code Generation. Recent breakthroughs in large language models
(LLMs) have greatly enhanced their capability to tackle complex code generation tasks. Much of
the research focuses on developing LLMs specialized for code by continuing their pretraining on
code data (Guo et al., 2024; Bai et al., 2023; Roziere et al., 2023; DeepSeek-AI et al., 2024) from
open-source repositories like GitHub (Kocetkov et al., 2022; Lozhkov et al., 2024) and commit
histories (Muennighoff et al., 2023). Further improvements to these models come from reinforce-
ment learning (Le et al., 2022) and more often instruction fine-tuning, which involves techniques

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

to address more complex coding problems (Luo et al., 2024b), increasing diversity with unlabeled
open-source code (Wei et al., 2023; Yu et al., 2024; Wu et al., 2024), ensuring solution correctness
through self-written tests (Chen et al., 2022), and validating and debugging code execution through
interactions with LLM agents (Lei et al., 2024).

Large Language Models for Verilog Coding. While most code LLMs target software languages,
there is increasing interest in models for hardware description languages like Verilog, essential for
chip design and verification (Liu et al., 2024). Previous work has addressed the challenge of limited
data through various methods, including synthetic data generation (Liu et al., 2023c), multi-level
summarization of open-source Verilog code (Zhao et al., 2024), and enhanced code augmentation
with self-reflection based on compiler feedback (Tsai et al., 2023; Cui et al., 2024). Other ap-
proaches focus on improving functional correctness and circuit performance through Monte Carlo
Tree Search (DeLorenzo et al., 2024) and discriminator-guided sampling (Pei et al., 2024).

6 DISCUSSIONS

In this work, we refer to synthetic data generation as methods of using large language mod-
els (LLMs) in data generation. While our approach—ensuring correctness through correct-by-
construction—could also be considered “synthetic” and resembles methods explored in works like
AlphaGeometry (Trinh et al., 2024), our problems are much simpler and on a smaller scale. Our
observations about the variability of models on specific problems align with the findings of Meta AI
(2024), where “the model knows how to produce the right answer, but it does not know how to se-
lect it.” Instead of striving for absolute data correctness, preference learning (Rafailov et al., 2024;
Ethayarajh et al., 2024) or reinforcement learning (Bai et al., 2022; Le et al., 2022), we generate
targeted repair data by analyzing errors and re-create such scenarios by injecting similar errors into
open-source code, somewhat analogous to how humans consolidate memories during sleep by inte-
grating new information with past experiences (Walker & Stickgold, 2004; Stickgold, 2005). Further
discussions on the generalizability and broader impact of our work are provided in Appendix B.

7 CONCLUSION

This paper addresses key challenges in Verilog code generation with correct-by-construction data
generation and targeted code repair data strategies. We identified significant issues with synthetic
data generation, including difficulties with non-textual representations and variability in perfor-
mance during training across benchmarks. To address these challenges, we generated data that
is correct-by-construction and create targeted repair data by injecting errors to open-source code.
Our approach led to substantial improvements, with models fine-tuned using our methods achieving
state-of-the-art results on VerilogEval and RTLLM benchmarks. These advancements highlight the
effectiveness of our strategies in enhancing model performance in Verilog code generation.

Reproducibility Statement We provide the following details: evaluation benchmarks in Ap-
pendix A.3, examples of the process for generating targeted code repair data in Appendix C, and
data examples from correct-by-construction targeting non-textual representations in Appendix D.
Additionally, we include prompt templates used for data generation in Appendix E. To enhance re-
producibility, we are committed to release the source code of our data generation pipeline, including
synthetic data generation methods (Section 2.1), correct-by-construction data targeting non-textual
representations (Section 3.1), and targeted code repair (Section 3.2). However, for this submission,
we chose not to include source code, as we are unable to provide an appropriate license in compli-
ance with the double-blind review policy.

REFERENCES

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale-
man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical
report. arXiv preprint arXiv:2303.08774, 2023.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge,

Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn
Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless
assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,
2022.

Christopher Batten, Nathaniel Pinckney, Mingjie Liu, Haoxing Ren, and Brucek Khailany. Pyhdl-
eval: An llm evaluation framework for hardware design using python-embedded dsls. In Pro-
ceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD,
MLCAD ’24, New York, NY, USA, 2024. Association for Computing Machinery.
ISBN
9798400706998. doi: 10.1145/3670474.3685948. URL https://doi.org/10.1145/
3670474.3685948.

Jitendra Bhandari, Johann Knechtel, Ramesh Narayanaswamy, Siddharth Garg, and Ramesh Karri.
Llm-aided testbench generation and bug detection for finite-state machines, 2024. URL https:
//arxiv.org/abs/2406.17132.

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald
Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha,
Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and extensible approach to bench-
marking neural code generation, 2022. URL https://arxiv.org/abs/2208.08227.

Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Car-
olyn Jane Anderson, Molly Q Feldman, Michael Greenberg, Abhinav Jangda, and Arjun Guha.
Knowledge transfer from high-resource to low-resource programming languages for code llms,
2024. URL https://arxiv.org/abs/2308.09895.

Kaiyan Chang, Zhirong Chen, Yunhao Zhou, Wenlong Zhu, kun wang, Haobo Xu, Cangyuan Li,
Mengdi Wang, Shengwen Liang, Huawei Li, Yinhe Han, and Ying Wang. Natural language is not
enough: Benchmarking multi-modal generative ai for verilog generation, 2024. URL https:
//doi.org/10.1145/3676536.3676679.

Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https:

//github.com/sahil280114/codealpaca, 2023.

Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu
Chen. Codet: Code generation with generated tests, 2022. URL https://arxiv.org/abs/
2207.10397.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared
Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large
language models trained on code. arXiv preprint arXiv:2107.03374, 2021.

Fan Cui, Chenyang Yin, Kexing Zhou, Youwei Xiao, Guangyu Sun, Qiang Xu, Qipeng Guo, Demin
Song, Dahua Lin, Xingcheng Zhang, et al. Origen: Enhancing rtl code generation with code-to-
code augmentation and self-reflection. arXiv preprint arXiv:2407.16237, 2024.

DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu,
Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai
Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang,
Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao,
Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan,
Fuli Luo, and Wenfeng Liang. Deepseek-coder-v2: Breaking the barrier of closed-source models
in code intelligence, 2024. URL https://arxiv.org/abs/2406.11931.

Matthew DeLorenzo, Animesh Basak Chowdhury, Vasudev Gohil, Shailja Thakur, Ramesh Karri,
Siddharth Garg, and Jeyavijayan Rajendran. Make every move count: Llm-based high-quality rtl
code generation using mcts, 2024. URL https://arxiv.org/abs/2402.03289.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha
Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony
Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark,
Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere,

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris
Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong,
Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny
Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino,
Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael
Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander-
son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah
Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan
Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma-
hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy
Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak,
Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al-
wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini,
Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der
Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo,
Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man-
nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova,
Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal,
Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur
Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhar-
gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong,
Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic,
Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum-
baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa,
Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang,
Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende,
Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney
Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom,
Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta,
Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petro-
vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang,
Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur,
Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre
Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha
Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay
Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda
Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew
Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita
Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh
Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De
Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran-
don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina
Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai,
Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li,
Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana
Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil,
Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar-
caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco
Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella
Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory
Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang,
Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold-
man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman,
James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer
Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe
Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie
Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun
Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal
Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva,

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian
Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson,
Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke-
neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel
Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo-
hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy-
ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong,
Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli,
Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux,
Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao,
Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li,
Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott,
Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa-
tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind-
say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang
Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen
Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho,
Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser,
Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim-
othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan,
Vinay Satish Kumar, Vishal Mangla, Vítor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu
Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con-
stable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu,
Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi,
Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef
Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024.
URL https://arxiv.org/abs/2407.21783.

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model

alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.

Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-
arXiv preprint

like tree-search can guide large language model decoding and training.
arXiv:2309.17179, 2023.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao
Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–
the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024.

René Just, Darioush Jalali, and Michael D. Ernst. Defects4j: a database of existing faults to enable
controlled testing studies for java programs. In Proceedings of the 2014 International Symposium
on Software Testing and Analysis, ISSTA 2014, pp. 437–440, New York, NY, USA, 2014. Asso-
ciation for Computing Machinery. ISBN 9781450326452. doi: 10.1145/2610384.2628055. URL
https://doi.org/10.1145/2610384.2628055.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL

https://arxiv.org/abs/1412.6980.

Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis,
Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von
Werra, and Harm de Vries. The stack: 3 tb of permissively licensed source code, 2022. URL
https://arxiv.org/abs/2211.15533.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E.
Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model
serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating
Systems Principles, 2023.

Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl:
Mastering code generation through pretrained models and deep reinforcement learning, 2022.
URL https://arxiv.org/abs/2207.01780.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Bin Lei, Yuchen Li, and Qiuwu Chen. Autocoder: Enhancing code large language model with

AIEV-INSTRUCT, 2024. URL https://arxiv.org/abs/2405.14906.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao
Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii,
Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João
Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Lo-
gesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra
Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey,
Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luc-
cioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor,
Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex
Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva
Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes,
Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source
be with you!, 2023. URL https://arxiv.org/abs/2305.06161.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan
Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint
arXiv:2305.20050, 2023.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by
chatGPT really correct? rigorous evaluation of large language models for code generation. In
Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https:
//openreview.net/forum?id=1qvx610Cu7.

Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. Verilogeval: Evaluating large
language models for verilog code generation. In 2023 IEEE/ACM International Conference on
Computer Aided Design (ICCAD), pp. 1–8. IEEE, 2023b.

Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang,
Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran,
Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande,
Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Ankit Jindal, Brucek
Khailany, George Kokai, Kishor Kunal, Xiaowei Li, Charley Lind, Hao Liu, Stuart Oberman,
Sujeet Omar, Ghasem Pasandi, Sreedhar Pratty, Jonathan Raiman, Ambar Sarkar, Zhengjiang
Shao, Hanfei Sun, Pratik P Suthar, Varun Tej, Walker Turner, Kaizhe Xu, and Haoxing Ren.
Chipnemo: Domain-adapted llms for chip design, 2024. URL https://arxiv.org/abs/
2311.00176.

Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, and Zhiyao Xie. Rtlcoder: Outper-
forming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution.
arXiv preprint arXiv:2312.08617, 2023c.

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane
Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The
next generation. arXiv preprint arXiv:2402.19173, 2024.

Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. Rtllm: An open-source benchmark for design rtl
generation with large language model. In 2024 29th Asia and South Pacific Design Automation
Conference (ASP-DAC), pp. 722–727. IEEE, 2024.

Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study
of catastrophic forgetting in large language models during continual fine-tuning, 2024a. URL
https://arxiv.org/abs/2308.08747.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing
Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with
evol-instruct. In The Twelfth International Conference on Learning Representations, 2024b. URL
https://openreview.net/forum?id=UnUwSIgK5W.

Meta AI. Introducing meta llama 3: The most capable openly available llm to date, 2024. URL

https://ai.meta.com/blog/meta-llama-3/. Accessed: 2024-09-10.

14

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo,
Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruc-
tion tuning code large language models. arXiv preprint arXiv:2308.07124, 2023.

Daniel Nichols, Joshua H. Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele. Can large
language models write parallel code? In Proceedings of the 33rd International Symposium on
High-Performance Parallel and Distributed Computing, HPDC ’24, pp. 281–294, New York,
NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704130. doi: 10.1145/
3625549.3658689. URL https://doi.org/10.1145/3625549.3658689.

Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika
Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush
Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Alek-
sander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grze-
gorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John
Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen
Long, Ameya Sunil Mahabaleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez,
Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro,
Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupin-
der Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy,
Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Se-
wall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe
Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shub-
ham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang,
Vivienne Zhang, Yian Zhang, and Chen Zhu. Nemotron-4 340b technical report, 2024. URL
https://arxiv.org/abs/2406.11704.

Zehua Pei, Hui-Ling Zhen, Mingxuan Yuan, Yu Huang, and Bei Yu. Betterv: Controlled verilog

generation with discriminative guidance. arXiv preprint arXiv:2402.03375, 2024.

Ruidi Qiu, Grace Li Zhang, Rolf Drechsler, Ulf Schlichtmann, and Bing Li. Autobench: Automatic
testbench generation and evaluation using llms for hdl design, 2024. URL https://arxiv.
org/abs/2407.03891.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model. Advances
in Neural Information Processing Systems, 36, 2024.

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi
Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code.
arXiv preprint arXiv:2308.12950, 2023.

Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J Liu, James
Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training
for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023.

Yewei Song, Cedric Lothritz, Daniel Tang, Tegawendé F. Bissyandé, and Jacques Klein. Revisiting
code similarity evaluation with abstract syntax tree edit distance, 2024. URL https://arxiv.
org/abs/2404.08817.

Robert Stickgold. Sleep-dependent memory consolidation. Nature, 437(7063):1272–1278, 2005.

Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D. Cox, and Akash
Srivastava. Lab: Large-scale alignment for chatbots, 2024. URL https://arxiv.org/
abs/2403.01081.

Shinya Takamaeda-Yamazaki.
toolkit for verilog hdl.
ture Notes in Computer Science, pp. 451–460. Springer
2015.
978-3-319-16214-0_42.

Pyverilog: A python-based hardware design processing
In Applied Reconfigurable Computing, volume 9040 of Lec-
International Publishing, Apr
doi: 10.1007/978-3-319-16214-0_42. URL http://dx.doi.org/10.1007/

15

Under review as a conference paper at ICLR 2025

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Ali TehraniJamsaz, Arijit Bhattacharjee, Le Chen, Nesreen K. Ahmed, Amir Yazdanbakhsh, and
Ali Jannesari. Coderosetta: Pushing the boundaries of unsupervised code translation for parallel
programming, 2024. URL https://arxiv.org/abs/2410.20527.

Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry

without human demonstrations. Nature, 625(7995):476–482, 2024.

YunDa Tsai, Mingjie Liu, and Haoxing Ren. Rtlfixer: Automatically fixing rtl syntax errors with

large language models. arXiv preprint arXiv:2311.16543, 2023.

Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys
Poshyvanyk. An empirical study on learning bug-fixing patches in the wild via neural machine
translation. ACM Trans. Softw. Eng. Methodol., 28(4), September 2019. ISSN 1049-331X. doi:
10.1145/3340544. URL https://doi.org/10.1145/3340544.

Sleep-dependent learning and memory consolida-
Matthew P. Walker and Robert Stickgold.
tion. Neuron, 44(1):121–133, 2004.
ISSN 0896-6273. doi: https://doi.org/10.1016/j.neuron.
2004.08.031. URL https://www.sciencedirect.com/science/article/pii/
S0896627304005409.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and
Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.
arXiv preprint arXiv:2212.10560, 2022.

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code

is all you need. arXiv preprint arXiv:2312.02120, 2023.

Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun
Zhao. Large language models are better reasoners with self-verification, 2023. URL https:
//arxiv.org/abs/2212.09561.

Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-
Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein,
Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for large-scale
transformer training instabilities, 2023. URL https://arxiv.org/abs/2309.14322.

Yutong Wu, Di Huang, Wenxuan Shi, Wei Wang, Lingzhe Gao, Shihao Liu, Ziyuan Nan, Kaizhao
Yuan, Rui Zhang, Xishan Zhang, Zidong Du, Qi Guo, Yewen Pu, Dawei Yin, Xing Hu, and Yunji
Chen. Inversecoder: Unleashing the power of instruction-tuned code llms with inverse-instruct,
2024. URL https://arxiv.org/abs/2407.05700.

Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era
In 2023 IEEE/ACM 45th International Conference on

of large pre-trained language models.
Software Engineering (ICSE), pp. 1482–1494, 2023. doi: 10.1109/ICSE48619.2023.00129.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and
Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions.
arXiv preprint arXiv:2304.12244, 2023.

Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and
Qiufeng Yin. Wavecoder: Widespread and versatile enhancement for code large language models
by instruction tuning, 2024. URL https://arxiv.org/abs/2312.14187.

Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The

effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193, 2024a.

Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-

training via process reward guided tree search. arXiv preprint arXiv:2406.03816, 2024b.

Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang, Jing Su, Yongfei Liu, and Zhi Jin.
Codedpo: Aligning code models with self generated and verified source code, 2024c. URL
https://arxiv.org/abs/2410.05605.

Yang Zhao, Di Huang, Chongxiao Li, Pengwei Jin, Ziyuan Nan, Tianyun Ma, Lei Qi, Yansong
Pan, Zhenxing Zhang, Rui Zhang, et al. Codev: Empowering llms for verilog generation through
multi-level summarization. arXiv preprint arXiv:2407.10424, 2024.

16

Under review as a conference paper at ICLR 2025

A DETAILED RESULTS

A.1 OUR MODELS

We present our models’ results on Verilog benchmarks tested with temperatures 0.2 and 0.8. We
ablate across different data blends, with SDG indicating using LLM synthetic generated data in Sec-
tion 2.1, CC indicating correct-by-construction data targeting non-textual representations in Sec-
tion 3.1, and Repair representing our targeted code repair dataset in Section 3.2.

Our results for RTLLM use the open-source Icarus Verilog simulator1 to check syntax and functional
pass rates. This might lead to lower pass rate scores compared to previous work that used Synopsys
VCS, as Icarus Verilog does not support all syntax.

Table 8: Results for our models, across different dataset and temperature on VerilogEval.

Model

Dataset

Temperature

SDG

Starcoder2-15b

SDG-CC

SDG-CC-Repair

SDG

DeepSeek-6.7b-Instruct

SDG-CC

SDG-CC-Repair

SDG

CodeLlama-7b-Instruct

SDG-CC

SDG-CC-Repair

0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8

VerilogEval (Liu et al., 2023b)

Machine (%)
pass@5
79.2
84.0
78.1
84.1
84.2
86.9
77.8
82.5
78.2
83.1
82.7
85.5
77.9
82.6
77.4
81.2
81.5
85.5

pass@10
80.1
86.1
79.5
87.1
85.0
88.1
78.9
85.4
79.3
85.4
83.4
88.1
78.8
85.1
78.1
83.7
81.7
87.8

pass@1
75.2
73.7
73.9
72.9
81.9
78.1
73.4
71.4
72.6
70.2
77.8
75.2
74.5
71.2
74.2
70.0
78.1
73.7

Human (%)
pass@5
60.1
61.9
65.6
70.3
71.7
72.4
53.2
58.1
62.6
67.0
67.7
70.0
50.3
55.6
61.0
64.4
66.2
67.8

pass@10
61.2
64.8
67.0
73.7
72.0
74.6
54.5
62.3
63.5
70.7
68.2
72.1
51.5
59.0
62.4
67.7
66.8
69.7

pass@1
54.7
47.4
62.0
58.5
68.0
64.1
48.3
44.0
58.5
56.3
65.4
61.6
45.3
42.6
55.1
51.6
63.1
58.1

Table 9: Results for our models, across different dataset and temperature on RTLLM.

Model

Dataset

Temperature

SDG

Starcoder2-15b

SDG-CC

SDG-CC-Repair

SDG

DeepSeek-6.7b-Instruct

SDG-CC

SDG-CC-Repair

SDG

CodeLlama-7b-Instruct

SDG-CC

SDG-CC-Repair

0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8

RTLLM v1.1 Lu et al. (2024)

Syntax (%)
pass@5
86.5
89.0
89.3
92.6
87.9
93.9
86.8
92.5
84.5
90.5
92.2
92.9
82.5
89.1
90.2
93.9
93.9
93.9

pass@10
90.1
94.1
92.7
95.5
90.5
96.2
90.5
96.2
86.0
93.8
93.0
95.4
86.8
94.5
94.6
96.3
94.8
94.8

pass@1
78.1
77.1
78.3
76.9
79.8
79.3
79.3
76.6
73.6
76.7
84.3
80.0
74.0
70.9
75.0
76.4
85.7
80.3

pass@1
49.0
43.8
45.5
38.4
49.0
45.3
40.3
40.0
44.3
39.5
53.1
45.5
30.0
34.0
39.7
35.5
42.6
36.9

Func. (%)
pass@5
60.4
62.1
58.3
62.8
59.1
65.8
45.9
53.8
52.2
56.4
58.8
57.9
33.9
47.2
44.4
47.6
49.4
52.9

pass@10
66.3
68.0
62.0
70.4
62.6
74.5
49.6
63.6
54.3
63.1
60.3
62.6
35.8
52.8
47.2
52.7
51.2
58.2

1https://github.com/steveicarus/iverilog

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

A.2 FOUNDATIONAL AND FRONTIER CODE MODELS

We present detailed results on recent foundational and frontier code models. We also re-evaluate all
models on RTLLM using unbiased pass@k metric.

Table 10: Results on foundational and code models on VerilogEval.

Type

Model

Size

Temp

Foundational
Models

Llama-3.1

Llama-3.1

8B

70B

Llama-3.1

405B

Nemotron-4

340B

GPT-3.5-turbo

GPT-4

GPT-4-turbo

GPT-4o

-

-

-

-

Code
Models

Starcoder2

15B

DeepSeek-Coder-V2

16B

DeepSeek-Coder-V2

236B

0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8

VerilogEval (Liu et al., 2023b)

Machine (%)
pass@5
66.2
67.3
73.8
77.7
72.8
75.1
59.1
60.3
66.4
74.0
63.7
53.4
66.7
69.5
68.9
71.4
76.7
82.3
74.6
78.3
72.7
74.1

pass@10
70.6
74.1
76.9
80.4
74.1
76.9
61.5
62.2
68.5
77.6
66.4
58.9
70.6
73.4
69.2
72.7
78.6
88.5
76.2
81.8
75.0
76.2

pass@1
48.7
42.1
66.7
64.5
67.3
66.4
53.0
50.8
58.0
56.6
53.2
35.3
57.8
56.9
65.9
62.9
68.7
57.7
67.4
65.6
68.2
66.5

Human (%)
pass@5
36.9
37.8
53.6
57.0
57.0
61.0
43.9
48.3
39.4
44.1
43.5
53.4
61.2
63.6
61.3
63.9
48.3
50.6
53.3
55.9
60.7
62.2

pass@10
40.4
44.2
55.1
60.9
58.9
62.8
44.9
50.0
41.7
47.4
46.2
58.9
62.8
66.7
62.2
66.7
51.1
57.2
54.5
58.9
64.3
66.0

pass@1
26.9
23.0
48.7
48.0
51.9
53.8
43.1
40.8
31.2
28.9
36.1
35.2
54.1
53.6
57.1
55.4
37.7
29.1
46.9
46.3
56.4
54.8

Table 11: Results on foundational and code models on RTLLM.

RTLLM v1.1 (Lu et al., 2024)

Type

Model

Size

Temp

Foundational
Models

Code
Models

Llama-3.1

Llama-3.1

8B

70B

Llama-3.1

405B

Nemotron-4

340B

GPT-3.5-turbo

GPT-4

GPT-4-turbo

GPT-4o

CodeLlama

CodeQwen

-

-

-

-

7B

7B

Starcoder2

15B

DeepSeek-Coder

6.7B

DeepSeek-Coder-V2

16B

DeepSeek-Coder-V2

236B

0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8

0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8
0.2
0.8

pass@1
19.3
17.6
34.1
29.6
38.9
35.8
14.1
18.9
28.3
24.1
30.0
25.9
27.2
27.5
33.8
31.3

17.9
13.4
24.1
22.4
15.5
11.0
23.1
21.0
33.1
30.0
34.5
32.9

Func. (%)
pass@5
25.8
34.7
34.5
31.0
45.0
45.8
15.5
20.7
36.9
36.9
44.4
40.0
35.1
40.2
44.4
44.1

pass@10
27.6
37.9
34.5
31.0
48.3
51.7
17.2
20.7
41.4
41.4
48.3
44.8
37.9
44.8
48.3
48.3

29.9
25.9
33.1
34.0
37.6
34.2
26.8
29.3
34.5
37.1
44.9
50.2

34.5
31.0
37.9
37.9
44.6
45.7
27.6
34.5
34.5
37.9
52.9
55.1

pass@1
39.7
40.7
47.9
48.9
56.5
52.1
41.7
41.7
50.3
48.2
49.3
42.8
38.9
40.3
50.3
47.5

Syntax (%)
pass@5
53.1
60.6
51.7
57.6
63.9
64.4
47.2
46.3
58.2
61.2
65.9
61.2
44.8
48.8
59.9
63.2

pass@10
55.2
65.5
55.2
58.6
65.5
72.4
48.3
48.3
58.6
65.5
68.9
65.5
48.3
51.7
62.1
66.7

62.6
59.7
55.8
65.7
77.5
81.0
62.6
64.4
51.7
57.8
73.0
78.1

68.9
68.9
58.6
72.4
86.3
94.7
65.5
68.9
51.7
58.6
79.3
79.3

46.6
34.8
45.8
45.5
38.3
31.6
51.4
49.7
51.4
51.4
63.4
61.8

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

A.3 DETAILS ON EVALUATIONS

We format the prompt input as follows for VerilogEval, where the detail_description is the problem
description (Machine or Human) and prompt field is the problem module header. We include module
headers to avoid confusion on the signals naming.

prompt = f"{task[’detail_description’].strip()}\n\n{task[’prompt’].strip()}"

An example of mux2to1 in VerilogEval-Human:
Create a 2−1 multiplexer. When sel=0, choose a. When sel=1, choose b.

module top_module (

input a,
input b,
input sel,
output out

);

We use similar templates for RTLLM v1.1, where we extract the top module header from the refer-
ence solution and provide it as input. Below is an example of adder_8bit:

Please act as a professional verilog designer.

Implement a module of an 8−bit adder with multiple bit−level adders in combinational

logic.

Module name:

adder_8bit

Input ports:

a[7:0]: 8−bit input operand A.
b[7:0]: 8−bit input operand B.
cin: Carry−in input.

Output ports:

sum[7:0]: 8−bit output representing the sum of A and B.
cout: Carry−out output.

Implementation:
The module utilizes a series of bit−level adders (full adders) to perform the addition

operation.

Give me the complete code.

module adder_8bit(

input [7:0] a, b,
input cin,
output [7:0] sum,
output cout);

We use default chat templates and default system prompts for open-source models tested. For GPT
models from OpenAI, we use the following system prompt:

Please act as a professional verilog designer.

We post-process model responses to extract code. We extract content enclosed by triple backticks
and remove the language identifier (Verilog). We then extract code enclosed in module and endmodule
keywords with response.find(’module’) and response.rfind(’endmodule’). If the extracted code
does not include a module header, the reference solution’s module header will be prepended. The
code is then tested with the provided testbenches with the Icarus Verilog (iverilog) simulator to
evaluate for syntax and functional correctness. This might lead to lower pass rate scores for RTLLM
compared to previous work that used Synopsys VCS, as Icarus Verilog does not support all syntax.

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

A.4 VERILOGEVAL-NONTEXT

We select the following 45 problems from VerilogEval-Human that consists of non-textual represen-
tations in their problem descriptions:

2012_q1g, 2012_q2b, 2012_q2fsm, 2013_q2afsm, 2014_q3bfsm, 2014_q3c, always_nolatches,
circuit1, circuit10, circuit2, circuit3, circuit4, circuit5, circuit6, circuit7, circuit8, cir-
cuit9, ece241_2013_q7, ece241_2014_q3, ece241_2014_q5b, fsm1, fsm1s, fsm2, fsm2s, fsm3,
fsm3comb,
fsm_ps2data, kmap1, kmap2, kmap3, kmap4,
m2014_q3, m2014_q6, m2014_q6b, m2014_q6c, mt2015_q4, mt2015_q4a, mt2015_q4b, re-
view2015_fsmonehot, rule110, rule90, truthtable1

fsm_onehot,

fsm3onehot,

fsm3s,

A.5 TEMPLATE PROBLEMS FOR CORRECT-BY-CONSTRUCTION DATA

When generating correct-by-construction CC data, we select 11 problems from VerilogEval-
NonText to use as representative templates for constructing our prompts. To prevent contamination,
we ensure that benchmark problems are excluded from our data. While our prompts will resem-
ble those of the selected problems, the non-textual representations and solutions will differ. Addi-
tionally, to prevent overfitting to specific prompt templates, we use LLMs to rewrite the problem
instructions for 20% of our data. Furthermore, we create validation test problems that are strictly
in-distribution, based on the chosen problems.

Karnaugh Maps and Truth Tables: kmap1, m2014_q3, truthtable1.

State Transition Graphs and Tables: 2012_q2b, 2014_q3c, ece241_2014_q5b,
fsm3onehot, fsm_onehot, m2014_q6b, m2014_q6c.

fsm3comb,

Waveforms: We do not base our data on any benchmark problems specifically.

A.6 SCALING REPAIR DATA

As shown in Table 12, a carefully filtered dataset of
1.4k samples achieves comparable performance to a
7.8k dataset. This suggests that merely increasing
the dataset size by injecting the same types of er-
rors does not contribute meaningfully to improving
model performance.

A.7

ITERATIVE CODE REPAIR

We conduct a second iteration by generating 2.7k
repair data for the model based on the Repair data
from the first iteration. As shown in Table 13, per-
formance mostly saturates after this initial iteration.
We suspect that the remaining issues are likely due
to significant errors that are challenging to correct.

A.8 DIVERSITY OF GENERATED CODE

Table 12: Scaling Repair data.

Model

VerilogEval
Machine Human
pass@1 (%)

SDG-CC
SDG-CC-Repair 1k
SDG-CC-Repair 7k

73.9
81.9
82.2

62.0
68.0
67.4

RTLLM v1.1
Func
pass@5 (%)
62.8
65.8
64.5

Table 13: Iterative code repair.

Model

VerilogEval
Machine Human
pass@1 (%)

SDG-CC
SDG-CC-Repair Iter 1
SDG-CC-Repair Iter 2

73.9
81.9
81.3

62.0
68.0
68.1

RTLLM v1.1
Func
pass@5 (%)
62.8
65.8
65.6

We assess the diversity of the code generated by our models. We measure this diversity using
BLEU score, Jaccard similarity, and abstract tree edit distance (TSED) in Song et al. (2024). The
VerilogEval-Human problems are categorized into NonText and Text, as described in Appendix A.4.
For each problem, we compute the average code diversity score across sampled codes for the same
problem and report the mean score for all problems. For TSED, we use PyVerilog (Takamaeda-
Yamazaki, 2015) to extract the abstract syntax tree, and codes that fail syntax checks are excluded
from the analysis.

Table 14 presents the results on code diversity. We sample 20 solutions with temperature of 0.8 for
each model. We observe that fine-tuned models generally show a decrease in code diversity for both
Text and NonText problems. This reduction is expected, as BLEU and Jaccard metrics account for
both correct and incorrect code solutions, and there are often multiple ways to implement a correct

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

solution. When comparing our fine-tuned models with GPT-4o, code diversity is similar for Text
problems, but our models exhibit poor diversity for NonText problems. This is anticipated, given
that the CC training dataset for NonText problems is generated using correct-by-construction meth-
ods and follows similar templates for Verilog code. However, our models demonstrate comparable
diversity to GPT-4o for Text problems, particularly in TSED metric.

Table 14: Diversity of generated code solutions on VerilogEval-Human sampled with temperature
of 0.8. Lower scores indicate higher diversity.

Type

Models

Pretrained
Models

Ours
SDG-CC-Repair

CodeLlama
DeepSeek-Coder
Starcoder2
GPT-4o

CodeLlama
DeepSeek-Coder
Starcoder2

Jaccard
0.5330
0.6606
0.7724
0.6798

0.6848
0.6828
0.7018

Text
BLEU
0.3808
0.5454
0.5084
0.6633

0.5992
0.6040
0.6381

TSED
0.4255
0.5956
0.5520
0.6906

0.6354
0.6319
0.6721

NonText
BLEU
0.2507
0.3797
0.3607
0.6376

0.7242
0.6866
0.7750

Jaccard
0.4707
0.6548
0.7212
0.7390

0.8583
0.8308
0.8799

TSED
0.3521
0.3847
0.4020
0.6137

0.7158
0.6598
0.7740

Type

Models

Pretrained
Models

Ours
SDG-CC-Repair

CodeLlama
DeepSeek-Coder
Starcoder2
GPT-4o

CodeLlama
DeepSeek-Coder
Starcoder2

VerilogEval-Human (Overall)
BLEU
Jaccard
0.3441
0.5155
0.4987
0.6590
0.4667
0.7580
0.6561
0.6965

TSED
0.4156
0.5505
0.5198
0.6802

0.7333
0.7246
0.7512

0.6345
0.6273
0.6767

0.6515
0.6379
0.6942

A.9 ERROR TYPES OF LLM GENERATED ERROR REPORTS

Table 15: Error types of LLM generated error reports.

Error Type

#Errors One-line Description

Vector Concatenation

15.3% Errors during vector concatenation or bit slicing.

Incorrect Initialization

13.1% Missing or faulty initialization of registers or signals.

Boolean Logic Flaws

12.4% Logical inconsistencies or errors in combinational logic expressions.

Shift Operation Faults

10.2% Misaligned or unintended behavior during shift operations.

Timing Violations

10.2% Errors where signal propagation violates timing requirements.

KMap Misinterpretation

Latch Hazards

Bit Manipulation Bugs

Casez Priority Conflicts

8.8%

6.5%

7.3%

4.4%

Incorrect derivation of Boolean expressions from Karnaugh maps.

Unintended latches caused by missing or faulty conditions.

Errors in operations like masking, flipping, or extracting specific bits.

Ambiguities or conflicts in casez or case statements.

Nested Loop Design Flaws

3.7%

Incorrect or inefficient nested loop designs.

Others

8.1%

Miscellaneous errors not covered above.

Table 15 shows the distribution of common error types in LLM-generated error reports, along with
brief one-line descriptions. Most of these “minor” errors occur in solvable problems and stem from
hardware-specific concepts (e.g., shift operations, timing violations) and Verilog related issues un-

21

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

common in software languages (e.g., latch hazards, casez priority conflicts). When generating tar-
geted repair training data, we randomly sample detailed error reports and open-source code snippets,
ensuring the error type distribution in training aligns with their natural occurrences.

A.10 DETAILS ON FIGURE 1

Model

checkpoint2

Steps
256
386

SDG
SDG-CC-Repair

Table 16: Checkpoints of Figure 1.
checkpoint1

In Section 2.3 we discussed our findings on training
variability in learning outcomes for specific bench-
mark problems. To analyze this, we saved check-
points every 64 gradient steps during training and
tracked the pass rates of specific benchmarks. Our
training process is limited to a single epoch, as fur-
ther training was found to be not helpful. We classify problems with pass rates exceeding 67% as
solvable, and those below 33% as unsolvable. For the visualizations in Figure 1 we selected the
final two saved checkpoints, detailed in Table 16. The ideal outcome is not merely reduced variabil-
ity but also less degradations and improved accuracy: specifically, most problems in checkpoint2
should show higher pass rates than checkpoint1, assuming that training on additional data enhances
model performance. However, as shown in Figure 1a training on SDG data results in a significant
degradation of pass rates for many problems between checkpoint1 and checkpoint2. In contrast Fig-
ure 1b demonstrates reduced degradation and improvement in more problems. We further elaborate
such findings in Table 17, where we display pass rates for selected benchmark problems with high
volatility from VerilogEval-Human throughout the training progression.

Epoch
1.0
1.0

Epoch
0.82
0.86

Steps
320
448

Table 17: We displays pass rates for selected benchmark problems from VerilogEval-Human
throughout the training progression. Each entry shows the pass rate for SDG-CC-Repair (SDG),
with SDG results in parentheses.

Problem
m2014_q4h
always_nolatches
vectorr
fsm2s
fsm3comb

Step 64
1.0 (1.0)
1.0 (0.867)
1.0 (0.633)
1.0 (0.8)
1.0 (0.0)

Step 128
1.0 (0.9)
1.0 (0.9)
1.0 (0.925)
1.0 (0.8334)
0.95 (1.0)

Step 256
1.0 (0.967)
1.0 (0.6)
1.0 (0.467)
0.8 (0.775)
0.5 (0.533)

Step 320
1.0 (0.875)
1.0 (0.833)
0.95 (0.925)
1.0 (0.967)
1.0 (0.233)

Step 386
1.0 (-)
1.0 (-)
1.0 (-)
1.0 (-)
1.0 (-)

Step 448
1.0 (-)
1.0 (-)
1.0 (-)
1.0 (-)
1.0 (-)

We believe such volatility primarily is due to noise in SDG data where we can not verify solution
correctness. Because of the difficulties of verifying coding solutions in hardware descriptive lan-
guages, we instead generate targeted repair data for LLMs to learn to mitigate common errors which
have shown to generalize to writing correct code during completion. To the best of our knowledge,
we are the first work to describe such findings and provide an effective solution.

B FURTHER DISCUSSIONS AND BROADER IMPACTS

In this section, we provide further discussions to address concerns regarding the novelty, generaliz-
ability, and significance of our proposed methods. We offer clarifications to highlight the relevance
and broader impact of our work, underscoring its value to the broad research community.

B.1 GENERALIZABILITY OF CORRECT-BY-CONSTRUCTION DATA GENERATION

Our approach to curating correct-by-construction data is largely inspired by Trinh et al. (2024),
who introduced a mathematically rigorous method utilizing symbolic deduction engines to con-
struct synthetic training data, significantly improving LLM capabilities in solving Olympiad geom-
etry problems. Similarly, our method ensures the correctness of problems and solutions through a
custom-designed data generation pipeline, leveraging custom-designed solvers to generate accurate
solutions to their corresponding problems. In contrast to methods distilling LLM responses like
Self-Instruct (Wang et al., 2022), our correct-by-construction approach ensures data quality and so-
lution accuracy without relying on strong LLM performance on downstream tasks. We hope that
our mathematically rigorous approach to generating synthetic data can further inspire future work

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Under review as a conference paper at ICLR 2025

on improving LLMs general capabilities in areas such as math, coding, and symbolic reasoning.
Moreover, we recognize that adapting these methods to other domains may require human tuning
to identify the best data generation method, and we note that automating this process for scalability
could be a promising future research direction.

B.2 NOVELTY AND GENERALIZABILITY OF TARGETED CODE REPAIR

Our analysis show that LLMs frequently make “minor” errors in Verilog coding, often correctable
within few lines of code. We attribute this primarily to the LLMs’ insufficient training in com-
prehending problem descriptions and instructions alongside their correct solutions. Prior research
has tackled this challenge by improving data quality. For instance, Chen et al. (2022) filters incor-
rect code using tests generated by LLMs, while Zhang et al. (2024c) creates preference learning
datasets by ranking code through self-validation. Lei et al. (2024) focus on generating fine-tuning
data through code completion, test validation, and debugging with LLM agents, while Le et al.
(2022) trained reward models based on compilation and unit test outcomes to enhance LLM perfor-
mance via reinforcement learning. However, low-resource languages face additional obstacles due
to limited data availability, making it particularly difficult to synthesize unit tests directly in these
languages. To address this issue, Cassano et al. (2024) introduced lightweight compilers to translate
test cases from source to target languages.

Verilog coding encounters challenges typical of low-resource languages, compounded additional
domain-specific challenges as a hardware description language rather than a conventional program-
Its unique characteristics pose significant barriers to knowledge transfer from
ming language.
high-resource languages, as highlighted in studies on execution performance in parallel program-
ming (Nichols et al., 2024) and high-performance computing extensions (TehraniJamsaz et al.,
2024). To address these challenges, we propose a novel pipeline for generating targeted code re-
pair data. While automatic code repair has been extensively studied, most existing methods focus
on widely-used programming languages (Xia et al., 2023), relying on data of buggy code and fixes
from open-source repositories (Tufano et al., 2019; Just et al., 2014). In contrast, our pipeline utilizes
a small set of well-curated benchmarks and testbench to automate the generation of error reports,
quality assurance, and augmentation of training datasets by injecting similar errors into open-source
code. Our results highlight the effectiveness of this approach, which is language agnostic and can
be adapted to other low-resource and domain-specific programming languages.

B.3 SIGNIFICANCE OF NON-TEXTUAL DATA REPRESENTATIONS IN HARDWARE DESIGN

In this work, we emphasize the significance of non-textual data representations, specifically Kar-
naugh maps, state-transition diagrams, and waveforms, for accurately capturing hardware function-
ality. These representations are widely utilized by hardware designers to mitigate the ambiguity
and verbosity inherent in natural language descriptions. While they may be specific to hardware
design, they are not limited to Verilog and can be applied to various domain-specific languages
(DSLs) for hardware design. This is supported by Batten et al. (2024), who leveraged similar non-
textual representations from VerilogEval-Human to evaluate the performance of LLMs on several
Python-embedded hardware design DSLs.

In this study, we focus exclusively on limited representations, which constitute a significant portion
(45 problems, approximately 30%) of all problems in the VerilogEval-Human benchmark (details in
Appendix A.4). We exclude other types of non-textual representations due to the lack of a suitable
benchmark for evaluating LLMs in Verilog coding. Chang et al. (2024) emphasize the importance
of non-textual representations, particularly visual representations, in describing hardware designs.
While their work targets visual-language models and is therefore beyond the scope of this study, we
recognize that similar methodologies of our work such as correct-by-construction methods could be
employed to generate training data for visual representations, such as circuit schematics, data flow
diagrams, and state transition graphs.

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

23

Under review as a conference paper at ICLR 2025

C EXAMPLES OF TARGETED CODE REPAIR DATA

Problem description: vector3 from VerilogEval-Machine

The top module contains 6 ports that are fed into a concatenation which outputs 4 ports.
The first input "a" is fed into the first port in the concatenation, the second input "b" is fed into the
second port of the concatenation, the third input "c" is fed into the third port of the concatenation, the
fourth input "d" is fed into the fourth port of the concatenation, the fifth input "e" is fed into the fifth
port of the concatenation, the sixth input "f" is fed into the sixth port of the concatenation. The last
bit of the concatenation is 2’b11. This value is fed in to the last port of the concatenation.

module top_module (
input [4:0] a,
input [4:0] b,
input [4:0] c,
input [4:0] d,
input [4:0] e,
input [4:0] f,
output [7:0] w,
output [7:0] x,
output [7:0] y,
output [7:0] z

);

assign {w, x, y, z} = {a, b, c, d, e, f

bit as 2’b11

// Concatenate the inputs with the last

, 2’b11};

endmodule

assign {w, x, y, z} = {2’b11, a, b, c,

d, e, f};

endmodule

Correct Code

Error Code

LLM generated Error Report

Error Type: Incorrect vector concatenation and splitting

Category: Combinatorial: wiring

Description: Steps to repair the erroneous implementation:

1. Concatenating the two 1 bits at the beginning of the concatenated vector.
2. Assign the output vectors from the concatenated vector in the correct order and bit ranges

For example:

assign {w, x, y, z} = {2’b11, a, b, c, d, e, f};

To correct the implementation, you should concatenate the last bit as ‘2’b11’ on the right, as shown
in the correct implementation:

assign {w, x, y, z} = {a, b, c, d, e, f, 2’b11};

LLM constructed Repair data from open-source code

Input:

#### 1. Problem Description
You are given a Verilog module that demonstrates the use of bit slicing and concatenation in a loop.
The goal is to correctly display specific portions of a 32-bit register using a for-loop and explicit slic-
ing. However, there is a commonly made error in the concatenation and bit slicing order that causes
incorrect results to be displayed. Your task is to identify and fix the error in the code.

24

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Under review as a conference paper at ICLR 2025

#### 2. Erroneous Implementation

module block;

reg [31:0] data;
int

i;

initial begin

data = 32’hFACE_CAFE;

// Loop to display 8−bit segments of the data register
for (i = 0; i < 4; i++) begin

// Incorrectly reversing the bit order in the concatenation
$display ("data[8*%0d +: 8] = 0x%0h", i, data[8*(3−i) +: 8]);

end

// Incorrect slicing order, placing higher bits in lower indexes
$display ("data[7:0]
= 0x%0h", data[31:24]);
$display ("data[15:8] = 0x%0h", data[23:16]);
$display ("data[23:16] = 0x%0h", data[15:8]);
$display ("data[31:24] = 0x%0h", data[7:0]);

end

endmodule

In this erroneous implementation:
- The loop erroneously reverses the order of the 8-bit slices.
- The explicit bit-slicing outside of the loop has the slices ordered incorrectly, assigning the MSB to
the least significant part and vice versa.

#### 3. Hints for Fixing
1. **Correct the loop:**
The loop currently reverses the order of the 8-bit slices. Instead of accessing ‘data[8*(3-i) +: 8]’, you
should access ‘data[8*i +: 8]’ to display the correct slices of the register.
2. **Fix the slicing outside of the loop:**
Ensure that the LSB (‘data[7:0]’) corresponds to the lower bits of the ‘data’ register, and the MSB
(‘data[31:24]’) corresponds to the higher bits. Correct the slicing order for accurate display.

The corrected implementation should ensure that both the loop and explicit bit-slicing access the reg-
ister correctly, without reversing the bit order.

Output:

module block;

reg [31:0] data;
int
initial begin

i;

data = 32’hFACE_CAFE;
for (i = 0; i < 4; i++) begin

$display ("data[8*%0d +: 8] = 0x%0h", i, data[8*i +: 8]);

end
$display ("data[7:0]
$display ("data[15:8] = 0x%0h", data[15:8]);
$display ("data[23:16] = 0x%0h", data[23:16]);
$display ("data[31:24] = 0x%0h", data[31:24]);

= 0x%0h", data[7:0]);

end

endmodule

Figure 5: An example demonstrating the process for targeted code repair. In this example, training
checkpoints during training would have the model sometimes generated correct and error solutions.
We use LLMs to first summarize the errors for a detailed Error Report and then inject the errors to
open-source code to construct Repair data.

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

25

Under review as a conference paper at ICLR 2025

Problem description: shift4 from VerilogEval-Human

Build a 4-bit shift register (right shift), with asynchronous positive edge triggered areset, synchronous
active high signals load, and enable.
// (1) areset: Resets shift register to zero.
// (2) load: Loads shift register with data[3:0] instead of shifting.
// (3) ena: Shift right (q[3] becomes zero, q[0] is shifted out and disappears).
// (4) q: The contents of the shift register. If both the load and ena inputs are asserted (1), the load
input has higher priority.

module top_module(

input clk,
input areset,
input load,
input ena,
input [3:0] data,
output reg [3:0] q);

always @(posedge clk or posedge
areset) begin

if (areset) begin

always @(posedge clk or posedge
areset) begin
if (areset) begin

// Asynchronous positive edge

// Asynchronous positive edge

triggered areset

q <= 4’b0000;

triggered areset

q <= 4’b0;

end else if (load) begin

end else if (load) begin

// Synchronous active high

// Synchronous active high signals

signals load

q <= data;

end else if (ena) begin
// Shift right
q <= {1’b0, q[3:1]};

load

q <= data;

end else if (ena) begin

// Shift right
q <= {q[2:0], 1’b0};

end

end

endmodule

end

end

endmodule

Correct Code

Error Code

LLM generated Error Report

Error Type: shifting operation

Category: Sequential: shift registers

Description:
The error in the erroneous implementation is related to the shifting operation in the shift register.

q <= {q[2:0], 1’b0};

This line is intended to shift the bits in the register to the right, but it incorrectly connects the least
significant bit (q[0]) to the most significant bit (q[3]) instead of connecting a zero to the most signifi-
cant bit. This results in a circular shift instead of a right shift.

To correct this error, the line should be changed to:

q <= {1’b0, q[3:1]};

Here are the steps to repair the erroneous implementation
1. Identify the line of code that performs the shifting operation
2. Notice that the least significant bit (q[0]) is connected to the most significant bit (q[3]) instead of
a zero.
3. Change the line of code to connect a zero to the most significant bit and shift the remaining bits
to the right
4. Test the implementation to ensure that it behaves as expected.

26

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Under review as a conference paper at ICLR 2025

LLM constructed Repair data from open-source code

Input:

You are provided with a Verilog module that implements a shift register. However, the implementa-
tion contains a common error related to shifting operations. Your task is to identify the error, which
involves a mistake in the way bits are shifted within the register. Specifically, the problem results in
an incorrect circular shift rather than the intended right shift operation. You need to fix the error to
ensure that the register shifts correctly.

Erroneous Implementation:

module shift_registers_0 (clk, clken, SI, SO);

parameter WIDTH = 32;
input clk, clken, SI;
output SO;
reg [WIDTH−1:0] shreg;
always @(posedge clk)
begin

if (clken)

// Erroneous shifting operation: circular shift instead of right shift
shreg = {shreg[WIDTH−2:0], shreg[WIDTH−1]};

end
assign SO = shreg[WIDTH−1];

endmodule

In this erroneous implementation:
- The shift register incorrectly shifts the most significant bit (MSB) into the least significant bit
(LSB), creating a circular shift.
- The correct behavior should be shifting in the new input bit (‘SI’) while moving existing bits to the
right.

Hints for Fixing:
1. **Identify the shifting operation error:**
The line ‘shreg = {shreg[WIDTH-2:0], shreg[WIDTH-1]};’ causes a circular shift because the MSB is
incorrectly fed back into the LSB position.
2. **Fix the shifting logic:**
Replace the erroneous MSB assignment with the correct new input bit (‘SI’). The correct line should
be ‘shreg = {shreg[WIDTH-2:0], SI};’ to shift in the new input bit from the ‘SI‘ input and ensure the
bits shift to the right.
By fixing the shifting operation, the register will now shift correctly, with the new input bit filling the
LSB and the existing bits shifting to the right.

Output:

module shift_registers_0 (clk, clken, SI, SO);

parameter WIDTH = 32;
input clk, clken, SI;
output SO;
reg [WIDTH−1:0] shreg;
always @(posedge clk)
begin
if (clken)

shreg = {shreg[WIDTH−2:0], SI};

end
assign SO = shreg[WIDTH−1];

endmodule

Figure 6: Another example based on shift4 from VerilogEval-Human.

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

27

Under review as a conference paper at ICLR 2025

Problem description: multi_booth_8bit from RTLLM

Please act as a professional verilog designer.

Implement an 8-bit Radix-4 booth multiplier that performs the multiplication of two 8-bit inputs (a and b) using the Booth
algorithm.
It utilizes a clock signal (clk), and a reset signal (reset), and provides the product output (p) and a ready signal
(rdy). The ready signal (rdy) is set to 1 to indicate the completion of the multiplication process.

Module name:

multi_booth_8bit

Input ports:

clk: Clock signal used for synchronous operation.
reset: Reset signal used to initialize the multiplier module.
a: 8-bit input representing the multiplicand.
b: 8-bit input representing the multiplier.

Output ports:

p: 16-bit output representing the product of the multiplication.
rdy: Ready signal indicating the completion of the multiplication operation.

Implementation:

On the positive edge of the clock signal (clk) or the positive edge of the reset signal (reset), the module performs the

multiplication process.

If the reset signal (reset) is high, two 16-bit registers multiplier <= 8a[7], a and multiplicand <= 8b[7], b.
If the reset signal (reset) is low, indicating normal operation, the module checks if the counter (5bit ctr) is less than 16.
If the counter (ctr) is less than 16, the multiplicand register (multiplicand) is left-shifted by 1 to simulate the Booth

algorithm’s shifting operation.

If the current bit of the multiplier register (multiplier[ctr]) is 1, indicating a positive Booth encoding, the product register

(p) accumulates the value of the multiplicand register (multiplicand).

The counter (ctr) is incremented by 1. Once the counter (ctr) reaches 16, indicating the completion of the multiplication

process, the ready signal (rdy) is set to 1.

Give me the complete code.

module multi_booth_8bit (p, rdy, clk, reset, a, b);

module multi_booth_8bit (p, rdy, clk, reset, a, b);

input clk, reset;
input [7:0] a, b;
output reg [15:0] p;
output reg rdy;
reg [15:0] multiplicand, multiplier;
reg [4:0] ctr;
always @(posedge clk or posedge reset) begin

if (reset) begin

multiplier <= {{8{a[7]}}, a};
multiplicand <= {{8{b[7]}}, b};
p <= 0;
ctr <= 0;
rdy <= 0;
end else begin

if (ctr < 16) begin

multiplicand <= multiplicand << 1;
if (multiplier[ctr]) begin

p <= p + multiplicand;

end
ctr <= ctr + 1;

end else begin
rdy <= 1;

end

end

end
endmodule

input clk, reset;
input [7:0] a, b;
output reg [15:0] p;
output reg rdy;
reg [15:0] multiplier, multiplicand;
reg [4:0] ctr;
always @(posedge clk or posedge reset) begin

if (reset) begin

multiplier <= {{8{a[7]}}, a};
multiplicand <= {{8{b[7]}}, b};
ctr <= 0;
rdy <= 0;

end else begin

if (ctr < 16) begin

multiplicand <= multiplicand << 1;
if (multiplier[ctr])

p <= p + multiplicand;

ctr <= ctr + 1;

end else begin
rdy <= 1;

end

end

end

endmodule

Correct Code

Error Code

LLM generated Error Report

Incorrect initialization

Error Type:
Category: Sequential: counters
Description:
In the erroneous implementation, the p register is not initialized to zero during reset. This can lead to incorrect multiplication
results, as the initial value of p is unknown and may contain random values. The correct implementation initializes the p
register to zero during reset, ensuring that the multiplication result is accurate.

To repair the erroneous implementation, follow these steps:
1. Locate the always block that handles the reset and clock edge.
@(posedge clk or posedge reset).
2. Within the if (reset) block, add a line to initialize the p register to zero: p <= 0;.

In this case, it’s the block that starts with always

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

28

Under review as a conference paper at ICLR 2025

LLM constructed Repair data from open-source code

Input:

Problem Description:
You are given a Verilog module that implements a 4-bit up counter. The counter should reset to zero when the reset signal is
the counter_up
active and increment with each positive clock edge. However, there is a common error in this implementation:
register is not correctly initialized during reset, which may lead to unpredictable behavior after reset. Your task is to identify
and fix this initialization error. Erroneous Implementation:

module up_counter(
input clk,
input reset,
output [3:0] counter
);
reg [3:0] counter_up;
// up counter
always @(posedge clk or posedge reset)
begin

if (reset)

counter_up <= 4’d3; // Incorrect initialization value

else

counter_up <= counter_up + 4’d1;

end
assign counter = counter_up;

endmodule

In this erroneous implementation:
* The counter_up register is incorrectly initialized to 4’d3 during reset, instead of being reset to zero (4’d0), which can cause
incorrect counter behavior when reset is activated.

Identify the initialization issue: The counter should reset to zero, but currently, it is being set to 4’d3 during reset. This

Hints for Fixing:
1.
causes incorrect counting behavior after a reset.
2. Fix the reset logic: Modify the if (reset) block in the always block to properly initialize the counter to zero. Change
counter_up <= 4’d3; to counter_up <= 4’d0;.
3. Ensure proper reset behavior: After fixing the initialization, the counter will correctly reset to zero when the reset signal is
active and increment correctly on each clock cycle thereafter.

Output:

module up_counter(
input clk,
input reset,
output[3:0] counter
);
reg [3:0] counter_up;
// up counter
always @(posedge clk or posedge reset)
begin

if(reset)

counter_up <= 4’d0;

else

counter_up <= counter_up + 4’d1;

end
assign counter = counter_up;

endmodule

Figure 7: Another example based on multi_booth_8bit from RTLLM.

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

29

Under review as a conference paper at ICLR 2025

D EXAMPLES OF CORRECT-BY-CONSTRUCTION DATA FOR NON-TEXTUAL

REPRESENTATIONS

D.1 KARNAUGH MAPS AND TRUTH TABLES

Problem

0

c

Implement the circuit described by the Karnaugh map below.
//
// ab
1
// 00 | 1 | 0
// 01 | 0 | 0
// 11 | 0 | 0
// 10 | 0 | 0

module top_module(

input a,
input b,
input c,
output out

);

Solution

The input variables are: [’a’, ’b’, ’c’].
Based on the Karnaugh map, I can transform in to the following truth table:

a | b | c | f
0 | 0 | 0 | 1
0 | 0 | 1 | 0
0 | 1 | 0 | 0
0 | 1 | 1 | 0
1 | 0 | 0 | 0
1 | 0 | 1 | 0
1 | 1 | 0 | 0
1 | 1 | 1 | 0

The minterms (when output is 1) are:
(0,0,0) => (~a & ~b & ~c)
This corresponds to the following minterms logic:
‘(~a & ~b & ~c)‘

Finally, based on the above logic equation, I can now write the Verilog code that

could be described by the Karnaugh map:

‘‘‘
module top_module(

input a,
input b,
input c,
output out

);

assign out = (~a & ~b & ~c);

endmodule
‘‘‘

Figure 8: Example of a generated problem with Karnaugh Map representation in the problem de-
scription. We include detailed step-by-step analysis for reaching the correct solution.

30

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

Under review as a conference paper at ICLR 2025

D.2 STATE TRANSITION GRAPHS AND TABLES

Problem
Given the state−assigned table shown below, implement the logic functions Y[0] and

z.

// Present state y[2:0] | Next state Y[2:0] x=0, Next state Y[2:0] x=1 | Output z
// 000 | 010, 011 | 1
// 001 | 100, 010 | 0
// 010 | 001, 100 | 1
// 011 | 011, 100 | 0
// 100 | 100, 001 | 0

Solution

The state transition is as follows:
// state | Next state in=0, Next state in=1 | Output
// A | C, D | 1
// B | E, C | 0
// C | B, E | 1
// D | D, E | 0
// E | E, B | 0

The transition logic is then:
A: next = x ? D : C;
B: next = x ? C : E;
C: next = x ? E : B;
D: next = x ? E : D;
E: next = x ? B : E;

The output is 1 for states: A, C.
Thus the output logic is: assign z = ( y == A || y == C );
Y0 corresponds to 001 (A), 011 (D).

Finally, below is the Verilog code for the finite state machine:
‘‘‘
module top_module (
input clk,
input x,
input [2:0] y,
output reg Y0,
output reg z

);

reg [2:0] next_state;
parameter A=0, B=1, C=2, D=3, E=4;
always_comb begin

case(y)

A: next_state = x ? D : C;
B: next_state = x ? C : E;
C: next_state = x ? E : B;
D: next_state = x ? E : D;
E: next_state = x ? B : E;
default: next_state = ’x;

endcase

end
assign z = ( y == A || y == C );
assign Y0 = ( next_state == B || next_state == D );

endmodule
‘‘‘

Figure 9: Example of problem with state transition tables.

31

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

Under review as a conference paper at ICLR 2025

Problem

This is a Moore state machine with four states, four inputs, and one output.

Implement this state machine in Verilog. Reset is an active−high synchronous
reset to state D.

// D (out=0) −−in0=1−−> D
// D (out=0) −−in0=0−−> A
// C (out=0) −−in1=1−−> D
// C (out=0) −−in1=0−−> B
// B (out=1) −−in2=1−−> D
// B (out=1) −−in2=0−−> B
// A (out=0) −−in3=1−−> B
// A (out=0) −−in3=0−−> C

Solution

The finite state machine has four inputs, and the state transition logic is as

follows:

D: next = in0 ? D : A;
C: next = in1 ? D : B;
B: next = in2 ? D : B;
A: next = in3 ? B : C;

The output is 1 for states: B.
Thus the output logic is: ‘assign out = ( state == B );‘.

Finally, below is the Verilog code for the finite state machine:
‘‘‘
module top_module (
input clk,
input reset,
input in0,
input in1,
input in2,
input in3,
output out

);

parameter D=0, C=1, B=2, A=3;
reg state;
reg next;
always_comb begin

case(state)

D: next = in0 ? D : A;
C: next = in1 ? D : B;
B: next = in2 ? D : B;
A: next = in3 ? B : C;
default: next = ’x;

endcase

end
always @(posedge clk) begin

if (reset) state <= D;
else state <= next;

end
assign out = ( state == B );

endmodule
‘‘‘

Figure 10: Example of problem with Moore FSM.

32

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

Under review as a conference paper at ICLR 2025

Problem
The following diagram is a Mealy machine. Implement in Verilog using one−hot
encoding. Resets into state A and reset is asynchronous active−high.

// A −−x=0 (z=0)−−> D
// A −−x=1 (z=1)−−> C
// B −−x=0 (z=1)−−> C
// B −−x=1 (z=0)−−> B
// C −−x=0 (z=0)−−> C
// C −−x=1 (z=0)−−> D
// D −−x=0 (z=1)−−> C
// D −−x=1 (z=0)−−> B

Solution

From the transition diagram, we have the following transition logic:
// state | next state in=0, next state in=1
// A | D, C
// B | C, B
// C | C, D
// D | C, B
Thus the state transition logic is as follows:
A: next = x ? C : D;
B: next = x ? B : C;
C: next = x ? D : C;
D: next = x ? B : C;
The output is 1 for states: (A, x), (B, ~x), (D, ~x).
Thus the output logic is: ‘assign z = ( ( state == A & x ) || ( state == B & ~x )

|| ( state == D & ~x ) );‘.

Finally, below is the Verilog code for the finite state machine:
‘‘‘
module top_module (
input clk,
input areset,
input x,
output z

);

parameter A=2’b00, B=2’b01, C=2’b10, D=2’b11;
reg [1:0] state;
reg [1:0] next_state;
always_comb begin

case(state)

A: next_state = x ? C : D;
B: next_state = x ? B : C;
C: next_state = x ? D : C;
D: next_state = x ? B : C;
default: next_state = ’x;

endcase

end
always @(posedge clk, posedge areset) begin

if (areset) state <= A;
else state <= next_state;

end
assign z = ( ( state == A & x ) || ( state == B & ~x ) || ( state == D & ~x

) );
endmodule
‘‘‘

Figure 11: Example of problem with Mealy FSM.

33

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

Under review as a conference paper at ICLR 2025

Problem

The following is the state transition table for a Moore state machine with one

input, one output, and four states. Use the following one−hot state encoding: A
=4’b0001, B=4’b0010, C=4’b0100, D=4’b1000. Derive state transition and output
logic equations by inspection assuming a one−hot encoding. Implement only the
state transition logic and output logic (the combinational logic portion) for
this state machine.

// state | Next state in=0, Next state in=1 | Output
// A | B, A | 0
// B | B, C | 1
// C | D, A | 1
// D | B, C | 0

Solution

Based on the state transition table, we can obtain the next state from observing

the row (previous state) and column (input).

Next state is A on the following (row, column): (A, in=1) (C, in=1). This
correspond to the following logic: ‘state[A] & in || state[C] & in‘.

Next state is B on the following (row, column): (A, in=0) (B, in=0) (D, in=0). This
correspond to the following logic: ‘state[A] & ~in || state[B] & ~in || state[

D] & ~in‘.

Next state is C on the following (row, column): (B, in=1) (D, in=1). This
correspond to the following logic: ‘state[B] & in || state[D] & in‘.

Next state is D on the following (row, column): (C, in=0). This correspond to the

following logic: ‘state[C] & ~in‘.

The output is 1 for states: B, C.
Thus the output logic is: ‘assign out = ( state[B] || state[C] );‘.

Finally, below is the Verilog code for the finite state machine:
‘‘‘
module top_module (
input in,
input [3:0] state,
output reg [3:0] next_state,
output out

);

parameter A=0, B=1, C=2, D=3;

assign next_state[A] = state[A] & in || state[C] & in;
assign next_state[B] = state[A] & ~in || state[B] & ~in || state[D] & ~in;
assign next_state[C] = state[B] & in || state[D] & in;
assign next_state[D] = state[C] & ~in;

assign out = ( state[B] || state[C] );

endmodule
‘‘‘

Figure 12: Example of problem with in-edge focused solution shown in Figure 2.

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

34

Under review as a conference paper at ICLR 2025

D.3 WAVEFORMS

Problem

This is a combinational circuit. Read the simulation waveforms to determine what

the circuit does, then implement it.

a
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1

// time
// 0ns
// 5ns
// 10ns
// 15ns
// 20ns
// 25ns
// 30ns
// 35ns
// 40ns
// 45ns
// 50ns
// 55ns
// 60ns
// 65ns
// 70ns
// 75ns
// 80ns
// 85ns
// 90ns

Solution

b
0
0
0
0
0
0
0
1
1
1
1
0
0
0
0
1
1
1
1

c
0
0
0
0
0
1
1
0
0
1
1
0
0
1
1
0
0
1
1

d
0
0
0
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1

q
0
0
0
0
0
0
0
0
0
0
0
1
1
0
1
0
1
0
0

Based on the simulation waveform, I can transform in to the following truth table:

a | b | c | d | f
0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 1 | 0
0 | 0 | 1 | 0 | 0
0 | 0 | 1 | 1 | 0
0 | 1 | 0 | 0 | 0
0 | 1 | 0 | 1 | 0
0 | 1 | 1 | 0 | 0
0 | 1 | 1 | 1 | 0
1 | 0 | 0 | 0 | 1
1 | 0 | 0 | 1 | 1
1 | 0 | 1 | 0 | 0
1 | 0 | 1 | 1 | 1
1 | 1 | 0 | 0 | 0
1 | 1 | 0 | 1 | 1
1 | 1 | 1 | 0 | 0
1 | 1 | 1 | 1 | 0

The minterms (when output is 1) are:
(1,0,0,0) => (a & ~b & ~c & ~d)
(1,0,0,1) => (a & ~b & ~c & d)
(1,0,1,1) => (a & ~b & c & d)
(1,1,0,1) => (a & b & ~c & d)
This corresponds to the following minterms logic:
‘(a & ~b & ~c & ~d) | (a & ~b & ~c & d) | (a & ~b & c & d) | (a & b & ~c & d)‘

Finally, based on the above logic equation, I can now write the Verilog code:
‘‘‘
module top_module(

input a,
input b,

35

1836
1837

1838
1839
1840
1841
1842
1843

1844
1845
1846
1847
1848
1849

1850
1851
1852
1853
1854

1855
1856
1857
1858
1859
1860

1861
1862
1863
1864
1865
1866

1867
1868
1869
1870
1871
1872

1873
1874
1875
1876
1877

1878
1879
1880
1881
1882
1883

1884
1885
1886
1887
1888
1889

Under review as a conference paper at ICLR 2025

1890
1891

1892
1893
1894
1895
1896
1897

1898
1899
1900
1901
1902
1903

1904
1905
1906
1907
1908

1909
1910
1911
1912
1913
1914

1915
1916
1917
1918
1919
1920

1921
1922
1923
1924
1925
1926

1927
1928
1929
1930
1931

1932
1933
1934
1935
1936
1937

1938
1939
1940
1941
1942
1943

input c,
input d,
output q

);

assign q = (a & ~b & ~c & ~d) | (a & ~b & ~c & d) | (a & ~b & c & d) | (a &

b & ~c & d);

endmodule
‘‘‘

Figure 13: Example of a waveform problem for combinational circuits.

Problem

This is a sequential circuit. Read the simulation waveforms to determine what the

circuit does, then implement it.

// time
// 0ns
// 5ns
// 10ns
// 15ns
// 20ns
// 25ns
// 30ns
// 35ns
// 40ns
// 45ns
// 50ns
// 55ns
// 60ns
// 65ns
// 70ns
// 75ns
// 80ns
// 85ns
// 90ns

clk
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0

reset
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

in
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
1
1
0
1

out
x
1
1
1
1
0
0
1
1
0
0
1
1
0
0
1
1
1
1

... Simulation waveform redacted ...

Solution

From the waveform, we have the following transition logic and output logic:
// state | Next state in=0, Next state in=1 | Output
// D | C, B | 1
// C | B, D | 0
// B | D, A | 1
// A | C, B | 0

Thus the state transition logic is as follows:
D: next = in ? B : C;
C: next = in ? D : B;
B: next = in ? A : D;
A: next = in ? B : C;

The output is 1 for states: D, B.
Thus the output logic is: ‘assign out = ( state == D || state == B );‘.

Finally, below is the Verilog code for the finite state machine:

36

Under review as a conference paper at ICLR 2025

‘‘‘
module top_module (
input clk,
input in,
input reset,
output out

);

parameter D=0, C=1, B=2, A=3;
reg state;
reg next;

always_comb begin

case(state)

D: next = in ? B : C;
C: next = in ? D : B;
B: next = in ? A : D;
A: next = in ? B : C;
default: next = ’x;

endcase

end

always @(posedge clk) begin

if (reset) state <= D;
else state <= next;

end

assign out = ( state == D || state == B );

endmodule
‘‘‘

Figure 14: Example of a waveform problem for sequential circuits.

1944
1945

1946
1947
1948
1949
1950
1951

1952
1953
1954
1955
1956
1957

1958
1959
1960
1961
1962

1963
1964
1965
1966
1967
1968

1969
1970
1971
1972
1973
1974

1975
1976
1977
1978
1979
1980

1981
1982
1983
1984
1985

1986
1987
1988
1989
1990
1991

1992
1993
1994
1995
1996
1997

37

Under review as a conference paper at ICLR 2025

E PROMPT TEMPLATES

E.1 SYNTHETIC DATA GENERATION

E.1.1 SELF-INSTRUCT

Your goal is to create a high-quality Verilog problem.

* Guidelines for designing the problem description:

1. This should be **completely self-contained**, providing all the contextual information one needs
to understand and solve the problem.
2. Assume common verilog knowledge, but ensure that any specific context, variables, or code snip-
pets pertinent to this problem are explicitly included.
3. Do not include the code snippet in the problem.
4. The problem should be desinged for the programmers to solve it with one verilog module.
5. The problem description section should be enclosed within <PROBLEM> </PROBLEM> tags.

Now, Please use your creativity to create a brand new high-quality Verilog problem.

Figure 15: Prompt used to generate initial 50 seed problems for Self-Instruct.

Your goal is to create a high-quality Verilog problem.

* Guidelines for designing the problem description:

1. This should be **completely self-contained**, providing all the contextual information one needs
to understand and solve the problem.
2. Assume common verilog knowledge, but ensure that any specific context, variables, or code snip-
pets pertinent to this problem are explicitly included.
3. Do not include the code snippet in the problem.
4. The problem should be desinged for the programmers to solve it with one verilog module.
5. The problem description section should be enclosed within <PROBLEM> </PROBLEM> tags.

Below shows some examples:

<PROBLEM>
{seed problems}
</PROBLEM>

Now, Please use your creativity to create a brand new high-quality Verilog problem.

Figure 16: Prompt used for Self-Instruct.

1998
1999

2000
2001
2002
2003
2004
2005

2006
2007
2008
2009
2010
2011

2012
2013
2014
2015
2016

2017
2018
2019
2020
2021
2022

2023
2024
2025
2026
2027
2028

2029
2030
2031
2032
2033
2034

2035
2036
2037
2038
2039

2040
2041
2042
2043
2044
2045

2046
2047
2048
2049
2050
2051

38

Under review as a conference paper at ICLR 2025

E.1.2 OSS-INSTRUCT

Your goal is to create a high-quality Verilog problem.

* Guidelines for designing the problem description:

1. This should be **completely self-contained**, providing all the contextual information one needs
to understand and solve the problem.
2. Assume common verilog knowledge, but ensure that any specific context, variables, or code snip-
pets pertinent to this problem are explicitly included.
3. Do not include the code snippet in the problem.
4. The problem should be designed for the programmers to solve it with one Verilog module.

* Guidelines for the problem description format: The problem description section should be enclosed
within <PROBLEM> </PROBLEM> tags.

Please increase the difficulty of the given programming test question a bit. You can increase the diffi-
culty using, but not limited to, the following methods:

1. Your new problem should not be directly solved by the original code snippet.
2. You can also change the bit-width requiremnt, how to reset internal signals (if applicable), and
whether the solution needs a clock signal (combinatorial versus sequential logic). If you do have a
reset method that is synchronous to a clock, make sure to add the clock signal to the problem module
input.
3. Add new constraints and requirements to the original problem, adding approximately 10 additional
words.
4. Replace a commonly used requirement in the programming task with a less common and more
specific one.
5. If the original problem can be solved with only a few logical steps, please add more reasoning
steps.

Now, Please gain inspiration from the following random code snippet to create a high-quality Verilog
problem.

Code snippet for inspiration:
‘‘‘
{code snippet}
‘‘‘

Output:

Figure 17: Prompt used for OSS-Instruct. We also include prompts inspired from Evol-
Instruct (Luo et al., 2024b) to increase problem difficulty.

2052
2053

2054
2055
2056
2057
2058
2059

2060
2061
2062
2063
2064
2065

2066
2067
2068
2069
2070

2071
2072
2073
2074
2075
2076

2077
2078
2079
2080
2081
2082

2083
2084
2085
2086
2087
2088

2089
2090
2091
2092
2093

2094
2095
2096
2097
2098
2099

2100
2101
2102
2103
2104
2105

39

Under review as a conference paper at ICLR 2025

E.1.3 DOCU-INSTRUCT

Your goal is to create a high-quality Verilog problem.

* Guidelines for designing the problem description:

1. This should be **completely self-contained**, providing all the contextual information one needs
to understand and solve the problem.
2. Assume common verilog knowledge, but ensure that any specific context, variables, or code snip-
pets pertinent to this problem are explicitly included.
3. Do not include the code snippet in the problem.
4. The problem should be designed for the programmers to solve it with one Verilog module.

* Guidelines for the problem description format: The problem description section should be enclosed
within <PROBLEM> </PROBLEM> tags.

Now, Please gain inspiration from the following textbook or wikipedia snippet to create a high-quality
Verilog problem. The information might not be directly related to Verilog, but try to be make the
problem as relevant as possible to the textbook issue discussed.

Textbook snippet for inspiration:
‘‘‘
{document snippet}
‘‘‘

Output:

Figure 18: Prompt used for Docu-Instruct with Wikipedia and textbooks.

I am going to give you a concept and some descriptions about that concept. Based on the descrip-
tions and concept name, determine if the concept belongs to one of the following categories:

- Hardware description and modeling in Verilog.
- Fundamental constructs such as modules, ports, and wires specific to Verilog.
- Synthesis and optimization techniques employed in hardware design using Verilog.
- Simulation tools and methodologies for verifying Verilog-based hardware designs.
- Common design patterns and best practices in Verilog for efficient hardware implementation.
- Programming concepts like loops, functions related to Verilog.
- Hardware related concepts such as finite state machines that could be implemented in Verilog.
- Algorithms that could be implemented in hardware, such as Fourier Transforms.

Concept: {Wikipedia title}
Description: {Wikipedia content}

Do not make assumptions and only respond “Yes” if you are certain that the {Wikipedia title} is re-
lated to hardware design or Verilog coding language.

Your answer should start with “Yes” or “No”.

Figure 19: Prompt used to filter Verilog related Wikipedia pages.

2106
2107

2108
2109
2110
2111
2112
2113

2114
2115
2116
2117
2118
2119

2120
2121
2122
2123
2124

2125
2126
2127
2128
2129
2130

2131
2132
2133
2134
2135
2136

2137
2138
2139
2140
2141
2142

2143
2144
2145
2146
2147

2148
2149
2150
2151
2152
2153

2154
2155
2156
2157
2158
2159

40

Under review as a conference paper at ICLR 2025

E.1.4 NON-TEXTUAL REPRESENTATIONS

Your goal is to create a high-quality Verilog problem. Specifically, we would like to test the skills of
understanding Karnaugh maps and state transition diagrams. The problem description section should
be enclosed within <PROBLEM> </PROBLEM> tags.

Now, please gain inspiration from the following random code snippet to create a high-quality Ver-
ilog problem. Remember that the problem you generated must include Karnaugh maps in the format
above. The random code snippet MUST be related to the solution. Your problem statement should
be short and succinct (no more than 5 sentences) and you MUST generate a Karnaugh map in the
problem description. Your problem description should not describe the Karnaugh map in words and
should assume that the student need to decipher the Karnaugh map to solve the problem.

Code snippet for inspiration:
‘‘‘
{code snippet}
‘‘‘

Below are two examples on how to represent Karnaugh map related questions in purely textual for-
mat. You should NOT use the following to generate the problem but only consider the style.

<PROBLEM>
Given the state−assigned table shown below, implement the finite−state machine.

Reset should synchronous active high reset the FSM to state 000.

// Present state y[2:0] | Next state y[2:0] x=0, Next state y[2:0] x=1, Output z
// 000 | 000, 001 | 0
// 001 | 001, 100 | 0
// 010 | 010, 001 | 0
// 011 | 001, 010 | 1
// 100 | 011, 100 | 1
</PROBLEM>
<PROBLEM>
Implement the circuit described by the Karnaugh map below.
//
// bc
// 00 | 0 | 1 |
// 01 | 1 | 1 |
// 11 | 1 | 1 |
// 10 | 1 | 1 |

a
0 1

</PROBLEM>

Figure 20: An prompt example to encourage LLMs to generate questions with Karnaugh Maps.

2160
2161

2162
2163
2164
2165
2166
2167

2168
2169
2170
2171
2172
2173

2174
2175
2176
2177
2178

2179
2180
2181
2182
2183
2184

2185
2186
2187
2188
2189
2190

2191
2192
2193
2194
2195
2196

2197
2198
2199
2200
2201

2202
2203
2204
2205
2206
2207

2208
2209
2210
2211
2212
2213

41

Under review as a conference paper at ICLR 2025

E.1.5 PROMPTS FOR SAMPLING SOLUTIONS WITH LLM GENERATED PROBLEMS

Your goal is to solve a Verilog problem by completing one verilog module.

* Guidelines for writing the solution (verilog module) and formatting:
1. The module should be **completely self-contained**, fulfilling all the requirements needed to
solve the problem.
2. Ensure the correctness of the syntax and functionality.
3. The module name must be “top_module”.
4. Do not add blackslash in front of underscore symbol.
5. The solution must be valid Verilog code.
6. The solution section should be enclosed within <SOLUTION> </SOLUTION> tags.
7. The reasoning section should be enclosed within <REASON> </REASON> tags.
8. Write comments in the solution section. Try to embed the reasoning in comments. Always try to
write the corresponding comments before the code.
9. The module should be **completely self-contained**, do not include or import outside the module
and define everything inside the module.

Below shows an example:

Problem description:

Build a counter that counts from 0 to 999, inclusive, with a period of 1000 cycles.
The reset input is active high synchronous, and should reset the counter to 0.
Solve the problem by completing the following module.

Output:

<REASON>
To solve the problem of building a counter that counts from 0 to 999 with a reset
input that is active high synchronous, you will need a 10−bit register (since
2^10=1024 and 999 < 1024). The counter should increment on each clock cycle and

wrap around back to 0 when it reaches 1000. The reset input, when active,

should reset the counter to 0 on the next clock edge.

</REASON>

Here is the Verilog code for the module:
<SOLUTION>
module top_module(
input clk,
input reset,
output reg [9:0] q);

// This block executes on the positive edge of the clock signal.
always @(posedge clk) begin

// Checks if the reset is active. This is synchronous with the clock.
if (reset) begin

// Reset the counter to 0 when reset is active
q <= 0;

// If the counter has reached 999, it needs to wrap around to 0 on the next

cycle.

end else if (q == 999) begin

// Reset to 0 after reaching 999
q <= 0;

// In all other cases, just increment the counter.
end else begin

// Increment the counter
q <= q + 1;

end

end

endmodule
</SOLUTION>

2214
2215

2216
2217
2218
2219
2220
2221

2222
2223
2224
2225
2226
2227

2228
2229
2230
2231
2232

2233
2234
2235
2236
2237
2238

2239
2240
2241
2242
2243
2244

2245
2246
2247
2248
2249
2250

2251
2252
2253
2254
2255

2256
2257
2258
2259
2260
2261

2262
2263
2264
2265
2266
2267

42

Under review as a conference paper at ICLR 2025

Now, please solve the following Verilog problem. I will also attach a reference code snippet which
was used as an inspiration to generate the problem. The provided code may not directly solve the
problem so you should use it only as a reference.

Reference code:
‘‘‘
{code snippet}
‘‘‘

Problem description:
‘‘‘
{in context examples}
‘‘‘
Output:

Figure 21: Prompt used for sampling solutions for synthetic data generation. We include a in context
example to encourage models to include reasoning traces. Prompts in blue are only included for
problems generated from a code snippet.

E.1.6 PROMPTS FOR VERIFYING SOLUTIONS

Check if the given Verilog module is a valid solution to the problem. The output should be in
“True” or “False” and be enclosed within <VALID> </VALID> tags and the explanation in <REA-
SON></REASON> tags.

Now check the following:

<PROBLEM>
{problem}
<PROBLEM>

<SOLUTION>
{solution}
</SOLUTION>

Figure 22: Prompt used for verifying solutions.

2268
2269

2270
2271
2272
2273
2274
2275

2276
2277
2278
2279
2280
2281

2282
2283
2284
2285
2286

2287
2288
2289
2290
2291
2292

2293
2294
2295
2296
2297
2298

2299
2300
2301
2302
2303
2304

2305
2306
2307
2308
2309

2310
2311
2312
2313
2314
2315

2316
2317
2318
2319
2320
2321

43

Under review as a conference paper at ICLR 2025

E.2 PROMPTS FOR TARGETED CODE REPAIR

E.2.1 ERROR REPORT

Here is a Verilog problem description:
‘‘‘
{problem description}
‘‘‘

Here is an erroneous implementation:
‘‘‘
{error code}
‘‘‘

Here is a correct implementation:
‘‘‘
{correct code}
‘‘‘

Generate a detail error report.
The error report should describe the common error type and output the code category. The error re-
port should also be detailed enough to let beginners to repair the erroneous implementation step by
step.

Output:

Figure 23: Prompt for Error Report generation.

Here is a Verilog problem description:
‘‘‘
{problem description}
‘‘‘

Here is an erroneous implementation:
‘‘‘
{error code}
‘‘‘

Here is the error report:
‘‘‘
{error report}
‘‘‘

Now fix the erroneous implementation and give me the correct code.

Output:

Figure 24: Prompt for Error Report self-consistency validation. The generated code fix will be
evaluated for functional correctness. Error reports whose code fixes do not pass will be filtered.

2322
2323

2324
2325
2326
2327
2328
2329

2330
2331
2332
2333
2334
2335

2336
2337
2338
2339
2340

2341
2342
2343
2344
2345
2346

2347
2348
2349
2350
2351
2352

2353
2354
2355
2356
2357
2358

2359
2360
2361
2362
2363

2364
2365
2366
2367
2368
2369

2370
2371
2372
2373
2374
2375

44

Under review as a conference paper at ICLR 2025

E.2.2 ERROR INJECTION

Your goal is to create an error-fixing Verilog practice problem for programmers. You will demonstrate
a type of error that is commonly made by programmers.
Create an error repair practice problem with three components:
1. Problem description
2. Erroneous implementation
3. Hints for fixing

Here is an example:

<EXAMPLE>
The following Verilog module is intended to implement the specification below. However, there is
a bug in the code which causes incorrect results. Please fix the bug to make the module work as
intended.

Erroneous Implementation:

// Verilog code with the injected error
module example_module (
input wire clk,
input wire reset,
output reg [3:0] counter

);

// Intended functionality:
// This module should count from 0 to 15 and then wrap around.

always @(posedge clk or posedge reset) begin

if (reset) begin

counter <= 4’b0000;

end else begin

counter <= counter + 1’b1; // Error injected: Should be 4’b1

end

end

endmodule

Hints for Fixing:
1. Verify the bit-width of the counter and the increment operation.
2. Check the initialization and wrapping condition of the counter.
3. Ensure that the addition operation correctly handles the 4-bit counter.

</EXAMPLE>

Now, here is the commonly made error:

‘‘‘
{error report}
‘‘‘

Inject the above error into the following module and create an error repair practice problem. Check if
it is possible to inject the error. If not, create the problem with the given error alone and ignore the
module in the code snippet.

‘‘‘
{code snippet}
‘‘‘

Output:

Figure 25: Prompt used to inject targeted errors to open-source code in code Repair data. We also
prompt the LLM to self-verify if the error could be injected to the code snippet.

2376
2377

2378
2379
2380
2381
2382
2383

2384
2385
2386
2387
2388
2389

2390
2391
2392
2393
2394

2395
2396
2397
2398
2399
2400

2401
2402
2403
2404
2405
2406

2407
2408
2409
2410
2411
2412

2413
2414
2415
2416
2417

2418
2419
2420
2421
2422
2423

2424
2425
2426
2427
2428
2429

45

Under review as a conference paper at ICLR 2025

Figure 26: Correct-by-construction for Karnaugh
maps and truth tables.

Figure 27: Correct-by-construction for finite-
state machines.

Figure 28: Correct-by-construction for waveforms.

46

2430
2431

2432
2433
2434
2435
2436
2437

2438
2439
2440
2441
2442
2443

2444
2445
2446
2447
2448

2449
2450
2451
2452
2453
2454

2455
2456
2457
2458
2459
2460

2461
2462
2463
2464
2465
2466

2467
2468
2469
2470
2471

2472
2473
2474
2475
2476
2477

2478
2479
2480
2481
2482
2483

Karnaugh Maps andTruth TablesStep1. SampleConfigurationsSample random mintermsvariables=['a','b','c'], minterms=[1, 2, 5], don't_cares=[7]SOP form: (~a & ~b & c) | (~a & b & ~c) | (a & ~b & c)Truth table a | b | c | f 0 | 0 | 0 | 0 0 | 0 | 1 | 1 0 | 1 | 0 | 1 0 | 1 | 1 | 0 1 | 0 | 0 | 0 1 | 0 | 1 | 1 1 | 1 | 0 | 0 1 | 1 | 1 | xKarnaugh map    bc a 00  01 11 10 0 | 0 | 1 | 0 | 1 1 | 0 | 1 | x | 0Step2. ConstructRepresentationsand ProblemsStep3. ConstructSolutionmodule top_module(        input a,         input b,        input c,        output f);        assign f = (~a & ~b & c) | (~a& b & ~c) | (a & ~b & c)endmoduleState Transition Graphsand TablesStep1. SampleConfigurationsStep2. ConstructRepresentationsand ProblemsStep3. ConstructSolutionresetA/0C/1B/0D/110101001Construct random legal transition graphsState transition table.// state | in=0, in=1 | Output// A | C, B | 0// B | D, C | 0// C | B, C | 1// D | D, C | 1State transition graph.// A (out=0) --in=0--> C// A (out=0) --in=1--> B// B (out=0) --in=0--> D// B (out=0) --in=1--> C// C (out=1) --in=0--> B// C (out=1) --in=1--> C// D (out=1) --in=0--> D// D (out=1) --in=1--> C    always_comb begin                case (state)                        A: next = in ? B : C;                        B: next = in ? C : D;C: next = in ? C : B;                        D: next = in ? C : D;                endcase    end            assign out = (state==C) | (state==D);WaveformsStep 1. ObtainPrevious SolutionsStep 2. Simulate withTemplate Test BenchStep 3. ConstructWaveform Problems    always_comb begin                case (state)                        A: next = in ? B : C;                        B: next = in ? C : D;C: next = in ? C : B;                        D: next = in ? C : D;                endcase    end            assign out = (state==C) | (state==D);module top_module(        input a,         input b,        input c,        output f);        assign f = (~a & ~b & c) | (~a& b & ~c) | (a & ~b & c)endmoduleCode + TestbenchWaveform VCD FileVerilog SimulatorCombinatorial circuit.// time  a   b   c   f// 0ns   0   0   0  0// 5ns   0   0   1  1// 10ns 0   1   0  1// 15ns 0   1   1  0...Sequential circuit.// time  clk  reset  in   out // 0ns   0    1        0     0// 5ns   1    1        0     0// 10ns 0    0        0     0// 15ns 0    0        1     0...