#  CodeV: Empowering LLMs for Verilog Generation through Multi-Level
Summarization

Yang Zhao1, 2 Di Huang1 Chongxiao Li1, 2 Pengwei Jin1, 2 Ziyuan Nan1, 2
Tianyun Ma1, 3 Lei Qi1, 3 Yansong Pan1, 2 Zhenxing Zhang1, 3 Rui Zhang1
Xishan Zhang1, 4 Zidong Du1 Qi Guo1 Xing Hu1, ✉ Yunji Chen1, 2, ✉

1SKL of Processors, Institute of Computing Technology, CAS
2University of Chinese Academy of Sciences
3University of Science and Technology of China
4Cambricon Technologies

https://iprc-dip.github.io/CodeV

###### Abstract

The increasing complexity and high costs associated with modern processor
design have led to a surge in demand for processor design automation.
Instruction-tuned large language models (LLMs) have demonstrated remarkable
performance in automatically generating code for general-purpose programming
languages like Python. However, these methods fail on hardware description
languages (HDLs) like Verilog due to the scarcity of high-quality instruction
tuning data, as even advanced LLMs like GPT-3.5 exhibit limited performance on
Verilog generation. Regarding this issue, we observe that (1) Verilog code
collected from the real world has higher quality than those generated by LLMs.
(2) LLMs like GPT-3.5 excel in summarizing Verilog code rather than generating
it. Based on these observations, this paper introduces CodeV, a series of
open-source instruction-tuned Verilog generation LLMs. Instead of generating
descriptions first and then getting the corresponding code from advanced LLMs,
we prompt the LLM with Verilog code and let the LLM generate the corresponding
natural language description by multi-level summarization. Experimental
results show that CodeV relatively surpasses the previous open-source SOTA by
14.4% (BetterV in VerilogEval) and 11.3% (RTLCoder in RTLLM) respectively, and
also relatively outperforms previous commercial SOTA GPT-4 by 22.1% in
VerilogEval.

✉✉footnotetext:  Corresponding author. Contact: {zhaoyang24s, huxing,
<EMAIL_ADDRESS>
##  1 Introduction

The design flow of processors is expensive and complex, especially the coding
of the hardware description language (HDL) like Verilog. Researchers have
therefore explored ways to automate this process. Traditional methods include
agile hardware development like Chisel [3], and high-level synthesis (HLS),
which compiles high-level programming languages such as C/C++ to Verilog [8,
20]. Recently, the rise of large language models (LLMs) has achieved excellent
performance in text-to-code tasks for general-purpose programming languages
(e.g., Python) [34, 33, 29, 51, 53, 31, 56] and shows a new perspective on
automatic Verilog generation from text or natural language descriptions.

A promising approach for adapting LLMs to Verilog generation is instruction
tuning, which has been widely adopted in code generation [10, 29, 51, 53, 31,
56, 41]. Instruction tuning constructs high-quality datasets consisting of
instruction-response (i.e. description-code) pairs and then fine-tunes LLMs on
them. These datasets are constructed either from human-crafted resources or
synthetic data [54], where human-crafted resources include public data
collection [31, 41] and human annotating [34], and synthetic data means to
obtain code responses by prompting advanced LLMs (e.g., GPT-3.5/4) with self-
generated descriptions [10, 29, 51, 53, 56, 41]. In Verilog generation, for
example, RTLCoder [26] synthesizes description-code pairs by prompting GPT-3.5
to generate and mutate descriptions and get the corresponding code from
GPT-3.5. BetterV [35] uses pre-defined descriptions for collected and self-
generated open-source Verilog code snippets.

Figure 1: The CodeV framework overview. We first collect and filter high-
quality Verilog modules from open-source codebases. The modules are then sent
to GPT-3.5 to request multi-level summaries. Pairing high-level descriptions
with corresponding modules, the high-quality dataset is utilized to fine-tune
base LLMs, yielding CodeV models. Figure 2: GPT-generated Verilog dataset
contains unrealistic code. (a) A synthetic description obtained from [26],
which is unrealistic. (b) The circuit’s block diagram according to the
description, contains illegal loops in the circuit. (c) The corresponding
incorrect Verilog code in the dataset.

However, there are still two challenges hindering the application of
instruction tuning to Verilog generation tasks. (1) The scarcity of human-
crafted description-code pairs in Verilog. The number of programming
competitions and large open-source projects in Verilog is far less than that
in general-purpose programming languages like Python [21]. Additionally, the
cost of human annotation for Verilog is much higher than for languages like
Python, as it requires annotators to be familiar with hardware circuits,
making the construction of comprehensive datasets via annotation unfeasible.
(2) The difficulty in obtaining high-quality Verilog responses from advanced
LLMs. Although advanced LLMs excel on general-purpose code generation tasks,
they perform poorly in Verilog generation tasks. For example, GPT-4 achieves
$88.4\%$ pass@1 in the Python benchmark HumanEval [11, 22], but only achieves
$43.5\%$ pass@1 in the VerilogEval-human benchmark [24].

Regarding these issues, we have observed the following: (1) Verilog code
collected from the real world is of higher quality than that generated by
LLMs. Verilog code collected from the real world has fewer syntax errors,
clearer semantics, and better conforms to the code distribution in practical
applications compared to Verilog code generated by LLMs. Figure 2a-c
illustrates an example observed from the GPT-generated Verilog dataset
provided by Liu et al. [26]. Figure 2a shows a description obtained by
combining two circuit descriptions. We draw a block diagram based on this
description, and as shown in Figure 2b, this description leads to a loop in a
combinational circuit, which is illegal. Figure 2c depicts the circuit
generated by GPT-3.5 based on this incorrect description, which fails to
integrate the shifter and multiplexer, and the internal implementation of the
shifter module is also incorrect. (2) LLMs excel in summarizing code rather
than generating it. Although GPT-3.5’s poor performance in generating Verilog
(only $26.7\%$ pass@1 in VerilogEval-human benchmarks), it indeed offers the
ability to describe the functionality of given Verilog code. Based on these
observations, we introduce CodeV, a series of instruction-tuned Verilog
generation LLMs that employ reverse instruction generation for dataset
construction. Instead of generating descriptions first and then getting the
corresponding code from an advanced LLM, we prompt the LLM with Verilog code
and let the LLM generate the corresponding natural language descriptions.
Figure 1 illustrates the CodeV’s framework overview. First, we crawl over
1500K Verilog codes from GitHub, filter them, and obtain a deduplicated, self-
contained, and syntax-correct set containing 165K high-quality Verilog
modules. Subsequently, we use a multi-level code summarization method to
bridge the semantic gap between high-level requirement descriptions and low-
level Verilog code. Specifically, we let GPT-3.5 to first generate fine-
grained functional descriptions based on given Verilog modules, and then
summarize high-level requirements, resulting in a dataset of high-quality
description-code pairs. Finally, we fine-tune base LLMs including CodeLlama,
DeepSeekCoder, and CodeQwen on this dataset to get CodeV.

Experimental results show that CodeV series achieve state-of-the-art (SOTA)
results on the VerilogEval [24] and RTLLM [28] benchmarks, relatively
averagely surpassing the ever-reported SOTA open-source instruction-tuned LLMs
by 14.4% (BetterV [35] in VerilogEval) and 11.3% (RTLCoder [26] in RTLLM)
respectively, and also relatively surpassing the previous SOTA commercial LLM
GPT-4 by 22.1% in VerilogEval. The major contributions of this paper are as
follows:

  * •

We propose an effective description-code dataset construction approach by
providing GPT-3.5 with Verilog code to summarize corresponding descriptions in
a multi-level manner. We show that this method outperforms prior data
construction work on Verilog generation tasks.

  * •

Based on this method, we present a series of SOTA Verilog generation LLMs,
namely CodeV. Among them, CodeV-CodeQwen achieves 77.6% pass@1 on the
VerilogEval-machine benchmark and 53.2% on the VerilogEval-human benchmark,
outperforming GPT-4 and previous SOTA model BetterV, and also achieves a 93.1%
syntax pass rate and 55.2% function pass rate on the RTLLM benchmark,
outperforming previous SOTA model RTLCoder.

  * •

We plan to open-source a series of CodeV models and an instruction tuning
dataset containing 165K high-quality description-code pairs, to support the
advancement and collaboration within the LLM, electronic design automation
(EDA), and programming language communities.

##  2 Related Work

###  2.1 LLM-based Verilog Generation

##### Models for verilog generation.

Thakur et al. [43] collected approximately 400MB of Verilog files from GitHub
and textbooks to fine-tune CodeGen-16B [32]. However, the dataset was not
refined, resulting in a blend of text and code, with some Verilog code
potentially containing syntax errors. ChipNemo [23] used LLaMA2 [46] as its
base model, performing domain-adaptive pre-training on public datasets and
NVIDIA’s internal chip design files (including HDL code, EDA scripts, and
configuration files, etc.), and fine-tuning on public instruction datasets and
an expert-crafted domain dataset with 1430 examples. However, likely due to
the dataset not being refined for Verilog generation tasks, the 70B version
model of ChipNemo even underperforms its base model on the VerilogEval-Human
benchmark. RTLCoder [25] uses GPT to synthesize instruction-code pairs,
starting by generating instructions based on a keywords pool and a source code
pool, then mutating these instructions to increase their number and diversity,
and finally having GPT generate Verilog code based on these synthesized
instructions to form a fine-tuning dataset. This approach may lead to
unrealistic instructions, and the code quality is limited by GPT’s generation
ability. BetterV [35] collects Verilog code from GitHub and expands its
dataset by translating Verilog code into C to create C-Verilog pairs,
leveraging the base model’s knowledge in C to enhance Verilog generation
capabilities. However, they did not validate the effectiveness of this
approach by experiments. In contrast to these methods, CodeV employs a novel
dataset construction approach, utilizing multi-level summarization to generate
high-quality instruction tuning datasets from high-quality Verilog code,
achieving SOTA results.

##### Frameworks on verilog generation.

Instead of training specific models, some work focuses on designing generation
frameworks for Verilog tasks to improve general-purpose LLMs’ performance.
Chip-Chat [5] and ChipGPT [9] use multi-round EDA and human feedback to assist
LLMs for Verilog generation but require excessive human intervention,
hindering automation. Lu et al. [28] introduced a self-planning method to
improve generation accuracy, where LLMs plan before generating Verilog.
However, the improvement is limited due to the deficiency of general-purpose
LLMs in Verilog Generation.

###  2.2 Instruction Tuning

Instruction tuning has emerged as a powerful strategy to enhance the
capabilities of LLMs by fine-tuning them with a mixture of instructions and
corresponding responses. Initially explored through models like T5 [36], which
was trained on a diverse array of public NLP tasks, this approach has seen
significant advancements over time. Notable developments include FLAN [50],
ExT5 [1], T0 [38], and UnifiedQA [19], which expanded the range of tasks to
improve cross-task generalization abilities of LLMs.

Using human annotators, OpenAI has developed models such as InstructGPT [34]
and ChatGPT by training on a large corpus of human-generated instructions,
aligning the model outputs more closely with real-world user intentions. This
dataset, however, remains proprietary and not publicly accessible. Meanwhile,
Vicuna [12] has explored using user-shared conversations from platforms like
ShareGPT.com for training, illustrating a community-driven approach to data
collection.

Without human annotators, Self-Instruct [47] utilize foundation LLMs like
GPT-3 [7] to generate synthetic instruction-response pairs. This methodology
allowed the same LLM to be instruction-tuned on its self-generated data,
thereby distilling knowledge from advanced LLMs without external inputs.
Following this, models like Alpaca [42] and Code Alpaca [10] applied Self-
Instruct to fine-tune Llama [44] using ChatGPT-generated instructions.
Furthermore, WizardLM [52] introduced Evol-Instruct to utilize heuristic
prompts to guide ChatGPT, creating more complex and diverse datasets.

For code generation, WizardCoder [29] introduced Code Evol-Instruct to adapt
Evol-Instruct to coding problems. Codellama-Instruct [37] are finetuned from
Codellama using Self-Instruct by prompting Llama 2 [45] for coding problems
and then ask Codellama for unit tests filtering and code solutions. OctoPack
[31] collects open-source commit information as a data source for instruction
tuning. Taking into account the scenario of modifying code through multiple
rounds of interaction, OpenCodeInterpreter [56] constructs a multi-turn
instruction tuning dataset by using GPT-3.5/GPT-4 to simulate human feedback
and incorporating execution feedback. Magicoder [51] integrates real-world
code snippets into the instruction generation process to enhance the
controllability, quality, and creativity in synthetic coding tasks.

However, due to the high cost of annotation and the generally poor performance
of advanced LLMs like GPT-4, these methods fail on the task of Verilog
generation. Our work aims to fill this gap using a reverse instruction method.

###  2.3 Code Generation Models for General-purposed Language

LLMs trained on extensive codebases have proven to be highly effective in code
generation. Among them, closed-source models including Codex by OpenAI [11],
PaLM-Coder by Google [13], and Code-Davinci by Microsoft [30] have
demonstrated remarkable capabilities in popular code generation benchmarks
like HumanEval [11] and MBPP [2]. On the other hand, there are also several
notable open-source works like InCoder [15], CodeGen series [32], CodeT5
series [48, 49], CodeGeeX [55], StarCoder series [21, 27], CodeLlama-Series
[37], PanGu-Coder series [14, 40], AIXCoder [17], Deepseek-Coder [16],
CodeQwen [4]. Although these models perform well with general-purpose
programming languages, they perform poorly with HDL like Verilog due to a lack
of training data. To address this issue, our work proposes an instruction
tuning method for fine-tuning these models on Verilog generation and
introduces CodeV, a series of SOTA Verilog generation models.

##  3 Methods

###  3.1 High-quality Verilog Code Collection

High-quality real-world Verilog code is the foundation of our approach. We
first crawl Verilog and SystemVerilog files from GitHub, ensuring each file
contains a complete module, i.e. contains at least one "module" and
"endmodule" pair. Files with external references, i.e. code containing
"include" or "import" keywords, are excluded. This is because reconstructing
these reference relationships from the crawled files is challenging, and we do
not want LLM to generate these non-self-contained Verilog codes after
training. Then we deal with the comments interspersed throughout the files.
These comments can be categorized into two types: those related to Verilog
implementation, such as functional descriptions, test cases, etc.; and those
unrelated to implementation, such as licenses, company and author information,
logs, etc. The former is essential for LLMs to understand the code and should
be kept, while the latter should be avoided. Therefore, we use regular
expressions to delete these implementation-unrelated comments. Next, files
exceeding $2096$ characters are deleted, as longer files would be truncated
during training, resulting in incomplete file content and degrading the
dataset quality. Following Thakur et al. [43], we also use MinHash and Jaccard
similarity to filter out duplicate code. Finally, we run syntax checks on all
Verilog code and remove any that fail the check, to further enhance the
quality of the dataset.

Through the above-mentioned effort, we obtain high-quality real-world Verilog
code. Next, we will construct requirement descriptions for them.

###  3.2 Multi-level Code Summarization

Figure 3: An actual example of the prompt for multi-level summarization. (a)
The prompt provided to GPT-3.5. (b) An example of the demonstrations, with
code, low-level descriptions, and high-level summaries. (c) Summaries
responded from GPT-3.5 with and (d) without multi-level summarization. Figure
4: The similarity distribution of our dataset against benchmark data. Data
with Rouge-L scores greater than $0.5$ are removed for decontamination.

Manual annotation is prohibitively time-consuming and costly. Hence, we
employed GPT-3.5 to generate high-level summaries for each Verilog module as
its requirement description. As analyzed in VerilogEval [24], when required
for summarising, LLMs often produce verbose descriptions, preferring line-by-
line explanations over high-level summaries. To address this issue, we
introduce a multi-level summarization method, employing few-shot learning [6]
to guide GPT-3.5 in first producing detailed descriptions and then abstracting
high-level summaries. Figure 3 presents an example from our actual process,
involving the generation of requirement descriptions for a demultiplexer
module collected from GitHub. Figure 3a illustrates our prompt, which includes
$n$ examples with code snippets, detailed descriptions, and problem summaries,
followed by the demultiplexer code, and a request for GPT to imitate the
examples by first providing a detailed description and then a problem summary.
Figure 3b illustrates an example used for GPT’s few-shot learning. Figure 3c
shows GPT’s response using multi-level summarization, while Figure 3d shows
the response without multi-level summarization by deleting the detailed
description in the demonstrations. We observe that the description in Figure
3c is more concise and aligns with actual Verilog development requirements. In
contrast, the description in Figure 3d is verbose and even inaccurately counts
the number of zeros in output signals, leading to a contradictory explanation.

###  3.3 Data Decontamination

To ensure a fair evaluation, we conduct decontamination for the training data
to remove sequences similar to benchmarks. Specifically, we use Rouge-L, a
widely used similarity metric, to measure the similarity between data. Rouge-L
$\in[0,1]$, with higher values indicating greater similarity. Following
RTLCoder [26], we remove data from the training set that have a Rouge-L score
greater than $0.5$ with any of the benchmarks, thereby preventing test data
leakage. The Rouge-L plot is shown in Figure 4.

###  3.4 Supervised Finetuning

Finally, we finetune base models on our dataset. Given a natural language
description, the LLM with parameter $\theta$ generates code repeatedly drawing
from $p_{\theta}(x_{t}|x_{<t})$ and using $x_{t}$ as part of the input for the
subsequent prediction. Following prior work, the LLM is designed to reduce the
negative log-likelihood across the dataset D $\\{x^{1},...,x^{|D|}\\}$ [18]:

$\mathcal{L}=-\frac{1}{|D|}\sum_{i=1}^{|D|}\frac{1}{n}\sum_{t=1}^{n}\log
p_{\theta}(x^{i}_{t}|x^{i}_{<t}).$ (1)

##  4 Experiments

We conduct comprehensive experiments demonstrating that CodeV achieves SOTA
results across diverse Verilog generation tasks, and we further analyze
CodeV’s performance as detailed below.

###  4.1 Experimental Setup

####  4.1.1 Instruction-tuning

We used CodeLlama-7b-Instruct [37], Deepseek-Coder-6.7b-Instruct [16], and
CodeQwen1.5-7B-Chat [4] as our base models. We fine-tuned these models for $3$
epochs using PyTorch’s Distributed Data Parallel (DDP) on 4 NVIDIA A100-80GB
SMX GPUs. We employed the Adafactor optimizer [39] with a linear learning rate
scheduler, setting the initial learning rate at $5e-5$ with $15$ warm-up
steps. We set global batch size $256$, with max sequence length $2048$.

####  4.1.2 Benchmarks and Metrics

We conduct comprehensive evaluations on the VerilogEval [24] and RTLLM [28]
benchmarks.

VerilogEval includes two parts: VerilogEval-machine and VerilogEval-human,
with $143$ GPT-generated and $156$ hand-crafted Verilog generation problems,
respectively. We follow the VerilogEval paper, using the pass@k metric to
measure the Verilog generation accuracy. The pass@k metric estimates the
proportion of problems that can be solved at least once in $k$ attempts:

$\mathbf{pass}@k:=\underset{\text{ problems
}}{\mathbb{E}}\left[\frac{1-\binom{n-c}{k}}{\binom{n}{k}}\right]$ (2)

where $n\geq k$ represents the total number of trials for each problem, and
$c$ represents the number of trials that pass the functional check. We set
$n=20$ according to the original settings.

RTLLM contains $29$ Verilog generation problems. Following prior work [26], we
test models on RTLLMv1.1 which is error-fixed. Also, following the original
settings, we generate Verilog code for each problem $5$ times and report the
success rate (proportion of problems that at least one trial passes the syntax
check and function check, respectively). We acknowledge that employing the
pass@k metric with a greater number of trials ($n$ in Eq. 2) yields less
random results; therefore, we extend our evaluation on the RTLLM benchmark in
Section 4.4.1, setting $n=20$ for each generation task which is similar to
VerilogEval.

Following RTLCoder [26], for all tests, we set the generation temperature of
each model to 0.2, 0.5, 0.8, and report the best performance across different
temperature settings, but we also present results on three temperatures
separately in Section B.0.1.

####  4.1.3 Baseline Methods

In our experiment, we compare our results with baseline methods, including:
(1) foundation models not tailored for Verilog generation tasks, including
closed-source commercial LLMs such as GPT-3.5 and GPT-4, and four open-source
models specialized in code generation tasks: Starcoder [21],
CodeLlama-7b-Instruct [37], DeepSeek-Coder-6.7b-Instruct [16], and
CodeQwen1.5-7B-Chat [4]; (2) models fine-tuned for Verilog generation tasks,
including Thakur et al. [43], RTLCoder [25] and BetterV [35].

###  4.2 Main Results

Table 1: Comparison of our CodeV models against various baseline models. Some
data are missing due to the models being close-sourced and the data not being
reported previously. The best results are highlighted in bold. In VerilogEval,
CodeV outperforms all previous methods across all metrics, relatively
surpassing previous open-source SOTA BetterV by 14.4% and GPT-4 by 22.1% on
average. In RTLLM, CodeV relatively surpasses previous open-source SOTA
RTLCoder by 11.3%.

Type | Model |  Open source | VerilogEval | RTLLM v1.1
---|---|---|---|---
Machine (%) | Human (%) |  Syntax (%) |  Func. (%)
pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10
Foundation models | GPT-3.5 | ○ | 46.7 | 69.1 | 74.1 | 26.7 | 45.8 | 51.7 | 89.7 | 37.9
GPT4 | ○ | 60.0 | 70.6 | 73.5 | 43.5 | 55.8 | 58.9 | 100.0 | 65.5
Starcoder | ● | 46.8 | 54.5 | 59.6 | 18.1 | 26.1 | 30.4 | 93.1 | 27.6
CodeLlama | ● | 43.1 | 47.1 | 47.7 | 18.2 | 22.7 | 24.3 | 86.2 | 31.0
DeepSeek-Coder | ● | 52.2 | 55.4 | 56.8 | 30.2 | 33.9 | 34.9 | 93.1 | 44.8
CodeQwen | ● | 46.5 | 54.9 | 56.4 | 22.5 | 26.1 | 28.0 | 86.2 | 41.4
IT baselines | ChipNeMo | ○ | 43.4 | - | - | 22.4 | - | - | - | -
Thakur et al. | ● | 44.0 | 52.6 | 59.2 | 30.3 | 43.9 | 49.6 | 86.2 | 24.1
RTLCoder-Mistral | ● | 62.5 | 72.2 | 76.6 | 36.7 | 45.5 | 49.2 | 96.6 | 48.3
RTLCoder-DeepSeek | ● | 61.2 | 76.5 | 81.8 | 41.6 | 50.1 | 53.4 | 93.1 | 48.3
BetterV-CodeLlama | ○ | 64.2 | 75.4 | 79.1 | 40.9 | 50.0 | 53.3 | - | -
BetterV-DeepSeek | ○ | 67.8 | 79.1 | 84.0 | 45.9 | 53.3 | 57.6 | - | -
BetterV-CodeQwen | ○ | 68.1 | 79.4 | 84.5 | 46.1 | 53.7 | 58.2 | - | -
CodeV | CodeV-CodeLlama | ● | 78.1 | 86.0 | 88.5 | 45.2 | 59.5 | 63.8 | 93.1 | 62.1
CodeV-DeepSeek | ● | 77.9 | 88.6 | 90.7 | 52.7 | 62.5 | 67.3 | 89.7 | 55.2
CodeV-CodeQwen | ● | 77.6 | 88.2 | 90.7 | 53.2 | 65.1 | 68.5 | 93.1 | 55.2

Table 1 compares the main results of our CodeV with baseline methods on the
VerilogEval and RTLLM benchmarks. We test CodeLlama, DeepSeek-Coder, and
CodeQwen on RTLLM, while other baseline results are sourced from RTLCoder [26]
or BetterV [35] paper. For a fair comparison, we also evaluate our models
trained on comparable-size datasets against RTLCoder. The results show that:

##### CodeV achieves SOTA on both benchmarks.

In the VerilogEval benchmark, CodeV outperforms all previous baseline methods
across all metrics, particularly in the most challenging pass@1 metric where
CodeV-CodeLlama achieves 78.1% in VerilogEval-machine, and CodeV-CodeQwen
achieves 53.1% in VerilogEval-human, significantly outperforming both GPT-4
and BetterV-CodeQwen. In the RTLLM v1.1 benchmark, CodeV-CodeLlama nearly
matches GPT-4’s functional check success rate, failing in just 1 more case out
of 29 circuits compared to GPT-4, and significantly outperforms all other
models. These results demonstrate that, contributed by a high-quality
instruction tuning dataset, CodeV exhibits significant superiority in Verilog
generation tasks.

##### The differences among the base models decrease after fine-tuning.

This is reasonable because the base models are trained on different training
data but use the same instruction tuning data, which reduces their
differences. Additionally, we find that after fine-tuning, CodeQwen surpasses
DeepSeek-Coder, and CodeLlama also outperformed the other two models on
VerilogEval-Machine pass@1 after fine-tuning. This indicates that the
performance of a base model in specific domains does not necessarily align
with its performance after instruction tuning.

##### There is still a large gap between Human and Machine on VerilogEval.

This stays the same as prior work. There are two reasons for this: First, the
descriptions for Machine are generated by GPT, so they are closer to the
distribution of LLMs compared with Human. Second, some of the Human
desciptions contain tabular data like truth tables, which are difficult to
obtain from the open-source codebase or generate from GPT, leading to a more
significant distribution difference in the descriptions.

##### Our method yields better data.

As shown in Table 2, under the same data size (i.e. 27K), the results obtained
from our approach surpass those of RTLCoder, especially for its directly
trained version. It is worth noting that in addition to data distillation,
RTLCoder also proposes a training method based on syntax checking, which is
orthogonal with our method. We cannot compare with BetterV because its data
size has not been disclosed. More analytical results on data size are shown in
Section 4.4.2.

Table 2: Comparison under same training set size. Trained with a randomly
chosen 27K dataset, CodeV outperforms RTLCoder in 5 out of 8 metrics and fully
surpasses its directly-trained version.

Model | VerilogEval | RTLLM v1.1
---|---|---
Machine (%) | Human (%) |  Syntax (%) |  Func. (%)
pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10
RTLCoder-DeepSeek | 61.2 | 76.5 | 81.8 | 41.6 | 50.1 | 53.4 | 93.1 | 48.3
RTLCoder-DeepSeek-Direct | 59.8 | 73.6 | 77.2 | 39.1 | 48.3 | 51.3 | 86.2 | 44.8
CodeV-DeepSeek-27K | 72.3 | 82.2 | 85.6 | 39.1 | 53.4 | 58.3 | 89.6 | 44.8

###  4.3 Ablation Study

In this section, we measured the effectiveness of multi-level summarization.
Due to budget constraints, we conducted this experiment only on a 10K dataset.
We sampled 10K data from the multi-level summarization dataset and generated
single-level summarization (i.e. summarize directly) for the same Verilog
code. The results in Table 3 show that the instruction data obtained through
multi-level summarization can effectively enhance model performance which is
consistent with our expectations.

Table 3: Ablation study on multi-level summarization (MLS). The models are
trained on 10K datasets due to budget constraints. Using multi-level
summarization methods, the performance of the CodeV increases in all metrics.

Model | Temp. | VerilogEval
---|---|---
Machine (%) | Human (%)
pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10
CodeV-CodeLlama without MLS | 0.2 | 59.0 | 66.3 | 68.9 | 28.3 | 37.5 | 40.3
0.5 | 59.5 | 72.1 | 75.8 | 25.1 | 40.4 | 45.7
0.8 | 54.7 | 72.7 | 77.5 | 21.4 | 39.2 | 45.0
CodeV-CodeLlama with MLS | 0.2 | 63.0 | 71.2 | 72.9 | 31.7 | 38.1 | 39.8
0.5 | 60.9 | 73.1 | 76.5 | 29.2 | 41.4 | 45.7
0.8 | 58.5 | 73.9 | 78.1 | 25.7 | 41.5 | 47.6

###  4.4 Further Analysis

####  4.4.1 Evaluating RTLLM with pass@k metric

As shown in Eq. 2, the pass@k metric provides more unbiased estimations. In
this section, we set $n=20$ to further evaluate our models on RTTLM using
pass@k metrics and conduct varying comparisons with baselines. It is
noteworthy that, even with 5 trials for each problem, the success rate metric
in original RTLLM settings does not equal pass@5 metric, hence one cannot
directly compare these two metrics. Table 4 illustrates that, CodeV exhibits
huge superiority compared to other baseline models, especially for functional
correctness.

Table 4: Evaluations on RTLLM using pass@k metrics. CodeV models still surpass
all baseline models with a more unbiased pass@k metric, especially for
functional correctness.

Type | Model | RTLLM v1.1
---|---|---
Syntax (%) | Func. (%)
pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10
foundation models | CodeLlama | 57.4 | 78.4 | 83.8 | 16.1 | 28.9 | 35.4
DeepSeek-Coder | 73.1 | 86.0 | 89.1 | 28.8 | 38.8 | 43.3
CodeQwen | 73.1 | 86.0 | 89.1 | 28.8 | 38.8 | 43.3
IT baselines | RTLCoder-Mistral | 64.6 | 73.7 | 78.3 | 24.5 | 37.3 | 42.3
RTLCoder-DeepSeek | 73.4 | 83.9 | 86.2 | 35.8 | 40.3 | 43.1
CodeV | CodeV-CodeLlama | 79.0 | 89.2 | 89.9 | 39.4 | 50.3 | 53.1
CodeV-DeepSeek | 78.3 | 87.4 | 89.1 | 42.4 | 51.5 | 53.2
CodeV-CodeQwen | 78.8 | 89.5 | 92.4 | 36.6 | 53.3 | 61.3

####  4.4.2 Impact of Data Size Scaling

We randomly sampled subsets of various sizes (27K, 50K, 100K) from the
complete 165K dataset and fine-tuned all three baseline models, reporting
their evaluation results on VerilogEval as shown in Figure 5. Except for
CodeV-CodeQwen on VerilogEval-machine, which peaks on 100K-size, we observed
that model performance generally improves as training set size increases, but
with diminishing returns. This suggests that for 7B models, the benefits of
further increasing the training set size may be limited, and future efforts
should focus more on enhancing the quality of the dataset.

Figure 5: Analysis on dataset sizes. We train CodeV on different-size datasets
and evaluate them on VerilogEval. The performance of models improves with the
increase in training set size, but the returns gradually diminish.

##  5 Conclusion

In this paper, we present CodeV, a series of Verilog generation LLMs. By
collecting Verilog code from the real world and utilizing multi-level
summarization to generate high-level descriptions, we successfully construct a
high-quality dataset and train multiple CodeV models based on it. Our
experimental results have shown CodeV’s effectiveness, showcasing superior
performance over both previous open-source and commercial state-of-the-art
models by significant margins. Specifically, CodeV relatively exceeds GPT-4 in
VerilogEval by 22.1%. This highlights its practical utility in the domain of
circuit design automation.

##  6 Limitations

Our work has two main limitations: first, CodeV is unable to directly generate
complex circuits without the assistance of a framework, such as the one
proposed by Lu et al. [28]; second, CodeV lacks the capability for circuit
optimization. However, our approach is orthogonal to theirs, as our model can
serve as the core of a Verilog Generation Agent or produce functionally
correct Verilog code for subsequent optimization. We acknowledge these
limitations and believe there is an opportunity to further enhance the model’s
performance in real-world Verilog generation applications, leaving these
improvements for future work.

## References

  * Aribandi et al. [2021] Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al.  Ext5: Towards extreme multi-task scaling for transfer learning.  _arXiv preprint arXiv:2111.10952_ , 2021.
  * Austin et al. [2021] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ , 2021.
  * Bachrach et al. [2012] Jonathan Bachrach, Huy Vo, Brian Richards, Yunsup Lee, Andrew Waterman, Rimas Avižienis, John Wawrzynek, and Krste Asanović.  Chisel: constructing hardware in a scala embedded language.  In _Proceedings of the 49th Annual Design Automation Conference_ , pages 1216–1225, 2012.
  * Bai et al. [2023] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_ , 2023.
  * Blocklove et al. [2023] Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce.  Chip-chat: Challenges and opportunities in conversational hardware design.  In _2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD)_. IEEE, sep 2023.  doi: 10.1109/mlcad58807.2023.10299874.  URL https://doi.org/10.1109%2Fmlcad58807.2023.10299874.
  * Brown et al. [2020a] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020a.
  * Brown et al. [2020b] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020b.
  * Canis et al. [2011] Andrew Canis, Jongsok Choi, Mark Aldham, Victor Zhang, Ahmed Kammoona, Jason H Anderson, Stephen Brown, and Tomasz Czajkowski.  Legup: high-level synthesis for fpga-based processor/accelerator systems.  In _Proceedings of the 19th ACM/SIGDA international symposium on Field programmable gate arrays_ , pages 33–36, 2011.
  * Chang et al. [2023] Kaiyan Chang, Ying Wang, Haimeng Ren, Mengdi Wang, Shengwen Liang, Yinhe Han, Huawei Li, and Xiaowei Li.  Chipgpt: How far are we from natural language hardware design, 2023.
  * Chaudhary [2023] Sahil Chaudhary.  Code alpaca: An instruction-following llama model for code generation.  https://github.com/sahil280114/codealpaca, 2023.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2(3):6, 2023.
  * Chowdhery et al. [2023] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113, 2023.
  * Christopoulou et al. [2022] Fenia Christopoulou, Gerasimos Lampouras, Milan Gritta, Guchun Zhang, Yinpeng Guo, Zhongqi Li, Qi Zhang, Meng Xiao, Bo Shen, Lin Li, et al.  Pangu-coder: Program synthesis with function-level language modeling.  _arXiv preprint arXiv:2207.11280_ , 2022.
  * Fried et al. [2022] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis.  Incoder: A generative model for code infilling and synthesis.  _arXiv preprint arXiv:2204.05999_ , 2022.
  * Guo et al. [2024] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al.  Deepseek-coder: When the large language model meets programming–the rise of code intelligence.  _arXiv preprint arXiv:2401.14196_ , 2024.
  * Hao et al. [2022] Yiyang Hao, Ge Li, Yongqiang Liu, Xiaowei Miao, He Zong, Siyuan Jiang, Yang Liu, and He Wei.  Aixbench: A code generation benchmark dataset.  https://github.com/aixcoder-plugin/aiXcoder-7B, 2022.
  * Keskar et al. [2019] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher.  Ctrl: A conditional transformer language model for controllable generation.  _arXiv preprint arXiv:1909.05858_ , 2019.
  * Khashabi et al. [2020] Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi.  Unifiedqa: Crossing format boundaries with a single qa system.  _arXiv preprint arXiv:2005.00700_ , 2020.
  * Lahti et al. [2018] Sakari Lahti, Panu Sjövall, Jarno Vanne, and Timo D Hämäläinen.  Are we there yet? a study on the state of high-level synthesis.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 38(5):898–911, 2018.
  * Li et al. [2023] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.  Starcoder: may the source be with you!  _arXiv preprint arXiv:2305.06161_ , 2023.
  * Liu et al. [2024] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Liu et al. [2023a] Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al.  Chipnemo: Domain-adapted llms for chip design.  _arXiv preprint arXiv:2311.00176_ , 2023a.
  * Liu et al. [2023b] Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren.  Verilogeval: Evaluating large language models for verilog code generation.  In _2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD)_ , pages 1–8. IEEE, 2023b.
  * Liu et al. [2023c] Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, and Zhiyao Xie.  Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution.  _arXiv preprint arXiv:2312.08617_ , 2023c.
  * Liu et al. [2023d] Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, and Zhiyao Xie.  Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution.  _arXiv preprint arXiv:2312.08617_ , 2023d.
  * Lozhkov et al. [2024] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al.  Starcoder 2 and the stack v2: The next generation.  _arXiv preprint arXiv:2402.19173_ , 2024.
  * Lu et al. [2024] Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie.  Rtllm: An open-source benchmark for design rtl generation with large language model.  In _2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC)_ , pages 722–727. IEEE, 2024.
  * Luo et al. [2023] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  Wizardcoder: Empowering code large language models with evol-instruct.  _arXiv preprint arXiv:2306.08568_ , 2023.
  * Microsoft [2023] Microsoft.  Azure openai service models.  https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models, 2023.
  * Muennighoff et al. [2023] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre.  Octopack: Instruction tuning code large language models.  _arXiv preprint arXiv:2308.07124_ , 2023.
  * Nijkamp et al. [2022] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  Codegen: An open large language model for code with multi-turn program synthesis.  _arXiv preprint arXiv:2203.13474_ , 2022.
  * OpenAI [2023] R OpenAI.  Gpt-4 technical report. arxiv 2303.08774.  _View in Article_ , 2023.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744, 2022.
  * Pei et al. [2024] Zehua Pei, Hui-Ling Zhen, Mingxuan Yuan, Yu Huang, and Bei Yu.  Betterv: Controlled verilog generation with discriminative guidance.  _arXiv preprint arXiv:2402.03375_ , 2024.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ , 21(140):1–67, 2020.
  * Roziere et al. [2023] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ , 2023.
  * Sanh et al. [2021] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al.  Multitask prompted training enables zero-shot task generalization.  _arXiv preprint arXiv:2110.08207_ , 2021.
  * Shazeer and Stern [2018] Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _International Conference on Machine Learning_ , pages 4596–4604. PMLR, 2018.
  * Shen et al. [2023] Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, et al.  Pangu-coder2: Boosting large language models for code with ranking feedback.  _arXiv preprint arXiv:2307.14936_ , 2023.
  * Shypula et al. [2023] Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh.  Learning performance-improving code edits.  _arXiv preprint arXiv:2302.07867_ , 2023.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Stanford alpaca: An instruction-following llama model, 2023.
  * Thakur et al. [2023] Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Hammond Pearce, Benjamin Tan, Ramesh Karri, Brendan Dolan-Gavitt, and Siddharth Garg.  Benchmarking large language models for automated verilog rtl code generation.  In _2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)_, pages 1–6. IEEE, 2023.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Touvron et al. [2023c] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023c.
  * Wang et al. [2022] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language models with self-generated instructions.  _arXiv preprint arXiv:2212.10560_ , 2022.
  * Wang et al. [2021] Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  _arXiv preprint arXiv:2109.00859_ , 2021.
  * Wang et al. [2023] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi.  Codet5+: Open code large language models for code understanding and generation.  _arXiv preprint arXiv:2305.07922_ , 2023.
  * Wei et al. [2021] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_ , 2021.
  * Wei et al. [2023] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang.  Magicoder: Source code is all you need.  _arXiv preprint arXiv:2312.02120_ , 2023.
  * Xu et al. [2023] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_ , 2023.
  * Yu et al. [2023] Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin.  Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation.  _arXiv preprint arXiv:2312.14187_ , 2023.
  * Zhang et al. [2023] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al.  Instruction tuning for large language models: A survey.  _arXiv preprint arXiv:2308.10792_ , 2023.
  * Zheng et al. [2023] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al.  Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x.  _arXiv preprint arXiv:2303.17568_ , 2023.
  * Zheng et al. [2024] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue.  Opencodeinterpreter: Integrating code generation with execution and refinement.  _arXiv preprint arXiv:2402.14658_ , 2024.

##  Appendix A Broader Impacts

Compared to existing models, CodeV has the highest accuracy in Verilog
generation tasks. This demonstrates that our proposed instruction tuning
dataset construction method and CodeV can further push the boundaries of
circuit design automation and lower the barrier to circuit design.

However, like other code models, CodeV may generate code that does not align
with user intentions and could be misused. On this point, we refer readers to
the comprehensively discussed broader impacts and hazard analysis in Chen et
al. [11]. Due to the lower usage threshold of code LLMs compared to
programming languages, there may be certain security issues. Therefore, we
recommend users conduct a thorough functionality verification and security
review of the code generated by CodeV to avoid potential losses.

##  Appendix B Additional Experimental Results

####  B.0.1 Impact of Generation Temperature

Figure 6: Analysis on generation temperature. We test our models on
VerilogEval, setting different temperatures. As temperature rises, the pass@1
metric decreases while pass@5 and pass@10 decrease. Figure 7: Token Count
Distribution of instructions generated by GPT-3.5, with and without multi-
level summarization, as well as corresponding code. Utilizing multi-level
summarization, the generated instructions are more concise. Figure 8: The
field distribution of our dataset summarized with GPT-3.5.

Figure 6 further analyzes the VerilogEval results of our models when setting
on different generation temperatures. Generally, as the temperature increases,
the pass@1 metric decreases while pass@5 and pass@10 increase. This is because
when the temperature rises, the model tends to generate more uncertain and
diverse responses, which has two effects: on the one hand, uncertain responses
are more likely to produce incorrect results, lowering the accuracy of
individual Verilog codes; on the other hand, the diversity enhances the
exploration of various potential responses, thereby increasing the probability
of at least one correct result being present among multiple Verilog codes.

###  B.1 Instructions token count distributions with and without multi-level
summarization

Figure 7 illustrates the token count distributions of 10K descriptions
generated by GPT-3.5, with or without multi-level summarization, and that of
the corresponding code. It shows that employing multi-level summarization
results in descriptions with fewer tokens, exhibiting a more left-skewed
distribution compared to those without summarization, which aligns better with
actual Verilog design tasks, demonstrating the effectiveness of our approach.

###  B.2 The field distribution of our dataset.

We sampled 27K data and utilized GPT to analyze the distribution of Verilog
code within our dataset and depict the statistics in Figure 8. It is evident
that the Verilog code we collected covers a wide range of fields, including
signal processing circuits, and arithmetic circuits, among others, reflecting
the high quality of our data set.

##  Appendix C Dataset Examples From the Study

There are some examples given by CodeV in Table 5.

Table 5: Instruction and response examples of CodeV. Instruction |  Response
---|---
Design a 16-to-1 multiplexer (MUX16) using Verilog. The multiplexer should take 16 input signals (I0-I15) and a 4-bit select signal (S3, S2, S1, S0). The output signal O should be the selected input signal based on the select signal. Use a case statement to implement the multiplexer logic. |  ⬇ module MUX16 ( input I0, I1, I2, I3, I4, I5, I6, I7, I8, input I9, I10, I11, I12, I13, I14, I15, input S0, S1, S2, S3, output reg O ); always @ (*) begin case ({S3, S2, S1, S0}) 4’b0000 : O = I0; 4’b0001 : O = I1; 4’b0010 : O = I2; 4’b0011 : O = I3; 4’b0100 : O = I4; 4’b0101 : O = I5; 4’b0110 : O = I6; 4’b0111 : O = I7; 4’b1000 : O = I8; 4’b1001 : O = I9; 4’b1010 : O = I10; 4’b1011 : O = I11; 4’b1100 : O = I12; 4’b1101 : O = I13; 4’b1110 : O = I14; 4’b1111 : O = I15; endcase end endmodule
Implement a Verilog module for a relu (Rectified Linear Unit) function. The module should take a clock signal, a reset signal, and a 32-bit signed input signal sum-in. It should output three signals - done, sum-out, and relu-out. The relu function should set all negative values in the input sum-in to zero and leave positive values unchanged. The done signal should indicate when the relu operation is completed, sum-out should pass through the input sum-in, and relu-out should be the result of the relu function. |  ⬇ module relu( input clock, input reset, input signed [31:0] sum_in, output reg done, output signed [31:0] sum_out, output [31:0] relu_out ); reg [31:0] relu_out_reg; reg [31:0] sum_out_reg; always @(posedge clock or posedge reset) begin if (reset) begin relu_out_reg <= 32’b0; done <= 1’b0; end else begin sum_out_reg <= sum_in; if (sum_in < 0) relu_out_reg <= 32’b0; else begin relu_out_reg <= sum_in; done <= 1’b1; end end end assign relu_out = relu_out_reg; assign sum_out = sum_out_reg; endmodule