# Integrating AI Tutors in a Programming Course

Iris Ma<EMAIL_ADDRESS>University of California, IrvineIrvineCaliforniaUSA
,  Alberto Krone Martins<EMAIL_ADDRESS>University of California,
IrvineIrvineCaliforniaUSA and  Cristina Videira Lopes<EMAIL_ADDRESS>University of California, IrvineIrvineCaliforniaUSA

###### Abstract.

RAGMan is an LLM-powered tutoring system that can support a variety of course-
specific and homework-specific AI tutors. RAGMan leverages Retrieval Augmented
Generation (RAG), as well as strict instructions, to ensure the alignment of
the AI tutors’ responses. By using RAGMan’s AI tutors, students receive
assistance with their specific homework assignments without directly obtaining
solutions, while also having the ability to ask general programming-related
questions.

RAGMan was deployed as an optional resource in an introductory programming
course with an enrollment of 455 students. It was configured as a set of five
homework-specific AI tutors. This paper describes the interactions the
students had with the AI tutors, the students’ feedback, and a comparative
grade analysis. Overall, about half of the students engaged with the AI
tutors, and the vast majority of the interactions were legitimate homework
questions. When students posed questions within the intended scope, the AI
tutors delivered accurate responses 98% of the time. Within the students used
AI tutors, 78% reported that the tutors helped their learning. Beyond AI
tutors’ ability to provide valuable suggestions, students reported
appreciating them for fostering a safe learning environment free from
judgment.

large language models, education, software engineering, LLMs

††conference: 1st ACM Virtual Global Computing Education Conference; December
2024; Virtual

##  1\. Introduction

The number of students enrolling in Computer Science has increased
significantly in the past decades. This surge is driven by the growing
reliance on software in modern society and the increasing appeal of software
careers. We teach in a large public University in the United States, and our
undergraduate enrollment in Computer Science has reached peak levels, with
close to 1,000 freshmen every year over the past 3 years. Providing
individualized support to a large number of students in introductory
programming classes, especially regarding mastery of challenging material, has
been challenging.

The emergence of conversational chatbots built on top of Large Language Models
(LLMs) presents a potential route for providing students with nteractive
learning experiences and on-demand, personalized assistance with their
homework assignments. We developed RAGMan, a Retrieval-Augmented Generation
(RAG) framework for building AI tutors, and deployed several of these tutors
in an introductory programming course, ICS32. The AI tutors were configured so
that they would not give out the solutions to the homework, and would nudge
the students towards possible next steps relative to their questions. We
collected anonymized usage data, and asked the students to fill out a survey
at the end. Additionally, we had access to the anonymized final grades.

The data from this deployment points to a positive outcome regarding the
benefit, and perceived benefit, of the AI tutors: there were statistically
significant fewer students who failed the course compared to the previous
year’s cohort, and the vast majority of students who chose to use the AI
tutors reported that they felt they helped their learning. But there are also
some warning signs, specifically: there was a very small, but marginally
significant decrease in the percentage of A-grades. As this is just one
deployment, and not a controlled or systematic experiment, we cannot assert
causation, or even correlation, between the use of AI tutors and the changes
in grades. Like many recent uses of AI tutors, ours is exploratory in nature;
its value is to help pave the way for future research.

We believe that we are entering a new world in Education. Only 21% of students
in this cohort reported not using AI tools at all for homework help, and 56%
of them reported using several AI tools. On the one hand, the use of
unconstrained AI tools, such as ChatGPT and Copilot, threatens to disrupt the
human learning process, as they can simply give out answers to homework
questions, bypassing the student’s productive struggles. On the other hand, it
is now possible to develop educational tools that help the students in their
learning process. Knowing where to draw the lines is an emerging art.

##  2\. Background

###  2.1. LLMs in Education

Large Language Models (LLMs) refer to extensive deep learning models
constructed on the foundation of the transformer architecture (Vaswani et al.,
2017) and pre-trained with a substantial volume of data. LLMs have proven
successful in solving a variety of Natural Language Processing (NLP) tasks
(Madaan et al., 2022; Steiss et al., 2024; MacNeil et al., 2023; Leinonen et
al., 2023a) and passing knowledge examinations crossing various fields (Katz
et al., 2024; Wang et al., 2023; Biswas, 2023). Given the versatility of LLMs
in solving problems across different domains, there has been a growing
interest in using LLMs in education based on their potential to improve
learning experiences, enable personalized education, and help with creating
and grading assignments. (Kasneci et al., 2023a).

####  2.1.1. Impacts of Using LLMs in Educational Settings

Several studies have explored the opportunities and challenges associated with
integrating AI tools into education (Warschauer et al., 2023; Kasneci et al.,
2023b; Becker et al., 2023; Weber et al., 2024). Jacob et al. (Jacob et al.,
2023) investigated the experience of a non-native English speaker using
ChatGPT to improve their English writing skills. The authors found that such
tools can positively impact second-language learners by facilitating writing
practice, rather than hindering their development.

While AI tools powered by LLMs offer potential benefits for student learning,
concerns regarding their inclusion, usability, technical limitations, ethical
implications, and user well-being have also been raised (Durall and Kapros,
2020). A study (Tlili et al., 2023) revealed that ChatGPT could exhibit
harmful behaviors like dishonesty, manipulation, and misinformation
dissemination. Thus, the development of responsible and accountable AI tools
for real-world applications is critical (Kuhail et al., 2023; Arrieta et al.,
2020).

####  2.1.2. LLMs in Computer Science Education

Research on the application of LLMs in computer science education is still in
its early stages. Studies have primarily focused on exploring their
capabilities (Kazemitabaar et al., 2024), particularly in programming task
solving (Denny et al., 2023; Finnie-Ansley et al., 2023) and educational
content generation (Leinonen et al., 2023b).

Three recent studies have implemented LLM-powered AI tools for programming
classes; all of these works are similar to ours, although with significant
differences.

Liffiton et al. (Liffiton et al., 2023) developed CodeHelp for entry-level
programming classes. CodeHelp presents the students a fixed number of forms
with which they can get help about specific blocks of code – explanations,
error messages, etc. It was designed with a number of guardrails that ensure
that the complete solution to programming problems is not presented to the
students, and it only responds to content related to programming. Also, code
blocks are removed from the responses. In order to implement these guardrails,
CodeHelp makes several requests to the LLM, with several prompts.The results
indicate that students found CodeHelp to be a valuable complement to direct
support from instructors and teaching assistants.

Kazemitabaar et al. (Kazemitabaar et al., 2024) developed CodeAid, a system
that leverages few-shot learning to assist students with programming
assignments and enhance their conceptual understanding. CodeAid is very
similar to CodeHelp, in that it presents the students with a fixed set of help
actions. Their prompts include examples (few-shot learning) for improved
responses. Their results indicate that the tool provided correct and helpful
answers in 9̃0% of the time.

Liu et al. (Liu et al., 2024) introduced a suite of AI-powered educational
tools within a web application named CS50.ai, designed to enhance learning
experiences in an introductory programming course. These tools provide
functionalities including code explanation, style checking, and a Q&A bot
integrated with Ed Discussion 111https://edstem.org/). The Q&A bot employs the
RAG technique, achieving an accuracy of 88% in responding to curriculum-
related queries and 77% in administrative queries.

Our AI tutors differ from both CodeHelp and CodeAid in a couple of important
aspects. First, our AI tutors are conversational agents, not single question
help tools. We wanted to provide free-text conversational AI tutors whose user
interface is similar to a human Teaching Assistant available over UIs such as
Slack. This is because, often, students do not know enough about their
questions to be able formulate them in the right way, or even classify them
properly; having a conversation with an AI tutor has the potential to help
students discover what their question really is, and to engage with the
student’s questions in a much deeper manner. Second, internally, our RAGMan
framework uses the RAG technique, which allows us to scope the responses to
the exact topics of the homework assignments.

As for CS50.ai, the difference is that we have implemented the RAG technique
in the conversations between students and AI tutors instead of using it to
answer administrative questions on Ed. Our study also analyzed these
interactions by evaluating the accuracy of responses generated by the AI
tutors using the RAG technique. This analysis not only sheds light on the
effectiveness of the RAG application in conversational educational settings
but also provides insights into potential areas for further enhancement of AI-
driven educational tools.

###  2.2. Retrieval-Augmented Generation (RAG)

When users want to explore detailed knowledge into current or specific topics
with an LLM, they may encounter responses that are hallucinatory and
misleading (Ji et al., 2023; Alkaissi and McFarlane, 2023; Li et al., 2023;
Lin et al., 2024). According to Li et al. (Li et al., 2023), about 19.5%
responses from ChatGPT are hallucinatory. To mitigate this, Lewis et al. have
developed RAG, which connects generative AI models to external knowledge
database (Lewis et al., 2020). RAG equips models with sources they can
reference to make their responses verifiable and more dependable.

##  3\. RAGMan and AI Tutors

RAGMan is our RAG framework for developing AI tutors, and it is fairly generic
– it can be used for education as well as other purposes. Each AI tutor built
on top of RAGMan has its own knowledge base, and behaves in ways that are
tuned for specific goals that are not necessarily all the same for all AI
tutors. Here we explain the generic framework, and how we obtain specific AI
tutors from it.

Figure 1 depicts the overall architecture. RAGMan is structured into two main
subsystems: the front-end and the back-end applications. The front-end is a
NextJS application that provides a familiar conversational user
interface222https://github.com/XXXXXX (anonymized for submission). The back-
end is a Python application that includes facilities for creating a vector
database from a set of documents, and a Web application that serves as the
mediator between the front-end application and LLM APIs. For this study we
used OpenAI.

Figure 1. RAGMan Architecture

Each AI tutor built on top of RAGMan consists of its own knowledge base – a
set of documents identified by the instructor as being important for the
purpose of the tutor. Additionally, each tutor’s prompts are highly customized
(again, by the instructor) in order to make the tutor’s responses align with
the purpose of the tutor.

The flow of information involving a student’s message is as follows. Upon
receiving the student’s message, the back-end server first retrieves related
texts from the database. It then constructs a complex prompt consisting of six
parts: (1) the role assigned to the AI tutor (in this case study, always a
Teaching Assistant), (2) the goal of the assignment, (3) a short assignment
specification, (4) the texts retrieved from the vector database, (5) the
conversation so far, ending with the student’s latest message, and (6)
instructions about how to respond. Additionally, we implemented one more
guardrail for these tutors in the form of a verification step to make sure the
final response to the student never included code.

Students access the AI tutors without a login – RAGMan does not collect any
personal information. We do, however, log all messages and responses on our
server for inspection, monitoring, and analysis purposes.

Knowledge Bases In this deployment, we provide the AI tutors with two sources
of knowledge context: (1) WP detailed descriptions and (2) anonymized student
discussion posts. Regarding the discussion posts, our University uses Ed
Discussion, a peer-to-peer forum where students can post and answer questions
related to assignments or general course policies. These interactions were
deemed a valuable resource to construct a dataset, as each year students tend
to ask similar questions and clarifications regarding the same assignments.

We collected all student interactions from the previous year’s ICS32 course on
the Ed Discussion platform. Posts relevant to WPs were then extracted (by
tag), and any personally identifiable information was removed to ensure
student privacy. This process resulted in a dataset of 195 discussion posts.
No posts are removed, so it is possible that there are wrong or poor-quality
answers in this dataset. Nevertheless, these discussions usually end up with a
good answer vetted by a member of the instructional staff, so they are
valuable.

OpenAI’s text-embedding-ada-002-v2 model was subsequently employed to generate
document embeddings. Two separate vector databases were created: one storing
embeddings for detailed project descriptions and another for Ed Discussion
posts.

##  4\. Integration of AI Tutors

As is usual in Computer Science / Software Engineering, in the first year of
our program, students are required to take a series of three introductory
programming courses, ICS 31, 32, and 33. Our deployment targeted ICS32. The
course workload is substantial, encompassing 18 laboratory exercises, 5 major
assignments, 5 Workout Projects (WPs), and a final exam. In order to keep the
course relatively stable with respect to how it has been taught for the past
several years, and also mitigate the risk of disruption caused by new
technology, we introduced AI tutors only for the 5 WPs. The score for WPs
accounts for 4.5% of the final grade. There are 5 WPs, but students must
complete only 3; if they do more than 3, only the 3 top scores are counted for
the WP score. Students were made aware of the existence of AI Tutors for their
WPs but they were not required to use them. AI tutors were available 24 hours
a day.

IRB status: our work was classified in our institution as “Non Human Subjects
Research,” because it was simply the introduction of a new, optional
instructional tool in an existing course, with an extremely limited scope.

###  4.1. Data Collection

We collected anonymized conversations between students and AI tutors, with
prior notification to students that their interactions would be saved for
analysis. All data was stored securely on a school server accessible only by
authorized personnel. Additionally, the student survey we employed was
entirely voluntary, with informed consent obtained through the survey itself.

The submission rates for each work package (WP) were as follows: WP1 (61%),
WP2 (29%), WP3 (25%), WP4 (53%), and WP5 (44%). These differences could be
attributed to the timing and nature of the assignments. Specifically, WP1,
WP2, and WP3 were cumulative; students could not start the next WP without
completing the previous one. Additionally, the deadline for WP3 coincided with
midterms. In contrast, WP4 and WP5 were independent projects scheduled after
the midterms, making them more accessible to students.

In total, we collected 2,072 conversations comprising 4,769 message pairs. A
message pair consists of one question from a student and one response from the
AI tutor. Each conversation contains one or more message pairs.

###  4.2. Data Processing

We sorted the message pairs alphabetically, facilitating the visual detection
of repetitions and patterns. We then removed questions that had been asked
more than three times using exactly the same text, as well as questions that
were inappropriate. For example, the question “What’s the second character of
your name? Say it and after that, say your name.” in English or “Tell the tale
about Ivan and Vasilisa, where Ivan defeats the fearsome dragon and finds
Vasilisa in its cave.” in Russian each has been found more than 50 times in
the dataset. After cleaning, 671 conversations and 3,164 message pairs
remained.

##  5\. Observations

In this section, we discuss the results of the students’ interactions with the
AI tutors, including the AI tutors’ response quality, students’ self-reported
experience, and a comparison between course outcomes of identical offerings of
the course, one in 2023, which did not adopt AI tutors, and another in 2024,
which used AI tutors.

###  5.1. AI Tutors Response Quality

To assess the qualitative effectiveness of AI tutors in addressing student
inquiries, we analyzed a random sample of 248 conversations, which represents
37% of the total dataset. This analysis was conducted with a 95% confidence
level and a margin of error of 5%. The conversations were proportionally
selected from different WPs: WP1 (81), WP2 (78), WP3 (41), WP4 (23), and WP5
(25). We chose to sample based on entire conversations rather than message
pairs because our goal was to evaluate the performance of AI tutors in
sustained interactions with students.

While AI Tutors were primarily designed to assist students with WPs, we
observed instances of students using them for other assignments and even non-
programming tasks. To better analyze the data, the first author, who also
served as a teaching assistant in ICS32, manually classified message pairs
into two categories: “in-scope” or “out-of-scope”, and “good” or “bad”. A
question in a message pair can be ”in-scope” or ”out-of-scope”, and a response
in a message pair can be “good” or “bad” quality.

Questions directly related to WPs, general programming knowledge, or greetings
were deemed “in-scope.” Examples of “in-scope” questions are provided in
Figure 2. Conversely, questions pertaining to specific unrelated assignments
or non-programming topics were classified as “out-of-scope.” Examples of “out-
of-scope” questions can be found in Figure 3.

sample 12: What does tell() do in workout project 1
sample 30: How to use sys.argv
sample 221: hi can you help me on workout project 4?

Figure 2. In-scope Questions

sample 116: Act like an SEO specialist and create a comprehensive list of
keywords…
sample 211: for a3, where should the token be stored? sample 254: Can you
please explain in-depth the instructions for assignment 4 part 1? Here are the
instructions …

Figure 3. Out-of-scope Questions

A good quality response from AI tutors should exhibit three key
characteristics, as exemplified in Figure 4. First, the response must
accurately and effectively address the query. Second, honesty is crucial: if
the AI tutor lacks sufficient context relevant to the specific assignment, it
should acknowledge this limitation. Finally, the AI tutor should be equipped
to identify and decline to answer queries that are either inappropriate or
unrelated to the domain of programming. Responses that fail to meet any of
these three criteria are considered ineffective.

The analysis indicated that 74% of student queries fell within the scope of AI
tutors’ capabilities, while conversely, 26% of queries were classified as out
of scope. Moreover, 94% of responses were considered “good.” Responses to
questions within the scope were more often “good” (98%) compared to responses
to out-of-scope messages (81%). For example, in Figure 4, a student inquired
about the functionality of the tell() function in WP1. The AI tutor
effectively addressed this query by first explaining the purpose of tell() and
the data it returns. Subsequently, the tutor provided a relevant example and
establishes a connection between the method and the student’s current project.
Finally, the AI tutor concluded by posing a question to the student.

Student: What does tell() do in workout project 1
TEACH: In Workout Project 1, the ‘tell()‘ method is used to find the current
position of the file pointer within the file you are working with. When you
open a file and read from it or write to it, the file pointer keeps track of
where you are in the file. The ‘tell()‘ method returns the current position as
an integer, which represents the number of bytes from the beginning of the
file. For example, if you are implementing the reverse cipher, you might want
to know the size of the file to determine how to reverse its contents. You
could use ‘tell()‘ after moving the file pointer to the end of the file to get
the total number of bytes in the file. This information can be used to decide
how to read the file in reverse order. Can you think of a scenario within your
project where knowing the current position of the file pointer might be
useful?

Figure 4. In Scope Question, Good Response (sample: 12)

Although rare, AI tutors sometimes fail to provide accurate responses, even
for inquiries within their intended scope. A deeper investigation revealed
instances where AI tutors offered potentially misleading advice. As
exemplified in Figure 5, a student inquired about retrieving the length of
text within a file for WP3. The AI tutor recommended using the len() function.
However, the len() function is not the most suitable choice for this specific
scenario. Firstly, employing len() necessitates loading the entire file
content into memory at once. This approach could be problematic for
exceptionally large files, potentially causing memory exhaustion. Secondly, a
core objective of this project was to facilitate student comprehension of
pointer concepts. Consequently, suggesting the use of seek() and tell()
functions to determine file length would be more aligned with this learning
objective.

Student: how do i get the length of text in the file for qp3
TEACH: … Here are the steps to get the length of the text in a file: 1\. Open
the file in read mode. 2\. Read the contents of the file into a string. 3\.
Use the ‘len()‘ function to get the length of the string, which represents the
length of the text in the file…

Figure 5. In Scope Question, Bad Response (sample: 187)

Interestingly, while 26% of questions fell outside the scope of AI tutors’
capabilities, 81% of responses were still rated as “good.” Two primary reasons
emerged for AI tutors’ effectiveness in handling out-of-scope questions.
First, when students sought assistance with assignments beyond WPs, AI tutors
typically guided them towards teaching assistants or requested additional
context specific to the assignment. AI tutors then redirected the conversation
back to WPs and inquired about further assistance in that area. Second, in
instances where students provided sufficient context for out-of-scope
questions, AI tutors leveraged this information to offer relevant support.

Student: No i need help for lab 4
TEACH: I apologize for any confusion, but as a teaching assistant for the
Python programming class, my guidance is specifically tailored for the Workout
Project 1 (WP1) related to implementing a reverse cipher. If you’re looking
for assistance with Lab 4, I recommend reaching out to the appropriate
teaching assistant or instructor for that particular lab assignment. For any
issues or questions related to the reverse cipher project (WP1), I’m here to
help. If you have questions about how to use ‘seek()‘, ‘tell()‘, or ‘read(1)‘
functions in Python, or how to structure your code to read from and write to
files as described in WP1, feel free to ask, and I’ll be glad to guide you
through it.

Figure 6. Out of Scope Question, Good Response (sample: 5)

###  5.2. Students’ Feedback

A survey was distributed one week before the end of instruction, with 88%
response rate. 79% of students who responded the survey indicated that they
have used at least one AI tool in ICS32, with AI tutors being the most popular
choice for these students (58%). 21% of them relied solely on tools like
ChatGPT. The use of AI tools in programming classes is likely an inevitable
trend, even in the absence of a designated classroom tool. However,
unconstrained access to AI tools that provide solutions without guardrails is
likely detrimental to student learning.

Then we asked about their overall experience with the AI tutors. This included
assessments of AI tutors’ perceived helpfulness, advantages, and
disadvantages. 79% of students reported the AI tutors had a positive impact on
their learning process in ICS32. A summary of the survey results is as
follows:

  1. (1)

The majority students found that AI tutors were helpful with WPs.
Specifically, 13% of them “strongly agree,” 45% “agree,” 30% “neutral,” 9%
“disagree,” and 3% “strongly disagree.”

  2. (2)

Even though AI tutors were not designed to assist with other assignments, some
students still found they are helpful with 5% “strongly agree,” 18% agree,”
45% “neutral,” 21% “disagree,” and 11% “strongly disagree.”

Most students appreciated AI tutors’ ability to clearly explain WPs and
recommend effective methods. Some valued the step-by-step guidance for
problem-solving and its contribution to their understanding of programming
concepts. Notably, a significant advantage of AI tutors was that students did
not not have to worry about being judged because of the questions they asked.

On the down side, most students expressed dissatisfaction with AI tutors’
slowness. The implementation we had at the time was, indeed, very slow, and
this affected the user experience. We have since fixed this issue, but not in
time for ICS32. Another concern raised by students was the occurrence of
contradictory responses. An example of this is illustrated in Figure 5, where
AI tutor contradicted itself. Some students felt that AI tutors lacked
detailed explanations. This could be attributed to the limitation of not
incorporating context from past homework assignments.

###  5.3. Impact on grades

Note: the results presented here stand on a highly complex, multi-variate
context. Nevertheless, since we had access to the grades of the previous
year’s course, we make a simple comparative analysis of the grades, to try to
identify any statistically significant effects of the AI tutors.

We compared the grade distributions of the cohorts from the Winter 2023
(without AI tutors) and Winter 2024 (with AI tutors) offerings of the class by
the same instructor. In this analysis, we compared the distribution of the
grades of each WP between the two cohorts, the distribution of the combined WP
grade, the grades of the final exam, and the distribution of the final course
letter grades.

To compare the continuous distribution of numerical grades of the WPs, we used
classical two-sample Kolmogorov-Smirnov tests (KS; Kolmogorov, 1933; Smirnov,
1939). We also used Wilcoxon–Mann–Whitney and Anderson-Darling tests,
resulting in similar conclusions. The two-sample KS tests indicated that the
grade distributions showed no statistically significant difference between the
cohorts in WP1, WP2, and WP5, with resulting p-values of $0.132,0.248,0.805$.
The other two WPs, WP3 and WP4, showed statistically significant differences,
with p-values of $7.64\times 10^{-14}$ and $0.011$, respectively. In WP3, the
2024 cohort showed a worse outcome, while in WP4, the 2024 cohort showed a
better outcome. As mentioned before, there was a modification in the deadline
for WP3 in 2024, which was increased by a week and made it overlap with
midterms. This WP had the lowest submission rate among all assignments.

In this course, the student grades of the individual WPs are combined in a
single WP grade, accounting for $4.5$% of the student’s final grade. To
calculate this WP grade, the two lowest-graded WPs of each student are
dropped, and the grades of the three highest-graded WPs are added together and
then rescaled to a maximum of $4.5$ points. The comparison of this combined WP
grade distribution between the two cohorts shows a statistically significant
difference with a p-value of $0.012$ and with the 2024 cohort displaying a
mode of 0.2% points higher than the 2023 cohort. The distribution of the
numerical grades of the final exam showed no statistically significant
difference between the two cohorts, with a p-value of $1.000$, indicating that
the impact of AI tutors in the student grades was reflected only in the WPs.

Grade | 2023 cohort | 2024 cohort
---|---|---
|  | (using AI tutors)
| Number of | Percent | Number of | Percent
| students | of class | students | of class
A | $158$ |  $44$% | $130$ |  $42$%
B | $97$ |  $27$% | $101$ |  $32$%
C | $41$ |  $11$% | $37$ |  $12$%
D | $13$ |  $4$% | $9$ |  $3$%
F | $51$ |  $14$% | $33$ |  $11$%

Table 1. Comparison of the final letter grade distribution between the 2023
and 2024 cohorts. AI Ttutors were only used by the 2024 cohort.

Table 1 shows the comparison of the final course letter grades between the two
cohorts. To test if the observed difference in the discrete letter grades is
significant, we perform a Monte-Carlo test by resampling 1000 distributions
from the normalized observed grade distribution of each of the two cohorts
(e.g. Hope, 1968), with each distribution being generated using the number of
students in the largest cohort (359). Then, we calculate Kolmogorov-Smirnov
tests for each pair of the resampled distribution between the cohorts.
Afterward, we calculate the mode of the distribution of the test results using
the Venter estimator (Venter, 1967). This test results in a mode of the
p-values of $0.414$, apparently indicating that the difference in the final
course letter grades between the two cohorts is not significant. However, the
higher grades (A) dominate the present letter grade distribution, and it is
well known in the literature that Kolmogorov-Smirnov tests are weak in
distinguishing effects on tails of the tested distributions (Mason and
Schuenemeyer, 1983). Accordingly, we applied the test to grades B and lower,
resulting in p-values with a mode of $0.027$ that indicates a statistically
significant difference at the tails of the grade distribution. These results
show that although a small difference was observed in the highest grades of
the course between the two cohorts, it is statistically insignificant. They
also indicate that a statistically significant difference between the two
cohorts is observed at the lower tail of the grade distribution: in the cohort
using AI tutors, the fraction of students receiving good grades (i.e. B)
increased at a high statistical significance, while the fraction of students
passing the course also improved (i.e. we observe a statistically significant
decrease in the F grades). The observed improvement in the number of students
passing the course can be partially explained by a possible impact of the AI
tutors in the WPs because even if the difference between the combined WP
grades is small (0.2 points, or $4.5$% higher in the 2024 cohort), that could
be enough to make some students move a final letter grade up.

We note, once again, that this data is not enough to draw any firm causation
effect between using AI tutors technologies and student grades. We are only
analyzing a single course design and only two individual points in time.
Moreover, our experiment is not entirely randomized nor performed under the
exact same conditions, as the cohorts of different years are formed by
students who may have been educated differently prior to the ICS32 course.
Thus, multiple hidden factors may have impacted the grade distributions.
Nevertheless, even if further studies are necessary for any causal inference
analysis, the current data can suggest that AI tutors could have an impact on
student grades that can be uneven among high-achieving students and low-
achieving students.

##  6\. Conclusion

This paper introduces RAGMan, an LLM-powered tutoring system designed to
support entry-level programming students with homework assignments. AI tutors
build on top of RAGMan to offer students a conversational interface to help
them with problem-solving without directly providing solutions.

We conducted a manual analysis to evaluate the quality of AI tutors’
responses. The results show that 93.5% were categorized as good, meaning they
correctly and effectively addressed student inquiries. We also distributed a
survey to assess student experiences using AI tutors. Over 78% of students who
have used AI tutors reported that the tutors facilitated their learning in
class, which indicates that regardless of any real impact on the course
outcomes, students felt that they are learning more when they are assisted by
such technology. The survey results also show that students appreciated the AI
tutors’ ability to provide helpful study advice in a judgment-free environment
where they felt comfortable asking questions. Efforts to improve AI tutors
will focus on minimizing the generation of contradictory responses.

Finally, we compared the grade distributions of a course cohort that used AI
tutors and another that did not, offered in two different years. This
comparison shows a statistically significant increase in the number of
students approved in the course and an increase in the number of students
receiving middle grades (i.e., B). This data suggests, although it does not
demonstrate, that AI tutors can positively impact student success and provide
important help, especially to students who would be struggling in challenging
courses.

## References

  * (1)
  * Alkaissi and McFarlane (2023) Hussam Alkaissi and Samy I McFarlane. 2023.  Artificial hallucinations in ChatGPT: implications in scientific writing.  _Cureus_ 15, 2 (2023).
  * Arrieta et al. (2020) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020\.  Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  _Information fusion_ 58 (2020), 82–115.
  * Becker et al. (2023) Brett A. Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023.  Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ (Toronto, ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 500–506.  https://doi.org/10.1145/3545945.3569759
  * Biswas (2023) Som Biswas. 2023.  Role of ChatGPT in Computer Programming.: ChatGPT in Computer Programming.  _Mesopotamian Journal of Computer Science_ 2023 (2023), 8–16.
  * Denny et al. (2023) Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023.  Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_. 1136–1142.
  * Durall and Kapros (2020) Eva Durall and Evangelos Kapros. 2020.  Co-design for a competency self-assessment chatbot and survey in science education. In _Learning and Collaboration Technologies. Human and Technology Ecosystems: 7th International Conference, LCT 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings, Part II 22_. Springer, 13–24.
  * Finnie-Ansley et al. (2023) James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023.  My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. In _Proceedings of the 25th Australasian Computing Education Conference_ (Melbourne, VIC, Australia) _(ACE ’23)_. Association for Computing Machinery, New York, NY, USA, 97–104.  https://doi.org/10.1145/3576123.3576134
  * Hope (1968) Adery C. A. Hope. 1968.  A Simplified Monte Carlo Significance Test Procedure.  _Journal of the Royal Statistical Society. Series B (Methodological)_ 30, 3 (1968), 582–598.  http://www.jstor.org/stable/2984263
  * Jacob et al. (2023) Sharin Jacob, Tamara Tate, and Mark Warschauer. 2023.  Emergent AI-assisted discourse: Case study of a second language writer authoring with ChatGPT.  _arXiv preprint arXiv:2310.10903_ (2023).
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _Comput. Surveys_ 55, 12 (2023), 1–38.
  * Kasneci et al. (2023a) Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023a.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and individual differences_ 103 (2023), 102274.
  * Kasneci et al. (2023b) Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci. 2023b.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and Individual Differences_ 103 (2023), 102274.  https://doi.org/10.1016/j.lindif.2023.102274
  * Katz et al. (2024) Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024.  Gpt-4 passes the bar exam.  _Philosophical Transactions of the Royal Society A_ 382, 2270 (2024), 20230254.
  * Kazemitabaar et al. (2024) Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Z Henley, Paul Denny, Michelle Craig, and Tovi Grossman. 2024.  CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs.  _arXiv preprint arXiv:2401.11314_ (2024).
  * Kolmogorov (1933) Andrey Nikolaevich Kolmogorov. 1933.  Sulla determinazione empirica di una legge di distribuzione.  _G. Ist. Ital. Attuari_ 4 (1933), 83–91.  https://cir.nii.ac.jp/crid/1571135650766370304
  * Kuhail et al. (2023) Mohammad Amin Kuhail, Nazik Alturki, Salwa Alramlawi, and Kholood Alhejori. 2023.  Interacting with educational chatbots: A systematic review.  _Education and Information Technologies_ 28, 1 (2023), 973–1018.
  * Leinonen et al. (2023a) Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023a.  Comparing code explanations created by students and large language models. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1_. 124–130.
  * Leinonen et al. (2023b) Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A Becker. 2023b.  Using large language models to enhance programming error messages. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_. 563–569.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020\.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ 33 (2020), 9459–9474.
  * Li et al. (2023) Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023.  HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.  https://openreview.net/forum?id=bxsrykzSnq
  * Liffiton et al. (2023) Mark Liffiton, Brad E Sheese, Jaromir Savelka, and Paul Denny. 2023.  Codehelp: Using large language models with guardrails for scalable support in programming classes. In _Proceedings of the 23rd Koli Calling International Conference on Computing Education Research_. 1–11.
  * Lin et al. (2024) Jionghao Lin, Ashish Gurung, Danielle R Thomas, Eason Chen, Conrad Borchers, Shivang Gupta, Kenneth R Koedinger, et al. 2024\.  Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation.  _arXiv preprint arXiv:2402.14594_ (2024).
  * Liu et al. (2024) Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J. Malan. 2024.  Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education. In _Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1_ (¡conf-loc¿, ¡city¿Portland¡/city¿, ¡state¿OR¡/state¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) _(SIGCSE 2024)_. Association for Computing Machinery, New York, NY, USA, 750–756.  https://doi.org/10.1145/3626252.3630938
  * MacNeil et al. (2023) Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023.  Experiences from using code explanations generated by large language models in a web software development e-book. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_. 931–937.
  * Madaan et al. (2022) Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022.  Language Models of Code are Few-Shot Commonsense Learners. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 1384–1403.  https://doi.org/10.18653/v1/2022.emnlp-main.90
  * Mason and Schuenemeyer (1983) David M. Mason and John H. Schuenemeyer. 1983.  A Modified Kolmogorov-Smirnov Test Sensitive to Tail Alternatives.  _The Annals of Statistics_ 11, 3 (1983), 933–946.  http://www.jstor.org/stable/2240655
  * Smirnov (1939) Nikolai Vasilyevich Smirnov. 1939.  Estimate of deviation between empirical distribution functions in two independent samples.  _Bull. Moscow Univ._ 2 (1939), 3–16.
  * Steiss et al. (2024) Jacob Steiss, Tamara Tate, Steve Graham, Jazmin Cruz, Michael Hebert, Jiali Wang, Youngsun Moon, Waverly Tseng, Mark Warschauer, and Carol Booth Olson. 2024.  Comparing the quality of human and ChatGPT feedback of students’ writing.  _Learning and Instruction_ 91 (2024), 101894.  https://doi.org/10.1016/j.learninstruc.2024.101894
  * Tlili et al. (2023) Ahmed Tlili, Boulus Shehata, Michael Agyemang Adarkwah, Aras Bozkurt, Daniel T Hickey, Ronghuai Huang, and Brighter Agyemang. 2023.  What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education.  _Smart Learning Environments_ 10, 1 (2023), 15.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Venter (1967) J. H. Venter. 1967.  On Estimation of the Mode.  _The Annals of Mathematical Statistics_ 38, 5 (1967), 1446–1455.  http://www.jstor.org/stable/2238960
  * Wang et al. (2023) Hongyan Wang, WeiZhen Wu, Zhi Dou, Liangliang He, and Liqiang Yang. 2023.  Performance and exploration of ChatGPT in medical examination, records and education in Chinese: Pave the way for medical AI.  _International Journal of Medical Informatics_ 177 (2023), 105173.
  * Warschauer et al. (2023) Mark Warschauer, Waverly Tseng, Soobin Yim, Thomas Webster, Sharin Jacob, Qian Du, and Tamara Tate. 2023.  The affordances and contradictions of AI-generated text for second language writers.  _Available at SSRN_ (2023).
  * Weber et al. (2024) Jason Lee Weber, Barbara Martinez Neda, Kitana Carbajal Juarez, Jennifer Wong-Ma, Sergio Gago-Masague, and Hadar Ziv. 2024.  Measuring CS Student Attitudes Toward Large Language Models. In _Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2_ (Portland, OR, USA) _(SIGCSE 2024)_. Association for Computing Machinery, New York, NY, USA, 1846–1847.  https://doi.org/10.1145/3626253.3635604


# Microwave spectroscopy of Andreev states in InAs nanowire-based hybrid
junctions using a flip-chip layout

Patrick Zellekens+<EMAIL_ADDRESS>Present address: RIKEN Center for
Emergent Matter Science and Advanced Device Laboratory, 351-0198 Saitama,
Japan  Peter Grünberg Institut (PGI-9), Forschungszentrum Jülich, 52425
Jülich, Germany  JARA-Fundamentals of Future Information Technology, Jülich-
Aachen Research Alliance, Forschungszentrum Jülich and RWTH Aachen University,
Germany  Russell Deacon+<EMAIL_ADDRESS>RIKEN Center for Emergent Matter
Science and Advanced Device Laboratory, 351-0198 Saitama, Japan  Pujitha Perla
Peter Grünberg Institut (PGI-9), Forschungszentrum Jülich, 52425 Jülich,
Germany  JARA-Fundamentals of Future Information Technology, Jülich-Aachen
Research Alliance, Forschungszentrum Jülich and RWTH Aachen University,
Germany  Detlev Grützmacher  Peter Grünberg Institut (PGI-9),
Forschungszentrum Jülich, 52425 Jülich, Germany  JARA-Fundamentals of Future
Information Technology, Jülich-Aachen Research Alliance, Forschungszentrum
Jülich and RWTH Aachen University, Germany  Mihail Ion Lepsa  Peter Grünberg
Institut (PGI-9), Forschungszentrum Jülich, 52425 Jülich, Germany  JARA-
Fundamentals of Future Information Technology, Jülich-Aachen Research
Alliance, Forschungszentrum Jülich and RWTH Aachen University, Germany  Thomas
Schäpers  Peter Grünberg Institut (PGI-9), Forschungszentrum Jülich, 52425
Jülich, Germany  JARA-Fundamentals of Future Information Technology, Jülich-
Aachen Research Alliance, Forschungszentrum Jülich and RWTH Aachen University,
Germany  Koji Ishibashi  RIKEN Center for Emergent Matter Science and Advanced
Device Laboratory, 351-0198 Saitama, Japan

###### Abstract

Josephson junctions based on semiconductor nanowires are potential building
blocks for electrically tunable qubit structures, e.g. the gatemon or the
Andreev qubit. However, an actual realization requires the thorough
investigation of the intrinsic excitation spectrum. Here, we demonstrate the
fabrication of low-loss superconducting microwave circuits that combine high
quality factors with a well-controlled gate architecture by utilizing a flip-
chip approach. This platform is then used to perform single-tone and two-tone
experiments on Andreev states in in-situ grown InAs/Al core/half-shell
nanowires with shadow mask defined Josephson junctions. In gate-controlled and
flux-biased spectroscopic measurements we find clear signatures of single
quasiparticle as well as quasiparticle pair transitions between discrete
Andreev bound states mediated by photon-absorption. Our experimental findings
are supported by simulations that show that the junction resides in the
intermediate channel length regime.

++footnotetext: These authors contributed equally to this work

Over the course of the last decade, Josephson junctions based on
superconductor-insulator/normal conductor-superconductor stacks have been used
as building blocks for various kinds of quantum bit structures [1] such as the
fluxonium, the transmon, or the gatemon as its electrically-tunable
counterpart. All these structures, including the gatemon, have in common that
they rely on the collective excitation of an extended circuit structure and
only utilize macroscopic properties such as the supercurrent to manipulate and
interact with the system.

In contrast to conventional Josephson circuits there are mesosopic systems,
which have been already realized by means of different superconductor hybrid
structures, including break junctions, carbon nanotubes, and semiconductor
nanowires. Here, the characteristics of the system can be determined by a
small number of well resolved microscopic excitations that are spatially
localized within the normal conducting section, so-called Andreev bound states
[2, 3]. Transitions between a pair of such states can be utilized in two
special types of quantum bits, i.e. the Andreev level qubit [4, 5, 6, 7] and
the Andreev spin qubit [8, 9, 10, 11, 12]. In addition study of the energy-
phase dispersion of Andreev bound states can provide insight into details of
the superconducting state of the system. This is of interest as the range of
hybrid systems being studies is expanding from metal and semiconductor based
systems to new topological insulator or two-dimensional systems, partly
motivated by proposals for novel superconducting states such as proximity
induced topological superconductivity [13, 14].

Figure 1: Schematic sketch of the measurement setup. a, The nanowire Josephson
junction on the device chip is inductively coupled to the $\lambda/4$ coplanar
waveguide (CPW) resonator on the sapphire-based superconducting microwave
circuit. The gate is connected to an external bias tee which supplies the dc-
bias and the rf signal of the second tone. b, Excitation representation of
Andreev levels for a short junction with $L/\xi<1$ (dashed line) and a long
junction with $L/\xi=2.74$ (solid lines). The transparency was set to typical
state-of-the-art values for epitaxial nanowire Josephson junctions
($\tau=0.895$). Here, $L$ corresponds to the effective junction length,
$\xi=\hbar v_{\mathrm{F}}/\Delta$ is the superconducting coherence length,
$v_{\mathrm{F}}$ the Fermi velocity and $\Delta$ the superconducting gap. The
blue and red arrows indicate possible single quasiparticle (odd) and
quasiparticle pair (even) transitions, respectively.

The successful realization of any system based on Andreev bound states
requires a deep and detailed knowledge of the energy-phase dispersion of the
Andreev levels. In the past, tunnelling measurements have been employed for
the spectroscopic analysis of single Andreev bound states [15, 16]. As an
alternative technique, the Andreev bound state spectrum can be detected
through coupling of a phase biased junction to a microwave cavity, e.g. a
coplanar waveguide (CPW) resonator. This two-tone spectroscopy may provide
evidence of the properties of sought after exotic phenomenon such as
topologically protected Majorana modes [17, 18, 13]. The coupling of the
device with the cavity can be readout using the dispersive shift of the
resonance single-tone response and transitions between Andreev bound states
can be mapped using two-tone spectroscopy techniques typically employed in
circuit quantum electrodynamics [19, 6, 20].

Each new hybrid system has its own set of material processing requirements,
e.g. substrate limitations, the sensitivity to chemicals and etchants or the
thermal budget. In particular, semiconductor nanowire based devices can
require special etching steps and the deposition of high-k gate dielectrics
that are known to be counterproductive to the performance of superconducting
microwave resonators. Additionally, state-of-the-art fabrication techniques of
superconducting microwave circuits such as deep trench etching are usually
highly detrimental for semiconductor devices [21]. Here, we propose a flip-
chip method, with the aim of separating the fabrication of the Josephson
junction circuit and the microwave circuit. Regarding the latter, the key is
the reduction of parasitic two-level systems by optimizing interfaces and
surfaces [22, 23, 24, 25, 21] and limiting the effective area on the microwave
circuit that is exposed to the dielectric. In addition, one can select low-
loss substrate materials such as sapphire to improve the intrinsic quality of
the superconducting layer without the risk of a degradation or destruction of
the nanowire Josephson junction due to electrostatic discharge. Our flip-chip
approach creates a platform for the coupling of superconducting circuits
containing high Q-factor coplanar waveguide resonators to various hybrid
devices, including selectively-grown structures such as topological insulators
[26], which can then be used to perform spectroscopy of Andreev states in any
highly transparent system. In order to demonstrate the feasibility and
performance of our flip-chip approach, we investigated the coupling of the
resonator to the device chip. Furthermore, we present single-tone as well as
two-tone spectroscopic measurements of an epitaxially grown InAs/Al half shell
nanowire with a shadow mask defined Josephson junction [27, 28]. We note that
a similar flip-chip method has recently been reported for Josephson junction
detector based spectrometers [29].

Figure 2: Flip-chip layout. a, Optical micrograph of an exemplary microwave
circuit with a $650\,\mu$m-thick sapphire wafer substrate in $c$-plane
orientation. All superconducting structures are made out of 50-nm-thick
molecular beam epitaxy grown Nb, which has been patterned by means of dry
etching. For the application of the microwave signal, a single $\lambda/4$
coplanar waveguide resonator is used, which is capacitively coupled (yellow
square) to a short transmission line and an impedance-matched superconducting
bonding pad. The electromagnetic tuning of the device is realized with a
combination of an on-chip flux bias line and a large bonding pad that acts as
the mechanical and electrical connection between the gate electrode on the
device chip and the fridge wiring. b, Scanning electron microscopy image of an
in-situ formed InAs/Al nanowire Josephson junction on top of a single bottom
gate electrode. The SQUID loop itself and the electrical contacts are made out
of 80-nm-thick NbTi. c, Optical micrograph of a flipped device. The image was
taken through the sapphire substrate mounted upon a polymer stamp during
alignment. The SQUID loop is aligned such that one side is located above the
termination point of the resonator. To ensure a constant and homogeneous
reference potential, the device is connected to a large bonding pad that is
electrically coupled to the ground plane of the sapphire chip with a
conducting epoxy. Figure 3: Resonator characterization. a, Single-tone
response of a strongly overcoupled resonator similar to the microwave circuit
shown in a with a sample chip attached to it and with the nanowire Josephson
junction set into pinch-off. The blue line corresponds to the measured change
in magnitude and the red line to the unwrapped phase, respectively. The latter
has been corrected by the frequency-dependent background phase
$\Theta_{b}(f)$. b, Extracted internal quality factor of the very same
resonator as a function of the applied microwave power. Measurements are
performed at $T\sim 15\,$mK and the rf-power is indicated with the fridge-
internal attenuation accounted for. The inset shows an exemplary circle fit of
the resonator in the IQ plane at an applied microwave power of $-80\,$dBm.

## Flip-chip technique

A schematic representation of our system, consisting of the microwave circuit
on sapphire (resonator chip) and the Josephson junction device on high-
resistivity silicon (100) (device chip), is shown in Fig. 1a. Here, a rf-
SQUID, comprised of a gate-controlled nanowire Josephson junction in a
superconducting loop, is inductively coupled to a quarter wavelength
($\lambda/4$) CPW resonator. By this approach, it is possible to make use of
well-defined pre-patterned surface gates and high-k dielectrics with low
defect density on the device chip and still preserve high quality microwave
structures. The superconducting Nb resonator is connected to a directional
coupler, thus providing the pump tone as the input signal and feeding the
reflected signal back to the measurement setup. The mesoscopic nanowire
Josephson junction is based on an epitaxial Al-InAs half-shell nanowire
Josephson junction device that has been prepared by utilizing an in-situ
shadow evaporation process [27]. In Fig. 2 an optical micrograph of the
microwave circuit together with details of the alignment of the device chip on
the resonator circuit as well as a micrograph of the nanowire-based Josephson
junction are depicted. Further details about the mounting procedure and an
image of a successfully mounted chip are provided in the Supplementary
Information.

When coupled to the microwave cavity, the resonator photons drive transitions
between Andreev levels and with sufficient coupling the state of the system is
detected via the dispersive shift of the cavity resonance [19], i.e. the
single-tone response. Here, two types of excitations of the Andreev system are
possible. First, the microwave photon assisted excitation of a pair from the
superconducting condensate into the Andreev doublet with total energy
$(E_{A,1}+E_{A,2})$, where $E_{A,n}$ is the energy of Andreev state $n$.
Second, the excitation of a single quasiparticle from one Andreev level to
another [10, 20]. Both transitions are in literature typically identified by
the parity of the number of quasi-particles involved, i.e. even (pair) or odd
(single). An example Andreev bound state spectrum in the excitation
representation is shown in Fig. 1b where each line indicates a doublet of spin
degenerate states. The details of the energy dispersion depend not just on the
superconducting gap and junction length but also properties of the weak link
system such as spin-orbit interaction or confinement potential. For longer
channels with $L/\xi>1$, higher Andreev bound states are folded into the phase
space, as indicated by solid lines Fig. 1b. The dashed line in Fig. 1b
corresponds to the single channel/short junction limit, which has been
utilized already to study Rabi oscillations in both atomic break junctions
[19] and semiconductor nanowire-based junctions [6]. In the limit of a short
junction, with length $L$ much smaller than the superconducting coherence
length $\xi$, a doublet of Andreev bound states appear at energies
$E_{\mathrm{A}}=\Delta\sqrt{1-\tau\sin^{2}{(\varphi/2)}}$, where $\tau$ is the
junction transparency and $\varphi$ the superconducting phase difference [30,
31, 32].

## Determination of resonator properties

As a first step to characterize the flip-chip set-up, the properties of the
$\lambda/4$ CPW resonators are determined. Figure 3a provides an example for
the single-tone response $S_{21}$ of a highly overcoupled
($Q_{\text{c}}\approx 20000$) resonator similar to the one shown in Fig. 2
with the nanowire device gated such that all states are detuned from the
cavity and measured in a dilution refrigerator at $T\sim 15\,$mK. For an
applied microwave power of -80 dBm, a narrow dip in magnitude (solid blue
line) can be observed at resonance. In addition, the resonator exhibits a
pronounced phase shift at this frequency (dashed red line). Here, the phase
signal has been corrected for the monotonically changing, frequency-dependent
background phase $\Theta_{b}$. Figure 3b shows the power-dependent internal
quality factor $Q_{i}$ of the very same chip as in Fig. 3a. The microwave
power is estimated at the resonator chip including $62\,$dB of cold
attenuation inside the dilution fridge and external attenuation. The chip
reveals a maximum of $Q_{\text{i}}=5\times 10^{5}$ in transmission-mode
$S_{21}$. Based on an estimated single photon limit of $\sim 150\,$dBm, our
resonator reveals a large internal $Q$ factor even at low photon numbers with
the decrease in $Q_{i}$ anticipated to arise from the increasing influence of
unsaturated two-level systems in the device. The characteristic parameters of
the resonator are thereby determined by means of the circle fit method [33]
(see inset in Figure 3b and the Supplementary Information for further
details).

## Interaction of cavity and short channel junction

Before we present data of spectroscopic measurements it is useful to take a
closer look at the predicted dispersive response of the cavity to a single
Andreev bound states in the short channel case (cf. dashed line in Fig. 1b)
[4, 20, 6]. Fig. 4a shows the energies for two-particle transitions with two
very slightly different values of transparency $\tau$. Shown in Figs. 4b and c
are the corresponding predicted dispersive shifts of the cavity frequency in
the weak coupling regime. The response is modelled applying a Jaynes-Cummings
type Hamiltonian to an effective two-level system comprised of the ground
state and the two quasi-particle excited state of the system, described by
Zazunov et al. [4] for which further details can be found in the Supplementary
Information. As can be seen in Fig. 4b the cavity response is sensitive to the
detuning of the two-particle transition to the Andreev bound state from the
cavity frequency, $2hf_{\mathrm{A}}(\varphi)-hf_{0}$. When this detuning is
small but positive the cavity resonance is dispersively shifted down in
frequency. In contrast if the detuning can be shifted from positive to
negative through control of $\varphi$ the resulting dispersive shift of the
cavity shifts from downward to upward, as shown in Fig. 4c. Thus, from the
type of cavity response, information on the detuning can be gained. Note that
in the strong coupling regime, where the effective coupling $g_{\mathrm{eff}}$
is much greater than the lifetime of photons and Andreev states, such avoided
crossing regions would manifest zero-mode Rabi splittings as has been
previously reported in both atomic [19] and nanowire [6] Josephson junctions.

Figure 4: Examples of dispersive coupling of cavity to a short channel. a,
Pair transition energies ($2hf_{A}$) for the short junction limit for two
different transparencies. Magenta and dark purple traces indicate the quasi-
particle transition energy $2hf_{A}$ for transparencies of $\tau=0.987$ and
$0.988$, respectively. b, c, Cavity frequency shift $\delta f$ from the bare
cavity frequency $f_{0}$ normalized by the frequency corresponding to the gap
edge $f_{\Delta}=\Delta/h$ for both $\tau=0.987$ and $0.988$, respectively.
The effective coupling strength at $\varphi=\pi$ is
$g_{\mathrm{eff}}=0.0139f_{\Delta}$ for both transparencies. As can be seen in
b, when the bound states and cavity detuning remains positive the cavity
frequency is dispersively shifted downward. If however the detuning is shifted
through zero via the phase as in c the cavity frequency exhibits both up and
down shifts. Model parameters for b and c are:
$\omega_{\mathrm{c}}=0.2235f_{\Delta}$, $\gamma=0.0067f_{\Delta}$,
$\kappa=6.7^{-6}f_{\Delta}$, $Z=2\times 10^{-5}$. Model details can be found
in the methods section.

## Spectroscopic measurements

After discussing the properties of the $\lambda/4$ resonator coupled to the
device chip, we now proceed with spectroscopic measurements on our InAs-
nanowire/Al junction. As a first step, gate-dependent experiments were
performed at a superconducting phase difference across the junction
$\varphi=\pi$ to identify regions with transitions between Andreev levels.
Figure 5a shows a two-tone spectrum as a function of gate voltage alongside a
measurement of the phase shift in the single-tone response with only probe
tone applied at the bare cavity frequency $f_{0}$ (cf. Fig. 5b).

Figure 5: Gate-dependent Andreev bound state spectrum. a, Two-tone spectrum at
$\varphi=\pi$ in a gate voltage range from $0.18$ to $0.23\,$V and from $0.37$
to $0.48\,$V. The phase shift of the resonator $\delta\Theta$ is color coded.
The probe frequency is indicated by a dashed black line. We assign the abrupt
discontinuities in the state dispersion, e.g. around $V_{\mathrm{g}}=0.43\,$V,
to random reconfigurations of the nanowire surface potential or changes in the
effective gate charge. The vertical dashed lines indicate the gate voltages
chosen for the flux-dependent measurements discussed later. b, Corresponding
resonator phase-shift obtained by single-tone spectroscopy in the same gate
voltage range. The slight shifts compared to the measurement in a are caused
by a time-dependent gate drift.

The phase shift of the single-tone response (shown in Fig. 5b) reflects the
detuning of the Andreev transitions from the cavity frequency. It reveals
dispersive shifts of the cavity resonance due to interaction with the Andreev
level(s), which are consistent with the states observed in the two-tone
spectrum. The observed modulation of the single-tone response as a function of
the applied gate voltage is caused by changes in the transition energy between
different Andreev levels. At gate voltages for which the Andreev level
transition energy is higher than the resonator photon energy (such as
$V_{\mathrm{g}}\sim 0.22\,$V), the cavity resonance frequency is dispersively
shifted down, giving a relative increase in phase. In contrast, at gate
voltages for which the detuning between Andreev transition energy and cavity
frequency is reduced to zero, an abrupt reversal of the frequency shift is
observed, inducing strong negative phase shifts (such as at
$V_{\mathrm{g}}\sim 0.41\,$V). Around zero detuning, despite the signature of
an avoided-crossing-like transition, we observe no zero-mode or vacuum Rabi
splitting that would indicate strong coupling. The most likely reason are the
comparably short lifetimes of the Andreev states being studied, as such the
device is being probed in the weak coupling dispersive regime.

The two-tone spectrums in Fig. 5a contains richer features than suggested in
the single-tone response. For careful selection of a gate voltage such as the
region around $V_{\mathrm{g}}\sim 0.195\,$V or $0.39\,$V the two-tone spectrum
reveals transitions to multiple Andreev bound states as well as multiple-
photon processes. The multiple photon replicas of the quasiparticle pair
transitions to the Andreev states are particularly evident in the low second
tone frequencies below the cavity frequency ($<5\,$GHz).

Figure 6: Flux-dependent state dispersion. a, Two-tone Andreev bound state
spectrum at a gate voltage of $V_{\mathrm{g}}=0.251\,$V (blue dashed line in
Fig. 4 a) as a function of the magnetic field induced phase difference
$\varphi$ across the junction. The red line is a fit to the pair transition
driven by the pump tone using the short channel model with parameters
$\tau=0.9562$, $\Delta=104\,\mu$eV. The purple lines correspond to a the
corresponding two photon process caused by the absorption or emission of an
additional resonator photon. The dashed blue line indicates another two-
particle transistion feature from another set of Andreev states. Dashed green
lines highlight a set of single-quasiparticle transitions between spin
polarized Andreev levels. b, Single-tone measurement of cavity response at the
same $V_{\mathrm{g}}$ as in a. The resonance frequency is extracted and
plotted in c alongside a fit to a Jaynes-Cummings Hamiltonian (solid black
line). Fit parameters are $\gamma=1.32\,$MHz, $Z=6.318\times 10^{-7}$,
$f_{\mathrm{offset}}=-29\,$kHz, $g_{\mathrm{eff}}(\pi)=60\,$MHz,
$\kappa=0.1964\,$MHz. d, Two-tone spectrum at a gate voltage of
$V_{\mathrm{g}}=0.4374\,$V (red dashed line in Fig. 4 a). Again a prominent
two particle transition is observed and fitted with a short channel model with
$\tau=0.9981$, $\Delta=132\,\mu$eV (solid red line). Two photon transitions
(solid purple lines) which utilize both cavity and pump photons are also
observed. Additionally, a solid blue line indicates a two photon pair
transition using two pump photons. The solid green line corresponds to a
higher-order transition due to the combination of a pump-induced single
resonator photon and the relaxation of an Andreev level. e, Single-tone
measurement of cavity response at the same $V_{\mathrm{g}}$ as in d. The
resonance frequency is extracted and plotted in f alongside a fit to a Jaynes-
Cummings Hamiltonian (solid black line). Fit parameters are $\gamma=466\,$MHz,
$Z=4.65\times 10^{-7}$, $f_{\mathrm{offset}}=-8.6\,$kHz,
$g_{\mathrm{eff}}(\pi)=23\,$MHz, $\kappa=0.1964\,$MHz.

Proceeding to the lower gate voltage section in Fig. 5a one finds a rather
complex spectrum with an abundance of transitions between
$V_{\mathrm{g}}=0.19$ and $0.20$ V. In addition, on its right side one
observes a single spectral trace which is attributed to a pair transition.
Once again, the coupling of the Andreev bound states in the junction to the
resonator is reflected in the single-tone measurement shown in Fig. 5b. A
strong modulation of the phase shift is observed at a gate voltage range from
$V_{\mathrm{g}}=0.19$ to $0.20$ V, where the transitions found in Fig. 5a
cross the resonator frequency. In addition, around $V_{\mathrm{g}}=0.22$ V
where the Andreev level approaches the resonator frequency the cavity phase
shift has a smooth maximum.

One finds that at larger gate biases in the presented measurement range
($V_{\mathrm{g}}=0.42-0.47$ V) only a single transition is observed in the
pump frequency window. We attribute the single transition found at gate biases
$V_{\mathrm{g}}>0.42$ V to a quasiparticle pair transition
$2E_{A}=hf_{\mathrm{pump}}$ from the ground state [20, 34]. The modulation
with the gate voltage can be attributed to potential fluctuation in the
nanowire channel, e.g. in a chaotic open dot with low charging energy coupled
to superconductors by tunnel barriers and a diffusive junction with
transparent normal superconducting interfaces, as discussed by Houzet and
Skvortsov [35]. Indeed, in comparable nanowire-based junctions critical
current fluctuations were observed, which were correlated to corresponding
conductance fluctuations in the normal state [36, 37]. Since ultimately the
critical current is determined by the Andreev levels a variation of these
levels with gate voltage will directly result in critical current variations.
While only a single pair transition is observed in the two-tone spectrum for
$V_{\mathrm{g}}>0.42$ V we deduce that further high lying transitions must
exist at frequencies beyond $f_{\mathrm{pump}}=20\,$GHz. The spectroscopy as a
function of flux (discussed below) reveals some characteristics typical of
long Josephson junctions and relatively large frequency shifts from the bare
cavity frequency across the measurement flux range are observed.

We next discuss the phase-dependent measurements and use a combination of
single and two-tone measurements to evaluate the coupling of the Andreev
levels. In Figs. 6a and d are shown for which we select gate voltages of
$V_{\mathrm{g}}=0.251\,$V and $0.4347\,$V, respectively. The spectum at
$V_{\mathrm{g}}=0.251\,$V corresponds to a gate condition at which the pair
transition frequency remains above the cavity frequency at $\varphi=\pi$. In
contrast at $V_{\mathrm{g}}=0.4347\,$V the pair transition frequency dips
below the cavity frequency at $\varphi=\pi$.111Owing to a gate drift the gate
voltage was slightly readjusted between phase-dependent and gate-dependent
measurements. The most prominent feature in both sets of phase data is the
pair transition where $2E_{\mathrm{A}}=hf_{\mathrm{pump}}$. The pair
transition can be fitted using solutions to a transcendental (non-algebraic)
expression for the Andreev level energy in a long junction with multiple
transverse 1-dimensional normal bands, as discussed in Refs. [20, 34] (see
Supplementary Information). Features can also be well fit using the short
channel analytical solution for the Andreev state energies if the gap $\Delta$
is taken as a free parameter which is reduced below the expected bulk gap
(assumed to be $\Delta_{\mathrm{Al}}\sim 180\,\mu$eV based on previous dc
transport studies [27]). We note both fitting methods produce comparable
results in the measurement parameters space but differ at higher pump
frequencies. In Figs. 6a and d fits are performed using the short channel
expression shown as solid red lines with parameters as indicated in the
caption and results are later used to evaluate the coupling strength assuming
coupling with only a single two-level system. The reduced gap required for
fitting is a consequence of the states themselves being intermediate between
the short channel limit and a long junction (see the Supplementary Information
for further discussion). In either case we observe a high transparency
$\tau=0.995$ and $0.998$ with fitted $\Delta\sim 104$ and $132\,\mu$eV in
Figs. 6a and b respectively. We note that fitting with the long junction
expression also produces comparable high transparencies (see Supplementary
Information).

Apart from the pair transition features in Figs. 6a and d, we also find weaker
transitions. First, we identify replicas of second order transitions involving
absorption or emission of a resonator photon, i.e. at $2E_{\mathrm{A}}\pm
f_{\mathrm{res}}=hf_{\mathrm{pump}}$, respectively (solid magenta lines).
Furthermore, one finds a transition involving a pair of pump photons, i.e.
$2E_{\mathrm{A}}=2hf_{\mathrm{pump}}$ (dashed blue line). The appearance of
these replicas is likely related to relatively high power of both pump and
probe tones. Interestingly, close to $\varphi=\pi$ we also find a transition
with an inverted dispersion (solid green line). We attribute this feature to a
higher-order transition where a single resonator photon is created upon pump
irradiation with the assistance of the Andreev energy. The relatively
prominence of this feature also suggests elevated thermal occupation of the
excited states of the Andreev system.

In addition to the two-tone measurement we present the single-tone response in
Figs. 6b and e as well as the extracted positions of the resonance frequency
plotted as shift $\delta f=f_{\mathrm{res}}-f_{0}$ in Figs. 6c and f. Here,
the resonances are fitted with a simple Lorentzian lineshape with a linear
background subtraction to extract $f_{\mathrm{res}}$. Note that a negative
frequency shift seen here is consistent with a positive phase shift in data
plotted in the previous gate spectrums (cf. Fig. 5) owing to the features of
the resonator. It can be seen that the direction of the resonance frequency
shift is related to the detuning of the pair transition frequency and bare
cavity frequency, as previously discussed. Most prominent modulation of the
resonance occurs when detuning is tuned through zero, as seen in Fig. 6f,
where the resonance frequency shift changes sign (cf. Fig. 4c). We fit the
extracted frequency shifts assuming that only the single pair transition
identified in the two-tone measurement contributes to the cavity response
using the known analytical expressions for the coupling in the short channel
limit [4, 19]. This approach is arguably an oversimplification of our system
as other states are inevitably involved as will be later discussed but
nevertheless serves as a indication of the lower limit of the coupling and a
point of comparison with other studies. Fits to Fig. 6e indicate an effective
coupling strength at $\varphi=\pi$ of $g_{\mathrm{eff}}(\pi)=60\,$MHz and a
two-level system lifetime $\gamma\sim 1.3\,$MHz. Fits to Fig. 6f indicate
$g_{\mathrm{eff}}(\pi)=23\,$MHz and a two-level system lifetime $\gamma\sim
466\,$MHz. We note that fitting includes an offset to the frequency shift
independent of $\varphi$ that is $-29\,$kHz and $-8.9\,$kHz in Figs. 6e and f,
respectively. This shift is likely already caused by the coupling with
additional states that are possibly beyond the range of applied pump tone.

Close inspection of the two-tone spectrum in Fig. 6 reveals a set of lines
(dashed green) around $15\,$GHz which can be assigned to single quasiparticle
transitions [10, 20, 34]. These four-fold transitions originate from modifying
the dispersion of the Andreev bound states by Rashba spin-orbit coupling. More
precisely, the coupling of lowest Rashba-splitted confinement state in the
nanowire to next higher ones results in a difference of Fermi velocities for
different spin orientations. This in turn results in an effective spin-
splitting of the Andreev bound states. Single quasiparticle transitions
between pairs of these Andreev bound states then lead to a four-fold
transition pattern in the two-tone spectroscopy, as found in our experiment.
Single particle transitions between two subbands of the Andreev spectrum
requires a relatively long junction and is thus inconsistent with the pair
transition states previously discussed. In addition a weak pair transition
like feature with a flatter dispersion (dashed blue line) is observed, again
consistent with a junction that is effectively longer. It is found to be
challenging to produce accurate fits to the experimentally observed features
using expressions for the long junction detailed by Tosi et al. [20, 34] which
assumes two bands with different Fermi velocities due to the inflence of spin-
orbit interaction characterised by parameter $\Lambda_{j=1,2}=L\Delta/\hbar
v_{F,j}$ for band $j$. A rough qualitatively matching is obtained by assuming
that the single quasiparticle transitions are in-fact a two photon process
involving a cavity and pump photon and an example of such a spectrum is shown
in the Supplementary Information. Model parameters include $\tau=0.64$ and
$\Lambda_{1}=3.17$, $\Lambda_{2}=2.06$ with $\Delta=180\,\mu$eV. We conclude
that these feature arise from a lower transparency and relatively longer
channel for which such single particle transitions may become visible owing to
the reduced separation in energy between the Andreev subbands. We propose two
origins for the coexistence of such long and short junction behavior. It is
possible that state are related to two different transverse bands of the
nanowire with a significant difference in the Fermi velocity. Alternatively,
the states may correspond to channels at different locations on the nanowire
surface. The nanowire is half covered with the Al shell such that transport on
the nanowire surface that is on the underside (the opposite face from regions
with aluminium deposition) may have an effective longer junction length.

## Conclusion

We have demonstrated a flip-chip method for the coupling of a rf-SQUID device
with a high-$Q$ coplanar waveguide microwave cavity for spectroscopic
measurements. The separate processing of the rf-SQUID device chip and
resonator chip simplifies fabrication and relaxes requirements, i.e. for the
resonator chip the low-loss substrate material could be chosen to improve the
intrinsic quality of the superconducting resonator without taking the risk of
degradation or destruction of the Josephson junction on the device chip. Using
a demonstrator flip-chip device containing an Al-InAs half-shell nanowire
Josephson junction, we were able to show that detailed information about
Andreev bound states can be obtained with our approach using the single- and
two-tone response. The concept presented here, can also be employed for more
delicate or complex device circuits, e.g. ones containing topological
insulator films, where the thermal budget is rather limited, or circuits with
a set of gatemon or Andreev qubits.

## Methods

###  .1 Fabrication of the microwave circuit

The circuit containing the resonator is fabricated onto a $c$-plane sapphire
substrate [25]. The $\lambda/4$ resonator is designed in a coplanar waveguide
(CPW) geometry with a bare resonance frequency of $f_{0}\sim 5.4\,$GHz. In
order to avoid parasitic effects of residual resist or lift-off induced
sidewalls, all superconducting structures have been fabricated out of a
globally deposited 50 nm thick sputtered niobium film subsequently patterned
with a SF6 dry etching procedure. The chosen resonator design utilizes a
single port and is arranged with meanders to occupy half of the available
$8\times 4\,$mm2 sapphire substrate, as shown in Fig. 2a. The remaining half
of the substrate provides ample space on which to flip the target device chip
while minimizing the overlap of device chip and resonator transmission line.
In addition, space for gate connections and an additional microwave line for
input of additional signals is provided. In order to stabilize induced
vortices in an out-of-plane magnetic field, a global array of $4\times
4\,\mu$m2 wide flux traps was etched into the ground plane and the termination
point of the resonator. In Fig. 2c a detail of the target device chip aligned
to the termination point of the $\lambda/4$ coplanar waveguide resonator
prepared on the sapphire substrate is shown.

###  .2 Fabrication of the device chip and the nanowire Josephson junction

The target device chips are fabricated on a Si(100) substrate with high
resistivity ($\rho>100\,$k$\Omega$cm) to reduce substrate-induced losses.
Local bottom gates are defined using Ti/Pt ($3/10\,$nm) and are covered with a
$3\,$nm$/12\,$nm thick Al2O3 and HfO2 dielectric layer that has been globally
deposited by atomic layer deposition. Using a CHF3/O2-based dry etching
procedure, the latter is then removed from the areas that are not used for the
actual formation or control of the SQUID loop or the nanowire Josephson
junction. Single InAs/Al nanowire-based junctions are picked up and positioned
on the back-gate using a micromanipulator mounted inside a scanning electron
microscope (Hitachi FIB NB-5000). The junctions were fabricated by employing a
fully in-situ shadow evaporation scheme, as reported in Ref. [27]. The InAs
nanowire has a diameter of $90\,$nm and a length of $6\,\mu$m. The $20$-nm-
thick Al half-shells are separated by $90\,$nm. After the nanowire placement,
a superconducting loop of sputtered NbTi with a thickness of $80\,$nm is then
fabricated using electron beam lithography and DC magnetron sputtering. In
addition to the coupling loop, a ground connection pad is added to ensure a
ground reference for the applied gate voltage. A scanning electron micrograph
of the device circuit is shown in Fig. 2b.

###  .3 Mounting procedure

The mounting of the target device chip upon the resonator is achieved using a
homemade polymer stamping system. The sapphire resonator is mounted on a
section of polymer gel sheet (Hakuto Gel Sheet WF-55-X4-A) stuck to a glass
slide. Viewing through the back of the sapphire resonator the operator is able
to align the device superconducting loop with the coupling section of the
$\lambda/4$ resonator and bring the resonator and device chip into contact.
The electrical connection between the two chips has been successfully achieved
using two methods. In one a 2-$\mu$m-thick indium pad is added to the gate and
ground contact regions of the sapphire chip and the device chip is heated to
$\sim 120\,^{\circ}$C when brought into contact with the sapphire. Once the
indium is observed to be displaced by the contact of the two chips the
temperature is reduced. Alternatively, it was found that a small amount of
silver epoxy (either Creative Materials 118-15 or Epo-tek E4110-PFC) at the
gate and ground contact regions is sufficient to adhere the two chips and
provide good electrical contact. Of the two methods the indium contacts
provide a stronger bond but the epoxy method is more forgiving in the event of
a mistake in alignment, as this can be corrected before the epoxy is cured by
carefully removing the epoxy, cleaning and retrying.

###  .4 Measurement setup

Spectroscopy measurements were performed in a BlueFors LD400 dilution
refrigerator with a base temperature of $14\,$mK. The phase-bias ($\varphi$)
across the junction was imposed by threading a small perpendicular magnetic
field which is generated by a home-made superconducting coil through the
superconducting NbTi loop. The magnetic flux is directly related to the
junction phase difference through $\varphi=2\pi\Phi/\Phi_{0}$, where
$\Phi_{0}$ is the magnetic flux quantum. The bottom-gate is connected to a
bias-tee in order to apply a dc gate-bias $V_{\mathrm{g}}$ as well as to
supply a microwave signal with frequency $f_{\mathrm{pump}}$ for two-tone
spectroscopy. The device is characterized by single-tone as well as by two-
tone spectroscopy. For the former the shift of the resonator frequency
$f_{\mathrm{r}}-f_{0}$ induced by the coupling to the weak link is measured by
applying a continuous signal to the $\lambda/4$ resonator. The measured bare
resonator frequency is $f_{0}=4.911019$ GHz. Characterisation of the bare
resonator at $T\sim 250\,$mK was performed in an Oxford Heliox He3
refridgerator. Additional details of the experimental setup can be found in
the Supplementary Information.

###  .5 Two-tone spectroscopy

Two-tone spectroscopy is similar to the concept of dispersive readout in terms
of an indirect scanning of the coupled quantum system. A key difference is the
way in which the second tone is applied to the device. Instead of the
resonator, the impedance-matched gate line is used as a second and independent
microwave channel. The measurement scheme is as follows: First, a frequency-
dependent measurement of the cavity response is performed to find the
resonance frequency $f_{0}$ of the resonator. The latter is then used as the
first tone (probe signal) $f_{\mathrm{probe}}$=$f_{0}$. In order to determine
a range of parameters in which the resonator couples to one or more Andreev
bound states, the single-tone response is mapped out with respect to the
applied gate voltage and magnetic field. Based on these results, a suitable
gate voltage Vg or flux induced phase difference $\varphi$ is selected which
is then kept constant throughout the measurement. Subsequently, the second
tone (pump signal) $f_{\mathrm{pump}}$ is swept over a wide frequency range
(here $\delta hf_{\mathrm{pump}}\approx\Delta_{\text{Al}}/4$). When the pump
frequency is resonante with a transition between Andreev states the system
will become excited through the absorption of a photon. This induces a
significant alteration of the inductance of the SQUID loop and, in the
following, induces a dispersive shift of the resonance frequency which may be
detected as a change if phase $\delta\Theta$ measured at the probe frequency.
Lastly, the remaining electromagnetic variable, i.e. gate voltage
V${}_{\text{g}}$ or magnetic flux $\Phi$, is varied to change the energy of
the bound state. The advantage of this approach is a very high sensitivity and
the possibility to directly map out the energy space of the combined system
far below and above the actual cavity resonance frequency. Assuming an
undercoupled system, the only limitation is set by the bandwidth of the
experimental setup and the wiring in the dilution refrigerator.

###  .6 Fitting cavity frequency shifts

Fitting of the single-tone response of the cavity is performed using the
modeled coupling of a short channel from prior studies[4, 19] in which the
system is simplified to a Jaynes-Cummings type Hamiltonian. Applying input-
output theory we find an expression for the cavity reflection:

$A=\frac{-i\kappa}{(\omega_{\mathrm{c}}-\omega_{\mathrm{probe}})-\frac{i\kappa}{2}+g_{\mathrm{eff}}(\varphi)\chi}-1,$

where $\kappa$ is the cavity loss including internal loss and the loss through
port 1, $\kappa=\kappa_{\mathrm{int}}+\kappa_{1}$. Term $\gamma$ is the
decoherence which can include both energy relaxation, poisoning and also pure
dephasing rate. Term $\omega_{\mathrm{probe}}$ is the probe frequency applied
to the cavity and $\omega_{\mathrm{c}}$ is the cavity resonance frequency.
Parameter $\chi$ is the susceptibility given as

$\chi=\frac{g_{\mathrm{eff}}(\varphi)}{-((E_{A}/\hbar)-\omega_{\mathrm{c}})+\frac{i\gamma}{2}}.$

The effective coupling $g_{\mathrm{eff}}$ is a function of $\varphi$ and is
given as [4, 19]

$g_{\mathrm{eff}}(\varphi)=M\frac{1}{\hbar}\sqrt{\frac{\hbar\omega_{\mathrm{c}}}{2l}}\frac{\Delta}{4\Phi_{0}}\frac{E_{A}(\pi,\tau)}{E_{A}(\varphi,\tau)}\tau\sin(\varphi)\tan(\varphi/2),$

where $M$ is the mutual inductance between cavity and SQUID loop and $l$ is
the resonator length and $\Phi_{0}=\hbar/2e$. For fitting and simulation the
expression is simplified to:

$g_{\mathrm{eff}}(\varphi)=\sqrt{Z}\frac{\Delta}{2}\frac{E_{A}(\pi,\tau)}{E_{A}(\varphi,\tau)}\tau\sin(\varphi)\tan(\varphi/2),$

where $Z=(M^{2}/\hbar^{2})(\hbar\omega_{\mathrm{c}}/2L)(1/2\Phi_{0})$. Further
details can be found in the Supplementary Information.

## References

  * Clarke and Wilhelm [2008] J. Clarke and F. K. Wilhelm, Nature 453, 1031 (2008).
  * Kulik [1969] I. O. Kulik, Zh. Eksp. Teor. Fiz. [Sov. Phys. JETP 30, 944–950 (1970)] 57, 1745 (1969).
  * Beenakker [1992] C. Beenakker, in  _Transport phenomena in mesoscopic systems_ (Springer, 1992).
  * Zazunov  _et al._ [2003] A. Zazunov, V. S., Shumeiko, E. N. Bratus, J. Lantz, and G. Wendin, Phys. Rev. Lett. 90, 087003 (2003).
  * Zazunov  _et al._ [2005] A. Zazunov, V. S. Shumeiko, G. Wendin, and E. N. Bratus’, Phys. Rev. B 71, 214505 (2005).
  * Hays  _et al._ [2018] M. Hays, G. de Lange, K. Serniak, D. J. van Woerkom, D. Bouman, P. Krogstrup, J. Nygård, A. Geresdi, and M. H. Devoret, Phys. Rev. Lett. 121, 047001 (2018).
  * Cerrillo  _et al._ [2021] J. Cerrillo, M. Hays, V. Fatemi, and A. L. Yeyati, Phys. Rev. Research 3, L022012 (2021).
  * Chtchelkatchev and Nazarov [2003] N. M. Chtchelkatchev and Y. V. Nazarov, Phys. Rev. Lett. 90, 226806 (2003).
  * Padurariu and Nazarov [2010] C. Padurariu and Y. V. Nazarov, Phys. Rev. B 81, 144519 (2010).
  * Park and Yeyati [2017] S. Park and A. L. Yeyati, Phys. Rev. B 96, 125416 (2017).
  * Hays  _et al._ [2020] M. Hays, V. Fatemi, K. Serniak, D. Bouman, S. Diamond, G. de Lange, Krogstrup, N. P., J. Geresdi, and M. H. Devoret, Nature Physics 16, 1103 (2020).
  * Hays  _et al._ [2021] M. Hays, V. Fatemi, D. Bouman, J. Cerrillo, S. Diamond, K. Serniak, T. Connolly, P. Krogstrup, J. Nygård, A. L. Yeyati, A. Geresdi, and M. H. Devoret, Science 373, 430 (2021).
  * Prada  _et al._ [2020] E. Prada, P. San-Jose, M. W. de Moor, A. Geresdi, E. J. Lee, J. Klinovaja, D. Loss, J. Nygård, R. Aguado, and L. P. Kouwenhoven, Nature Reviews Physics 2, 575 (2020).
  * Hasan and Kane [2010] M. Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 (2010).
  * Deacon  _et al._ [2010] R. S. Deacon, Y. Tanaka, A. Oiwa, R. Sakano, K. Yoshida, K. Shibata, K. Hirakawa, and S. Tarucha, Phys. Rev. Lett. 104, 076805 (2010).
  * Pillet  _et al._ [2010] J.-D. Pillet, C. H. L. Quay, P. Morfin, C. Bena, A. L. Yeyati, and P. Joyez, Nature Physics 6, 965–969 (2010).
  * Peng  _et al._ [2016] Y. Peng, F. Pientka, E. Berg, Y. Oreg, and F. von Oppen, Phys. Rev. B 94, 085409 (2016).
  * van Heck  _et al._ [2017] B. van Heck, J. I. Väyrynen, and L. I. Glazman, Phys. Rev. B 96, 075404 (2017).
  * Janvier  _et al._ [2015] C. Janvier, L. Tosi, L. Bretheau, Ç. Girit, M. Stern, P. Bertet, P. Joyez, D. Vion, D. Esteve, M. Goffman, _et al._ , Science 349, 1199 (2015).
  * Tosi  _et al._ [2019] L. Tosi, C. Metzger, M. F. Goffman, C. Urbina, H. Pothier, S. Park, A. L. Yeyati, J. Nygård, and P. Krogstrup, Phys. Rev. X 9, 011010 (2019).
  * Bruno  _et al._ [2015] A. Bruno, G. de Lange, S. Asaad, K. L. van der Enden, N. K. Langford, and L. DiCarlo, Applied Physics Letters 106, 182601 (2015), http://dx.doi.org/10.1063/1.4919761 .
  * Gao  _et al._ [2007] J. Gao, J. Zmuidzinas, B. A. Mazin, H. G. LeDuc, and P. K. Day, Applied Physics Letters 90, 102507 (2007).
  * Göppl  _et al._ [2008] M. Göppl, A. Fragner, M. Baur, R. Bianchetti, S. Filipp, J. M. Fink, P. J. Leek, G. Puebla, L. Steffen, and A. Wallraff, Journal of Applied Physics 104, 113904 (2008).
  * Barends  _et al._ [2009] R. Barends, H. Hortensius, T. Zijlstra, J. J. Baselmans, S. Yates, J. Gao, and T. M. Klapwijk, IEEE transactions on applied superconductivity 19, 936 (2009).
  * Megrant  _et al._ [2012] A. Megrant, C. Neill, R. Barends, B. Chiaro, Y. Chen, L. Feigl, J. Kelly, E. Lucero, M. Mariantoni, P. J. O’Malley, _et al._ , Applied Physics Letters 100, 113510 (2012).
  * Schüffelgen  _et al._ [2019] P. Schüffelgen, D. Rosenbach, C. Li, T. W. Schmitt, M. Schleenvoigt, A. R. Jalil, S. Schmitt, J. Kölzer, M. Wang, B. Bennemann, U. Parlak, L. Kibkalo, S. Trellenkamp, T. Grap, D. Meertens, M. Luysberg, G. Mussler, E. Berenschot, N. Tas, A. A. Golubov, A. Brinkman, T. Schäpers, and D. Grützmacher, Nature Nanotechnology 14, 825 (2019).
  * Zellekens  _et al._ [2020] P. Zellekens, R. Deacon, P. Perla, H. A. Fonseka, T. Mörstedt, S. A. Hindmarsh, B. Bennemann, F. Lentz, M. I. Lepsa, A. M. Sanchez, D. Grützmacher, K. Ishibashi, and T. Schäpers, Phys. Rev. Applied 14, 054019 (2020).
  * Perla  _et al._ [2021] P. Perla, H. A. Fonseka, P. Zellekens, R. Deacon, Y. Han, J. Kölzer, T. Mörstedt, B. Bennemann, A. Espiari, K. Ishibashi, D. Grützmacher, A. M. Sanchez, M. I. Lepsa, and T. Schäpers, Nanoscale Adv. 3, 1413 (2021).
  * Griesmar  _et al._ [2021] J. Griesmar, R. H. Rodriguez, V. Benzoni, J.-D. Pillet, J.-L. Smirr, F. Lafont, and Ç.Ö. Girit, arXiv:2106.02632 (2021).
  * Beenakker [1991] C. W. J. Beenakker, Phys. Rev. Lett. 67, 3836 (1991).
  * Furusaki and Tsukada [1991] A. Furusaki and M. Tsukada, Phys. Rev. B 43, 10164 (1991).
  * Furusaki [1999] A. Furusaki, Superlattices and Microstructures 25, 809 (1999).
  * Probst  _et al._ [2015] S. Probst, F. B. Song, P. A. Bushev, A. V. Ustinov, and M. Weides, Review of Scientific Instruments 86, 024706 (2015).
  * Metzger  _et al._ [2021] C. Metzger, S. Park, L. Tosi, C. Janvier, A. A. Reynoso, M. Goffman, C. Urbina, A. L. Yeyati, and H. Pothier, Physical Review Research 3, 013036 (2021).
  * Houzet and Skvortsov [2008] M. Houzet and M. A. Skvortsov, Phys. Rev. B 77, 024525 (2008).
  * Doh  _et al._ [2005] Y.-J. Doh, J. A. van Dam, A. L. Roest, E. P. A. M. Bakkers, L. P. Kouwenhoven, and S. D. Franceschi, Science 309, 272 (2005).
  * Günel  _et al._ [2012] H. Y. Günel, I. E. Batov, H. Hardtdegen, K. Sladek, A. Winden, K. Weis, G. Panaitov, D. Grützmacher, and T. Schäpers, Journal of Applied Physics 112, 034316 (2012).

## Acknowledgements

The authors thank Yasuhiro Nakamura for fruitful discussions. Support from Dr.
Florian Lentz, Dr. Stefan Trellenkamp, Matthias Geitner and Rainer Benczek
with the required e-beam lithography and deposition of the superconducting
material is gratefully acknowledged. Most of the fabrication has been
performed in the Helmholtz Nano Facility. This work was partly funded by
Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under
Germany’s Excellence Strategy—Cluster of Excellence Matter and Light for
Quantum Computing (ML4Q) EXC 2004/1—390534769. This work was supported by JSPS
Grant-in-Aid for Scientific Research (B) (No. 19H02548), Grants-in-Aid for
Scientific Research (S) (No. 19H05610) and Grant-in-Aid for Scientific
Research (A) (No. 19H00867).

## Author contribution

P. P., B. B. and M. I. L. performed the nanowire growth by MBE and the
deposition of the superconducting shell. P. Z. and R. D. fabricated the
samples for transport experiments and carried out the low-temperature
experiments and the subsequent data analysis. P. Z. R. D. and T. S. wrote the
manuscript. All authors contributed to the discussions.