# A three-dimensional Josephson parametric amplifier

Imran Mahboob<EMAIL_ADDRESS>NTT Basic Research Laboratories,
NTT Corporation, Atsugi-shi, Kanagawa 243-0198, Japan  Hiraku Toida  NTT Basic
Research Laboratories, NTT Corporation, Atsugi-shi, Kanagawa 243-0198, Japan
Kousuke Kakuyanagi  NTT Basic Research Laboratories, NTT Corporation, Atsugi-
shi, Kanagawa 243-0198, Japan  Yasunobu Nakamura  Research Center for Advanced
Science and Technology, The University of Tokyo, Meguro-ku, Tokyo 153-8904,
Japan  Center for Emergent Matter Science, RIKEN, Wako-shi, Saitama 351-0198,
Japan  Shiro Saito  NTT Basic Research Laboratories, NTT Corporation, Atsugi-
shi, Kanagawa 243-0198, Japan

###### Abstract

A Josephson parametric amplifier (JPA) is executed in a three-dimensional (3D)
microwave cavity by coupling it to a superconducting quantum interference
device (SQUID) that is embedded in a two-dimensional resonator. The JPA is
activated in a three-wave mixing configuration by injecting ac magnetic flux,
at twice the 3D cavity frequency, into the SQUID. An 8.3 GHz cavity is
measured in a non-degenerate phase-insensitive configuration which yields
gains in excess of 40 dB, where a 20 dB gain results in an operational
bandwidth of 0.4 MHz, a 1 dB compression point of $-115$ dBm with half a
quantum of added noise.

The Josephson parametric amplifier, a superconducting resonator shunted with
nonlinear inductance from Josephson junctions (JJs), was initially pioneered
in the late 70s and early 80s to yield signal gain with low-noise and to study
quantum optics with microwave flavoured photons Wahlsten and Claeson (1978);
Yurke  _et al._ (1988). The JPA re-emerged in the late 2000s and it has become
an indispensable component for quantum limited measurements at microwave
frequencies Caves (1982); Castellanos-Beltran and Lehnert (2007); Yamamoto
_et al._ (2008); Bergeal  _et al._ (2010). Indeed, JPAs have facilitated
quantum non-demolition measurements of superconducting qubits Vijay, Slichter,
and Siddiqi (2011); Lin  _et al._ (2013), to monitor their relaxation and
corresponding trajectories Lin  _et al._ (2013), to even apply feedback to
this process Vijay  _et al._ (2012); Risté  _et al._ (2013) as well as being
instrumental in the precision measurements of mechanical vibrations,
semiconductor quantum dots and surface acoustic waves Teufel  _et al._ (2009);
Stehlik  _et al._ (2015); Noguchi  _et al._ (2017). Moreover, the squeezed
microwave radiation generated by the JPA has not only enabled quantum optics
to be explored at ever greater extremes of non-classicality Castellanos-
Beltran  _et al._ (2008); Mallet  _et al._ (2011); Eichler  _et al._ (2011);
Flurin  _et al._ (2012); Eichler  _et al._ (2014); Wilson  _et al._ (2011) but
it has also been employed to excite quantum states of a harmonic oscillator
and to perform sub-vacuum noise measurements Kono  _et al._ (2017); Bienfait
_et al._ (2017).

Meanwhile a concerted effort has taken place to develop a multi-
superconducting-qubit architecture to enable the implementation of quantum
error correction protocols which could ultimately lead to a fault-tolerant
quantum computer Reed  _et al._ (2012); Barends  _et al._ (2014); Risté  _et
al._ (2015); Blumoff  _et al._ (2016); Takita  _et al._ (2016); Paik  _et al._
(2016); Neill  _et al._ (2017). To that end, the ability to handle the large
cumulative signal power from multiple qubits via a single JPA is essential for
the realisation of this ambition. However, the second generation of JPAs from
the late 2000s suffered from poor power handling capacity, whose gain became
easily saturated, whilst operating over relatively narrow bandwidths. These
shortcomings led to the emergence of third generation JPAs which sought to
address these objectives simultaneously by either reducing the resonator’s
quality factor $Q$ Mutus  _et al._ (2013); Roy  _et al._ (2016), diluting the
nonlinearity of the inductance with SQUID arrays Zhou  _et al._ (2014);
Eichler and Wallraff (2014) or by eliminating the resonator entirely White
_et al._ (2015); Macklin  _et al._ (2015). Most recently, kinetic inductance
parametric amplifiers have arisen which can operate without JJs and they have
achieved unprecedented performance levels Malnou  _et al._ (2021); Parker  _et
al._ (2022).

Here an alternative strategy is pursued, which is geared to simply increasing
the dynamic range of the JPA, whilst neglecting the resultant bandwidth
performance. To that end particular focus is paid to the source of gain
saturation which stems from the anharmonicity in the restoring potential of
the JPA and whose potency is a consequence of the direct incorporation of the
nonlinear inductance into the amplifying resonator Zhou  _et al._ (2014);
Manucharyan  _et al._ (2007). By separating the participation of the nonlinear
inductance in the signal amplifying resonator, by means of a dispersive
interaction, its efficacy can be diluted. On the other hand, the high $Q$ of
the 3D cavity Paik  _et al._ (2011) still enables tangible frequency
modulations from the diluted nonlinear inductance thus enabling access to
parametric amplification Castellanos-Beltran and Lehnert (2007); Yamamoto  _et
al._ (2008). For this 3D variant of a JPA, gains in excess of 40 dB are
observed, which are large amongst the most competitive parametric
amplification factors reported to date Bergeal  _et al._ (2010).

Figure 1: (a) A photograph of two 3D JPAs consisting of a pair of cavities
that are milled from a split aluminium block (left) which enables access to
the SQUID chip (top right) located in the centre where the SQUID (bottom
right) is capacitively coupled to both cavities via the large rectangular
antennas. The SQUID can be ac flux pumped and/or dc flux biased via
application of current to the coplanar waveguide located between the cavities.
TE101 modes are hosted by the cavities that are probed via the SMA connectors
to which external signals are coupled. (b) An equivalent circuit diagram of
the 3D JPA and a schematic of the measurement circuit used to probe the
cavities [HF (red) and LF (blue)]. The reflection coefficient $S_{11}$ is
measured in a vector network analyser, VNA (green). The SQUID (yellow) has a
nearby flux line (purple) to which external dc and ac current signals can be
applied through a bias-tee (magenta).

The 3D JPAs developed in this work are shown in Fig. 1(a) and they consist of
two 3D cavities that are capacitively coupled to a magnetic flux pumped SQUID
Sirois  _et al._ (2015), whose equivalent electrical circuit is shown in the
lower portion of Fig. 1(b). The smaller high-frequency (HF) cavity is detailed
here and the larger lower-frequency (LF) cavity in Supplementary Materials SM
. It should be noted that both cavities can be operated as JPAs independently
of each other. The HF cavity has a volume of $27\times 5\times 24$ mm3 and is
machined from aluminium (A1050) in a split configuration enabling access to
the SQUID chip and a coplanar waveguide chip to inject magnetic flux into the
former as shown in Fig. 1(a). A single SQUID is patterned via electron beam
lithography on a high-resistance silicon substrate and metalized with double
angle aluminium evaporation where the first aluminium layer is oxidised to
form the JJs and the resultant chip is placed in a tunnel interconnecting the
two cavities. The SQUID has a loop area of $20\times 5$ $\mu$m2 and the JJs
have a nominal area of $0.3\times 0.1$ $\mu$m2 with $I_{\rm{c}}\sim 1.6$
$\mu$A. The SQUID is galvanically connected to two $2.5\times 1.8$ mm2 antenna
pads that are located on either side of the SQUID, each via a $110\times 2500$
$\mu$m2 line, that position them in the centre of the cavities to which they
capacitively couple.

Figure 2: (a, b) The dipole-like electromagnetic field distribution of the
SQUID resonator and the TE101 mode in the HF cavity respectively with the
SQUID mimicking lumped element inductance set at 1.2 nH. (c, d) The frequency
of the SQUID resonator and the response of the HF cavity’s resonance
respectively as the SQUID mimicking lumped element inductance is varied.

In order to understand the dynamics of this 3D variant of a JPA, its resonant
electromagnetic field distributions are numerically evaluated via the RF
module in COMSOL CS . Specifically all the electromagnetic resonances were
extracted as a function of the SQUID inductance, which is modelled as a lumped
element, and the model included both cavities, the SQUID chip, the ceramic
chip incorporating the gold-plated copper waveguide and the SMA connectors.
The simulations reveal the circuit embedding the SQUID between the large
antennas exhibits a resonant electromagnetic mode with a dipole-like field
distribution in addition to the cavities sustaining resonant transverse
electromagnetic (TE) modes as shown in Figs. 2(a) and 2(b) respectively SM .
As the SQUID’s inductance is varied via the lumped element, from 0.1 nH
determined by $I_{\rm{c}}$ at zero magnetic field, the dipole-like resonance
in the SQUID resonator at 6.2 GHz is reduced by $\sim 1$ GHz as shown in Fig.
2(c), and in response, the HF cavity exhibits a red shift over 90 MHz starting
at 8.22 GHz as shown in Fig. 2(d). The apparent linear frequency shift of the
former stems from the lumped element inductance variations being much smaller
than the numerically evaluated 8.6 nH inductance of this circuit from Fast
Henry FH . The COMSOL simulation indicates a circuit quantum electrodynamics-
like dispersive interaction between the SQUID resonator and the 3D cavity with
the latter inheriting a Kerr anharmonicity in its restoring potential that is
diluted by the large frequency separation of these resonances Blais  _et al._
(2004); Wallraff  _et al._ (2004); Kirchmair  _et al._ (2013). Indeed, the
dispersive interaction scales with this frequency separation and the LF cavity
exhibits a stronger response as consequence of its spectral proximity to the
SQUID resonance SM .

Figure 3: (a) The raw reflection coefficient $|S_{11}|$ as a function of
frequency for the HF cavity acquired with a VNA using a probe power of $-110$
dBm at the cavity input. (b) The corresponding normalised complex reflection
coefficient (points) and the analytical fit (line) used to extract the
internal loss and external coupling rates. (c) $|S_{11}|$ as a function of
frequency measured via a VNA with a probe power of $-115$ dBm, whilst dc
current is applied to the flux line to modulate the SQUID inductance.

Experimentally, in order to modulate the SQUID’s inductance, a current line is
defined 1 $\mu$m away, as shown in the bottom right of Fig. 1(a), which
corresponds to a numerically evaluated mutual inductance of $\approx 5$ pH
Sirois  _et al._ (2015); FH . This line is connected to a coplanar waveguide
with 50 $\Omega$ impedance which is also patterned onto the SQUID chip via
electron-beam lithography. This in turn is wire-bonded to a second coplanar
waveguide patterned on a temperature stable ceramic (TMM10i, Rogers
Corporation), which is located in another tunnel between the cavities. Finally
this is connected to an SMP connector which enables external ac and dc
currents to be inductively coupled to the SQUID thus enabling it to be
externally flux biased and pumped at frequency $\omega_{\rm{p}}$ namely
$\Phi_{\rm{ext}}=\Phi_{\rm{dc}}+\Phi_{\rm{ac}}\cos(\omega_{\rm{p}}t)$.
Conversely, the cavities are probed via homodyne spectroscopy from SMA
connectors directly mounted on the cavities whose external coupling rates were
tuned at room temperature by simply adjusting the length of their centre pins.

The experiments were performed in a dry dilution refrigerator, shielded from
magnetic noise, with the sample mounted on the base plate at $<10$ mK and
measured using a standard microwave setup as detailed in Fig. 1(b). The HF
cavity was probed in the spectral region suggested by the COMSOL simulation in
Fig. 2(d) via a reflection measurement, using the circuit depicted in green in
Fig. 1(b), whilst the flux line was deactivated. The result, shown in Fig.
3(a), reveals a microwave resonance around the anticipated frequency. The
resonance linewidth was evaluated by fitting the corresponding normalised
complex reflection coefficient from standard microwave circuit theory, as
shown in Fig. 3(b), which revealed the HF cavity is over-coupled with an
external coupling rate of 1.1 MHz and internal loss rate 0.42 MHz Pozar
(2012).

To appraise the evolution of the cavity resonance as the capacitively coupled
SQUID resonator’s inductance is modulated, the previous measurement is
repeated but now with the application of dc current bias ($I_{\rm{dc}}$) to
the flux line for $\Phi_{\rm{dc}}$, as depicted in magenta in Fig. 1(b). The
result shown in Fig. 3(c) reveals the cavity frequency can be periodically
modulated as the magnetic flux piercing the SQUID is varied. The experimental
nonlinear and periodic magnetic flux dependence of the SQUID resonance cannot
be modelled in COMSOL and hence the profile of the HF cavity’s dispersive
response is also not accurately captured in Fig. 2(d). Nevertheless, at the
half magnetic flux quantum $\Phi_{0}/2$, the cavity frequency shift of $\sim
90$ MHz is in-line with the COMSOL simulation, indicating that the SQUID’s
inductance has a maximal variation of 1.1 nH.

Given the flux dependent frequency modulations in the 3D cavity, the
availability of parametric amplification can now be explored where in addition
to a weak signal input at frequency $\omega_{\rm{s}}$, the flux line is
simultaneously pumped to give $\Phi_{\rm{ac}}$ at frequency
$\omega_{\rm{p}}=2\omega_{\rm{s}}+\Delta$ whilst $\Phi_{\rm{dc}}$ is varied.
This results in signal amplification in the non-degenerate regime when
$\Delta\neq 0$ and the concomitant generation of an idler at
$\omega_{\rm{i}}=\omega_{\rm{s}}+\Delta$ Castellanos-Beltran and Lehnert
(2007); Bergeal  _et al._ (2010). The result of this measurement, shown in
Fig. 4(a), reveals that the signal (idler which is not shown) can only be
amplified (generated) at specific dc flux biases. This observation arises from
the absence of flux dependent frequency modulations at
$\Phi_{\rm{dc}}/\Phi_{0}\approx$ an integer. On the other hand when
$\Phi_{\rm{dc}}/\Phi_{0}\approx(2n+1)/2$, where $n=$ integer, the SQUID’s
critical current becomes so small that it can easily be quenched by the
circulating current in the SQUID induced via $\Phi_{\rm{ac}}$.

Figure 4: (a) $|S_{11}|$ as a function of frequency and dc flux bias with
signal power = $-120$ dBm, pump power = $-34$ dBm and $\Delta/2\pi=30$ Hz. (b)
The maximum signal power gains (points) extracted from the data shown in the
inset, with the cavity measured via signal power = $-130$ dBm,
$\Phi_{\rm{dc}}/\Phi_{0}=0.3$ and $\Delta/2\pi=20$ Hz. Also shown is an
analytical fit (line) detailed in the main text. (c) The non-degenerate signal
power gain measured at $\Phi_{\rm{dc}}/\Phi_{0}=0.3$ for various ac pump and
signal powers. (d) The noise power density measured in a spectrum analyser via
a -140 dBm signal tone with: (i) the pump de-activated, (ii) activated at
$-34$ dBm. The latter configuration yields $\sim 20$ dB parametric
amplification and a concomitant signal-to-noise ratio improvement. The inset
indicates in the absence of a signal tone: (iii) the cryogenic amplifier’s
noise when the 3D JPA is de-activated (iv) activating the pump at $-32.8$ dBm
results in a parametric resonance.

In order to explore optimal non-degenerate parametric amplification, a dc flux
bias is selected between these extremes [Fig. 4(a)]. In the inset to Fig.
4(b), signal amplification in the HF cavity can be observed as the ac pump
power $P$ is increased. The resultant maximum power gain is plotted in Fig.
4(b) and is limited by the activation of a parametric resonance above a
critical pump power $P_{\rm{c}}$, namely when the cavity loss is completely
neutralised by the parametric gain, as shown in the inset to Fig. 4(d)
Yamamoto  _et al._ (2008); Lin  _et al._ (2013). In the limit
$\Delta\rightarrow 0$, the signal power gain can be approximated to:
$1+(4P/P_{\rm{c}})/(1-P/P_{\rm{c}})^{2}$ which results in a fit, shown in Fig.
4(b), that can reproduce the experimental response and thus confirm the
availability of non-degenerate parametric amplification Zhou  _et al._ (2014).
Finally as the gain increases the cavity linewidth also narrows and at a
signal power gain of 20 dB, a 0.4 MHz instantaneous bandwidth can be extracted
Bergeal  _et al._ (2010).

Now to evaluate the enhancement in the dynamic range of the 3D JPA as a linear
amplifier, the signal power gain over a large range of input powers is
measured with a number of pump powers as summarised in Fig. 4(c). Non-
degenerate gains in excess of 40 dB are achieved for signal powers below
$-140$ dBm. On the other hand, the minimum gain needed to overwhelm the
following commercial cryogenic amplifier noise is typically 20 dB Bergeal  _et
al._ (2010) and this gain can be achieved with a 1 dB compression point of
approximately $-115$ dBm.

To exploit this performance, the noise added by the 3D JPA needs to be
evaluated to confirm if this amplification is available close to the quantum
limit. For this purpose the approach of signal-to-noise ratio improvement is
employed Castellanos-Beltran and Lehnert (2007); White  _et al._ (2015). To
that end, the noise power in the spectral region around the cavity resonance
is acquired via a spectrum analyser, with the 3D JPA off and on where the
latter is set to yield $\sim 20$ dB gain, as shown in Fig. 4(d). This
measurement reveals a mean noise rise of $4.8\pm 0.25$ dB with one standard
deviation uncertainty and signal gain $G_{\rm{JPA}}=17.8$ dB, which is less
than expected due to the signal tone being slightly detuned from the cavity
resonance. A $13\pm 8.5$ dB improvement in the signal-to-noise ratio is
observed when the 3D JPA is activated and this can be equated to
$1/((T_{\rm{JPA}}/T_{\rm{cryo}})+(1/G_{\rm{JPA}}))$ Mutus  _et al._ (2013);
White  _et al._ (2015). The cryogenic amplifier’s noise temperature,
$T_{\rm{cryo}}=4.4\pm 0.3$ K, is calibrated using the approach detailed in
Ref. 47 and is summarised in Supplemental Materials SM . The 3D JPA’s noise
temperature $T_{\rm{JPA}}$ is then determined to be $0.148\pm 0.098$ K and
this can be contrasted with vacuum noise $\hbar\omega/2k_{B}=0.199$ K at 8.3
GHz where $\hbar$ is the reduced Planck constant and $k_{B}$ is the Boltzmann
constant. Consequently, the 3D JPA adds only half a quantum of noise during
signal amplification, the minimum possible for a non-degenerate phase-
insensitive parametric amplifier Caves (1982); Castellanos-Beltran and Lehnert
(2007); Yamamoto  _et al._ (2008); Zhou  _et al._ (2014).

To put these performance metrics into context, the highest 1 dB compression
point of $-99$ dBm, with a power gain of $\sim 20$ dB in a few GHz bandwidth,
was observed in a travelling wave parametric amplifier Macklin  _et al._
(2015); Roy  _et al._ (2016). Although the 3D JPA can offer even greater gains
as well as a respectable 1 dB compression point, it suffers in terms of its
limited bandwidth. However, the bandwidth available to the 3D JPA can easily
be broadened by increasing the external coupling rate by simply adjusting the
length of the SMA connector. For parametric amplification to still occur, this
enhanced bandwidth needs to be narrower than the dc current induced frequency
modulations which suggests that at least 10 MHz should be possible.

On the other hand, operating the 3D JPA in a degenerate phase-sensitive
configuration should not only yield gains in excess of 40 dB Bergeal  _et al._
(2010) but it could also achieve unprecedented levels of vacuum squeezing
which could have applications for metrology and this is a natural future
direction of study Castellanos-Beltran  _et al._ (2008); Zhou  _et al._
(2014). Finally, the relative ease at which a parametric resonance can be
activated in the low-loss 3D cavity, as shown in the inset to Fig. 4(d), paves
the way towards higher-order parametric resonances which could yield non-
trivial macroscopic quantum states Svensson  _et al._ (2018).

A JPA has been executed by capacitively coupling a flux pumped SQUID resonator
to a 3D microwave cavity. The dispersive interaction between them results in
the 3D cavity’s resonance inheriting a diluted Kerr nonlinearity which enables
the resultant JPA to yield non-degenerate signal gains exceeding 40 dB. On the
other hand, a 20 dB gain can be achieved with half a quantum of added noise
and a 1 dB compression point of $-115$ dBm. The 3D JPA also lays the
foundations to study quantum optics in massive 3D microwave cavities.

The authors thank S. Kono and A. Noguchi of the University of Tokyo for their
support during the inception of this project. This work was partially
supported by CREST(JPMJCR1774) and ERATO (JPMJER1601), JST.

## References

  * Wahlsten and Claeson (1978) S. Wahlsten and S. R. T. Claeson, J. Appl. Phys. 49, 4248 (1978).
  * Yurke  _et al._ (1988) B. Yurke, P. G. Kaminsky, R. E. Miller, E. A. Whittaker, A. D. Smith, A. H. Silver, and R. W. Simon, Phys. Rev. Lett. 60, 764 (1988).
  * Caves (1982) C. M. Caves, Phys. Rev. D 26, 1817 (1982).
  * Castellanos-Beltran and Lehnert (2007) M. A. Castellanos-Beltran and K. W. Lehnert, Appl. Phys. Lett. 91, 083509 (2007).
  * Yamamoto  _et al._ (2008) T. Yamamoto, K. Inomata, M. Watanabe, K. Matsuba, T. Miyazaki, W. D. Oliver, Y. Nakamura, and J. S. Tsai, Appl. Phys. Lett. 93, 042510 (2008).
  * Bergeal  _et al._ (2010) N. Bergeal, F. Schackert, M. Metcalfe, R. Vijay, V. E. Manucharyan, L. Frunzio, D. E. Prober, R. J. Schoelkopf, S. M. Girvin, and M. H. Devoret, Nature 465, 64 (2010).
  * Vijay, Slichter, and Siddiqi (2011) R. Vijay, D. H. Slichter, and I. Siddiqi, Phys. Rev. Lett. 106, 110502 (2011).
  * Lin  _et al._ (2013) Z. R. Lin, K. Inomata, W. D. Oliver, K. Koshino, Y. Nakamura, J. S. Tsai, and T. Yamamoto, Appl. Phys. Lett. 103, 132602 (2013).
  * Vijay  _et al._ (2012) R. Vijay, C. Macklin, D. H. Slichter, S. J. Weber, K. W. Murch, R. Naik, A. N. Korotkov, and I. Siddiqi, Nature 490, 77 (2012).
  * Risté  _et al._ (2013) D. Risté, M. Dukalski, C. A. Watson, G. de Lange, M. J. Tiggelman, Y. M. Blanter, K. W. Lehnert, R. N. Schouten, and L. DiCarlo, Nature 502, 350 (2013).
  * Teufel  _et al._ (2009) J. D. Teufel, T. Donner, M. A. Castellanos-Beltran, J. W. Harlow, and K. W. Lehnert, Nat. Nano. 4, 820 (2009).
  * Stehlik  _et al._ (2015) J. Stehlik, Y. Y. Liu, C. M. Quintana, C. Eichler, T. R. Hartke, and J. R. Petta, Phys. Rev. Applied 4, 014018 (2015).
  * Noguchi  _et al._ (2017) A. Noguchi, R. Yamazaki, Y. Tabuchi, and Y. Nakamura, Phys. Rev. Lett. 119, 180505 (2017).
  * Castellanos-Beltran  _et al._ (2008) M. A. Castellanos-Beltran, K. D. Irwin, G. C. Hilton, L. R. Vale, and K. W. Lehnert, Nat. Phys. 4, 929 (2008).
  * Mallet  _et al._ (2011) F. Mallet, M. A. Castellanos-Beltran, H. S. Ku, S. Glancy, E. Knill, K. D. Irwin, G. C. Hilton, L. R. Vale, and K. W. Lehnert, Phys. Rev. Lett. 105, 220502 (2011).
  * Eichler  _et al._ (2011) C. Eichler, D. Bozyigit, C. Lang, M. Baur, L. Steffen, J. M. Fink, S. Filipp, and A. Wallraff, Phys. Rev. Lett. 107, 113601 (2011).
  * Flurin  _et al._ (2012) E. Flurin, N. Roch, F. Mallet, M. H. Devoret, and B. Huard, Phys. Rev. Lett. 109, 183901 (2012).
  * Eichler  _et al._ (2014) C. Eichler, Y. Salathe, J. Mlynek, S. Schmidt, and A. Wallraff, Phys. Rev. Lett. 113, 110502 (2014).
  * Wilson  _et al._ (2011) C. M. Wilson, G. Johansson, A. Pourkabirian, M. Simoen, J. R. Johansson, T. Duty, F. Nori, and P. Delsing, Nature 479, 376 (2011).
  * Kono  _et al._ (2017) S. Kono, Y. Masuyama, T. Ishikawa, Y. Tabuchi, R. Yamazaki, K. Usami, K. Koshino, and Y. Nakamura, Phys. Rev. Lett. 119, 023602 (2017).
  * Bienfait  _et al._ (2017) A. Bienfait, P. Campagne-Ibarcq, A. H. Kiilerich, X. Zhou, S. Probst, J. J. Pla, T. Schenkel, D. Vion, D. Esteve, J. L. L. Morton, K. Moelmer, and P. Bertet, Phys. Rev. X 7, 041011 (2017).
  * Reed  _et al._ (2012) M. D. Reed, L. DiCarlo, S. E. Nigg, L. Sun, L. Frunzio, S. M. Girvin, and R. J. Schoelkopf, Nature 482, 382 (2012).
  * Barends  _et al._ (2014) R. Barends, J. Kelly, A. Megrant, A. Veitia, D. Sank, E. Jeffrey, T. C. White, J. Mutus, A. G. Fowler, B. Campbell, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, C. Neill, P. O’Malley, P. Roushan, A. Vainsencher, J. Wenner, A. N. Korotkov, A. N. Cleland, and J. M. Martinis, Nature 508, 500 (2014).
  * Risté  _et al._ (2015) D. Risté, S. Poletto, M. Z. Huang, A. Bruno, V. Vesterinen, O. P. Saira, and L. DiCarlo, Nat. Commun. 6, 7983 (2015).
  * Blumoff  _et al._ (2016) J. Z. Blumoff, K. Chou, C. Shen, M. Reagor, C. Axline, T. Brierley, P. Silveri, C. Wang, B. Vlastakis, E. Nigg, L. Frunzio, H. Devoret, L. Jiang, M. Girvin, and J. Schoelkopf, Phys. Rev. X 6, 031041 (2016).
  * Takita  _et al._ (2016) M. Takita, A. D. Córcoles, E. Magesan, B. Abdo, M. Brink, A. Cross, J. M. Chow, and J. M. Gambetta, Phys. Rev. Lett. 117, 210505 (2016).
  * Paik  _et al._ (2016) H. Paik, A. Mezzacapo, M. Sandberg, D. T. McClure, B. Abdo, A. D. Córcoles, O. Dial, D. F. Bogorin, B. L. T. Plourde, M. Steffen, A. W. Cross, J. M. Gambetta, and J. M. Chow, Phys. Rev. Lett. 117, 250502 (2016).
  * Neill  _et al._ (2017) C. Neill, P. Roushan, K. Kechedzhi, S. Boixo, S. V. Isakov, V. Smelyanskiy, R. Barends, B. Burkett, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, A. Fowler, B. Foxen, R. Graff, E. Jeffrey, J. Kelly, E. Lucero, A. Megrant, J. Mutus, M. Neeley, C. Quintana, D. Sank, A. Vainsencher, J. Wenner, T. C. White, H. Neven, and J. M. Martinis, arXiv:1709.06678 (2017).
  * Mutus  _et al._ (2013) J. Y. Mutus, T. C. White, E. Jeffrey, D. Sank, R. Barends, J. Bochmann, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, J. Kelly, A. Megrant, C. Neill, P. J. J. O’Malley, P. Roushan, A. Vainsencher, J. Wenner, I. Siddiqi, R. Vijay, A. N. Cleland, and J. M. Martinis, Appl. Phys. Lett. 103, 122602 (2013).
  * Roy  _et al._ (2016) T. Roy, S. Kundu, M. Chand, A. M. Vadiraj, A. Ranadive, N. Nehra, M. P. Patankar, J. Aumentado, A. A. Clerk, and R. Vijay, Appl. Phys. Lett. 107, 262601 (2016).
  * Zhou  _et al._ (2014) X. Zhou, V. Schmitt, P. Bertet, D. Vion, W. Wustmann, V. Shumeiko, and D. Esteve, Phys. Rev. B 89, 214517 (2014).
  * Eichler and Wallraff (2014) C. Eichler and A. Wallraff, EPJ Quantum Technol. 1, 2 (2014).
  * White  _et al._ (2015) T. C. White, J. Y. Mutus, I. C. Hoi, R. Barends, B. Campbell, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, E. Jeffrey, J. Kelly, A. Megrant, C. Neill, P. J. J. O’Malley, P. Roushan, D. Sank, A. Vainsencher, J. Wenner, S. Chaudhuri, J. Gao, and J. M. Martinis, Appl. Phys. Lett. 106, 242601 (2015).
  * Macklin  _et al._ (2015) C. Macklin, K. O’Brien, D. Hover, M. E. Schwartz, V. Bolkhovsky, X. Zhang, W. D. Oliver, and I. Siddiqi, Science 350, 307 (2015).
  * Malnou  _et al._ (2021) M. Malnou, M. R. Vissers, J. Wheeler, J. Aumentado, J. Hubmayr, J. N. Ullom, and J. Gao, PRX Quantum 2, 010302 (2021).
  * Parker  _et al._ (2022) D. J. Parker, M. Savytskyi, W. Vine, A. Laucht, T. Duty, A. Morello, A. L. Grimsmo, and J. J. Pla, Phys. Rev. Applied 17, 034064 (2022).
  * Manucharyan  _et al._ (2007) V. E. Manucharyan, E. Boaknin, M. Metcalfe, R. Vijay, I. Siddiqi, and M. Devoret, Phys. Rev. B 76, 014524 (2007).
  * Paik  _et al._ (2011) H. Paik, D. I. Schuster, L. S. Bishop, G. Kirchmair, G. Catelani, A. P. Sears, B. R. Johnson, M. J. Reagor, L. Frunzio, L. I. Glazman, S. M. Girvin, M. H. Devoret, and R. J. Schoelkopf, Phys. Rev. Lett. 107, 240501 (2011).
  * Sirois  _et al._ (2015) A. J. Sirois, M. A. Castellanos-Beltran, M. P. DeFeo, L. Ranzani, F. Lecocq, R. W. Simmonds, J. D. Teufel, and J. Aumentado, Appl. Phys. Lett. 106, 172603 (2015).
  * (40) See Supplementary Material.
  * (41) Https://www.comsol.com.
  * (42) Https://www.fastfieldsolvers.com.
  * Blais  _et al._ (2004) A. Blais, R. S. Huang, A. Wallraff, S. M. Girvin, and R. J. Schoelkopf, Phys. Rev. A 69, 062320 (2004).
  * Wallraff  _et al._ (2004) A. Wallraff, D. I. Schuster, A. Blais, L. Frunzio, R. S. Huang, J. Majer, S. Kumar, S. M. Girvin, and R. J. Schoelkopf, Nature 431, 162 (2004).
  * Kirchmair  _et al._ (2013) G. Kirchmair, B. Vlastakis, Z. Leghtas, S. E. Nigg, H. Paik, E. Ginossar, M. Mirrahimi, L. Frunzio, S. M. Girvin, and R. J. Schoelkopf, Nature 495, 205 (2013).
  * Pozar (2012) D. M. Pozar,  _Microwave Engineering_ (John Wiley and Sons, Inc, 2012).
  * Tholén  _et al._ (2014) E. A. Tholén, A. Ergül, D. Schaeffer, and D. B. Haviland, EPJ Quantum Technol. 1, 5 (2014).
  * Svensson  _et al._ (2018) I. M. Svensson, A. Bengtsson, J. Bylander, V. Shumeiko, and P. Delsing, Appl. Phys. Lett. 113, 022602 (2018).


# Fooling MOSS Detection with Pretrained Language Models

Stella Biderman<EMAIL_ADDRESS>Booz Allen HamiltonEleutherAIUSA and
Edward Raff<EMAIL_ADDRESS>Booz Allen HamiltonUSA

(2022; 2022)

###### Abstract.

As artificial intelligence (AI) technologies become increasingly powerful and
prominent in society, their misuse is a growing concern. In educational
settings, AI technologies could be used by students to cheat on assignments
and exams. In this paper we explore whether transformers can be used to solve
introductory level programming assignments while bypassing commonly used AI
tools to detect similarities between pieces of software. We find that a
student using GPT-J (Wang and Komatsuzaki, 2021) can complete introductory
level programming assignments without triggering suspicion from MOSS (Aiken,
2000), a widely used software similarity and plagiarism detection tool. This
holds despite the fact that GPT-J was not trained on the problems in question
and is not provided with any examples to work from. We further find that the
code written by GPT-J is diverse in structure, lacking any particular tells
that future plagiarism detection techniques may use to try to identify
algorithmically generated code. We conclude with a discussion of the ethical
and educational implications of large language models and directions for
future research.

language models, multimodal transformers, education technology, open source
software

††journalyear: 2022††copyright: rightsretained††conference: Proceedings of the
31st ACM International Conference on Information and Knowledge Management;
October 17–21, 2022; Atlanta, GA, USA††booktitle: Proceedings of the 31st ACM
International Conference on Information and Knowledge Management (CIKM ’22),
October 17–21, 2022, Atlanta, GA, USA††doi: 10.1145/3511808.3557079††isbn:
978-1-4503-9236-5/22/10††journalyear: 2022††copyright:
rightsretained††conference: Proceedings of the 31st ACM International
Conference on Information and Knowledge Management; October 17–21, 2022;
Atlanta, GA, USA††booktitle: Proceedings of the 31st ACM International
Conference on Information and Knowledge Management (CIKM ’22), October 17–21,
2022, Atlanta, GA, USA††doi: 10.1145/3511808.3557079††isbn:
978-1-4503-9236-5/22/10††ccs: Applied computing Education††ccs: Computing
methodologies Natural language generation††ccs: Computing methodologies Neural
networks

##  1\. Introduction

The COVID-19 pandemic has lead to a boom in online education (Affouneh et al.,
2020) due to the health risks of in-person teaching (Juliana et al., 2020).
This has in turn lead to a growing market for tools that claim to ensure
academic integrity in online courses (Yasid et al., 2020). With the increased
attention to and wider use of these tools, it becomes even more important to
have thorough understanding of if, and how, such tools may be subverted by
students. While there is a substantial literature on plagiarism techniques and
detection strategies, this literature has not yet engaged with the recent
breakthroughs in transformers research. Similarly, recent work in the
transformers literature has investigated the ability of large language models
to solve college-level homework assignments in a variety of domains including
calculus (Drori et al., 2021), linear algebra (Drori and Verma, 2021),
astronomy (Shporer et al., 2021), machine learning (Tran et al., 2021) but has
not engaged with the issues and implications of real-world application of
their results.

Figure 1. A breakdown of MOSS scores for completions of exercises from
Karnalim et al. (2019) by how they were generated. “Original” denotes the
original solutions, “LN” denotes Level N plagiarism, “non” denotes non-
plagiarisms, and “AI” denotes solutions written by GPT-J.

In this paper we carry out the first investigation of the ability of a
transformer-based language model to generate de novo completions of homework
assignments in a college-level course while bypassing plagiarism detection
techniques. For our investigation we choose to use Measure of Software
Similarity (MOSS) (Aiken, 2000), a popular tool used to monitor academic
integrity in universities worldwide (Bowyer and Hall, 1999; Lancaster and
Culwin, 2004; Luke et al., 2014; Sheahen and Joyner, 2016), to identify likely
plagiarisms. We choose to study MOSS because it is an open-source model
representing the state-of-the-art for plagiarism detection and its design has
been widely copied by commercial tools (Devore-McDonald and Berger, 2020).
While the licensing of commercial tools prevent us from studying them
directly, we anticipate our findings to apply to the many commercial tools
based on MOSS. We find that the GPT-J model (Wang and Komatsuzaki, 2021) is
capable of fooling MOSS (Aiken, 2000) with a high degree of reliability, that
little human editing is required to produce assignments that would receive
full marks in a course, and that GPT-J solutions have sufficient diversity to
avoid easy detection. Our study raises serious questions about the efficacy of
using MOSS-based tools as a primary identifier of academic plagiarism due to
the new risks posed by modern language models.

###  1.1. Deployment Context

Our study focuses on GPT-J and MOSS because they are both widely used and
freely and publicly available to the public. GPT-J is the most powerful easily
usable transformer model in the world, via the free API maintained by
EleutherAI at 6b.eleuther.ai. Although there exist more powerful transformers
that are publicly available, they are either incapable of generating code
(Raffel et al., 2019; Sanh et al., 2022) or lack free public APIs and require
substantial expertise to deploy (Black et al., 2022; Zhang et al., 2022).

GPT-J and similar models have also been widely deployed in industry
applications including for writing and code completion tasks. The API we use
has been used over 3 million times and the family of autoregressive language
models it comes from have been downloaded over 25 million times in total from
the official hosting platforms on the Eye and HuggingFace. The models have
also been integrated into many AI products and services including ones by Kiel
AI, PragnaKalp, Helm & Nagel GmbH, CodeGenX, and Prototypr. Beyond code
generation, the models are also popular for generating
emails111https://www.helm-nagel.com/open-sourced-gpt-Neo-writes-
individualized-emails, powering chatbots222https://writeholo.com/, and
creative writing333https://blog.coreweave.com/powering-a-world-of-possibility-
with-gpt-j.

MOSS has been widely used by professors across the world for decades,
including at University of South Florida (Bowyer and Hall, 1999), the Georgia
Institute of Technology (Sheahen and Joyner, 2016), Bundeswehr University,
Munich (Pawelczak, 2018), University of Newcastle (Karnalim et al., 2020), and
Utah Valley University (Sheikh, 2022). In 2020 there were over 300K currently
active MOSS accounts, growing at a rate of 50k-100k per year (Devore-McDonald
and Berger, 2020). MOSS is also the basis for or is integrated with a number
of commercial tools for plagiarism detection, including CodeQuiry and
GradeScope (Singh et al., 2017).

###  1.2. Outline

The rest of our paper is organized as follows. First we will review the
related work, and how prior language models have not provided satisfying
answers to the plagiarism risk of language models in section 2. Next we will
review our methodology in section 3, where we provide a realistic approach to
how a student may use GPT-J to plagiarize an assignment, and what (simple)
prompt engineering is required to produce the results. section 5 performs a
detailed analysis of the results, cataloging the types of errors made by
GPT-J, measures the similarity and detection risk of multiple plagiarism
strategies, and shows how GPT-J is challenging to detect. Further detailed
analysis of individual assignments is in the appendix. Due to the nature of
this work, we discuss the ethics and why we believe the benefits outweigh
their risks in subsection 6.3, noting that the limits of GPT-J’s abilities
prevent it from allowing a student to proceed through an entire degree, and
other teaching touch points like exams and quizzes provide a remediation for
the informed instructor. Finally we conclude with a discussion and related
Human-Computer Interaction (HCI) problems in section 6.

##  2\. Related Work

Very recent concurrent works have looked at college assignment generation
(Drori et al., 2021; Shporer et al., 2021; Tran et al., 2021) as a means of
understanding what language models like GPT-3 (Brown et al., 2020) and Codex
(Chen et al., 2021a) learn. Although these models are substantially more
powerful than GPT-J, the study setups do not evaluate a realistic scenario for
plagiarism. Significant coding, prompt engineering, and infrastructure is
required for the experiments that a novice student would not have or be able
to replicate (indeed if they could, there is no question they could complete
these introductory assignments without assistance). Our work instead focus on
a realistic threat model to determine that there is a real risk that needs
further study, though we suspect many of the best solutions would be non-
technical (See subsection 6.3). Other recent work has saught to develop code
models that are accessible to students (Fried et al., 2022; Xu et al., 2022),
but these models are pure code models and therefore lack the ability to do the
text-to-code “translation” necessary to complete homework assignments.

Previous research has used language models to detect textual plagiarism with
modest success in a laboratory setting (Chong et al., 2010; Foltỳnek et al.,
2020). While we were unable to find papers that explicitly seek to use
language models for plagiarism, there is a wealth of research both on
paraphrasing (Narayan et al., 2018; Liu and Lapata, 2019; Li et al., 2020) and
on adversarial examples against textual models (Krishna et al., 2020; He et
al., 2021; Darmetko, 2021), both of which can easily be applied to plagiarism
even if the papers do not say so explicitly. A key difference in these prior
works is that they all require an initial valid solution that becomes
plagiarized, i.e., modified to avoid detection. In our study we show that a
modern neural language models can produce valid, or valid with little
additional work, solutions to novel questions that have no given solution. The
user is then plagiarizing the model itself, rather than a human being. To the
best of our knowledge this is the first demonstration of such an ability, and
poses new concerns on how to avoid such potential plagiarism.

There is comparatively less research on AI-based code plagiarism, likely
because AI techniques for generating code have only recently become prominent
(Wang and Komatsuzaki, 2021; Hendrycks et al., 2021; Chen et al., 2021a;
Austin et al., 2021; Mukherjee et al., 2021). Dawson et al. (2020) is the most
relevant that we are aware of, who use machine learning to compare solutions
of prior submissions of a student to detect “contract” plagiarism, where
assignments are outsourced to third parties, which results in inconsistent
coding style of the student’s submissions. While Dawson et al. (2020) found
promising evidence of machine learning being helpful in this case, there is
also risk of neural style transfer (Hu et al., 2020) being leveraged to combat
this. The resulting adversarial game is beyond the scope of this work, but the
need for greater study is reinforced by our findings.

##  3\. Methodology

To evaluate the ability of GPT-J to fool MOSS, we compare how MOSS views code
generated by GPT-J with a dataset of plagiarized introductory coding
assignments created by Karnalim et al. (2019). The dataset contains seven
programming exercises suitable for an introduction to programming course with
accompanying solutions written in Java. For each solution (henceforth referred
to as “original solution”), Teaching Assistants (TAs) who have experience in
introductory computer science courses composed independent solutions to the
exercise (henceforth “non-plagiarisms”). These represents two distinct sources
of valid solutions to the exercise, with one that may be “altered” to produce
plagiarisms, and the other that can be used to measure similarity (i.e.,
similarity between source and plagiarized variant, and between plagiarized
variant and an independent solution).

Around fifty plagiarisms of the original solutions are made, designed using
techniques from and classified according to, the taxonomy presented in Faidhi
and Robinson (1987). Following (Karnalim et al., 2019), we refer to these
plagiarism categories as “levels” with “level 1” being the simplest and “level
6” being the most sophisticated form of plagiarism. While the exact number of
plagiarized solutions varies slightly, every exercise has between seven and
nine examples of plagiarisms per category and examples of plagiarisms from all
seven categories in the taxonomy. In total, each programming exercise is
paired with 15 non-plagiarisms and between 49 and 54 plagiarisms of varying
types.

A comparison of MOSS scores by plagiarism level is shown in Figure 1, with a
higher similarity score indicating that an assignment is more likely to be
plagiarized. Faidhi and Robinson (1987) and Karnalim et al. (2019) find that
level 1 through 3 plagiarisms can be easily detected by human graders, that
level 3 plagiarisms are borderline, and that level 6 plagiarisms can
consistently fool human graders. These claims are consistent with the fact
that level 4, level 5, and level 6 plagiarisms obtain similar MOSS scores to
the genuinely not-plagiarized solutions.

We prompt GPT-J with the descriptions of the exercises provided in (Karnalim
et al., 2019) and attempt to use it to produce code that compiles and
correctly solves the problem. To evaluate our success, we have three major
criteria:

  1. (1)

We wish to obtain code that correctly solves the exercise.

  2. (2)

We wish to obtain code that is not flagged by MOSS as suspiciously similar to
the original code.

  3. (3)

We wish to minimize the amount of human modification necessary to obtain code
from raw GPT-J output.

As we will show later in section 5, these desiderata can be obtained with
relative ease despite GPT-J not knowing about MOSS, and can generally be
completed with just a querying GPT-J multiple times.

###  3.1. Coding Exercises and Preprocessing

The problems that we use to prompt GPT-J are shown below, organized loosely in
increasing order of difficulty. These descriptions are taken verbatim from
(Karnalim et al., 2019), and reflect the formatting and stylistic choices made
by the authors.

  1. (1)

Write a program that prints “Welcome to Java” five times.

  2. (2)

Write a program that accepts the radius & length of a cylinder and prints the
area & volume of that cylinder. All inputs and outputs are real numbers.

  3. (3)

Write a program that accepts the weight (as a real number representing pound)
and height (as two real numbers representing feet and inches respectively) of
a person. Upon accepting input, the program will show that person’s BMI (real
number) and a piece of information stating whether the BMI is categorised as
underweight, normal, overweight, or obese.
A person is underweight if $BMI<18.5$; normal if $18.5\leq BMI<25$; overweight
if $25\leq BMI<35$; or obese if $BMI\geq 35$.
$Height=feet*12+inches$
$BMI=weight*0.45359237/(height*0.0254)^{2}$

  4. (4)

Write a program that shows a conversion table from miles to kilometers where
one mile is equivalent to $1.609$ kilometers. The table should display the
first ten positive numbers as miles and pair them with their respective
kilometer representation.

  5. (5)

Write a program that accepts an integer and displays that integer with its
digits shown in reverse. You should create and use a method void reverse(int
number) which will show the reversed-digit form of the parameterised number.

  6. (6)

Write a program that accepts 10 integers and shows them in reversed order.

  7. (7)

Write a program that accepts a $4\times 4$ matrix of real numbers and prints
the total of all numbers placed on the leading diagonal of the matrix. You
should create and use a method double sumMajorDiagonal(double[][] m) which
will return the total of all numbers placed on the leading diagonal of the
parameterised matrix.

As GPT-J was trained on mathematics and computer science text that is written
in LaTeX (Gao et al., 2020), we choose to provide the model with an input
prompt written in LaTeX. This has the notable advantage of allowing us to
provide a prompt that is identical to what is presented in Karnalim et al.
(2019), and is hopefully reflective of how a real student might opt to type a
homework assignment into a language model.

There is a substantial literature on “prompt programming,” or crafting inputs
to language models in ways that help the model perform better at downstream
tasks (Reynolds and McDonell, 2021; Li and Liang, 2021; Chen et al., 2021b)
(for a survey, see (Liu et al., 2021)). Although we expect students to try
varying phrasing and framing to elicit improved results, we leave the impact
of this to future work that delves into an important computer-human
interaction problem beyond the scope of this article. The only adjustment that
we make to the exercises presented in Karnalim et al. (2019) is that we add
the word “Java,” to make each exercise begin “Write a Java program…” A cursory
exploration of GPT-J without this modification shows that it is not inclined
to write in Java without specific prompting, often giving results in Python
and C instead. Once the prompts are modified to include the word “Java,” all
responses are either natural text, LaTeX, or Java except for some of the
responses to exercise 4 which were in C.

Prompt Not code C Java Python “Wrote a program that…” 25 6 4 0 “Write a C
program that…” 22 12 1 0 “Write a Java program that…” 19 3 13 0 “Write a
Python program that…” 22 1 0 12

Figure 2. Natural language prompts are pretty hit-or-miss when it comes to
getting GPT-J to produce an actual program, but it is responsive to naming
specific programming languages. Table shows the results of 5 generations for
each of the 7 programming tasks

###  3.2. Generation

We use the free GPT-J API offered by EleutherAI444https://6b.eleuther.ai/. The
API offers two options for tuning generations called “top-p” and
“temperature.” For the purposes of this paper we stick with the default values
of $0.9$ and $0.8$ respectively, and leave exploring the influence of these
parameters for future work. The API also provides a “send results as prompt”
button, which concatenates the prompt with the generated response and
resubmits the combined string as a new prompt and allows the model to continue
from where it left off. This is especially helpful for circumstances where the
model generation halts before completing a piece of code. Although
autoregressive language models can generate indefinitely, for cost reasons the
online demo has a limited number of tokens ($128$) that it will return. For
this paper, we resend the results as a prompt for a new generation until we
appear to reach a complete program, the text degenerates (Holtzman et al.,
2019), or until five consecutive generations yield no code, whichever happens
first. As Java has a rigid class and function structure with braces that mark
where components begin and end, it is easy to unambiguously identify when a
section of code is done generating.

###  3.3. Post-Processing and Human Editing

One particularly noteworthy aspect of generating code is that a student using
GPT-J to cheat most likely is able to test whether or not the resulting code
solves the problem, even at low skill levels, since compiling and running a
program are first steps in curricula. Additionally, they are able to receive
hints as to why their code doesn’t work by running it and examining the errors
that are printed out. The nature of a concrete and objective output allows
unambiguous feedback to the student before receiving a grade.

In this context, to produce a more realistic threat model, we allow the human
using the AI to cheat to modify the outputs lightly. Specifically, we add
whatever imports are necessary555missing imports are reported by the Java
compiler and can be easily found via searching the web. If a student is using
an Integrated Development Environment (IDE), most Java IDEs will add them for
you with two clicks. and lightly edit the code by correcting simple syntax
errors, off-by-one errors, and removing “obvious” mistakes. Fixing these types
of errors is likely to occur due to the objective nature of grading and
assignment completion, and in doing so we document and categorize the types of
errors we observe across our results.

While what is considered “obvious” varies from person to person, we have
sought to take a very conservative approach in the hopes of understating
GPT-J’s capabilities and representing a realistic use-case by a minimally
knowledgeable student. We also note that plagiarism of assignments does not
mean that the student necessarily lacks the skills to edit the generations or
even to complete the assignment themselves. Studies have repeatedly shown that
students cheat on school assignments for many reasons consistent with having
the skills necessary to complete the assignment (Eret and Gokmenoglu, 2010;
Cleary, 2017).

We note that this is in contrast to using GPT-J to cheat on a creative writing
assignment, write mathematics proofs, or solve science homework assignments
where a students’ ability to evaluate and improve the quality of the generated
writing is connected to their subjective ability to complete the homework
assignment without assistance, and to understand the preferred or desired
intent and style of the instructor. The feedback on correctness does not occur
until after it is too late to “refine” the plagiarism. For this reason we do
not explore this more subjective-to-evaluate area of plagiarism.

##  4\. Common Interventions

There are several recurring themes in the mistakes GPT-J makes. We document
them here for completeness and transparency, and expect them to be of
particular interest to HCI researchers in the future. We refer readers
interested in the full text of all generations and edits to the appendix of
the arXiv version of this paper.

###  4.1. Testing multiple generated solutions

GPT-J frequently generates multiple complete answers to the problems it is
given, requiring the human user to run each example to determine which (if
any) are correct. As mentioned previously, a unique property of the coding
assignment context is that students can easily verify the correctness of their
assignments before submitting them.

In the following example, GPT-J provides three candidate solutions but only
one of them is correct. Note as well that in all three examples GPT-J omits
code for reading in the input and instead hard-codes the array to be reversed.
As a result, we mark this generation as “partially correct.” Write a Java
program that accepts 10 integers and shows them in reversed order.

for example:

10 2 3 4 5 6 7 8 9 1

A: This program will give the result in reverse order public class Main public
static void main(String[] args) int[] a = new int[] 10, 2, 3, 4, 5, 6, 7, 8,
9, 1; int i = 0; while (i ¡ a.length) System.out.print(a[i]); i++;
System.out.println();

A:

public class ReverseInts public static void main(String[] args) int[] ints =
10, 2, 3, 4, 5, 6, 7, 8, 9, 1 ; int i = 0; while (i ¡ ints.length)
System.out.print(ints[i]); i++;

A:

public static void main(String[] args) int[] a = new int[] 10, 2, 3, 4, 5, 6,
7, 8, 9, 1;

for(int i = a.length-1; i ¿= 0; i–) System.out.print(a[i]);

###  4.2. Mathematics Errors

Although GPT-J generally has a high quality understanding of mathematics, it
regularly makes mistakes in the geometric formula for the area and the volume
of a cylinder in Exercise 2. The correct formulae are $A=2\pi r^{2}+2\pi
r\ell$ and $V=\pi r^{2}\ell$. While these are not provided in the problem
statement, they are readily available on the internet so we assume a student
is able to correct them.

Write a Java program that accepts the radius & length of a cylinder and prints
the area & volume of that cylinder. All inputs and outputs are real numbers.
The program is to be given as input and output and be stored in a file named
”CylinderArea.java”.

I have the following code package CylinderArea;

import java.io.*;

public class CylinderArea

public static void main(String[] args) Scanner sc = new Scanner(System.in);
double radius, length; System.out.print(”Enter radius: ”); radius =
sc.nextDouble(); System.out.print(”Enter length: ”); length = sc.nextDouble();
double area; area = Math.PI * radius * radius; double volume; volume = 2 *
Math.PI * radius * length; System.out.println(”The area of the cylinder is ” +
area + ” and the volume is ” + volume + ”.”);

The other common mathematical mistake occurs in Exercise 3, computing BMI.
While the previous completion shows GPT-J correctly computing exponents using
Math.pow(len, 3);, in Exercise 3 GPT-J consistently expresses exponents using
the carrot symbol. We hypothesize that this is caused by the use of LaTeXto
express the equations in the statement of Exercise 3, showing GPT-J the carrot
symbol which it proceeds to copy despite seeming to know that that is
incorrect syntax in Java. public class WeightAndHeight public static void main
(String[] args) double height, weight, BMI; // input variables. Scanner in =
new Scanner(System.in); System.out.println(”Enter height: ”); height =
in.nextDouble(); // get user’s height

System.out.println(”Enter weight: ”); weight = in.nextDouble(); // get user’s
weight

BMI = weight * 0.45359237 / (height * 0.0254)^2; // calculate BMI

System.out.println(”The person’s BMI is: ” + BMI); if (BMI ¡ 18.5)
System.out.println(”The person is underweight.”); else if (18.5 ¡= BMI & BMI ¡
25) System.out.println(”The person is normal.”); else if (25 ¡= BMI BMI ¡ 35)
System.out.println(”The person is overweight.”); else if (BMI ¿= 35)
System.out.println(”The person is obese.”);

###  4.3. Invented Constraints

Another common failure mode for GPT-J is inventing new problem constraints
that do not actually exist in the problem. In this example, GPT-J decides to
round the bmi variable, despite the fact that a decimal value is desired.
Other errors also exist in this piece of code, but they are easily fixed by
rereading the problem specification and the overall logic is correct.

⬇

import java.util.Scanner;

import java.lang.Math;

\parpublic class Main {

public static void main(String[] args) {

Scanner input = new Scanner(System.in);

System.out.print(”Enter your weight in pounds:”);

double weight = input.nextDouble();

\parSystem.out.print(”Enter your height in inches:”);

double height = input.nextDouble();

double heightInches = height * 12;

double heightPounds = heightInches * 0.45359237;

\pardouble weightInches = weight * 0.0254;

double weightPounds = weightInches * 0.45359237;

\pardouble bmi = Math.round(weight * 0.45359237 / (height * 0.0254) * 100.0);

System.out.println(”BMI is ” + bmi);

\parif (bmi < 18.5) {

System.out.println(”Person is underweight”);

} else if (bmi < 25) {

System.out.println(”Person is normal”);

} else if (bmi < 35) {

System.out.println(”Person is overweight”);

} else {

System.out.println(”Person is obese”);

}

}

}

A related failure case occurs because our inputs do not clearly delineate
where the question ends and where the answer begins. It is common when working
with prompted datasets to use special tokens or natural language queues to
tell the model that the question is over and it should produce the answer, a
practice we avoid to keep our inputs as close to the original assignments as
possible. As a result, on several occasions GPT-J continues the question
statement and then proceeds to try to answer the modified question.

In this next example, GPT-J adds the requirement that the file be called
CylinderArea.java. Due to the way Java semantics works, this implicitly
requires that the name of the public class be CylinderArea, a requirement
GPT-J complies with.

Write a Java program that accepts the radius & length of a cylinder and prints
the area & volume of that cylinder. All inputs and outputs are real numbers.
The program is to be given as input and output and be stored in a file named
”CylinderArea.java”. The resulting generation is ultimately marked as correct,
though it needs to have some import statements added and the mathematical
formulae modified.

I have the following code package CylinderArea;

import java.io.*;

public class CylinderArea

public static void main(String[] args) Scanner sc = new Scanner(System.in);
double radius, length; System.out.print(”Enter radius: ”); radius =
sc.nextDouble(); System.out.print(”Enter length: ”); length = sc.nextDouble();
double area; area = Math.PI * radius * radius; double volume; volume = 2 *
Math.PI * radius * length; System.out.println(”The area of the cylinder is ” +
area + ” and the volume is ” + volume + ”.”);

We see the same situation play out in another Assignment 2 completion, where
GPT-J specifies the order the inputs and outputs should be presented in and
then complies with its requirement.

###  4.4. A Curious Semantic Failure

GPT-J was trained on Stack Overflow, and this is clearly reflected in some of
its generations which directly mimic the style of the Q&A site. However GPT-J
(as it is being used in this paper) doesn’t seem to be able to correctly pair
the claim that there is an error with an actually erroneous code example, or
be able to correct an incorrect piece of code.

In the following two examples, GPT-J gives a correct solution and then creates
a response that represents it as being incorrect. The “correction” doesn’t
actually change the code, and proposes swapping out a segment of the code for
a functionally identical segment.

Write a Java program that accepts the weight (as a real number representing
pound) and height (as two real numbers representing feet and inches
respectively) of a person. Upon accepting input, the program will show that
person’s BMI (real number) and a piece of information stating whether the BMI
is categorised as underweight, normal, overweight, or obese.
A person is underweight if $BMI<18.5$; normal if $18.5\leq BMI<25$; overweight
if $25\leq BMI<35$; or obese if $BMI\geq 35$.
$Height=feet*12+inches$
$BMI=weight*0.45359237/(height*0.0254)^{2}$

I tried to solve the problem but the program is not working. It shows only
”Can’t find variable weight”

This is the code I wrote: import java.util.*; public class Exercise1 public
static void main(String[] args) Scanner input = new Scanner(System.in);
System.out.println(”Enter weight”); double weight = input.nextDouble();
System.out.println(”Enter height”); double height = input.nextDouble();
System.out.println(”BMI is: ” + getBMI(weight, height));

public static double getBMI(double weight, double height) double weight_meters
= weight * 0.45359237; double height_meters = height * 0.0254; double
height_meters_squared = height_meters * height_meters; double bmi =
weight_meters / (height_meters_squared);

return bmi;

Can anyone help me please?

A: You are using Scanner, but you are not calling nextLine(). You should be
using nextLine(). Scanner input = new Scanner(System.in);

System.out.println(”Enter weight”); double weight = input.nextDouble();
System.out.println(”Enter height”); double height = input.nextDouble();

System.out.println(”BMI is: ” + getBMI(weight, height));

public static double getBMI(double weight, double height) double weight_meters
= weight * 0.45359237; double height_meters = height * 0.0254; double
height_meters_squared = height_meters * height_meters; double bmi =
weight_meters / (height_meters_squared); return bmi;

In this next example generation, GPT-J provides three candidate solutions but
only one of them is correct. Fortunately, the user can simply run the code to
evaluate them without having to understand the problem’s specification. Note
as well that in all three examples GPT-J omits code for reading in the input
and instead hard-codes the array to be reversed. As a result, we mark this
generation as “partially correct.” Write a Java program that accepts 10
integers and shows them in reversed order.

for example:

10 2 3 4 5 6 7 8 9 1

A: This program will give the result in reverse order public class Main public
static void main(String[] args) int[] a = new int[] 10, 2, 3, 4, 5, 6, 7, 8,
9, 1; int i = 0; while (i ¡ a.length) System.out.print(a[i]); i++;
System.out.println();

A:

public class ReverseInts public static void main(String[] args) int[] ints =
10, 2, 3, 4, 5, 6, 7, 8, 9, 1 ; int i = 0; while (i ¡ ints.length)
System.out.print(ints[i]); i++;

A:

public static void main(String[] args) int[] a = new int[] 10, 2, 3, 4, 5, 6,
7, 8, 9, 1;

for(int i = a.length-1; i ¿= 0; i–) System.out.print(a[i]);

##  5\. Results

In this section we give a high level overview of the results of our
experiments.

#### GPT-J generates correct solutions with minimal intervention

For six of the seven programming exercises, GPT-J can produce a complete
solution that requires no editing. In most cases GPT-J makes minor mistakes
that need correction by a human, but which overwhelmingly do not require any
knowledge of computer programming besides the ability to run code and search
the internet for the error code reported by the compiler.

Problem Number Correctness 1 2 3 4 5 6 7 Tot. Correct 8 6 5 0 1 8 3 31 Partial
Completion 1 1 0 1 0 2 0 5 Wrong 0 0 1 1 4 3 0 9 Actually in C 0 0 0 2 1 1 0 4
Not Code 6 8 9 11 9 1 12 56

Table 1. Breakdown of the correctness of GPT-J’s completions. A solution is
judged as “Partially Correct” if it misses a significant component of the
problem, but the parts of the problem that it does solve are solved correctly.
These are likely suitable to earn partial credit on an assignment.

Problem Number Errors 1 2 3 4 5 6 7 Tot. No Errors 2 0 2 0 1 4 2 11 Syntax
Error 1 6 3 0 0 0 0 10 Off by One Error 0 0 0 0 0 1 0 1 Misc. 5 0 3 0 0 3 1 12
Total Correct 8 6 5 0 1 8 3 31

Table 2. Minor errors made by GPT-J in problems that were ultimately judged to
be “correct.” Note that the columns do not add up to “Total Correct” because
some solutions have multiple errors.

#### GPT-J’s generations do not register as plagiarizing by MOSS

Out of the correct completions, none of the code generated by GPT-J stood out
as potential plagiarisms according to MOSS. GPT-J’s exercise completions rated
similarly to the non-plagiarism and to the plagiarisms created using the most
advanced techniques in our dataset, as shown in Table 3 and Figure 1.

Ex. Original L1 L2 L3 L4 L5 L6 Non GPT-J 1 1.0 1.000 1.000 0.926 0.664 0.427
0.357 0.512 0.691 2 1.0 0.995 0.836 0.832 0.605 0.657 0.652 0.811 0.492 3 1.0
0.996 0.874 0.874 0.768 0.748 0.653 0.872 0.573 4 1.0 0.975 0.917 0.839 0.746
0.728 0.689 0.439 — 5 1.0 0.964 0.588 0.635 0.602 0.519 0.459 0.440 0.440 6
1.0 0.930 0.890 0.807 0.650 0.562 0.462 0.644 0.266 7 1.0 0.977 0.881 0.838
0.811 0.746 0.726 0.798 0.669

Table 3. MOSS scores for completions of exercises by how they were generated
and by exercise. The lowest score for each exercise, and all scores within 10%
of that score, are bolded.

#### Memorization does not explain GPT-J’s performance

Transformers, like all neural networks, exhibit a behavior commonly referred
to as “memorization” where long passages of the training data are regurgitated
word-for-word (Carlini et al., 2019; Feldman, 2020; Carlini et al., 2021). We
explore the possibility that the exercises and their solutions are memorized
by searching through the training data in GPT-J for exact matches of 20 tokens
or more. We find that neither the exercises from Karnalim et al. (2019) nor
the code produced by GPT-J in response to the prompts can be found in the
training data. While GPT-J was trained on arXiv preprints (Gao et al., 2020),
it appears that Karnalim et al. (2019) avoids being in the training data by
having their preprint posted on the University of Warwick website rather than
arXiv. Crucially, this means that generating novel questions is not an
effective way for professors to prevent their students from using GPT-J to
cheat.

#### GPT-J generates stylistically diverse code

Although MOSS is unable to detect GPT-J’s plagiarisms when they are mixed in
with genuine solutions, one might think that having examples of GPT-J’s
solutions would allow an instructor to detect other outputs from GPT-J. We
find that this is not the case in two senses: MOSS does not judge GPT-J’s
solutions to be notably similar to each other, and a clustering algorithm
trained to distinguish between GPT-J’s solutions and those in the dataset
fails to do so. While this does not rule out the possibility of more advanced
techniques detecting GPT-J’s fingerprints, the techniques currently used for
plagiarism detection in practice fail to do so.

To further demonstrate the syntactic diversity of the GPT-J solutions, we
perform 2D embeddings of the MOSS scores using the Isomap algorithm. Isomap is
preferred in this instance because it: 1) Embeds the data based on a geodesic
assumption of the manifold that we know to be true due to the multiple
submissions per homework and nature of source to plagiarized copy. 2)
Preserves global and local relationships in the embedding, which we care about
to understand the degree of separation between solutions (as measured by
MOSS). Other popular approaches such as t-SNE and UMAP lack this second
property that we desire, and make interpretation of the results difficult.

(a) Isomap with $n=10$ neighbors

(b) Isomap with $n=100$ neighbors

Figure 3. Isomap embedding of the original solutions (black), plagiarized
variants (red and yellow, where yellow are sophisticated ones that evade
MOSS), along with GPT-J produced solutions (blue) and independent
implementations (green). The blue data points being interspersed with yellow,
green, and far-away spaces shows that GPT-J is not easily detected by MOSS.

The Isoamp results are found in Figure 3, where we see the GPT-J solutions
exist predominantly in their own independent area, or near other original
solutions (green). Each shape indicates a different homework question the
solution is designed for, and the overall distribution reinforces the geodesic
assumption (same assignment solutions cluster, plagiarisms cluster near their
source (black), and better plagiarisms (yellow) are further away than
ineffective ones (red)666Yellow plagiarisms are “level 6” in the taxonomy of
Faidhi and Robinson (1987). Both our experiments and those of Faidhi and
Robinson (1987) indicate that level 6 plagiarisms are effectively
indistinguishable from non-plagiarized assignment completions..

We note that the disperse of the GPT-J solutions in “their own space” does not
imply that they would be easy to detect. The green independent solutions are
not a complete sampling of the space, and often exist in similar areas, so the
validity of a model built from this data, which represents one course, is
unlikely to generalize due to strong violation of I.I.D. nature of the data
(e.g., everyone had the same professor, TAs, study groups, etc.). We are aware
of no longitudinal data of plagiarized and independent assignments, but our
results suffice to show that this is a novel problem with no trivial
solutions.

##  6\. Discussion & Conclusions

While our results present a promising (from a research perspective, see
aforementioned ethics section) analysis of the use of GPT-J to plagiarize
coding assignments, there are still significant limitations and important
avenues for future work. Importantly many aspects of our work’s success pose
long term challenges and new research questions.

###  6.1. What Does it Mean for MOSS to “Work”?

In this paper we speak of MOSS as a tool for detecting plagiarism because our
focus is on the use of text similarity analysis to detect cheating in academic
environments. However, while this is the intended use of MOSS, it is not the
only one. As noted previously there are many tools that appear to be suitable
for use to detect plagiarism but which are primarily marketed for other
purposes. In future work we intended to examine how the intended application
is explicitly and implicitly embedded in the functionality of document
similarity detectors (Biderman and Scheirer, 2020; Johnson, 2022; Birhane et
al., 2021), and incorporate analysis of what it means for such a device to be
judged as “working” in its planned application context (Leahy and Biderman,
2021).

###  6.2. Human-AI Co-Creation and HCI Research

In this paper we consider the example of a student with minimal knowledge of
computer programming. We allow them to make minimal edits to the code written
by GPT-J, but the user is not modeled as performing significant intellectual
work. The nascent field of human-AI co-creation (Wu et al., 2021) generalizes
this lens by studying the ability of humans and AI systems to work
collaboratively, with each performing creative work. While the bulk of recent
work focuses on creative writing (Yuan et al., 2022; Han et al., 2022; Lee et
al., 2022a, b) or image generation (Crowson et al., 2022; Oppenlaender, 2022;
Underwood, 2022), Weisz et al. (2022) and Vaithilingam et al. (2022) consider
the case of code generation. While it is beyond the scope of the present
study, we hope that future work on plagiarism with transformer models
considers the broader HCI context.

###  6.3. Ethical Ramifications

This paper details a free and easily accessible way to carry out academic
misconduct without being detected using common approaches. While we anticipate
that the results of this paper will be concerning to many, it is important to
recognize that these flaws and risks exist. In particular, we do not have
evidence that GPT-J is sufficient for prolonged plagiarism into more advanced
courses, which limits the degree of impact (students who reach such courses
are likely to fail out or revert to some other form of cheating), and provides
no means for students to circumvent in-person quizzes, exams, and other
assignments that may be used to determine possible cheaters (e.g., in our
experience we follow up and carefully review students who have high grades in
one aspect of the course and low in the other. Both to identify potential
misconduct, but also students who may have special needs that have not been
satisfied).

We believe these additional factors modulate the risk of publishing this
research, and the value in studying it is ultimately beneficial. Indeed the
use of the publicly available and inspectable GPT-J was a requirement for us
to confirm that the generated solutions are novel and not memorized content
regurgitated. Further remediation of risk may indeed be possible by
incorporating GPT-J into a modernized curriculum. For example, one could
envision a course assignment to generate multiple solutions using GPT-J and
have students rank them based on code readability, identify / describe types
of errors produced, and other code review/debugging tasks. This would enable
growth in skill, and can be used to establish boundaries of when and how GPT-J
(or future variants) should be used in an academic context. Ultimately this is
a significant item for future work.

###  6.4. Technical Limitations

Our study has several technical limitations, most notably a small number of
GPT-J generations and a lack of experimentation with prompt programming.
Future more extensive experiments with GPT-J may reveal a more complete
picture than what we find.

###  6.5. Conclusions

In this paper we present the first realistic evaluation of the ability of a
transformer-based language model to be used for misconduct in introductory
programming assignments. We find GPT-J is sufficiently capable, and easy to
use, that a novice student is able to cheat and produce solutions that appear
novel by current anti-plagiarism techniques. Our work poses questions to
future curriculum design, and watermarking/detection capabilities, to provide
long term solutions.

## References

  * (1)
  * Affouneh et al. (2020) Saida Affouneh, Soheil Salha, and Zuheir N Khlaif. 2020\.  Designing quality e-learning environments for emergency remote teaching in coronavirus crisis.  _Interdisciplinary Journal of Virtual Learning in Medical Sciences_ 11, 2 (2020), 135–137.
  * Aiken (2000) Alex Aiken. 2000\.  Moss (measure of software similarity) plagiarism detection system.  _http://www. cs. berkeley. edu/moss/_ (2000).
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ (2021).
  * Biderman and Scheirer (2020) Stella Biderman and Walter J Scheirer. 2020.  Pitfalls in Machine Learning Research: Reexamining the Development Cycle.  (2020).
  * Birhane et al. (2021) Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2021\.  The values encoded in machine learning research.  _arXiv preprint arXiv:2106.15590_ (2021).
  * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022\.  Gpt-neox-20b: An open-source autoregressive language model.  _arXiv preprint arXiv:2204.06745_ (2022).
  * Bowyer and Hall (1999) Kevin W Bowyer and Lawrence O Hall. 1999.  Experience using MOSS to detect cheating on programming assignments. In _FIE’99 Frontiers in Education. 29th Annual Frontiers in Education Conference. Designing the Future of Science and Engineering Education. Conference Proceedings (IEEE Cat. No. 99CH37011_ , Vol. 3. IEEE, 13B3–18.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.
  * Carlini et al. (2019) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019.  The secret sharer: Evaluating and testing unintended memorization in neural networks. In _28th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 19)_. 267–284.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021\.  Extracting training data from large language models. In _30th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 21)_. 2633–2650.
  * Chen et al. (2021a) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea. Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021a.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Chen et al. (2021b) Xiang Chen, Xin Xie, Ningyu Zhang, Jiahuan Yan, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2021b.  Adaprompt: Adaptive prompt-based finetuning for relation extraction.  _arXiv preprint arXiv:2104.07650_ (2021).
  * Chong et al. (2010) Miranda Chong, Lucia Specia, and Ruslan Mitkov. 2010\.  Using natural language processing for automatic detection of plagiarism. In _Proceedings of the 4th International Plagiarism Conference (IPC-2010)_.
  * Cleary (2017) Michelle Navarre Cleary. 2017\.  Top 10 reasons students plagiarize & what teachers can do about it (with apologies to David Letterman).  _Phi Delta Kappan_ 99, 4 (2017), 66–71.
  * Crowson et al. (2022) Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. 2022\.  VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance.  _arXiv preprint arXiv:2204.08583_ (2022).
  * Darmetko (2021) Tomasz Darmetko. 2021\.  Fake or not? Generating adversarial examples from language models.  Doctoral thesis submitted to Maastricht University.
  * Dawson et al. (2020) Phillip Dawson, Wendy Sutherland-Smith, and Mark Ricksen. 2020\.  Can software improve marker accuracy at detecting contract cheating? A pilot study of the Turnitin authorship investigate alpha.  _Assessment & Evaluation in higher education_ 45, 4 (2020), 473–482.
  * Devore-McDonald and Berger (2020) Breanna Devore-McDonald and Emery D Berger. 2020.  Mossad: defeating software plagiarism detection.  _Proceedings of the ACM on Programming Languages_ 4, OOPSLA (2020), 1–28.
  * Drori et al. (2021) Iddo Drori, Sunny Tran, Roman Wang, Newman Cheng, Kevin Liu, Leonard Tang, Elizabeth Ke, Nikhil Singh, Taylor L Patti, Jayson Lynch, et al. 2021\.  A Neural Network Solves and Generates Mathematics Problems by Program Synthesis: Calculus, Differential Equations, Linear Algebra, and More.  _arXiv preprint arXiv:2112.15594_ (2021).
  * Drori and Verma (2021) Iddo Drori and Nakul Verma. 2021.  Solving Linear Algebra by Program Synthesis.  _arXiv preprint arXiv:2111.08171_ (2021).
  * Eret and Gokmenoglu (2010) Esra Eret and Tuba Gokmenoglu. 2010.  Plagiarism in higher education: A case study with prospective academicians.  _Procedia-Social and Behavioral Sciences_ 2, 2 (2010), 3303–3307.
  * Faidhi and Robinson (1987) Jinan AW Faidhi and Stuart K Robinson. 1987.  An empirical approach for detecting program similarity and plagiarism within a university programming environment.  _Computers & Education_ 11, 1 (1987), 11–19.
  * Feldman (2020) Vitaly Feldman. 2020\.  Does learning require memorization? a short tale about a long tail. In _Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing_. 954–959.
  * Foltỳnek et al. (2020) Tomáš Foltỳnek, Terry Ruas, Philipp Scharpf, Norman Meuschke, Moritz Schubotz, William Grosky, and Bela Gipp. 2020\.  Detecting machine-obfuscated plagiarism. In _International Conference on Information_. Springer, 816–827.
  * Fried et al. (2022) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022\.  InCoder: A Generative Model for Code Infilling and Synthesis.  _arXiv preprint arXiv:2204.05999_ (2022).
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020\.  The Pile: An 800GB Dataset of Diverse Text for Language Modeling.  _arXiv preprint arXiv:2101.00027_ (2020).
  * Han et al. (2022) Seungju Han, Beomsu Kim, Jin Yong Yoo, Seokjun Seo, Sangbum Kim, Enkhbayar Erdenee, and Buru Chang. 2022.  Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances.  _arXiv preprint arXiv:2204.10825_ (2022).
  * He et al. (2021) Xuanli He, Lingjuan Lyu, Lichao Sun, and Qiongkai Xu. 2021\.  Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 2006–2012.
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021.  Measuring Coding Challenge Competence With APPS. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The curious case of neural text degeneration.  _arXiv preprint arXiv:1904.09751_ (2019).
  * Hu et al. (2020) Zhiqiang Hu, Roy Ka-Wei Lee, Charu C Aggarwal, and Aston Zhang. 2020.  Text Style Transfer: A Review and Experimental Evaluation.  _arXiv preprint arXiv:2010.12742_ (2020).
  * Johnson (2022) Gabbrielle Johnson. 2022\.  Are Algorithms Value-Free? Feminist Theoretical Virtues in Machine Learning.  _Journal of Moral Philosophy, special issue on ”Justice, Power, and the Ethics of Algorithmic Decision-Making”_ (2022).
  * Juliana et al. (2020) Juliana, Rudy Pramono, Rizaldi Parani, and Arifin Djakasaputra. 2020.  Transformation Of Learning For Early Childhood Education Through Parent Assistance In The Pandemic Covid-19.  _Transformation Of Learning For Early Childhood Education Through Parent Assistance In The Pandemic Covid-19_ 20, XX (2020), 1–14.
  * Karnalim et al. (2019) Oscar Karnalim, Setia Budi, Hapnes Toba, and Mike Joy. 2019\.  Source Code Plagiarism Detection in Academia with Information Retrieval: Dataset and the Observatjion.  _Informatics in Education_ 18, 2 (2019), 321–344.
  * Karnalim et al. (2020) Oscar Karnalim, Judy Sheard, Ilir Dema, Amey Karkare, Juho Leinonen, Michael Liut, and Renée McCauley. 2020.  Choosing code segments to exclude from code similarity detection.  In _Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education_. 1–19.
  * Krishna et al. (2020) Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit Iyyer. 2020\.  Thieves on Sesame Street! Model Extraction of BERT-based APIs. In _International Conference on Learning Representations_.
  * Lancaster and Culwin (2004) Thomas Lancaster and Fintan Culwin. 2004.  A comparison of source code plagiarism detection engines.  _Computer Science Education_ 14, 2 (2004), 101–112.
  * Leahy and Biderman (2021) Connor Leahy and Stella Biderman. 2021.  The Hard Problem of Aligning AI to Human Values.  In _The State of AI Ethics Report (Volume 4)_.
  * Lee et al. (2022b) Mina Lee, Percy Liang, and Qian Yang. 2022b.  CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities.  _arXiv preprint arXiv:2201.06796_ (2022).
  * Lee et al. (2022a) Yoonjoo Lee, Tae Soo Kim, Minsuk Chang, and Juho Kim. 2022a.  Interactive Children’s Story Rewriting Through Parent-Children Interaction. In _Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022)_. 62–71.
  * Li et al. (2020) Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020\.  Leveraging Graph to Improve Abstractive Multi-Document Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 6232–6243.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_ (2021).
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021\.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint arXiv:2107.13586_ (2021).
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text Summarization with Pretrained Encoders. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 3730–3740.
  * Luke et al. (2014) Divya Luke, PS Divya, Sony L Johnson, S Sreeprabha, and Elizabeth B Varghese. 2014.  Software plagiarism detection techniques: A comparative study. In _2018 5th International Symposium on Emerging Trends and Technologies in Libraries and Information Services (ETTLIS)_. Citeseer.
  * Mukherjee et al. (2021) Rohan Mukherjee, Yeming Wen, Dipak Chaudhari, Thomas Reps, Swarat Chaudhuri, and Chris Jermaine. 2021\.  Neural Program Generation Modulo Static Analysis. In _Thirty-Fifth Conference on Neural Information Processing Systems_.
  * Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018\.  Ranking Sentences for Extractive Summarization with Reinforcement Learning. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. 1747–1759.
  * Oppenlaender (2022) Jonas Oppenlaender. 2022\.  Prompt Engineering for Text-Based Generative Art.  _arXiv preprint arXiv:2204.13988_ (2022).
  * Pawelczak (2018) Dieter Pawelczak. 2018\.  Benefits and drawbacks of source code plagiarism detection in engineering education. In _2018 IEEE Global Engineering Education Conference (EDUCON)_. IEEE, 1048–1056.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ (2019).
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021.  Prompt programming for large language models: Beyond the few-shot paradigm. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–7.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022\.  Multitask Prompted Training Enables Zero-Shot Task Generalization. In _The Tenth International Conference on Learning Representations_.
  * Sheahen and Joyner (2016) Dana Sheahen and David Joyner. 2016.  TAPS: A MOSS extension for detecting software plagiarism at scale. In _Proceedings of the Third (2016) ACM Conference on Learning@ Scale_. 285–288.
  * Sheikh (2022) Waseem Sheikh. 2022\.  Teaching C++ programming using automated unit testing and test-driven development—Design and efficacy study.  _Computer Applications in Engineering Education_ (2022).
  * Shporer et al. (2021) Avi Shporer, Sunny Tran, Nikhil Singh, Brandon Kates, Jayson Lynch, and Iddo Drori. 2021\.  Learning Methods for Solving Astronomy Course Problems.  _preprint_ (2021).  https://www.cs.columbia.edu/~idrori/Learning_Methods_for_Solving_Astronomy_Course_Problems.pdf
  * Singh et al. (2017) Arjun Singh, Sergey Karayev, Kevin Gutowski, and Pieter Abbeel. 2017.  Gradescope: a fast, flexible, and fair system for scalable assessment of handwritten work. In _Proceedings of the fourth (2017) acm conference on learning@ scale_. 81–88.
  * Tran et al. (2021) Sunny Tran, Pranav Krishna, Ishan Pakuwal, Prabhakar Kafle, Nikhil Singh, Jayson Lynch, and Iddo Drori. 2021.  Solving machine learning problems. In _Asian Conference on Machine Learning_. PMLR, 470–485.
  * Underwood (2022) Ted Underwood. 2022\.  Mapping the Latent Spaces of Culture.  _To appear in Startwords_ (2022).  https://doi.org/10.17613/faaa-1r21
  * Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022\.  Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In _CHI Conference on Human Factors in Computing Systems Extended Abstracts_. 1–7.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.
  * Weisz et al. (2022) Justin D Weisz, Michael Muller, Steven I Ross, Fernando Martinez, Stephanie Houde, Mayank Agarwal, Kartik Talamadupula, and John T Richards. 2022.  Better together? an evaluation of ai-supported code translation. In _27th International Conference on Intelligent User Interfaces_. 369–391.
  * Wu et al. (2021) Zhuohao Wu, Danwen Ji, Kaiwen Yu, Xianxu Zeng, Dingming Wu, and Mohammad Shidujaman. 2021\.  AI Creativity and the Human-AI Co-creation Model. In _International Conference on Human-Computer Interaction_. Springer, 171–190.
  * Xu et al. (2022) Frank F Xu, Uri Alon, Graham Neubig, and Vincent J Hellendoorn. 2022.  A systematic evaluation of large language models of code.  _arXiv preprint arXiv:2202.13169_ (2022).
  * Yasid et al. (2020) Muhammad Yasid, Gomgom T.P. Siregar, and Muhammad Ansori Lubis. 2020.  The Policy of Credit Payment Relaxation in Overcoming the Impact of Covid-19 Spread to the Economic Society in Indonesia.  _Journal of Advanced Research in Dynamical and Control Systems_ (2020).
  * Yuan et al. (2022) Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022\.  Wordcraft: Story Writing With Large Language Models. In _27th International Conference on Intelligent User Interfaces_. 841–852.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022\.  OPT: Open Pre-trained Transformer Language Models.  _arXiv preprint arXiv:2205.01068_ (2022).