# Quantum Simulation of an Open System:
A Dissipative 1+1D Ising Model

Erik Gustafsona, Michael Hiteb, Jay Hubiszc, Bharath Sambasivamc, Judah
Unmuth-Yockeya aDepartment of Theoretical Physics, Fermi National Accelerator
Laboratory, Batavia, IL 60510  bUniversity of Iowa Department of Physics and
Astronomy, Iowa City, IA 52242  cDepartment of Physics, Syracuse University,
Syracuse, NY 13244

###### Abstract

The 1+1D Ising model is an ideal benchmark for quantum algorithms, as it is
very well understood theoretically. This is true even when expanding the model
to include complex coupling constants. In this work, we implement quantum
algorithms designed for the simulation of open or complex coupling quantum
field theories on IBM devices with a focus on the measurement of the Lee-Yang
edge singularity. This feature corresponds (at large volumes) to a phase
transition, and our successful reproduction of the transition represents a
non-trivial test for current hardware and its ability to distinguish features
of interest in quantum field theories.

††preprint: APS/123-QED

##  I  Introduction

With the development of quantum algorithms designed for the simulation of
lattice quantum field theory, it is of crucial importance to perform
benchmarking tests. These tests should push the limits of current quantum
hardware, and ideally inspire confidence that features in simulation output
correspond to physics of interest, as opposed to noise in the hardware.
Quantum simulating simple, well-understood theories on contemporary noisy
devices is thus useful for two main reasons:

  * 1.

It reveals the amount of resources required to simulate them and motivates the
construction of more resource-efficient digitization methods.

  * 2.

It teaches us about the types of noise in NISQ-era devices, and enables us to
compare various error mitigation methods.

The 1+1D Ising model is very well understood theoretically, even in the
complex plane of coupling parameters, and has a rich phenomenology. It is thus
an excellent model for benchmarking, and is the model of focus in our work.

Dissipative models are particularly interesting due to their role in
characterizing the behavior of effective theories where some fraction of a
complete system has been traced out, such as in cosmological scenarios, or in
the case of a thermal bath or chemical potential. They are typically difficult
to study using classical monte-carlo methods due to the sign problem. In this
work, we consider the well-studied (see, e.g. [1, 2, 3, 4]) 1+1D transverse
quantum Ising model with imaginary longitudinal field, where a dissipative
term in an effective Hamiltonian has been included:

$\hat{H}=-\lambda\sum_{i=1}^{N_{s}-1}\hat{\sigma}^{z}_{i}\hat{\sigma}^{z}_{i+1}-h_{x}\sum_{i=1}^{N_{s}}\hat{\sigma}^{x}_{i}+i\Theta\sum_{i=1}^{N_{s}}\hat{\sigma}^{z}_{i}.$
(1)

In quantum mechanics, this sort of effective Hamiltonian can be understood as
characterizing the time evolution of an incomplete Hilbert space, and the time
evolution of a system according to this Hamiltonian must be completed into a
trace-preserving set of quantum operations.

In this paper, we focus our study on the quantum simulation of exceptional
points of this effective Hamiltonian, which are associated with the Lee-Yang
edge singularity in the infinite volume limit [4]. Exceptional points
correspond to a reduction in the effective rank of the system at which the
Hamiltonian is diagonalizable only up to Jordan-block form. The study of non-
Hermitian Hamiltonians was first made popular by Bender et. al. via the study
of PT-symmetric Hamiltonians [5, 6]. More recently, they have been
demonstrated experimentally [7].

Over the past decade, quantum computers have become more accessible through
cloud-based platforms like IBM Quantum. There has been significant progress in
the reduction of qubit and gate noise, and the improvement of coherence times;
however, the level of noise is not low enough to be able to implement quantum
error correction successfully. That being said, there have been great strides
in understanding the types of errors in hardware, converting some types to
another, and development and implementation of various error mitigation
strategies. Some well-studied examples are dynamical decoupling [8, 9],
measurement error mitigation [10, 11, 12], Pauli twirling [13, 14], zero-noise
extrapolation [15, 16, 17], mitigation circuits [18], and techniques involving
machine learning [15, 19, 20]. See [21] for a statistical analysis and scaling
of error mitigation techniques with system size. Some of these protocols are
now integrated into qiskit [22] and other platforms, making it easy to choose
the level of error mitigation while implementing models. This has enabled the
implementation of the dynamics and ground state preparation of physical
theories using longer circuit depths than previously possible.

In this work, we show that, using current error mitigation techniques, the
exceptional points of the Hamiltonian in Eq. 1 can be observed in quantum
simulation on IBM hardware for the 1D Ising chain with an imaginary
longitudinal magnetic field for small system sizes.

There exist several algorithms for simulating non-Hermitian Hamiltonians [23,
24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]. For example, quantum
imaginary time evolution (QITE) [25] is a hybrid quantum NISQ-friendly
algorithm that simulates imaginary time evolution of a Hamiltonian using a
classical optimization technique to find a unitary that performs the correct
Trotter evolution of the system. This is followed by state tomography, and
then evolution again. Close to regions of criticality, when the correlation
length diverges, tomography is very expensive, so QITE is not very effective
near such interesting critical features. There has also been some work done on
the variational quantum eigensolver [26, 27, 28] techniques for non-Hermitian
Hamiltonians. The linear combination of unitaries [29] algorithm can simulate
non-unitary dynamics up to controllable tolerance, but requires many ancillary
qubits and oracles, and is not suited for NISQ-era implementation.

There are a set of probabilistic algorithms that make use of the quantum
channels formalism [31, 32, 33, 34, 35]. In this paper, we implement one of
the algorithms in [33] (the damping channel) on real quantum hardware at IBM,
and demonstrate that the interesting features of non-unitary dynamics can be
observed even on noisy hardware. The broad idea behind this quantum channel is
to expand the system of interest using ancillary qubits. The extended system
has unitary dynamics, and careful measurements on the ancillary qubits evolves
the system of interest under the non-Hermitian Hamiltonian of interest with a
certain probability of success. Non-Hermitian Hamiltonians have been studied
using other methods on quantum hardware in Refs. [36, 37, 38, 39].

The paper is organized as follows: In section II, we review the damping
channel used to simulate non-Hermitian dynamics using ancillary qubits. In
III, we provide the Hamiltonian for the model we study in this work, and
review the unique features of non-Hermitian Hamiltonians we are looking to
observe, i.e., the exceptional points, and showcase an example of the 1D Ising
chain with an imaginary magnetic field. In IV, we briefly review the error
mitigation strategies we use, and present the results from quantum simulation
of the aforementioned modified Ising model. Finally, in V, we conclude and
offer further directions to pursue.

##  II The Damping Channel

In this section, we review the quantum channel used to perform a Trotter step
according to the anti-Hermitian part of the Hamiltonian. The full details of
the channel can be found in  Ref [33]. This channel simulates a single qubit
anti-Hermitian Hamiltonian

$\hat{k}_{j}=i\Theta\,\hat{\sigma}_{z},$ (2)

where $j$ is the index of the qubit this term is acting on. Any single qubit
anti-Hermitian Hamiltonian can be brought to this form using simple rotations.
Moreover, if the anti-Hermitian Hamiltonian of interest acts on multiple
qubits, under the Trotter approximation, it can be decomposed into a unitary
acting on the relevant qubits and the non-unitary piece acting on just a
single qubit [33]. The Kraus operator set that defines the damping channel is

$\hat{E}_{0}^{\text{DC}}=\begin{pmatrix}1&0\\\
0&\sqrt{1-\gamma}\end{pmatrix}\hskip 14.22636pt\text{and}\hskip
14.22636pt\hat{E}_{1}^{\text{DC}}=\begin{pmatrix}0&0\\\
0&\sqrt{\gamma}\end{pmatrix},$ (3)

where $\gamma=1-e^{-4\Theta\delta t}$. These Kraus operators satisfy the usual
completeness relation

$\hat{E}_{0}^{\text{DC}\dagger}\hat{E}_{0}^{\text{DC}}+\hat{E}_{1}^{\text{DC}\dagger}\hat{E}_{1}^{\text{DC}}=\mathbb{1}.$
(4)

The first Kraus operator, $\hat{E}_{0}^{\text{DC}}$ is defined such that its
action on the system qubit gives the desired anti-Hermitian dynamics, and
$\hat{E}_{1}^{\text{DC}}$ is the inevitable “quantum jump” term that comes
from the requirement of trace completion. In the language of open quantum
systems, $\hat{E}_{1}^{\text{DC}}$ is related to the Lindblad operators.

This channel acts on the system qubit, and requires an ancillary qubit to
implement, which extends the system Hilbert space to make the dynamics of the
combined system unitary. This is achieved by the controlled-$y$ rotation gate,
$CR_{y}(\varphi)$, where $\varphi=-2\arcsin{\sqrt{\gamma}}$, with the
ancillary qubit as the target. In the computational basis, this is the matrix

$\hat{U}^{\text{DC}}=\mathbb{1}\otimes\hat{E}_{0}^{\text{DC}}-i\hat{\sigma}^{y}\otimes\hat{E}_{1}^{\text{DC}}=\begin{pmatrix}1&0&0&0\\\
0&\sqrt{1-\gamma}&0&\sqrt{\gamma}\\\ 0&0&1&0\\\
0&-\sqrt{\gamma}&0&\sqrt{1-\gamma}\end{pmatrix},$ (5)

where the first term in the tensor products acts on the ancillary qubit and
the second term acts on the system qubit. After one action of this unitary,
with the ancillary qubit prepared in the $\ket{0}$ state, we get

$\hat{U}^{\text{DC}}\rho_{\text{tot}}\hat{U}^{\text{DC}\dagger}=\ket{0}\bra{0}\otimes\hat{E}_{0}^{\text{DC}}\rho\hat{E}_{0}^{\text{DC}\dagger}+\ket{1}\bra{1}\otimes\hat{E}_{1}^{\text{DC}}\rho\hat{E}_{1}^{\text{DC}\dagger}.$
(6)

Following this, a measurement is performed on the ancillary qubit. If it is
found to be in the $\ket{0}$ state, the density matrix of the system has been
acted on by the desired evolution according to $\hat{k}_{j}$ in Eq. (2)

$\frac{\hat{E}_{0}^{\text{DC}}\rho\hat{E}_{0}^{\text{DC}\dagger}}{\Tr\left(\hat{E}_{0}^{\text{DC}}\rho\hat{E}_{0}^{\text{DC}\dagger}\right)}=\frac{e^{\Theta\hat{\sigma}_{z}\delta
t}\rho e^{\Theta\hat{\sigma}_{z}\delta
t}}{\Tr\left(e^{2\Theta\hat{\sigma}_{z}\delta t}\right)}.$ (7)

The probability of measuring the ancillary qubit in the $\ket{0}$ state is
$p_{0}=\Tr\left(e^{2\Theta\hat{\sigma}_{z}\delta t}\rho\right)\approx
1-\frac{\gamma}{2}(1-r\cos\theta)$, where $(r,\theta,\phi)$ are the Bloch ball
coordinates for a single qubit. If, on the other hand, the ancillary qubit is
measure in the $\ket{1}$ state, the system is taken out of the physical
subspace. After a single Trotter step, the ancillary qubit is reset to the
$\ket{0}$ state and the quantum channel is applied again. For a large number
of time-steps, this probability approaches $e^{-4\Theta\delta t}$; it becomes
increasingly likely the system will leave the physical subspace. Nevertheless,
we will show in this work that interesting properties of the system can be
extracted at short time-scales on NISQ era devices.

##  III Model

Consider the 1D Ising model with an imaginary longitudinal magnetic field with
open boundary conditions, given by the Hamiltonian

$\hat{H}_{\text{all}}=-\lambda\sum_{i=1}^{N_{s}-1}\hat{\sigma}^{z}_{i}\hat{\sigma}^{z}_{i+1}-h_{x}\sum_{i=1}^{N_{s}}\hat{\sigma}^{x}_{i}+i\Theta\sum_{i=1}^{N_{s}}\hat{\sigma}^{z}_{i}.$
(8)

Under the Trotter approximation, the dynamics according to this Hamiltonian
can be broken up into three pieces

$\hat{U}_{\text{full}}=e^{-i\hat{H}_{\text{all}}\delta
t}\approx\hat{M}_{\text{U}}\hat{M}_{\text{NU}}\approx\hat{U}_{\text{NN}}\hat{U}_{\text{X}}\left(\hat{M}_{\text{NU}}\right),$
(9)

with

$\displaystyle\hat{U}_{\text{NN}}$ $\displaystyle=e^{i\lambda\delta
t\sum_{i=1}^{N_{s}-1}\hat{\sigma}^{z}_{i}\hat{\sigma}^{z}_{i+1}},$ (10)
$\displaystyle\hat{U}_{\text{X}}$ $\displaystyle=e^{ih_{x}\delta
t\sum_{i=1}^{N_{s}}\hat{\sigma}^{x}_{i}},$ (11)

and where the last piece is the evolution according to the anti-Hermitian
longitudinal magnetic field,

$\displaystyle\hat{M}_{\text{NU}}=e^{\Theta\delta
t\sum_{i=1}^{N_{s}}\hat{\sigma}^{z}_{i}}.$ (12)

The unitary terms are straightforward to implement using standard digital
gates. The non-unitary terms require one ancilla qubit per system qubit to
implement via the channel outlined in Sec. II. Therefore, the full unitary
implemented on the extended (system + ancillas) Hilbert space is

$\hat{U}_{\text{full}}\approx\left(\mathbb{1}\otimes\hat{U}_{\text{NN}}\right)\left(\mathbb{1}\otimes\hat{U}_{\text{X}}\right)\left(\hat{U}^{\text{DC}}\right)$
(13)

where $\hat{U}^{\text{DC}}$ acts on all system and ancillary qubits.

Figure 1: The decomposition into a basis gate set $[\text{CNOT, RZ,
}\sqrt{\text{X}},\text{X}]$ of time evolution according to (in reading order)
the nearest-neighbour term, the transverse magnetic field term, and the
unitary $U^{\text{DC}}$ acting on the extended system (with one ancillary
qubit), corresponding to the imaginary longitudinal magnetic field.

One could in-principle use Quantum Shannon Decomposition (QSD) [40] to
decompose the full time-evolution unitary $e^{-i\hat{H}\,t}$ for arbitrary $t$
for small system sizes. QSD decomposes any $N$-qubit unitary into a circuit
that has at-most $\mathcal{O}(4^{N})$ CNOT gates. However, finding the actual
decomposition is a very difficult task for larger systems.

The gate decomposition of these into a standard gate set is shown in Fig. 1.
These circuits form the building blocks of the full quantum circuit. The
bottleneck in terms of hardware error rates comes from the two-qubit CNOT
gates in NISQ-era devices. The scaling of the number of CNOT gates with the
number of system qubits per Trotter step (first-order) is shown in column 2 of
Table 1

Term | CNOTs (all) | CNOTs (alt)
---|---|---
$\hat{U}_{\text{NN}}$ | $2(N-1)$ | $2(N-1)$
$\hat{U}_{\text{X}}$ | $0$ | $0$
$\hat{U}^{\text{DC}}$ | $2N$ | $N$
Total | $2(2N-1)$ | $2(3/2N-1)$
Table 1: Scaling of number of CNOTs with system size to implement one step of
the dynamics of the Trotterized Hamiltonians in Eq. (8) (column 2) and Eq.
(14)(column 3) Figure 2: Ideal hardware topology for quantum simulation of the
Hamiltonian in Eq. (8)

We use IBM Quantum devices to simulate the Hamiltonian in this work. These
devices have fixed topologies that are not ideal for simulating the
Hamiltonian in Eq. (8) using the damping channel. The ideal topology for the
Hamiltonian is a ‘comb’ shown in Fig. 2. This mismatch necessitates the use of
the expensive SWAP gates, requiring $3$ CNOT gates each. Of course, this extra
cost is not present for very small system sizes like two or four sites, but
starts to show up for even six system sites. To avoid this extra cost (and
noise), we restrict ourselves to the case where the anti-Hermitian piece in
Eq. 8 acts only on alternate sites of the 1D Ising chain

$\hat{H}_{\text{alt}}=-\lambda\sum_{i=1}^{N_{s}-1}\hat{\sigma}^{z}_{i}\hat{\sigma}^{z}_{i+1}-h_{x}\sum_{i=1}^{N_{s}}\hat{\sigma}^{x}_{i}+i\Theta\sum_{i=1}^{\lfloor
N_{s}/2\rfloor}\hat{\sigma}^{z}_{2i}.$ (14)

This modified Hamiltonian also has the interesting features of non-Hermitian
Hamiltonians we would like to observe through quantum computation. The scaling
of the CNOT gate cost per (first-order) Trotter step for this modified
Hamiltonian is shown in column 3 in Table 1.

###  III.1 Exceptional points

Non-Hermitian Hamiltonians have interesting features that we hope to see using
the algorithm we have reviewed so far. In this subsection, we will briefly
discuss what exceptional points are and why it is still possible to see them
using noisy quantum devices.

Non-Hermitian Hamiltonians can have complex eigenvalues and non-orthonormal
eigenvectors. In some regions of parameter space, the ground state of the
Hamiltonian can be real. As one approaches an exceptional point from this
region, the ground state and the first excited state approach each other and
become degenerate on the exceptional point. Moreover, the corresponding
eigenvectors merge and the rank of the Hamiltonian is reduced, allowing it to
only be diagonalizable up to a Jordan block.

In Fig 3, we show the line of exceptional points of the Hamiltonian in Eq.
(14) for 2, 4, 6, and 8 sites. $h_{\text{x}}$ and $\Theta$ are respectively
the strength of the transverse magnetic field and the imaginary longitudinal
magnetic field that acts on even sites. When $\Theta=0$, the usual Ising model
is required. In the large system-size limit, the line of exceptional points
deviates from the $\Theta=0$ line at $h_{\text{x}}=1$, which is the well-known
phase transition of the 1D quantum Ising chain. In this limit, this line of
exceptional points is called the Lee-Yang edge.

Figure 3: The line of exceptional points for the Hamiltonian in Eq. (14) for
(from top to bottom) 2, 4, 6, 8 site systems

When you go beyond this line of exceptional points, the now degenerate ground
state splits off into the complex plane as a complex conjugate pair. The state
corresponding to the positive imaginary part of the eigenvalue becomes an
attractor for the dynamics—time evolution according to the Hamiltonian of any
state leads to this state eventually in this region of parameter space beyond
the exceptional line. This is an appealing feature when simulating on noisy
devices: Even if the noise kicks the state off the correct evolution, further
time-evolution nudges the state back towards the attractor. So, the line of
exceptional points should be a relatively robust feature visible even on noisy
devices.

The purity of the state of the system beyond the line of exceptional points
will be close to unity, whereas the for states below the line of exceptional
points will be mixed. An observable like the Rènyi entropy would be able to
detect this feature. However, the purity of the state is not accessible
without at least some level of tomography to find the off-diagonal elements of
the reduced density matrix. This would not scale well for larger systems, so
we have identified a simple diagonal observable that is sensitive to the line
of exceptional points

$\hat{\mathcal{O}}=\frac{1}{N}\sum_{i}\hat{\sigma}_{i}^{z},$ (15)

where $N$ is the number of system qubits, and the sum runs over all the system
sites.

##  IV Results

In this section, we briefly discuss the types of expected errors and how we
mitigate them. Following that, we show the results from quantum computation on
the ibmq_kolkata device.

Very broadly, there are two sources of errors in this system: The inevitable
quantum jumps stemming from the fact that the system of interest has non-
unitary dynamics and the hardware error. To deal with quantum jumps we only
post-select the counts from the quantum computation where all the measurements
of the ancillary qubits yielded $0$. Typically, this leads to ‘shot-noise’,
since the number of effective shots left over is a fraction of the total shots
of the computation. This is remedied by taking a large number of shots per
circuit, $100\,000$. Each circuit requires around $50$ seconds of quantum
runtime usage.

The leading sources of hardware errors are measurement errors (we need a total
of $N\left(\frac{1}{2}N_{t}+1\right)$ measurements, where $N_{t}$ is the
number of Trotter steps), qubit decoherence, and CNOT gate errors. In this
work we stick to a relatively light error mitigation strategy. We use the M3
(Matrix-free Measurement Mitigation) protocol [11] and dynamical decoupling
[41, 42, 43, 44, 45] to respectively tackle measurement errors and qubit
decoherence. See [46] for an analysis of different error mitigation strategies
for quantum simulations. One could add heavier error mitigation strategies to
deal with CNOT gate errors like zero-noise extrapolation, but the focus of
this work is to demonstrate the resilience of exceptional points to hardware
noise.

###  IV.1 M3 mitigation

During the measurement stage of a quantum computation, there is a probability
of readout assignment error. For a single qubit, this occurs when a $0(1)$ is
incorrectly measured as a $1(0)$. The reason this happens is because qubit
measurement is a process that takes a finite amount of time, allowing the
qubit to interact with the environment. This type of error would affect the
characterization of the state after several shots, and thus the observables
being measured. Typically, this type of error is quantified by performing
calibration measurements by state preparation and measurement to create a
‘confusion matrix’ whose entries $p_{ij}$ are the probability of incorrectly
measuring the $i^{\text{th}}$ bit-string as the $j^{\text{th}}$ bit-string.
Then, one inverts this matrix and maps the noisy probability distribution to
the ideal one. The full confusion matrix is $2^{N}\times 2^{N}$-dimensional,
requiring an exponential amount of calibration measurements. For a large
number of qubits, the matrix inversion process also becomes resource-
intensive. The M3 protocol addresses both of these issues by restricting the
confusion matrix only to the populated bitstrings in the noisy probability
distribution and only considering errors between bitstrings that are a small
Hamming distance away from each other. This makes the scaling, at most, linear
in the number of calibration measurements and the matrix inversion procedure
easier. For instance, for small enough systems, one could use the LU-
decomposition to aid in matrix inversion.

###  IV.2 Dynamical Decoupling

In current devices, the decoherence of a qubit via unwanted interaction with
the environment is a significant source of noise. This type of noise is
quantified using two decay constants- $T_{1}$ and $T_{2}$, which are
respectively associated with the $\ket{1}$ state, and the $\ket{+}$ state
decaying, thereby losing information about energy and phase. On IBM devices,
both $T_{1,2}\sim\mathcal{O}(100)\mu s$. This type of noise is suppressed
using dynamical decoupling. The broad idea is to hit idle qubits with a series
of timed quick pulses (or single-qubit gates) that move the state around in
the Bloch sphere in a way that cancels the action of the environment on
average. The assumption here is that the pulses are quicker than the time
scale in which the state of the environment changes. There are several
implementations of the dynamical decoupling, with different choices of pulse
sequences. See [9] for a detailed study comparing various implementations.

###  IV.3 Hardware Results

The topology of the hardware is shown in Fig. 4. We evolve the system
according to the first-order Trotterized version of the Hamiltonian in Eq.
(14) for 2, 4, and 6 system qubits, with $dt=0.3$. In Table 2, we show the
particular qubits of the hardware we use, the number of Trotter steps, and the
number of CNOT gates required for the circuits corresponding to various system
sizes. In Fig 5 we show the fraction of post-selected statistics that do not
have any quantum jumps for two, four, and six system sites. We provide the
qiskit notebooks for building the circuits and running them on the hardware we
used in [47].

Figure 4: Hardware topology of ibmq_kolkata $N$ | Hardware qubits | $N_{t}$ | CNOTs
---|---|---|---
$2$ | [21, 23, (24)] | $7$ | $28$
$4$ | [10, 12, 15, (18, 13, 17)] | $5$ | $50$
$6$ | [10, 12, 15, 18, 21, 23, | $4$ | $64$
| (13, 17, 24)] |  |
Table 2: Details of the quantum simulation on ibmq_kolkata for various system
sizes. The hardware qubits chosen as the ancillas are in parentheses.

Figure 5: The fraction of statistics with no quantum jumps for (from top to
bottom) 2, 4, and 6 system sites after evolution by 7, 5, and 4 trotter steps
respectively. The different colored lines correspond to values of $h_{x}$
shown in the legend

Following the evolution, we measure the diagonal observable in Eq. (15) for
$100$ equally spaced points in the $(\Theta,h_{\text{x}})$ parameter space. In
Figs. 7, 8, and 9, respectively, we show the circuits for a Trotter step, the
exact results, and quantum hardware computations on ibmq_kolkata for two,
four, and six system sites. The line of exceptional points is overlayed on top
of both the exact and hardware simulation results. Comparing the quantum
hardware results with the exact evolution, we see that the values of
observables don’t exactly match. This can be improved by using better error
mitigation techniques. That being said, it is clear that the degree of
sharpness of the exceptional point transition in the hardware computation is
similar to the exact evolution.

This is an example of what we are looking for—features of the model that are
relatively robust to hardware noise. It is a promising sign that such features
in non-Hermitian dynamics can be seen on NISQ-era devices.

Figure 6: The circuit for one Trotter step for 4 system sites (and 2
ancillas).

Figure 7: $\langle\mathcal{O}\rangle$ from exact evolution, and the error-
mitigated values from quantum computation for 2 system sites

Figure 8: $\langle\mathcal{O}\rangle$ from exact evolution, and the error-
mitigated values from quantum computation for 4 system sites

Figure 9: $\langle\mathcal{O}\rangle$ from exact evolution, and the error-
mitigated values from quantum computation for 6 system sites

##  V Conclusions and Outlook

In this paper, we studied the hardware implementation of open quantum systems
with non-Hermitian dynamics using the example of a 1D Ising model with an
imaginary longitudinal magnetic field. We used a quantum channel approach
developed in [33] where the system of interest is embedded in a larger Hilbert
space using ancillary qubits, with the enlarged system undergoing unitary
evolution. The particular channel demonstrated in this work requires one
ancillary qubit per system site that has the imaginary field acting on it. We
study the dynamics under the first-order Trotter approximation, wherein the
ancillary qubits are measured after each Trotter step. This process of
measuring qubits during a computation can be expensive for a large number of
Trotter steps since measurement of a qubit takes time. This creates a
limitation due to the coherence times of contemporary qubits. The entanglement
between the system and the ancillary qubits is set up in such a way that this
measurement result reveals if the system has evolved according to the desired
non-Hermitian Hamiltonian. The catch is that there is a small probability the
system would undergo a quantum jump, putting the system in an undesired state.
This can be combated using post-selection of results, which in-turn requires a
large number of shots to ensure sufficient statistics in the physical subspace
of interest.

There is another quantum channel, called the ‘Random Walk through time’ also
developed in [33], where the quantum jumps are controllable—they cause the
system to Trotter evolve backwards in time instead. This could be useful from
the standpoint of error mitigation in a way similar to the self-mitigation
approach in [48]. Instead of building separate mitigation circuits, one could
use some of the statistics of runs that had quantum jumps. Particularly, these
would be the runs that evolved equally forward and backwards in time and could
serve as a way to characterize the amount of noise in the circuit, provided
the hardware errors are lesser than the Trotter error.

For some systems described by non-Hermitian effective Hamiltonians, there are
interesting features in the complex parameter space. Exceptional points, which
give rise to an attractor evolution, are one such feature. We argue that as a
result, exceptional points should be relatively resistant to hardware noise,
and show for the example of a 1D Ising chain with an imaginary longitudinal
magnetic field that it can be observed with some mild error mitigation
strategies. The authors are hopeful that this would lead to the study of other
interesting physical models using quantum computers without the need for heavy
error mitigation protocols.

##  VI Acknowledgements

Jay Hubisz and Bharath Sambasivam would like to thank the IBM-Q hub at
Brookhaven National Laboratory for providing access to IBM quantum computers
on a premium plan. This project would not have been possible without access to
these valuable hardware resources. Jay Hubisz and Bharath Sambasivam are
supported in part by the U.S. Department of Energy (DOE), Office of Science,
Office of High Energy Physics, under Award Number DE-SC0009998. Bharath
Sambasivam would like to thank Dr. Muhammad Asaduzzaman and Prof. Jason
Pollack for helpful discussions. Michael Hite was supported in part by the
Department of Energy DE-SC0019139 and NSF award DMR-1747426. This manuscript
has been authored by Fermi Research Alliance, LLC under Contract No. DE-
AC02-07CH11359 with the U.S. Department of Energy, Office of Science, Office
of High Energy Physics. This work is supported by the Department of Energy
through the Fermilab Theory QuantiSED program in the area of “Intersections of
QIS and Theoretical Particle Physics.”

## References

  * Uzelac  _et al._ [1979] K. Uzelac, P. Pfeuty, and R. Jullien, Yang-lee edge singularity from a real-space renormalization-group method, Phys. Rev. Lett. 43, 805 (1979).
  * Itzykson and Drouffe [1989] C. Itzykson and J.-M. Drouffe,  _Statistical Field Theory_, Cambridge Monographs on Mathematical Physics, Vol. 1 (Cambridge University Press, 1989).
  * Uzelac  _et al._ [1980] K. Uzelac, P. Pfeuty, and R. Jullien, 1d transverse ising model in a longitudinal real or complex field, Journal of Magnetism and Magnetic Materials 15-18, 1011 (1980).
  * von Gehlen [1991] G. von Gehlen, Critical and off-critical conformal analysis of the ising quantum chain in an imaginary field, Journal of Physics A: Mathematical and General 24, 5371 (1991).
  * Bender and Boettcher [1998] C. M. Bender and S. Boettcher, Real spectra in non-hermitian hamiltonians having pt symmetry, Phys. Rev. Lett. 80, 5243 (1998).
  * Bender  _et al._ [1999] C. M. Bender, S. Boettcher, and P. N. Meisinger, PT-symmetric quantum mechanics, Journal of Mathematical Physics 40, 2201 (1999), https://pubs.aip.org/aip/jmp/article-pdf/40/5/2201/8169295/2201_1_online.pdf .
  * Guo  _et al._ [2009] A. Guo, G. J. Salamo, D. Duchesne, R. Morandotti, M. Volatier-Ravat, V. Aimez, G. A. Siviloglou, and D. N. Christodoulides, Observation of PT-Symmetry Breaking in Complex Optical Potentials, Phys. Rev. Lett. 103, 093902 (2009).
  * Hahn [1950] E. L. Hahn, Spin echoes, Phys. Rev. 80, 580 (1950).
  * Ezzell  _et al._ [2023] N. Ezzell, B. Pokharel, L. Tewala, G. Quiroz, and D. A. Lidar, Dynamical decoupling for superconducting qubits: a performance survey (2023), arXiv:2207.03670 [quant-ph] .
  * Bravyi  _et al._ [2021] S. Bravyi, S. Sheldon, A. Kandala, D. C. Mckay, and J. M. Gambetta, Mitigating measurement errors in multiqubit experiments, Physical Review A 103, 10.1103/physreva.103.042605 (2021).
  * Nation  _et al._ [2021] P. D. Nation, H. Kang, N. Sundaresan, and J. M. Gambetta, Scalable mitigation of measurement errors on quantum computers, PRX Quantum 2, 040326 (2021).
  * Nachman  _et al._ [2020] B. Nachman, M. Urbanek, W. A. de Jong, and C. W. Bauer, Unfolding quantum computer readout noise (2020), arXiv:1910.01969 [quant-ph] .
  * Bennett  _et al._ [1996a] C. H. Bennett, D. P. DiVincenzo, J. A. Smolin, and W. K. Wootters, Mixed-state entanglement and quantum error correction, Phys. Rev. A 54, 3824 (1996a).
  * Bennett  _et al._ [1996b] C. H. Bennett, G. Brassard, S. Popescu, B. Schumacher, J. A. Smolin, and W. K. Wootters, Purification of noisy entanglement and faithful teleportation via noisy channels, Phys. Rev. Lett. 76, 722 (1996b).
  * Temme  _et al._ [2017] K. Temme, S. Bravyi, and J. M. Gambetta, Error mitigation for short-depth quantum circuits, Phys. Rev. Lett. 119, 180509 (2017).
  * Li and Benjamin [2017] Y. Li and S. C. Benjamin, Efficient variational quantum simulator incorporating active error minimization, Phys. Rev. X 7, 021050 (2017).
  * Kandala  _et al._ [2019] A. Kandala, K. Temme, A. D. Córcoles, A. Mezzacapo, J. M. Chow, and J. M. Gambetta, Error mitigation extends the computational reach of a noisy quantum processor, Nature 567, 491 (2019).
  * Urbanek  _et al._ [2021] M. Urbanek, B. Nachman, V. R. Pascuzzi, A. He, C. W. Bauer, and W. A. de Jong, Mitigating depolarizing noise on quantum computers with noise-estimation circuits, Physical Review Letters 127, 10.1103/physrevlett.127.270502 (2021).
  * OBr [2023] Purification-based quantum error mitigation of pair-correlated electron simulations, Nature Physics 10.1038/s41567-023-02240-y (2023).
  * Strikis  _et al._ [2021] A. Strikis, D. Qin, Y. Chen, S. C. Benjamin, and Y. Li, Learning-based quantum error mitigation, PRX Quantum 2, 10.1103/prxquantum.2.040330 (2021).
  * Qin  _et al._ [2023] D. Qin, Y. Chen, and Y. Li, Error statistics and scalability of quantum error mitigation formulas, npj Quantum Information 9, 10.1038/s41534-023-00707-7 (2023).
  * Qiskit contributors [2023] Qiskit contributors, Qiskit: An open-source framework for quantum computing (2023).
  * Su and Li [2020] H.-Y. Su and Y. Li, Quantum algorithm for the simulation of open-system dynamics and thermalization, Physical Review A 101, 10.1103/physreva.101.012328 (2020).
  * Guimarães  _et al._ [2023] J. D. Guimarães, J. Lim, M. I. Vasilevskiy, S. F. Huelga, and M. B. Plenio, Noise-assisted digital quantum simulation of open systems (2023), arXiv:2302.14592 [quant-ph] .
  * Motta  _et al._ [2019] M. Motta, C. Sun, A. T. K. Tan, M. J. O’Rourke, E. Ye, A. J. Minnich, F. G. S. L. Brandão, and G. K.-L. Chan, Determining eigenstates and thermal states on a quantum computer using quantum imaginary time evolution, Nature Physics 16, 205 (2019).
  * Xie  _et al._ [2023] X.-D. Xie, Z.-Y. Xue, and D.-B. Zhang, Variational quantum eigensolvers for the non-hermitian systems by variance minimization (2023), arXiv:2305.19807 [quant-ph] .
  * Zhang  _et al._ [2022] D.-B. Zhang, B.-L. Chen, Z.-H. Yuan, and T. Yin, Variational quantum eigensolvers by variance minimization, Chinese Physics B 31, 120301 (2022).
  * Liu  _et al._ [2023] H. Liu, X. Yang, K. Tang, L. Che, X. Nie, T. Xin, J. Li, and D. Lu, Practical quantum simulation of small-scale non-hermitian dynamics, Physical Review A 107, 10.1103/physreva.107.062608 (2023).
  * Childs and Wiebe [2012] A. M. Childs and N. Wiebe, Hamiltonian simulation using linear combinations of unitary operations, 12, 901–924 (2012).
  * Schlimgen  _et al._ [2022] A. W. Schlimgen, K. Head-Marsden, L. M. Sager-Smith, P. Narang, and D. A. Mazziotti, Quantum simulation of open quantum systems using density-matrix purification (2022), arXiv:2207.07112 [quant-ph] .
  * Wei  _et al._ [2016] S.-J. Wei, D. Ruan, and G.-L. Long, Duality quantum algorithm efficiently simulates open quantum systems, Scientific Reports 6, 10.1038/srep30727 (2016).
  * Hu  _et al._ [2020] Z. Hu, R. Xia, and S. Kais, A quantum algorithm for evolving open quantum dynamics on quantum computing devices, Scientific Reports 10, 10.1038/s41598-020-60321-x (2020).
  * Hubisz  _et al._ [2021] J. Hubisz, B. Sambasivam, and J. Unmuth-Yockey, Quantum algorithms for open lattice field theory, Phys. Rev. A 104, 052420 (2021).
  * Hu  _et al._ [2022] Z. Hu, K. Head-Marsden, D. A. Mazziotti, P. Narang, and S. Kais, A general quantum algorithm for open quantum dynamics demonstrated with the Fenna-Matthews-Olson complex, Quantum 6, 726 (2022).
  * Head-Marsden  _et al._ [2021] K. Head-Marsden, S. Krastanov, D. A. Mazziotti, and P. Narang, Capturing non-markovian dynamics on near-term quantum computers, Phys. Rev. Res. 3, 013182 (2021).
  * Naghiloo  _et al._ [2019] M. Naghiloo, M. Abbasi, Y. N. Joglekar, and K. W. Murch, Quantum state tomography across the exceptional point in a single dissipative qubit, Nat. Phys. 15, 1232 (2019).
  * Dogra  _et al._ [2021] S. Dogra, A. A. Melnikov, and G. S. Paraoanu, Quantum simulation of parity–time symmetry breaking with a superconducting quantum processor, Commun. Phys. 4 (2021).
  * Lin  _et al._ [2022] Z. Lin, L. Zhang, X. Long, Y. ang Fan, Y. Li, K. Tang, J. Li, X. Nie, T. Xin, X.-J. Liu, and D. Lu, Experimental quantum simulation of non-hermitian dynamical topological states using stochastic schrödinger equation, npj Quantum Information 8, 10.1038/s41534-022-00587-3 (2022).
  * Powers  _et al._ [2023] C. Powers, L. Bassman Oftelie, D. Camps, and W. A. de Jong, Exploring finite temperature properties of materials with quantum computers, Sci. Rep. 13, 1986 (2023).
  * Shende  _et al._ [2006] V. Shende, S. Bullock, and I. Markov, Synthesis of quantum-logic circuits, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 25, 1000 (2006).
  * Viola and Lloyd [1998] L. Viola and S. Lloyd, Dynamical suppression of decoherence in two-state quantum systems, Phys. Rev. A 58, 2733 (1998).
  * Zanardi [1999] P. Zanardi, Symmetrizing evolutions, Physics Letters A 258, 77 (1999).
  * Vitali and Tombesi [1999] D. Vitali and P. Tombesi, Using parity kicks for decoherence control, Phys. Rev. A 59, 4178 (1999).
  * Duan and Guo [1999] L.-M. Duan and G.-C. Guo, Suppressing environmental noise in quantum computation through pulse control, Physics Letters A 261, 139 (1999).
  * Viola  _et al._ [1999] L. Viola, E. Knill, and S. Lloyd, Dynamical decoupling of open quantum systems, Phys. Rev. Lett. 82, 2417 (1999).
  * Charles  _et al._ [2023] C. Charles, E. J. Gustafson, E. Hardt, F. Herren, N. Hogan, H. Lamm, S. Starecheski, R. S. V. de Water, and M. L. Wagman, Simulating $\mathbb{Z}_{2}$ lattice gauge theory on a quantum computer (2023), arXiv:2305.02361 [hep-lat] .
  * [47] E. Gustafson, M. Hite, J. Hubisz, B. Sambasivam, and J. Unmuth-Yokey, Github-repository for Quantum simulation of non-Hermitian 1D Ising model.
  * A Rahman  _et al._ [2022] S. A Rahman, R. Lewis, E. Mendicelli, and S. Powell, Self-mitigating trotter circuits for su(2) lattice gauge theory on a quantum computer, Phys. Rev. D 106, 074502 (2022).


# A Practical Toolkit for Multilingual Question and Answer Generation

Asahi Ushio  Fernando Alva-Manchego  Jose Camacho-Collados
Cardiff NLP, School of Computer Science and Informatics, Cardiff University,
UK
<EMAIL_ADDRESS>

###### Abstract

Generating questions along with associated answers from a text has
applications in several domains, such as creating reading comprehension tests
for students, or improving document search by providing auxiliary questions
and answers based on the query. Training models for question and answer
generation (QAG) is not straightforward due to the expected structured output
(i.e. a list of question and answer pairs), as it requires more than
generating a single sentence. This results in a small number of publicly
accessible QAG models. In this paper, we introduce AutoQG, an online service
for multilingual QAG, along with lmqg, an all-in-one Python package for model
fine-tuning, generation, and evaluation. We also release QAG models in eight
languages fine-tuned on a few variants of pre-trained encoder-decoder language
models, which can be used online via AutoQG or locally via lmqg. With these
resources, practitioners of any level can benefit from a toolkit that includes
a web interface for end users, and easy-to-use code for developers who require
custom models or fine-grained controls for generation.

##  § 1 Introduction

Question and answer generation (QAG) is a text generation task seeking to
output a list of question-answer pairs based on a given paragraph or sentence
(i.e. the context). It has been used in many NLP applications, including
unsupervised question answering modeling Lewis et al. (2019); Zhang and Bansal
(2019); Puri et al. (2020), fact-checking Ousidhoum et al. (2022), semantic
role labeling Pyatkin et al. (2021), and as an educational tool Heilman and
Smith (2010); Lindberg et al. (2013). The most analysed setting in the
literature, however, has been question generation (QG) with pre-defined
answers, as this simplifies the task and makes the evaluation more
straightforward.

Despite its versatility, QAG remains a challenging task due to the difficulty
of generating compositional outputs containing a list of question and answer
pairs as shown in Figure 1, with recent works mainly relying on extended
pipelines that include several ad-hoc models Lewis et al. (2021); Bartolo et
al. (2021). These works integrate QAG into their in-house software, preventing
models to be publicly released, and their complex pipelines make them hard to
reproduce and use by practitioners.

Figure 1: An example of question and answer generation given a paragraph as
context.

In this paper, we introduce an open set of software tools and resources to
assist on the development and employment of QAG models for different types of
users. We publicly release the following resources:111All the resources except
for the datasets are released under an open MIT license, while the datasets
follow the license of their original release.

  * •

lmqg,222https://github.com/asahi417/lm-question-generation a Python package
for QAG model fine-tuning and inference on encoder-decoder language models
(LMs), as well as evaluation scripts, and a deployment API hosting QAG models
for developers;

  * •

16 models for English, and three diverse models for each of the seven
languages integrated into our library, all fine-tuned on QG-Bench Ushio et al.
(2022) and available on the HuggingFace hub Wolf et al.
(2020);333https://huggingface.co/lmqg

  * •

AutoQG (https://autoqg.net), a website where developers and end users can
interact with our multilingual QAG models.

##  § 2 Resources: Models and Datasets

Our QAG toolkit makes use of pre-existing models and datasets, fully
compatible with the HuggingFace hub. This makes our library easily extendable
in the future as newer datasets and better models emerge. In this section, we
describe the datasets (§ 2.1) and models (§ 2.2) currently available through
lmqg and AutoQG.

###  § 2.1 Multilingual Datasets

Our toolkit integrates all QG datasets available in QG-Bench Ushio et al.
(2022). QG-Bench is a multilingual QG benchmark consisting of a suite of
unified QG datasets in different languages. In particular, we integrate the
following datasets: SQuAD (English), SQuADShifts Miller et al. (2020)
(English), SubjQA Bjerva et al. (2020) (English), JAQuAD So et al. (2022)
(Japanese), GerQuAD Möller et al. (2021) (German), SberQuAd Efimov et al.
(2020) (Russian), KorQuAD Lim et al. (2019) (Korean), FQuAD d’Hoffschmidt et
al. (2020) (French), Spanish SQuAD Casimiro Pio et al. (2019) (Spanish), and
Italian SQuAD Croce et al. (2018) (Italian). QG-Bench is available through our
official lmqg HuggingFace project page and
GitHub444https://github.com/asahi417/lm-question-
generation/blob/master/QG_BENCH.md.

###  § 2.2 Models

Aiming to make QAG models publicly accessible in several languages, we used
lmqg to fine-tune LMs using QG-Bench (§ 2.1). First, we defined a pipeline QAG
model architecture consisting of two independent models: one for answer
extraction (AE) and one for question generation (QG). During training, the AE
model learns to find an answer in each sentence of a given paragraph, while
the QG model learns to generate a question given an answer from a paragraph.
To generate question-answer pairs at generation time, the AE model first
extracts answers from all the sentences in a given paragraph, and then these
are used by the QG model to generate a question for each answer. While not
directly evaluated in this paper, we also integrated other types of QAG
methods such as multitask and end2end QAG Ushio et al. (2023), all available
via the lmqg library (§ 3) as well as AutoQG (§ 5).

As pre-trained LMs, we integrated T5 Raffel et al. (2020), Flan-T5 Chung et
al. (2022), and BART Lewis et al. (2020) for English; and mT5 Xue et al.
(2021) and mBART Liu et al. (2020) for non-English QAG models. The pre-trained
weights were taken from checkpoints available in the HuggingFace Hub as below:

  * •

t5-{small,base,large}

  * •

google/flan-t5-{small,base,large}

  * •

facebook/bart-{base,large}

  * •

google/mt5-{small,base}

  * •

facebook/mbart-large-cc25

All the fine-tuned QAG models are publicly available in our official
HuggingFace Hub. While we initially integrated these models, users can easily
fine-tune others using lmqg, as we show in § 3.

##  § 3 lmqg: An All-in-one QAG Toolkit

In this section, we introduce lmqg (Language Model for Question Generation), a
Python library for fine-tuning LMs on QAG (§ 3.1), generating question-answer
pairs (§ 3.2), and evaluating QAG models (§ 3.3). Additionally, with lmqg, we
build a REST API to host QAG models to generate question and answer
interactively (§ 5). lmqg is inter-operable with the HuggingFace ecosystem, as
it can directly make use of the datasets and models already shared on the
HuggingFace Hub.

###  § 3.1 QAG Model Fine-tuning

Fine-tuning is performed via GridSearcher, a class to run encoder-deocoder LM
fine-tuning with hyper-parameter optimization (see Appendix A for more
details). For example, the following code shows how we can fine-tune T5 Raffel
et al. (2020) on SQuAD Rajpurkar et al. (2016), with the QAG model explained
in § 2.2. Since we decomposed QAG into AE and QG, two models need to be fine-
tuned independently.

⬇

from lmqg import GridSearcher

# instantiate AE trainer

trainer_ae = GridSearcher(

dataset_path="lmqg/qg_squad",

input_types="paragraph_sentence",

output_types="answer",

model="t5-large")

# train AE model

trainer_ae.train()

# instantiate QG trainer

trainer_qg = GridSearcher(

dataset_path="lmqg/qg_squad",

input_types="paragraph_answer",

output_types="question",

model="t5-large")

# train QG model

trainer_qg.train()

The corresponding dataset,
lmqg/qg_squad,555https://huggingface.co/datasets/lmqg/qg_squad has as columns:
paragraph_answer (answer-highlighted paragraph), paragraph_sentence (sentence-
highlighted paragraph), question (target question), and answer (target
answer). The input and the output to the QG model are paragraph_answer and
question, while those to the AE model are paragraph_sentence and answer. The
inputs and the outputs can be specified by passing the name of each column in
the dataset to the arguments, input_types and output_types when instantiating
GridSearcher.

###  § 3.2 QAG Model Generation

In order to generate question-answer pairs from a fine-tuned QAG model, lmqg
provides the TransformersQG class. It takes as input a path to a local model
checkpoint or a model name on the HuggingFace Hub in order to generate
predictions in a single line of code. The following code snippet shows how to
generate a list of question and answer pairs with the fine-tuned QAG model
presented in § 2.2. TransformersQG decides which model to use for each of AE
and QG based on the arguments model_ae and model.

⬇

from lmqg import TransformersQG

# instantiate model

model = TransformersQG(

model="lmqg/t5-base-squad-qg",

model_ae="lmqg/t5-base-squad-ae"

)

# input paragraph

x = """William Turner was an English

painter who specialised in watercolour

landscapes. One of his best known

pictures is a view of the city of

Oxford from Hinksey Hill."""

# generation

model.generate_qa(x)

[

(

"Who was an English painter

specialised in watercolour

landscapes?",

"William Turner"

),

(

"Where is William Turner’s

view of Oxford?",

"Hinksey Hill."

)

]

###  § 3.3 QAG Model Evaluation

Similar to other text-to-text generation tasks, we implement an evaluation
mechanism that compares the set of generated question-answer pairs
$\mathcal{\tilde{Q}}_{p}=\\{(\tilde{q}^{1},\tilde{a}^{1}),(\tilde{q}^{2},\tilde{a}^{2}),\dots\\}$
to a reference set of gold question-answer pairs
$\mathcal{Q}_{p}=\\{(q^{1},a^{1}),(q^{2},a^{2}),\dots\\}$ given an input
paragraph $p$. Let us define a function to evaluate a single question-answer
pair to its reference pair as

$\displaystyle d_{q,a,\tilde{q},\tilde{a}}$
$\displaystyle=s\big{(}t(q,a),t(\tilde{q},\tilde{a})\big{)}$ (1)
$\displaystyle t(q,a)$
$\displaystyle=``\texttt{question:}\\{q\\},\texttt{answer:}\\{a\\}\textrm{'}\textrm{'}$
(2)

where $s$ is a reference-based metric, and we compute the $F_{1}$ score as the
final metric as below:

$\displaystyle F_{1}$ $\displaystyle=2\frac{R\cdot P}{R+P}$ (3) $\displaystyle
R$ $\displaystyle={\rm
mean}\Big{(}\Big{[}\max_{(q,a)\in\mathcal{Q}_{c}}\big{(}d_{q,a,\tilde{q},\tilde{a}}\big{)}\Big{]}_{(\tilde{q},\tilde{a})\in\mathcal{\tilde{Q}}_{c}}\Big{)}$
(4) $\displaystyle P$ $\displaystyle={\rm
mean}\Big{(}\Big{[}\max_{(\tilde{q},\tilde{a})\in\mathcal{\tilde{Q}}_{c}}\big{(}d_{q,a,\tilde{q},\tilde{a}}\big{)}\Big{]}_{(q,a)\in\mathcal{Q}_{c}}\Big{)}$
(5)

Conceptually, the recall (4) and precision (5) computations attempt to “align”
each generated question-answer pair to its “most relevant” reference pair. As
with traditional precision and recall metrics, precision is aimed at
evaluating whether the predicted question-answer pairs are correct (or in this
case, aligned with the reference question-answer pairs), and recall tests
whether there are enough high-quality question-answer pairs. Thus, we refer to
the score in (3) as the QAAligned F1 score. The quality of the alignment
directly depends on the underlying metric $s$. Furthermore, the complexity of
QAAligned is no more than the complexity of the underlying metric, and
invariant to the order of generated pairs because of the alignment at
computing recall and precision.

Out-of-the-box, lmqg implements two variants based on the choice of
base_metric $s$ (used for evaluation in § 4): QAAligned BS using BERTScore
Zhang et al. (2019) and QAAligned MS using MoverScore Zhao et al. (2019). We
selected these two metrics as they correlate well with human judgements in QG
Ushio et al. (2022). Nevertheless, the choice of base_metric is flexible and
users can employ other natural language generation (NLG) evaluation metrics
such as BLEU4 Papineni et al. (2002), METEOR Denkowski and Lavie (2014), or
ROUGEL Lin (2004).

With lmqg, QAAligned score can be computed with the QAAlignedF1Score class as
shown in the code snippet below:

⬇

from lmqg import QAAlignedF1Score

# gold reference and generation

ref = [

"question: What makes X?, answer: Y",

"question: Who made X?, answer: Y"]

pred = [

"question: What makes X?, answer: Y",

"question: Who build X?, answer: Y",

"question: When X occurs?, answer: Y"]

# compute QAAligned BS

scorer = QAAlignedF1Score(

base_metric="bertscore")

scorer.get_score(pred, ref)

# compute QAAligned MS

scorer = QAAlignedF1Score(

base_metric="moverscore"

)

scorer.get_score(pred, ref)

##  § 4 Evaluation

We rely on the QAG models and datasets included in the library (see § 2). The
individual QG components of each model (i.e. the generation of a question
given an answer in a paragraph) were extensively evaluated in Ushio et al.
(2022). For this evaluation, therefore, we focus on the quality of the
predicted questions and answers given a paragraph (i.e. the specific answer is
not pre-defined). For each model, we fine-tune, make predictions and compute
their QAAligned scores via lmqg.

###  § 4.1 Results

#### Monolingual evaluation (English).

Table 1 presents the test results on SQuAD for seven English models based on
BART, T5 and Flan-T5. The QAG model based on BARTLARGE proves to be the best
aligned with gold reference question and answers among most of the metrics. As
with other QG experiments and NLP in general, the larger models prove more
reliable.

#### Multilingual evaluation.

Table 2 shows the test results of three multilingual models (mBART, mT5SMALL
and mT5BASE) in seven languages other than English, using their corresponding
language-specific SQuAD-like datasets in QG-Bench for fine-tuning and
evaluation.666The result of mBART in German is zero. Upon further inspection,
we found that the fine-tuned answer extraction module did not learn properly,
probably due to the limited size of the German dataset. T5 models, however,
proved more reliable in this case. In this evaluation, no single LM produces
the best results across the board, yet QAG models based on mT5SMALL and
mT5BASE are generally better than those based on mBART.

Model | QAAligned BS | QAAligned MS
---|---|---
BARTBASE | 92.8 / 93.0 / 92.8 | 64.2 / 64.1 / 64.5
BARTLARGE |  93.2 / 93.4 / 93.1 |  64.8 / 64.6 / 65.0
T5SMALL | 92.3 / 92.5 / 92.1 | 63.8 / 63.8 / 63.9
T5BASE | 92.8 / 92.9 / 92.6 | 64.4 / 64.4 / 64.5
T5LARGE | 93.0 / 93.1 / 92.8 | 64.7 / 64.7 / 64.9
Flan-T5SMALL | 92.3 / 92.1 / 92.5 | 63.8 / 63.8 / 63.8
Flan-T5BASE | 92.6 / 92.5 / 92.8 | 64.3 / 64.4 / 64.3
Flan-T5LARGE | 92.7 / 92.6 / 92.9 | 64.6 / 64.7 / 64.5

Table 1: QAAligned scores ($F_{1}$/$P$/$R$) on the test set of SQuAD dataset
by different QAG models, where the best score in each metric is shown in
boldface.

| Language | QAAligned BS | QAAligned MS
---|---|---|---
mT5SMALL | German |  81.2 / 80.0 / 82.5 |  54.3 / 54.0 / 54.6
Spanish | 79.9 / 77.5 / 82.6 | 54.8 / 53.3 / 56.5
French |  79.7 / 77.6 / 82.1 |  53.9 / 52.7 / 55.3
Italian | 81.6 / 81.0 / 82.3 |  55.9 / 55.6 / 56.1
Japanese | 79.8 / 76.8 / 83.1 | 55.9 / 53.8 / 58.2
Korean | 80.5 / 77.6 / 83.8 |  83.0 / 79.4 / 87.0
Russian | 77.0 / 73.4 / 81.1 | 55.5 / 53.2 / 58.3
mT5BASE | German | 76.9 / 76.3 / 77.6 | 53.0 / 52.9 / 53.1
Spanish |  80.8 / 78.5 / 83.3 |  55.3 / 53.7 / 57.0
French | 68.6 / 67.6 / 69.7 | 47.9 / 47.4 / 48.4
Italian |  81.7 / 81.3 / 82.2 | 55.8 / 55.7 / 56.0
Japanese |  80.3 / 77.1 / 83.9 |  56.4 / 54.0 / 59.1
Korean | 77.3 / 76.4 / 78.3 | 77.5 / 76.3 / 79.0
Russian | 77.0 / 73.4 / 81.2 | 55.6 / 53.3 / 58.4
mBART | German | 0 / 0 / 0 | 0 / 0 / 0
Spanish | 79.3 / 76.8 / 82.0 | 54.7 / 53.2 / 56.4
French | 75.6 / 74.0 / 77.2 | 51.8 / 51.0 / 52.5
Italian | 40.1 / 40.4 / 39.9 | 27.8 / 28.1 / 27.5
Japanese | 76.7 / 74.8 / 78.9 | 53.6 / 52.3 / 55.1
Korean |  80.6 / 77.7 / 84.0 | 82.7 / 79.0 / 87.0
Russian |  79.1 / 75.9 / 82.9 |  56.3 / 54.0 / 58.9

Table 2: QAAligned scores ($F_{1}$/$P$/$R$) on the test set of QG-Bench by
different QAG models, where the best score in each language is shown in
boldface.

Gold | BARTB | BARTL | T5S | T5B | T5L | Flan-T5S | Flan-T5B
---|---|---|---|---|---|---|---
4.9 | 4.1 | 4.2 | 4.2 | 4.3 | 4.3 | 4.2 | 4.3

Table 3: Average number of generated question and answer pairs per paragraph
on the test set of SQuAD by different QAG models.

Language | Gold | mT5SMALL | mT5BASE | mBART
---|---|---|---|---
German | 4.6 | 10.1 | 8.4 | 0.0
Spanish | 1.3 | 4.6 | 4.8 | 4.7
French | 1.3 | 4.9 | 3.6 | 5.4
Italian | 3.8 | 4.7 | 4.6 | 2.5
Japanese | 1.3 | 6.6 | 6.8 | 3.6
Korean | 1.3 | 6.7 | 6.3 | 6.7
Russian | 1.3 | 4.8 | 4.9 | 4.7

Table 4: The averaged number of generated question and answer pairs per
paragraph on the test set of QG-Bench for each language.

###  § 4.2 Number of Generated Questions and Answers

Table 3 and Table 4 show the averaged number of generated question-answer
pairs and compare it to the number in the gold dataset. For English, there is
a small difference across all QAG models, with all generating fewer pairs than
the gold dataset, but with a limited margin. For other languages, however,
there are clear differences across QAG models, with the numbers of question-
answer pairs generated by the QAG models always being larger than those in the
gold dataset. When comparing the number of pairs generated by the QAG models
with their QAAligned scores, in languages such as German, Spanish, and Korean,
QAG models that generated a larger number question-answer pairs achieved
higher scores, not only recall-wise but also generally for F1.

##  § 5 AutoQG

Finally, we present AutoQG (https://autoqg.net), an online QAG demo where
users can generate question-answer pairs for texts in eight languages
(English, German, Spanish, French, Italian, Japanese, Korean, Russian) by
simply providing a context document. We deploy the QAG models described in §
2. In addition to the features described above, the online demo shows
perplexity computed via lmppl,777https://pypi.org/project/lmppl a Python
library to compute perplexity given any LM architecture. This feature helps us
provide a ranked list of generation to the user. Although we can compute
perplexity for non-English generations based on the QAG models in each
language, it entails large memory requirements on the the hosting server. As
such, we compute a lexical overlap between the question and the document as a
computationally-light alternative to the perplexity, which is defined as:

$\displaystyle 1-\frac{|q\cap p|}{|q|}$ (6)

where $|\cdot|$ is the number of characters in a string, and $q\cap p$ is the
longest sub-string of the question $q$ matched to the paragraph $p$.

Figure 2: A screenshot of AutoQG with an example of question and answer
generation over a paragraph. Figure 3: A screenshot of AutoQG with an example
of question and answer generation over a paragraph in Japanese. Figure 4: A
screenshot of AutoQG when an answer is specified by the user.

Figure 2 and Figure 3 show examples of the interface with English and Japanese
QAG, where there is a tab to select QAG models, language, and parameters at
generation including the beam size and the value for nucleus sampling Holtzman
et al. (2020). Optionally, users can specify an answer and generate a single
question on it with the QG model, as shown in Figure 4. A short introduction
video to AutoQG is available at https://youtu.be/T6G-D9JtYyc.

##  § 6 Conclusion

In this paper, we introduced lmqg, a Python package to fine-tune, evaluate and
deploy QAG models with a few lines of code. The library implements the QAG
task as an efficient integration of answer extraction and question generation,
and includes automatic reference-based metrics for model evaluation. Finally,
we showcase AutoQG, an online demo where end users can benefit from QAG models
without any programming knowledge. AutoQG enables the selection of features
going from different models and languages to controlling the diversity of the
generation.

## Limitations

The focus on this paper was introducing software to make QAG models available
to as many practitioners as possible, but there are a couple of limitations in
the models and evaluation metrics we proposed.

First, our released QAG models assume a paragraph up to around 500 tokens as
an input, and longer documents can not be directly fed into the models.
Additionally, the released QAG models were fine-tuned on questions that
require one-hop reasoning only, so they are unable to generate multi-hop
reasoning.

Second, the QAAligned score is a framework to extend any NLG metric to match
the prediction to the reference when they are different in size, where we
employed two well-established metrics (BERTScore and MoverScore) as underlying
metrics. Since those underlying metrics are already proven to be effective
Zhang et al. (2019); Zhao et al. (2019); Ushio et al. (2022), we have not
conducted any human annotation for QAG specifically. Nonetheless, an extended
human evaluation could help provide more insights on other limitations of the
model not detected by the automatic evaluation.

## Ethics Statement

While the QAG models are fine-tuned on pre-trained language models, which are
known to contain some toxic contents Schick et al. (2021), an internal check
does not reveal any toxic generation. However, there is a potential risk that
the QAG model could generate toxic text due to the underlying LMs.

## References

  * Bartolo et al. (2021) Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021.  Improving question answering model robustness with synthetic adversarial data generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8830–8848, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Bjerva et al. (2020) Johannes Bjerva, Nikita Bhutani, Behzad Golshan, Wang-Chiew Tan, and Isabelle Augenstein. 2020.  SubjQA: A Dataset for Subjectivity and Review Comprehension.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5480–5494, Online. Association for Computational Linguistics.
  * Casimiro Pio et al. (2019) Carrino Casimiro Pio, Costa-jussa Marta R., and Fonollosa Jose A. R. 2019.  Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question Answering.  _arXiv e-prints_ , page arXiv:1912.05200v1.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Croce et al. (2018) Danilo Croce, Alexandra Zelenanska, and Roberto Basili. 2018.  Neural learning for question answering in italian.  In _AI*IA 2018 – Advances in Artificial Intelligence_ , pages 389–402, Cham. Springer International Publishing.
  * Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014.  Meteor universal: Language specific translation evaluation for any target language.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 376–380, Baltimore, Maryland, USA. Association for Computational Linguistics.
  * d’Hoffschmidt et al. (2020) Martin d’Hoffschmidt, Wacim Belblidia, Quentin Heinrich, Tom Brendlé, and Maxime Vidal. 2020.  FQuAD: French question answering dataset.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1193–1208, Online. Association for Computational Linguistics.
  * Efimov et al. (2020) Pavel Efimov, Andrey Chertok, Leonid Boytsov, and Pavel Braslavski. 2020.  Sberquad–russian reading comprehension dataset: Description and analysis.  In _International Conference of the Cross-Language Evaluation Forum for European Languages_ , pages 3–15. Springer.
  * Heilman and Smith (2010) Michael Heilman and Noah A. Smith. 2010.  Good question! statistical ranking for question generation.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 609–617, Los Angeles, California. Association for Computational Linguistics.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lewis et al. (2019) Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019.  Unsupervised question answering by cloze translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4896–4910, Florence, Italy. Association for Computational Linguistics.
  * Lewis et al. (2021) Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021.  PAQ: 65 million probably-asked questions and what you can do with them.  _Transactions of the Association for Computational Linguistics_ , 9:1098–1115.
  * Lim et al. (2019) Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019.  Korquad1. 0: Korean qa dataset for machine reading comprehension.  _arXiv preprint arXiv:1909.07005_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lindberg et al. (2013) David Lindberg, Fred Popowich, John Nesbit, and Phil Winne. 2013.  Generating natural language questions to support learning on-line.  In _Proceedings of the 14th European Workshop on Natural Language Generation_ , pages 105–114, Sofia, Bulgaria. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Miller et al. (2020) John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. 2020.  The effect of natural distribution shift on question answering models.  In _International Conference on Machine Learning_ , pages 6905–6916. PMLR.
  * Möller et al. (2021) Timo Möller, Julian Risch, and Malte Pietsch. 2021.  Germanquad and germandpr: Improving non-english question answering and passage retrieval.
  * Ousidhoum et al. (2022) Nedjma Ousidhoum, Zhangdie Yuan, and Andreas Vlachos. 2022.  Varifocal question generation for fact-checking.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2532–2544, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Puri et al. (2020) Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020\.  Training question answering models from synthetic data.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5811–5826, Online. Association for Computational Linguistics.
  * Pyatkin et al. (2021) Valentina Pyatkin, Paul Roit, Julian Michael, Yoav Goldberg, Reut Tsarfaty, and Ido Dagan. 2021.  Asking it all: Generating contextualized questions for any semantic role.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1429–1441, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Schick et al. (2021) Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021.  Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP.  _Transactions of the Association for Computational Linguistics_ , 9:1408–1424.
  * So et al. (2022) ByungHoon So, Kyuhong Byun, Kyungwon Kang, and Seongjin Cho. 2022.  Jaquad: Japanese question answering dataset for machine reading comprehension.  _arXiv preprint arXiv:2202.01764_.
  * Ushio et al. (2022) Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados. 2022.  Generative language models for paragraph-level question generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , Abu Dhabi, U.A.E. Association for Computational Linguistics.
  * Ushio et al. (2023) Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados. 2023.  An empirical comparison of lm-based question and answer generation methods.  In _Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics_ , Toronto, Canada. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Zhang and Bansal (2019) Shiyue Zhang and Mohit Bansal. 2019.  Addressing semantic drift in question generation for semi-supervised question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2495–2509, Hong Kong, China. Association for Computational Linguistics.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Grid Search

Figure 5: An overview of the hyper-parameter search implemented as
GridSearcher.

To fine-tune LMs on QAG, one can use the GridSearcher class of lmqg, which
performs LM fine-tuning with a two-stage optimization of hyper-parameter, a
set of parameters to be used at fine-tuning such as learning rate or batch
size, as described in Figure 5. Let us assume that we want to find an optimal
combination of the learning rate and random seed from a list of candidates
[1e-4,1e-5] and [0,1] for learning rate and random seed respectively on QG as
an example. We also assume a training and a validation dataset to train a
model on the task and an evaluation score that reflects a performance of a
model (eg. BLEU4Papineni et al. (2002)), and we define a search-space as a set
including all the combinations of those candidates, i.e. {(1e-4, 0), (1e-4,
1), (1e-5, 0), (1e-5, 1)}. The goal of the GridSearcher is to find the best
combination to train a model on the training dataset for the target task over
the search-space with respect to the evaluation score computed on the
validation dataset.

Brute-force approach such as to train model over every combination in the
search-space can be a highly-inefficient, so GridSearcher employs a two-stage
search method to avoid training for all the combinations, while being able to
reach to the optimal combination as possible. To be precise, given an epoch
size $L$ (epoch), the first stage fine-tunes all the combinations over the
search-space, and pauses fine-tuning at epoch $M$ (epoch_partial). The top-$K$
combinations (n_max_config) are then selected based on the evaluation score
computed over the validation dataset, and they are resumed to be fine-tuned
until the last epoch. Once the $K$ chosen models are fine-tuned at second
stage, the best model is selected based on the evaluation score, which is kept
being fine-tuned until the evaluation score decreases.

The dataset for training and validation can be any datasets shared in the
HuggingFace Hub, and one can specify the input and the output to the model
from the column of the dataset by the arguments input_types and output_types
at instantiating GridSearcher. For example, the following code shows how we
can fine-tune T5 Raffel et al. (2020) on question generation, a sub-task of
QAG, with SQuAD Rajpurkar et al. (2016), where the dataset lmqg/qg_squad is
shared at https://huggingface.co/datasets/lmqg/qg_squad on the HuggingFace
Hub, which has columns of paragraph_answer, that contains a answer-highlighted
paragraph, and question, which is a question corresponding to the answer
highlighted in the paragraph_answer. We choose them as the input and the
output to the model respectively by passing the name of each column to the
arguments, input_types and output_types.

⬇

from lmqg import GridSearcher

# instantiate the trainer

trainer = GridSearcher(

dataset_path="lmqg/qg_squad",

input_types="paragraph_answer",

output_types="question",

model="t5-large",

batch_size=128,

epoch=10,

epoch_partial=2,

n_max_config=3,

lr=[1e-4,1e-5],

random_seed=[0,1])

# train model

trainer.train()