# Applying Magnetic Curvature to to MMS data to identify thin current sheets
relative to tail reconnection

###### Abstract

Magnetic reconnection X-lines have been observed to be more common duskward of
midnight. Thin current sheets have also been postulated to be a necessary
precondition for reconnection onset. We take advantage of the MMS tetrahedral
formation during the 2017–2020 MMS tail seasons to calculate the thickness of
the cross-tail neutral sheet relative to ion gyroradius. While a similar
technique was applied to Cluster data, current sheet thickness over a broader
range of radial distances has not been robustly explored before this study. We
compare this to recent theories regarding mechanisms of tail current sheet
thinning and to recent simulations. We find MMS spent more than twice as long
in ion-scale thin current sheets in the pre-midnight sector than post-
midnight, despite nearly even plasma sheet dwell time. The dawn-dusk asymmetry
in the distribution of Ion Diffusion Regions, as previously reported in
relation to regions of thin current sheets, is also analyzed.

Geophysical Research Letters

Space Science Center, University of New Hampshire, Durham, New Hampshire
Microsoft Corporation, Redmond, Washington

Anthony Jason<EMAIL_ADDRESS>
Geomagnetic tail current sheet thickness during MMS 2017–2020 tail seasons
estimated using magnetic field line curvature

Location of tail current sheets with thickness at or below ion gyro-scale
compared to location of reconnection-related Ion Diffusion Regions

Observations compared to recent PIC simulations

## Plain Language Summary

Magnetic reconnection is an important mechanism for energy transfer in the
magnetosphere. In order for reconnection to begin, however, the reconnecting
current sheet must first become very thin. In the geomagnetic tail
reconnection and related phenomena have been observed closer to dusk than dawn
on the nightside, although the reasons for this have not been clearly
understood. Recent simulations of the geomagnetic tail suggest that the
central current sheet in the tail should be thinner pre-midnight than post-
midnight, possibly explaining why reconnection happens more often on the pre-
midnight than the post-midnight sector. We use nineteen months of MMS data in
the tail, comprising the tail seasons of four years from 2017–2020, to
estimate the thickness of the tail neutral sheet relative to relevant ion
scales from dawn flank to dusk flank and both closer to and further away from
the Earth than has been done in the past. We then compare the thickness we
measure with the simulation predictions and with the location of previously
identified reconnection locations in the same time period.

##  1 Introduction

In Dungey’s model of the open magnetosphere (1961, 1963), energy stored in the
interplanetary magnetic field (IMF) is transferred to kinetic energy of the
magnetospheric plasma in a process called magnetic reconnection. On the
dayside, reconnection is between the IMF and that of the closed terrestrial
magnetospheric field lines. In the tail, where the flux transported from the
dayside loads and thins the tail current sheet (CS, embedded in the plasma
sheet), reconnection closes the open flux and returns it to the dayside,
leading to a two-cell plasma convection (see review by Cowley, 1981). This
model has received abundant observational support.

Phenomena associated with reconnection in the geomagnetic tail such as auroral
substorms, dipolarization fronts, and bursty-bulk flows (BBFs) as well as in
situ observations of reconnection in the tail have provided an important part
of this support. However, the locations of these phenomena have also shown a
significant asymmetry in the dawn-dusk direction, being more common on the
dusk-side of midnight by significant ratios (Nagai et al., 1998; Posch et al.,
2007; Xiao et al., 2017). A statistical study using THEMIS (Sibeck and
Angelopoulous, 1998) was performed by Imber et al. (2011) and showed that 81%
of flux ropes and traveling compression regions associated with reconnection
in the magnetotail were found in the dusk sector. Rogers, Farrugia, and
Torbert (2019) showed that reconnection regions (ion diffusion regions or
IDRs) during the 2017 MMS tail season (May – September) were preferentially
observed in situ on the dusk-side of midnight by a far larger ratio (91.7%
pre-midnight, 8.3% post-midnight) than could be accounted for by differences
in spacecraft dwell time on either side of midnight (56.5% pre-midnight, 43.5%
post-midnight). This suggests that the observed asymmetrical distribution of
reconnection and related phenomena is due to underlying physical processes and
not observational bias.

Reconnection theory (Sönnerup, 1979) and simulations (e.g. Birn, 1980; Liu et
al., 2019) suggest that a thin current sheet, i.e. one with a thickness on the
order of an ion inertial length ($d_{i}$) or less, is necessary for
reconnection onset to occur. Ion-scale thin current sheets in the neighborhood
of tail reconnection sites have been anecdotally observed (e.g. Runov et al.,
2003), which supports a correlation between extended regions of thin current
sheets and magnetic reconnection in the geomagnetic tail. The distribution of
ion-scale current sheets within the central tail plasma sheet is thus
pertinent to the question of reconnection location in the tail.

Attempts to estimate the thickness of the tail current sheet have been
occasionally made using single-spacecraft measurement techniques (e.g.
Artemyev et al., 2011; S. Lu et al., 2019). Many of these techniques relied
upon a ratio of measurements of the magnetic field and particle current
density. These were sometimes made at substantially different points in time,
which assumes that the current sheet being measured is essentially a quiet,
Harris-type current sheet (Harris, 1962) throughout the measurement period.
However, reconnection and related substorm phenomena often occur during times
of a disturbed geomagnetic field, violating the assumption of a Harris-like
current sheet.

Rong et al. (2011) utilized the four spacecraft of the Cluster mission in
tetrahedral formation to calculate the radius of curvature ($R_{C}$) of the
magnetic field at the barycentre of the fleet:

$R_{C}=\frac{1}{|(\hat{b}\cdot\nabla)\hat{b}|}$ (1)

where $\hat{b}$ is the unit vector $\vec{B}/|\vec{B}|$. They then estimated
the thickness of the neutral sheet by scaling the radius of curvature with the
current sheet tilt, as described by Shen et al. (2008). This method was
superior to previous estimates using single-spacecraft techniques. A
statistically significant dawn-dusk asymmetry in current sheets thinner than
1000km was found, with thinner currents sheets more common duskward of
midnight. However, due to the nature of its near-polar orbit, Cluster only
encountered the tail current sheet at a radial distance approximately 20 Earth
radii ($R_{E}$) from the Earth within a narrow band $\approx 2R_{E}$ wide.
Current sheet thickness in the geomagnetic tail has not been explored using
such robust techniques at other radial distances before our current study.

Current sheet thinning has also been addressed in numerical simulations.
Recent hybrid (Lu et al., 2016) and particle-in-cell (PIC) simulations (Lu et
al., 2018) suggest a mechanism for preferential thinning of the tail current
sheet in the pre-midnight sector. In both simulations, external drivers cause
a global thinning of the tail current sheet to approximately ion scales, at
which point partial demagnetization of ions drives charge separation from the
still frozen-in electrons, leading to Hall electric fields. The duskward
cross-tail current of the tail current sheet is then enhanced by the
$\vec{E}\times\vec{B}$ drifting of frozen-in electrons and diamagnetic drift
of partially demagnetized ions, leading to progressive thinning of the current
sheet on the pre-midnight. These simulations suggest the asymmetric thinning
should be robust across a broad range of radial distance.

The MMS mission (Burch et al. 2015) launched in 2015 typically flies in a
tetrahedron formation that is capable of the 4-point measurements necessary
for curvature calculation. We utilize a technique particularly suited to
identifying current sheets which may be preferentially thinned as suggested by
simulations. Buc̈hner and Zelenyi (1989) described a scalar parameter $K$ to
identify whether the local magnetic field could support adiabatic motion of
charged particles in a plasma. We adapt $K$ to apply this test against the
thermal ion population in the vicinity of the MMS observatories:

$K_{i}=\sqrt{\frac{R_{C}}{\rho_{g,i}}}$ (2)

where $\rho_{g,i}=\frac{\sqrt{2m_{i}T_{\perp,i}k_{b}}}{|q_{i}||\vec{B}|}$ is
the thermal ion gyroradius. Where $K_{i}<1$ the gyroradius of the ions is
larger than the radius of curvature of the local magnetic field, implying that
the majority ions will not remain frozen-into the magnetic field and adiabatic
motion of those ions cannot be supported. Under such conditions Hall electric
fields may form between the demagnetized ions and the still-frozen-in
electrons, supporting the process described by Lu et al.

We calculate the value of $K_{i}$ throughout the geomagnetic tail near the
plasma sheet for the entire 2017–2020 MMS tail seasons (Phases 2b, 3b, 4b, and
5b). As $K_{i}$ is a measure of the relative thickness of the neutral sheet,
recalling that $R_{C}$ is proportional to neutral sheet thickness (see section
3.8 Rong et al. 2011), we use these data to test recent theories regarding
mechanisms for causing thinning of the tail current sheet by comparing MMS
observations to predictions made in simulations. We also investigate the
relationship between thin (ion gyroradius scale) current sheet locations and
the occurrence of Ion Diffusion Regions (IDRs) associated with reconnection.

##  2 Instruments and methods

The MMS spacecraft measure electric and magnetic fields using the FIELDS
instrument suite (Torbert et al. 2016). The analog and digital fluxgate
magnetometers (AFG/DFG) measure magnetic fields in the frequency range from DC
up to 64 Hz (Russell et al. 2016). Level 2 fluxgate magnetometer (FGM) data of
version 5.86 and higher (highest available as of submission) were used
throughout this study.

The Fast Plasma Instrument (FPI) provides MMS with high cadence electron and
ion distributions in the energy/charge range of 10 eV/q up to 30 keV/q. Each
MMS satellite is equipped with eight FPI spectrometers which, combined with
electrostatic control of the field-of-view, allows FPI to sample the full
electron and ion distributions (Pollock et al. 2016). It is important to note
that core ion distributions can extend beyond the range of FPI, meaning that
actual ion temperatures may be higher than what is calculated using FPI data.
Level 2 FPI ion moments of version 3.3.0 were used throughout this study.

Positions of the individual spacecraft in the MMS fleet are provided using
Magnetic Ephemeris and Coordinates (MEC) data products (Morley, 2015) and are
calculated using the LANLGeoMag suite (Henderson et al. 2018). All instrument
data used in this study is available from the MMS Science Data Center
(https://lasp.colorado.edu/mms/sdc). Level 2 fast survey data was used
throughout this study. Calculations of the magnetic field line curvature were
made using the mms-curvature library and is publicly available
(https://github.com/unh-mms-rogers/mms-curvature).

In order to ensure that the formation of the MMS fleet was appropriate for the
calculation of spatial gradients a minimum value of the Tetrahedron Quality
Factor (TQF: Fusilier et al. 2016) was required where $TQF\geq 0.8$. For
similar reasons, this survey was performed only on data collected while the
MMS fleet was at least 8 Earth radii ($R_{E}$) from the Earth to avoid
deformations of the regular tetrahedron as the fleet approached perigee.
Observations in this study were also limited to regions with a measured ion
density of $\rho_{i}\geq 0.05cc^{-1}$ to ensure observations were made in the
plasma sheet (Rogers, Farrugia, Torbert, 2019; Raj et al. 2002; Baumjohann,
1993).

Figure 1: The total amount of time in seconds spent by MMS in a TCS, under
conditions outlined in the text, in sectors of dimension $0.5R_{E}\times
0.5R_{E}$ (a, b, c) and Magnetic Local Time (MLT) (d). a) Distribution of TCS
dwell times in the GSM X–Y plane, with times summed over the GSM Z axis. b)
TCS dwell times in the GSM X –Z plane, with times summed over the GSM Y axis.
c) TCS dwell times in the GSM Z – Y plane, with times summed over the GSM X
axis. d) TCS dwell times across all radial distances by MLT location. Magenta
circles represent Ion Diffusion Region locations (see text)

##  3 Observations and Analysis

Figure 1 shows the distribution of ion-scale thin current sheets as measured
by MMS over the combined 2017–2020 tail seasons where data from all four
spacecraft were available ($\approx$ 550 days). Colors represent the amount of
time which MMS spent in each region with a value of $K<1.0$, i.e. the dwell
time of MMS in a thin current sheet (TCS). The effects of orbital bias on
these dwell time measurements are small as MMS spent approximately equal time
in the plasma sheet on either side of midnight: 51.7% pre-midnight, 48.3%
post-midnight. IDRs previously identified by Rogers, Farrugia, and Torbert
(2019) from the 2017 season (Phase 2b), as well as additional IDRs identified
using the same technique, have been overlaid as magenta circles on the TCS
dwell times in Figure 1. All of these IDRs are found in regions where MMS
spent at least a moderate amount of time in a TCS. It should be noted that
some IDR markers in Figure 1 and following figures totally obscure the dwell
time indicator for the region where they are located. A listing of all
identified IDRs shown here is provided in the supplementary materials.

Figure 1d shows the total TCS dwell time as a function of Magnetic Local Time
(MLT). This sub-figure is comparable with similar plots showing the global MLT
distribution of other substorm-related phenomena such at Pi1B pulsations
(Posch et al., 2007, Fig.12) and dipolarization fronts (Xiao et al., 2017,
Fig. 6), all of which show a strong preference for substorm-related activity
to occur duskward of midnight. The sharp delineation at midnight shown in
Figure 1d is similar to distributions of substorm related phenomena reported
in the previous studies mentioned.

Figure 2 shows the same TCS data derived from the parameter $K$ but here
normalized by the total dwell time MMS spent in the tail plasma sheet, as
indicated by the measured ion density $n_{i}>0.05cc^{-1}$. As expected, the
majority of the time spent by MMS in the plasma sheet was not near an ion-
scale TCS, as indicated by the bulk of the distribution reflecting a ratio of
TCS time to plasma sheet dwell time of much less than one. The distribution of
known IDRs during the same time frame is also laid over the map of normalized
TCS dwell times in Figure 2.

Figure 2: Dwell time MMS spent in a TCS normalized by the amount of time MMS
spent in the plasma sheet. Regions and projection are in the style of Figure
1a.

The total time spent in a TCS is significantly higher on the pre-midnight of
the tail ($187.8hrs$) than the post-midnight ($77.68hrs$) (see Figure 1d).
While MMS spent an approximately equal amount of time in the plasma sheet on
both the pre- and post-midnight sectors ($51.7\%/48.3\%$), this contrasts with
the uneven amount of time spent in a thin current sheet by MMS; $70.7\%$ of
total time spent in TCSs was in the pre-midnight sector versus $29.3\%$ post-
midnight. The majority of the time spent in a TCS on the post-midnight is
found near apogee across all seasons (see Fig.1) where spacecraft velocity was
lowest and the bulk of dwell time each orbit was spent regardless of other
factors.

The distribution of time spent in a TCS pre-midnight is far more varied in
radial distance and is not confined to the apogee of each orbit. Figure 3
shows the dwell time of MMS in a TCS relative to dwell time in moderate-to-
high ion density as a function of MLT in bands of radial distance, each
$2R_{E}$ wide. The center of the relative TCS dwell time distribution is at
$\approx$21 MLT at smaller radial distances (Fig. 3a,b). The distribution both
broadens towards midnight as MMS increases in radial distance (Fig. 3c,d),
although the peak remains more duskward. The large relative TCS dwell times at
the dawn and dusk flanks (Fig.2) are interpreted as encounters with the flank
magnetopause where current sheets and increased ion density are expected. Sub-
figure $3c)$ represents the approximate region of the geomagnetic tail sampled
by Cluster as in Rong et al. (2011).

The asymmetry in the locations of Ion Diffusion Regions (IDRs; magenta
circles, Figure 1) associated with magnetic reconnection is even more
pronounced than that of relative TCS dwell time. 19 IDRs were confidently
identified across all four seasons on the dusk-side of midnight, while only
six were observed on the dawn-side, equating to 76.9% of reconnection events
observed duskward of midnight and 23.1% dawnward. In Figure 2 we see that the
majority of previously identified IDRs lie not only in areas where MMS spent a
great deal of time in ion-scale TCS, but also where the ratio of TCS dwell
time to total plasma sheet dwell time was greatest. This supports the
intuitive interpretation that a thinner current sheet is more likely to
support reconnection, and that reconnection is more likely where thin current
sheets are more common. However, the long periods of time spent in both the
plasma sheet and thin current sheet where no reconnection was observed, such
as -16$R_{E}\hat{x}$, 11$R_{E}\hat{y}$ in Figure 2, indicate that an ion-scale
TCS is an insufficient condition for reconnection.

Figure 3: Dwell time MMS spent in a TCS by MLT for narrow ($2R_{E}$ wide)
bands across the geomagnetic tail, normalized by the total amount of time MMS
spent in the plasma sheet in that same MLT $\times$ R sector. e) Is the
approximate band which has been studied by the Cluster mission (e.g. Rong et
al. 2011).

##  4 Discussion

A feature of tail reconnection that stand in observations is that IDRs and
related reconnection phenomena occurs preferentially in the pre-midnight
sector (Eastwood et al. 2010, Nagai et al. 2013, Genestreti et al. 2013). The
explanation for this has been glossed over somewhat since the question of
observational bias due to orbital variations was not often addressed (see
discussion in Imber et al. 2011). If a spacecraft spends more time at dusk
then one supposes its chances of seeing IDRs or other phenomena is increased
relative to other regions, all other things being equal. Thus while studies
based on previous missions showed a dawn-dusk asymmetry in observed
reconnection-related features, we could not be sure it was not due to an
observational bias. This question was addressed by Rogers, Farrugia, and
Torbert (2019) who confirmed that the asymmetric distribution of observed IDRs
by MMS was not a function of observational bias but is a result of magnetotail
physics.

The strong asymmetry in typical current sheet thickness is not entirely
surprising. In addition to previous studies which observed a similar asymmetry
but were more limited in extent (Rong et al. 2011) or utilized more indirect
methods of calculating current sheet thickness (e.g. S. Lu et al. 2019),
mechanisms for the source of this asymmetry have also been mooted. Lu et al.
(2018) and Pritchett and Lu (2018) have hypothesized that the cause of this
dusk-preference was preferential thinning on the pre-midnight of the tail due
an externally-driven convective electric field and enhanced by Hall electric
fields (normal to the current sheet) which formed as the current sheet
approached ion scales. Comparing our observations to this model we find the
ratio of dwell time in an ion-scale TCS, 70.7% pre-midnight to 29.3% post-
midnight; a 2 to 1 ratio with twice as much time spent encountering an ion-
scale current sheet pre-midnight as post-midnight, qualitatively supports the
model of Lu et al. A comparison of our Figures 2 and 3 to Figure 2 in Lu et
al. (2018) is even more encouraging as the radial variations in our
observations are qualitatively similar to those shown in the PIC simulations.

Figure 4: Plots of the proportion of the time MMS spent in a TCS relative to
total time spent in the plasma sheet as well as the location of identified
IDRs, both as a function of MLT, for the 2017 tail season (a) and the combined
2017–2020 tail seasons (b).

We can also see the pre-/post-midnight TCS asymmetry clearly in Figure 4 which
plots the time MMS spent in a TCS as a percentage of the total time spent in
the plasma sheet, along with the number of IDRs identified in each MLT region.
It is unsurprising that the number of IDRs identified increases with the
greater proportional time spent in a TCS, but less expected is the greater
duskward extent of the TCS than observed IDRs.

Recent 3D PIC simulations (Liu et al. 2019) suggest that the near-midnight
preference of observed reconnection may be due to the effects of Hall
reconnection in three dimensions. Liu et al. suggest that reconnection on a
given thin current sheet will be suppressed on the side of the current sheet
opposite the direction of net current flow for a region on the scale of 10s of
ion inertial lengths (see Fig. 7, Liu et al., 2019). In the geomagnetic tail,
with a net dawnward flow, this would lead to suppression of reconnection onset
on the duskward side of an existing thin current sheet. As we have observed,
ion-scale TCSs capable of supporting possible reconnection are twice as likely
to be found on the pre-midnight of midnight as on the post-midnight. Thus
while reconnection is more likely to occur duskward of midnight in the tail
due to ion-scale TCSs occurring more frequently there, it is more likely to be
suppressed as the TCS extends further duskward away from midnight (see Fig. 9,
Liu et al., 2019). This would have the practical effect of concentrating
reconnection onset in the region at and near midnight in the pre-midnight
region. The location of nearly half of all identified IDRs within the region
of 22-24MLT supports this behavior.

Also of note is the duskward expansion of both TCS and IDR locations over the
four tail seasons of this study. 2017-2019 were in the declining phase of the
solar cycle. Figure 4a shows the distribution of proportional time in a TCS as
well as IDR distribution in MLT for the 2017 tail season with a 2017 annual
average Dst of $-8.09nT$. Figure 4b shows the same distributions for the full
four seasons of this study; during which time the Dst decreased to an annual
average of $-5.14nT$ in 2019 and averaged only $-5.54nT$ for the whole of
2018-2020 (Nose et al. 2015). This implies not only that the process of plasma
sheet compression is a global process dependant on solar activity, known since
Dungey (1963), but that the mesoscale process of asymmetric TCS distribution
is also a function of solar driving. More observations of tail TCS
distributions during the increasing solar cycle may provide more insight into
this question, or at least provide better statistics.

##  5 Conclusion

Locations in the geomagnetic tail where the neutral sheet thickness is reduced
to ion scales have been mapped during four MMS tail seasons (2017–2020) using
the ratio of radius of magnetic curvature to the thermal ion gyroradius. The
routine calculation of magnetic field curvature was made possible due to the
high-resolution magnetic field measurements available on all four spacecraft
of the MMS fleet, as well as the regular tetrahedron geometry of their
formation. Ion-scale thin current sheets were found to be twice as common on
the pre-midnight side of the geomagnetic tail as on the dawn-side. Locations
of common thin current sheets were compared to the distribution of
reconnection Ion Diffusion Regions previously identified for the same time
span and implications for their coincidence were discussed. Possible
mechanisms for the formation of both thin current sheets and reconnection
suggested by recent PIC simulations were compared to these observations and
qualitative agreement with simulations was found.

###### Acknowledgements.

The authors would like to thank Zach Dykstra and Phil Doroff For computational
support. All instrument data used in this study is publically available at the
MMS Science Data Center (https://lasp.colorado.edu/mms/sdc/public/). This work
has been supported by NASA via contract 499878Q

## References

### — To be replaced with BibTeX later —

Artemyev, A. V., et al. “Cluster Statistics of Thin Current Sheets in the
Earth Magnetotail: Specifics of the Dawn Flank, Proton Temperature Profiles
and Electrostatic Effects.” J. Geophys. Res. Sp. Phys., vol. 116, no. 9, 2011,
pp. 1–9, doi:10.1029/2011JA016801.

Baumjohann, Wolfgang. “The Near-Earth Plasma Sheet: An AMPTE/IRM Perspective.”
Space Science Reviews, vol. 64, no. 1–2, 1993, pp. 141–63. DOI.org (Crossref),
doi:10.1007/BF00819660.

Birn, J. “Computer Studies of the Dynamic Evolution of the Geomagnetic Tail.”
Journal of Geophysical Research: Space Physics, vol. 85, no. A3, Mar. 1980,
pp. 1214–22. DOI.org (Crossref), doi:10.1029/JA085iA03p01214.

Büchner, Jörg, and Lev M. Zelenyi. “Regular and Chaotic Charged Particle
Motion in Magnetotaillike Field Reversals: 1. Basic Theory of Trapped Motion.”
J. Geophys. Res., vol. 94, no. A9, 1989, pp. 11821–42,
doi:10.1029/ja094ia09p11821.

Burch, J. L., et al. “Magnetospheric Multiscale Overview and Science
Objectives.” Space Science Reviews, vol. 199, no. 1–4, Mar. 2016, pp. 5–21.
DOI.org (Crossref), doi:10.1007/s11214-015-0164-9.

Dungey, J.W. “Interactions of Solar Plasma with the Geomagnetic Field.”
Planetary and Space Science, vol. 10, Jan. 1963, pp. 233–37. DOI.org
(Crossref), doi:10.1016/0032-0633(63)90020-5.

Dungey, J. W. “Interplanetary Magnetic Field and the Auroral Zones.” Phys.
Rev. Lett., vol. 6, no. 2, 1961, pp. 47–48, doi:10.1103/PhysRevLett.6.47.

Eastwood, J. P., et al. “Average Properties of the Magnetic Reconnection Ion
Diffusion Region in the Earth’s Magnetotail: The 2001-2005 Cluster
Observations and Comparison with Simulations.” J. Geophys. Res. Sp. Phys.,
vol. 115, no. A08215, 2010, doi:10.1029/2009JA014962.

Fuselier, S. A., et al. “Magnetospheric Multiscale Science Mission Profile and
Operations.” Space Science Reviews, vol. 199, no. 1–4, Mar. 2016, pp. 77–103.
DOI.org (Crossref), doi:10.1007/s11214-014-0087-x.

Genestreti, K. J., et al. “The Location and Rate of Occurrence of Near-Earth
Magnetotail Reconnection as Observed by Cluster and Geotail.” J. Atmos. Solar-
Terrestrial Phys., vol. 121, 2014, pp. 98–109,
doi:10.1016/j.jastp.2014.10.005.

Harris, E. G. “On a Plasma Sheath Separating Regions of Oppositely Directed
Magnetic Field.” Il Nuovo Cimento, vol. 23, no. 1, Jan. 1962, pp. 115–21.
DOI.org (Crossref), doi:10.1007/BF02733547.

Henderson, Mike, et al. LANLGeoMag: V1.5.16 (Version v1.5.16). Zenodo, 10 Mar.
2018, http://doi.org/10.5281/zenodo.1195041.

Imber, S. M., et al. “A THEMIS Survey of Flux Ropes and Traveling Compression
Regions: Location of the near-Earth Reconnection Site during Solar Minimum.”
J. Geophys. Res. Sp. Phys., vol. 116, no. A02201, 2011, p. 12,
doi:10.1029/2010JA016026.

Liu, Yi Hsin, et al. “Three-Dimensional Magnetic Reconnection With a Spatially
Confined X-Line Extent: Implications for Dipolarizing Flux Bundles and the
Dawn-Dusk Asymmetry.” J. Geophys. Res. Sp. Phys., vol. 124, no. 4, 2019, pp.
2819–30, doi:10.1029/2019JA026539.

Lu, San, P. L. Pritchett, et al. “Formation of Dawn-Dusk Asymmetry in Earth’s
Magnetotail Thin Current Sheet: A Three-Dimensional Particle-In-Cell
Simulation.” J. Geophys. Res. Sp. Phys., vol. 123, no. 4, Blackwell Publishing
Ltd, Apr. 2018, pp. 2801–14, doi:10.1002/2017JA025095.

Lu, San, Y. Lin, et al. “Hall Effect Control of Magnetotail Dawn-Dusk
Asymmetry: A Three-Dimensional Global Hybrid Simulation.” J. Geophys. Res. Sp.
Phys., vol. 121, no. 12, 2016, p. 11,882-11,895, doi:10.1002/2016JA023325.

Morley, Steve. Magnetic Ephemeris and Coordinates. Zenodo.
doi:10.5281/zenodo.2594027

Nagai, T, et al. “Structure and Dynamics of Magnetic Reconnection for Substorm
Onsets with Geotail Observations.” J. Geophys. Res., vol. 103, A3, 1998,
doi:10.1029/97JA02190.

Nagai, T., et al. “Three-Dimensional Structure of Magnetic Reconnection in the
Magnetotail from Geotail Observations.” J. Geophys. Res. Sp. Phys., vol. 118,
2013, pp. 1667–78, doi:10.1002/jgra.50247.

M. Nose, T. Iyemori, M. Sugiura, T. Kamei, World Data Center for Geomagnetism,
Kyoto. (2015), Geomagnetic Dst index, doi:10.17593/14515-74000.

Pollock, C., et al. “Fast Plasma Investigation for Magnetospheric Multiscale.”
Space Science Reviews, vol. 199, no. 1–4, Mar. 2016, pp. 331–406. DOI.org
(Crossref), doi:10.1007/s11214-016-0245-4.

Posch, J. L., et al. “Statistical Observations of Spatial Characteristics of
Pi1B Pulsations.” Journal of Atmospheric and Solar-Terrestrial Physics, vol.
69, no. 15, Nov. 2007, pp. 1775–96. DOI.org (Crossref),
doi:10.1016/j.jastp.2007.07.015.

Pritchett, P. L., and San Lu. “Externally Driven Onset of Localized Magnetic
Reconnection and Disruption in a Magnetotail Configuration.” J. Geophys. Res.
Sp. Phys., vol. 123, no. 4, 2018, pp. 2787–800, doi:10.1002/2017JA025094.

Raj, Arjun, et al. “Wind Survey of High-Speed Bulk Flows and Field-Aligned
Beams in the near-Earth Plasma Sheet.” J. Geophys. Res. Sp. Phys., vol. 107,
no. A12, 2002, doi:10.1029/2001JA007547.

Rogers, A. J., et al. “Numerical Algorithm for Detecting Ion Diffusion Regions
in the Geomagnetic Tail With Applications to MMS Tail Season 1 May to 30
September 2017.” Journal of Geophysical Research: Space Physics, vol. 124, no.
8, Aug. 2019, pp. 6487–503. DOI.org (Crossref), doi:10.1029/2018JA026429.

Runov, A., et al. “Current Sheet Structure near Magnetic X-Line Observed by
Cluster.” Geophys. Res. Lett., vol. 30, no. 11, 2003,
doi:10.1029/2002GL016730.

Russell, C. T., et al. “The Magnetospheric Multiscale Magnetometers.” Space
Sci. Rev., vol. 199, 2016, doi:10.1007/s11214-014-0057-3.

Sibeck, D. G., and V. Angelopoulos. “THEMIS Science Objectives and Mission
Phases.” Space Science Reviews, vol. 141, no. 1–4, Dec. 2008, pp. 35–59.
DOI.org (Crossref), doi:10.1007/s11214-008-9393-5.

Shen, C., et al. “Magnetic Configurations of the Tilted Current Sheets in
Magnetotail.” Annales Geophysicae, vol. 26, no. 11, Nov. 2008, pp. 3525–43.
DOI.org (Crossref), doi:10.5194/angeo-26-3525-2008.

Sonnerup, B. U. Ö. “Magnetic Field Reconnection.” Sol. Syst. Plasma Phys.,
edited by L. J. Lanzerotti et al., North-Holland Publishing Co., 1979, pp.
45–108.

Torbert, R. B., et al. “The FIELDS Instrument Suite on MMS: Scientific
Objectives, Measurements, and Data Products.” Space Sci. Rev., vol. 199, 2016,
doi:10.1007/s11214-014-0109-8.

Xiao, Sudong, et al. “Occurrence Rate of Dipolarization Fronts in the Plasma
Sheet: Cluster Observations.” Ann. Geophys., vol. 35, no. 4, 2017, pp.
1015–22, doi:10.5194/angeo-35-1015-2017.


# Multi-Objective Linguistic Control of Large Language Models

Dang Nguyen, Jiuhai Chen, Tianyi Zhou
University of Maryland, College Park
{dangmn, jchen169<EMAIL_ADDRESS>
Project: https://github.com/tianyi-lab/mctune

###### Abstract

Large language models (LLMs), despite their breakthroughs on many challenging
benchmark tasks, lean to generate verbose responses and lack the
controllability of output complexity, which is usually preferred by human
users in practice. In this paper, we study how to precisely control multiple
linguistic complexities of LLM output by finetuning using off-the-shelf data.
To this end, we propose multi-control tuning (MCTune), which includes multiple
linguistic complexity values of ground-truth responses as controls in the
input for instruction tuning. We finetune LLaMA2-7B on Alpaca-GPT4 and
WizardLM datasets. Evaluations on widely used benchmarks demonstrate that our
method does not only improve LLMs’ multi-complexity controllability
substantially but also retains or even enhances the quality of the responses
as a side benefit.

Multi-Objective Linguistic Control of Large Language Models

Dang Nguyen, Jiuhai Chen, Tianyi Zhou University of Maryland, College Park
{dangmn, jchen169<EMAIL_ADDRESS>Project: https://github.com/tianyi-
lab/mctune

##  1 Introduction

Large language models have achieved remarkable success in generating free-from
texts for different downstream tasks or human instructions Xu et al. (2023a);
Chowdhery et al. (2022); Touvron et al. (2023b). However, existing LLMs still
lack precise control over the linguistic complexity of their outputs Soatto et
al. (2023); Chen et al. (2024b); Li et al. (2024a), e.g., the total number of
nouns, the variation of verbs, etc. Linguistic controllability is crucial to
creating personalized outputs since those complexity indices directly reflect
human reading complexity in multiple aspects. For example, a short yes/no
answer is required by some users while a detailed explanation is preferred by
others. Moreover, recent studies Singhal et al. (2023); Zhao et al. (2024);
Lambert et al. (2024) have discovered a spurious correlation between the
quality reward used in LLM alignment and the output length (i.e., a specific
linguistic complexity). Consequently, LLMs favor generating verbose responses
due to the length bias, which may increase unnecessary reading complexity. It
is still an open problem to mitigate the bias without hurting the output
quality.

While existing LLM finetuning techniques such as instruction-tuning Xu et al.
(2023a); Stiennon et al. (2020); Li et al. (2024b) and reinforcement learning
from human feedback (RLHF) Christiano et al. (2017); Bai et al. (2022) have
been demonstrated to be effective in aligning the output with human intent or
preference, they only focus on maximizing a single objective. Instead,
achieving the controllability of multiple complexity indices requires a non-
trivial multi-objective optimization that has not been thoroughly studied on
LLMs. Rather than solely maximizing or minimizing the complexities, it aims to
reach different target complexity values on the Pareto frontier. This requires
LLMs to adjust the trade-off among objectives and capture their potential
correlations or constraints in the text generation process. In addition, due
to the huge space of possible combinations of complexity indices, it could be
expensive to collect training data for multi-objective control.

In this paper, we take the first step towards multi-objective control of the
linguistic complexity of LLM outputs. Instead of collecting new data, our
strategy allows the reuse of existing instruction-tuning data. In particular,
we annotate the ground-truth responses in a dataset by their complexity
metrics evaluated using tools developed in computational linguistics. The
multiple complexity indices and their values are appended as tags to the input
so finetuning an LLM on the linguistic-label augmented data helps build a
strong connection between the input tags and the linguistic complexity of the
output, hence enforcing the LLM to adhere to the complexity requirements
during sequential decoding. Surprisingly, we observe that randomly sampling a
small subset of tags for each training example suffice to obtain
controllability over all the complexities of test examples, thereby reducing
the required amount of training data.

We examine our approach by finetuning LLaMA2-7B Touvron et al. (2023a) using a
linguistic-complexity labeled Alpaca-GPT4 dataset Peng et al. (2023) (i.e.,
the prompts are from the original Alpaca dataset while the responses are
composed by GPT-4) and a WizardLM dataset Xu et al. (2023b). We do not only
observe an expected substantial improvement in linguistic controllability but
also a side benefit of enhanced response quality, indicating that finetuning
under multiple linguistic constraints can improve the LLM for general
purposes. Compared to unconstrained RLHF and IFT methods, which suffer from
the length bias (Chen et al., 2024a), our approach does not introduce the bias
but can improve both the controllability and quality simultaneously.

##  2 Related Work

##### Linguistic Features and Complexity.

The exploration of linguistic features and complexity in language models
encompasses a diverse range of research. Seminal studies have investigated the
syntactic abilities of LSTMs (Linzen et al., 2016). Ficler and Goldberg (2017)
has introduced methods for manipulating the stylistics and syntactic output of
test generation models. Additionally, linguistic style transfer (Shen et al.,
2017) has also showcased the adaptability of language models to capture and
replicate varied linguistic features. Building on the previous efforts to
understand the multifaceted nature of linguistics complexity, our work
concentrates on producing responses endowed with specific linguistic
characteristics, which is less explored in the previous work.

##### Controllability of LLMs.

The topic of personalized language modeling has attracted significant
attention across various research papers. Techniques such as user embedding
have become common for customizing language models to individual needs Welch
et al. (2020); Rocca and Yarkoni (2022). More recently, Mireshghallah et al.
(2022); Oba et al. (2023) propose prompt based personalized fine-tuning for
specific users, and producing personalized responses. Our research shifts the
focus from personalization for specific users to the broader goal of
controlling large language models (LLMs) to produce outputs with linguistic
diversity and complexity, addressing a gap not explored by the aforementioned
works.

Another prevalent technique for guiding the output of large language models
involves Tagging (Korbak et al., 2023; Prabhumoye et al., 2023; Lu et al.,
2022). This method incorporates appending human-readable text during the
training of LLMs. Contrary to previous studies that concentrated on managing
aspects like toxicity (Korbak et al., 2023; Prabhumoye et al., 2023; Li et
al., 2024a) and controlling repetition (Lu et al., 2022), our approach employs
tagging techniques to control linguistic features across multiple attributes.
Additionally, we utilize multiple tags to enable simultaneous consideration of
various attributes, a difference from earlier work that primarily uses of a
single tag.

##### Finetuning and Alignment of LLMs.

With the emergence of large-scale language models, such as those in the GPT
series, aligning language models has become prevalent. Studies like those by
(Zhou et al., 2023; Xu et al., 2023b; Li et al., 2023, 2024c; Chen and
Mueller, 2024) have concentrated on the process of data curation for
instruction fine-tuning to enhance models’ instructions following
capabilities. Unlike these data-centric approaches, we keep the original
instruction dataset (Chen and Mueller, 2023) but augment instructions with
various tags to introduce a richer array of linguistic features, thereby
elevating the instruction-following capabilities of the models.

##  3 Finetuning LLMs for Linguistic Controllability

In this section, we delineate our approach to multi-control tuning. Section
3.1 outlines the linguistic features of interest and describes how we extract
them from a given text segment. In Section 3.2, we explain how the extracted
features are incorporated into the multi-control tuning process.

###  3.1 Handcrafted Linguistic Features

ID | Name | Description
---|---|---
1 | t_word | number of words
2 | n_noun | number of nouns
3 | n_verb | number of verbs
4 | n_adj | number of adjectives
5 | t_uword | number of unique words
6 | n_unoun | number of unique nouns
7 | n_uverb | number of unique verbs
8 | n_uadj | number of unique adjectives
9 | ttr | type-token ratio
10 | noun_var | noun variation
11 | verb_var | verb variation
12 | adj_var | adjective variation
13 | fkre | Flesch-Kincaid reading ease
14 | rt_average | average reading time

Table 1: The list of linguistic features for controllability tuning. The
second column shows the name of each feature with the corresponding
descriptions in the third column. We include a detailed explanation of how
these features are computed in Appendix A.

We are specifically interested in controlling the handcrafted linguistic
properties of the model’s generation. This type of feature has been used
throughout the NLP field Bogdanova et al. (2017); Anshika Choudhary (2021);
Lee et al. (2021) and is loosely defined in Lee and Lee (2023) as “a single
numerical value produced by a uniquely identifiable method on any natural
language.” An example of a linguistic feature not considered handcrafted is
text embeddings produced by deep neural networks, which usually take the form
of a vector. We extract such features from a text segment using the LFTK
package proposed in Lee and Lee (2023). It encompasses a diverse set of 220
features that are grouped into different linguistic families. Within the scope
of this paper, we sample a reasonably-sized set of 14 features for multi-
control tuning, which are presented in Table 1. These features are selected to
cover most of the feature families while being simple to understand and verify
by human users.

Formally, given a text segment $\mathbf{x}=[x_{1},x_{2},...,x_{l}]$, with
$x_{i}$ being the $i$-th token of $\mathbf{x}$, we denote the feature
extractor as a function $f:\mathcal{X}\rightarrow\mathbb{R}^{d}$ that maps
$\mathbf{x}$ to a $d$-dimensional vector
$f(\mathbf{x})=\left[f_{1}(\mathbf{x}),...,f_{d}(\mathbf{x})\right]$, where
$f_{j}(\mathbf{x})$ represents the $j$-th linguistic feature of interest and
$\mathcal{X}$ represents the space of all texts. In this paper, the function
$f$ refers to the LFTK feature extractor.

###  3.2 Multi-Objective Control Tuning

Consider the standard instruction tuning setting where an LLM, denoted as
$p_{\theta}(\mathbf{x})=\prod^{l}_{i=1}p_{\theta}(x_{i}\mid x_{<i})$, is
trained on a dataset of $N$ instruction-output pairs,
$D_{\text{train}}=\left\\{(\mathbf{x}_{i},\mathbf{y}_{i})\right\\}^{N}_{i=1}$.
The training objective is to generate $\mathbf{y}_{i}$ given $\mathbf{x}_{i}$;
thus, the loss is

$\mathcal{L}=-\sum^{N}_{i=1}\sum^{|\mathbf{y}_{i}|}_{k=1}\log{p_{\theta}(\mathbf{y}_{i,k}\mid\mathbf{x}_{i},\mathbf{y}_{i,<k})}$

We address multi-control tuning by framing it as a conditional instruction
tuning problem, where we utilize the target response $\mathbf{y}_{i}$ from
$D_{\text{train}}$ to generate a linguistic control vector $f(\mathbf{y}_{i})$
and append it to $\mathbf{x}_{i}$. The model is then trained to generate
$\mathbf{y}_{i}$ condition on both $\mathbf{x}_{i}$ and $f(\mathbf{y}_{i})$.
However, to enhance data diversity and better simulate real-world scenarios,
where a user may wish to control only a few features, we do not utilize all
features for every data example. For each pair
$(\mathbf{x}_{i},\mathbf{y}_{i})\in D_{\text{train}}$, we randomly sample an
integer $n_{i}\sim\operatorname{Uniform}\left\\{1,\ldots,m\right\\},\,m\leq
d$. The set of $n_{i}$ feature indices, denoted by $C_{i}$, is then randomly
sampled from the pool of all feature indices, i.e.,
$C_{i}\sim\operatorname{Uniform}\left(\left\\{A\subseteq\left\\{1,\ldots,d\right\\}:|A|=n_{i}\right\\}\right)$.
The linguistic control vector used for the $i$-th example is denoted as
$f_{C_{i}}(\mathbf{y}_{i})$, where
$f_{C_{i}}(\mathbf{y}_{i})=\left[f_{C_{i,1}}(\mathbf{y}_{i}),\ldots,f_{C_{i,n_{i}}}(\mathbf{y}_{i})\right]$.
The resulting training loss becomes

$\mathcal{L}=-\sum^{N}_{i=1}\sum^{|\mathbf{y}_{i}|}_{k=1}\log{p_{\theta}(\mathbf{y}_{i,k}\mid\mathbf{x}_{i},f_{C_{i}}(\mathbf{y}_{i}),\mathbf{y}_{i,<k})}$

The benefits of our training strategy are twofold: (1) it improves the
controllability and instruction-following capability simultaneously, thus
avoiding the catastrophic forgetting problem that may degrade the model’s
generation quality. In fact, we will later show in Section 5.6.2 that LLMs
trained with our approach achieve even stronger instruction-following ability
compared to those trained with vanilla instruction tuning; (2) it allows us to
utilize off-the-shelf datasets without the need to collect new ones.

###  3.3 Prompt Template

Figure 1: An example of how data is formatted before being fed into LLMs in
this paper. The first paragraph presents a system prompt containing a complete
list of feature descriptions. Our preliminary results indicate that including
descriptions enhances the effectiveness of our approach.

In practice, we format both the instruction $\mathbf{x}$ and the output
$\mathbf{y}$ into a predefined template before they are input into the LLMs.
This paper adopts the template outlined in Taori et al. (2023), wherein
$\mathbf{x}$ is decomposed into an instruction and an input component.
Regarding linguistic controls, we format them into a sequence of the form
[name_1: value_1] ... [name_n: value_n], aiming for conciseness by utilizing
the features’ abbreviated names listed in Table 1. To assist the LLM in better
understanding these abbreviations, we provide a comprehensive list of feature
descriptions within the system prompt, which has been confirmed to enhance the
effectiveness of our approach in preliminary experiments. This sequence of
controls is subsequently attached to the input component. Figure 1 illustrates
a detailed example of how an input prompt is constructed.

##  4 Evaluation of Linguistic Controllability

During evaluation on a reserved test dataset $D_{\text{test}}$, we aim to
measure how the model performs as the linguistic control vector $f$ changes.
To conduct such an evaluation, we develop a sampling strategy to sample
different control vectors for an instruction $\mathbf{x}_{i}$. Specifically,
we need a sampling strategy that: (1) is specific to an instruction
$\mathbf{x}_{i}$, i.e., the sampled control vectors should not stray too far
from the reasonable range for a specific $\mathbf{x}_{i}$. For example, an
instruction to "Generate a short story" should not have a large value for
t_word; (2) ensures the sampled control vectors are always valid, meaning that
no linguistic controls conflict with each other (e.g., t_uword should always
be less than or equal to t_word), and no control is out-of-bound (e.g., fkre
should always be less than or equal to 121.22).

To achieve the first goal, we utilize $\mathbf{y}_{i}$’s linguistic feature
vector as a reference point and sample new control vectors $f^{\prime}$ from
the Gaussian distribution centered at $f(\mathbf{y}_{i})$, i.e.,
$f^{\prime}\sim\mathcal{N}(f(\mathbf{y}_{i}),I\sigma^{2})$. However, since
each feature $f_{i}(\mathbf{y})$ has a different range, a small $\sigma$ for
some features may be large for others. To avoid this inconsistency, we
standardize all features to unit variance before sampling. More formally, the
new control vector $f^{\prime}$ is computed by

$f^{\prime}=z^{-1}(z(f(\mathbf{y}_{i}))+\sigma\epsilon),\,\,\,\,\epsilon\sim\mathcal{N}(0,I)$

where $z:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ standardizes each feature to
unit variance, and $z^{-1}$ is the inverse operation.

While sampling a valid control vector can be a challenging problem, verifying
its validity is straightforward. This can be done using a simple rule-based
method, which we outline in Appendix B. We utilize this observation to achieve
the second goal by performing rejection-based sampling, i.e., keep resampling
a new control vector $f^{\prime}$ until we find a valid one. We will show in
Section 5.7.3 that $\sigma$ can serve as a hyperparameter to control the
evaluation difficulty.

Lastly, for each example in $D_{\text{test}}$, we randomly sample a number
$n$, a set of $n$ feature indices $C$, and $K$ new control vectors
$f^{\prime}_{C}$ for controllability and generation quality evaluation.

##  5 Experiments

###  5.1 Implementation Details

By default, we set $K=5$ and $\sigma=0.1$ unless specified otherwise. We set
the maximum number of linguistic controls per example to $m=5$ and will show
in Section 5.7.2 that limiting $m$ to 5 does not affect the model’s
controllability, even when more than 5 controls are used in the evaluation.
For all datasets, we fine-tune LLaMA2-7B using the AdamW optimizer with a
linear learning rate schedule. We set the learning rate to $2\times 10^{-5}$,
the batch size to 128, and the number of warmup steps to 100. All models are
trained on 8 RTX A6000 GPUs for 5 epochs.

###  5.2 Datasets.

##### Data Preprocessing.

Because LFTK cannot extract linguistic features with perfect accuracy, it
sometimes generates invalid control vectors $f(\mathbf{y}_{i})$ (e.g.,
producing an fkre greater than 121.22). This situation complicates the process
of sampling new control vectors, as $f(\mathbf{y}_{i})$ may deviate
significantly from the feasible region, drastically reducing the probability
of sampling a valid one. Therefore, we exclude data examples with invalid
$f(\mathbf{y}_{i})$.

##### Alpaca-GPT4.

We utilize the Alpaca-GPT4 dataset Peng et al. (2023) for instruction tuning
and evaluation. This dataset shares the same set of instructions as the Alpaca
dataset Taori et al. (2023), but it employs OpenAI’s GPT-4 to generate high-
quality responses. Our preliminary experiments demonstrate that training with
this dataset yields better results in terms of both controllability and
generation quality compared to training with the original Alpaca dataset.
After preprocessing, the dataset is divided into a training set of 45,000
examples and a test set of 2,000 examples.

##### WizardLM.

In addition to Alpaca-GPT4, we also evaluate our method on the WizardLM
dataset Xu et al. (2023b). The original dataset contains 70,000 examples of
instruction-output pairs that were automatically generated by ChatGPT. We
subsampled 50,000 examples from the original dataset. Similar to Alpaca-GPT4,
after preprocessing, we split the data into a training set of 40,000 examples
and a test set of 2,000 examples.

###  5.3 Models.

We use LLaMA2-7B Touvron et al. (2023b) as the base model for multi-control
tuning in all experiments.

###  5.4 Baselines

To evaluate the impact of our method on controllability and generation
quality, we conduct comparisons with the same LLaMA2-7B base model but trained
with regular instruction fine-tuning. We also include OpenAI’s GPT-3.5 Turbo
(gpt-3.5-turbo-0125) as a baseline to compare our model against state-of-the-
art proprietary LLMs in terms of controllability. For each baseline, we
control the linguistic complexity of their responses using the prompt template
shown in Figure 1 in a zero-shot manner.

###  5.5 Evaluation Metrics

##### Controllability Error.

Given an instruction $\mathbf{x}$, a linguistic control vector $f_{C}$, and a
generated response $\hat{\mathbf{y}}\sim
p_{\theta}(\mathbf{y}\mid\mathbf{x},f_{C})$, we measure controllability error
by computing the $L_{1}$ error between the specified control vector $f_{C}$
and the response’s linguistic feature vector $f_{C}(\hat{\mathbf{y}})$. We
denote this error as
$\mathbf{e}=\left|f_{C}(\hat{\mathbf{y}})-f_{C}\right|\in\mathbb{R}^{|C|}$.

##### Quality Score.

To evaluate generation quality, we follow Zheng et al. (2023) in using a
powerful LLM (e.g., ChatGPT-4 Turbo) as a judge to assign quality scores
ranging from 1 to 10.

###  5.6 Main Results and Analysis

####  5.6.1 Linguistic Controllability Evaluation

Figure 2: Comparison of controllability error (the average normalized $L_{1}$
error) of ChatGPT (gpt-3.5-turbo-0125), IFT (finetuning without controls), and
MCTune (ours) on AlpacaGPT-4 and WizardLM datasets and three test settings of
target linguistic complexity with increasing $\sigma$ (difficulty). To
visualize each linguistic feature’s average error on the same radar plot, we
apply min-max normalization to normalize them to a similar scale. To reduce
the effect of outliers, the minimum $L_{1}$ error of the $i$-th feature is the
minimum among all baselines, and the maximum $L_{1}$ error refers to the 95th
percentile of errors among all baselines.

This section presents the evaluation of linguistic controllability between our
method and various baselines. We consider three evaluation settings: Easy
($\sigma=0.1$), Medium ($\sigma=0.2$), and Hard ($\sigma=0.3$). Our goal is to
measure the controllability error of each baseline on each linguistic control
and then visualize all of them on the same radar plot for comparison.
Formally, for each linguistic control $i$, and each baseline $j$, we maintain
a list
$\mathbf{e}_{i,j}=\left[e^{1}_{i,j},\ldots,e^{|\mathbf{e}_{i,j}|}_{i,j}\right]$
of $L_{1}$ errors made by $j$, where the length of this list depends on the
number of text examples that contain $i$. The matrix of all $L_{1}$ errors for
feature $i$ is denoted as

$\mathbf{E}_{i}=\begin{pmatrix}e^{1}_{i,1}&\ldots&e^{|\mathbf{e}_{i,1}|}_{i,1}\\\
\vdots&\ddots&\vdots\\\ e^{1}_{i,J}&\ldots&e^{|\mathbf{e}_{i,J}|}_{i,J}\\\
\end{pmatrix}$ (1)

It is challenging to directly visualize $\mathbf{E}_{i}$ for all $i$ on the
same radar plot because each linguistic control $i$ has a different range of
values. Therefore, we normalize each element in $\mathbf{E}_{i}$ using min-max
normalization, with the minimum being $\min{E_{i}}$ and the maximum being the
95th percentile of $\mathbf{E}_{i}$, or $P_{95}(\mathbf{E}_{i})$. For each
baseline $j$, its average normalized $L_{1}$ error on control $i$ is
calculated as $\sum_{k}\operatorname{norm}(e^{k}_{i,j})/|\mathbf{e}_{i,j}|$,
where $\operatorname{norm}(\cdot)$ denotes the normalization described above.

As shown in Figure 2, our method consistently outperforms the other baselines
across all settings. Surprisingly, a state-of-the-art model like ChatGPT
underperforms in controllability compared to the instruction-finetuned
LLaMA2-7B. Upon manual inspection, we observed that ChatGPT tends to produce
more verbose responses regardless of the linguistic controls applied, which
may explain the observed poor performance. Another possible reason for this
discrepancy is that LLaMA2-7B is finetuned on data sharing the same
distribution as the test set, giving it an advantage over ChatGPT. Note that
for noun_var, verb_var, and adj_var, the differences between each baseline’s
errors are not significant. We hypothesize that, in contrast to other
linguistic complexities, the descriptions of these features are somewhat more
ambiguous, and we also did not provide a clear description in the system
prompt, which may not be helpful for the model to understand and follow these
features.

Figure 3: Trade-off between linguistic controllability and generation quality
in three increasingly difficult test settings. Each dot represents a model’s
response $\hat{\mathbf{y}}$ to a specific query
$\left[\mathbf{x},f_{C}\right]$. The response is given a quality score from 1
to 10 by a judge LLM (GPT-4 Turbo) based on how well $\hat{\mathbf{y}}$
addresses $\mathbf{x}$. A controllability error is measured for
$\hat{\mathbf{y}}$, which is computed by taking the average of normalized
$L_{1}$ errors across all linguistic controls in $f_{C}$. Blue and orange dots
respectively represent responses from models trained by IFT and MCTune using
Alpaca-GPT4 dataset.

####  5.6.2 Generation Quality Evaluation

Figure 4: Comparison between MCTune and IFT-trained models on MT-Bench. We
finetune LLaMA2-7B on Alpaca-GPT4 dataset and GPT-4 Turbo is the judge in the
test. The average score per axis ranges from 1 to 10 and are given by the
judge.

A natural concern when fine-tuning for controllability is how it affects the
model’s generation quality. In this section, we answer this question by
comparing our method with standard instruction fine-tuning on MT-Bench Zheng
et al. (2023), a widely used benchmark for evaluating LLMs on multi-turn open-
ended questions. Specifically, we use GPT-4 Turbo (gpt-4-0125-preview) as a
judge and compare the two methods in both single-answer and pairwise settings.
In the single-answer setting, the judge assigns a single numeric score from 1
to 10 for each model’s answer. In the pairwise setting, the judge receives two
answers from both baselines and returns either a win, a loss, or a tie. The
result for the single-answer setting is shown in Figure 4. As shown in the
figure, our method does not degrade the model’s general language ability but
even improves it in most categories. Note that the performance of both
baselines on Coding and Math questions is poor because the AlpacaGPT-4 dataset
does not have a strong specialty in questions of this category. When
evaluating under the pairwise setting, MCTune achieves a 51.25% win rate over
instruction fine-tuning. These results suggest that training for
controllability is beneficial for improving LLMs’ natural language
performance.

###  5.7 Analyses

####  5.7.1 Relationship Between Linguistic Controllability and Generation
Quality

This experiment focuses on studying the relationship between generation
quality and the controllability error of LLM responses. To start off, we
randomly select a set of 5 linguistic controls $C$ and fix it throughout the
experiment for simplicity. Given an LLM $p_{\theta}$ and the Alpaca-GPT4 test
dataset
$D_{\text{test}}=\left\\{(\mathbf{x}_{i},\mathbf{y}_{i})\right\\}^{N}_{i=1}$,
we select 50 examples from $D_{\text{test}}$ then sample 50 responses
$\hat{\mathbf{y}}_{i}$ from
$p_{\theta}(\mathbf{y}_{i}\mid\mathbf{x}_{i},f_{C}(\mathbf{y}_{i}))$. Each
$\hat{\mathbf{y}}_{i}$ is then evaluated by GPT-4 Turbo on how well it
satisfies $\mathbf{x}_{i}$ and a controllability error of
$\hat{\mathbf{y}}_{i}$ is computed by taking the average of the normalized
$L_{1}$ errors across all controls in $C$. We also consider three different
settings where $\sigma=0.1$, $\sigma=0.3$, and $\sigma=0.5$ to see how the
pattern shifts as $\sigma$ changes. The results are shown in Figure 3. We can
see that, compared to IFT, our method achieves better quality and
controllability simultaneously. As $\sigma$ increases, the controllability
error increases, which is expected and consistent with the results in Section
6. We then start to notice a slight degradation in quality at $\sigma=0.5$,
where there are no responses with a score of 10, and more responses with low
scores begin to appear. This observation suggests that there might be a
positive correlation between controllability and generation quality.

####  5.7.2 Analysis of Model Controllability with Varying Linguistic Controls

Figure 5: Linguistic controllability error on the test set as the number of
linguistic controls $n$ per sample in MCTune’s training increases. The solid
curves represent the linguistic controllability error averaged over all
linguistic complexities (lower is better) with the shaded areas represent the
95%-confidence interval. The dotted vertical line indicates the maximum number
of controls (5) used during MCTune’s training.

In this experiment, we are interested in studying how the model’s
controllability is affected as the number of linguistic controls increases. We
conduct this experiment using the Alpaca-GPT4 dataset, from which we randomly
sample 100 examples from the test set. For each example, we sample five new
control vectors $f^{\prime}$ of length $n$, compute the average normalized
$L_{1}$ error per control, and then average these over all controls to obtain
a single numeric score. We repeat this process for $n=1$ to $n=14$. The
results are shown in Figure 5. Surprisingly, controllability is worse when $n$
is small. This is true for both MCTune and the IFT baseline, suggesting that
this is not an inherent limitation of our approach. Notice that, as the number
of linguistic controls increases beyond five, the maximum number of controls
used during training, the controllability error of IFT fluctuates, while
MCTune remains approximately stable. This demonstrates that our model, trained
with MCTune, is able to generalize to a larger number of controls than it has
seen in training.

####  5.7.3 Analyzing the Impact of $\sigma$ on Linguistic Controllability
Evaluation

Figure 6: Analysis of how the average normalized $L_{1}$ error varies when
changing the test difficulty level via $\sigma$. We use the same model and
only vary $\sigma$.

This section examines the effect of $\sigma$ on controllability evaluation.
Given a model fine-tuned with MCTune and a test dataset, we follow the process
described in Section 4 to sample new linguistic control vectors for evaluating
controllability. We consider three settings: $\sigma=0.1$, $\sigma=0.2$, and
$\sigma=0.3$. The results are depicted in Figure 6. As illustrated by the
results, as $\sigma$ increases, the average normalized $L_{1}$ error uniformly
increases across all linguistic controls. This indicates that $\sigma$ can
serve as a hyperparameter to adjust the difficulty of the evaluation, which is
intuitive since a larger $\sigma$ enables the sampling of control vectors that
are more distant from the reference control vector $f(\mathbf{y})$.

###  5.8 Case Studies

Figure 7: Examples of MCTune model responding to the same instruction under
different linguistic controls. The linguistic complexities of the responses
are extracted using LFTK with exact values shown at the bottom of each
example.

This section presents examples of how different values of controls affect the
model’s responses. As shown in Figure 7, increasing the t_word control leads
to a longer response and larger values of n_verb and n_uverb increase the
variation in verb choice. Although it is challenging to strictly control the
complexities of the response, the linguistic controls can serve as soft
constraints on the model’s generation.

##  6 Discussion and Conclusions

In this paper, we advance the precise control of linguistic complexities in
LLMs through multi-objective control tuning. Our method is straightforward and
can be seamlessly integrated with existing instruction-tuning datasets without
the need to gather new ones. Our training objective concurrently optimizes the
LLM’s ability to follow instructions and its controllability. Through our
experiments, we find that incorporating this dual-focus strategy significantly
improves the LLM’s generative quality, surpassing the results of instruction
fine-tuning alone. This finding suggests that controllability and instruction-
following ability may have a complementary effect on each other. Additionally,
we observe that while state-of-the-art LLMs achieve impressive natural
language performance, they are not easy to control. This emphasizes the need
for studying methods that improve controllability in LLMs.

##  7 Limitations

While our method significantly improves controllability compared to regular
instruction fine-tuning, we observe that there is still room for improvement.
The model trained with MCTune is able to loosely follow the linguistic
controls but struggles to produce responses with the exact complexities. An
interesting next step would be to improve controllability in a strict setting.
Another limitation we observe is the fact that LFTK sometimes extracts
incorrect linguistic complexities for a given text. This leads to noisy
controls that may confuse and reduce the controllability of the model during
training.

## References

  * Ahmadian et al. (2024) Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024.  Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in llms.  _CoRR_ , abs/2402.14740.
  * Anshika Choudhary (2021) Anuja Arora Anshika Choudhary. 2021.  Linguistic feature based learning model for fake news detection and classification.  _Expert Systems with Applications_.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _CoRR_ , abs/2204.05862.
  * Bogdanova et al. (2017) Dasha Bogdanova, Jennifer Foster, Daria Dzendzik, and Qun Liu. 2017.  If you can’t beat them join them: Handcrafted features complement neural nets for non-factoid answer reranking.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 121–131, Valencia, Spain. Association for Computational Linguistics.
  * Chen and Mueller (2023) Jiuhai Chen and Jonas Mueller. 2023.  Quantifying uncertainty in answers from any language model via intrinsic and extrinsic confidence assessment.  _CoRR_ , abs/2308.16175.
  * Chen and Mueller (2024) Jiuhai Chen and Jonas Mueller. 2024.  Automated data curation for robust language model fine-tuning.  _CoRR_ , abs/2403.12776.
  * Chen et al. (2024a) Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024a.  ODIN: disentangled reward mitigates hacking in RLHF.  _CoRR_ , abs/2402.07319.
  * Chen et al. (2024b) Yihan Chen, Benfeng Xu, Quan Wang, Yi Liu, and Zhendong Mao. 2024b.  Benchmarking large language models on controllable generation under diversified instructions.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.  _CoRR_ , abs/2204.02311.
  * Christiano et al. (2017) Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 4299–4307.
  * Ficler and Goldberg (2017) Jessica Ficler and Yoav Goldberg. 2017.  Controlling linguistic style aspects in neural language generation.  In _Proceedings of the Workshop on Stylistic Variation_ , pages 94–104, Copenhagen, Denmark. Association for Computational Linguistics.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.  _CoRR_ , abs/2310.06825.
  * Korbak et al. (2023) Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. 2023.  Pretraining language models with human preferences.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 17506–17533. PMLR.
  * Lambert et al. (2024) Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024.  Rewardbench: Evaluating reward models for language modeling.  _CoRR_ , abs/2403.13787.
  * Lee et al. (2021) Bruce W. Lee, Yoo Sung Jang, and Jason Hyung-Jong Lee. 2021.  Pushing on text readability assessment: A transformer meets handcrafted linguistic features.
  * Lee and Lee (2023) Bruce W. Lee and Jason Lee. 2023.  LFTK: Handcrafted features in computational linguistics.  In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_ , pages 1–19, Toronto, Canada. Association for Computational Linguistics.
  * Li et al. (2024a) Ming Li, Jiuhai Chen, Lichang Chen, and Tianyi Zhou. 2024a.  Can llms speak for diverse people? tuning llms via debate to generate controllable controversial statements.  _CoRR_ , abs/2402.10614.
  * Li et al. (2024b) Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. 2024b.  Selective reflection-tuning: Student-selected data recycling for LLM instruction-tuning.  _CoRR_ , abs/2402.10110.
  * Li et al. (2024c) Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. 2024c.  Superfiltering: Weak-to-strong data filtering for fast instruction-tuning.  _CoRR_ , abs/2402.00530.
  * Li et al. (2023) Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2023.  From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning.  _ArXiv_ , abs/2308.12032.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of lstms to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Lu et al. (2022) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022.  Quark: Controllable text generation with reinforced unlearning.  _Advances in neural information processing systems_ , 35:27591–27609.
  * Mireshghallah et al. (2022) Fatemehsadat Mireshghallah, Vaishnavi Shrivastava, Milad Shokouhi, Taylor Berg-Kirkpatrick, Robert Sim, and Dimitrios Dimitriadis. 2022.  UserIdentifier: Implicit user representations for simple and effective personalized sentiment analysis.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3449–3456, Seattle, United States. Association for Computational Linguistics.
  * Oba et al. (2023) Daisuke Oba, Naoki Yoshinaga, and Masashi Toyoda. 2023.  Perplm: Personalized fine-tuning of pretrained language models via writer-specific intermediate learning and prompts.  _arXiv preprint arXiv:2309.07727_.
  * Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023.  Instruction tuning with gpt-4.  _arXiv preprint arXiv:2304.03277_.
  * Prabhumoye et al. (2023) Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2023.  Adding instructions during pretraining: Effective way of controlling toxicity in language models.  _arXiv preprint arXiv:2302.07388_.
  * Rocca and Yarkoni (2022) Roberta Rocca and Tal Yarkoni. 2022.  Language as a fingerprint: Self-supervised learning of user encodings using transformers.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 1701–1714, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Shen et al. (2017) Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017.  Style transfer from non-parallel text by cross-alignment.  _Advances in neural information processing systems_ , 30.
  * Singhal et al. (2023) Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023.  A long way to go: Investigating length correlations in RLHF.  _CoRR_ , abs/2310.03716.
  * Soatto et al. (2023) Stefano Soatto, Paulo Tabuada, Pratik Chaudhari, and Tian Yu Liu. 2023.  Taming AI bots: Controllability of neural states in large language models.  _CoRR_ , abs/2305.18449.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020.  Learning to summarize with human feedback.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023a) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023a.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.
  * Welch et al. (2020) Charles Welch, Jonathan K. Kummerfeld, Verónica Pérez-Rosas, and Rada Mihalcea. 2020.  Exploring the value of personalized word embeddings.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6856–6862, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Xu et al. (2023a) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023a.  Wizardlm: Empowering large language models to follow complex instructions.  _CoRR_ , abs/2304.12244.
  * Xu et al. (2023b) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023b.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_.
  * Zhao et al. (2024) Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024.  Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning.  _CoRR_ , abs/2402.04833.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023.  Lima: Less is more for alignment.  _arXiv preprint arXiv:2305.11206_.

##  Appendix A Computing Linguistic Features

In this section, we describe in greater detail how linguistic features are
computed. We use LFTK as a feature extractor, which is based on spaCy. Given a
string, spaCy tokenizes it into a sequence of tokens along with their
annotations (e.g., part-of-speech). The number of words t_word is the number
of tokens. The n_noun, n_verb, and n_adj are computed based on the POS
provided by spaCy. The t_uword, n_unoun, n_uverb, and n_uadj are computed
accordingly. The type-token ratio, noun variation, verb variation, and
adjective variation are $\frac{\texttt{t\\_uword}}{\texttt{t\\_word}}$,
$\frac{\texttt{n\\_unoun}}{\texttt{n\\_noun}}$,
$\frac{\texttt{n\\_uverb}}{\texttt{n\\_verb}}$, and
$\frac{\texttt{n\\_uadj}}{\texttt{n\\_adj}}$, respectively. Let t_sent be the
number of sentences and t_syll is the number of syllables, the formula to
compute Flesch-Kincaid reading ease is

$206.835-1.015\left(\frac{\texttt{t\\_word}}{\texttt{t\\_sent}}\right)-84.6\left(\frac{\texttt{t\\_syll}}{\texttt{t\\_word}}\right)$

Lastly, the average reading time rt_average is
$\frac{\texttt{t\\_word}}{240}$.

##  Appendix B Verifying Validity of Control Vectors

Given a linguistic control vector, it is straightforward to check its validity
by iterating through a list of pre-defined rules.

  * •

$\texttt{t\\_word}>0$

  * •

$\texttt{t\\_word}\geq\texttt{n\\_noun}+\texttt{n\\_verb}+\texttt{n\\_adj}$

  * •

$\texttt{t\\_word}\geq\texttt{t\\_uword}$

  * •

$\texttt{n\\_noun}\geq 0$

  * •

$\texttt{n\\_noun}\geq\texttt{n\\_unoun}$

  * •

$\texttt{n\\_verb}\geq 0$

  * •

$\texttt{n\\_verb}\geq\texttt{n\\_uverb}$

  * •

$\texttt{n\\_adj}\geq 0$

  * •

$\texttt{n\\_adj}\geq\texttt{n\\_uadj}$

  * •

$\texttt{t\\_uword}>0$

  * •

$\texttt{t\\_uword}\geq\texttt{n\\_unoun}+\texttt{n\\_uverb}+\texttt{n\\_uadj}$

  * •

$\texttt{n\\_unoun}\geq 0$

  * •

$\texttt{n\\_uverb}\geq 0$

  * •

$\texttt{n\\_uadj}\geq 0$

  * •

$\texttt{fkre}\leq 121.22$

If any of the rules above is violated, we conclude that the control vector is
invalid.

Method | t_word | n_noun | n_verb | n_adj | t_uword | n_unoun | n_uverb | n_uadj | ttr | noun_var | verb_var | adj_var | fkre | rt_average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
IFT (Zero-shot) | 0.31 | 0.33 | 0.34 | 0.32 | 0.38 | 0.35 | 0.33 | 0.35 | 0.30 | 0.30 | 0.31 | 0.32 | 0.34 | 0.31
IFT (Few-shot) | 0.33 | 0.32 | 0.33 | 0.26 | 0.39 | 0.41 | 0.31 | 0.28 | 0.38 | 0.30 | 0.33 | 0.32 | 0.39 | 0.31
GPT-3.5 (Zero-shot) | 0.37 | 0.50 | 0.36 | 0.44 | 0.55 | 0.34 | 0.33 | 0.54 | 0.26 | 0.33 | 0.48 | 0.29 | 0.39 | 0.47
GPT-3.5 (Few-shot) | 0.48 | 0.53 | 0.65 | 0.37 | 0.67 | 0.60 | 0.39 | 0.44 | 0.40 | 0.29 | 0.35 | 0.28 | 0.40 | 0.48
GPT-4 (Zero-shot) | 0.64 | 0.52 | 0.34 | 0.51 | 0.63 | 0.73 | 0.46 | 0.39 | 0.21 | 0.32 | 0.38 | 0.28 | 0.49 | 0.51
GPT-4 (Few-shot) | 0.40 | 0.40 | 0.47 | 0.31 | 0.46 | 0.51 | 0.35 | 0.36 | 0.36 | 0.28 | 0.30 | 0.27 | 0.40 | 0.32
RLHF (REINFORCE) | 0.25 | 0.24 | 0.27 | 0.40 | 0.27 | 0.25 | 0.25 | 0.27 | 0.29 | 0.29 | 0.38 | 0.30 | 0.83 | 0.36
MCTune | 0.18 | 0.22 | 0.25 | 0.25 | 0.21 | 0.23 | 0.22 | 0.25 | 0.27 | 0.26 | 0.29 | 0.26 | 0.31 | 0.20

Table 2: Additional comparison of controllability error with GPT-4, few-shot
and RL-based baselines.

Method | t_word | n_noun | n_verb | n_adj | t_uword | n_unoun | n_uverb | n_uadj | ttr | noun_var | verb_var | adj_var | fkre | rt_average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
IFT | 0.37 | 0.37 | 0.33 | 0.35 | 0.41 | 0.38 | 0.33 | 0.35 | 0.44 | 0.40 | 0.43 | 0.47 | 0.40 | 0.36
MCTune | 0.34 | 0.32 | 0.32 | 0.29 | 0.37 | 0.35 | 0.32 | 0.34 | 0.47 | 0.41 | 0.42 | 0.44 | 0.33 | 0.31

Table 3: Comparison of controllability error when training Mistral-7B-v0.1
with IFT and MCTune (ours).

##  Appendix C Additional Experiments With More Baselines

In this section, we conduct experiments with other baselines that include:

##### GPT-4.

Similar to other baselines, we query GPT-4 (gpt-4-0125-preview) using the
prompt template in Figure 1 in a zero-shot manner. Since GPT-4 is very
expensive, we only evaluate it on 500 test examples.

##### Few-shot baselines.

To introduce stronger baselines, we consider the few-shot setting where a few
exemplars are included in the prompt to boost the performance of the
underlying model. We carefully select 5 exemplars from the training set that
can cover all the linguistic complexities.

##### RL-based methods.

We compare with a variant of RLHF proposed in Ahmadian et al. (2024), where
instead of training the LLM policy with PPO, they use REINFORCE. To adapt this
method to our task, we use the negative controllability error as the rewards.
Since they can be computed deterministically, we do not need to use neural
networks for reward modeling. More specifically, given
$(\mathbf{x},\mathbf{y},f_{C})\sim D$ and a response $\hat{\mathbf{y}}$, the
reward is

$R_{f_{C}}(\hat{\mathbf{y}},\mathbf{y})=-\frac{1}{|C|}\sum_{i=1}^{|C|}\frac{|f_{C_{i}}(\hat{\mathbf{y}})-f_{C_{i}}(\mathbf{y})|}{M_{i}-m_{i}}$

where $M_{i}$ and $m_{i}$ are the maximum and minimum values of the $i$-th
linguistic complexity in the training set. The REINFORCE training objective
is:

$\mathbb{E}_{(\mathbf{x},\mathbf{y},f_{C})\sim
D}\left[(R_{f_{C}}(\hat{\mathbf{y}},\mathbf{y})-b)\nabla_{\theta}\log{p_{\theta}(\hat{\mathbf{y}}|\mathbf{x})}\right]$

where $\hat{\mathbf{y}}\sim p_{\theta}(\cdot|\mathbf{x},f_{C}(\mathbf{y}))$
and $b$ is the average of all achieved rewards in previous steps. For the
Fine-grained RLHF, we are actively adapting their codebase to our task and
will report the results shortly. As shown in Table 2, MCTune exhibits the
lowest controllability errors on most linguistic complexities.

##  Appendix D Additional Experiments With Other LLM Architectures

In this section, we evaluate our method on Mistral-7B-v0.1 Jiang et al. (2023)
to see how it generalizes to LLM architectures other than LLaMA. As shown in
Table 3, MCTune keeps improving the controllability of Mistral on most
attributes.