# Rotating-modulated Higher-Order Topological States in a Split-ring Photonic
Insulator

Hui Chang Li  School of Physics and Materials Science, Nanchang University,
Nanchang 330031, China;  Xiang Zhou  School of Physics and Materials Science,
Nanchang University, Nanchang 330031, China;  Hai Lin Chi  School of Physics
and Materials Science, Nanchang University, Nanchang 330031, China;  Wen Wen
Wang  School of Physics and Materials Science, Nanchang University, Nanchang
330031, China;  Yun Shen<EMAIL_ADDRESS>School of Physics and Materials
Science, Nanchang University, Nanchang 330031, China;  Institute of Space
Science and Technology, Nanchang University, Nanchang 330031, China  Xiao Hua
Deng<EMAIL_ADDRESS>School of Physics and Materials Science, Nanchang
University, Nanchang 330031, China;  Institute of Space Science and
Technology, Nanchang University, Nanchang 330031, China

###### Abstract

The emerging field of topology has brought device effects to a new level.
Higher-order topological insulators (HOTIs) go beyond traditional descriptions
of bulk-edge correspondence, broadening the understanding of topologically
insulating phases. In this paper, a second-order split-ring photonic crystal
(SSPC) with zero-dimensional ($0$D) corner states and one-dimensional ($1$D)
edge states is proposed. Based on the coupling strength determined by the
opening direction between the split-rings, the electronic transition strength
of the electronic system is imitated, and the topological trivial and non-
trivial transformation of the topological two-dimensional ($2$D) SSH model are
realized by using the rotating split-ring lattice. Theory and simulation find
that SSPC has non-trivial topological edge states that can be quantified by
bulk polarization. As the opening direction of the split-rings gradually
changes within one period, there will be transitions between four different
topological polarizations of the lowest energy bands, which can be
conveniently used to achieve transitions between different topological phases.
Our research can be extended to higher dimensions and broaden research paths
for higher-order photonic topological insulators and semimetals.

††preprint: APS/123-QED

##  I Introduction

Recent years, the research on topological insulators in condensed matter has
flourished. Researchers have conducted extensive and in-depth research on
colorful topological physical effects, from the initial quantum Hall effect to
the fractional quantum Hall effect [1], quantum spin Hall effect [2, 3],
quantum valley Hall effect [4], and then extended many strange properties,
such as bulk-boundary correspondence [5, 6, 7, 8], non-Hermitian skin effect
[9, 10, 11, 12, 13], Floquet topology [14, 15, 16, 17], Anderson local [18,
19], Majorana fermion [20], non-reciprocity [21, 22], non-Abelian topology
[23, 24, 25], hyperbolic metasurface topology [26], et al. The combination of
the excellent characteristics of the topological insulator of the post-
electronic system and a variety of classical systems has pushed the research
of classical systems to a whole new height, such as photons, phonons,
circuits, cold atoms, mechanical and other systems [27, 28, 29, 30, 31, 32,
33, 34].

The higher-order topological phase breaks through the bulk-boundary
correspondence in traditional topology, and introduces a more abstract and
complex concept involving higher-dimensional topological systems. Formally,
the topological state of the lowest dimension contained in an $m$-D
topological insulator system is $n$-D, so this topological insulator is called
a $m$-$n$-order topological insulator.

For example, if the lowest dimension of a $2$D system containing topological
states is $0$D, then it is called $(2-0)=2$-order topological insulator.
Although the topological phase of matter originates from condensed matter
systems, it has proven to be a ubiquitous property in a wide range of wave
systems, and naturally, topological concepts can be extended to optical boson
systems. As a classical optical analogue of electrons in quantum systems,
photonic crystal platforms have the advantages of flexibility and diversity,
making them ideal for realizing high-order topological insulators and
exploring the basic mechanisms. [35, 36, 37, 38, 39, 40].

The coupling size between split-rings can be conveniently controlled by
adjusting the opening direction [41, 42, 43, 44, 45, 46, 47, 48, 49, 50]. In
photonic crystals that can be manipulated flexibly, this property can provide
different rotational degrees of freedom for the exploration of photon platform
topology. The research content of this paper is based on the rotation degrees
of freedom of the four split-rings. Firstly, the emergence of photonic band
gaps caused by the rotation of the split-rings is introduced. Then, the
topological edge states are analyzed intuitively by constructing the
interfaces of topologically trivial and non-trivial structures characterized
by the bulk polarization. Then the higher-order corner states and its
characteristic topological invariant - corner charge, are analyzed, and a
brief summary is made.

##  II MODEL

In this section, we construct the SSPC utilizing the principles of the
classical $2$D Su-Schrieffer-Heeger (SSH) model from condensed matter physics.

As shown in Fig. 1(a), we use split-rings to analogy tight-binding lattice,
and consider the case that the opening orientations of two split-rings are
different. The coupling strength when the opening orientations are far away
from the center of the two rings is higher than that when the opening
orientations are near the center of the two rings.

In this way, the SSPC can be used to compare with the lattice model of the
classical $2$D SSH model. The two cases respectively correspond to the weak
coupling strength $\omega$ and strong coupling strength $\upsilon$. We
simultaneously adjust the opening orientation of four split-rings in a unit
cell to achieve the coupling strength regulation between the split-rings.

In Fig. 1(b), the rotation angles of the four split-rings are $\theta$; the
lattice constant and inner and outer radius of the ring of SSPC are $a$,
$r_{1}$ and $r_{2}$, respectively; the boundary between the ring and the
background material is an ideal electrical conductor, and the relative
dielectric constant of the background material is 11.56. By calculating the
$\Gamma-X-M-Y-\Gamma$ path in the first Brillouin zone (FBZ), the band
structure at different $\theta$ is obtained. When
$\theta=45^{\circ}/225^{\circ}$, there is no band gap, and there is band
degeneracy in the $M-Y$ segment; when $\theta=135^{\circ}/315^{\circ}$, there
is no band gap in the $X-M$ segment; when
$\theta=0^{\circ}/90^{\circ}/180^{\circ}/270^{\circ}$, there exist a band gap
between the first and second bands. As shown in Figure. 1(c).

In addition, the model can be characterized by the Hamiltonian matrix of a
tight-binding approximation model:

$\begin{pmatrix}0&0&\omega_{1}+\upsilon_{1}\cdot e^{-i\cdot
k_{y}}&\omega_{2}+\upsilon_{2}\cdot e^{-i\cdot k_{x}}\\\
0&0&\omega_{2}+\upsilon_{2}\cdot e^{i\cdot k_{x}}&\omega_{1}+\upsilon_{1}\cdot
e^{i\cdot k_{y}}\\\ \omega_{1}+\upsilon_{1}\cdot e^{i\cdot
k_{y}}&\omega_{2}+\upsilon_{2}\cdot e^{-i\cdot k_{x}}&0&0\\\
\omega_{2}+\upsilon_{2}\cdot e^{i\cdot k_{x}}&\omega_{1}+\upsilon_{1}\cdot
e^{-i\cdot k_{y}}&0&0\end{pmatrix}$ (1)

Where, $\omega_{1}=t_{0}+t_{1}\cdot\cos\left(\pi/4+\theta\right)$ represents
the intracellular transition between grid points $1$ and $3$, $2$ and $4$,
$\omega_{2}=t_{0}+t_{1}\cdot\cos\left(-\pi/4+\theta\right)$ represents the
intracellular transition between grid points $1$ and $4$, $2$ and $3$,
$\upsilon_{1}=t_{0}+t_{1}\cdot\cos\left(-3\pi/4+\theta\right)$ represents the
intercellular transition between grid points $1$ and $3$, $2$ and $4$,
$\upsilon_{2}=t_{0}+t_{1}\cdot\cos\left(3\pi/4+\theta\right)$ represents the
intercellular transition between grid points $1$ and $4$, $2$ and $3$, where
$t_{0}=1$, $t_{1}=0.8$, where the exponential silver $\theta$ is the periodic
change factor of the modulation coupling intensity. The period is $[0,2\pi]$,
the band structure described by the Hamiltonian can also open a band gap
between the first and second bands (Appendix 1).

Figure 1: (a) Analogy between the SSPC and the $2$D SSH model in condensed
matter. The opening direction between each two split-rings determines the
coupling strength of the two rings. The coupling strength is smaller when the
openings are oriented towards opposite directions compared to when they are
oriented towards the same direction. Correspondingly, in the $2$D SSH model,
the coupling strength $\omega$ of intracellular lattice points is smaller than
$\upsilon$ of intercellular lattice points. (b) Unit cell and the FBZ of SSPC.
The lattice constant is $a=103um$, the inner and outer radius of the split-
ring $r_{1}=0.13a$, $r_{2}=0.18a$, the rotation angle is $\theta$, and the
rotation directions and rotation centers are denoted by red solid arrows and
red dots, respectively. The FBZ is marked with a black dotted line and the red
solid rectangular box is the path to calculate the band structure. (c)
Photonic bands of the SSPC for $\theta=45^{\circ}/225^{\circ}$ (blue solid
curves), $\theta=135^{\circ}/315^{\circ}$ (green solid curve). The photonic
bands of the SSPC for $\theta=0^{\circ}/90^{\circ}/180^{\circ}/270^{\circ}$ is
marked by a black solid curve, and the band gap marked in yellow appears.

##  III Topological invariant

The topological properties of the different shapes of the preceding band
structure at different rotation angles can be described by the parities at the
highly symmetric point $X/Y$. [42]

Where, $p_{x/y}$ represents the volume polarization in the $x/y$ direction,
and $\eta_{m}(\Gamma/X/Y)$ represents the parity of the characteristic state
at the highly symmetric point $\Gamma/X/Y$ of the m band. The parity is $+1$
and the parity is $-1$. The band gap of the model in this paper occurs between
the first and second energy bands, so only the case of $m=1$ is considered
here

As shown in Fig. 2(c) and d), the two figures are phase transition diagrams of
the first and second frequencies at highly symmetric points $X$ and $Y$ with
respect to the Angle of $\theta$. In a period of change of $\theta$
($0^{\circ}-360^{\circ}$), the first frequency and the second frequency
experience two degeneracy breaks, and their parity also changes with $\theta$.
This results in four intervals with different topological properties within a
period of change.

$\eta(\Gamma)=+1$ over a period of change. In the interval
$0^{\circ}-45^{\circ}$ and $315^{\circ}-360^{\circ}$ $\eta(X)=\eta(Y)=+1$, in
the interval $45^{\circ}-135^{\circ}$ $\eta(X)=+1$, $\eta(Y)=-1$, in the
interval $135^{\circ}-225^{\circ}$ $\eta(X)=\eta(Y)=-1$, and in the interval
$225^{\circ}-315^{\circ}$ $\eta(X)=-1$, $\eta(Y)=+1$, it is easy to know that
the bulk polarization of these four regions is
$\left(p_{x},p_{y}\right)=(0,0)\mid(0,\frac{1}{2})\mid(\frac{1}{2},\frac{1}{2})\mid(\frac{1}{2},0)$
respectively. Next, we will use numerical calculation of topological invariant
– volume polarization to verify the results of the above analysis

Figure 2: (a) The unit cell is marked with a solid black line, and the real-
space lattice vector are marked by black arrows. (b) The FBZ is marked by a
solid red line and the reciprocal lattice vectors are marked by red arrows.
The FBZ is divided into $(M-1)*(N-1)$ regions. Phase transition diagram of the
bottom two eigenvalues as a function of $\theta$ (c) at $X$ and (d) at $Y$.
Light red and light blue curves respectively represent the odd parity and even
parity of the eigenstates. By analyzing the parity of the first eigenvalue,
the corresponding bulk polarization ($p_{x}$, $p_{y}$) can be obtained. The
interval of 1-360° consists of four sections with different bulk polarization,
which are: $\theta=0^{\circ}-45^{\circ}$ and $315^{\circ}-360^{\circ}$ is
$(0,0)$, $45^{\theta}={\circ}-135^{\circ}$ is $(0,1/2)$,
$\theta=135^{\circ}-225^{\circ}$ is $(1/2,1/2)$,
$\theta=225^{\circ}-315^{\circ}$ is $(1/2,0)$.

In this part, we will analyze the related topological invariants of SSPC,
namely bulk polarization derived from Wilson loop and edge polarization derive
from corner charge.

Based on the Berry phase of electrons in condensed matter, the topological
polarization theory of Bloch photons in photonic crystals has been developed.
Here, since the SSPC satisfies mirror symmetry, the Wannier center of the
states can be shifted by $0$ or $1/2$ lattice constant from the center of the
unit cell by designing the structure parameters, while these two deviations
mark the trivial and nontrivial polarization, respectively. In addition, the
finite polarization of the bulk states results in the emergence of the edge
states in the band gap, while the finite polarization of the edge states
results in the emergence of the corner states [35, 37, 38, 39].

Firstly, we define the real space lattice basis vector of SSPC:
$a_{1}=(a,0,0)$ and $a_{2}=(0,a,0)$. To consider $a$ unit vector
$a_{3}=(0,0,1)$ in the z direction and solve for the reciprocal lattice
vector: $b_{1}=\frac{2\pi\big{(}a_{2}\times
a_{3}\big{)}}{a_{1}\cdot\big{(}a_{2}\times
a_{3}\big{)}}=\bigg{(}\frac{2\pi}{a},0\bigg{)}$, $\quad
b_{2}=\frac{2\pi\big{(}a_{3}\times a_{1}\big{)}}{a_{1}\cdot\big{(}a_{2}\times
a_{3}\big{)}}=\bigg{(}0,\frac{2\pi}{a}\bigg{)}$.

The corresponding FBZ can be obtained as shown in Fig. 2(b). The FBZ is
divided into $M\dot{N}$ $k$ points along $b_{1}$ and $b_{2}$ directions
respectively (we use $M=N$ in all calculations). Then, the Wilson loop can be
defined:

$W_{\omega,\upsilon}^{x,y}=F_{\omega,\upsilon}^{x,y}\left(k\right)\cdot
F_{\omega,\upsilon}^{x,y}\left(k+\Delta k\right)\cdots
F_{\omega,\upsilon}^{x,y}\left(k+\left(N-1\right)\cdot\Delta k\right)$ (2)

where
$\left.F_{\omega,\upsilon}^{x,y}\left(k\right)=\left\langle\varphi_{\omega}^{x,y}\left(k\right)\right|\varphi_{\upsilon}^{x,y}\left(k+\Delta
k\right)\right\rangle$ are the Wilson loop elements, in which $\Delta
k=\frac{2\pi}{N}$, the $\left|\varphi_{\omega,\upsilon}^{x,y}(k+\Delta
k)\right\rangle$ is the periodic part of the wave functions of the
$\omega$-th, $\upsilon$-th order band with wave vector $k$ along $x$ and $y$
direction, respectively. $\varepsilon\left(r\right)$ is the position-dependent
electric permittivity.

The Wilson loop operator along and directions can be diagonalized as:
$\left.\left.W_{\omega,\upsilon}^{y}\left|\chi\left(k_{x}\right)\right\rangle=\exp\left(i\cdot
2\pi\Re_{y}\left(k_{x}\right)\right)\right|\chi\left(k_{x}\right)\right\rangle$
and
$\left.W_{\omega,\nu}^{x}\left|\left.\chi\left(k_{y}\right)\right)=\exp\left(i\cdot
2\pi\Re_{x}\left(k_{y}\right)\right)\right|\chi\left(k_{y}\right)\right\rangle$,
where $\left|\chi(k_{x})\right\rangle$ and $\left|\chi(k_{y})\right\rangle$
are the eigenvectors which depends on the Wilson loop, and the phase
$\Re_{y}(k_{x})$ and $\Re_{x}(k_{y})$ are the elements that form the Wilson
loop.

Now, the bulk polarization can be defined as

$\displaystyle p_{x}$
$\displaystyle=\frac{1}{N}\sum_{k_{x}}\Re_{y}\left(k_{x}\right)$ (3)
$\displaystyle p_{y}$
$\displaystyle=\frac{1}{N}\sum_{k_{y}}\Re_{x}\left(k_{y}\right)$

We calculated the four band gap regions mentioned above by using the method of
calculation and above. The results obtained are consistent with the results of
parity analysis above, and the polarization of the four band gap regions is
$(p_{x},p_{y})=(0,0)|\left(0,1/2\right)|(1/2,1/2)|(1/2,0)$.

##  IV Topological edge and corner states

Our proposed model exhibits second-order corner states. Here, we construct a
finite lattice system using models with $(p_{x},p_{y})=(0,0)$ and
$(p_{x},p_{y})=(1/2,1/2)$, as shown in Fig. 3(a). We surround the model with
$(p_{x},p_{y})=(1/2,1/2)$ using a structure with $(p_{x},p_{y})=(0,0)$. The
internal structure has a bulk polarization of $1/2$ in both the $x$ and $y$
directions, which represents a nontrivial topological phase. At the boundary
interface with the trivial topological structure, the existence of topological
edge states can be supported by the non-zero bulk polarization. The band
structures of the horizontal and vertical supercells, shown in Fig. 3(b),
demonstrate the existence of edge states within the band gap. We then
calculate the characteristic frequencies of this finite system, as shown in
Fig. 3(d). Besides the topological edge states, there are four topological
corner states within the band gap. The field distribution of the corner states
is shown in Fig. 3(c), which is localized in the four corners of the model
with $(p_{x},p_{y})=(1/2,1/2)$. If the topological edge states can be
understood as ”edge of the bulk,” then the topological properties of the
corner states can be understood as ”edge of the edge.” Due to the bulk
polarizations of $1/2$ in both $x$ and $y$ directions of the enclosed lattice
model, there can be a description of the corner charge $Q_{c}$ of the corner
mode:

$Q_{c}=4\cdot P_{x}\cdot P_{y}$ (4)

When the corner charge of the wrapped model is non-zero, there will be a
topological angular mode, that is to say, only when the bulk polarization of
wrapped model is $(p_{x},p_{y})=(1/2,1/2)$, $Q_{c}=1$, the topological corner
states will appear.

Figure 3: (a) Schematic diagram of the SSPC finite model. Light red and light
blue represent the SSPC region of the non-trivial topology and the trivial
topology respectively, and the corresponding bulk polarization are
$(p_{x},p_{y})=(0,0)$ and $(p_{x},p_{y})=(1/2,1/2)$. (b) Photonic bands of the
supercell which composed of $(p_{x},p_{y})=(0,0)$ and
$(p_{x},p_{y})=(1/2,1/2)$. The black points are the bulk states, and the red
solid curve is the edge state. (c) $|TE|$ field distribution of corner state,
represents the corner charge interpretation of the corner state: the
polarization of the edge leads to the appearance of the topological corner
state. (d) The eigenfrequency diagram corresponding to the finite model. The
red dots, green dots and black dots represent corner states, edge states and
bulk states, respectively.

When $(p_{x},p_{y})=(0,0)$ and $(p_{x},p_{y})=(1/2,1/2)$, the topology trivial
and non-trivial properties of the model are corresponding respectively. Such
topological distinction can be corresponding to $p_{x}$ and $p_{y}$ and they
will not affect each other. In other words, when the two models with
$(p_{x},p_{y})=(0,0)$ and $(p_{x},p_{y})=(0,1/2)$ are used to form the
interface system, since the $x$ direction is only topologically trivial, while
the y direction is topologically trivial and non-trivial, there does not exist
topological edge state in the $x$ direction. As shown in Fig. 4(a-c), in the
$y$ direction, there exist topological edge states described by non-zero bulk
polarization. Similarly, in the interface system consisting of two models with
$(p_{x},p_{y})=(0,0)$ and $(p_{x},p_{y})=(1/2,0)$, there only exist
topological edge states described by non-zero bulk polarization in the $x$
direction, but does not in the y direction, as shown in Fig. 4(d-f).

Figure 4: (a) Schematic diagram of the vertical supercell and its band
structure (composed of $(p_{x},p_{y})=(0,0)$ (light blue area) and
$(p_{x},p_{y})=(0,1/2)$ (light green area)). The black dots represent the bulk
states and the solid green curve represents the edge state. (b)
Eigenfrequencies corresponding to the finite models. Black dots and green dots
represent bulk and edge states. (c) $|TE|$ field distribution diagram of edge
states. (d) The horizontal supercell diagram and its band structure (composed
of $(p_{x},p_{y})=(0,0)$ and $(p_{x},p_{y})=(1/2,0)$ (light yellow area)),
with black dots representing bulk states and green solid curve representing
edge state. (e) Eigenfrequencies corresponding to the finite models. Black
dots and yellow dots represent bulk and edge states. (f) $|TE|$ field
distribution diagram of edge states.

##  V Conclusion and discussions

We propose a photonic crystal model composed of four split-rings. The coupling
strength between the split-rings can be corresponded by the lattice point
transition in condensed matter. When the four split-rings are rotated at the
same time, there exist two ”open-degenerate-open” processes between the first
frequency and the second frequency at the highly symmetric points $X$ and $Y$
in a 2$\pi$ rotation period. During the period, there exist four types band
gaps with different topological properties, corresponding to different bulk
polarization. When the bulk polarization in the $x/y$ direction are equal to
$1/2$, topological edge state appearing at the $x/y$ interface composed of
topologically trivial structures. In addition, when the bulk polarization in
the $x$ and $y$ directions are both equal to $1/2$, ”edge states of edge
states” appear, that is, corner states. Our model can be further extended to
other classical systems, and the concept of ”coupling can be changed by
rotation” can be applied to dynamic continuously modulated systems, such as
Floquet topology and synthetic dimension. Our research content enriches the
research content of photonic crystal higher-order topology and broadens the
road for the follow-up research.

## ACKNOWLEDGMENTS

This work was supported by the National Natural Science Foundation of China
(Grant numbers 61865009, 61927813).

## References

  * Klitzing  _et al._ [1980] K. v. Klitzing, G. Dorda, and M. Pepper, New method for high-accuracy determination of the fine-structure constant based on quantized hall resistance, Phys. Rev. Lett. 45, 494 (1980).
  * Kane and Mele [2005a] C. L. Kane and E. J. Mele, Quantum spin hall effect in graphene, Phys. Rev. Lett. 95, 226801 (2005a).
  * Kane and Mele [2005b] C. L. Kane and E. J. Mele, ${Z}_{2}$ topological order and the quantum spin hall effect, Phys. Rev. Lett. 95, 146802 (2005b).
  * Xiao  _et al._ [2007] D. Xiao, W. Yao, and Q. Niu, Valley-contrasting physics in graphene: Magnetic moment and topological transport, Phys. Rev. Lett. 99, 236809 (2007).
  * Chen  _et al._ [2023] C. Chen, R.-Z. Liu, J. Wu, Z.-E. Su, X. Ding, J. Qin, L. Wang, W.-W. Zhang, Y. He, X.-L. Wang, C.-Y. Lu, L. Li, B. C. Sanders, X.-J. Liu, and J.-W. Pan, Berry curvature and bulk-boundary correspondence from transport measurement for photonic chern bands, Phys. Rev. Lett. 131, 133601 (2023).
  * Edvardsson  _et al._ [2019] E. Edvardsson, F. K. Kunst, and E. J. Bergholtz, Non-hermitian extensions of higher-order topological phases and their biorthogonal bulk-boundary correspondence, Phys. Rev. B 99, 081302 (2019).
  * Masłowski and Sedlmayr [2023] T. Masłowski and N. Sedlmayr, Dynamical bulk-boundary correspondence and dynamical quantum phase transitions in higher-order topological insulators, Phys. Rev. B 108, 094306 (2023).
  * Xiao  _et al._ [2020] L. Xiao, T. Deng, K. Wang, G. Zhu, Z. Wang, W. Yi, and P. Xue, Non-hermitian bulk–boundary correspondence in quantum dynamics, Nature Physics 16, 761 (2020).
  * Yao and Wang [2018] S. Yao and Z. Wang, Edge states and topological invariants of non-hermitian systems, Phys. Rev. Lett. 121, 086803 (2018).
  * Li  _et al._ [2022] Y. Li, C. Liang, C. Wang, C. Lu, and Y.-C. Liu, Gain-loss-induced hybrid skin-topological effect, Phys. Rev. Lett. 128, 223903 (2022).
  * Zhang  _et al._ [2022] K. Zhang, Z. Yang, and C. Fang, Universal non-hermitian skin effect in two and higher dimensions, Nature communications 13, 2496 (2022).
  * Zhang  _et al._ [2021] X. Zhang, Y. Tian, J.-H. Jiang, M.-H. Lu, and Y.-F. Chen, Observation of higher-order non-hermitian skin effect, Nature communications 12, 5377 (2021).
  * Xin  _et al._ [2023] H. Xin, W. Song, S. Wu, Z. Lin, S. Zhu, and T. Li, Manipulating the non-hermitian skin effect in optical ring resonators, Phys. Rev. B 107, 165401 (2023).
  * Rechtsman  _et al._ [2013] M. C. Rechtsman, J. M. Zeuner, Y. Plotnik, Y. Lumer, D. Podolsky, F. Dreisow, S. Nolte, M. Segev, and A. Szameit, Photonic floquet topological insulators, Nature 496, 196 (2013).
  * Ke  _et al._ [2023] S. Ke, W. Wen, D. Zhao, and Y. Wang, Floquet engineering of the non-hermitian skin effect in photonic waveguide arrays, Phys. Rev. A 107, 053508 (2023).
  * Vu [2022] D. Vu, Dynamic bulk-boundary correspondence for anomalous floquet topology, Phys. Rev. B 105, 064304 (2022).
  * Li  _et al._ [2023] M. Li, C. Li, L. Yan, Q. Li, Q. Gong, and Y. Li, Fractal photonic anomalous floquet topological insulators to generate multiple quantum chiral edge states, Light: Science & Applications 12, 262 (2023).
  * Ren  _et al._ [2024] M. Ren, Y. Yu, B. Wu, X. Qi, Y. Wang, X. Yao, J. Ren, Z. Guo, H. Jiang, H. Chen, _et al._ , Realization of gapped and ungapped photonic topological anderson insulators, Physical Review Letters 132, 066602 (2024).
  * Stützer  _et al._ [2018] S. Stützer, Y. Plotnik, Y. Lumer, P. Titum, N. H. Lindner, M. Segev, M. C. Rechtsman, and A. Szameit, Photonic topological anderson insulators, Nature 560, 461 (2018).
  * Han  _et al._ [2023] S. Han, Y. Chua, Y. Zeng, B. Zhu, C. Wang, B. Qiang, Y. Jin, Q. Wang, L. Li, A. G. Davies, _et al._ , Photonic majorana quantum cascade laser with polarization-winding emission, Nature Communications 14, 707 (2023).
  * Duggan  _et al._ [2020] R. Duggan, S. A. Mann, and A. Alù, Nonreciprocal photonic topological order driven by uniform optical pumping, Phys. Rev. B 102, 100303 (2020).
  * Rechtsman [2023] M. C. Rechtsman, Reciprocal topological photonic crystals allow backscattering, Nature Photonics 17, 383 (2023).
  * Qian  _et al._ [2024] L. Qian, W. Zhang, H. Sun, and X. Zhang, Non-abelian topological bound states in the continuum, Phys. Rev. Lett. 132, 046601 (2024).
  * You  _et al._ [2022] O. You, S. Liang, B. Xie, W. Gao, W. Ye, J. Zhu, and S. Zhang, Observation of non-abelian thouless pump, Phys. Rev. Lett. 128, 244302 (2022).
  * Parto  _et al._ [2023] M. Parto, C. Leefmans, J. Williams, F. Nori, and A. Marandi, Non-abelian effects in dissipative photonic topological lattices, Nature Communications 14, 1440 (2023).
  * Huang  _et al._ [2024] L. Huang, L. He, W. Zhang, H. Zhang, D. Liu, X. Feng, F. Liu, K. Cui, Y. Huang, W. Zhang, _et al._ , Hyperbolic photonic topological insulators, Nature Communications 15, 1647 (2024).
  * Lin  _et al._ [2023] Z.-K. Lin, Q. Wang, Y. Liu, H. Xue, B. Zhang, Y. Chong, and J.-H. Jiang, Topological phenomena at defects in acoustic, photonic and solid-state lattices, Nature Reviews Physics 5, 483 (2023).
  * Zhang  _et al._ [2024] H. Zhang, Z. Gu, L. Si, and J. Ding, Realization of terahertz frequency selecting based on topological edge states with kagome photonic crystals, Results in Physics (2024).
  * Lin  _et al._ [2024] Y.-C. Lin, Y.-C. Chien, and W.-J. Hsueh, Topological slow-light in one-dimensional conjugated photonic systems, Optics Communications (2024).
  * Fan  _et al._ [2024] L. Fan, Y. Chen, J. Zhu, and Z. Su, Multi-band topological valley modes of flexural waves in micro-perforated phononic plates, International Journal of Mechanical Sciences 266, 108916 (2024).
  * Zou  _et al._ [2021] D. Zou, T. Chen, W. He, J. Bao, C. H. Lee, H. Sun, and X. Zhang, Observation of hybrid higher-order skin-topological effect in non-hermitian topolectrical circuits, Nature Communications 12, 7201 (2021).
  * González-Cuadra  _et al._ [2020] D. González-Cuadra, A. Dauphin, P. R. Grzybowski, M. Lewenstein, and A. Bermudez, Dynamical solitons and boson fractionalization in cold-atom topological insulators, Phys. Rev. Lett. 125, 265301 (2020).
  * Mugel  _et al._ [2016] S. Mugel, A. Celi, P. Massignan, J. K. Asbóth, M. Lewenstein, and C. Lobo, Topological bound states of a quantum walk with cold atoms, Phys. Rev. A 94, 023631 (2016).
  * Gao  _et al._ [2024] Z. Gao, Y. Zhang, X. Li, X. Zhang, X. Chen, G. Du, F. Hou, B. Gu, Y. Lun, Y. Zhao, Y. Zhao, Z. Qu, K. Jin, X. Wang, Y. Chen, Z. Liu, H. Huang, P. Gao, M. Mostovoy, J. Hong, S.-W. Cheong, and X. Wang, Mechanical manipulation for ordered topological defects, Science Advances 10, eadi5894 (2024), https://www.science.org/doi/pdf/10.1126/sciadv.adi5894 .
  * Li  _et al._ [2020] M. Li, D. Zhirihin, M. Gorlach, X. Ni, D. Filonov, A. Slobozhanyuk, A. Alù, and A. B. Khanikaev, Higher-order topological states in photonic kagome crystals with long-range interactions, Nature Photonics 14, 89 (2020).
  * Rao  _et al._ [2024] M. Rao, F. Shi, Z. Rao, J. Yang, C. Song, X. Chen, J. Dong, Y. Yu, and S. Yu, Single photon emitter deterministically coupled to a topological corner state, Light: Science & Applications 13, 19 (2024).
  * Mittal  _et al._ [2019] S. Mittal, V. V. Orre, G. Zhu, M. A. Gorlach, A. Poddubny, and M. Hafezi, Photonic quadrupole topological phases, Nature Photonics 13, 692 (2019).
  * Xie  _et al._ [2018a] B.-Y. Xie, H.-F. Wang, H.-X. Wang, X.-Y. Zhu, J.-H. Jiang, M.-H. Lu, and Y.-F. Chen, Second-order photonic topological insulator with corner states, Phys. Rev. B 98, 205147 (2018a).
  * He  _et al._ [2020] L. He, Z. Addison, E. J. Mele, and B. Zhen, Quadrupole topological photonic crystals, Nature communications 11, 3119 (2020).
  * Xie  _et al._ [2018b] B. Xie, G. Su, H. Wang, H. Su, X. Shen, P. Zhan, M.-H. Lu, Z. Wang, and Y.-F. Chen, Visualization of higher-order topological insulating phases in two-dimensional dielectric photonic crystals., Physical review letters 122 23, 233903 (2018b).
  * Guo  _et al._ [2021] Z. Guo, J. Jiang, H. Jiang, J. Ren, and H. Chen, Observation of topological bound states in a double su-schrieffer-heeger chain composed of split ring resonators, Phys. Rev. Res. 3, 013122 (2021).
  * Chen  _et al._ [2022] C. Chen, T. Chen, W. Ding, X. Xiang, F. Mao, and J. Zhu, Split-ring resonator coupling-induced tunable acoustic second-order topological insulators, Phys. Rev. B 106, 045403 (2022).
  * Jiang  _et al._ [2020] J. Jiang, J. Ren, Z. Guo, W. Zhu, Y. Long, H. Jiang, and H. Chen, Seeing topological winding number and band inversion in photonic dimer chain of split-ring resonators, Phys. Rev. B 101, 165427 (2020).
  * Mittal  _et al._ [2021] S. Mittal, G. Moille, K. Srinivasan, Y. K. Chembo, and M. Hafezi, Topological frequency combs and nested temporal solitons, Nature Physics 17, 1169 (2021).
  * Guo  _et al._ [2018] Z. Guo, H. Jiang, Y. Sun, Y. Li, and H. Chen, Asymmetric topological edge states in a quasiperiodic harper chain composed of split-ring resonators, Opt. Lett. 43, 5142 (2018).
  * Ao  _et al._ [2020] Y. Ao, X. Hu, Y. You, C. Lu, Y. Fu, X. Wang, and Q. Gong, Topological phase transition in the non-hermitian coupled resonator array, Phys. Rev. Lett. 125, 013902 (2020).
  * Zhao  _et al._ [2018] H. Zhao, P. Miao, M. H. Teimourpour, S. Malzard, R. El-Ganainy, H. Schomerus, and L. Feng, Topological hybrid silicon microlasers, Nature communications 9, 981 (2018).
  * Lin  _et al._ [2021] Z. Lin, S. Ke, X. Zhu, and X. Li, Square-root non-bloch topological insulators in non-hermitian ring resonators, Opt. Express 29, 8462 (2021).
  * Dutt  _et al._ [2020] A. Dutt, M. Minkov, I. A. Williamson, and S. Fan, Higher-order topological insulators in synthetic dimensions, Light: Science & Applications 9, 131 (2020).
  * Lazarides and Tsironis [2020] N. Lazarides and G. P. Tsironis, Topological split-ring resonator based metamaterials with $\mathcal{PT}$ symmetry relying on gain and loss, Phys. Rev. B 102, 064306 (2020).


# Target Variable Engineering

Jessica Clark
Decisions, Operations, and Information Technology Department
Robert H. Smith School of Business
University of Maryland - College Park
College Park, MD 20912
<EMAIL_ADDRESS>
http://www.jessicamarieclark.com

###### Abstract

How does the formulation of a target variable affect performance within the ML
pipeline? The experiments in this study examine numeric targets that have been
binarized by comparing against a threshold. We compare the predictive
performance of regression models trained to predict the numeric targets vs.
classifiers trained to predict their binarized counterparts. Specifically, we
make this comparison at every point of a randomized hyperparameter
optimization search to understand the effect of computational resource budget
on the tradeoff between the two. We find that regression requires
significantly more computational effort to converge upon the optimal
performance, and is more sensitive to both randomness and heuristic choices in
the training process. Although classification can and does benefit from
systematic hyperparameter tuning and model selection, the improvements are
much less than for regression. This work comprises the first systematic
comparison of regression and classification within the framework of
computational resource requirements. Our findings contribute to calls for
greater replicability and efficiency within the ML pipeline for the sake of
building more sustainable and robust AI systems.

_Keywords_ Machine Learning $\cdot$ Target Variables $\cdot$ Hyperparameter
Optimization

##  1 Introduction

Target variables for machine learning applications should be formulated to
support a specific decision, and in research contexts are usually treated as a
fixed part of the ML pipeline. However, even given a specific task, there can
be flexibility in the formulation of the target variable. Specifically, there
are many applications where either numeric or categorical predictions could be
equally suitable. To use a classic example, if a company wants to address
customers who are likely to “churn", i.e. suspend their services in the near
future, they could train regression models to predict each customer’s numeric
future usage of their service. Alternatively, the company could binarize the
target variable based on whether usage is above or below some threshold. Then,
they would train binary classifiers to predict the resulting categorical
target. Beyond the obvious differences between the two formulations, such as
choosing the appropriate evaluation metric, the choice between these two
potential target variable formulations is not usually discussed in the extant
ML literature.

This work studies the fundamental but previously unanswered research question
of how regression vs. classification models differ, in terms of both resource
requirements within the ML pipeline and replicability of results. In
applications where either formulation could be used interchangeably, the
choice is usually approached heuristically. To systematize this choice, we
conduct an experimental comparison of various parts of the ML pipeline given a
numeric target versus the categorical target variable that results from
binarizing using a threshold. Thus, the predictive problems compared are
identical, save for the formulation of the target variable.

A key component of the experiments is that we compare the performance for the
two task types as a function of Hyperparameter Optimization (HPO) random
search budgets. Across feature sets, target variables, and model families, we
consistently find that regression tasks require significantly more computation
to converge on optimal parameters than their classification counterparts.
Digging into these results reveals that regression is more sensitive not only
to HPO budget, but to all of the heuristic choices across the ML pipeline that
we investigate. HPO budget, model selection, choice of grid search algorithm,
and amount of training data all yield significantly more variation in terms of
test performance than they do for classifiers. The performance of regression
models are also more sensitive to randomness and therefore prone to
overfitting.

Thus, in applications where either formulation could be appropriate, choosing
classification enables use of smaller HPO budgets and yields more
straightforward generalization of results. In general, modelers planning
regressions should ensure a large budget for HPO and use repeated sampling to
ensure generalizability. Modelers conducting classification don’t need to use
such large grids as are used for regression, and can also use smaller amounts
of training data to reach nearly optimal results. Given the substantial carbon
emissions associated with HPO, recent work has pointed out that prioritizing
computationally efficient algorithms can lead to significant reductions in
environmental impact (Strubell et al., 2019; Schwartz et al., 2020). Our
findings thus contribute to research in sustainable AI not by developing more
efficient algorithms, but by streamlining other parts of the ML pipeline. Our
findings also contribute to advances in automated machine learning (AutoML) by
systematizing some heuristic choices. Finally, we also contribute to work
dealing with the crisis of replicability in science broadly, and ML more
specifically (Bouthillier et al., 2021). Simply put, classification results
are easier to replicate. They are less susceptible to overfitting, less
sensitive to both randomness and heuristic choices. Although the state-of-the-
art is relatively better (on average) for regression, making decisions
systematically and reporting all parameters is of more critical importance for
this task.

##  2 Related Work

Practical guides for applied machine learning emphasize the importance of
formulating the target variable to align with some decision that is being
supported (Provost and Fawcett, 2013). For instance, CRISP-DM, a widely-used
business framework for applying machine learning, includes formulating the
target variable as part of the “business understanding" phase (Chapman et al.,
2000). It has been acknowledged that tasks related to the business
understanding phase are not widely studied in the literature (Baier et al.,
2019), and most research in ML (both applied and theoretical) assumes that the
target variable is a fixed concept.

There are a few common practices relating to modifying the target variable to
make prediction problems easier. For example, if the distribution of numeric
values has a heavy right tail, it can be log-transformed. If a binary-valued
target variable has a strong class imbalance, oversampling or undersampling
can be used to improve predictive performance. Target variables can be
specially formulated for particular applications, such as causal effect
estimation (Fernández-Loría and Provost, 2022). The field of _prompt
engineering_ includes constructing tasks to elicit the best-possible
classifications or predictions from large language models such as ChatGPT
(i.e. Sorensen et al. (2022), Brown et al. (2020), Liu et al. (2021), Zhou et
al. (2022)).

There are many examples of past work which have implicitly compared
classification and regression for a particular task by providing reasons for
binarizing numerical target variables. First, it may be easier to acquire
binary, rather than numerical, labels, especially when the labels are user-
generated (Sparling and Sen, 2011). Second, although what is being measured
directly may be numeric, typical use of that variable involves a categorical
decision (Liu et al., 2020). Binarization may result in a simpler problem
(Zhang and Moe, 2021) or yield desirable evaluation metrics such as a
confusion matrix (Abbasi et al., 2019). Recent work has also studied how and
why reformulating a regression problem as classification can result in
improved performance of neural networks (Stewart et al., 2023), which they
term “the binning phenomenon."

The question of whether it’s ever appropriate to binarize a numeric dependent
variable has also been debated in the traditional statistics literature, and
is generally viewed as a bad practice (Royston et al., 2006; Fitzsimons,
2008). Binarizing has been found to lead to misleading results in the size and
direction of coefficients in regression analysis (Maxwell and Delaney, 1993).
Although binarizing the response variable makes results easier to explain and
present to non-practitioners, it can also lead to a loss of information and
statistical power (Irwin and McClelland, 2003). The field has continued to
discuss the role of dichotomization in statistics (Pham, 2015). The results in
this work do not directly contradict past findings; however, we find that
there are positive benefits to binarizing in predictive contexts.

##  3 EXPERIMENTS

The core experiments in this paper seek to compare the process and performance
models trained to predict numerical target variables (regression task) versus
binary categorical target variables (classification task). The experimental
framework relies on the idea that we can compute a binarized counterpart to
any numeric target variable by comparing to a threshold. Thus, all of the
other parameters of the experiment are kept as similar as possible such that
the only difference is the two target variable data types.

###  3.1 Grid Search

Following the notation developed by Dodge et al. (2019), we denote
$\mathcal{M}$ to indicate the _model family_ , meaning a general induction
algorithm with a set of $k$ hyperparmeters that can be optimized. Each
$k$-tuple of values of individual hyperparameters forms one _hyperparameter
value_ $h$, and the set of all possible hyperparameter values forms
$\mathcal{H}_{\mathcal{M}}$. In our experiments, we choose model families that
can be adapted to predict either numeric or categorical target variables, and
thus $\mathcal{H}_{\mathcal{M}}$ is the same for both task types.

The grid searches completed in this paper conduct $B$ random draws from
$\mathcal{H}_{\mathcal{M}}$, and are randomly initialized $S$ times. Let
$\mathcal{A}\left(\mathcal{M},h,s,\mathcal{D}_{T},\mathcal{D}_{P}\right)$
denote an algorithm that returns the performance in some prediction data
$\mathcal{D}_{P}$ using a model from $\mathcal{M}$ with hyperparameter value
$h$ trained on $\mathcal{D}^{T}$, given random initialization state
$s\in\\{1,\cdots,S\\}$. For draw $b$ from $\mathcal{H}_{\mathcal{M}}$, define
the validation and test performance for draw $b$ as:

$\displaystyle v_{b}$
$\displaystyle=\mathcal{A}\left(\mathcal{M},h_{b},s,\mathcal{D}_{T},\mathcal{D}_{V}\right)$
(1) $\displaystyle t_{b}$
$\displaystyle=\mathcal{A}\left(\mathcal{M},h_{b},s,\mathcal{D}_{T},\mathcal{D}_{TE}\right)$
(2)

We report the cumulative maximum validation performance after $B$ grid search
iterations $v^{*}_{B}$, the best hyperparameter value $h^{*}_{B}$ and test
performance using those best hyperparameters $t^{*}_{B}$:

$\displaystyle v^{*}_{B}$
$\displaystyle=\max_{h\in\\{h_{1},...,h_{B}\\}}\mathcal{A}\left(\mathcal{M},h,s,\mathcal{D}_{T},\mathcal{D}_{V}\right)$
(3) $\displaystyle h^{*}_{B}$
$\displaystyle=\operatorname*{arg\,max}_{h\in\\{h_{1},...,h_{B}\\}}\mathcal{A}\left(\mathcal{M},h,s,\mathcal{D}_{T},\mathcal{D}_{V}\right)$
(4) $\displaystyle t^{*}_{B}$
$\displaystyle=\mathcal{A}\left(\mathcal{M},h^{*}_{B},s,\mathcal{D}_{T},\mathcal{D}_{TE}\right)$
(5)

For the experiments in this paper, we set $B=400$ and $S=15$. The first draw
for each search always comprises the default hyperparameters for
$\mathcal{M}$, yielding a reasonable estimate of off-the-shelf performance.
$\mathcal{H}_{\mathcal{M}}$ (including the default parameters) is specific to
each $\mathcal{M}$ and were drawn from Hyperopt-Sklearn (Komer et al., 2014)
and have been used in past work on HPO (Grinsztajn et al., 2022; Gorishniy et
al., 2021).

We experiment with both a standard random search (Bergstra and Bengio, 2012)
as well as the Tree-Structured Parzen Estimator algorithm, a Bayesian
optimization algorithm (Turner et al., 2021). We use the Optuna library in
Python for managing this grid search (Akiba et al., 2019).

###  3.2 Datasets

The data testbed uses three feature sets gathered from publicly-available
online data: Airbnb.com111http://insideairbnb.com/get-the-data/, accessed
March 2018, Kickstarter.com 222https://webrobots.io/kickstarter-datasets/,
accessed Dec 2015, and Yelp.com 333https://www.yelp.com/dataset, accessed Jan
2022. We engineered a tabular feature set of size approximately 2000 from
each.444Note that other tabular benchmarking datasets (Grinsztajn et al.,
2022) mostly have considerably fewer features; having larger feature sets
allows us to experiment with feature set size.

We derived 10 numeric target variables from each domain, which were
standardized using $z$-score normalization such that each one has mean 0 and
standard deviation 1. We further created a binarized counterpart to each
numeric target by thresholding at the mean value. That is, the binarized
target is positive if the numeric target is greater than 0, and negative
otherwise. Table 1 contains detailed descriptions of the datasets and target
variables.

Domain |  Feature Set Description |  Numeric Target Variables
---|---|---
Airbnb |  Information and descriptions of listings from Airbnb.com. |  (1) Number of guests accommodated (2) Availability in the next 30 days (3) Availability in the next 60 days (4) Availability in the next 90 days (5) Availability in the next 365 days (6) Host listings count (7) Number of reviews (8) Price (9) Average rating (10) Average reviews per month
Kickstarter |  Information and descriptions of completed crowdfunding campaigns from Kickstarter.com. |  (1) Dollars pledged (2) Number of backers (3) Dollar goal amount (4) Number of reward levels for contributors (5) Minimum amount to receive an award (6) Maximum amount to receive an award (7) Standard deviation of reward amounts (8) Time between campaign creation and campaign launch (9) Number of sentences in description (10) Average length of sentences in description
Yelp |  Information about business which have received reviews on Yelp.com. |  (1) Total number of reviews (2) Average star rating (3) Average "useful" review rating (4) Average "funny" review rating (5) Average "cool" review rating (6) Average review count of reviewers (7) Percent of reviewers with "elite" status (8) Percent of male reviewers (9) Number of checkins (10) Number of tips
Table 1: Description of feature sets and numeric target variables.

For consistency of comparison, each feature set contains $30,000$ instances,
yielding “medium"-sized data.555We have experimented with much larger datasets
in terms of both instance and feature set sizes and found very consistent
results but due to computational resource constraints we have excluded a full
comparison from this paper. Most results in the paper, other than those
presented in Section 4.2, divide each feature set into three:

  1. 1.

$\mathcal{D}^{T}$: $10,000$ training instances.

  2. 2.

$\mathcal{D}^{V}$: $5,000$ validation instances, used for tuning
hyperparameters.

  3. 3.

$\mathcal{D}^{TE}$: $15,000$ test instances, used for evaluation.

###  3.3 Model Families

The experiments in this paper use three families of induction algorithms that
can be suitable for either the regression or classification task. First,
ensemble methods such as XGBoost (Chen et al., 2015) are currently regarded as
the state-of-the-art ML model for tabular data (Grinsztajn et al., 2022;
Borisov et al., 2022; Shwartz-Ziv and Armon, 2022) and so most of our main
results use XGBoost for modeling. Second, although ensemble methods currently
have superior performance, deep learning for tabular data is an area of active
research, and a recent survey found that ResNet and other deep learning models
can achieve comparable or superior performance to XGBoost on benchmarking
datasets, although they generally take far longer to train (Gorishniy et al.,
2021). Third, we include $L_{2}$-regularized linear methods (linear regression
and logistic regression) as a third model family because they are simple to
understand and interpret, are in common use across a wide variety of fields
and applications, and have been found to achieve decent performance in past
work (Rudin, 2019; Clark and Provost, 2019).

###  3.4 Evaluation

We measure the $R^{2}$ regression score between the actual numeric values and
numerical predictions. $R^{2}$ is normally between 0 and 1. Our results
include numerous modeling settings yielding negative $R^{2}$ due to
overfitting to the training data. To ensure a fair comparison for such
results, we truncated the reported $R^{2}$ to 0. For the corresponding
classifiers, we measured the AUC (Area under the ROC Curve), which represents
the ability of a classifier’s scores to rank positive instances above negative
ones (Provost and Fawcett, 2001) and is usually between 0.5 and 1. AUCs of
less than 0.5 in the validation or test data were truncated to 0.5.

A direct “apples-to-apples" comparison of regression and classification
results is challenging for two reasons. First, the two performance measures
are on different scales and measure different things. Second, even among
target variables of the same type, performance is not necessarily comparable;
some tasks are easier and some are harder. Therefore, we normalize both
$R^{2}$ and AUC relative to the maximum value achieved in each random
initialization of grid search. That is, we compare progress from the minimum
possible value to the maximum value as a function of the HPO budget. If
$v_{min}$ is the minimum achievable value (0 for $R^{2}$ and $0.5$ for AUC),
then define:

$\displaystyle vnorm_{b}$
$\displaystyle=\frac{v_{b}-v_{min}}{v^{*}_{B}-v_{min}}$ (6) $\displaystyle
tnorm_{b}$ $\displaystyle=\frac{t_{b}-v_{min}}{v^{*}_{B}-v_{min}}$ (7)

Thus, $vnorm_{b}$ starts somewhere between 0 and 1 with the default $h$ value.
As $b$ increases to $B$, $vnorm_{b}$ increases to 1. We expect that
$tnorm(b,s)$ is less than $vnorm(b,s)$, and shows the relative
generalizability of each random search run by comparing the test performance
to the expected maximum (validation) performance.

##  4 RESULTS

The experiments in this paper illustrate key differences in how the model
selection and training process plays out for two types of predictive tasks:
regression (predicting numerical targets) vs. classification (predicting their
binarized counterparts). In summary, this section shows that regression
requires more time and data resources to reach optimal performance, and is
also more sensitive to various settings in the process.

Unless stated otherwise, most of the results in this section use feature set
sizes of approximately 2000, a random sampling algorithm for HPO, and XGBoost
as the model family. Section 4.3 probes the effect of these three choices.

###  4.1 Hyperparameter Optimization

Using the formulas given in Equations 6 and 7, Figure 2 plots the normalized
cumulative maximum validation and test performance for each numeric target
variable vs. its binarized counterpart across 400 HPO budgets. The lines show
the average performance across 30 target variables and 15 random
initializations, and the shaded regions show the average difference across
target variables between the minimum and maximum initializations. The
validation performance of regression tasks (in blue) not only has relatively
worse performance given default $h$, but also requires a higher budget to
approach $v^{*}$ . Table 2 summarizes the average number of trials required to
reach $90$, $95$, and $100\%$ of $v^{*}$ for each target variable and random
initialiation. The differences between numeric and binarized targets are all
significant for $\alpha>.99$.

Figure 1: Cumulative maximum validation and test performance by HPO budget for
numeric and binarized target variables, normalized relative to the maximum
overall validation performance.

Figure 2: Average $t^{*}$ given various budgets, relative to a budget of 400
iterations, also including maximum performance across all random seeds.

Table 2: Trials Required for Grid Search Convergence  $\%$ of Max | Num | Bin | Diff | Std Err | p-val
---|---|---|---|---|---
90% | 48.19 | 0.60 | 47.59 | 3.85 | $<.001$
95% | 86.92 | 6.41 | 80.51 | 4.92 | $<.001$
99% | 162.94 | 110.21 | 52.73 | 7.49 | $<.001$

Furthermore, the HPO process for regression is less generalizable. Note in
Figure 2 that the test performance for regression is lower than for
classification, relative to what would be expected given the validation
performance. After 400 search iterations, the average regression $t^{*}$ is
$0.88$. For classification, it is $0.96$. A t-test for the difference between
these two means is significant for $\alpha=0.99$. The average gap between the
minimum and maximum $tnorm*$ is also larger for regression, so the test
performance has greater variation relative to the expected validation
performance. After 400 iterations, the average $tnorm*$ range is 0.27 for
regression and 0.04 for classification. Again, a $t$-test for the difference
in these two means is significant for $\alpha=0.99$. These results suggests
that regression would benefit from increased HPO budgets, i.e. more
computational resources. Certainly, classification and regression should not
use the same sizes of grid.

These results are further emphasized in Figure 2, which compares the average
$t^{*}$ given a budget of 400 iterations versus various other budgets: the
default HP, 10, and 100 iterations as well as the overall best $h$ found
across all random initializations for each target variable. For many numeric
variables, there is a substantial loss in performance for the default $h$ and
other smaller budgets. Furthermore, even the average $t^{*}$ after 400
iterations is still far from the true best possible performance across random
searches, again emphasizing the tendency of regression to overfit. These
differences are not present for classification. Table 3 summarizes the average
percent difference versus a budget of 400 iterations for numeric and binarized
targets. All differences between numeric and binarized tasks are significant
for $\alpha=0.99$ based on paired t-tests. Not tuning, or using a smaller
grid, affects classification significantly less than it affects regression.
Also, the best possible outcome is substantially larger for regression than
the average, again calling the replicability and generalizability of
regression results into question.

Table 3: Mean Difference vs. 400 Tuning Iterations  Iterations | Num (std) | Bin (std)
---|---|---
0 | -39.12$\%$ (0.425) | -0.56$\%$ (0.004)
10 | -30.64$\%$ (0.333) | -0.51$\%$ (0.003)
100 | -5.33$\%$ (0.091) | -0.19$\%$ (0.001)
Best Overall | 13.91$\%$ (0.199) | 0.58$\%$ (0.002)

###  4.2 Learning Curves

The results in this section used the process given by Perlich et al. (2003) to
create learning curves that show generalization performance with respect to
the amount of training data for regression vs. classification. In order to
experiment with larger quantities of training data (up to $20,000$ training
instances), we recombined $\mathcal{D}^{T}$, $\mathcal{D}^{V}$, and
$\mathcal{D}^{TE}$, then randomly selected $5,000$ test instances for each
target. To create learning curves, we repeated the following steps 30 times.

  1. 1.

Randomly draw $k$ training instances, where $k$ is between $100$ and $20,000$.

  2. 2.

Using the training set of size $k$, train an XGB model using a best $h$ to
predict the numeric target. Estimate predictions in the test set and measure
the $R^{2}$.

  3. 3.

Using the training set of size $k$, train an XGB model using a best $h$ to
predict the binarized target. Estimate predictions in the test set and measure
the AUC.

  4. 4.

Normalize each $R^{2}$ and $AUC$ such that 0 is the minimum possible
performance and 1 is the maximum observed performance across all $k$ for that
target.

Figure 4 shows the normalized progress to the maximum performance averaged
across 30 target variables and 30 random draws for each. The shaded regions
represent a +/- 1 standard deviation interval around the average. This chart
provides evidence that the learning curves for regression are “steeper" with
respect to the amount of training data. That is, for any number of training
data, classification tends to have relatively closer performance to the
maximum than regression. For instance, the average normalized performance for
regression with a 100-instance training set is $4\%$ of the maximum observed
and $17\%$ for classification. With a 1000-instance training set, regression
is at $39\%$ and classification is at $59\%$.

We also expect that learning curves will level out as the marginal benefit of
more data diminishes. At the high end of training set sizes, the
classification learning curves appear to be flattening, while the regression
curves are apparently still increasing. This implies that regression models
receive relatively more benefit from more data; once again, classification
requires fewer resources to perform at the highest level.

Figure 3: Average normalized learning curves for regression vs.
classification. On average, regression has steeper learning curves, meaning
that the performance is more sensitive to training set size.

Figure 4: Validation and test performance across feature set sizes.

###  4.3 Other parts of the pipeline

There are other heuristic choices in the ML pipeline besides HPO budget. This
section probes the effects of the size of the feature set, the choice of
sampling algorithm in the HPO grid search, and the choice of the model family
for regression vs. classification.

####  4.3.1 Feature set size

The models trained in the prior sections were trained using approximately 2000
features each. What happens if there are fewer features? Figure 4 replicates
Figure 2 with 5 and 100 features. With 5 features, the relative differences
between regression and classification are less dramatic, although still
present. The differences are quite evident when there are 100 features.
Anecdotally, we note that we have conducted preliminary experiments both on
published benchmark tabular datasets (Grinsztajn et al., 2022) as well as
feature sets with up to $200,000$ features, and the results of these
preliminary experiments confirm the main results in this paper.

####  4.3.2 Grid Search Sampling Algorithm

Figure 6 compares the test performance across two grid search sampling
algorithms: simple random sampling, and Tree-Structured Parzen Estimator
(TPE), a Bayesian sampler which has been found to yield improved results
(Turner et al., 2021). As before, the difference between the two samplers are
much smaller for classification than for regression. We also note that for
regression, the choice of which sampler performs better would depend on the
HPO budget.

Figure 5: Relative performance of random sampling vs. Tree Parzen Estimator.

Figure 6: Test performance of linear and ResNet models relative to XGBoost.

####  4.3.3 Model Selection

All of the prior results in this paper have used XGBoost as the model family;
however, we find that model selection is also more impactful for regression
than for classification, as can be seen in Figure 6.666Note that due to
resource constraints, these results include only 19 of the target variables
examined in the prior sections. This chart compares the average best tuned
test performance of Linear and ResNet models relative to XGBoost. The
differences between the best and worst-performing model families for
classification are significantly less than those associated with regression.
The average percent improvement for regression tasks between the worst and
best-performing model family is 245.79$\%$. For classification, the average
percent improvement is 6.34$\%$. The paired differences are statistically
significant for $\alpha=0.90$.777The core results for this paper have also
been replicated for Linear and ResNet models and are included in the
supplemental material.

Taken along with the results in Section 4.3.2, the implication is that
heuristic choices in all parts of the ML pipeline matter relatively more for
regression than for classification. The selection of models included in model
selection are more consequential. Model selection can be significantly
shortcut for classification because the best model is closer in performance to
the worst and/or default model. With implications for replicability, the
models chosen to benchmark performance in research proposing a new algorithm
for regression also take on increased importance. These results also present
an interesting tradeoff for researchers. Although regression requires more
resources, there are also potentially larger benefits to be found when
developing new regression algorithms (for all parts of the ML pipeline).

##  5 DISCUSSION

Our results bring additional nuance to the current understanding of the
importance of HPO in the machine learning pipeline, particularly as it
pertains to replicability of ML findings, sustainable AI, and automation of ML
heuristics. HPO is necessary to achieve optimal performance in ML models
(Bischl et al., 2023), to the point where ML benchmarking results can be
reversed depending on the extent of HPO conducted (Bouthillier et al., 2021;
Dodge et al., 2019). This has contributed to a lack of replicability in the ML
literature and calls for increased detail in reporting of experimental
parameters (Dodge et al., 2019). HPO budget, i.e. number of search iterations
or total time, is also a framework that has been used for evaluating the
differences between induction algorithms; for instance, deep learning methods
have been found to achieve comparable performance to tree-based ensemble
methods on tabular data, but deep learning methods require far more
computational resources (Gorishniy et al., 2021). Our results leverage HPO
budget as a dimension by which to compare the relative resources required by
regression and classifications and reveal the large discrepancy in
computational requirements between the two tasks. Our focus on HPO also
highlights the fact that regression is more sensitive to both heuristic
choices and randomness. This both makes regression modeling findings around
regression harder to replicate and calls for larger grids (and even more
computation) to be used in such contexts.

A major cost associated with HPO is the computation time that it requires,
especially in the modern age of large language models and neural architecture
search (NAS) (Strubell et al., 2019). A full grid search trains and evaluates
models using all possible hyperparameter combinations, although randomized
grid search and its variations have been shown to be just as effective but
much faster (Bergstra and Bengio, 2012). Still, given the criticality of
conducting a thorough grid search, HPO uses a tremendous amount of resources.
These resource requirements leads to egregious quantities of carbon emissions
(Strubell et al., 2019; Schwartz et al., 2020) and also inequities in who is
able to contribute to the ML field (Strubell et al., 2019). Our findings
contribute to recent calls for more efficient ML algorithms (Strubell et al.,
2019; Schwartz et al., 2020; Dodge et al., 2019) by improving the efficiency
of the ML pipeline rather than any specific modeling algorithm: assuming that
regression and classification are interchangeable from the perspective of
performance in a downstream application, we show that classification requires
a smaller grid search and fewer resources in general.

The other cost of HPO is one that is common to the entire ML pipeline. There
are numerous heuristic choices involved, such as which induction algorithms to
try for comparison or optimization, which features to use, how much training
data to acquire, how to set the HPO budget, which hyperparameters to tune, the
size of the grid, and more. These choices are usually made by knowledgeable
data scientists, who are in short supply (He et al., 2021). AutoML attempts to
automate some of these choices, thereby streamlining the number of heuristic
choices in the pipeline (He et al., 2021). For instance, recent work has
focused on determining which hyperparameters for each common model family are
tunable (i.e. where HPO effort is best spent) (Probst et al., 2019). This
paper makes a fundamental contribution to the AutoML literature by instead
evaluating tunability based on an underlying characteristic of the data being
modeled: the formulation of the target variable. We find that regression tasks
are overall more tunable, which has previously observed but not systematically
evaluated (Sipper, 2022).

This work makes the significant assumption that regression and classification
can be used interchangeably in some contexts and studies the effect of this
choice on the resources required by the ML pipeline.888Of course, there are
also situations where either formulation could be reasonably used but the
downstream outcomes will differ; we leave a thorough exploration of the choice
between regression and classification in terms of outcomes to future work.
Thus, it provides insight into the choice of whether or not to binarize by
conducting a systematic comparison. Although past work in statistics has
demonstrated that binarization leads to issues in traditional analyses, it
frequently occurs in applied ML. We demonstrate that regression tasks are
particularly costly in terms of required modeling effort; they require a
higher HPO budget and greater amounts of training data, and the model
selection process is less generalizable. Classification should be chosen when
possible for the sake of efficiency, and smaller grids can be used. On the
other hand, regression may present a greater opportunity for researchers who
wish to publish impactful results; however, sufficiently large grid search,
ensembling, and repeated sampling should be used to ensure replicability.

There are a few other apparent limitations in this work. First, most of our
results use XGBoost to demonstrate the salient differences between the two
tasks. We assert that using XGBoost may actually yield conservative results
based on preliminary experiments with ilnear models and ResNet deep learning
models. Second, our datasets are relatively small compared to the data
typically used for truly computationally burdensome ML tasks. Once again, we
believe that the performance differences between regression and classification
seen in our results may be conservatively estimated compared to what would be
seen with larger datasets, both in number of instances and number of features,
based on the results in Sections 4.2 and 4.3. Finally we also note that
tabular datasets of medium size are quite common in business applications.
Future work could verify our findings with larger datasets and other model
types.

##  6 CONCLUSION

We have experimentally compared the effect of choosing numeric regression vs.
binary classification on the required resources and resulting performance in
the ML pipeline. We show that choosing a numeric target variable consistently
requires more time, computation, and data resources, and yields results that
are more sensitive to randomness and model selection. We present actionable
recommendations for ML researchers, users, and consumers of models.

## References

  * Abbasi et al. (2019) Ahmed Abbasi, Jingjing Li, Donald Adjeroh, Marie Abate, and Wanhong Zheng.  Don’t mention it? analyzing user-generated content signals for early adverse event warnings.  _Information Systems Research_ , 30(3):1007–1028, 2019.
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A next-generation hyperparameter optimization framework.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 2623–2631, 2019.
  * Baier et al. (2019) Lucas Baier, Fabian Jöhren, and Stefan Seebacher.  Challenges in the deployment and operation of machine learning in practice.  In _ECIS_ , volume 1, 2019.
  * Bergstra and Bengio (2012) James Bergstra and Yoshua Bengio.  Random search for hyper-parameter optimization.  _Journal of machine learning research_ , 13(2), 2012.
  * Bischl et al. (2023) Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix, et al.  Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 13(2):e1484, 2023.
  * Borisov et al. (2022) Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci.  Deep neural networks and tabular data: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022\.
  * Bouthillier et al. (2021) Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al.  Accounting for variance in machine learning benchmarks.  _Proceedings of Machine Learning and Systems_ , 3:747–769, 2021.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chapman et al. (2000) Pete Chapman, Julian Clinton, Randy Kerber, Thomas Khabaza, Thomas Reinartz, Colin Shearer, Rüdiger Wirth, et al.  Crisp-dm 1.0: Step-by-step data mining guide.  _SPSS inc_ , 9(13):1–73, 2000.
  * Chen et al. (2015) Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, et al.  Xgboost: extreme gradient boosting.  _R package version 0.4-2_ , 1(4):1–4, 2015.
  * Clark and Provost (2019) Jessica Clark and Foster Provost.  Unsupervised dimensionality reduction versus supervised regularization for classification from sparse data.  _Data Mining and Knowledge Discovery_ , 33:871–916, 2019\.
  * Dodge et al. (2019) Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith.  Show your work: Improved reporting of experimental results.  _arXiv preprint arXiv:1909.03004_ , 2019.
  * Fernández-Loría and Provost (2022) Carlos Fernández-Loría and Foster Provost.  Causal classification: Treatment effect estimation vs. outcome prediction.  _The Journal of Machine Learning Research_ , 23(1):2573–2607, 2022.
  * Fitzsimons (2008) Gavan J Fitzsimons.  Death to dichotomizing, 2008.
  * Gorishniy et al. (2021) Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.  Revisiting deep learning models for tabular data.  _Advances in Neural Information Processing Systems_ , 34:18932–18943, 2021.
  * Grinsztajn et al. (2022) Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux.  Why do tree-based models still outperform deep learning on typical tabular data?  _Advances in Neural Information Processing Systems_ , 35:507–520, 2022.
  * He et al. (2021) Xin He, Kaiyong Zhao, and Xiaowen Chu.  Automl: A survey of the state-of-the-art.  _Knowledge-Based Systems_ , 212:106622, 2021.
  * Irwin and McClelland (2003) Julie R Irwin and Gary H McClelland.  Negative consequences of dichotomizing continuous predictor variables.  _Journal of Marketing Research_ , 40(3):366–371, 2003.
  * Komer et al. (2014) Brent Komer, James Bergstra, and Chris Eliasmith.  Hyperopt-sklearn: automatic hyperparameter configuration for scikit-learn.  In _ICML workshop on AutoML_ , volume 9, page 50. Citeseer Austin, TX, 2014.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, 2021.
  * Liu et al. (2020) Yuanyang Liu, Gautam Pant, and Olivia RL Sheng.  Predicting labor market competition: Leveraging interfirm network and employee skills.  _Information Systems Research_ , 31(4):1443–1466, 2020.
  * Maxwell and Delaney (1993) Scott E Maxwell and Harold D Delaney.  Bivariate median splits and spurious statistical significance.  _Psychological bulletin_ , 113(1):181, 1993.
  * Perlich et al. (2003) Claudia Perlich, Foster Provost, and Jeffrey Simonoff.  Tree induction vs. logistic regression: A learning-curve analysis.  2003\.
  * Pham (2015) Michel Tuan Pham.  Is it ok to dichotomize? a research dialogue.  _Journal of Consumer Psychology_ , 25(4):650–651, 2015.
  * Probst et al. (2019) Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl.  Tunability: Importance of hyperparameters of machine learning algorithms.  _The Journal of Machine Learning Research_ , 20(1):1934–1965, 2019.
  * Provost and Fawcett (2001) Foster Provost and Tom Fawcett.  Robust classification for imprecise environments.  _Machine learning_ , 42:203–231, 2001.
  * Provost and Fawcett (2013) Foster Provost and Tom Fawcett.  _Data Science for Business: What you need to know about data mining and data-analytic thinking_.  " O’Reilly Media, Inc.", 2013.
  * Royston et al. (2006) Patrick Royston, Douglas G Altman, and Willi Sauerbrei.  Dichotomizing continuous predictors in multiple regression: a bad idea.  _Statistics in medicine_ , 25(1):127–141, 2006\.
  * Rudin (2019) Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature machine intelligence_ , 1(5):206–215, 2019.
  * Schwartz et al. (2020) Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni.  Green ai.  _Communications of the ACM_ , 63(12):54–63, 2020\.
  * Shwartz-Ziv and Armon (2022) Ravid Shwartz-Ziv and Amitai Armon.  Tabular data: Deep learning is not all you need.  _Information Fusion_ , 81:84–90, 2022.
  * Sipper (2022) Moshe Sipper.  High per parameter: A large-scale study of hyperparameter tuning for machine learning algorithms.  _Algorithms_ , 15(9):315, 2022.
  * Sorensen et al. (2022) Taylor Sorensen, Joshua Robinson, Christopher Michael Rytting, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate.  An information-theoretic approach to prompt engineering without ground truth labels.  _arXiv preprint arXiv:2203.11364_ , 2022.
  * Sparling and Sen (2011) E Isaac Sparling and Shilad Sen.  Rating: how difficult is it?  In _Proceedings of the fifth ACM conference on Recommender systems_ , pages 149–156, 2011.
  * Stewart et al. (2023) Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean-Philippe Vert.  Regression as classification: Influence of task formulation on neural network features.  In _International Conference on Artificial Intelligence and Statistics_ , pages 11563–11582. PMLR, 2023.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for deep learning in nlp.  _arXiv preprint arXiv:1906.02243_ , 2019.
  * Turner et al. (2021) Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, and Isabelle Guyon.  Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020.  In _NeurIPS 2020 Competition and Demonstration Track_ , pages 3–26. PMLR, 2021.
  * Zhang and Moe (2021) Kunpeng Zhang and Wendy Moe.  Measuring brand favorability using large-scale social media data.  _Information Systems Research_ , 32(4):1128–1139, 2021.
  * Zhou et al. (2022) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.  Large language models are human-level prompt engineers.  _arXiv preprint arXiv:2211.01910_ , 2022.