# Solving Challenging Control Problems
Using Two-Staged Deep Reinforcement Learning

Nitish Sontakke and Sehoon Ha Georgia Institute of Technology, Atlanta, GA
30318 USA. e-mail<EMAIL_ADDRESS>
###### Abstract

We present a two-staged deep reinforcement learning algorithm for solving
challenging control problems. Deep reinforcement learning (deep RL) has been
an effective tool for solving many high-dimensional continuous control
problems, but it cannot effectively solve challenging problems with certain
properties, such as sparse reward functions or sensitive dynamics. In this
work, we propose an approach that decomposes the given problem into two
stages: _motion planning_ and _motion imitation_. The motion planning stage
seeks to compute a feasible motion plan with approximated dynamics by directly
sampling the state space rather than exploring random control signals. Once
the motion plan is obtained, the motion imitation stage learns a control
policy that can imitate the given motion plan with realistic sensors and
actuations. We demonstrate that our approach can solve challenging control
problems - _rocket navigation_ and _quadrupedal locomotion_ \- which cannot be
solved by the standard MDP formulation. The supplemental video can be found
at: https://youtu.be/FYLo1Ov_8-g

###### Index Terms:

Quadrupedal Locomotion, Deep Reinforcement Learning, Motion Planning, Motion
Imitation

##  I Introduction

In recent years, deep reinforcement learning (deep RL) has demonstrated
impressive performance on a wide range of sequential decision problems ranging
from the Go game [1, 2] to robot control [3, 4, 5]. However, deep RL in
practice still requires significant engineering efforts on tuning algorithms
and reward functions to find an effective control policy. Conditions such as
sparse reward functions and sensitive dynamics amplify the complexity of the
problem. These properties degrade the accuracy of the policy gradient descent
method by hiding informative signals and injecting noises. Therefore,
engineers often try to address these issues by designing denser and smoother
reward functions or combining a policy with hand-designed controllers based on
their prior knowledge.

Our approach to tackling challenging control problems is to divide them into
easier sub-problems. Inspired by traditional motion planning algorithms [6, 7,
8] that optimize plans with approximated dynamics, we pre-solve an
approximated problem with deep RL and use the solution as a hint for solving
the original problem. We believe that this decomposition provides substantial
improvements to current deep RL methods for robotics. First, we can directly
_sample a sequence of states_ rather than sampling control signals, which
allows the learning algorithm to investigate a broader range of experiences.
In addition, we can efficiently compute transitions in the motion planning
stage, which makes the computation much more efficient than detailed physics-
based simulation. We can also easily incorporate the desired prior knowledge,
such as desired footfall patterns for locomotion tasks, which are not
guaranteed during deep RL with physics-based simulation. Although this
approach might initially sacrifice the validity of the physics slightly, the
imitation phase eventually recovers these missing details.

Figure 1: Two challenging environments used for evaluation: _Rocket
Navigation_ and _Quadrupedal Locomotion_.

In this work, we propose a technique for decomposing the original control
problem into two subproblems: _motion planning_ and _motion imitation_. First,
we solve the motion planning problem with the same reward function as the
original problem but with a simplified transition function. We rapidly explore
various trajectories by sampling in the state space rather than sampling
action signals. We suppress physics errors by introducing a new dynamics term,
which is similar to the formulation in traditional motion planning algorithms.
Once we generate the reference trajectory, we train a new motion imitation
policy to track the given motion plan in a full-scale physics simulation.

We evaluate the proposed algorithm in two environments: Rocket Navigation and
Quadrupedal Locomotion, where both tasks are designed to have sparse reward
functions and non-linear, sensitive dynamics. The experimental results
indicate that our two-staged approach can achieve much higher rewards than
solving it as a monolithic problem. We also demonstrate the robustness of the
resulting policies by applying random perturbations at testing time.

##  II Related Work

Motion Planning Algorithms. Traditional motion planning encompasses various
algorithms such as roadmaps, cell decomposition, potential field methods, and
sampling-based methods. For domains such as quadrupedal locomotion, sampling
tends to be the method of choice because of its efficacy in high-dimensional
spaces. The Rapidly-Exploring Random Tree with its variants has gained
significant traction as an effective tool for tackling this problem [6, 7, 8].
Ratliff et al. proposed CHOMP [9], which uses signed distance fields to model
obstacles and Hamiltonian Monte Carlo Sampling instead of traditional
sampling-based planning methods. There has also been a string of optimization-
based methods [10, 11, 12, 13, 14, 15]. Instead of using optimization, we
solve motion planning problems using deep RL.

Deep Reinforcement Learning. Learning-based approaches have proven to be
effective for solving various control problems, including control of
quadrupedal robots. Some of the earliest work to leverage reinforcement
learning is by Kohl and Stone [16], who proposed a policy gradient-based
method to achieve a faster gait on the Sony Aibo robot. More recently, deep RL
has been employed to train agents to learn feedback control [3, 17, 5]. The
learned policy has been successfully deployed on real robots by using domain
randomization (DR) [3], system identification [5], or real-world adaptation
[18]. Alternatively, researchers [4, 19] have investigated learning policies
directly from real-world experience, which can intrinsically overcome sim-to-
real gaps. Sample efficiency is a critical challenge for deep RL approaches,
which can be improved by leveraging model-based control strategies [17, 20,
21]. Recently, Lee et al. [22] employed a teacher-student framework for
training their agent. In addition, deep RL can solve a motion imitation
problem [23, 24] to track the given reference motion, typically obtained by
motion capture. Instead of using motion capture data, our algorithm aims to
imitate the given motion trajectory obtained at the planning stage.

Hierarchical Reinforcement Learning. Hierarchical Reinforcement Learning (HRL)
[25, 26] has also proven to be an effective tool in tackling the locomotion
problem. Furthermore, researchers [27] have developed HRL strategies for
controlling legged characters in simulation, which have been further combined
with adversarial learning [28] to achieve high-level control.

Reinforcement Learning with Motion Planning. Traditional model-free RL
algorithms are effective for many domains, but they often suffer from high
sample complexity. They also have difficulty converging to optimal policies in
the case of long-horizon tasks, and encounter safety concerns in environments
cluttered with obstacles. On the other hand, motion planners need accurate
models of the environment and are vulnerable to the curse of dimensionality.
Their strength lies in their capability to compute optimal collision-free
trajectories.

Therefore, there exists a lot of prior work to combine the best of both
domains. Jurgenson and Tamar [29] propose a modification to the popular DDPG
algorithm that allows them to train neural motion planners more efficiently.
Luo et al.  [30] create demonstrations by planning to states already visited
by the policy during the exploration phase. They train their policy by
sampling experience tuples from this demonstration buffer and the original
replay buffer. Xia et al. [31] propose a two-level hierarchical approach,
where the higher-level module is a learned policy that outputs the subgoal
that the lower-level motion planner can then reach. Yamada et al. propose
MoPA-RL [32], which switches between direct execution of policy actions and a
path output by a motion planner, depending on the action magnitude. Our
approach draws inspiration from the given concept but instead uses deep RL for
the motion planning component and combines it with the motion imitation
framework.

##  III Two-staged Reinforcement Learning

Figure 2: Overview of our two-staged Reinforcement Learning method. We
decompose our original problem into two stages - motion planning and motion
imitation. Motion planning uses deep RL to learn an approximate motion plan
using a simplified model. Motion imitation learns a control policy to track
the generated reference plan in physics simulation.

This section will describe our algorithm for solving challenging control
problems with sparse reward signals and sensitive dynamics, which are
difficult to solve with naive formulations. Our key insight into solving this
problem is to adopt a two-staged approach by employing an _RL based motion
planning_ stage and a _motion imitation_ stage (Figure 2). During the _motion
planning_ stage, we aim to find an approximate motion plan for the given
problem using simplified dynamics. During the _motion imitation_ stage, we
train a control policy with the original, complex action space to imitate the
given motion plan. This approach allows us to solve complex problems in a much
more sample-efficient manner, which is difficult to achieve using standard
deep RL formulations.

###  III-A Background: MDP and Deep RL

In reinforcement learning, we formulate the given problem as a Markov Decision
Process (MDP) $(\mathcal{S},p_{0},\mathcal{A},\mathcal{T},R,\gamma)$. We
define an MDP with a set of states $\mathcal{S}$, an initial state
distribution $p_{0}$, a set of actions $\mathcal{A}$, a transition function
$\mathcal{T}=p(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t})$, a reward
function $R(\mathbf{s}_{t},\mathbf{a}_{t})$, and a discount factor $\gamma$,
where the subscript $t$ denotes the time step.

For each time step, an agent observes the current state $\mathbf{s}_{t}$ and
samples an action $\mathbf{a}_{t}$ from a policy
$\pi_{\theta}(\mathbf{a}_{t}|\mathbf{s}_{t})$ that is parameterized by
$\theta$. Our goal is to find the optimal policy parameter $\theta^{*}$ that
maximizes the cumulative reward:

$J(\pi)=\mathbb{E}_{\tau\sim\rho_{\pi}(\tau)}\left[\sum_{t=0}^{T}\gamma^{t}r_{t}\right],$
(1)

where $T$ is the planning horizon. Typically, deep RL utilizes a policy
gradient that is estimated from a large number of trajectories to find the
optimal parameters. This defines our baseline, monolithic RL, that solves the
given problem using a standard, unmodified deep RL algorithm. We use the
Proximal Policy Optimization (PPO) [33] algorithm in our work.

However, deep RL cannot easily solve a challenging control problem when it is
modeled using a standard MDP formulation. First, it often fails to solve a
sparse reward function. Typically, it is easier to describe the desired
behavior of the agent using sparse rewards by simply specifying the success
condition or the final desired state. However, this easily confuses the deep
RL algorithm because it does not receive informative signals during
exploration. The second challenge is when we need to deal with high-
dimensional, sensitive dynamics. For instance, in case of the bipedal
locomotion problem, the majority of control signals will cause the robot to
fall immediately. In this case, the control policy must be carefully tuned to
generate meaningful experiences. To mitigate these challenges, we propose a
two-stage approach that uses deep RL for the motion planning stage as well.

###  III-B Motion Planning Stage

We re-utilize the deep RL approach, retaining the original problem definition
as much as possible. In addition, using deep RL allows us to effectively solve
the motion planning problem with an unknown time horizon, which is required
for typical optimization-based motion planning algorithms.

We define a motion planning problem as a new MDP
$(\mathcal{S}_{plan},p_{0},\mathcal{A}_{plan},\mathcal{T}_{plan},R_{plan},\gamma)$
by patching four components: the state space $\mathcal{S}_{plan}$, the action
space $\mathcal{A}_{plan}$, the transition function $\mathcal{T}_{plan}$, and
the reward function $R_{plan}$. First, we define a compact state space
$\mathcal{S}_{plan}$ that is typically a subset of $\mathcal{S}$. We then
redefine the action space to directly sample the state space, which allows the
learning algorithm to sample more meaningful states than random action
signals, particularly for highly sensitive control problems. More
specifically, we define our action to be the difference to the state,
$\mathbf{a}\in\mathcal{A}_{plan}=\Delta\mathcal{S}_{plan}$, while limiting the
gap between two states using box constraints. This gives rise to the new
transition function
$\mathbf{s}_{t+1}=\mathcal{T}_{plan}(\mathbf{s}_{t},\mathbf{a}_{t})=\mathbf{s}_{t}+\mathbf{a}_{t}$.
The simplicity of the transition function allows us to take a much larger step
in the motion planning stage, such as $0.1$ second, compared to the typical
time steps of $0.002$ seconds to $0.0005$ seconds in physics-based simulation.
In our experience, this gives us a significant boost in terms of computational
resources and wall clock time.

However, this action space formulation only considers the smoothness of the
trajectory by limiting the difference between two consecutive states. It may
be suitable for simple dynamics, but it does not guarantee the physical
validity for challenging problems such as legged locomotion. Therefore, we use
an augmented reward for motion planning, $R_{plan}=R+R_{dyn}$, where $R$ is
the original task reward and $R_{dyn}$ is the dynamics term that incentivizes
the physical validity of the state transitions. We can implement $R_{dyn}$ in
various ways, such as using the simplified model, just like other
optimization-based motion planning algorithms. By solving the given problem,
the algorithm finds the trajectory of the states without control signals. This
reference trajectory
$\bar{\tau}=(\bar{\mathbf{s}}_{0},\bar{\mathbf{s}}_{1},\cdots,\bar{\mathbf{s}}_{T})$
is used as an input to the next stage.

###  III-C Imitation learning framework

Once we obtain a reasonable trajectory, our motion imitation stage is used to
find a control policy that can generate feasible and effective control signals
to follow the given trajectory. Once again, we define a motion imitation MDP
$(\mathcal{S},p_{0},\mathcal{A},\mathcal{T},R_{imi},\gamma)$ by only replacing
the reward function $R$ of the original MDP with the motion imitation reward
$R_{imi}$. The reward $R_{imi}(\mathbf{s}_{t})$ is designed to measure the
similarity between the current state and given reference state
$\bar{\mathbf{s}}_{t}$. A typical choice would be to measure the difference
between them,
$R_{imi}(\mathbf{s}_{t})=|\mathbf{s}_{t}-\bar{\mathbf{s}}_{t}|^{2}$. For more
complex robots, such as the quadruped robots in our experiments, we can design
a slightly more complicated reward function that measures the differences of
various features, such as joint positions, joint velocities, and end-effector
positions, inspired by the formulation in the DeepMimic paper [23]. We will
describe more details of our formulation in the experiments section.

Note that our two-staged approach allows us to convert sparse reward signals
$R$ to dense and informative rewards, $R_{imi}$, with minimal prior knowledge.
This invention allows us to find a control policy more efficiently. In
addition, the motion imitation framework is known to find a robust control
policy by repetitively collecting a large number of samples near the reference
trajectory. We will further evaluate the benefit of our approach in the
following section.

##  IV Experiments

We designed our experiments to evaluate the proposed two-staged learning
framework on difficult control problems. Particularly, we selected two
challenging problems, _Rocket Navigation_ and _Quadrupedal Locomotion_ (Figure
1), to answer the following research questions:

  * •

Can the proposed method solve challenging control problems in a more sample
efficient manner?

  * •

Can the proposed method allow us to incorporate prior knowledge into a control
policy?

  * •

Is the resulting policy robust to external perturbations?

For all the learning experiments, we use the stable-baselines [34]
implementation of PPO [33] as our learning algorithm. The hyperparameters for
all the experiments, such as learning weights or network architectures, are
tuned with a wide range using Bayesian hyperparameter search.

###  IV-A Rocket Navigation

Our first environment is _Rocket Navigation_ , where the task is to control
the rocket so that it can reach the target location, the moon. This
environment is inspired by the OpenAI gym [35] Lunar Lander environment and
implemented using an open-source 2D physics simulator, PyBox2d [36] with a
frequency of $50$ Hz. The rocket is controlled by three engines, a single main
engine responsible for thrust, and two side engines responsible for rotations.
All the engines have highly non-linear relationships between control signals
and actual exerted forces. We use a gravity value of $-2.5m/s^{2}$, and model
air resistance using the relationship $F_{ar}=-\lambda\dot{\mathbf{x}}$, where
$\dot{\mathbf{x}}$ is the rocket velocity, and the coefficient $\lambda=2.5$
in our experiments.

Original MDP. In the original problem, the observation is defined as a $10$
dimensional vector that includes the rocket’s global position, orientation,
velocity, angular velocity, relative location of the goal, current distance
from the goal, and angle difference between the current rocket heading and
goal direction. We use a two-dimensional continuous action space with values
in the range $[-1,1]$. The two dimensions correspond to the main and side
engine thrusts respectively. The main engine only works when the action
sampled is in $[0.5,1.0]$. The side engines work when the action sampled is in
the range $[-1.0,-0.5]\ \bigcup\ [0.5,1.0]$: a negative value leads to clock-
wise rotation whereas a positive value leads to counter-clockwise rotation. We
scale our action values by factors $50$ and $10$ for the main and side engines
respectively so that the rocket has enough thrust to complete the task
successfully.

For reward design, we use an exponential function that gives us explicit
control over the smoothness of the reward. We intentionally design a sparse
reward function with the agent receiving rewards and penalties only when the
rocket gets very close to the goal or obstacles:

$R(\mathbf{s}_{t})=w_{1}\text{exp}(k_{1}||\mathbf{x}_{t}-\mathbf{x_{g}}||_{2})\\\
-w_{2}\text{exp}(k_{2}||\mathbf{x}_{t}-\mathbf{o_{1}}||_{2})-w_{3}\text{exp}(k_{3}||\mathbf{x}_{t}-\mathbf{o_{2}}||_{2}).$

Here, $\mathbf{x}_{t}$ is the current rocket position, $\mathbf{x_{g}}$ is the
goal location, and $\mathbf{o_{1}}$ and $\mathbf{o_{2}}$ are the obstacle
locations. All of these lie in $\mathcal{S}$. In our experiments, we set
$w_{1}=0.9,w_{2}=w_{3}=0.05,k_{1}=k_{2}=k_{3}=-0.5$. The first term encourages
the agent to get closer to the goal while the other two terms discourage
collisions with the obstacles. We depict our sparse reward function using heat
maps in Figure 3 Left. As a result, it is very challenging to solve the given
control problem using a naive RL algorithm.

We terminate the episode when (1) the rocket goes out of bounds, or (2) the
episode reaches the maximum number of steps.

Motion Planning MDP. During the motion planning stage, we aim to find a sparse
state trajectory that runs at $20$ Hz. We use a seven dimensional state space
that is a subset of the original formulation without the velocities. The
action $\mathbf{a}\in\mathcal{A}_{plan}$ can directly change the position and
orientation of the rocket by a maximum of $0.5$ m and $0.05^{\circ}$ at each
step. These values are chosen such that the resulting trajectory is realizable
for the motion imitation policy. The reward function for this stage,
$R_{plan}$ is the same as the original reward function $R$ without any
additional $R_{dyn}$.

Motion Imitation MDP. During the motion imitation, we will find a control
policy that imitates the reference trajectory generated by the motion planning
stage. Therefore, our reward function is design to imitate the reference
state:

$R_{imi}(\mathbf{s}_{t})=\text{exp}(k_{imi}||\mathbf{x}_{t}-\bar{\mathbf{x}}_{t}]||_{2})R(\bar{\mathbf{s}}_{t})-0.2I_{c},$

where $\mathbf{x}_{t}$ is the current rocket position and
$\bar{\mathbf{x}}_{t}$ is the point on the reference trajectory closest to
$\mathbf{x}_{t}$. We adjust the scale by multiplying the original reward
$R(\bar{\mathbf{s}}_{t})$ for $\bar{\mathbf{x}}_{t}$ in $R_{plan}$. The binary
flag $I_{c}$ is set to $1$ if there is a collision, and $0$ otherwise. We set
$k_{imi}=-0.625$. The heat map for this reward function is depicted on the
right side in Figure 3. Here, the state space is the same as the baseline
formulation.

Figure 3: Heat maps of the reward functions in Rocket Navigation. Left: An
original sparse reward $R$. Right: A dense imitation reward $R_{imi}$ to track
the given reference motion.  Figure 4: Learning curves for two Rocket
Navigation and two Quadrupedal Locomotion environments. Our approach has two
stages, motion planning (orange) and motion imitation (green), while the
standard RL has one learning curve in blue. The rewards are normalized by
their theoretical maximum value.

Results. We investigated two scenarios, beginning with a simple navigation
task, followed by the addition of two obstacles in the environment to make the
task harder. For the simpler task, we omit the terms corresponding to the
obstacles from the reward and set $k_{1}=-1$. In Figure 4, we compared the
learning curves of the proposed method against the baseline algorithm for
solving the given original MDP directly. We allowed one million time steps for
each motion planning and imitation stage while allowing two million time steps
for the baselines. Please note that one step in the motion planning stage is
$5$ to $10$ times computationally cheaper for evaluation than the others.
Since all 3 MDPs have slightly different reward functions, we normalized them
by dividing by the respective theoretical maximum rewards for fair comparison.

Our method was able to obtain reasonable solutions for both tasks, while the
baseline failed to solve either. We attribute this to the lack of meaningful
reward signal received by the agent for moving toward the goal, owing to the
sparsity of the reward. The agent tried to explore nearby regions with random
actuation signals, which cannot generate any meaningful thrusts and sinks to
the bottom due to gravity. On the other hand, the simplified dynamics of the
motion planning stage allowed the agent to explore a much larger region of the
state space and finally discover a reasonable path to the goal. Although the
learned path should be sub-optimal under the given dynamics (e.g., it does not
consider acceleration or deceleration), the agent can convert it to a
dynamically feasible trajectory by imitating it. Please note that the only
prior knowledge we injected is that the trajectory must be smooth, which has
been exploited during the motion planning stage.

The results can be interpreted from multiple points of view. First, we
observed that the motion imitation stage could convert the given reasonable
kinematic trajectory to a physically valid motion, as observed in the prior
work. Additionally, our motion planning stage converts a sparse reward signal
to a dense reward formulation by providing a reasonable reference path, as can
be observed from Figure 3.

###  IV-B Quadrupedal Locomotion

Figure 5: The learned Laikago trotting motions. Our two-staged approach finds
a valid, periodic motion (Top) while the baseline barely balances without any
forward movement (Bottom). The two motions are captured at the same time
stamps.

Next, we evaluate the method on a higher dimensional control problem,
_Quadrupedal Locomotion_. The task is to make the quadrupedal robot, Laikago
from Unitree [37], walk with a specified footfall pattern. We consider two
gaits - walking and trotting. We utilize PyBullet [38] for simulation. We use
a simulation frequency of $600$ Hz and a control frequency of $30$ Hz. Since
the reward function does not include the detailed guidance for joint
movements, it can still be considered sparse. In addition, the robot must
maintain its balance during locomotion. Therefore, this reactive control task
is challenging to solve for legged robots with $12$ degrees of freedom without
any additional prior knowledge, such as trajectory generators [17].

Original MDP. In the original problem, the observation includes the robot’s
joint positions, joint velocities, bodies’ linear and angular positions and
velocities, a phase variable, and the distance to the goal. The 12-dimensional
actions are the PD targets for all the joints. We define our multiplicative
reward function as follows:

$R(\mathbf{s}_{t},\mathbf{a}_{t})=r_{dist}\cdot r_{ns}\cdot r_{ee\\_z}\cdot
r_{com},$

where the distance, no slip, end-effector ground z, foot clearance, and Center
of Mass (CoM) terms respectively are

$\displaystyle r_{dist}$
$\displaystyle=\text{exp}\left(k_{dist}||\mathbf{x}_{t}-\mathbf{g}||_{2}\right)$
$\displaystyle r_{ns}$
$\displaystyle=\text{exp}\left(k_{ns}\sum_{j=1}^{4}c(j,t)||\mathbf{e(j)}_{t}-\mathbf{e(j)}_{t-1}||_{2}\right)$
$\displaystyle r_{ee\\_z}$
$\displaystyle=\text{exp}\left(k_{ee\\_z}\sum_{j=1}^{4}||z(j)_{t}-\bar{z}(c(j,t))||_{2}\right)$
$\displaystyle r_{com}$
$\displaystyle=\text{exp}\left(k_{com}||\mathbf{C}-\mathbf{P}||_{2}\right)$

where $\mathbf{x}_{t}$ is the current base position, $\mathbf{g}$ is the goal
location, $c(j,t)$ is the desired contact flag for end-effector $j$ at time
$t$, $\mathbf{e(j)}_{t}$ is its position, $z(j)_{t}$ its current z-coordinate
and $\bar{z}(j)_{t}$ represents the desired z-coordinate which is $0$ cm when
$c(j,t)$ is on and $10$ cm when off. $\mathbf{C}$ and $\mathbf{P}$ represent
the respective positions of the CoM and the centroid of the end-effectors,
both projected onto the ground plane. The first term encourages the robot to
move towards the goal, the second term discourages foot slipping, the third
describes the desired foot height for the given footfall pattern, and the last
term is used to increase motion stability. We terminate the episode when the
robot reaches the goal location, deviates from the goal location by more than
the starting distance, its CoM drops below $0.32$m, or the episode reaches the
maximum steps, which is set to $59$ in our experiments. We set
$k_{dist}=-2.5,k_{ns}=-10,k_{ee\\_z}=-20$ and $k_{com}=-4$ in our experiments.

Motion Planning MDP. We employ the abstract state space that consists of the
body position and end-effector positions. Based on our definition in Section
III-B, the action can directly change the kinematic positions of the abstract
state. In addition, we directly control vertical positions of end-effectors
based on the given footfall pattern: the foot height is set to $0$ cm if the
foot should be in contact and $10$ cm if not. The base can move at a maximum
velocity of $0.264$ m/s and $0.63$ m/s while the end-effectors can move at
maximum angular speeds of $1.65$ rad/s and $1.26$ rad/s for walking and
trotting respectively. The reward function is defined as

$R_{plan}=r_{dist}\cdot r_{com}\cdot r_{x\\_pen}\cdot r_{com\\_z}$

where $r_{dist}$ and $r_{com}$ are the same as before. We have

$\displaystyle r_{x\\_pen}$
$\displaystyle=\text{exp}\left(k_{ee\\_x}\sum_{j=1}^{4}||x(j)_{t}-\bar{x}(j)||_{2}\right)$
$\displaystyle r_{com\\_z}$
$\displaystyle=\text{exp}\left(k_{com\\_z}||\mathbf{C}_{z,t}-\bar{\mathbf{C}}_{z,t}||_{2}\right)$

The first term penalizes the unnecessary lateral movements of the end-
effectors by setting $\bar{x}(j)$ as the x positions at the first frame. The
second term encourages the CoM z-coordinate $\mathbf{C}_{z,t}$ to stay close
to the desired CoM z-coordinate value $\bar{\mathbf{C}}_{z,t}$.

Motion Imitation MDP. During motion imitation, our goal is once again to find
an imitation policy that can mimic the given reference motion from the
previous stage. We use the same state and action spaces as the baseline and
exactly the same reward function used in the original DeepMimic [23]
formulation. Additionally, we use a Butterworth filter to smooth our actions.
For more details, please refer to the original paper.

Results. As before, we compared the _normalized_ learning curves of our two-
staged approach with the standard formulation in Figure 4. We set the same
budget of $10$M time steps for both stages of our method and $20$M time steps
for the baseline. Similar to the previous rocket navigation environment, the
baseline failed to learn meaningful locomotion policies. Our method could find
an effective trajectory during the motion planning stage and converts it to a
control policy that works with full-scaled physics-based simulation with on-
board sensors. Please refer to Figure 5 and the supplemental video for the
detailed motions.

Please note that our formulation has almost the same prior knowledge as the
baseline, such as the footfall patterns or the desired speed. However, in our
experience, the formulation makes a huge impact on learning results. In
typical MDP formulations, the desired contact flags can only be exploited as
the reward term that has an indirect impact on learning. However, our
framework allows us to directly couple them with the desired z positions of
the feet.

In addition, the motion imitation framework is known to generate a robust
control policy by repetitively sampling around the given target trajectory. We
checked the robustness of the learned walking policy by applying external
perturbations. Based on our experiments, the policy can endure up to the $100$
N force for a duration of $0.1$ second. We demonstrate this in the
supplemental video.

##  V Conclusion and Future Work

We presented a deep reinforcement learning framework that takes a two-staged
approach for solving challenging control problems. Our key idea is to split
the given control problem into two sub-problems: _motion planning_ and _motion
imitation_. The motion planning phase is designed to quickly compute a
feasible motion plan using approximated dynamics, while the motion imitation
stage learns a control policy to imitate the generated motion plan. We
evaluated the proposed algorithm on two challenging control problems, _Rocket
Navigation_ and _Quadruped Locomotion_ , which have difficult reward functions
and dynamics. The experimental results indicate that our approach can discover
effective control policies for both problems while the baseline formulation
makes negligible progress on these problems.

Although the proposed framework is designed to solve general control problems,
it still requires some domain-specific knowledge. For instance, the motion
planning stage for the quadrupedal locomotion problem requires tuning the
action representations and the additional approximated dynamics term, which
can significantly affect the performance. In the future, we plan to
investigate the proposed algorithm on more complex scenarios, such as
locomotion with obstacles or manipulation.

In this work, we studied the proposed algorithm on static environments with
the fixed starting state. One interesting future research direction is to
extend the motion planning phase to learn a resilient motion plan that can
cope with dynamic perturbations. For instance, we can learn a large number of
motion plans for a quadrupedal robot that can deal with a wide range of
external perturbations. This might require large-scale reinforcement learning
to imitate many motor skills simultaneously.

## ACKNOWLEDGMENT

This work is supported by the National Science Foundation under Award
#2024768.

## References

  * [1] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al., “Mastering the game of go with deep neural networks and tree search,” nature, vol. 529, no. 7587, pp. 484–489, 2016.
  * [2] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al., “Mastering the game of go without human knowledge,” nature, vol. 550, no. 7676, pp. 354–359, 2017\.
  * [3] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, “Sim-to-real: Learning agile locomotion for quadruped robots,” arXiv preprint arXiv:1804.10332, 2018.
  * [4] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine, “Learning to walk via deep reinforcement learning,” arXiv preprint arXiv:1812.11103, 2018.
  * [5] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” Science Robotics, vol. 4, no. 26, 2019.
  * [6] M. Wermelinger, P. Fankhauser, R. Diethelm, P. Krüsi, R. Siegwart, and M. Hutter, “Navigation planning for legged robots in challenging terrain,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1184–1189, IEEE, 2016.
  * [7] D. Belter, P. Łabęcki, and P. Skrzypczyński, “Adaptive motion planning for autonomous rough terrain traversal with a walking robot,” Journal of Field Robotics, vol. 33, no. 3, pp. 337–370, 2016.
  * [8] J. Norby and A. M. Johnson, “Fast global motion planning for dynamic legged robots,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3829–3836, IEEE, 2020.
  * [9] N. Ratliff, M. Zucker, J. A. Bagnell, and S. Srinivasa, “Chomp: Gradient optimization techniques for efficient motion planning,” in 2009 IEEE International Conference on Robotics and Automation, pp. 489–494, IEEE, 2009\.
  * [10] V. Megaro, B. Thomaszewski, M. Nitti, O. Hilliges, M. Gross, and S. Coros, “Interactive design of 3d-printable robotic creatures,” ACM Transactions on Graphics (TOG), vol. 34, no. 6, pp. 1–9, 2015.
  * [11] F. Farshidian, M. Neunert, A. W. Winkler, G. Rey, and J. Buchli, “An efficient optimal planning and control framework for quadrupedal locomotion,” in 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 93–100, IEEE, 2017.
  * [12] P. Fankhauser, M. Bjelonic, C. D. Bellicoso, T. Miki, and M. Hutter, “Robust rough-terrain locomotion with a quadrupedal robot,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5761–5768, IEEE, 2018.
  * [13] J. Carius, R. Ranftl, V. Koltun, and M. Hutter, “Trajectory optimization for legged robots with slipping motions,” IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 3013–3020, 2019.
  * [14] C. Mastalli, I. Havoutis, M. Focchi, D. G. Caldwell, and C. Semini, “Motion planning for quadrupedal locomotion: Coupled planning, terrain mapping, and whole-body control,” IEEE Transactions on Robotics, vol. 36, no. 6, pp. 1635–1648, 2020.
  * [15] Y. Ding, C. Li, and H.-W. Park, “Kinodynamic motion planning for multi-legged robot jumping via mixed-integer convex program,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3998–4005, IEEE, 2020.
  * [16] N. Kohl and P. Stone, “Policy gradient reinforcement learning for fast quadrupedal locomotion,” in IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA’04. 2004, vol. 3, pp. 2619–2624, IEEE, 2004.
  * [17] A. Iscen, K. Caluwaerts, J. Tan, T. Zhang, E. Coumans, V. Sindhwani, and V. Vanhoucke, “Policies modulating trajectory generators,” in Conference on Robot Learning, pp. 916–926, PMLR, 2018.
  * [18] Y. Yang, K. Caluwaerts, A. Iscen, T. Zhang, J. Tan, and V. Sindhwani, “Data efficient reinforcement learning for legged robots,” in Conference on Robot Learning, pp. 1–10, PMLR, 2020.
  * [19] S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan, “Learning to walk in the real world with minimal human effort,” arXiv preprint arXiv:2002.08550, 2020\.
  * [20] V. Tsounis, M. Alge, J. Lee, F. Farshidian, and M. Hutter, “Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3699–3706, 2020.
  * [21] Z. Xie, X. Da, B. Babich, A. Garg, and M. van de Panne, “Glide: Generalizable quadrupedal locomotion in diverse environments with a centroidal model,” arXiv preprint arXiv:2104.09771, 2021.
  * [22] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” Science robotics, vol. 5, no. 47, 2020.
  * [23] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne, “Deepmimic: Example-guided deep reinforcement learning of physics-based character skills,” ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp. 1–14, 2018.
  * [24] X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,” arXiv preprint arXiv:2004.00784, 2020.
  * [25] D. Jain, A. Iscen, and K. Caluwaerts, “Hierarchical reinforcement learning for quadruped locomotion,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7551–7557, IEEE, 2019.
  * [26] D. Jain, A. Iscen, and K. Caluwaerts, “From pixels to legs: Hierarchical learning of quadruped locomotion,” arXiv preprint arXiv:2011.11722, 2020\.
  * [27] X. B. Peng, G. Berseth, K. Yin, and M. Van De Panne, “Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning,” ACM Transactions on Graphics (TOG), vol. 36, no. 4, pp. 1–13, 2017.
  * [28] Y.-S. Luo, J. H. Soeseno, T. P.-C. Chen, and W.-C. Chen, “Carl: Controllable agent with reinforcement learning for quadruped locomotion,” ACM Transactions on Graphics (TOG), vol. 39, no. 4, pp. 38–1, 2020.
  * [29] T. Jurgenson and A. Tamar, “Harnessing reinforcement learning for neural motion planning,” arXiv preprint arXiv:1906.00214, 2019.
  * [30] S. Luo, H. Kasaei, and L. Schomaker, “Self-imitation learning by planning,” arXiv preprint arXiv:2103.13834, 2021.
  * [31] F. Xia, C. Li, R. Martín-Martín, O. Litany, A. Toshev, and S. Savarese, “Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation,” arXiv preprint arXiv:2008.07792, 2020\.
  * [32] J. Yamada, Y. Lee, G. Salhotra, K. Pertsch, M. Pflueger, G. S. Sukhatme, J. J. Lim, and P. Englert, “Motion planner augmented reinforcement learning for robot manipulation in obstructed environments,” arXiv preprint arXiv:2010.11940, 2020.
  * [33] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017\.
  * [34] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu, “Stable baselines.” https://github.com/hill-a/stable-baselines, 2018.
  * [35] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540, 2016.
  * [36] E. Catto, “pybox2d,” URL https://github. com/pybox2d/pybox2d, vol. 11, 2012\.
  * [37] X. Wang, “Laikago, unitree robotics.” https://www.unitree.com/products/laikago.
  * [38] E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” 2016.


# Chiral emission of vortex microlasers enabled by collective modes of guided
resonances

Ye Chen1,# Mingjin Wang2,4,5,# Jiahao Si2 Zixuan Zhang1 Xuefan Yin1 Jingxuan
Chen2 NianYuan Lv1 Chenyan Tang2 Wanhua Zheng2,4,5,6,7,∗ Yuri Kivshar3,∗ Chao
Peng1,8∗

1State Key Laboratory of Advanced Optical Communication Systems and Networks,
School of Electronics & Frontiers Science Center for Nano-optoelectronics,
Peking University, Beijing 100871, China
2Laboratory of Solid State Optoelectronics Information Technology, Institute
of Semiconductors, CAS, Beijing 100083, China
3Nonlinear Physics Centre, Research School of Physics, The Australian National
University, Canberra ACT 2601, Australia
4State Key Laboratory on Integrated Optoelectronics, Institute of
Semiconductors, CAS, Beijing 100083, China
5Hangzhou Institute for Advanced Study, University of Chinese Academy of
Sciences, Hangzhou 310024, China
6College of Future Technology, University of Chinese Academy of Sciences,
Beijing 101408, China
7Center of Materials Science and Optoelectronics Engineering, University of
Chinese Academy of Sciences, Beijing 100049, China
8Peng Cheng Laboratory, Shenzhen 518055, China

#These authors contributed equally to this work.
∗To whom correspondence should be addressed; E-mail: Wanhua Zheng
(whzheng@semi.ac.cn), Yuri Kivshar (yuri.kivshar@anu.edu.au), Chao Peng
(pengchao@pku.edu.cn).

> Vortex lasers have attracted substantial attention in recent years owing to
> their wide array of applications such as micromanipulation, optical
> multiplexing, and quantum cryptography. In this work, we propose and
> demonstrate chiral emission of vortex microlaser leveraging the collective
> modes from omnidirectionally hybridizing the guided mode resonances (GMRs)
> within photonic crystal (PhC) slabs. Specifically, we encircle a central
> uniform PhC with a heterogeneous PhC that features a circular lateral
> boundary. Consequently, the bulk GMRs hybridize into a series of collective
> modes due to boundary scatterings, resulting in a vortex pattern in real
> space with a spiral phase front in its radiation. Benefiting from the long
> lifetime of GMRs as quasi-bound state in the continuum and using asymmetric
> pumping to lift the chiral symmetry, we demonstrate stable single-mode
> lasing oscillation with a low optical pumping threshold of
> $18~{}\mathrm{kW/cm^{2}}$ at room temperature. We identify the real-space
> vortex through polarization-resolved imaging and self-interference patterns,
> showing a vivid example of applying collective modes to realize compact and
> energy-efficient vortex microlasers.

## Introduction

An optical vortex is a beam of photons that travels with a helical phase
wavefront and carries quantized orbital angular momentum (OAM) [?]. Due to
their unique ability to spatially differentiate photons, optical vortices find
applications in optical sensing [?], micromanipulation [?,?], and optical
communications in the classical [?,?,?] and quantum [?] realms. In recent
years, many efforts have paid to shrinking vortex generators to a compact
footprint to facilitate chip-scale data transmission and quantum processing
[?,?,?,?,?,?,?,?,?,?]. Exciting advances in vortex microlasers have emerged
using phase singularities in real [?,?,?,?,?,?,?,?,?,?] or momentum [?,?,?]
spaces. For example, in real-space implementations, microring resonators
incorporated angular grating structures [?,?,?,?,?] and vertical-cavity
surface-emitting devices [?,?,?] using planar spiral nanostructures extract
the OAM modes into free space. Although real-space vortex microlasers are
compact with notable emission directionality and great efficiency, they face
challenges from significant scattering losses that reduce the quality factors
($Q$s) of laser cavities. In contrast, momentum-space implementations utilize
the Pancharatnam-Berry (PB) phase in PhC slabs to generate vortex beams [?].
By employing photonic bound states in the continuum (BICs) [?,?,?], namely, a
class of discrete states with infinite lifetimes embedded in the radiation
continuum, momentum-space vortex lasers exhibit low lasing thresholds due to
the high-$Q$ nature of BICs [?], but they are relatively bulky in size and
have difficulties in supporting high-order vertices since the lattice symmetry
restrains the PB phases.

Recently, the advance of vortex generation in a photonic disclination cavity
shows the remarkable light control capability of periodic photonic structures
[?], inspiring the potential of PhC to achieve spiral phases in real space,
even though the light path is not explicitly defined. It was reported that a
group of individual resonances can collectively oscillate as a whole and
exhibit distinctive features that are different from single resonances
[?,?,?,?,?] . For instance, an array of nanoparticles with Mie resonances
localized in real space can couple via the far field, leading to collective
modes in the momentum space. According to the law of duality, we expect that
the GMRs with discrete momentum can form collective modes in real space,
thereby paving the way for compact and low-threshold vortex microlasers for
on-chip integration.

In this work, we propose and demonstrate chiral emission of vortex microlaser
by utilizing a class of collective modes of GMRs in PhC slabs, in which the
optical modes circulate in real space and radiate as helical phase fronts.
Specifically, we design a square lattice PhC encircled by a circular lateral
boundary on an active InGaAsP membrane. As a consequence of the circular
boundary’s scattering, the GMRs that originally propagated along specific
Bloch wavevectors in bulk PhC omnidirectionally hybridize into two-fold
degenerate collective modes with opposite chirality, and they are favorable
for lasing because of the protection of quasi-BICs. By applying the technique
of asymmetric pumping to break chiral symmetry, we experimentally demonstrate
stable single-mode lasing behavior with an optical-pumping threshold of
$18~{}\mathrm{kW/cm^{2}}$, and identify the phase vortex features in real
space through polarization-resolved imaging and self-interference patterns.

## Principle and design

We schematically present our vortex microlaser design in Fig. 1a, in which
square latticed air holes are patterned on a suspended InGaAsP membrane. Six
layers of multiple quantum wells (MQWs) are embedded in the center of the
membrane as optical gain materials at the C-band of the telecom wavelength.
The details of the MQW epitaxial wafer are presented in the Suppl. Section 1.
We use the photonic bandgap (PBG) to localize the light in the transverse
direction [?]. Namely, the central region (region A with the radius $r_{A}$)
is surrounded by a heterogeneous PhC (region B with the radius $r_{B}$) to
form a circular boundary of PBG for lateral confinement. Then, the scatterings
at the circular boundary would drive the bulk GMRs that are aligned with the
iso-frequency contour in momentum space to collectively oscillate. As a
result, we found a set of collective modes that counter-intuitively circulate
in real space, enabling a helical phase wavefront upon the radiation in the
out-of-plane direction.

To elaborate on the principle of vortex generation, we first present the
detailed geometry and band diagram of the designed PhCs (Fig. 1b), focusing on
the transverse electric band A (TE-A) of region A in the vicinity of the 2nd-
order $\Gamma$ point within the bandgap of region B. It is worth noting that,
in infinite periodic PhCs, the bulk GMRs $|k\rangle$ and $|k^{\prime}\rangle$
with Bloch wavevectors $k\neq k^{\prime}$ are mutually orthogonal.
Consequently, they propagate independently in straight paths along specified
directions due to the translational symmetry of the PhC (left panel, Fig. 1c),
preventing them from altering propagation directions to create a well-defined
closed-loop in the real space like in microring resonators. However, the
introduction of a non-uniform boundary in the PhC can give rise to additional
momenta $\Delta k=k-k^{\prime}$ [?,?], leading to the coupling between
$|k\rangle$ to $|k^{\prime}\rangle$ according to the scattering effect (right
panel, Fig. 1c). Essentially, the modes within the circular boundary PhC can
be viewed as a collectively linear combination of a set of bulk GMRs due to
the boundary’s scattering (known as collective modes), allowing for well-
defined energy flows in the real space to generate unique phase fronts in the
radiation (lower-panel, Fig. 1c).

Specifically, we present a 3D visualization of the TE-A band’s dispersion in
Fig. 1d, revealing a quadratic curvature near the $\Gamma$ point. To ensure
the harmonic stability of collective modes in the time domain, the GMRs
involved in the linear combination must possess identical frequencies, namely
they should align with the iso-frequency contour of the band. As depicted in
Fig. 1d (red circle), this contour is nearly circular with a radius of $\Delta
k_{r}=\mu_{m}^{l}/R$, where $\mu_{m}^{l}$ denotes the $l$-th zeros of the
$m$-th Bessel function $J_{m}(\mu_{m}^{l})=0$ and $R$ denotes the cavity’s
radius in real space. When $R$ is sufficiently large, the real space boundary
tends towards roundness, irrespective of the discreteness of the PhC lattice.
Simultaneously, the contour of $\Delta k_{r}$ in momentum space shrinks,
resulting in nearly isotropic band dispersion. Based on our design featuring a
circular boundary of $R=10a$, we observe that both the boundary and dispersion
remain almost invariant under the rotations of any angle. Consequently, the
bulk GMRs ranging from $|k,0^{\circ}\rangle$ to $|k,360^{\circ}\rangle$ on the
iso-frequency contour propagate and scatter omnidirectionally as illustrated
schematically in Fig. 1d (see Suppl. Section I for detail discussions), which
facilitates their hybridization into a set of collective modes that retains
continuous rotation symmetry akin to a microring resonator, leading to paired,
degenerate modes rotating clockwise (CW) and counter-clockwise (CCW) in real
space without the requirements for physical light transmission paths.
Furthermore, through the diffraction of the PhC lattice, a spiral phase front
of radiation, namely a vortex beam, is generated.

The collective modes are identified via numerical simulation (COMSOL
Multiphysics, Fig. 2a), consistent with the analytical solutions derived from
the coupled-wave theory (CWT) [?,?] in polar coordinates (see the details in
Suppl. Section 2). The slow-varying envelopes of collective modes are
represented by the Bessel function $J_{m}(r)$, denoted by the quantum numbers
$m$ and $l$ in the azimuth and radial directions, respectively. It’s worth
noting that, except for $m=0$, each quantum number $(m,l)$ corresponds to a
pair of two-fold degenerate modes that rotate in either CW or CCW directions.
As an illustration, we choose the CW mode of $(1,1)$ for further analysis,
which is our candidate for lasing oscillation. We calculate the snapshot
magnetic field at fixed time intervals within one oscillation cycle, ranging
from $0$ to $T$, clearly demonstrating the rotating in real space (Fig. 2b).
This rotational motion generates a helical phase front spanning from $0$ to
$2\pi$ in out-of-plane radiation, akin to the motion of a propeller driving
water. Furthermore, the distribution of electrical field strength ($|E|$) is
plotted in the center of Fig. 2b, showing a donut-shape pattern in real space,
as a consequence of temporal averaging of energy flow. When a linear polarizer
is placed before observing the donut beam, two lobes with equal density
intensities would appear along the direction perpendicular to the polarizer’s
major axis, thus providing a distinctive feature to identify the vortex beam
(Fig. 2c).

The circular collective modes have the potential to achieve low-threshold
lasing because their radiation losses can be suppressed by using quasi-BICs
[?,?]. By tuning the structural parameters $a$ and $r_{A}$, we found eight
off-$\Gamma$ BICs and one symmetry-protected BIC in the momentum space,
carrying integer topological charges on their far-field polarization (Fig.
3a). We calculate the $Q$s of the TE-A band in the bulk PhC (Fig. 3b), showing
that the lifetime diverges to infinity at $k_{\text{BIC}}=0.034$, giving rise
to a ring of high $Q$ values at $|k_{\text{BIC}}|=0.034$. As we reported
previously [?,?,?], the radiations in finite-size PhC can be highly
directional, determined by the sizes and shapes of the boundaries. With a
circular boundary, the radiations show equal weights along the contour of
$\Delta k_{r}$, and they can be effectively suppressed by aligning the in-
plane momentum of radiation with the high-$Q$ ring as $\Delta
k_{r}=k_{\text{BIC}}$ (Fig. 3a). Further, we fine-tune the structural
parameters of circular the PBG in region B to best suppress the radiation
while keeping the mode wavelength fixed at $\sim 1550$ nm. As presented in
Fig. 3c, we obtain an optimal $Q$ of $\sim 10^{6}$ at the parameters of
$a=537$ nm, $r_{A}=164$ nm, and $r_{B}=154$ nm, which is sufficient for low-
threshold lasing.

When the PhC cavity is perfectly round, the boundary and dispersion own
continuous rotational symmetry, allowing for the two-fold degeneracy of CW and
CCW modes, represented by $|\sigma_{\pm}\rangle$, respectively, which have
identical complex eigenfrequencies $\lambda_{\pm}$ described by
$\mathbf{H}|\sigma_{\pm}\rangle=\lambda_{\pm}|\sigma_{\pm}\rangle$, where
$\mathbf{H}$ denotes the Hamiltonian of the system. However, in realistic
samples, imperfections in fabrication inevitably affect the real part of the
permittivity, while the asymmetric alignment during optical pumping introduces
non-uniform gain, leading to a spontaneous chiral symmetry breaking depicted
by a perturbation of Hamiltonian $\Delta\mathbf{H}$ (Fig. 3d). Accordingly,
the CW and CCW modes would experience a non-Hermitian mutual coupling of
$\kappa=\langle\sigma_{\mp}|\mathbf{\Delta H}|\sigma_{\pm}\rangle$ that lifts
the degeneracy, giving rise to different frequencies and $Q$s of
$|\sigma_{\pm}\rangle$ [?,?,?,?,?], confirmed by the numerical simulations
(Fig. 3e). When combined with the optical gain of InGaAsP material that is
depicted by the photoluminescence (PL) spectrum (as schematically shown in
Fig. 3f), the difference in the wavelengths and $Q$s would make one of
$|\sigma_{\pm}\rangle$ prevailing over the mode competition, to enable chiral
emission of single-mode lasing under appropriate asymmetric pumping
conditions. It is worth noting that, by tuning the coupling strength, the
collective modes $|\sigma_{\pm}\rangle$ can possibly operate at the exception
point (EP) region at which their frequencies are the same but $Q$s are
different to support directional lasing as reported in microring lasers [?,?].
However, such a specific condition is not mandatory for our design since the
mode difference is already sufficient to support single-mode oscillation. More
discussions are presented in Suppl. Section 3.

## Sample fabrication and experimental setup

To verify the principle and design of our vortex microlaser, we fabricate the
samples from an InGaAsP MQWs wafer on the InP substrate. The PhC
nanostructures are exposed by electron beam lithography (EBL) followed by
inductively coupled plasma (ICP) dry etching. We subsequently remove the
sacrificial layer underneath the MQWs by using hydrochloric acid to restore
the mirror symmetry in the $z$ direction required by the BICs (See Methods for
details). The fabricated sample is observed by using the scanning electron
microscope (SEM). The top view (Fig. 4a) shows that the microlaser has a total
footprint of $26.8\times 26.8~{}\mu$m2, in which the regions A and B are
separated by the circular dashed line. We further cleave a part of the PhC by
a focused ion beam (FIB), to observe the detailed top and side views as
presented in Fig. 4b and c, respectively. The structural parameters of PhC are
characterized as the periodicity of $a=537$ nm, diameter of $r_{A}=164$ nm,
and height of $h=622$ nm, matching well with our design.

The schematic of the measurement setup for vortex beam observation is shown in
Fig. 4d. The sample is optically pumped at room temperature by using a pulsed
laser at the wavelength of 1064 nm, with a repetition rate of 10 kHz and a
pulse duration of 2 ns. Before shining upon the sample, the pump laser beam is
tailored to be transversely asymmetric by an iris diaphragm, to enable the
asymmetric pumping condition required by single-mode lasing. Further, the pump
beam is attenuated by an absorptive neutral density (ND) filter with an
attenuation ratio of -24 dB to avoid sample damage. Accordingly, the average
pump power is measured by a power meter before the ND filter, to minimize
power fluctuation and noise disturbance, particularly when the pump power is
low.

The sample reaches lasing oscillation when a sufficiently strong pump beam is
focused on the sample surface by an objective lens (X50), consequently,
generating a vortex beam emitting in the vertical direction. The vortex and
reflected pump beams are collected by the same objective lens and then pass
through a lens (L1) to form a real space image at its focal point. Further,
the real space image is enlarged 6 times by a 4$f$ system composed of the
lenses L2 and L3, following a long-pass filter with a cut-on wavelength of
1300 nm to get rid of the reflected pump beam. Finally, the vortex beam is
split and recombined to form a Mach-Zehnder interferometer for off-center
self-interference observation, which is composed of two beam splitters (BSs)
and two gold-coated mirrors (M1 and M2). The optical path difference is
controlled by adjusting the attitude angle of M2. The real-space image of the
vortex beam is captured by the complementary metal oxide semiconductor (CMOS)
camera, enabling the characterizing of its real-space pattern, polarization
distribution, as well as self-interference fringes to identify the expected
spiral phase features. At the same time, the emission spectrum is detected by
a monochromator in the range from 1540 nm to 1570 nm with a resolution of
$\sim$ 0.08 nm. See Methods for more details about the measurement system.

## Experimental results

To achieve lasing oscillation, we first apply a symmetric pump beam with a
circular spot (upper panel, Fig. 5a) whose diameter is about
$5~{}\mu\mathrm{m}$. At a low pump power of 16 kW/cm2, spontaneous emission is
notable, along with two small peaks near the wavelength of 1555 nm,
corresponding to the CW and CCW collective modes that compete in lasing. By
increasing the pump power to 18 kW/cm2, lasing oscillation is observed,
characterized by the distinct lasing peaks amidst the strongly suppressed
spontaneous emission spectrum. It is important to highlight that the peaks of
CW and CCW modes coexist in the lasing spectrum with different but comparable
magnitudes, indicating a lack of dominance in mode competition. When we
further elevate the pump power to a higher value of 48 kW/cm2, the emission
power increases significantly, while the presence of the two peaks persists,
suggesting the laser is not operating under single-mode oscillation.

The above observation aligns with our theoretical prediction, showing that
spontaneous chiral symmetry breaking raised by fabrication imperfection is not
adequate for one of the chiral modes to dominate the mode competition.
Consequently, we opt for asymmetric pumping with an elliptical spot on the
sample to deliberately break the chiral symmetry, which should give rise to
different $Q$ values for the collective modes in opposite chirality as we
proposed in the theory. The establishment process of lasing oscillation is
similar to the case of symmetric pumping during the increasing of the pump
power from 16 kW/cm2 to 48 kW/cm2 (lower panel, Fig. 5a). Below the lasing
threshold, we still found two peaks residing in the spontaneous emission
background. Differently, only one lasing peak is evident in the spectrum when
the pump power exceeds the lasing threshold, proving the suppression of
another chiral mode during mode competition that facilitates single-mode
lasing and vortex generation.

Further, we measure the pump-emission power curve of the sample at room
temperature (Fig. 5b), in which three data points are included (labeled A to
C), each corresponding to different pump power densities with their spectra
illustrated in the lower panel of Fig. 5a, respectively. Notably, a low lasing
threshold of $18~{}\mathrm{kW/cm^{2}}$ is observed, attributed to the high-$Q$
characteristics of the collective modes of quasi-BICs. The inset in Fig. 5b
displays the real-space pattern of the vortex microlaser, revealing a donut
shape that agrees well with our theoretical predictions. The pattern is
rotationally uniform along the azimuth direction and features one peak in the
radial direction, thus corresponding to the Bessel function of $(m,l)=(1,1)$.
We further insert a linear polarizer before the CMOS camera to characterize
the polarization distribution of the vortex beam along the orientation of
$0^{\circ}$, $45^{\circ}$, $90^{\circ}$, and $135^{\circ}$, respectively. As
shown in Fig. 5c, the pattern manifests identical densities along arbitrary
directions, confirming the rotational invariability of the vortex beam. For a
given polarized direction, the dominant polarization aligns along the azimuth
rather than the radial direction, which is quite different from the micro-ring
vortex laser. This is because the collective modes primarily propagate along
the axis of the circle as transverse waves but gradually rotate their
directions due to boundary scatterings. It is also worth noting that besides
the lasing mode of $(1,1)$, other orders of collective modes have also been
observed in the experiment. However, they are less favorable for lasing since
their wavelengths and $Q$s values are not optimized in purpose. More data and
discussion about the vortex lasing are presented in Suppl. Section 4.

Finally, we validate the spiral phase of the vortex beam by using the self-
interference technique. We apply asymmetric pumping to select one of the
chiral modes for lasing. Subsequently, we split the vortex beam evenly into
two parts and overlapped them in real space based on a Mach-Zehnder
interferometer setup. A pair of fork patterns with opposite orientations are
observed, showing the distinctive feature of phase singularity. The
dislocation points of these forks align with the center of each vortex beam,
at which a single fringe separates into two individual branches, confirming
the vortex beam’s quantum number of $|m|=1$. Notably, the orientation of the
fork pattern signifies the chirality of the lasing mode, which can be either
CW or CCW, depending on how chiral symmetry is broken. As a result, we
ascertain that the data shown in Fig. 5d represent the lasing of the CW mode,
while we also observe the lasing of the CCW mode in another sample with the
fork orientation flipped. It is worth noting that phase vortex, polarization
vortex, and their combination can exhibit different features in self-
interference patterns. For additional data and discussion, see Suppl. Section
5 for the details.

## Discussion

Collective modes are ubiquitous phenomena in photonic [?,?,?], plasmonic
[?,?,?], and quantum wave systems [?,?], which refer to a cluster of
individual modes collectively oscillating as a whole but showing distinctive
differences in their field distribution and $Q$s than the single ones. The key
to creating collective modes is to induce couplings between the individual
modes. Although most reported collective modes are realized by coupling tight-
binding modes in real space to enable collective behavior in momentum space,
we propose that a similar philosophy also holds for the momentum-space
localized mode (such as GMRs) to generate exotic distributions in real space,
according to the duality between the real and momentum spaces. We anticipate
that, beyond boundary scatterings, other mechanisms such as random disorders,
defects, and superlattices can also contribute to the couplings between single
individual modes even in non-Hermitian or non-Abelian manners, thus leading to
rich and unexplored phenomena of collective oscillation, showing as a direct
example in photonic to reflect the famous remarks of “more is different”.

From a technical perspective, the collective modes of GMRs could be a
promising way for on-chip vortex generation towards practical applications.
Firstly, compared to the micro-ring vortex laser [?,?,?,?,?], our approach
offers a lower lasing threshold due to the high-$Q$ characteristics of quasi-
BICs. Moreover, its surface-emitting nature [?,?,?,?,?] can facilitate
sufficiently high output power, which is crucial for optical manipulation,
detection, and communication. Secondly, the proposed architecture is readily
compatible with electrical pumping [?,?,?], which is an important stride
toward practical lasers. This feasibility arises because the strict $z$-mirror
symmetry required by off-$\Gamma$ BICs can be relaxed if quasi-BICs near a
single symmetric-protected BIC can give sufficient high $Q$s to sustain lasing
oscillation, therefore the vortex microlaser can be fabricated by using
standard semiconductor laser process.

## Conclusion

In summary, we propose and demonstrate an on-chip vortex microlaser by
utilizing the collective modes of GMRs. Leveraging the scatterings from
circular boundaries, a succession of bulk GMRs that are protected by quasi-
BICs collectively oscillate in real space and feature a spiral phase front in
their radiation. We achieve a low lasing threshold of
$18~{}\mathrm{kW/cm^{2}}$ at room temperature, and further, observe single-
mode lasing behavior attributed to the spontaneous breaking of chiral symmetry
under fabrication imperfections and asymmetric pumping. The chiral emission
nature of the lasing beam is characterized by using polarization-resolve
imaging and self-interference technique, confirming the presence of the spiral
phase front. Our findings demonstrate that the coupling effect in momentum
space enables the emergence of exotic collective modes in real space, thus
offering a promising approach for on-chip vortex generation to fascinate many
applications ranging from optical imaging, sensing, and manipulating, to
large-volume optical communication.

## Supplementary materials

## Methods

## Numerical simulations

The numerical simulation is based on the finite-element method (FEM, COMSOL
Multiphysics). The bandstructures in Fig.1b and the quality factors of TE-A
band in Fig. 3a and b, are calculated in the 3D unit-cell structures with
Floquet periodic boundaries on the sidewalls and perfectly matched layers on
both top and bottom. Fig.2 and Fig.3c are performed based on 3D simulations of
our microlaser design and the direct results obtained by simulations are two-
fold degenerate eigenmodes independent of time. We remix the two-fold
degenerate eigenmodes with a ratio of $1:e^{\pm i\frac{\pi}{2}}$ to generate
CW or CCW modes. Though different remixing ratios are also permitted
theoretically, these cases can be regarded as mode splitting due to asymmetric
pumping. More details are presented in Suppl. Section 3.

## Sample fabrication

The vortex microlaser with undercut PhC structure is designed on an InP-based
epi-wafer. As shown in Fig.S1a, the active region consisting of six $7.5$ nm
compressively strained InGaAsP quantum wells and seven $12$ nm lattice-matched
InGaAsP barriers, is sandwiched by undoped $246.5$ nm InGaAsP cladding layers.
An undoped $1500$ nm InP is used as wet-etching layer for the fabrication of
undercut structure, leading to $z$-direction symmetry for the whole PhC. An
undoped $100$ nm InGaAsP underneath the etching layer is designed as etching
stop layer. The PhC structures locate at the top most of the epi-wafer. A
circular central region with a radius of $10\times a$ (periodicity $a=537$ nm,
and radius $r_{A}=164$ nm) is surrounded by a $50\times 50$ array of square
latticed circular holes (periodicity $a=537$ nm, and radius $r_{B}=154$ nm).
The PhC structures are patterned by EBL and inductively coupled plasma (ICP)
in Cl2 at $240^{\circ}$C. $1:1=$ HCl $:$ H2O solution is used to etch InP for
the undercut structure under a wet-etching rate of $700$ nm$/$min at room
temperature.

## Measurement and data processing

We use a 1064-nm pulsed laser with a repetition frequency of 10 kHz and a
pulse duration of 2 ns, to pump our vortex microlaser at room temperature. As
shown in Fig.4d, a $\times 50$ objective lens (IOPAMI137150X-NIR, Mitutoyo) is
used to converge the pump light to a spot with the diameter of $\sim 5$ um and
also collect the vortex beam generated from the sample. The following lens L1
($f=200~{}\mathrm{mm}$) confocal with the objective lens, as well as L2
($f=50~{}\mathrm{mm}$) and L3 ($f=300~{}\mathrm{mm}$) forming a 4$f$ system,
enlarges the real space image of samples by 300 times in total to generate
more in-plane optical path difference in the self-interference experiment. The
real space images are captured by an InGaAs infrared CMOS camera (IMX990,
Sony). Meanwhile, they are also focused into a monochromator with a 600g/mm
optical grating and an infrared array detector (IsoPlane SCT320 and NIRvana
640, Princeton Instruments). The images, after deducting CMOS intrinsic noise
and ambient background noise, are applied data-smoothing with a window size of
30 pixels.

## References

## References

  * 1. Allen, L., Beijersbergen, M. W., Spreeuw, R. J. C. & Woerdman, J. P.  Orbital angular momentum of light and the transformation of Laguerre-Gaussian laser modes.  _Phys. Rev. A_ 45, 8185–8189  (1992).
  * 2. Lavery, M. P. J., Speirits, F. C., Barnett, S. M. & Padgett, M. J.  Detection of a Spinning Object Using Light’s Orbital Angular Momentum.  _Science_ 341, 537–540  (2013).
  * 3. Paterson, L. _et al._ Controlled Rotation of Optically Trapped Microscopic Particles.  _Science_ 292, 912–914  (2001).
  * 4. MacDonald, M. P. _et al._ Creation and Manipulation of Three-Dimensional Optically Trapped Structures.  _Science_ 296, 1101–1103  (2002).
  * 5. Wang, J. _et al._ Terabit free-space data transmission employing orbital angular momentum multiplexing.  _Nat. Photon._ 6, 488–496  (2012).
  * 6. Lei, T. _et al._ Massive individual orbital angular momentum channels for multiplexing enabled by Dammann gratings.  _Light Sci. Appl._ 4, e257–e257  (2015).
  * 7. Xie, Z. _et al._ Ultra-broadband on-chip twisted light emitter for optical communications.  _Light Sci. Appl._ 7, 18001–18001  (2018).
  * 8. Chen, B. _et al._ Integrated optical vortex microcomb.  _Nat. Photon._ (2024).
  * 9. Cai, X. _et al._ Integrated Compact Optical Vortex Beam Emitters.  _Science_ 338, 363–366  (2012).
  * 10. Liu, J. _et al._ Direct fiber vector eigenmode multiplexing transmission seeded by integrated optical vortex emitters.  _Light Sci. Appl._ 7, 17148–17148  (2017).
  * 11. Shao, Z., Zhu, J., Chen, Y., Zhang, Y. & Yu, S.  Spin-orbit interaction of light induced by transverse spin angular momentum engineering.  _Nat. Commun._ 9, 926  (2018).
  * 12. Zhou, N. _et al._ Ultra-compact broadband polarization diversity orbital angular momentum generator with 3.6 × 3.6 $M$m2 footprint.  _Sci. Adv._ 5, eaau9593  (2019).
  * 13. Sroor, H. _et al._ High-purity orbital angular momentum states from a visible metasurface laser.  _Nat. Photon._ 14, 498–503  (2020).
  * 14. Wang, X.-Y. _et al._ Vortex radiation from a single emitter in a chiral plasmonic nanocavity.  _Nanophotonics_ 11, 1905–1911  (2022).
  * 15. Mei, F. _et al._ Cascaded metasurfaces for high-purity vortex generation.  _Nat. Commun._ 14, 6410  (2023).
  * 16. Liu, Y. _et al._ Integrated vortex soliton microcombs.  _Nat. Photon._ 18, 632–637  (2024).
  * 17. Li, H. _et al._ Orbital angular momentum vertical-cavity surface-emitting lasers.  _Optica_ 2, 547–552  (2015).
  * 18. Miao, P. _et al._ Orbital angular momentum microlaser.  _Science_ 353, 464–467  (2016).
  * 19. Carlon Zambon, N. _et al._ Optically controlling the emission chirality of microlasers.  _Nat. Photon._ 13, 283–288  (2019).
  * 20. Sun, W. _et al._ Lead halide perovskite vortex microlasers.  _Nat. Commun._ 11, 4862  (2020).
  * 21. Yang, Z.-Q., Shao, Z.-K., Chen, H.-Z., Mao, X.-R. & Ma, R.-M.  Spin-Momentum-Locked Edge Mode for Topological Vortex Lasing.  _Phys. Rev. Lett._ 125, 013903  (2020).
  * 22. Zhang, Z. _et al._ Tunable topological charge vortex microlaser.  _Science_ 368, 760–763  (2020).
  * 23. Zhang, Z. _et al._ Ultrafast control of fractional orbital angular momentum of microlaser emissions.  _Light Sci. Appl._ 9, 179  (2020).
  * 24. Chen, B. _et al._ Bright solid-state sources for single photons with orbital angular momentum.  _Nat. Nanotechnol._ 16, 302–307  (2021).
  * 25. Piccardo, M. _et al._ Vortex laser arrays with topological charge control and self-healing of defects.  _Nat. Photon._ 16, 359–365  (2022).
  * 26. Zhang, Z. _et al._ Spin–orbit microlaser emitting in a four-dimensional Hilbert space.  _Nature_ 612, 246–251  (2022).
  * 27. Huang, C. _et al._ Ultrafast control of vortex microlasers.  _Science_ 367, 1018–1021  (2020).
  * 28. Mohamed, S. _et al._ Controlling Topology and Polarization State of Lasing Photonic Bound States in Continuum.  _Laser & Photon. Rev._ 16, 2100574  (2022).
  * 29. Hwang, M.-S. _et al._ Vortex nanolaser based on a photonic disclination cavity.  _Nat. Photon._ 18, 1–8  (2023).
  * 30. Hsu, C. W. _et al._ Observation of trapped light within the radiation continuum.  _Nature_ 499, 188–191  (2013).
  * 31. Zhen, B., Hsu, C. W., Lu, L., Stone, A. D. & Soljačić, M.  Topological Nature of Optical Bound States in the Continuum.  _Phys. Rev. Lett._ 113, 257401  (2014).
  * 32. Jin, J. _et al._ Topologically enabled ultrahigh-Q guided resonances robust to out-of-plane scattering.  _Nature_ 574, 501–504  (2019).
  * 33. Auguié, B. & Barnes, W. L.  Collective Resonances in Gold Nanoparticle Arrays.  _Phys. Rev. Lett._ 101, 143902  (2008).
  * 34. Kuznetsov, A. I., Miroshnichenko, A. E., Brongersma, M. L., Kivshar, Y. S. & Luk’yanchuk, B.  Optically resonant dielectric nanostructures.  _Science_ 354, aag2472  (2016).
  * 35. Limonov, M. F., Rybin, M. V., Poddubny, A. N. & Kivshar, Y. S.  Fano resonances in photonics.  _Nat. Photon._ 11, 543–554  (2017).
  * 36. Yesilkoy, F. _et al._ Ultrasensitive hyperspectral imaging and biodetection enabled by dielectric metasurfaces.  _Nat. Photon._ 13, 390–396  (2019).
  * 37. Hoang, T. X. _et al._ Collective nature of high-Q resonances in finite-size photonic metastructures.  _arXiv_ 2405.01034  (2024).
  * 38. Chen, Z. _et al._ Observation of miniaturized bound states in the continuum with ultra-high quality factors.  _Sci. Bull._ 67, 359–366  (2022).
  * 39. Notomi, M.  Theory of light propagation in strongly modulated photonic crystals: Refractionlike behavior in the vicinity of the photonic band gap.  _Phys. Rev. B_ 62, 10696–10705  (2000).
  * 40. Notomi, M.  Negative refraction in photonic crystals.  _Opt. Quantum Electron._ 34, 133–143  (2002).
  * 41. Liang, Y., Peng, C., Sakai, K., Iwahashi, S. & Noda, S.  Three-dimensional coupled-wave model for square-lattice photonic crystal lasers with transverse electric polarization: A general approach.  _Phys. Rev. B_ 84, 195119  (2011).
  * 42. Chen, Z. _et al._ Analytical theory of finite-size photonic crystal slabs near the band edge.  _Opt. Express_ 30, 14033–14047  (2022).
  * 43. Ren, Y. _et al._ Low-threshold nanolasers based on miniaturized bound states in the continuum.  _Sci. Adv._ 8, eade8817  (2022).
  * 44. Wiersig, J., Kim, S. W. & Hentschel, M.  Asymmetric scattering and nonorthogonal mode patterns in optical microspirals.  _Phys. Rev. A_ 78, 053809  (2008).
  * 45. Wiersig, J. _et al._ Nonorthogonal pairs of copropagating optical modes in deformed microdisk cavities.  _Phys. Rev. A_ 84, 023845  (2011).
  * 46. Wiersig, J.  Structure of whispering-gallery modes in optical microdisks perturbed by nanoparticles.  _Phys. Rev. A_ 84, 063828  (2011).
  * 47. Peng, B. _et al._ Chiral modes and directional lasing at exceptional points.  _Proc. Natl. Acad. Sci._ 113, 6845–6850  (2016).
  * 48. Chen, W., Kaya Özdemir, Ş., Zhao, G., Wiersig, J. & Yang, L.  Exceptional points enhance sensing in an optical microcavity.  _Nature_ 548, 192–196  (2017).
  * 49. Feng, L., Wong, Z. J., Ma, R.-M., Wang, Y. & Zhang, X.  Single-mode laser by parity-time symmetry breaking.  _Science_ 346, 972–975  (2014).
  * 50. Miroshnichenko, A. E. & Kivshar, Y. S.  Fano Resonances in All-Dielectric Oligomers.  _Nano Lett._ 12, 6459–6463  (2012).
  * 51. Yan, J., Yuan, Z. & Gao, S.  End and Central Plasmon Resonances in Linear Atomic Chains.  _Phys. Rev. Lett._ 98, 216602  (2007).
  * 52. Giannini, V., Vecchi, G. & Gómez Rivas, J.  Lighting Up Multipolar Surface Plasmon Polaritons by Collective Resonances in Arrays of Nanoantennas.  _Phys. Rev. Lett._ 105, 266801  (2010).
  * 53. Dai, X. _et al._ Two-Dimensional Double-Quantum Spectra Reveal Collective Resonances in an Atomic Vapor.  _Phys. Rev. Lett._ 108, 193201  (2012).
  * 54. Macha, P. _et al._ Implementation of a quantum metamaterial using superconducting qubits.  _Nat. Commun._ 5, 5146  (2014).
  * 55. Kodigala, A. _et al._ Lasing action from photonic bound states in continuum.  _Nature_ 541, 196–199  (2017).
  * 56. Gao, X. _et al._ Dirac-vortex topological cavities.  _Nat. Nanotechnol._ 15, 1012–1018  (2020).
  * 57. Yang, L., Li, G., Gao, X. & Lu, L.  Topological-cavity surface-emitting laser.  _Nat. Photon._ 16, 279–283  (2022).
  * 58. Contractor, R. _et al._ Scalable single-mode surface-emitting laser via open-Dirac singularities.  _Nature_ 608, 692–698  (2022).
  * 59. Luan, H.-Y., Ouyang, Y.-H., Zhao, Z.-W., Mao, W.-Z. & Ma, R.-M.  Reconfigurable moiré nanolaser arrays with phase synchronization.  _Nature_ 624, 282–288  (2023).
  * 60. Hirose, K. _et al._ Watt-class high-power, high-beam-quality photonic-crystal lasers.  _Nat. Photon._ 8, 406–411  (2014).
  * 61. Yoshida, M. _et al._ Double-lattice photonic-crystal resonators enabling high-brightness semiconductor lasers with symmetric narrow-divergence beams.  _Nat. Mater._ 18, 121–128  (2019).
  * 62. Yoshida, M. _et al._ High-brightness scalable continuous-wave single-mode photonic-crystal laser.  _Nature_ 618, 727–732  (2023).

Figure 1: Principle of collective modes (a) Schematic of the microlaser
showing vortex generation under optical pumping. The central PhC region A is
surrounded by a heterogeneous PhC region B with a circular boundary. (b) The
PhC geometry is defined by lattice constant $a$, air-hole radius $r$, and slab
thickness $h$ (left panel), which gives rise to a band structure accordingly
(right panel), where the TE-A band of region A near the 2nd-$\Gamma$ point is
embedded in the bandgap of region B. (c) The principle of boundary scattering.
The Bloch waves propagate along straight directions in an infinite periodic
PhC (left upper panel), while the boundary scatterings give additional momenta
$\Delta k$ to alter their directions (right upper panel). As a result, an
omnidirectional mixing of Bloch waves happens at the ideal circular boundary
(lower panel). (d) A 3D visualization of the TE-A band’s dispersion, showing
an isotropic iso-frequency contour at the Brillouin zone center to support the
collective modes.

Figure 2: Spiral phase in the real space  (a) The amplitudes of collective
modes in our circular PhC cavity are Bessel functions $J_{m}(r)$ with quantum
numbers $m$ and $l$ in the azimuth and radial directions, respectively (upper
panel), and their $H_{z}$ field distributions are plotted (lower panel). (b)
As the candidate for lasing oscillation, the field strength ($|E|$) of
collective mode $(1,1)$ is illustrated as a donut pattern. Moreover, the
snapshots of magnetic fields $H_{z}$ at a sequence of time intervals in one
oscillation cycle $[0,T]$ show the temporal rotating of the mode that
generates a spiral phase from $0$ to $2\pi$. (c) The polarization-solved
distribution of collective mode $(1,1)$ exhibits two lobes with equal density
intensities as a distinct feature of the vortex beam.

Figure 3: High-$Q$ and chiral nature of collective modes (a) The $Q$
distribution in momentum space for bulk GMRs in the PhC slab, showing eight
off-$\Gamma$ BICs and one symmetry-protected BIC appear near the Brillouin
zone center, represented by integer topological charges where the $Q$s diverge
to infinite. (b) The detailed $Q$s of the TE-A band, the off-$\Gamma$ BICs are
found at $|k_{\text{BIC}}|=0.034$. (c) The $Q$s of collective mode $(1,1)$ in
the PhC with circular boundary, the $Q$s are optimized by tuning structural
parameters while keeping the wavelength almost fixed at $\sim 1550$, the
maximum $Q$$\sim 1\times 10^{6}$ appears at $a=537$ nm. (d) Under asymmetric
pumping, the CW and CCW collective modes behave differently under an
elliptical pump spot (blue). (e) The real and imaginary eigenfrequencies under
asymmetric pumping, showing differences in the wavelengths and $Q$s for mode
competition. (f) The principle of single-mode oscillation. The differences of
CW and CCW collective modes upon the PL gain specturm enable one chiral mode
to prevail in lasing.

Figure 4: Fabricated sample and measurement setup (a-c) The scanning electron
microscope images of the fabricated sample show the hetero-structures PhC
consisting of region A and region B from top and side views. FIB is used to
cleave the PhC for a side view of the undercut structure. (d) Schematic of the
experimental setup. Iris, iris diaphragm; ND, absorptive neutral density; BS,
beam splitter; L, lens; Pol, linear polarizer; M, gold-coated mirror.

Figure 5: Characterization of the vortex lasing (a) The establishing process
of lasing oscillation by increasing the pump powers from
$16~{}\mathrm{kW/cm^{2}}$ to $48~{}\mathrm{kW/cm^{2}}$, for symmetric (upper
panel) and asymmetric (lower panel) pump conditions, respectively. Three
points from A to C are marked, showing the status from spontaneous emission,
threshold lasing, to single-mode lasing. (b) The power curve of the vortex
microlaser is measured, indicating a low threshold of
$18~{}\mathrm{kW/cm^{2}}$. The inset shows a donut shape of the lasing beam in
real space. (c) The polarization-resolved distribution of the vortex beam
along $0^{\circ}$, $45^{\circ}$, $90^{\circ}$, and $135^{\circ}$,
respectively. (d) The off-center self-interference pattern of the vortex beam
is observed, showing two reversely oriented forks (marked by white arrows)
located at the phase singularities as a distinct feature of a phase vortex
beam.