Vishnoi and Simoni

# Surrogate-based Real-time Curbside Management for Ride-hailing and Delivery
Operations

Suyash C. Vishnoi, Corresponding Author
Ph.D. Candidate
Department of Civil  Architectural  and Environmental Engineering
The University of Texas Austin
<EMAIL_ADDRESS>
Michele D. Simoni
Assistant Professor
Department of Urban Planning and Environment
KTH Royal Institute of Technology
<EMAIL_ADDRESS>
## Abstract

The present work investigates surrogate model-based optimization for real-time
curbside traffic management operations. An optimization problem is formulated
to minimize the congestion on roadway segments caused by vehicles stopping on
the segment (e.g., ride-hailing or delivery operations) and implemented in a
model predictive control framework. A hybrid simulation approach where main
traffic flows interact with individually modeled stopping vehicles is adopted.
Due to its non-linearity, the optimization problem is coupled with a meta-
heuristic. However, because simulations are time expensive and hence
unsuitable for real-time control, a trained surrogate model that takes the
decision variables as inputs and approximates the objective function is
employed to replace the simulation within the meta-heuristic algorithm.
Several modeling techniques (i.e., linear regression, polynomial regression,
neural network, radial basis network, regression tree ensemble, and Gaussian
process regression) are compared based on their accuracy in reproducing
solutions to the problem and computational tractability for real-time control
under different configurations of simulation parameters. It is found that
Gaussian process regression is the most suited for use as a surrogate model
for the given problem. Finally, a realistic application with multiple ride-
hailing vehicle operations is presented. The proposed approach for controlling
the stop positions of vehicles is able to achieve an improvement of 20.65%
over the uncontrolled case. The example shows the potential of the proposed
approach in reducing the negative impacts of stopping vehicles and favorable
computational properties.

Keywords: Traffic Optimization, Curbside Management, Surrogate Modeling,
Simulation-based Optimization

##  1 1 Introduction

Due to recent changes in mobility, such as the growth of ride-hailing and
delivery services, urban curbside is facing increasing demand [1]. In urban
areas already burdened by limited road capacity and parking, it is crucial to
find new solutions to maximize the efficiency of these new operations. Real-
time traffic management can reduce congestion by dynamically affecting
vehicles’ movements through advanced sensing and communication technologies
[2]. For example, in the specific case of curbside, it can provide directions
for curbside stop positions on a street to minimize traffic disruptions. This
strategy can be formulated as a dynamic optimization (or control) problem in
which decision variables (e.g., curbside allocations) are determined based on
time-varying information on traffic conditions and curbside demand.

Given the above requirements, in the present work, we propose a novel
optimization formulation for controlling the stop positions of vehicles
intending to make curbside stops on a multi-lane road segment to reduce the
adverse effects of the stops on the traffic flow of the segment. A surrogate-
modeling-based solution approach to the problem is proposed, analyzed, and
demonstrated through an example application in allocating pick-up and drop-off
operations for ride-hailing service vehicles.

Simulation-based optimization (SO) can be employed to solve control problems.
Within this framework, simulation is adopted to evaluate the objective
function in relation to the decision variables and integrated with a generic
optimization approach to identify well-performing solutions. Memetic
algorithms, simulated annealing, and genetic algorithms (GAs) are all examples
of evolutionary algorithms that can be coupled to optimization for traffic
management purposes [3]. However, depending on the nature of the problem,
evolutionary algorithms typically require a large number of simulation
assessments and, ultimately, a significant computational effort to reach high-
quality solutions. Therefore, it is challenging to implement these solutions
for real-time control problems.

Surrogate modeling creates a statistical model (called surrogate model) to
represent the simulation’s output and can be integrated with optimization in
place of the original simulation. This paper develops alternative surrogate
model-based optimization approaches for managing road curbside to maximize its
traffic throughput and analyzes their accuracy and computational efficiency.
The studied problem is formulated as a control problem that can be solved
through a model predictive control (MPC) framework [4]. The present paper
introduces novel research on optimizing the positions of vehicle stoppages
while considering traffic dynamics by means of surrogate modeling, which, to
the best of the authors’ knowledge, has not been previously studied. The
contributions of the paper are summarized as follows:

  1. 1.

We present a novel optimization problem to control the curbside vehicle stop
positions with the objective of minimizing the adverse effects of the
congestion caused by the stops on the traffic flow. The problem is implemented
within an MPC framework based on a macroscopic traffic model.

  2. 2.

We investigate a surrogate-based solution approach for solving the given
optimization problem considering various alternatives of the surrogate model.
A numerical analysis is performed to determine the best surrogate model,
considering solution accuracy and computation time.

  3. 3.

We present an illustrative application of the proposed control approach in the
context of determining stop positions of pick-up and drop-off operations in
ride-hailing services. The control approach proves effective in improving the
considered traffic performance objective by $20.65\%$ by reducing the
spillback caused by vehicle stoppages upstream of the road segment.

The rest of the paper is organized as follows- Section 2 presents a review of
the literature related to the current work. Section 3 presents the traffic
control problem along with a description of the traffic dynamics and the
optimization problem. The surrogate-based approach to solve the optimization
problem is described in Section 4. Finally, a numerical example of the
application of the proposed control approach is presented in Section 5
followed by a conclusion in Section 6.

##  2 2 Related Work

Different studies have tackled the issue of ride-hailing operations’ traffic
impacts and related curbside management by employing traffic simulation. Roca-
Riu et al. (2017) [5] proposes ‘Dynamic Delivery Parking Spots’, delivery
facilities positioned on a link’s shoulder lane that are activated to minimize
traffic delays. Gonzales and Christofa (2017) [6] focus on re-optimizing
traffic signals in downstream intersections to maintain undersaturated
conditions. Ye et al. (2020) [7] develop a simulation-based GA to identify
parking lanes. Stueger et al. (2022) [8] evaluate two alternative strategies
for enforcing stop positions to reduce delays. All of these studies rely on
microscopic traffic simulation. Another solution approach is offered in [3],
which models the interactions of freight vehicles by combining the LWR model
with moving bottlenecks’ theory, and integrating it into a meta-heuristic.

For solving SO problems, iterative approaches involving multiple simulations
are usually time-consuming. As a result, surrogate-based optimization
strategies that eliminate the requirement for several simulation runs have
gained popularity and have been extensively investigated in the literature,
including their application in the transportation field. For instance, Chen et
al. (2014) [9] compare several surrogate-based optimization strategies for
optimizing toll charges to minimize the average network travel time. A
surrogate-based optimization method using a regression Kriging model is
proposed in [10] to solve a congestion pricing problem, which requires the
intense computation of the corresponding objective function. A surrogate-based
cooperative signal optimization method is proposed in [11] using a radial
basis function.

Osorio (2019) [12] utilizes available knowledge about the traffic system to
form an analytical surrogate model which is used in combination with a simple
statistical model to guide an optimization problem for calibration of traffic
microsimulation models. It has the benefit of not having to entirely rely on
black-box statistical models and using efficient optimization methods like
gradient-descent given a closed-form differentiable surrogate model. While
this approach is attractive for the aforementioned reasons, it is non-trivial
to find a closed-form analytical model for every problem setup, which is the
case in the present study. Hence, in this work, we rely on a purely
statistical approach. We select several available techniques from the
literature and compare them based on accuracy and time consumption for solving
a vehicle stop position control problem introduced in the following sections.

##  3 3 Problem definition and formulation

Figure 1: Flowchart of controller implementation.

The control problem considered in this work is defined as follows:
Given a set of vehicles destined to make stops on a segment of the road,
control their longitudinal stop positions from the upstream end of the segment
to minimize the adverse effects of the stops on traffic flow.

Throughout this work, the vehicles referred to in the above problem such as
trucks, or ride-hailing cars approaching the road segment of their drop-off or
pick-up operation are called approaching vehicles and the corresponding road
segment is called the stop segment. Each such vehicle is assumed to be
connected to an operator who identifies and transmits a stop position to the
vehicles with the help of a controller using the knowledge of each vehicle’s
position and the associated traffic dynamics. The conceptual framework for the
controller is illustrated in Figure 1. The controller is formulated in the MPC
framework wherein the optimal stop positions are decided based on their impact
on a predefined objective function over a fixed duration into the future
(known as the prediction horizon). To do so, the network operator would
typically solve an SO problem with the latest set of known inputs. The problem
is solved at fixed intervals and updated stop positions are communicated to
the approaching vehicles based on the latest knowledge about the traffic
conditions. Recommendations on the stop positions are allowed to be updated
until the vehicle enters the stop segment and no later to avoid confusing the
drivers. The solution to the SO problem is only initiated once an approaching
vehicle is detected within a certain distance upstream of the stop segment
referred to as the MPC range in Figure 1. The simulation framework is
described briefly in the ensuing section followed by a detailed description of
the SO problem and the proposed solution approach.

###  3.1 3.1 Traffic Simulation Framework

The simulation approach consists of a hybrid model where the main flows are
modeled macroscopically, according to the Lighthill-Whitham-Richards (LWR)
model [13, 14] whereas the dropoff/pick-up operations are microscopically
modeled as temporary bottlenecks based on [15].

For a given time $t$ and position $x$, $\rho(t,x)$ represents the traffic
density in vehicles per unit length and $Q(\rho(t,x))$ corresponds to the
Fundamental diagram (FD) which maps the traffic density to traffic flow
$q(t,x)$ defined in vehicles per unit time. The FD is defined on
$[0,\rho_{m}]$ where $\rho_{m}$ is the jam density. In this study, we adopt
the triangular FD which is defined as follows:

$q(t,x)=Q(\rho(t,x)):=\left\\{\begin{array}[]{ll}v_{f}\rho(t,x)&{\rm
if}\;\;\rho(t,x)\leq\rho_{c}\\\ w_{c}(\rho(t,x)-\rho_{m})&{\rm
otherwise},\end{array}\right.$ (1)

where $v_{f}$ denotes the free-flow speed of traffic, $w_{c}$ denotes the
congestion wave speed, and $\rho_{c}$ denotes the critical density such that
the maximum traffic flow or capacity $q_{m}=v_{f}\rho_{c}$. The spatial domain
of the stop segment is defined as $[0,~{}L]$ meters, and the time domain is
defined as $[0,~{}T]$ seconds. In this paper, we adopt a Hamilton-Jacobi (HJ)
partial differential equation (PDE) formulation of the LWR model given as
follows:

$\frac{\partial M(t,x)}{\partial t}+Q\left(-\frac{\partial M(t,x)}{\partial
x}\right)=0$ (2)

where $M(t,x)$ represents the cumulative vehicle count. To solve the HJ PDE
(2) we define value conditions $c(\cdot,\cdot)$ as follows:

$c(t,x)=\begin{cases}c^{l}_{ini}(x)&t=0,~{}x\in[(l-1)\Delta x,l\Delta x]\\\
c^{j}_{up}(t)&x=0,~{}t\in[(j-1)\Delta t,j\Delta t]\\\
c^{j}_{down}(t)&x=L,~{}t\in[(j-1)\Delta t,j\Delta t]\\\
c^{n}_{int}(t,x)&x\in[x_{b,n},x_{e,n}],~{}t\in[t_{b,n},t_{e,n}].\end{cases}$
(3)

Here, $c^{l}_{ini}(\cdot)$ represents an initial density condition with
$l\in\\{1,~{}\dots,~{}\frac{L}{\Delta x}\\}$ where $\Delta x$ is the length of
each initial density condition, $c^{j}_{up}(\cdot)$ and $c^{j}_{down}(\cdot)$
represent upstream and downstream boundary flow conditions with
$j\in\\{1,~{}\dots,~{}\frac{T}{\Delta t}\\}$ where $\Delta t$ represents the
duration of each boundary flow condition, and $c^{n}_{int}(\cdot,\cdot)$
represents internal boundary conditions with $n\in\\{1,~{}\dots,~{}n_{i}\\}$
where $n_{i}$ denotes the number of internal boundary conditions considered in
the current simulation, $x_{b,n}$ and $t_{b,n}$, and $x_{e,n}$ and $t_{e,n}$
denote the position and time of the start and end of the internal condition,
respectively. Figure 2 presents a schematic of the different value condition
blocks on a road segment.

Figure 2: Schematic of different value condition blocks on a road segment.

The complete solution ${\bf M}(\cdot,\cdot)$ to the LWR model corresponding to
given value conditions can be obtained using the formulation presented in
[16]. Here, the upstream and downstream boundary conditions are computed using
traffic demand and supply which are assumed known for the segment. The
internal conditions correspond to the bottlenecks caused by vehicle stops and
are computed using Algorithms 1 and 2 from [3].

The minimum value of the length $L$ is decided by the Courant-Friedrichs-Lewy
(CFL) condition [17]: $\Delta t\leq L/v_{f}$ which ensures the stability of
the simulation. The next section presents the SO problem description
considering the above simulation framework.

###  3.2 3.2 Simulation-based Optimization Formulation

The controller is formulated in the MPC framework where an optimal decision is
made by predicting the future behavior of the system over the prediction
horizon with the help of a model. Here, the network operator aims to regulate
the stop positions of approaching vehicles which form the decision vector to
minimize their adverse effects on the traffic flow. The notation used for
defining the optimization problem is presented in Table 1. The SO problem can
be defined as follows:

Table 1: Notation used for SO problem formulation. Parameter/ |  Description
---|---
Variable |
$t$ |  time at which control is executed
$N_{p}$ |  prediction horizon duration (time steps into the future up to which the impact of control is evaluated)
$N(t)$ |  number of approaching vehicles at time $t$
$x_{i}$ |  longitudinal stop position for vehicle $i$
$X_{t}$ |  vector of stop positions for approaching vehicles such that $X_{t}=\\{x_{1},\dots,x_{N(t)}\\}$
$u_{t}$ |  non-controlled simulation parameters for prediction period starting at time $t$
$k$ |  time step index within prediction horizon
$f(X_{t},u_{t})$ |  objective function computed from simulation
$q_{\text{out}}(k;X_{t},u_{t})$ |  simulated traffic outflow over time-step $k$
$q_{\text{in}}(k;X_{t},u_{t})$ |  simulated traffic inflow over time-step $k$
$D(k)$ |  upstream traffic demand for time-step $k$
$X_{D}$ |  vector of desired stop positions for approaching vehicles
$X_{US},X_{DS}$ |  vectors of maximum distance upstream and downstream, respectively from $X_{D}$ based on operation requirements
$w_{SB}$ |  objective penalty weight on congestion spillback
$W_{D}$ |  vector of objective penalty weights corresponding to detour from desired stop positions

$\displaystyle\min_{X_{t}}$ $\displaystyle
f(X_{t},u_{t})=-\sum_{k=1}^{N_{p}}q_{\text{out}}(k;X_{t},u_{t})+w_{SB}\sum_{k=1}^{N_{p}}\left(D(k)-q_{\text{in}}(k;X_{t},u_{t})\right)+W_{D}\cdot|X_{D}-X_{t}|$
(4) such that $\displaystyle x_{i}\hskip 2.84526pt\in(w_{c}\Delta
t,L-v_{f}\Delta t)~{}\forall i\in\\{1,\dots,N(t)\\}$ (5) $\displaystyle
X_{t}\in[X_{D}-X_{US},~{}X_{D}+X_{DS}]$ (6)

Here, (4) is the objective function which is minimized. The three terms of the
objective function represent the total traffic outflow over the prediction
horizon, the traffic demand blocked by the stop segment’s traffic, due to
congestion spillback upstream, and the cost of inconvenience due to detour
from the desired stop positions for all approaching vehicles, respectively.

Equation (5) specifies the bounds on the decision vector required to maintain
the stability of the model similar to the CFL condition preventing the
interaction between internal and boundary conditions within the time step of
their origination. Equation (6) bounds the maximum detour in stop position to
an acceptable range. These bounds are useful when the detour cost within a
certain distance of the desired position is small but the operation cannot be
performed beyond that distance. Additionally, depending on the time resolution
for simulation and the minimum distance traveled by a vehicle per time step,
the candidate stop positions can be reduced to a set of discrete values to
reduce the solution search space. For instance, with a time resolution of 1
second, the stop positions are approximately in multiples of $v_{f}$.

The objective function (4) is computed from the simulation described in
Section 3.1 for different $X_{t}$ given a set of non-controlled simulation
parameters $u_{t}$. Here, $u_{t}$ represents all possible parameters that
affect the simulation including spatial limits of the segment, FD parameters,
initial density, upstream demand $D(k)$, downstream supply, traffic signal
cycle times, approaching vehicles’ arrival time, and approaching vehicles’
stop durations. The stop positions and durations for vehicles intending to
make a stop and already present on the stop segment at the beginning of the
simulation are also included in $u_{t}$. Hereafter, these are referred to as
on-segment vehicles.

The objective function (4) lacks a direct analytical gradient due to the use
of simulation for evaluation. Therefore, the SO problem (4)-(6) cannot be
solved using efficient approaches like gradient descent. Instead, meta-
heuristic evolutionary algorithms such as GA can be used to find an optimal
solution where the values of $X_{t}$ are iteratively improved in order to
reach well-performing solutions. Since each evaluation of the objective
function (4) requires a simulation, meta-heuristics would likely require
significant computation efforts to achieve high-quality solutions making the
approach less attractive for real-time control. This forms the basis of the
surrogate-based solution approach for the SO problem which seeks to address
this issue and is discussed in the following section.

##  4 4 Surrogate modeling

In the proposed approach, the objective function (4) is approximated through a
suitable surrogate model which replaces the simulation in the SO problem
(4)-(6). This study considers two types of surrogate models- local models and
global models. Local models are created for individual runs of the controller
and are trained online with a new training sample each time the SO problem is
solved in a new setting. Global models on the other hand are trained offline
using a large training sample that is generated in advance considering
plausible scenarios that would arise during different controller runs. Figure
3 presents an outline of the steps involved with surrogate model development.
The following section describes the inputs considered by both types of models
and the corresponding sampling technique for training and testing.

Figure 3: Steps in surrogate model development.

###  4.1 4.1 Input and Sampling

To solve the SO problem using an evolutionary algorithm, like a GA, multiple
feasible solutions are ranked by evaluating them through the objective
function. The surrogate model output is intended to replace the objective
function when solving the SO problem with a GA. The model takes the set of
decision variables $X_{t}$ as input and generates an output equivalent to the
corresponding value of the objective function. In the MPC framework, the SO
problem is solved whenever an approaching vehicle is detected within the MPC
range. The number of approaching vehicles at any given time affects the number
of inputs for the surrogate model. The objective function (4) also includes
the non-controlled simulation parameters $u_{t}$ as inputs that should be
considered in the surrogate model. For local models, the effect of a given
$u_{t}$ on the surrogate model output is captured implicitly in the
simulations used to produce the dependent variables in the training data. For
global models, since $u_{t}$ varies across different controller runs, the
parameters included in $u_{t}$ mentioned in Section 3.2 are also taken as
inputs to these models. Multiple global surrogate models with different
numbers of inputs can be trained offline and can be applied to cases with
different numbers of approaching vehicles as required.

For local models, each data point in the training sample consists of a
distinct set of stop positions $X_{t}$ and the corresponding output from (4)
given a fixed $u_{t}$. Every stop position is selected randomly from a set of
predefined positions based on the minimum distance traveled in free flow that
is in multiples of $v_{f}$. For global models, the stop positions are selected
similarly. The sampling for parameters within $u_{t}$ is discussed in the
remainder of this section. The stop durations are selected randomly from
multiples of 30 seconds (this type of time discretization would be
realistically available from the driver) up to a reasonable upper bound. For
other parameters, the values are selected randomly from uniform distributions
with the following bounds- $[0,q_{m}]$ for the demand and supply parameters,
$[0,\rho_{m}]$ for the initial density parameters, and $[-T_{r},T_{g}]$ for
the red signal start time where $T_{r}$ and $T_{g}$ are the magnitude of the
red time and green time in a traffic signal cycle respectively. The negative
lower bound allows for cases in which the start time of the prediction horizon
is in the middle of the red phase of the signal. The start and stop positions,
and stop durations of the on-segment vehicles are selected in the same way as
those of the approaching vehicles.

###  4.2 4.2 Surrogate Model Selection

The following methods are considered for surrogate modeling:

  1. 1.

Linear regression (LR)

  2. 2.

Polynomial regression (PR)

  3. 3.

Regression-based tree ensemble (RTE)

  4. 4.

Shallow neural network (NN)

  5. 5.

Radial basis neural network (RBNN)

  6. 6.

Gaussian process regression (GPR)

A description of the techniques is omitted from this paper for brevity. Among
the above techniques, LR and PR are selected as surrogate models as they
result in simple linear and quadratic optimization problems when used as
surrogates (linear and quadratic mixed integer problems when the decision
variables are discrete). These can be solved using freely available standard
solvers, making them highly suitable for real-time control. RTE, NN with a
sigmoid transfer function, and RBNN with the radial basis function, are
considered valuable for capturing complex nonlinear relationships between
inputs and output. GPR is selected due to its popularity in surrogate-based
optimization studies, as evident in [9], as well as its demonstrated ability
to outperform NNs [18]. However, it is important to note that these techniques
require the use of meta-heuristics or heuristic algorithms to solve the
optimization problem. Since straightforward methods like LR and PR are often
unable to effectively learn complex relationships between inputs and output
like the other candidate techniques, there exists a trade-off between the
accuracy and time required to find a well-performing solution. In the analysis
section, we further assess the models in terms of their usefulness in real-
time control.

###  4.3 4.3 Analysis of Surrogate Models

####  4.3.1 4.3.1 Comparison metrics

The SO problem is solved by means of a meta-heuristic algorithm that relies on
accurately ranking the problem solutions based on the corresponding values of
the objective function. Therefore, it is important that the outputs of the
surrogate model can be used to correctly rank the solutions even if the
magnitude of the outputs themselves is not close to the value of the objective
function. Standard metrics such as the root mean squared error, which
evaluates the model based on the magnitude of the outputs alone, are not
sufficient for this study. A suitable metric for our problem is the Ranking
Error (RE) [19] which is defined as follows:

$\mathrm{RE}=\dfrac{n_{SP}}{N(t)^{2}},$ (7)

where $n_{SP}$ denotes the number of swapped pairs (pairs of solutions which
are incorrectly ordered) obtained from:

$n_{SP}=|\\{(i,j):(f_{t}(X_{t}^{i})-f_{t}(X_{t}^{j}))\times(\hat{f}_{t}(X_{t}^{i})-\hat{f}_{t}(X_{t}^{j}))<0\\}|$
(8)

where $\hat{f}_{t}(\cdot)$ denotes the output of the surrogate model, and
$N(t)$ is the number of solutions in the validation set. A larger RE implies
that a larger number of solutions is incorrectly ranked therefore resulting in
a higher possibility of the meta-heuristic producing a sub-optimal solution.

Since the goal of solving the SO problem is real-time traffic control which
requires fast computation of the solution, therefore, besides the RE, we also
consider the solution time as an evaluation metric for the model choice. In
the case of local models, the total time required to reach an optimal solution
includes the time to produce training samples, train the model, and solve the
problem using the trained model while for global models, it only includes the
time for solving the optimization problem.

####  4.3.2 4.3.2 Analysis of local models

To test surrogate model performance, we create different scenarios with
varying numbers of approaching and on-segment vehicles and different
simulation parameter values. Since the number of vehicles affects the number
of simulation parameters, scenarios are first classified by the number of
vehicles followed by division into sub-scenarios based on other parameter
values. Scenarios based on the numbers of vehicles are denoted by C1, C2, C3,
C4, and C5, with 4, 4, 4, 6, and 6 approaching vehicles, and 0, 2, 4, 0, and 2
on-segment vehicles, respectively. Further, various non-controlled simulation
parameters including traffic demand and supply, initial density, traffic
signal red phase start time, entry time and stop duration of approaching
vehicles, and start and stop position and stop duration of on-segment vehicles
are varied to produce 12 sub-scenarios within scenarios C1 and C4 and 13 sub-
scenarios within scenarios C2, C3, and C5. The goal is to create a diverse set
of conditions to represent various simulation settings that can be encountered
when solving the SO problem over different control periods. Details of sub-
scenario settings are omitted from the paper for brevity.

For this analysis, the length of the stop segment is 450 m with 2 lanes and a
traffic signal at the downstream end. One of the lanes is used for stopping.
The cycle time and red time for the traffic signal are 120 seconds and 48
seconds, respectively. The FD parameters are $v_{f}=14$ m/sec, $\rho_{c}=0.04$
veh/m, $\rho_{m}=0.24$ veh/m, $w_{c}=2.8$ m/sec, and $q_{m}=0.56$ veh/sec. The
MPC range is 150 seconds with a prediction horizon of 600 seconds. The
spillback penalty term $w_{SB}=0.1$ and the penalty vector $W_{D}=\textbf{0}$
that is no desired stop positions considered.

The performance of models trained with different methods is compared using 5
sample sets of sizes 100, 200, 300, 400, and 500 to determine the method that
works better with fewer samples. For each size, 5 sets of data are produced
for training to account for randomness in data generation. To test the models,
an independent sample set with 1000 samples is produced for each sub-scenario.
An average RE is computed for each combination of modeling method, training
sample size, and scenario by averaging the RE over the sub-scenarios and the 5
training sets produced per sample size. The accuracy of the models is
illustrated in Figure 4. The performance of all the modeling techniques
improves with the increase in the size of the training set. The largest
improvement of 0.10 in going from 100 to 500 samples is observed for NN over
all scenarios, followed by RTE and GPR which both show an average improvement
of 0.06. RTE has the lowest RE for all scenarios followed by GPR. The least
accurate model is LR followed by RBNN which hardly shows any improvement with
the number of samples. Generally, the performance is worse for scenarios C2,
C3, and C5, characterized by on-segment vehicles due to a more complex
relationship between the decision variables and the parameters.

The difference between the objective function value obtained by solving the SO
problem with simulation and that with surrogate models is also evaluated. A
single error value is obtained for each combination of modeling method,
training set size, and scenario, and is called the Mean Error (ME). The
purpose is to validate the observations obtained from the RE since RE is only
a proxy for the true performance in optimization. The ME is presented in
Figure 4 for the local models. The time step duration used for evaluating (4)
is 10 seconds. A solution using the LR model can be obtained simply by
observing the signs of the decision variable coefficients while an integer
programming solver is used with the PR model. A GA is adopted for all other
models. For the GA, the functional tolerance value is set to 0.5 which means
that the algorithm terminates if the average change in the objective function
value is below 0.5 over a pre-defined number of generations which here is set
to 50 by default. The other parameters are also kept as the default values for
the MATLAB GA function [20]. The maximum run time for the GA is set at 30
minutes. Note that 30 minutes is already high for solving a real-time control
problem, but it is adopted here to allow for the GA to terminate naturally by
the specified termination criteria. In cases where the simulation-based
objective is larger than the surrogate-based objective (due to GA not reaching
the optimal solution using simulation), the error is taken as 0 since it is
favorable in this setting. It is observed in Figure 4 that between modeling
techniques, the ME is lowest for RTE and GPR models and worst for RBNN and LR.
This is consistent with the observation from examining the RE and shows that
RE works as a reasonable proxy for actual error values in optimization. In
general, the ME decreases with an increase in sample size although in some
cases there is no significant trend such as LR and RBNN in scenario C1.

Figure 4: Average RE [top] and average ME [bottom] for different modeling
techniques and scenarios for local models with varying number of training
samples.

The time taken to train the models is negligible compared to that for
producing the samples. The mean training times are 0.006, 0.29, 0.046, 0.329,
0.007, and 0.122 seconds for LR, NN, RBNN, RTE, PR, and GPR, respectively. The
time spent in solving the optimization problem is 0.01, 12.50, 15.90, 42.40,
0.60, and 0.45 seconds, respectively in the same order of techniques. The RTE
model takes the longest to converge using a meta-heuristic algorithm. This can
be attributed to the tree structure which is more computationally intensive to
evaluate for each candidate solution. Among the techniques that are integrated
into meta-heuristic approaches, GPR is the fastest to converge with its
solving time being comparable to that using the PR model which is solved using
standard solvers. This is because of the more straightforward evaluation of
candidate solutions through GPR. The time to produce training samples
increases with the number of stopping vehicles such that 100 samples of
scenarios C1 to C5 take around 9.5, 13.4, 17.5, 15.4, and 19.8 seconds,
respectively when produced sequentially. The average time taken to solve the
problem using the simulation directly for scenarios C1 to C5 is 262, 312, 399,
552, and 601 seconds, respectively. Therefore, the surrogate-based
optimization with the online generation of training samples still saves a lot
of time as compared to the simulation-based solution.

####  4.3.3 4.3.3 Analysis of global models and comparison with local models

Since global models are trained with a larger number of variables, they
require a larger training set to understand the influence of all variables on
the objective function. Unlike local models, global models are trained to
account for the variations in the simulation parameter values. Therefore the
same model can be used in a broader range of test cases. Scenarios with
different numbers of approaching and on-segment vehicles still require
separate models as they have different numbers of variables.

Training sets of sizes 5000, 10,000, and 15,000 samples are produced using the
approach mentioned in Section 4.1. Testing is done using the same test sets
used with local models. The trends in the RE and ME for global models with
increasing training set sizes are presented in Figure 5. In this case, an
increase in the number of samples does not improve model performance. A reason
for this is that only a subset of the training variables is used in the
testing process and there is no guarantee of improvement corresponding only to
that subset. Similar to local models, scenarios C1 and C4 with no on-segment
vehicles show smaller values for RE as compared to other scenarios. Overall,
compared to Figure 4, the RE is much larger when using the global models as
compared to local models irrespective of the number of samples especially when
compared for RTE and GPR models. Similarly, the ME for the global is also
worse than the best results obtained using the local RTE and GPR models. The
results with the other modeling techniques are similar to those obtained for
local models.

Figure 5: Average RE [top] and average ME [bottom] for the different modeling
techniques and scenarios for global models with varying number of training
samples.

The solution times for LR, NN, RBNN, RTE, PR, and GPR are 0.013, 24.10, 19.49,
124.3, 0.46, and 10.09 seconds, respectively. We do not consider the sampling
and training time since the models are intended to be trained offline.
Interestingly, the meta-heuristic dependent models require more time than when
used in a local model context. RTE method takes almost thrice the time taken
using local models. This is possible because of the more complicated and
computationally intensive model obtained by training with a much larger set of
variables which also includes the simulation parameters. LR and PR have
comparable solving times to the local models as they are solved directly using
a solver. GPR still takes the lowest time among meta-heuristic dependent
methods although it is 20 times slower than in the local context.

Overall, as expected, optimization using surrogate models has considerable
computational advantages over simulation-based solutions. The local models
perform considerably better than the global models in terms of accuracy
although they require additional time to generate the samples and train the
model. However, since the training samples are independent and do not need to
be produced sequentially, it is possible to parallelize the simulations used
to create the training samples. Therefore, real-time control is possible using
these local models. RTE and GPR perform the best in terms of accuracy, and
between the two, GPR is also considerably faster. LR and PR models are fast
but not accurate and hence are not recommended for solving the SO problem.

In the following section, we present an example of a realistic application of
the control implemented using the surrogate-based optimization approach using
the GPR model versus directly using the simulation to solve the optimization
problem.

##  5 5 Application: Management of ride-hailing vehicles’ pickup/drop-off
operations

To illustrate the effectiveness of the proposed control approach on traffic
system performance as well as to compare the surrogate-based approach for
obtaining the optimal controls with the direct approach using the simulation,
we present an example application for controlling the pick-up/drop-off
positions of ride-hailing vehicles on a stretch of a signalized urban road. We
also study the impact of using different magnitudes of the penalty weight for
the detour caused by control. The local GPR surrogate model is selected for
this study due to its suitability to the optimization problem. We test two
training set sizes for the models, namely 50 and 500 to test low and high
computation costs for surrogate-model development. The results are benchmarked
against the case with randomly selected stop positions without control and a
case where the SO problem is solved directly using the simulation with a
termination time limit of 10 minutes allowing the solver to reach an optimal
solution. This solution may not be reached with the time duration available
for real-time control but is considered here to find the best control
performance and compare the surrogate-based solution to it. Surrogate-based
optimization is carried out under five different random seed values for each
sample size to reduce any bias in results due to the training sample.

control scenario |  detour penalty weight range | objective | outflow |  spillback penalty |  detour penalty
---|---|---|---|---|---
no-control |  - | 44.68 | 28.11 |  72.78 |  0.0
simulation-based |  | 20.65% | 2.38% |  11.74% |  0.0
GPR with 50 samples |  0.0 | 17.94% | 1.74% |  10.33% |  0.0
GPR with 500 samples |  | 17.08% | 1.22% |  10% |  0.0
simulation-based |  | 19.15% | 2.39% |  11.47% |  -0.43
GPR with 50 samples |  $[10^{-4},10^{-3}]$ | 17.27% | 1.52% |  10.41% |  -0.30
GPR with 500 samples |  | 14.96% | 0.58% |  9.37% |  -0.31
simulation-based |  | 12.68% | 1.71% |  10.22% |  -2.26
GPR with 50 samples |  $[10^{-3},10^{-2}]$ | 5.55% | -0.17% |  5.74% |  -1.66
GPR with 500 samples |  | 5.72% | -0.09% |  5.73% |  -1.60
simulation-based |  | 0.58% | 0.53% |  2.34% |  -1.60
GPR with 50 samples |  $[10^{-2},10^{-1}]$ | -1.94% | -0.02% |  1.49% |  -1.95
GPR with 500 samples |  | -1.83% | -0.26% |  1.25% |  -1.67
simulation-based |  | 0.00% | 0.00% |  0.00% |  0.0
GPR with 50 samples |  $[10^{-1},1.0]$ | -24.22% | -0.12% |  0.15% |  -10.9
GPR with 500 samples |  | -2.75% | -0.06% |  -0.08% |  -1.16
Table 2: Objective function values for the different cases in the numerical
example.

The studied application considers a total simulation duration of 900 seconds
with 10 approaching vehicles with entry times at 30, 60, 90, 120, 250, 270,
300, 330, 520, and 550 seconds and stop durations 60, 120, 90, 60, 60, 120,
90, 60, 90, and 120 seconds, respectively. The MPC range is set to 150
seconds, and the prediction horizon is maintained at 600 seconds. The FD
parameters and spatial domain of the segment are the same as described in
Section 4.3.2. The simulation starts with the traffic signal’s red cycle in
the first 20 seconds. The initial density is set at 0.02 veh/m throughout the
length of the segment. The demand and supply remain equal to $q_{m}$ for the
entire simulation duration, defined over 10-second intervals. The duration of
the time step used for computing the objective function (4) is also 10
seconds.

For analysis of the control under varying detour penalty weights, five sets of
weights are considered. Four of these sets are created by randomly picking the
weights from uniform distributions with bounds mentioned in Table 2, while in
the fifth case, all the weights are set to 0. We only consider the stability
constraints from Section 3.2 in this example.

The objective function values for the no-control case are presented in Table
2. For the GPR model-based approach with both sample sizes, the average values
over the five runs with different random seeds are presented. The detour
penalty is computed by using the stop positions from the no-control case as
the desired positions. Percentage improvements through control are presented
for the overall objective, outflow, and spillback penalty while absolute
improvement is presented for the detour penalty (since the value in the no-
control case is 0). The negative signs indicate a worsening of the component
compared to the no-control case which is not desirable. In general, the
spillback and detour penalty is supposed to be minimized while the total
outflow is supposed to be maximized which results in the overall minimization
of the objective function.

(a)

(b)

(c) Figure 6: Traffic evolution considering (a) randomly selected stop
positions, (b) optimally selected stop positions based on simulation, and (c)
optimally selected stop positions based on the GPR model with 500 samples. The
color bar represents traffic density in veh/m.

With a 0 detour penalty, the simulation-based solution is observed to produce
a significant improvement of $20.65\%$ in the objective value. The majority of
the improvement comes from the spillback penalty terms while the outflow is
also observed to improve. The corresponding improvement using the surrogate
model is smaller but still significant at $17.94\%$ and $17.08\%$ with 50 and
500 samples, respectively. The improvement in the objective function value
decreases with an increase in the value of the detour penalty weights for all
controllers. The detour penalty itself decreases for all controllers after a
point. This is because the increasing cost of detours offsets any improvement
in traffic and eventually forces the controller to allow vehicles to stop at
their desired positions. A detour in the cases with a high detour cost would
only result in a large penalty resulting in a lower objective value than the
no-control case. The simulation-based solutions are able to avoid this in all
cases with a minimum improvement of $0.0\%$ in the case with the highest
detour costs. The GPR-based solutions do face the issue of a deterioration of
the objective under control at large values of detour penalty weights. This is
especially true with 50 samples which produces a worse solution than the no-
control case in the two cases with the highest detour penalty weights. The
controller with 500 samples also causes a deterioration in the two cases but
it is smaller than the controller with 50 samples. This makes sense since a
larger detour penalty creates a more complex relationship between the stop
positions and objective function as it counters the improvement from the other
terms in the objective. More samples result in better learning of the
relationship and hence better results. Note, that the best performance of the
controllers for the case with the highest detour costs are $7.27\%$ with 50
samples, and $0.0\%$ with 500 samples, respectively which shows that the
models are indeed able to learn the relationships between the decision
variables and the objectives better than what the averages convey although the
performance is not always consistent. In this study, we utilize the parallel
computing toolbox in MATLAB [21] to generate the training samples with a pool
of 6 parallel workers since the sampled points are independent of each other.
The average computation time over the five cases with different detour
penalties and over different seed values using the surrogate-based controllers
with 50 and 500 samples are 6.55 and 21.26 seconds, respectively.

Figure 6 shows the space-time-density diagrams for the simulated traffic for
both the no-control case and the control cases using simulation and the GPR
model with 500 samples, with 0 detour penalty weights. Figures 6b and 6c show
that the controller strategically arranges vehicle stops to overlap congestion
waves, resulting in reduced spillback duration at the upstream end. In Figure
6b, Vehicle 1 stops further downstream compared to the no-control case and
Vehicles 2, 3, and 4 make stops in the direction of propagation of the
congestion wave of Vehicle 1 (highlighted with a dashed rectangle around the
vehicle stops). The delay in Vehicle 1’s stop allows future vehicles to
rearrange their stops so the congestion waves may overlap, preventing
additional congestion caused by their stops. Notably, the controlled case
exhibits a more significant congestion wave from the fourth red cycle of the
traffic signal, leading to a longer upstream congestion duration. This
congestion was not observed in the no-control case as the traffic approaching
the signal was blocked by vehicle stops, causing earlier congestion. By
preventing early additional congestion, the controlled case reduces the
cumulative time vehicles are blocked upstream (accounted for in the spillback
penalty) resulting in a lower objective function value. Both, the simulation-
based and surrogate-based controllers achieve an improvement by arranging
vehicle stops to overlap congestion waves and delay congestion, albeit at
different positions within the segment. As such, solutions from both
controllers are valid and high-performing in this context.

##  6 6 Conclusion

This work focuses on the optimal control of stop positions of vehicles
intending to make a curbside stop on a road segment to minimize congestion and
maximize the total outflow. An MPC framework is developed to perform real-time
control using an LWR simulation solved by means of an extended Lax-Hopf
approach to model bottlenecks in the traffic stream. This work employs
surrogate modeling to avoid computationally demanding SO strategies.
Alternative surrogates are explored to replace the simulation in the solution
approach. Analysis shows that the time consumed in solving the SO problem
using the surrogate is much smaller than that consumed using the simulation
thus making it a time-efficient approach suitable for real-time control. The
performed tests highlight that GPR outperforms other techniques in terms of
computational time and accuracy. A numerical study demonstrates that the
performance of controls derived through surrogate-based optimization is close
to a simulation-based solution while requiring much less time.

The proposed method can be extended to the problem of optimal speed regulation
of connected and autonomous cars in order to reduce traffic congestion and
increase throughput. Besides, the inclusion of capacity drop in the simulation
framework is another possible extension of the work which will make it more
reliable for real-world control.

## References

  * Marsden et al. [2020] Marsden, G., I. Docherty, and R. Dowling, Parking futures: Curbside management in the era of ‘new mobility’services in British and Australian cities. _Land Use Policy_ , Vol. 91, 2020, p. 104012.
  * Rizwan et al. [2016] Rizwan, P., K. Suresh, and M. R. Babu, Real-time smart traffic management system for smart cities by using Internet of Things and big data. In _2016 international conference on emerging technological trends (ICETT)_ , IEEE, 2016, pp. 1–7.
  * Simoni and Claudel [2017] Simoni, M. D. and C. G. Claudel, A fast simulation algorithm for multiple moving bottlenecks and applications in urban freight traffic management. _Transportation Research Part B: Methodological_ , Vol. 104, 2017, pp. 238–255.
  * Kouvaritakis and Cannon [2016] Kouvaritakis, B. and M. Cannon, Model predictive control. _Switzerland: Springer International Publishing_ , Vol. 38, 2016.
  * Roca-Riu et al. [2017] Roca-Riu, M., J. Cao, I. Dakic, and M. Menendez, Designing dynamic delivery parking spots in urban areas to reduce traffic disruptions. _Journal of Advanced Transportation_ , Vol. 2017, 2017.
  * Gonzales and Christofa [2017] Gonzales, E. J. and E. Christofa, Real-time signal control accounting for urban freight deliveries. In _2017 5th IEEE International Conference on Models and Technologies for Intelligent Transportation Systems (MT-ITS)_ , IEEE, 2017, pp. 810–815.
  * Ye et al. [2020] Ye, Q., S. M. Stebbins, Y. Feng, E. Candela, M. Stettler, and P. Angeloudis, Intelligent management of on-street parking provision for the autonomous vehicles era. In _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_ , IEEE, 2020, pp. 1–7.
  * Stueger et al. [2022] Stueger, P. N., F. Fehn, and K. Bogenberger, Minimizing the Effects of Urban Mobility-on-Demand Pick-Up and Drop-Off Stops: A Microscopic Simulation Approach. _Transportation Research Record_ , 2022, p. 03611981221101894.
  * Chen et al. [2014] Chen, X., L. Zhang, X. He, C. Xiong, and Z. Li, Surrogate-based optimization of expensive-to-evaluate objective for optimal highway toll charges in transportation network. _Computer-Aided Civil and Infrastructure Engineering_ , Vol. 29, No. 5, 2014, pp. 359–381.
  * Chen et al. [2016] Chen, X. M., C. Xiong, X. He, Z. Zhu, and L. Zhang, Time-of-day vehicle mileage fees for congestion mitigation and revenue generation: A simulation-based optimization method and its real-world application. _Transportation Research Part C: Emerging Technologies_ , Vol. 63, 2016, pp. 71–95.
  * Liang et al. [2021] Liang, Y., Z. Ren, L. Wang, H. Liu, and W. Du, Surrogate-assisted cooperative signal optimization for large-scale traffic networks. _Knowledge-Based Systems_ , Vol. 234, 2021, p. 107542.
  * Osorio [2019] Osorio, C., High-dimensional offline origin-destination (OD) demand calibration for stochastic traffic simulators of large-scale road networks. _Transportation Research Part B: Methodological_ , Vol. 124, 2019, pp. 18–43.
  * Lighthill and Whitham [1955] Lighthill, M. J. and G. B. Whitham, On kinematic waves II. A theory of traffic flow on long crowded roads. _Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences_ , Vol. 229, No. 1178, 1955, pp. 317–345.
  * Richards [1956] Richards, P. I., Shock waves on the highway. _Operations research_ , Vol. 4, No. 1, 1956, pp. 42–51.
  * Simoni and Claudel [2018] Simoni, M. D. and C. G. Claudel, A simulation framework for modeling urban freight operations impacts on traffic networks. _Simulation Modelling Practice and Theory_ , Vol. 86, 2018, pp. 36–54.
  * Claudel and Bayen [2010] Claudel, C. G. and A. M. Bayen, Lax–Hopf Based Incorporation of Internal Boundary Conditions Into Hamilton-Jacobi Equation. Part II: Computational Methods. _IEEE Transactions on Automatic Control_ , Vol. 55, No. 5, 2010, pp. 1158–1174.
  * Courant et al. [1967] Courant, R., K. Friedrichs, and H. Lewy, On the partial difference equations of mathematical physics. _IBM journal of Research and Development_ , Vol. 11, No. 2, 1967, pp. 215–234.
  * Tavassoli et al. [2022] Tavassoli, A., Y. Waghei, and A. Nazemi, Comparison of Kriging and artificial neural network models for the prediction of spatial data. _Journal of Statistical Computation and Simulation_ , Vol. 92, No. 2, 2022, pp. 352–369.
  * Joachims [2005] Joachims, T., A support vector method for multivariate performance measures. In _Proceedings of the 22nd international conference on Machine learning_ , 2005, pp. 377–384.
  * The Mathworks [2021] The Mathworks, I., _MATLAB Global Optimization Toolbox Release_. Natick, Massachusetts, United States, 2021.
  * The Mathworks Inc [2021] The Mathworks Inc, _MATLAB Parallel Computing Toolbox Release_. Natick, Massachusetts, United States, 2021.


# Intent Detection and Entity Extraction from BioMedical Literature

###### Abstract

Biomedical queries have become increasingly prevalent in web searches,
reflecting the growing interest in accessing biomedical literature. Despite
recent research on large-language models (LLMs) motivated by endeavors to
attain generalized intelligence, their efficacy in replacing task and domain-
specific natural language understanding approaches remains questionable. In
this paper, we address this question by conducting a comprehensive empirical
evaluation of intent detection and named entity recognition (NER) tasks from
biomedical text. We show that Supervised Fine Tuned approaches are still
relevant and more effective than general-purpose LLMs. Biomedical transformer
models such as PubMedBERT can surpass ChatGPT on NER task with only 5
supervised examples.

Intent Detection and Entity Extraction from BioMedical Literature

Ankan Mullick1*, Mukur Gupta2*, ††thanks: *Authors contributed equally Pawan
Goyal1
---
1Computer Science and Engineering Department, IIT Kharagpur, India
2Computer Science Department, Columbia University, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Research on large-language models has sky-rocketed in the post-ChatGPT era.
Researchers are now aiming for generalized intelligence by increasing model
size Brown et al. (2020); Chowdhery et al. (2022); Hoffmann et al. (2022),
expanding & rearranging pretraining data Touvron et al. (2023a, b); Sarkar and
Gupta (2021) and incorporating human feedback Ouyang et al. (2022); Dubois et
al. (2023). It is shown that the adoption of GPT-4 OpenAI (2023) can
potentially affect up to 80% of the U.S. workforce Eloundou et al. (2023).
These generalization reasoning demonstrations raise an important question for
the research community - does this mark an end to the task and domain-specific
natural language understanding approaches? While some research places LLMs as
“General Purpose Technologies" Eloundou et al. (2023); Zhang et al. (2023a)
for solving a range of complicated tasks, we show that these models struggle
to perform well on domain-specific complex tasks and specialized Supervised
Fine-tuned (SFT) models are still needed to solve language understanding use-
cases.

Over the past two decades, web searches have evolved dramatically
transitioning from generic interfaces to more intent-specific and entity-aware
systems capable of immediately displaying diverse multi-modal responses.
Particularly, biomedical inquiries, spanning topics such as medical treatment,
medical diagnosis, disease, etc. have seen a surge in popularity across search
engines. Fig. 1 shows the increase in the percentage of Biomedical queries on
Bing search and Google trends111Google trends data of last 10 years on five
topics (Health, Medical Treatment, Medical diagnosis, Disease, Pharmaceutical
drug) was gathered from Google Trends (https://trends.google.com/trends/).

As large volumes of biomedical data continue to be generated every second on
various online platforms the role of information retrieval systems in
processing domain-specific texts becomes increasingly important. However,
handling biomedical text data presents unique challenges, as the medical
queries on search engines and online medical forums are often incomplete, do
not follow a specific structure, and contain hard-to-interpret context-
specific medical terminologies, as shown in Table 1. While recent research is
centered around the development of general-purpose LLMs, that are shown to
exhibit exceptional Common Sense Reasoning capabilities Touvron et al.
(2023b), we show that these models face challenges in transferring their
performance to intricate biomedical domains. To this end, we focus on two
crucial natural language understanding tasks of intent detection and named
entity recognition from biomedical text.

(a) Google yearly Trend

(b) Bing Query yearly

Figure 1: Biomedical query search Statistics

Biomedical Text Intent Pharmacokinetic properties of abacavir were not altered
by the addition of either lamivudine or zidovudine or the combination of
lamivudine and zidovudine. Drug Canavan disease, or spongy degeneration of the
brain, is a severe leukodystrophy caused by deficiency of aspartoacylase
(ASPA). Disease

Table 1: Intent & corresponding Entity (highlighted) examples from DDI and
NCBI Disease datasets.

For the past two decades, different directions of intent detection and
corresponding entity extraction have been explored. Sun et al. (2016); Wang et
al. (2020); Mu et al. (2017b, a) demonstrate intent detection in the form of
out-of-domain data detection. Other research works explore methods like few
shot (Xia et al., 2021), zero-shot Xia et al. (2018), and clustering
frameworks Mullick et al. (2022b). Yani et al. (2022); Zhao et al. (2021);
Fetahu et al. (2022) explore entity recognition task in various settings. In
the medical domain, Zhou et al. (2021) focuses on smart healthcare and Galea
et al. (2018); Giorgi and Bader (2019); Lee et al. (2019) inspect transformer
based models for biomedical literature. Mullick et al. (2023, 2022a); Mullick
(2023b, a) aims at intent detection and entity extraction and Zhang et al.
(2017) explore medical query intents by applying graph-based frameworks.
Mullick et al. (2024); Guha et al. (2021) work on domain specific entity and
corresponding relation extraction. Mullick et al. (2017b, 2016, 2018a, 2018b,
2019, a) aim at opinion-fact entity extraction.

There is no unified and exhaustive comparison of existing approaches with the
recent LLMs for intent detection and entity extraction tasks across various
datasets in biomedical literature. Our work differs from the prior research in
two ways: we present a thorough empirical evaluation of the intent detection
on three datasets and corresponding named entity extraction (NER) approaches
on 27 unique entities covered in 5 biomedical datasets spanning across domains
like drugs, diseases, chemicals, genetics and, human anatomy. We evaluate
various supervised approaches (transformer-based, handcrafted features, etc.)
and benchmark them against two widely used large language models in the
biomedical domain. Our experiments reveal that the biomedical transformer-
based PubMedBERT model outperforms few-shot prompted ChatGPT (Turbo 3.5) on 4
biomedical NER benchmarks with just 5 supervised examples. We make our code
publicly available.222https://github.com/bioNLU-coling2024/biomed-NER-
intent_detection

## 2\. Datasets

We show our comparative study on a variety of datasets, which are widely used
as benchmarks in the biomedical domain. We use five different Named Entity
Recognition datasets: JNLPBA Collier and Kim (2004), DDI (combining DDI-
Drugbank and DDI-Medline) Segura-Bedmar et al. (2013), BC5CDR Smith et al.
(2008), NCBI-Disease Li et al. (2016) and AnatEM Ohta et al. (2012). Dataset
statistics including the entity types, count, and train-test splits are
outlined in Table 2. We use the pre-defined train-test divisions from the
respective manuscripts.

Along with the two popular intent detection datasets - CMID (Chen et al.,
2020) and KUAKE-QIC (part of the CBLUE (Zhang et al., 2021) benchmark), we
combine the three of the above five NER datasets (JNLPBA, DDI, and NCBI-
Disease) with respective intent labels (DDI for drugs, NCBI-Disease for
disease and JNLPBA for Genetics) for intent classification task - termed as
“Intent-Merged” dataset. Dataset statistics are summarized in Table 3.

CMID and KUAKE-QIC datasets, which are originally in Chinese, are translated
to English using Google Translation API. For translation validation, a random
sample of 400 translated (to English) examples of each dataset are validated
manually by two Chinese experts (ALA Language Center Company) with HSK Level-3
proficiency. The human-validation shows 91.75% and 97.0% translation accuracy
for CMID and KUAKE-QIC respectively. Hence, we use the translated English data
along with their pre-defined intent labels for our experiments. The inter-
annotator agreement is 0.89.

Dataset Entity Type # Entities #Train #Test JNLPBA Gene & Protein 5 2000 404
DDI Drug 4 714 112 BC5CDR Chem & Diesease 2 1000 500 NCBI-Disease Disease 4
693 100 AnatEM Anatomy 12 300 200

Table 2: Statistics of the NER datasets. We use the pre-defined train-test
split as mentioned in the papers.

Dataset #Train #Test Size #Intents CMID 9558 2696 4 KUAKE-QIC 6931 1955 11
Intent-Merged 3905 909 3

Table 3: Statistics of Intent Detection datasets.

## 3\. Experimental Settings

### 3.1. Intent Detection

Intent detection is a multi-class classification task where we evaluate the
accuracy of instruction-tuned ChatGPT (gpt-3.5-turbo-instruct) against various
SFT models on three English datasets: CMID, KUAKE-QIC, and Intent-Merged.

1\. Large Language Models: To ensure consistency with prior works, we employ a
$k$-shot prompt design, wherein $k$ examples per class from the training set
are used in the prompt. Given the larger text sizes of the Intent-Merged
dataset and the limited context window of LLMs, we use $k=1$ for all datasets.
We note no significant performance improvement with increasing $k$ for CMID
and KUAKE-QIC datasets. Further details on the prompt template are included in
the GitHub repository.

2\. Supervised Fined-Tuned Models: For SFT, we finetune - BERT (bert-base-
uncased) Devlin et al. (2018), RoBERTa (roberta-base) Liu et al. (2019),
PubMedBERT Gu et al. (2021), FastText Chen et al. (2020) and TextCNN Kim
(2014).
The empirical evaluation is shown in Table 4.

Model CMID KUAKE-QIC Intent-Merged Mean BERT 72.26 75.91 96.37 81.51 RoBERTa
72.88 78.16 99.11 83.38 PubMedBERT 72.70 76.88 97.90 82.49 Llama-2 51.11 42.50
39.54 44.38 ChatGPT 42.36 44.04 64.44 50.28 Fasttext 68.43 72.48 96.80 79.24
TextCNN 70.69 75.19 96.15 80.68

Table 4: Accuracy (in %) of intent classification tasks on three datasets.

All the SFT approaches consistently outperform the instruction-tuned ChatGPT.
The poor performance of LLMs on the Intent-Merged dataset, which is quite easy
for all the SFT approaches, reflects their deficiency in domain-specific
knowledge within their general-purpose pretraining datasets. This also shows
that models like FastText can outperform ChatGPT, given domain-specific
finetuning. We note that transformer architectures give better performance on
the translated corpus compared to FastText and TextCNN, which are shown to
work well on Chinese data Chen et al. (2020). RoBERTa gives the highest
accuracy across overall mean and individual datasets.

Model/Dataset DDI JNLPBA BC5CDR NCBI Disease AnatEM Mean BERT (A) $83.94\pm
0.17$ $72.60\pm 0.13$ $87.13\pm 0.24$ $77.44\pm 0.75$ $78.66\pm 0.35$ 79.95
RoBERTa (A) $87.13\pm 0.44$ $74.91\pm 0.11$ $89.50\pm 0.09$ $81.67\pm 0.51$
$81.90\pm 0.60$ 83.06 BioBERT (B) $88.06\pm 0.08$ $74.02\pm 0.40$ $90.19\pm
0.13$ $81.91\pm 0.80$ $83.43\pm 0.36$ 83.52 PubMedBERT (B) $88.84\pm 0.12$
$75.15\pm 0.06$ $90.77\pm 0.08$ $82.34\pm 0.16$ $84.21\pm 0.21$ 84.26 BioMed
RoBERTa (B) $88.76\pm 0.31$ $75.14\pm 0.25$ $90.24\pm 0.18$ $82.13\pm 0.92$
$82.70\pm 0.17$ 83.80 Clinical BERT (B) $83.79\pm 0.22$ $72.54\pm 0.07$
$87.90\pm 0.17$ $76.34\pm 0.64$ $73.47\pm 0.53$ 78.80 LSTM (C) $73.00\pm 0.01$
$67.00\pm 0.01$ $79.01\pm 0.01$ $70.00\pm 0.01$ $74.01\pm 0.01$ 72.60 LSTM +
CRF (C) $74.75\pm 0.04$ $70.67\pm 0.01$ $80.47\pm 0.02$ $73.13\pm 0.02$
$77.39\pm 0.05$ 75.23 CNN (C) $73.07\pm 0.01$ $68.04\pm 0.01$ $80.00\pm 0.01$
$67.09\pm 0.01$ $72.08\pm 0.01$ 72.06 CNN + CRF (C) $73.29\pm 0.08$ $70.84\pm
0.11$ $81.27\pm 0.14$ $73.59\pm 0.09$ $75.15\pm 0.13$ 74.83 Logistic
Regression (D) $78.63\pm 0.01$ $57.03\pm 0.01$ $78.20\pm 0.01$ $56.36\pm 0.01$
$65.48\pm 0.01$ 67.14 XGBoost (D) $73.55\pm 0.01$ $53.06\pm 0.01$ $67.86\pm
0.01$ $52.62\pm 0.01$ $59.91\pm 0.01$ 61.40 BINDER-BioBERT (E) $89.01\pm 0.01$
$76.63\pm 0.19$ $91.59\pm 0.09$ $\textbf{85.47}\pm 0.36$ $86.71\pm 0.25$ 85.88
BINDER-PubMedBERT (E) $\textbf{89.12}\pm 0.01$ $77.01\pm 0.01$
$\textbf{91.88}\pm 0.01$ $85.25\pm 0.02$ $\textbf{86.95}\pm 0.02$ 86.04
BINDER-RoBERTa (E) $87.98\pm 0.01$ $\textbf{77.08}\pm 0.01$ $90.48\pm 0.03$
$84.62\pm 0.06$ $83.91\pm 0.05$ 84.81 ChatGPT (F) $42.94\pm 3.10$ $24.5\pm
1.89$ $44.68\pm 2.78$ $19.65\pm 1.21$ $2.92\pm 0.07$ 26.94

Table 5: Experiment results (Macro average F1-Scores and corresponding
standard deviations) on different NER systems trained/finetuned and tested on
5 biomedical Datasets.

### 3.2. Named Entity Recognition

For NER, we apply a strict match between the predicted entity class and the
entity word boundaries and report strict F1-score (as in CoNLL shared task
(Tjong Kim Sang and De Meulder, 2003)). We run all models 5 times with
different random initialization and report micro-average F1-score along with
standard deviations. We also report the overall mean for each approach. For a
fair comparison, a maximum sequence length of 512 tokens was used for all
models, hence the texts larger the token length were further broken into
multiple texts.

#### 3.2.1. Supervised Fine-Tuned Models:

We thoroughly examine five different settings on the five biomedical datasets.

Setting A: Fine-tuned BERT and RoBERTa models are used (pre-trained on general
English corpus) without domain pretraining.

Setting B: Transformer systems with continued pretraining on biomedical text.
We fine-tune BioBERT, PubMedBERT, BioMed RoBERTa, and ClinicalBERT.

Setting C: LSTM and Convolutional Neural Networks (with/without CRF) are used
to generate the word embeddings and softmax classifiers for tag prediction.

Setting D: Hand-crafted word level features with ML classifier: (i) POS tag
(ii) shallow parsing features like chunk tag (iii) orthographic boolean
features like all capital, is alphanumeric, etc. (iv) n-gram features, etc. We
use the GENIA tagger333http://www.nactem.ac.uk/GENIA/tagger/ for POS and Chunk
tag extraction. We apply XGBoost and a multi-label logistic regression model
for NER tag prediction.

Setting E: We use state-of-the-art NER model BINDER Zhang et al. (2023b) along
with domain-specific (PubMedBERT and BioBERT) and RoBERTa encoders.

#### 3.2.2. Large Language Models:

We use instruction-tuned ChatGPT (gpt-3.5-turbo-instruct).

Setting F: We modify Wang et al. (2023) for reconditioning NER as a Tag
generation problem. In addition to the prompt design proposed by Wang et al.
(2023), following Zhang et al. (2023b) we also add a short description for the
entity. Each prompt infers only a single entity tag. Hence, each text instance
is passed multiple times for tagging all the entity types. We provide two
examples from the train set in each prompt. To motivate both high recall and
prevent hallucination in Entity identification, we specifically pick examples
with the median number of entity tags in the training dataset.

The evaluation outcomes are shown in Table 5. We find:

a) SFTs outperform LLMs: We observe that all SFT approaches surpass ChatGPT by
a big margin. Further, from Table 6, it’s evident that PubMedBERT can easily
outperform ChatGPT on most benchmarks with just five supervised examples.

b) Transformer SFT Models: i) PubMedBERT learns good embedding vectors due to
the largest pretraining corpus. BINDER combined with PubMedBERT gives the best
F1 score as it is able to leverage high-quality embeddings along with entity
descriptions which pushes the similar entity tokens closer in the embedding
space with a contrastive loss objective. (ii) LSTM/CNN-based neural embedding
and traditional ML-based models - XGBoost and Logistic Regression perform
poorly because they fail to capture contexts and do not leverage domain-
specific pretraining.

c) Feature-based SFT Models: (i) ML-based model performs better than CNN/LSTM
embedding systems on the DDI dataset, implying that it might be possible to
beat the performances on other datasets if the right feature set is selected,
which is usually an expensive process. (ii) The range of performance (best F1
- worst F1) for NCBI-Diease corpus is highest, showing that there is a big
difference between the selected feature set and the features captured by the
neural models. (iii) The addition of a CRF prediction layer on CNN/LSTM
improves the performance significantly.

d) Dataset Quality: In most of the cases, low F1 is observed on the entities
having fewer examples in the training set. For example, entities
"CompositeMention" and "Disease Class" show poor performance due to less
number of samples in training data. We note that the tag generation problem is
difficult for instruction-tuned LLMs. We also experiment with Llama-2 (7b)
model444https://ai.meta.com/llama/ and observe that vanilla Llama-2-7b does
not achieve good results as it was unable to follow the specified output
structure and most of times, ended up hallucinating text. So, we omit vanilla
Llama-2 results and will explore further in future.

#### 3.2.3. Experimental Setup:

We experiment on Tesla T4 16GB GPU, 6 Gbps clock cycle and GDDR5 memory. All
experiments (entity extraction and intent detection) took $\sim$60 minutes for
training. We fine-tune the models for a maximum of 20 epochs with a learning
rate of 5e-5 with AdamW optimizer and 10% warm-up steps. The batch size is 16.
Additional details are included in the GitHub Repository.

(a) PubMedBERT

(b) Binder-PubMedBERT

Figure 2: Ablation: Varying Training Size

# Shots DDI JNLPBA BC5CDR NCBI AnatEM 5 2.8 / 60.05 2.0 / 39.09 1.1 / 64.53
3.2 / 27.56 2.6 / 1.5 10 5.8 / 65.54 5.6 /50.56 2.1 /69.23 7.05 /34.32 6.1 /
3.5 30 45.49 /73.71 61.07 / 58.01 54.07 / 77.97 60.63 / 48.16 21.74 / 10.9 50
81.39 / 76.85 71.04 / 62.23 83.3 / 82.78 75.83 / 49.51 40.69 / 30.96 100 83.56
/ 80.94 74.24 / 68.02 88.58 / 85.76 82.38 / 72.78 73.34 / 52.73

Table 6: BINDER-PubMedBERT / PubMedBERT F1-score in K-shot setting

## 4\. Ablations

We study the relationship of SFT models with domain-specific finetuning data:

Varying Training Data Size: We vary the size of training data (10%, 25%, 50%,
75% and 100%), while keeping the test set constant and show the performances
of PubMedBERT, Binder (PubMedBERT) models in Figure 2. We observe that, unlike
raw PubMedBERT, Binder (PubMedBERT) attains a high performance with only 10%
of training data. Transformer-based models can learn with very little training
data and performance does not decrease much even with 25% training data. Due
to domain pre-training, PubMedBERT learns with much fewer samples and
saturates faster. This quick-learning behavior seems to be originating from
transfer learning. However, LSTM and CNN models suffer from poor performances
in low-data settings due to no pretraining (details in GitHub).

Few Shot: In Table 6, we show the performances (F1 Score) of the Binder
(PubMedBERT) and PubMedBERT models on a few shot settings with different
numbers of training samples. PubMedBERT embeddings perform better in very low-
resource setups (5, 10 shots). However, when training examples increase
further (30 shots onwards), BINDER (PubMedBERT) outperforms PubMedBERT because
of the Bi-Encoder architecture trained on contrastive learning objectives.

Error Type Entity Text Label Prediction Boundary 3-[(…)ethynyl] pyridine
B-D,I-D B-D,B-D Entity type heparinase III B-D,I-D B-G,I-G Entity Miss
Hyaluronan lyase B-D,I-D O,O

Table 7: Errors by BINDER-PubMedBERT on entity "drug_n" of DDI dataset.
Following abbreviations are used - B-D: B-DRUG_N, I-D: I-DRUG_N, B-G:B-Group,
I-G: I-GROUP

## 5\. Error Analysis

We do a detailed analysis on errors as following:

A) Some errors are due to model failure like RoBERTa’s failure to classify 52%
of the "other" intents from the KUAKE-QIC dataset. For example, a query such
as “I have a cyst in the corner of my right eye and it grows bigger and
bigger.” is classified wrongly as “diagnosis” intent but it is of “other”
category.

B) Three types of errors are observed for entity extraction (examples from the
DDI dataset are shown in Table 7).

C) Some models fail to identify the entity “drug_n” which represents new or
unapproved drugs so a periodic model update is required.

D) Relaxing entity-type error by considering exact F1-score instead of strict
F1, we observe an uplift of 4.57% in mean F1.

## 6\. Conclusion

The biomedical sector has matured significantly in the past few years. We show
instead of relying on general-purpose LLMs, it is important to design an
intent detection and entity extraction task for processing domain-specific
texts. In this work, we show that fine-tuned RoBERTa and BINDER (PubMedBERT)
can work efficiently to detect intents and extract named entities across
various benchmark datasets in biomedical literature. In the future, we aim to
extract intent and entity jointly as a relation tuple and inspect the
performances of various cross-domain scenarios.

## Limitations

Our dataset needs to be scaled up in terms of different languages, sizes, and
intent labels which we aim to do in the near future. The approach needs to be
updated as a single model for jointly extracting intents and entities for
multilingual scenarios which we aim to do as a part of future work.

## Ethical Concerns

We propose to release the algorithmic details and work on public datasets that
neither reveal any personal sensitive information nor any toxic statement. So
there are no ethical concerns in this work.

##

  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Chen et al. (2020) Nan Chen, Xiangdong Su, Tongyang Liu, Qizhi Hao, and Ming Wei. 2020.  A benchmark dataset and case study for chinese medical question intent classification.  _BMC Medical Informatics and Decision Making_ , 20.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.
  * Collier and Kim (2004) Nigel Collier and Jin-Dong Kim. 2004.  Introduction to the bio-entity recognition task at JNLPBA.  In _Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP)_ , pages 73–78, Geneva, Switzerland. COLING.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Alpacafarm: A simulation framework for methods that learn from human feedback.
  * Eloundou et al. (2023) Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023.  Gpts are gpts: An early look at the labor market impact potential of large language models.  _arXiv preprint arXiv:2303.10130_.
  * Fetahu et al. (2022) Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2022.  Dynamic gazetteer integration in multilingual models for cross-lingual and cross-domain named entity recognition.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2777–2790, Seattle, United States. Association for Computational Linguistics.
  * Galea et al. (2018) Dieter Galea, Ivan Laponogov, and Kirill Veselkov. 2018.  Exploiting and assessing multi-source data for supervised biomedical named entity recognition.  _Bioinformatics_ , 34(14):2474–2482.
  * Giorgi and Bader (2019) John M Giorgi and Gary D Bader. 2019.  Towards reliable named entity recognition in the biomedical domain.  _Bioinformatics_ , 36(1):280–286.
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare_ , 3(1):1–23.
  * Guha et al. (2021) Souradip Guha, Ankan Mullick, Jatin Agrawal, Swetarekha Ram, Samir Ghui, Seung-Cheol Lee, Satadeep Bhattacharjee, and Pawan Goyal. 2021.  Matscie: An automated tool for the generation of databases of methods and parameters used in the computational materials science literature.  _Computational Materials Science (Comput. Mater. Sci.)_ , 192:110325.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022.  Training compute-optimal large language models.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  _arXiv preprint arXiv:1408.5882_.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Li et al. (2016) Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016.  BioCreative V CDR task corpus: a resource for chemical disease relation extraction.  _Database (Oxford)_ , 2016:baw068.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mu et al. (2017a) Xin Mu, Kai Ming Ting, and Zhi-Hua Zhou. 2017a.  Classification under streaming emerging new classes: A solution using completely-random trees.  _IEEE Transactions on Knowledge and Data Engineering_ , 29(8):1605–1618.
  * Mu et al. (2017b) Xin Mu, Feida Zhu, Juan Du, Ee-Peng Lim, and Zhi-Hua Zhou. 2017b.  Streaming classification with emerging new class by class matrix sketching.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31.
  * Mullick (2023a) Ankan Mullick. 2023a.  Exploring multilingual intent dynamics and applications.  _IJCAI Doctoral Consortium_.
  * Mullick (2023b) Ankan Mullick. 2023b.  Novel intent detection and active learning based classification (student abstract).  _arXiv e-prints_ , pages arXiv–2304.
  * Mullick et al. (2024) Ankan Mullick, Akash Ghosh, G Sai Chaitanya, Samir Ghui, Tapas Nayak, Seung-Cheol Lee, Satadeep Bhattacharjee, and Pawan Goyal. 2024.  Matscire: Leveraging pointer networks to automate entity and relation extraction for material science knowledge-base construction.  _Computational Materials Science_ , 233:112659.
  * Mullick et al. (2018a) Ankan Mullick, Surjodoy Ghosh D, Shivam Maheswari, Srotaswini Sahoo, Suman Kalyan Maity, and Pawan Goyal. 2018a.  Identifying opinion and fact subcategories from the social web.  In _Proceedings of the 2018 ACM International Conference on Supporting Group Work_ , pages 145–149.
  * Mullick et al. (2016) Ankan Mullick, Pawan Goyal, and Niloy Ganguly. 2016.  A graphical framework to detect and categorize diverse opinions from online news.  In _Proceedings of the Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media (PEOPLES)_ , pages 40–49.
  * Mullick et al. (2017a) Ankan Mullick, Pawan Goyal, Niloy Ganguly, and Manish Gupta. 2017a.  Extracting social lists from twitter.  In _Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017_ , pages 391–394.
  * Mullick et al. (2018b) Ankan Mullick, Pawan Goyal, Niloy Ganguly, and Manish Gupta. 2018b.  Harnessing twitter for answering opinion list queries.  _IEEE Transactions on Computational Social Systems_ , 5(4):1083–1095.
  * Mullick et al. (2017b) Ankan Mullick, Shivam Maheshwari, Pawan Goyal, and Niloy Ganguly. 2017b.  A generic opinion-fact classifier with application in understanding opinionatedness in various news section.  In _Proceedings of the 26th International Conference on World Wide Web Companion_ , pages 827–828.
  * Mullick et al. (2023) Ankan Mullick, Ishani Mondal, Sourjyadip Ray, R Raghav, G Chaitanya, and Pawan Goyal. 2023.  Intent identification and entity extraction for healthcare queries in indic languages.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1825–1836.
  * Mullick et al. (2022a) Ankan Mullick, Abhilash Nandy, Manav Nitin Kapadnis, Sohan Patnaik, and R Raghav. 2022a.  Fine-grained intent classification in the legal domain.  _arXiv preprint arXiv:2205.03509_.
  * Mullick et al. (2019) Ankan Mullick, Sourav Pal, Projjal Chanda, Arijit Panigrahy, Anurag Bharadwaj, Siddhant Singh, and Tanmoy Dam. 2019.  D-fj: Deep neural network based factuality judgment.  _Technology_ , 50:173.
  * Mullick et al. (2022b) Ankan Mullick, Sukannya Purkayastha, Pawan Goyal, and Niloy Ganguly. 2022b.  A framework to generate high-quality datapoints for multiple novel intent detection.  _arXiv preprint arXiv:2205.02005_.
  * Ohta et al. (2012) Tomoko Ohta, Sampo Pyysalo, Jun’ichi Tsujii, and Sophia Ananiadou. 2012.  Open-domain anatomical entity mention detection.  In _Proceedings of the Workshop on Detecting Structure in Scholarly Discourse_ , pages 27–36, Jeju Island, Korea. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Sarkar and Gupta (2021) Dipankar Sarkar and Mukur Gupta. 2021.  Tso: Curriculum generation using continuous optimization.  _CoRR_.
  * Segura-Bedmar et al. (2013) Isabel Segura-Bedmar, Paloma Martínez, and María Herrero-Zazo. 2013.  SemEval-2013 task 9 : Extraction of drug-drug interactions from biomedical texts (DDIExtraction 2013).  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 341–350, Atlanta, Georgia, USA. Association for Computational Linguistics.
  * Smith et al. (2008) Larry Smith, Lorraine K Tanabe, Rie Johnson Nee Ando, Cheng-Ju Kuo, I-Fang Chung, Chun-Nan Hsu, Yu-Shi Lin, Roman Klinger, Christoph M Friedrich, Kuzman Ganchev, Manabu Torii, Hongfang Liu, Barry Haddow, Craig A Struble, Richard J Povinelli, Andreas Vlachos, William A Baumgartner, Jr, Lawrence Hunter, Bob Carpenter, Richard Tzong-Han Tsai, Hong-Jie Dai, Feng Liu, Yifei Chen, Chengjie Sun, Sophia Katrenko, Pieter Adriaans, Christian Blaschke, Rafael Torres, Mariana Neves, Preslav Nakov, Anna Divoli, Manuel Maña-López, Jacinto Mata, and W John Wilbur. 2008.  Overview of BioCreative II gene mention recognition.  _Genome Biol._ , 9 Suppl 2(S2):S2.
  * Sun et al. (2016) Yu Sun, Ke Tang, Leandro L Minku, Shuo Wang, and Xin Yao. 2016.  Online ensemble learning of data streams with gradually evolved classes.  _IEEE Transactions on Knowledge and Data Engineering_ , 28(6):1532–1545.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.
  * Wang et al. (2020) Min Wang, Ke Fu, Fan Min, and Xiuyi Jia. 2020.  Active learning through label error statistical methods.  _Knowledge-Based Systems_ , 189:105140.
  * Wang et al. (2023) Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023.  Gpt-ner: Named entity recognition via large language models.
  * Xia et al. (2021) Congying Xia, Wenpeng Yin, Yihao Feng, and Philip Yu. 2021.  Incremental few-shot text classification with multi-round new classes: Formulation, dataset and system.  _arXiv preprint arXiv:2104.11882_.
  * Xia et al. (2018) Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S Yu. 2018.  Zero-shot user intent detection via capsule neural networks.  _arXiv preprint arXiv:1809.00385_.
  * Yani et al. (2022) Mohammad Yani, Adila Alfa Krisnadhi, and Indra Budi. 2022.  A better entity detection of question for knowledge graph question answering through extracting position-based patterns.  _Journal of Big Data_ , 9(1):1–26.
  * Zhang et al. (2023a) Chaoning Zhang, Chenshuang Zhang, Chenghao Li, Yu Qiao, Sheng Zheng, Sumit Kumar Dam, Mengchun Zhang, Jung Uk Kim, Seong Tae Kim, Jinwoo Choi, Gyeong-Moon Park, Sung-Ho Bae, Lik-Hang Lee, Pan Hui, In So Kweon, and Choong Seon Hong. 2023a.  One small step for generative ai, one giant leap for agi: A complete survey on chatgpt in aigc era.
  * Zhang et al. (2017) Chenwei Zhang, Nan Du, Wei Fan, Yaliang Li, Chun-Ta Lu, and S Yu Philip. 2017.  Bringing semantic structures to user intent detection in online medical queries.  In _2017 IEEE International Conference on Big Data (Big Data)_ , pages 1019–1026. IEEE.
  * Zhang et al. (2021) Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, Luo Si, Yuan Ni, Guotong Xie, Zhifang Sui, Baobao Chang, Hui Zong, Zheng Yuan, Linfeng Li, Jun Yan, Hongying Zan, Kunli Zhang, Buzhou Tang, and Qingcai Chen. 2021.  Cblue: A chinese biomedical language understanding evaluation benchmark.
  * Zhang et al. (2023b) Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. 2023b.  Optimizing bi-encoder for named entity recognition via contrastive learning.
  * Zhao et al. (2021) Lingyun Zhao, Lin Li, Xinhao Zheng, and Jianwei Zhang. 2021.  A bert based sentiment analysis and key entity detection approach for online financial texts.  In _2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD)_ , pages 1233–1238. IEEE.
  * Zhou et al. (2021) Binggui Zhou, Guanghua Yang, Zheng Shi, and Shaodan Ma. 2021.  Natural language processing for smart healthcare.  _arXiv preprint arXiv:2110.15803_.