# The Importance of Directional Feedback
for LLM-based Optimizers

Allen Nie
Stanford University
<EMAIL_ADDRESS>
&Ching-An Cheng
Microsoft Research
<EMAIL_ADDRESS>
Andrey Kolobov
Microsoft Research
<EMAIL_ADDRESS>
&Adith Swaminathan
Microsoft Research
<EMAIL_ADDRESS>

###### Abstract

We study the potential of using large language models (LLMs) as an interactive
optimizer for solving maximization problems in a text space using natural
language and numerical feedback. Inspired by the classical optimization
literature, we classify the natural language feedback into directional and
non-directional, where the former is a generalization of the first-order
feedback to the natural language space. We find that LLMs are especially
capable of optimization when they are provided with directional feedback.
Based on this insight, we design a new LLM-based optimizer that synthesizes
directional feedback from the historical optimization trace to achieve
reliable improvement over iterations. Empirically, we show our LLM-based
optimizer is more stable and efficient in solving optimization problems, from
maximizing mathematical functions to optimizing prompts for writing poems,
compared with existing techniques.

##  1 Introduction

Owing to their capability to produce a diverse range of outputs similar to
that of humans, large language models (LLMs) are a powerful component for
solving many difficult problems involving language, including planning (Ichter
et al., 2023), interacting with users, understanding documents (Kojima et al.,
2022), and producing executable code (Gur et al., 2023). In addition to
harnessing LLMs in these _generative_ roles, several recent works have used
LLMs for _optimization_. So far, these efforts, such as APO (Pryzant et al.,
2023) and OPRO (Yang et al., 2023), have focused on optimization of a very
specific kind – employing LLMs to produce prompts that improve (another) LLM’s
performance. In this work, we argue that LLMs’ potential extends much further,
to general optimization problems. We showcase that LLMs are capable of
optimizing entities as dissimilar as mathematical functions and poems if they
are provided with _directional feedback_.

The notions of directional and non-directional feedback arise naturally in
many interactive decision-making domains and are tied to the classical
optimization literature (Boyd and Vandenberghe, 2004). Typically, a numerical
optimizer iterates over two steps. The first step aims to identify a “search
direction” for improvement. This information is provided to the optimizer by
an oracle, oftentimes a first-order oracle, and can be viewed as directional
feedback. The second step decides what to change about the input. The
applicability of various optimization methods depends on whether the
directional feedback information is available or not. Scenarios without
directional feedback are confined to black-box optimization methods such as
evolutionary search (Mitchell, 1998), Bayesian optimization (Mockus, 1998), or
policy gradient (Sutton et al., 1999). However, when the directional feedback
is available, one can choose the much more efficient gradient-based
optimization method, such as stochastic gradient descent or exact line search
(Boyd and Vandenberghe, 2004). This insight motivates our use of directional
feedback in LLM-driven optimization.

As we show, the presence or absence of directional feedback and the
possibility to access them is crucial for LLM-based optimization. For a
systematic study of factors that make an optimization process challenging, we
choose one of the difficult tasks proposed in the work on OPRO (listed as
failure cases in Appendix A of OPRO (Yang et al., 2023)) – navigating a bumpy
loss function landscape. We discover that an LLM-based optimizer agent’s
performance varies with the information the feedback carries, and, given
proper feedback, LLMs can strategically improve over past outputs, which makes
this previously unsolvable task solvable. In addition, we demonstrate that
using LLMs to “synthesize” feedback from a history of observations and prompts
can help optimization too.

We also explore LLMs’ optimization potential in a completely different
setting. Given the importance of feedback type in the LLM-based optimization
process and the lack of benchmarks that generate verbal feedback
automatically, we create a synthetic poem writing environment, where one can
programmatically create feedback for the LLMs. The poem environment is a
family of tasks where an LLM is asked to write a poem. A distinguishing
feature of this benchmark is that the poems must satisfy some constraints,
such as the number of syllables per line. By leveraging and synthesizing
feedback, we show that an LLM can sequentially optimize a poem-generation
prompt to yield a high success rate of producing constraint-satisfying poems.
Our results highlight the importance of studying the role of feedback in the
broader LLM-based text optimization landscape.

##  2 Preliminaries: Prompt Optimization for LLM-based Agent

An LLM-based agent’s behavior is modulated through the prompts used as inputs
to the LLM. We describe the interactive decision-making problem encountered by
an LLM-based agent, and how prompt optimization through interactive feedback
can improve the agent over time. In the following, uppercase letters, e.g.
$X$, denote random variables or sets. Lowercase letters denote realizations of
the random variables or set elements, e.g. “$X=x$" states that a r.v. $X$
takes on value $x$. Greek letters, e.g., $\xi$, denote parameters indexing
probability distributions.

Consider an agent encountering a complex task such as generating a poem with
logical constraints. The task is communicated to the agent via a text prompt
$p_{\text{task}}=\text{``Generate a poem with a rhyming scheme of abcba''}$.
The LLM produces output text $o_{1}\sim\Pr_{\tau}(O\mid
p_{\text{task}},p_{\text{tunable}})$ (e.g., a poem, or plans, or executable
code, or other texts as prompted), where $\tau$ captures LLM hyper-parameters
like sampling temperature, and $p_{\text{tunable}}$ contains any orchestrated
text inputs from other modules surrounding the LLM. In the sequel, we will
develop modules that will incorporate information gathered over time to update
$p_{\text{tunable}}$. By analogy with tunable parameters of an ML model, we
view $p_{\text{tunable}}$ as the tunable “parameters” of an LLM-based agent.

Based on the generated output $o_{1}$, a scalar reward $r_{1}$ and optionally
feedback $f_{1}$ are generated from the environment (e.g., human user
response, or logs generated by executing code in a programming environment)
and passed to the agent. For ease of notation, we assume $r\sim R$ and $f\sim
F$, but we do not make specific assumptions of the underlying distributions.
The reward can be a task success or failure boolean from the environment, or
user-provided thumbs-up/down signal. This interaction process iterates
$o_{1}\rightsquigarrow\\{r_{1},f_{1}\\},\dots,o_{t}\rightsquigarrow\\{r_{t},f_{t}\\}$,
until the environment terminates the interaction session. Figure 1(a)
illustrates the interaction process; for example, in Minecraft Voyager (Wang
et al., 2023), a prompt $p_{\text{task}}=\text{``Build a house''}$ is
translated into a code-generation request using internal orchestration that
prepends a specific $p_{\text{tunable}}$111In Voyager, these prompts are hand-
engineered and not automatically tuned.. The produced code $o_{t}$ is executed
in the Minecraft environment to generate error/debug/return messages $f_{t}$
as well as task completion flag $r_{t}$ that are returned to Voyager to refine
the code $o_{t+1}$ in subsequent iterations. The interaction session ends when
the user prompting the Voyager agent terminates it. Note that
$p_{\text{task}}$ can be interactively updated within a session (e.g., user
providing additional hints or rephrasing the task), and we only assume that
the rewards and feedbacks observed are consistent with the task that the agent
is prompted to solve.

LLM Agent Task Prompt $p_{\text{task}}$ Tunable Prompt $p_{\text{tunable}}$
Output $o$ Feedback $f$ Reward $r$

(a) Schematic of LLM-based agent. $p_{tunable}$ can be updated from feedback
and/or previous experiences via our sequential optimization.

LLM-Optimizer Optimizer Prompt $p_{\text{optimizer}}$ Interaction History
$\\{o_{t},r_{t},f_{t}\\}$ Tuned prompt ${p^{\prime}}_{\text{tunable}}$

(b) LLM-based Optimizer is a specific LLM-based agent that incorporates
previous experiences into the tunable prompt $p_{\text{tunable}}$ of another
LLM-based agent.

Define an LLM agent as $\pi:P_{\text{task}}\times
P_{\text{tunable}}\rightarrow O$. The distribution $O$ is defined by
$p_{\text{tunable}}$ alone, which we regard as the parameter of the LLM agent.
The optimization problem we need to solve is to find
$p_{\text{tunable}}^{\star}\coloneqq\operatorname*{arg\,max}_{p_{\text{tunable}}}\mathbb{E}_{o}\left[r\mid\pi(p_{\text{task}},p_{\text{tunable}})\right]$.
We can define an optimizer $g:P_{\text{tunable}}\times F\times R\rightarrow
P_{\text{tunable}}$, where the goal is to find $p_{\text{tunable}}^{\star}$
through a limited number of times that $\pi$ attempts the task. An optimal
optimizer $g^{\star}$ can find $p_{\text{tunable}}^{\star}$ with the fewest
number of attempts.

##  3 Optimizing LLM-based Agents

The LLM-based Optimizer is a specific instance of an LLM-based agent. It can
be used to improve another LLM-based agent using collected experience so that
the generated outputs have higher expected reward $\mathbb{E}_{o}\left[r\mid
o,p_{\text{task}}\right]$. The LLM-based optimizer takes a collection of
Output-Reward-Feedback $(o,r,f)$ tuples via its tunable prompt (see Figure
1(b)), and is tasked with generating a prompt ${p^{\prime}}_{\text{tunable}}$
for the LLM-based agent.

###  3.1 Fundamentals of LLM Optimization

The most common approach for optimization is through an iterative solver that
improves monotonically. However, in order to construct an iterative solver,
the optimization problem needs to satisfy a few assumptions. To establish
intuitions, we start with numerical optimization in a function approximation-
based supervised learning setting. Given a hypothesis $h$ and data sample
$(x,y)$, let $\tilde{y}=h_{\theta}(x)$. With a loss function $\ell:X\times
Y\times\Theta\rightarrow\mathbb{R}$, we can define
$L(\theta)=\mathbb{E}_{(x,y)}\left[\ell(\theta,x,y)\right]$. The goal is to
find $\theta^{\star}=\operatorname*{arg\,min}_{\theta}L(\theta)$. To achieve
this goal, a valid optimization procedure proposes a new $\theta$ for $k$
number of times. They usually consist of two steps:

  1. S1

Finding Valid Search Direction: We need to find useful information, such as a
descent direction $\Delta\theta^{(k)}$ that can inform the update step. The
usefulness of the information is tightly coupled with what the update step is.

  2. S2

Decide Update Rules: We need to decide how to update $\theta$. A typical
update procedure is simply:
$\theta^{(k+1)}=\theta^{(k)}+t^{(k)}\Delta\theta^{(k)}$, if
$\Delta\theta^{(k)}$ is informative, where $t_{k}$ is the step size.

If $L$ is convex, then the criteria to determine whether $\Delta\theta$ is
informative is quite simple: we can use the gradient of $L$. From convexity,
we know that $\nabla L(\theta^{(k)})^{T}(\theta^{(k+1)}-\theta^{(k)})\geq 0$
implies $L(\theta^{(k+1)})\geq L(\theta^{(k)})$. Then we can set the descend
direction $\Delta\theta^{(k)}$ so as to satisfy $-\nabla
L(\theta^{(k)})^{T}\Delta\theta^{(k)}\geq 0$. A simple way to satisfy this
criterion is let $\Delta\theta^{(k)}\coloneqq-\nabla L(\theta^{(k)})$, which
is the gradient descent method (GD). However, more complicated update rules
can be used, such as backtracking line search (Boyd and Vandenberghe, 2004).
Note that we do not always need to satisfy S2. For example, in an evolutionary
search algorithm, many candidates are proposed and the update rule simply
keeps the candidate with the best score.

Then we can contrast the setting with an LLM optimization problem. If we want
to have an iterative descent algorithm to find the optimal prompt for an LLM
agent, then we need to consider the following properties:

  1. S1

Search Direction: We should obtain useful information, analogous to $\nabla
L(\theta)$, to help inform the optimizer on how to update the parameter
$p_{\text{tunable}}$.

  2. S2

Update Parameter: Unlike the numerical case, where basic algebra can be
applied to update parameters, it is unclear whether there is a predefined
notion of $\Delta p_{\text{tunable}}^{(k)}$ in text space. This distance
$\Delta p_{\text{tunable}}^{(k)}$ in the best case, can be assessed by human
intuition over the semantics of the text, in the worst case, can be completely
arbitrary.

To propose an algorithm using LLM as an optimizer, we make the following
assumptions:

  1. A1

Permissible Search Direction: There exists useful information, which we
describe as feedback, $f$, for an LLM-based optimizer $g$ such that $g$ can
propose a $p_{\text{tunable}}^{k+1}$ where
$\mathbb{E}_{o}\left[r\mid\pi(p_{\text{task}},p_{\text{tunable}}^{k+1})\right]\geq\mathbb{E}_{o}\left[r\mid\pi(p_{\text{task}},p_{\text{tunable}}^{k})\right]$.

  2. A2

Valid Update: The LLM-based optimizer $g$ can modify $p_{\text{tunable}}$
based on $f$, where direction of change: $\Delta p_{\text{tunable}}$ is
determined by the information contained in $f$ (i.e., not a random text edit).

In the next few sections, we describe a few possible settings where these
assumptions can be satisfied or need not be. We assume A2 is always satisfied.
The first setting we discuss is that it is possible that LLM itself acts as a
black-box optimizer. For example, it might obtain a valid search direction by
implicitly computing finite differences between inputs and outputs in text
space.

#### LLM Might Implicitly Perform Newton’s Method

Similar to Newton’s Method using finite differences as an approximate
gradient, perhaps the LLM can implicitly compute a “function”:

$\displaystyle\nabla R=\lim_{\Delta p_{\text{tunable}}^{(k)}\rightarrow
0}\frac{\mathbb{E}_{o}\left[r\mid\pi(p_{\text{task}},p_{\text{tunable}}^{k})\right]-\mathbb{E}_{o}\left[r\mid\pi(p_{\text{task}},p_{\text{tunable}}^{k+1})\right]}{\Delta
p_{\text{tunable}}^{(k)}}$ (1)

If we think this is possible, then the input to the LLM can be tuples of
$(p_{\text{tunable}}^{1},r_{1}),...,(p_{\text{tunable}}^{k},r_{k})$. Although
it is unclear if this is truly the case, this shows that the optimizer needs
to retain a history of how past prompts $p$ have caused the reward $r$.

#### Feedback Can Be Directional

The other possibility, not relying on the black box “magic” of the LLM’s
internal process, is to hope that somehow a permissible search direction $f$
is given to us from an external source. Humans give directional feedback quite
often: “This coffee is too hot for me.” or “Can you lower the room
temperature?” The first feedback implicitly asks the agent to make a cooler
coffee (but not saying exactly how cool it should be). The second feedback
asks the agent to turn down the room temperature (but without specifying which
temperature to set). Imagine if the agent’s action (output space $O$) for both
cases is to write API calls to set temperature; then we know immediately what
$\Delta O$ should be – keep everything the same, but enter a lower temperature
value. After the adjustment, a user might say: “This coffee is now too cold
for me.” or “I’m freezing!” Making this kind of feedback very similar to the
gradient information we get from numerical optimization. This suggests that,
in some cases, incorporating feedback (or somehow obtaining directional
feedback) could be helpful for the optimization procedure. In such cases, we
can provide the LLM optimizer with examples of
$(p_{\text{tunable}}^{1},f_{1},r_{1}),(p_{\text{tunable}}^{2},f_{2},r_{2}),...,(p_{\text{tunable}}^{k},f_{k},r_{k})$
(i.e. additionally providing the feedback in each round to the optimizer).

#### Non-directional Feedback

There is another type of feedback we can consider. This type of feedback
contains useful information but is not directional because they do not
directly inform us how to change the input $O$. For example, feedback like “I
can’t drink this coffee because the temperature is not quite right.” This
feedback clearly states that the attribute of “temperature” is important to
the user and is not satisfactory. However, it does not tell us whether we
should make a coffee that’s hotter or colder. Coffee can have many attributes,
such as “temperature”, “acidity”, “roast”, “sweetness”, or “cream-level”. This
feedback is more useful than a scalar reward because it _explains_ attributes
that affect $R$ and allows us to focus more on a single dimension of many
attributes.

#### Reward as Feedback / No Feedback

Unlike directional and non-directional feedback, which usually contains
information about how to change $p_{\text{tunable}}$, score-based feedback
only gives back a numerical value indicating how well $p_{\text{tunable}}$
performs. In our setup, this means LLM-Optimizer only observes reward $r$
without $f$. This is often referred to as 0th-order feedback.

###  3.2 Sequential Prompt Optimization

Inspired by the descent method in numerical optimization, we propose an
algorithm that aims to satisfy the requirement of descent methods such that we
can reach the extremum. We define the following optimization loop with an LLM-
based agent $\pi$. Agent $\pi$ with an initial tunable prompt
$p_{\text{tunable}}$ takes a task description $p_{\text{task}}$ from the
environment and samples an output $o_{1}$. The environment returns a reward
$r_{1}$ and feedback $f_{1}$. An LLM-optimizer stores
$(o_{1},r_{1},f_{1},p_{\text{tunable}})$ in a history buffer $\mathbb{H}$.
When the buffer becomes large, we subsample $H$ from $\mathbb{H}$. The LLM-
optimizer proposes a new tunable parameter $p_{\text{tunable}}^{\prime}$. We
make an explicit decision on whether to replace $p_{\text{tunable}}$ with
$p_{\text{tunable}}^{\prime}$ based on the reward evaluated on the
distribution $o$ and $o^{\prime}$. The full procedure is in Algorithm 1. We
now describe the implementation choices for each component of our iterative
solver.

#### Policy

Policy is an LLM that takes in a tunable instruction prompt
$p_{\text{tunable}}$, description of task $p_{\text{task}}$ and produces an
output $o$. It does not see the history of interaction it did with the
environment. This design decision prohibits the policy from controlling its
own prompts. It is different from some other existing work. For example,
Voyager (Wang et al., 2023) would allow the policy to see all of its
interaction histories (and errors they make). React (Yao et al., 2023) also
allows the policy to see the full error trace. Allowing the policy to see its
past errors is a specific design choice on $p_{\text{tunable}}$. We let the
optimizer decide what $p_{\text{tunable}}$ should be (and its decision space
includes the human-engineered choices in prior works).

#### Prompt Proposal

We define the prompt proposal module as $\Delta:P_{\text{task}}\times
P_{\text{tunable}}\times F\times R\rightarrow P_{\text{tunable}}$. This module
looks at the task description, past prompts, and the feedback each prompt
receives and proposes a new prompt. If the environment provides directional or
non-directional feedback, this module should take in $F$ as well. Even though
past prompts were included, this module is allowed to produce completely new
prompts.

Input: Given $s,R$, an LLM-based agent $\pi$, an LLM-based prompt proposal
module $\Delta$, $p_{\text{tunable}}^{0}$, $p_{\text{task}}$, and $K$
iterations.  Output: $p_{\text{tunable}}^{K}$ $\mathbb{H}=\emptyset$ for
_$k\leftarrow 0...K$_ do
$o_{k},r_{k}\sim\pi(p_{\text{task}},p_{\text{tunable}}^{(k)}),R$ $f_{k}\sim F$
or $\hat{F}(p_{\text{task}},o_{k},r_{k})$
$H=\textnormal{{Sample}}(\mathbb{H})$
$p^{(k+1)}=\Delta(H,\\{p_{\text{task}},p_{\text{tunable}}^{k},f_{k},r_{k}\\})$
if
_$\mathbb{E}_{o}\left[r\mid\pi(p_{\text{task}},p_{\text{tunable}}^{k+1})\right]\geq\mathbb{E}_{o}\left[r\mid\pi(p_{\text{task}},p_{\text{tunable}}^{k})\right]$_
then $p_{\text{tunable}}^{k+1}=p_{\text{tunable}}^{k}$ end if
$\mathbb{H}=\mathbb{H}\cup\\{o_{k},r_{k},f_{k},p_{\text{tunable}}^{k}\\}$ end
for  return _$p_{\text{tunable}}^{K}$_ Algorithm 1 Sequential Prompt
Optimization

#### Feedback Synthesizer

If numerical feedback is the only type of feedback given, we can design a
feedback synthesizing module $\hat{F}:P_{\text{task}}\times O\times
R\rightarrow F$. It takes in $(o_{1},r_{1}),...,(o_{k},r_{k})$ and produces
feedback to the prompt proposal module. We designed this module to ask the
question, “How should the input be changed to have a greater effect on the
objective/output?” We note that we more specifically prompt the LLM to think
about the difference in the input space that would impact the difference in
output space, different from APO, where they ask the LLM to “give reasons why
the prompt could have gotten these examples wrong.”

#### Prompt Selector

In order to satisfy the descent method assumption A1 (permissible search
direction), we need to guarantee that $p_{\text{tunable}}^{\prime}$ is an
improvement over $p_{\text{tunable}}$. This can be achieved by setting a
selection criterion that requires $p^{(k+1)}$ to get a higher reward than
$p^{(k)}$. A simple criterion is to improve the average performance over the
distribution of $O$:
$\mathbb{E}_{o}\left[r\mid\pi(p_{\text{task}},p_{\text{tunable}}^{k+1})\right]\geq\mathbb{E}_{o}\left[r\mid\pi(p_{\text{task}},p_{\text{tunable}}^{k})\right]$.

##  4 Experiments

Figure 2: We visualize the optimization trajectory path made by the Optimizer
Agent with GPT-3.5 and GPT-4. The loss landscape on the left is the Rosenbrock
Function, and on the right is the Six-Hump Camel Function.

###  4.1 Numerical Optimization

We test if LLM can possibly do optimization and what are the necessary
ingredients for it to find the optimal solution in an optimization problem. We
set up this task as a more controlled study before studying prompt
optimization. In prompt optimization, it is often hard to know what the
optimal prompt or a good search direction is. With a numerical optimization
problem instead, both the optimal solution and a good search direction are
well-defined. We use a set of classic optimization
problems222https://www.sfu.ca/~ssurjano/optimization.html that require LLMs to
find $x$, a 2-dimensional vector. The implementation used in this paper is
included in LLF-Bench (Cheng et al., 2023).

  1. 1.

Task: Given a task description $p_{\text{task}}$ and a function $J$ (which is
hidden from the LLM), we sample a random starting point $(x_{0},J(x_{0}))$,
$x_{0}\sim X$. An LLM is asked to produce $x$ to minimize $J$.

  2. 2.

Optimizable variable: $X$. The LLM is asked to output $x$ directly. Here,
$p_{\text{tunable}}$ is the same as $x$.

  3. 3.

Output process $O$: the output module takes $x$ and directly outputs $x$, an
identity function.

  4. 4.

Reward $R$: $R(x)=-J(x)$.

  5. 5.

Feedback $F$:

     * •

Directional Feedback: $\nabla R(x)=\frac{dR}{dx}$, the first-order derivative
of the output.

     * •

Non-directional Feedback: We compare the partial derivatives $\frac{\partial
R}{\partial x_{1}}$ and $\frac{\partial R}{\partial x_{2}}$, and tell the LLM
which dimension of $x$ should be changed to accomplish the task (but without
telling LLM in which direction to change).

This experiment does not use the full setup of Algorithm 1. We only test our
LLM-based optimizer $\Delta$ and our feedback synthesizer $\hat{F}$. We choose
four functions: Booth, McCormick, Rosenbrock, and Six-Hump-Camel Function.
They were chosen because the optimal $x$ that minimizes these functions is not
$[0,0]$. In our initial experiments, we found that the LLM is quick to guess
$[0,0]$ for any problem, which trivializes the optimization problems where the
function attains its minimum at $[0,0]$.

We define simple regret as $\text{Reg}(\Delta)=|J(x_{T})-J(x^{\star})|$, where
$J$ is the function we try to minimize and $T$ is the number of optimization
steps we allow the optimizer $\Delta$ to take. We define cumulative regret as
$\text{CuReg}(\Delta)=\sum_{t=1}^{T}|J(x_{t})-J(x^{\star})|$. Intuitively,
simple regret corresponds to how close is an optimizer’s final answer $x_{T}$
to the correct answer $x^{\star}$. Cumulative regret describes how “efficient”
is the optimizer at finding the $x^{\star}$. We compare three models: $\Delta$
with GPT-3.5, GPT-4, and a stochastic gradient descent algorithm (SGD) with a
small yet fixed learning rate. We list all of the prompts we used and
additional experimental details in Appendix A.1. In the reported results, we
run 10 trials and allow $\Delta$ to take at most 10 optimization steps.

Figure 3: We plot the average Cumulative Regret and Simple Regret of each
condition over 10 trials. Each algorithm is allowed to take 10 steps. We tuned
the SGD learning rate to ensure that it was not too large or too small. The
result is aggregated over 4 loss functions.

  1. RQ1

Can an LLM Implicitly Perform Newton’s Method, given
$(x_{1},J(x_{1}),...,(x_{k},J(x_{k}))$?

From Figure 2, we can see that LLM, as an optimizer, has a rough sense of
direction given a history of past explorations. In Figure 2 (a), we note that
in both loss landscapes, although GPT-3.5 often fails to find the minimal
point without feedback (green lines), GPT-4 is able to understand the past
history and make new proposals of $x$ that incrementally minimizes $J(x)$.
This suggests that even though there is no explicit gradient computation, LLM
can be asked to “improve” based on a history of observations.

  1. RQ2

Does directional Feedback help the optimization process? Do other types of
feedback help as much?

We designed the prompt space for the LLM-based optimizer $\Delta$ to insert
feedback text right after the observation text and with an additional wording
that reads, “You should incorporate the suggestion.” Besides this change, the
optimizer agent prompt stays the same between no feedback and with feedback
conditions. The full prompt is available in Appendix A.1.

From Figure 2 (b) and Figure 3, we can see that both GPT-3.5 and GPT-4 are
able to take advantage of the additional feedback information and improve
their search direction. Feedback can help both a weaker model (GPT-3.5) and a
strong model (GPT-4). A stronger model can improve more, even if the feedback
has less information (see the comparison between Non-directional Feedback and
Directional Feedback in Section 3.1).

Figure 4: We plot the average Cumulative Regret and Simple Regret of each
condition over 10 trials and compare different feedback types. Synthetic
Feedback is generated by the same LLM as the optimizer.

Although loss minimization is a challenging task for LLMs, with some amount of
feedback, LLMs are able to find a final $x$ that is similar to a classic
optimization algorithm like SGD (see Figure 3b) – the simple regret is similar
across the two methods. It is worth noting that GPT-4’s final proposed $x$ is
not as close to the optimal as GPT-3.5. This is potentially because both
models decide their own step size, and we are limiting the optimization
horizon to 10 steps.

  1. RQ3

If directional feedback is missing, can we replace it with an LLM module to
enhance whatever feedback is available?

Oftentimes, direct and useful feedback might be missing from the environment.
In this experiment, we design a feedback synthesizer module (described in Sec
3.2) that takes the output from the model and the reward and synthesizes
feedback to improve the next output. Different from methods such as self-
reflection, self-criticism, or thinking step-by-step, the feedback synthesizer
asks questions similar to “What should I change about $x$ that will result in
a larger change in $y$?”, whereas self-reflection usually asks the model to
reflect upon past “mistakes” and critique what they did wrong.

In Figure 4, we show that we can synthesize feedback from a history of past
outputs and rewards that is able to guide the optimizer LLM to find a better
solution. Synthesized feedback is not as informative as directional feedback
that comes from the environment, but it can easily outperform settings where
no feedback is given.

Figure 5: We show the reward for each policy after each round of interaction
with the environment. OptAgent (our algorithm) is in red.

###  4.2 Poem Generation

In Section 4.1, we have validated the importance of directional feedback. We
now validate our optimization setup on a more practical domain, where we have
to optimize over a prompt that controls how another LLM-based agent produces
output. Unlike a typical mathematical reasoning or language task, the optimal
output $o^{\star}$ in the poem generation environment is defined over a text
distribution. Even for a single task $p_{\text{task}}$, we do not have direct
access to the optimal distribution $o^{\star}$ that can obtain the highest
expected reward. This is because, for a text constraint satisfaction problem,
many generated texts can obtain the full reward. We discuss the implementation
details below. This task is now included in LLF-Bench (Cheng et al., 2023).

A formal poem is a writing assignment that requires the creation of a poem to
satisfy some requirements regarding its form. For example, Haiku is a type of
formal poem that asks for three lines that form a 5-7-5 syllable pattern. This
is a challenging task for both GPT-3.5 and GPT-4, but easy for us to verify
whether the generated poem has satisfied the constraint. We can almost treat
this as a code optimization problem, where we can check if each line of the
poem, or the entire poem satisfies the constraint, and provide line-by-line
feedback if needed. The mechanical aspect of this task makes it a perfect toy
environment for prompt optimization.

  1. 1.

Task: Generate a poem with a given constraint sampled from a set of
constraints.

  2. 2.

Optimizable variable: $P_{\text{tunable}}$: This is the prompt
$p_{\text{tunable}}$ for the LLM-based agent that we want to update and
optimize.

  3. 3.

Output process $O$: the LLM agent takes the prompt $p_{\text{tunable}}$ and
follows its suggestion and a task description $p_{\text{task}}$ to produce a
poem $o$.

  4. 4.

Reward $R$: The fraction of lines in the generated poem that satisfy the
constraint described by $p_{\text{task}}$. $r\in[0,1]$.

  5. 5.

Feedback $F$:

     * •

Directional Feedback: We print out the number of syllables in the current line
and whether LLM needs to increase or decrease the number of syllables in that
line.

     * •

Non-directional Feedback: We print out how many lines violate the poem writing
constraints.

In the following experiment, we use the full setup of Algorithm 1. We allow
each agent to take 10 optimization steps. We name our agent OptAgent. It
produces an instruction that will be sent to the poem generation agent to
produce a poem. The poem-generation agent will not see the history of mistakes
or any other information. We additionally evaluate Reflexion agent (Shinn et
al., 2023). We set up four tasks: generating poems that contain 7, 8, 9, or 10
syllables for each line.

We show that in Figure 5, we can reliably select prompts that improve the
policy performance for each task. The prompt selection step in our
optimization algorithm ensures monotonic improvement in rewards. Otherwise, it
will reject the updated prompt and keep the previous prompt. This differs from
the Reflexion Agent which is not guaranteed to improve after each interaction.

##  5 Conclusion

This paper argues that LLMs can successfully optimize a wide range of entities
ranging from mathematical functions to prompts for textual tasks if provided
with directional feedback. We empirically show on challenging numerical
optimization scenarios and constrained text generation tasks that utilizing
either environment-provided or synthesized feedback is a crucial piece in LLM-
based optimization. We emphasize that this is an early work on general LLM-
based optimizers. LLMs’ potential in this role is still to be realized with
new methods for directional feedback generation.

## Acknowledgments and Disclosure of Funding

We would like to thank Ahmed Awadallah, Jennifer Neville, and Ricky Loynd for
their feedback and discussions. The work was performed between June and
September of 2023 during the first author’s internship at Microsoft Research
in Redmond.

## References

  * Boyd and Vandenberghe (2004) Stephen P Boyd and Lieven Vandenberghe.  _Convex optimization_.  Cambridge university press, 2004.
  * Cheng et al. (2023) Ching-An Cheng, Andrey Kolobov, Dipendra Misra, Allen Nie, and Adith Swaminathan.  LLF-Bench: Benchmark for interactive learning from language feedback, 2023.
  * Gur et al. (2023) Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust.  A real-world webagent with planning, long context understanding, and program synthesis.  _arXiv preprint arXiv:2307.12856_ , 2023.
  * Ichter et al. (2023) Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu.  Do as i can, not as i say: Grounding language in robotic affordances.  In _CORL_ , 2023.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  In _NeurIPS_ , 2022.
  * Mitchell (1998) Melanie Mitchell.  _An introduction to genetic algorithms_.  MIT press, 1998.
  * Mockus (1998) Jonas Mockus.  The application of bayesian methods for seeking the extremum.  _Towards global optimization_ , 2:117, 1998.
  * Pryzant et al. (2023) Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng.  Automatic prompt optimization with "gradient descent" and beam search.  _arXiv preprint arXiv:2305.03495_ , 2023.
  * Shinn et al. (2023) Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.  Reflexion: Language agents with verbal reinforcement learning.  _arXiv preprint arXiv:2303.11366_ , 2023.
  * Sutton et al. (1999) Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.  Policy gradient methods for reinforcement learning with function approximation.  In _NeurIPS_ , 1999.
  * Wang et al. (2023) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv:2305.16291_ , 2023.
  * Yang et al. (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen.  Large language models as optimizers.  _arXiv preprint arXiv:2309.03409_ , 2023.
  * Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  In _ICLR_ , 2023.

##  Appendix A Appendix

###  A.1 Loss Optimizing Experiment Details

We designed the prompt for two agents. The prompt is written in a Handlebar
syntax, where “{{}}” indicate variables to be replaced. A brief guide on this
syntax is available here333https://github.com/guidance-ai/guidance.

For the LLM-Optimizer:

⬇

{{#system~}}

You are trying to minimize the output (y) of a function by choosing input (x).

{{~/system}}

{{#user~}}

{{task_description}}

This is what you have previously chosen for x and what the ys were:

{{observation}}

{{feedback}}

You should incorporate the suggestion to output the next x.

Please output the next x that will make this function output the smallest y.

You cannot repeat the same x, doing so will result in a penalty.

Format: x = [x1, x2]

Output:

{{~/user}}

For the feedback synthesizing LLM:

⬇

{{#system~}}

You are trying to minimize the output (y) of a function by choosing input (x).

{{~/system}}

{{#user~}}

You are trying to minimize the output (y) of a function by choosing input (x).

You get to observe y once you choose the value of x, where x is a
2-dimensional vector.

This means x = [x1, x2], where x1 and x2 are real numbers.

The goal is to choose x such that y is as small as possible.

Here is a list of x and how it affects y:

{{#each history}}

{{this.action}}

{{this.observation}}

===================

{{~/each}}

For x = [x1, x2]

What are the suggestions you can give to the user to make y smaller?

For example, here are some of the things you can suggest:

- Changing x1 seems to have a bigger effect on y than changing x2.

- Make a larger change on x2

- Increase x1 by 1.2

- Decrease x2 by 0.5

- Try to increase x1 and decrease x2 at the same time

Or any other kind of suggestion. Do not make a suggestion that’s the form of a
question.

You should only make a one-sentence suggestion that’s brief and short.

Suggestion:

{{~/user}}

###  A.2 Poem Experiment Details

For the LLM-agent that generates the poem, we use the following prompt:

⬇

{{#system~}}

You are a student and your teacher gives you an assignment to write a poem.

{{~/system}}

{{#user~}}

The assignment is:

{{assignment}}

{{#if exists_intrusction}}

In addition, here are some helpful advice and guidance:

{{instruction}}

{{/if}}

{{~/user}}

\end{verbatim}

\end{small}

For the \textbf{feedback synthesizer module}, we use this prompt:

\begin{small}

\begin{Verbatim}[breaklines=true]

{{#system~}}

You are a helpful assistant who aims to provide feedback to a student who’s
writing a poem

according to some instructions.

It is important to let the student know if they did satisfy the instruction or
not and why.

{{~/system}}

{{#user~}}

This is the history of past generated poems and how well they did with respect
to instructions.

{{#each history}}

Instruction: {{this.observation}}

Poem:

{{this.action}}

Feedback from the teacher:

{{this.feedback}}

\---------------

{{~/each}}

{{~/user}}

{{#user~}}

Now, the student writes a new poem.

New instruction: {{observation}}

Poem:

{{action}}

What changes can you make to the poem to help it conform to the instructions?

{{~/user}}

{{#assistant~}}

{{gen ’exp_feedback’ temperature=0.7}}

{{~/assistant}}

For the LLM-based optimizer, we use this prompt:

⬇

{{#system~}}

You are a helpful assistant that wants to come up with instructions to a
student to help them write a poem that is satisfactory to a teacher’s
assignment.

The student’s poem needs to satisfy the requirement of this assignment.

{{~/system}}

{{#user~}}

This is the history of how you have been helping this student and whether your
instructions have succeeded.

Teacher’s feedback is the most important feedback, because the student needs
to meet the teacher’s criteria.

However, another student’s feedback can provide helpful information too.

{{#each history}}

The Assignment: "{{this.assignment}}"

Your Instruction:

{{this.prompt}}

Student’s Poem:

{{this.action}}

Teacher’s Feedback:

{{this.feedback}}

Feedback from another student:

{{this.exp_feedback}}

\---------------

{{~/each}}

{{~/user}}

{{#user~}}

Your previous instruction didn’t work \-- the students didn’t write a poem
that satisfied the teacher’s criteria.

Based on your interaction with the students, can you come up with better
instructions that can help this student write a poem that matches the
teacher’s criteria?

Keep in mind that telling the student what to do step-by-step might be very
helpful!

However, you need to be brief and to the point.

{{~/user}}

###  A.3 Implementation Detail of the Tasks

For both tasks, the implementation details can be found in Cheng et al.
(2023), and the code is available at https://github.com/microsoft/LLF-Bench.


††footnotetext: *Corresponding author

# Blending type Approximations by Kantorovich variant of $\alpha$-Schurer
operators

Nadeem Rao Department of Mathematics, Faculty of Science
Jamia Millia Islamia
New Delhi-110025
Delhi
India<EMAIL_ADDRESS>Mamta Rani Department of Mathematics
Shree Guru Gobind Singh Tricentenary University
Gurugram-122505, Haryana
India<EMAIL_ADDRESS>Adem Kiliçman Department of Mathematics &
Statistics, Faculty of Science
Universiti Putra Malaysia
43400 UPM Serdang
Selangor
Malaysia<EMAIL_ADDRESS>Pradeep Malik Department of Mathematics
Shree Guru Gobind Singh Tricentenary University
Gurugram-122505, Haryana
India<EMAIL_ADDRESS>Mohammad Ayman-Mursaleen School of Information &
Physical Sciences
The University of Newcastle
University Drive, Callaghan
New South Wales 2308
Australia
Department of Mathematics & Statistics, Faculty of Science
Universiti Putra Malaysia
43400 UPM Serdang
Selangor
Malaysia<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

In the present manuscript, we present a new sequence of operators, $i.e.$,
$\alpha$-Bernstein-Schurer-Kantorovich operators depending on two parameters
$\alpha\in[0,1]$ and $\rho>0$ for one and two variables to approximate
measurable functions on $[0:1+q],q>0$. Next, we give basic results and discuss
the rapidity of convergence and order of approximation for univariate and
bivariate of these sequences in their respective sections. Further, Graphical
and numerical analysis are presented. Moreover, local and global approximation
properties are discussed in terms of first and second order modulus of
smoothness, Peetre’s K-functional and weight functions for these sequences in
different spaces of functions.

###### Key words and phrases:

Schurer operators, modulus of continuity, rate of convergence, Gamma-function

###### 1991 Mathematics Subject Classification:

41A10, 41A25, 41A28, 41A35, 41A36

##  1\. Introduction

Bernstein (1912) [1] proposed the Bernstein polynomials as:

$\mathbb{B}_{l}(h;v)=\sum\limits_{\nu=0}^{l}P_{l,\nu}(v)h\bigg{(}\frac{\nu}{l}\bigg{)},\textrm{
$l\in\mathbb{N}$},$ (1.1)

where $P_{l,\nu}(v)=\binom{l}{\nu}v^{\nu}(1-v)^{\nu-l}$. For the operators
given by (1.1), he showed that $\mathbb{B}(g;v)$ converges to $g$ uniformly
where $g\in C[0,1]$. In 1962, Schurer [2] modified operators given in (1.1)
as: for $q>0$, a real number

$\mathbf{B}_{l,q}(g;v)=\sum\limits_{\nu=0}^{l+q}\binom{l+q}{\nu}v^{\nu}(1-v)^{\nu+q-l}g\bigg{(}\frac{\nu}{l}\bigg{)},\textrm{
$v\in[0,1+q]$},$ (1.2)

where $g\in C[0,1+q]$. One can note that for $q=0$, the polynomials presented
in (1.2) reduces to polynomials given by (1.1). The operators are introduced
in (1.1) and (1.2) are restricted for continuous functions only and are
different in respect to the domain of function $f$. Several researchers,
$e.g.$, Mursaleen et al. ([3], [4],) Acar et al. ([5], [6]), Mohiuddine et al.
[7], Ana et al. [8], İçöz et al. ([9]), [10]), Kajla et al. ([11], [12])
constructed new sequences of linear positive operators to investigate the
rapidity of convergence and order of approximation in different functional
spaces in terms of several generating functions. In the recent past, for
$g\in[0,1],m\in\mathbb{N}$ and $\lambda\in[-1,1]$, Chen et al. [13]
constructed a sequence of new linear positive operators as:

$T_{m,\lambda}(g;y)=\sum_{i=0}^{m}g\left(\frac{i}{m}\right)p_{m,i}^{\lambda}(y)\qquad(y\in[0,1]),$
(1.3)

where $p_{1,0}^{(\lambda)}=1-y$, $p_{1,1}^{(\lambda)}=y$ and

$\displaystyle p_{m,i}^{\lambda}(y)$ $\displaystyle=$
$\displaystyle\left[(1-\lambda)y\binom{m-2}{i}+(1-\lambda)(1-y)\binom{m-2}{i-2}+\lambda
y(1-y)\binom{m}{i}\right]$ (1.4) $\displaystyle
y^{i-1}(1-y)^{m-i-1}\quad(m\geq 2).$

The operators defined in (1.3) are named as $\lambda-$Berstein operator of
order m.

######  Remark 1.1.

One can not that for $\lambda=1$, the relation (1.3) reduces to classical
Bernstein operators [1].

Later, Aral and Erbay [14] introduced a parametric extension of Baskakov
operators. Recently, Özger et al. [15] constructed a sequence
$\lambda$-Bernstein-Schurer operators as: For every $g\in C_{B}[0,\infty)$
where $C_{B}[0,\infty)$ stands for the continuous and bounded function,

$\displaystyle\Psi_{n,\nu,\lambda}(g;u)$ $\displaystyle=$
$\displaystyle\sum_{k=0}^{n+\nu}g_{k}q_{n,\nu,k}^{(\lambda)}(u)$ (1.5)
$g_{k}=g\left(\frac{k}{n}\right)$

Now, we construct the $\lambda-$ Bernstein- Schurer kantorovich operators and
their moments

$\displaystyle\Psi_{n,\nu,\lambda}^{*}(g;u)$ $\displaystyle=$
$\displaystyle(n+1)\sum_{k=0}^{n+\nu}q_{n,\nu,k}^{(\lambda)}(u)\int_{\frac{k}{n+1}}^{\frac{k+1}{n+1}}g(s)ds$
(1.6) $\displaystyle\mathcal{Q}_{n,k}^{(\lambda)}(u)$ $\displaystyle=$
$\displaystyle\frac{u^{k-1}}{(1+u)^{n+k-1}}\Bigg{\\{}\frac{\lambda
u}{1+u}\binom{n+k-1}{k}-(1-\lambda)(1+u)\binom{n+k-3}{k-2}$ $\displaystyle+$
$\displaystyle(1-\lambda)u\binom{n+k-1}{k}\Bigg{\\}},$

with $\binom{n-3}{-2}=\binom{n-2}{-1}=0$. Motivating by the above development,
we introduce positive linear operators as follows:

$\displaystyle K_{n,\lambda}^{\rho}(f;y)$ $\displaystyle=$
$\displaystyle\sum_{i=0}^{m+q}p_{m,i}^{\lambda}(y)\int_{0}^{1}g\left(\frac{i+t^{\rho}}{m+1}\right)ds,$
(1.7) $\displaystyle p_{m,i}^{\lambda}(y)$ $\displaystyle=$
$\displaystyle\Big{[}(1-\lambda)y\binom{m+q-2}{i}+(1-\lambda)(1-y)\binom{m+q-2}{i-2}$
$\displaystyle+$ $\displaystyle\lambda
y(1-y)\binom{m+q}{i}\Big{]}y^{i-1}(1-y)^{m+q-i-1}\quad(m\geq 2).$

where $\rho>o$ and $\mathcal{Q}_{n,k}^{(\lambda)}(u)$ is given by (1).
In the subsequent sections, we investigate basic Lemmas, rate of convergence,
order of approximation results. Locally and globally approximation results in
terms of modulus of continuity, Peetre‘s K-functional, second order modulus of
smoothness, Lipschitz class and Lipschitz maximal function, weight functions.
Further, $\lambda$-Bivariate Schurer Kantorovich operators are constructed and
their pointwise and uniform approximation results are investigated.

##  2\. Basic Estimates

######  Lemma 2.1.

[15]For the operator defined in (1.5), one has

$\displaystyle\Psi_{n,\nu,\lambda}(e_{0};u)$ $\displaystyle=1,$
$\displaystyle\Psi_{n,\nu,\lambda}(e_{1};u)$ $\displaystyle=u+\frac{\nu}{n}u,$
$\displaystyle\Psi_{n,\nu,\lambda}(e_{2};u)$
$\displaystyle=u^{2}+\frac{(n+\nu+2(1-\lambda))(u-u^{2})}{n^{2}}+\frac{\nu(\nu+2n)u^{2}}{n^{2}}.$

######  Lemma 2.2.

Let $e_{k}(s)=s^{k}$, $k\in\\{0,1,2\\}$. Then, for the operators defined in
(1.6), we have

$\displaystyle\Psi_{n,\nu,\lambda}^{*}(e_{0};u)$ $\displaystyle=1,$
$\displaystyle\Psi_{n,\nu,\lambda}^{*}(e_{1};u)$
$\displaystyle=\bigg{(}\frac{n+\nu}{n+1}\bigg{)}u+\frac{1}{2(n+1)}$
$\displaystyle\Psi_{n,\nu,\lambda}^{*}(e_{2};u)$
$\displaystyle=\left[\frac{n^{2}}{(n+1)^{2}}-\frac{n+\nu+2(1-\lambda)}{(n+1)^{2}}+\frac{\nu(\nu+2n)}{(n+1)^{2}}\right]u^{2}$
$\displaystyle+\left[\frac{n+\nu+2(1-\lambda)}{(n+1)^{2}}+\frac{n+\nu}{(n+1)^{2}}\right]u+\frac{1}{3(n+1)^{2}}.$

######  Lemma 2.3.

For the operator defined in (1.7), we have

$\displaystyle K_{n,\lambda}^{\rho}(e_{0};u)$ $\displaystyle=1,$
$\displaystyle K_{n,\lambda}^{\rho}(e_{1};u)$
$\displaystyle=\frac{n+2(\lambda-1)}{n+1}u+\frac{(\lambda+1)(\rho+1)+1}{2(\rho+1)(n+1)},$
$\displaystyle K_{n,\lambda}^{\rho}(e_{2};u)$
$\displaystyle=\left(1+\frac{4\lambda-3}{n}\right)\frac{n^{2}u^{2}}{(n+1)^{2}}$
$\displaystyle+\frac{[(\rho+1)(n(2\lambda+3)+(\lambda-1)(2\lambda+7))]+4(\lambda-1)}{(\rho+1)(n+1)^{2}}u$
$\displaystyle+\frac{2n(2\rho+1)+(\lambda+1)(2\rho+1)((\lambda+2)(\rho+1)+2)+\rho+1}{(2\rho+1)(\rho+1)(n+1)^{2}}.$

###### Proof.

Using Lemma 2.3, one can easily prove Lemma 2.3. ∎

######  Lemma 2.4.

Let $e_{k}(s)=(e_{1}(s)-u)^{k}=\psi_{u}^{k}(s)$, $k\in{\mathbb{N}}$ be the
central moments of $K_{n,\lambda}^{\rho}(.;.)$ constructed in (1.7). Then, we
have

$\displaystyle K_{n,\lambda}^{\rho}((e_{1}(s)-u);u)$
$\displaystyle=\frac{2\lambda-3}{n+1}u+\frac{(\lambda+1)(\rho+1)+1}{(\rho+1)(n+1)},$
$\displaystyle K_{n,\lambda}^{\rho}((e_{1}(s)-u)^{2};u)$
$\displaystyle=\left[\left(1+\frac{4\lambda-3}{n}\right)\frac{n^{2}}{(n+1)^{2}}-\frac{2n+4\lambda-1}{n+1}+1\right]u^{2}$
$\displaystyle+\frac{[(\rho+1)(n(2\lambda+3)+(\lambda-1)(2\lambda+7)-2(\lambda+1))]+\lambda-6}{(\rho+1)(n+1)^{2}}u$
$\displaystyle+\frac{2n(2\rho+1)+(\lambda+1)(2\rho+1)((\lambda+2)(\rho+1)+2)+\rho+1}{(2\rho+1)(\rho+1)(n+1)^{2}}.$

###### Proof.

Using Lemma 2.2, Lemma 2.3 can easily be proved. ∎

##  3\. Convergence behaviour of $K_{n,\lambda}^{\rho}(.;.)$

######  Definition 3.1.

[7] For $g\in C[0,1+q],q>0$, the modulus of continuity for a uniformly
continuous function $g$ is defined as:

$\displaystyle\omega(g;\delta)=\sup_{|r_{1}-r_{2}|\leq\delta}|g(r_{1})-g(r_{2})|,\hskip
28.45274ptr_{1},r_{2}\in[0,1+p],q>0.$

Let $g$ be a uniformly continuous function in $C[0,1+q],q>0$ and $\delta>0$.
Then, one get

$\displaystyle|g(r_{1})-g(r_{2})|\leq\bigg{(}1+\frac{(r_{1}-r_{2})^{2}}{\delta^{2}}\bigg{)}\omega(g;\delta).$
(3.1)

######  Theorem 3.1.

Let $K_{n,\lambda}^{\rho}(.;.)$ be sequence of operators proposed by (1.7).
Then, $K_{n,\lambda}^{\rho}$ converges uniformly to $f$ on each bounded subset
of $[0,1+q],q>0$ where $f\in C[0,1+q],q>0\bigcap\Bigg{\\{}f:u\geq
0,\frac{f(u)}{1+u^{2}}$ converges as $u\rightarrow\infty\Bigg{\\}}.$

###### Proof.

To prove this result, it is adequate to prove that

$\displaystyle K_{n,\lambda}^{\rho}(e_{i};u)\rightarrow e_{i}(u),\textrm{ for
}i\in\\{0,1,2\\}.$

Using Lemma 2.3, it is clear that $K_{n,\lambda}^{\rho}(e_{i};u)\rightarrow
e_{i}(u)$ for $i=0,1,2$ as $n\rightarrow\infty$. Hence, Theorem 3.1 is proved.
∎

######  Example.

One can note that, for the following set of parameters $q=5$, $\rho=0.1$ and
$\lambda=0.5$, the operator $K_{n,\lambda}^{\rho}(f;x)$ converges uniformly to
the function $f(y)=y^{3}-5y^{2}+6y+2$ as $n$ increases which is illustrated in
the figure1.

Figure 1. Approximation by operator $K_{n,\lambda}^{\rho}(;,;)$ Figure 2. Error estimation of operators $K_{n,\lambda}^{\rho}(;,;)$ for the different values of $n$ $x$ | $E_{20,\lambda}^{\rho}(f;x)$ | $E_{30,\lambda}^{\rho}(f;x)$ | $E_{40,\lambda}^{\rho}(f;x)$
---|---|---|---
0.1 | 0.2717372121 | 0.1887446733 | 0.1445360958
0.2 | 0.2677254718 | 0.1886482134 | 0.1455017073
0.3 | 0.2412358918 | 0.1732429202 | 0.1348677403
0.4 | 0.1951644878 | 0.1444179547 | 0.1140349291
0.5 | 0.1324072752 | 0.1040624783 | 0.0844040078
0.6 | 0.0558602697 | 0.0540656519 | 0.0473757106
0.7 | 0.0315805132 | 0.0036833631 | 0.0043507716
0.8 | 0.1270190580 | 0.0672954058 | 0.0432700748
0.9 | 0.2275593491 | 0.134881315 | 0.0940860947
1 | 0.3303053711 | 0.2045519293 | 0.1466965539
Table 1. Error estimation table

The figure 2 and Table 1 are also demonstrated our analytical results.

######  Theorem 3.2.

(See [16]) Let $L:C([a,b])\rightarrow B([a,b])$ be a linear and positive
operator and let $\varphi_{x}$ be the function defined by

$\displaystyle\varphi_{x}(t)=|t-x|,\textrm{ }(x,t)\in[a,b]\times[a,b].$

If $f\in C_{B}([a,b])$ for any $x\in[a,b]$ and any $\delta>0$, the operator
$L$ verifies:

$\displaystyle|(Lf)(x)-f(x)|$ $\displaystyle\leq|f(x)||(Le_{0})(x)-1|$
$\displaystyle+\\{(Le_{0})(x)+\delta^{-1}\sqrt{(Le_{0})(x)(L\varphi_{x}^{2})(x)}\\}\omega_{f}(\delta).$

######  Theorem 3.3.

Let the operators $K_{n,\lambda}^{\rho}(.;.)$ be introduced by (1.7) and $f\in
C_{B}[0,1+p],q>0$, we have

$\displaystyle|K_{n,\lambda}^{\rho}(f;u)-f(u)|\leq 2\omega(f;\delta),$

where $\delta=\sqrt{K_{n,\lambda}^{\rho}(\psi_{u}^{2};u)}.$

###### Proof.

In view of Theorem 3.2, Lemma 2.3 and Lemma 2.4, one has

$\displaystyle|K_{n,\lambda}^{\rho}(f;u)-f(u)|\leq\\{1+\delta^{-1}\sqrt{K_{n,\lambda}^{\rho}(f;u)(\psi_{u}^{2};u)}\\}\omega(f;\delta).$

On choosing $\delta=\sqrt{K_{n,\lambda}^{\rho}(\psi_{u}^{2};u)}$, we completes
the proof of this result. ∎

##  4\. Pointwise Approximation results

Here, we consider the Lipschitz type space [19] as

$\displaystyle
Lip_{M}^{k_{1},k_{2}}(\gamma)\\!:=\\!\Big{\\{}f\\!\in\\!C_{B}[0,1+p],q>0:|f(t)\\!-\\!f(u)|\\!\leq\\!M\frac{|t\\!-\\!u|^{\gamma}}{(t\\!+\\!k_{1}u\\!+\\!k_{2}u^{2})^{\frac{\gamma}{2}}}:u,t\in(0,\infty)\Big{\\}},$

where $M\geq 0$ is a real valued constant number, $k_{1},k_{2}>0,$ $\rho>0$
and $\gamma\in(0,1]$.

######  Theorem 4.1.

For $f\in Lip_{M}^{k_{1},k_{2}}(\gamma)$, one yield

$\displaystyle|K_{n,\lambda}^{\rho}(f;u)-f(u)|\leq
M\Bigg{(}\frac{\eta_{n}^{*}(u)}{k_{1}u+k_{2}u^{2}}\Bigg{)}^{\frac{\gamma}{2}},$
(4.1)

where $u>0$ and $\eta_{n}^{*}(u)=K_{n,\lambda}^{\rho}(\psi_{u}^{2};u)$.

###### Proof.

For $\gamma=1$, we have

$\displaystyle|K_{n,\lambda}^{\rho}(f;u)-f(u)|$ $\displaystyle\leq
K_{n,\lambda}^{\gamma}{(|f(t)-f(u)|)}(u)$ $\displaystyle\leq
MK_{n,\lambda}^{\rho}\left(\frac{|t-u|}{(t+k_{1}u+k_{2}u^{2})^{\frac{1}{2}}};u\right).$

Since $\frac{1}{t+k_{1}u+k_{2}u^{2}}<\frac{1}{k_{1}u+k_{2}u^{2}}$ for all
$t,u\in(0,\infty)$, we get

$\displaystyle|K_{n,\lambda}^{\rho}(f;u)-f(u)|$
$\displaystyle\leq\frac{M}{(k_{1}u+k_{2}u^{2})^{\frac{1}{2}}}(K_{n,\lambda}^{\rho}((t-u)^{2};u))^{\frac{1}{2}}$
$\displaystyle\leq
M\Bigg{(}\frac{\eta_{n}^{*}(u)}{k_{1}u+k_{2}u^{2}}\Bigg{)}^{\frac{1}{2}}.$

This implies that for $\gamma=1,$ this result stand good. Now, for
$\gamma\in(0,1)$ and using Hölder’s inequality with $p=\frac{2}{\gamma}$ and
$q=\frac{2}{2-\gamma}$, one obtain

$\displaystyle|K_{n,\lambda}^{\rho}(f;u)-f(u)|$
$\displaystyle\leq\left(K_{n,\lambda}^{\rho}((|f(t)-f(u)|)^{\frac{2}{\gamma}};u)\right)^{\frac{\gamma}{2}}$
$\displaystyle\leq
M\left(K_{n,\lambda}^{\rho}\left(\frac{|t-u|^{2}}{(t+k_{1}u+k_{2}u^{2})};u\right)\right)^{\frac{\gamma}{2}}.$

Since $\frac{1}{t+k_{1}u+k_{2}u^{2}}<\frac{1}{k_{1}u+k_{2}u^{2}}$ for all
$t,u\in(0,\infty)$, we obtain

$\displaystyle|K_{n,\lambda}^{\rho}(f;u)-f(u)|$ $\displaystyle\leq
M\Bigg{(}\frac{\mathcal{P}_{n}^{\mu,\nu}\left(|t-u|^{2};u\right)}{k_{1}u+k_{2}u^{2}}\Bigg{)}^{\frac{\gamma}{2}}$
$\displaystyle\leq
M\Big{(}\frac{\eta_{n}^{*}(u)}{k_{1}u+k_{2}u^{2}}\Big{)}^{\frac{\gamma}{2}}.$

Hence, we arrive at the desired result. ∎

##  5\. Global Approximation

From [17], we recall some notation to prove the global approximation results.
For the weight function $1+u^{2}$ and $0\leq u<\infty$, we have
$B_{1+u^{2}}[0,1+p],q>0=\\{f(u):|f(u)|\leq M_{f}(1+u^{2}),$ $M_{f}$ is
constant depending on $f$}.
$C_{1+u^{2}}[0,1+p],q>0\subset B_{1+u^{2}}[0,1+p],q>0$ space of continuous
functions endowed with the norm
$\|f\|_{1+u^{2}}=\sup\limits_{u\in[0,1+p],q>0}\frac{|f|}{1+u^{2}}$.
and

$C_{1+u^{2}}^{k}[0,1+p],q>0=\\{f\in
C_{1+u^{2}}:\lim\limits_{u\rightarrow\infty}\frac{f(u)}{1+u^{2}}=k,\textrm{
where }k\textrm{ is a constant}\\}.$

######  Theorem 5.1.

Let the $K_{n,\lambda}^{\rho}(.;.)$ be the operators given by (1.7) and
$K_{n,\lambda}^{\rho}(.;.):C_{{1+u^{2}}}^{k}[0,1+p],q>0\rightarrow
B_{1+u^{2}}[0,1+p],q>0.$ Then, we have

$\displaystyle\lim\limits_{n\rightarrow\infty}\|K_{n,\lambda}^{\rho}(f;u)-f\|_{1+u^{2}}=0,$

where $f\in C_{{1+u^{2}}}^{k}[0,1+p],q>0.$

###### Proof.

To prove this result, it is sufficient to show that

$\displaystyle\lim\limits_{n\rightarrow\infty}\|K_{n,\lambda}^{\rho}(e_{i};u)-u^{i}\|_{1+u^{2}}=0,\textrm{
}i=0,1,2.$

From Lemma 2.3, we get

$\displaystyle\|K_{n,\lambda}^{\rho}(e_{0};u)-u^{0}\|_{1+u^{2}}=\sup\limits_{u\in[0,1+p],q>0}\frac{|K_{n,\lambda}^{\rho}(1;u)-1|}{1+u^{2}}=0\textrm{
for }i=0.$

For $i=1$

$\displaystyle\|K_{n,\lambda}^{\rho}(e_{1};u)\\!-\\!u^{1}\|_{1+u^{2}}$
$\displaystyle\\!=\\!\sup\limits_{u\in[0,1+p],q>0}\frac{\frac{n+2(\lambda-1)}{n+1}u+\frac{(\lambda+1)(\rho+1)+1}{2(\rho+1)(n+1)}}{1+u^{2}}$
$\displaystyle\\!=\bigg{(}\frac{n+2(\lambda-1)}{n+1}-1\bigg{)}\sup\limits_{u\in[0,1+p],q>0}\frac{u}{1+u^{2}}$
$\displaystyle+\frac{(\lambda+1)(\rho+1)+1}{2(\rho+1)(n+1)}\sup\limits_{u\in[0,1+p],q>0}\frac{1}{1+u^{2}}.$

Which implies that
$\|K_{n,\lambda}^{\rho}(e_{1};u)-u^{1}\|_{1+u^{2}}\rightarrow 0$ an
$n\rightarrow\infty$.
Similarly, we see that
$\|K_{n,\lambda}^{\rho}(e_{2};u)-u^{2}\|_{1+u^{2}}\rightarrow 0$ as
$n\rightarrow\infty$. ∎

##  6\. Construction of bivariate Szász-Durrmeyer-Operators
$H_{n_{1},n_{2}}^{\ast}(.;.)$ and their Basic Estimates

Take $\mathcal{I}^{2}=\\{(y_{1},y_{2}):0\leq y_{1}<1+q_{1},\;0\leq
y_{2}<1+q_{2}\\}$ and $C\left(\mathcal{I}^{2}\right)$ is the class of all
continuous functions on $\mathcal{I}^{2}$ equipped with the norm
$||g||_{C\left(\mathcal{I}^{2}\right)}=\sup_{(y_{1},y_{2})\in\mathcal{I}^{2}}|g(y_{1},y_{2})|$.
Then for all $h\in C\left(\mathcal{I}^{2}\right)$ and
$n_{1},n_{2}\in\mathbb{N}$, we construct sequence of bivariate bivariate
generalized baskakov operators as follows:

$\displaystyle K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(f;y_{1},y_{2})$
$\displaystyle=$
$\displaystyle\sum_{i_{1}=0}^{m_{1}+q_{1}}p_{m_{1},i_{1}}^{\lambda_{1}}(y_{1})\sum_{i_{2}=0}^{m_{2}+q_{2}}p_{m_{2},i_{2}}^{\lambda_{2}}(y_{2})$
$\displaystyle\int_{0}^{1}\int_{0}^{1}g\left(\frac{i_{1}+t_{1}^{\rho}}{m_{1}+1},\frac{i_{2}+t_{2}^{\rho}}{m_{2}+1}\right)dt_{1}dt_{2},$

where

$\displaystyle p_{m_{k},i_{k}}^{\lambda_{k}}(y_{k})$ $\displaystyle=$
$\displaystyle\Bigg{[}(1-\lambda)y_{k}\binom{m_{k}+q_{k}-2}{i_{k}}+(1-\lambda_{k})(1-y_{k})\binom{m_{k}+q_{k}-2}{i_{k}-2}$
$\displaystyle+$ $\displaystyle\lambda
y_{k}(1-y_{k})\binom{m_{k}+q_{k}}{i_{k}}\Bigg{]}y_{k}^{i_{k}-1}(1-y_{k})^{m_{k}+q_{k}-i_{k}-1}\quad(m_{1},m_{2}\geq
2).$

######  Lemma 6.1.

Let ${e_{i,j}}=y_{1}^{i}y_{2}^{j}$ be the central moments. Then, for the
operators (6), we have

$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0,0};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle 1,$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{1,0};y_{i},y_{2})$
$\displaystyle=$
$\displaystyle\frac{m_{1}+2(\lambda_{1}-1)}{m_{1}+1}y_{1}+\frac{(\lambda_{1}+1)(\rho+1)+1}{2(\rho+1)(m_{1}+1)},$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0,1};y_{i},y_{2})$
$\displaystyle=$
$\displaystyle\frac{m_{1}+2(\lambda_{1}-1)}{m_{2}+1}y_{2}+\frac{(\lambda_{2}+1)(\rho+1)+1}{2(\rho+1)(m_{2}+1)},$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{1,1};y_{i},y_{2})$
$\displaystyle=$
$\displaystyle\frac{m_{1}+2(\lambda_{1}-1)}{m_{1}+1}y_{1}+\frac{(\lambda_{1}+1)(\rho+1)+1}{2(\rho+1)(m_{1}+1)}$
$\displaystyle\times$
$\displaystyle\frac{m_{2}+2(\lambda_{2}-1)}{m_{2}+1}y_{2}+\frac{(\lambda_{2}+1)(\rho+1)+1}{2(\rho+1)(m_{2}+1)},$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{2,0};y_{i},y_{2})=\left(1+\frac{4\lambda_{1}-3}{m_{1}}\right)\frac{m_{1}^{2}y_{1}^{2}}{(m_{1}+1)^{2}}$
$\displaystyle+$
$\displaystyle\frac{[(\rho+1)(m_{1}(2\lambda_{1}+3)+(\lambda_{1}-1)(2\lambda_{1}+7))]+4(\lambda_{1}-1)}{(\rho+1)(m_{1}+1)^{2}}y_{1}$
$\displaystyle+$
$\displaystyle\frac{2m_{1}(2\rho+1)+(\lambda_{1}+1)(2\rho+1)((\lambda_{1}+2)(\rho+1)+2)+\rho+1}{(2\rho+1)(\rho+1)(m_{1}+1)^{2}},$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0,2};y_{i},y_{2})=\left(1+\frac{4\lambda_{2}-3}{m_{2}}\right)\frac{m_{2}^{2}y_{2}^{2}}{(m_{2}+1)^{2}}$
$\displaystyle+$
$\displaystyle\frac{[(\rho+1)(m_{2}(2\lambda_{2}+3)+(\lambda_{2}-1)(2\lambda_{2}+7))]+4(\lambda_{2}-1)}{(\rho+1)(m_{2}+1)^{2}}y_{2}$
$\displaystyle+$
$\displaystyle\frac{2m_{2}(2\rho+1)+(\lambda_{2}+1)(2\rho+1)((\lambda_{2}+2)(\rho+1)+2)+\rho+1}{(2\rho+1)(\rho+1)(m_{2}+1)^{2}}.$

###### Proof.

In the light of Lemma (2.3) and linearity property, we have

$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0,0};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0};y_{i},y_{2}),$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{1,0};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{1};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0};y_{i},y_{2}),$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0,1};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{1};y_{i},y_{2}),$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{1,1};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{1};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{1};y_{i},y_{2}),$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{2,0};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{2};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0};y_{i},y_{2}),$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0,2};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{0};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(e_{2};y_{i},y_{2}),$

which proves Lemma (6.1). ∎

######  Lemma 6.2.

Let
$\Psi_{y_{1},y_{2}}^{i,j}(t,s)=\eta_{i,j}(t,s)=(t-y_{1})^{i}(s-y_{2})^{j},\quad
i,j\in\\{0,1,2\\}$ be the central moments functions. Then from the operators
$K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(.;.)$ defined by (6) satisfies
the following identities

$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0,0};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle 1$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{1,0};y_{i},y_{2})$
$\displaystyle=$
$\displaystyle\frac{2\lambda_{1}-3}{m_{1}+1}y_{1}+\frac{(\lambda_{1}+1)(\rho+1)+1}{(\rho+1)(m_{1}+1)},$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0,1};y_{i},y_{2})$
$\displaystyle=$
$\displaystyle\frac{2\lambda_{1}-3}{m_{2}+1}y_{2}+\frac{(\lambda_{2}+1)(\rho+1)+1}{(\rho+1)(m_{2}+1)}$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{1,1};y_{i},y_{2})$
$\displaystyle=$
$\displaystyle\frac{2\lambda_{1}-3}{m_{1}+1}y_{1}+\frac{(\lambda_{1}+1)(\rho+1)+1}{(\rho+1)(m_{1}+1)}$
$\displaystyle\times$
$\displaystyle\frac{2\lambda_{1}-3}{m_{2}+1}y_{2}+\frac{(\lambda_{2}+1)(\rho+1)+1}{(\rho+1)(m_{2}+1)}$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{2,0};y_{i},y_{2})=\left[\left(1+\frac{4\lambda_{1}-3}{m_{1}}\right)\frac{m_{1}^{2}}{(m_{1}+1)^{2}}-\frac{2m_{1}+4\lambda_{1}-1}{m_{1}+1}+1\right]y_{1}^{2}$
$\displaystyle+$
$\displaystyle\frac{[(\rho+1)(m_{1}(2\lambda_{1}+3)+(\lambda_{1}-1)(2\lambda_{1}+7)-2(\lambda_{1}+1))]+\lambda_{1}-6}{(\rho+1)(m_{1}+1)^{2}}y_{1}$
$\displaystyle+$
$\displaystyle\frac{2m_{1}(2\rho+1)+(\lambda_{1}+1)(2\rho+1)((\lambda_{1}+2)(\rho+1)+2)+\rho+1}{(2\rho+1)(\rho+1)(m_{1}+1)^{2}},$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0,2};y_{i},y_{2})=\left[\left(1+\frac{4\lambda_{2}-3}{m_{2}}\right)\frac{m_{2}^{2}}{(m_{2}+1)^{2}}-\frac{2m_{2}+4\lambda_{2}-1}{m_{2}+1}+1\right]y_{2}^{2}$
$\displaystyle+$
$\displaystyle\frac{[(\rho+1)(m_{1}(2\lambda_{2}+3)+(\lambda_{2}-1)(2\lambda_{2}+7)-2(\lambda_{2}+1))]+\lambda_{2}-6}{(\rho+1)(m_{2}+1)^{2}}y_{2}$
$\displaystyle+$
$\displaystyle\frac{2m_{2}(2\rho+1)+(\lambda_{2}+1)(2\rho+1)((\lambda_{2}+2)(\rho+1)+2)+\rho+1}{(2\rho+1)(\rho+1)(m_{2}+1)^{2}}.$

###### Proof.

In the light of Lemma (6.1) and linearity property, we have

$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0,0};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0};y_{i},y_{2}),$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{1,0};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{1};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0};y_{i},y_{2}),$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0,1};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{1};y_{i},y_{2}),$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{1,1};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{1};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{1};y_{i},y_{2}),$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{2,0};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{2};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0};y_{i},y_{2}),$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0,2};y_{i},y_{2})$
$\displaystyle=$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{0};y_{i},y_{2})K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\eta_{2};y_{i},y_{2}),$

which proves Lemma (6.2). ∎

##  7\.  Degree of Convergence

For any $g\in C(\mathcal{I}^{2})$ and $\delta>0$ modulus of continuity of
order second is given by

$\omega\left(g;\delta_{n_{1}},\delta_{n_{2}}\right)=\sup\\{\mid
g(t,s)-g(y_{1},y_{2})\mid:(t,s),(y_{1},y_{2})\in\mathcal{I}^{2}\\}$

with $\mid t-y_{1}\mid\leq\delta_{n_{1}},\;\mid s-y_{2}\mid\leq\delta_{n_{2}}$
with the partial modulus of continuity defined as:

$\omega_{1}\left(g;\delta\right)=\sup_{0\leq y_{2}\leq\infty}\sup_{\mid
x_{1}-x_{2}\mid\leq\delta}\\{\mid g(x_{1},y_{2})-g(x_{2},y_{2})\mid\\},$

$\omega_{2}\left(g;\delta\right)=\sup_{0\leq y_{1}\leq\infty}\sup_{\mid
y_{1}-y_{2}\mid\leq\delta}\\{\mid g(y_{1},y_{1})-g(y_{1},y_{2})\mid\\}.$

######  Theorem 7.1.

For any $g\in C(\mathcal{I}^{2})$ we have

$\mid
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(g;y_{1},y_{2})-g(y_{1},y_{2})\mid\leq
2\bigg{(}\omega_{1}(g;\delta_{y_{1},n_{1}})+\omega_{2}(g;\delta_{n_{2},y_{2}})\bigg{)}.$

###### Proof.

In order to give the prove of Theorem 7.1, in general we use well-known
Cauchy-Schwarz inequality. Thus we see that

$\displaystyle\mid
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(g;y_{1},y_{2})$
$\displaystyle-$ $\displaystyle g(y_{1},y_{2})\mid\leq
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\mid
g(t,s)-g(y_{1},y_{2})\mid;y_{1},y_{2}\right)$ $\displaystyle\leq$
$\displaystyle K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\mid
g(t,s)-g(y_{1},s)\mid;y_{1},y_{2}\right)$ $\displaystyle+$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\mid
g(y_{1},s)-g(y_{1},y_{2})\mid;y_{1},y_{2}\right)$ $\displaystyle\leq$
$\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\omega_{1}(g;\mid
t-y_{1}\mid);y_{1},y_{2}\right)$ $\displaystyle+$ $\displaystyle
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\omega_{2}(g;\mid
s-y_{2}\mid);y_{1},y_{2}\right)$ $\displaystyle\leq$
$\displaystyle\omega_{1}(g;\delta_{n_{1}})\left(1+\delta_{n_{1}}^{-1}K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\mid
t-y_{1}\mid;y_{1},y_{2})\right)$ $\displaystyle+$
$\displaystyle\omega_{2}(g;\delta_{n_{2}})\left(1+\delta_{n_{2}}^{-1}K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\mid
s-y_{2}\mid;y_{1},y_{2})\right)$ $\displaystyle\leq$
$\displaystyle\omega_{1}(g;\delta_{n_{1}})\left(1+\frac{1}{\delta_{n_{1}}}\sqrt{K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}((t-y_{1})^{2};y_{1},y_{2})}\right)$
$\displaystyle+$
$\displaystyle\omega_{2}(g;\delta_{n_{2}})\left(1+\frac{1}{\delta_{n_{2}}}\sqrt{K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}((s-y_{2})^{2};y_{1},y_{2})}\right).$

If we choose
$\delta_{n_{1}}^{2}=\delta_{n_{1},y_{1}}^{2}=K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}((t-y_{1})^{2};y_{1},y_{2})$
and
$\delta_{n_{2}}^{2}=\delta_{n_{2},y_{2}}^{2}=K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}((s-y_{2})^{2};y_{1},y_{2}),$
then we easily to reach our desired results. ∎

Here, we find convergence in terms of the Lipschitz class for bivariate
function. For $M>0$ and $\tau,\tau\in[0,1+p],q>0,$ Lipschitz maximal function
space on $E\times E\subset\mathcal{I}^{2}$ defined by

$\displaystyle\mathcal{L}_{\tau,\tau}(E\times E)$ $\displaystyle=$
$\displaystyle\bigg{\\{}g:\sup(1+t)^{\tau}(1+s)^{\tau}\left(g_{\tau,\tau}(t,s)-g_{\tau,\tau}(y_{1},y_{2})\right)$
$\displaystyle\leq$ $\displaystyle
M\frac{1}{(1+y_{1})^{\tau}}\frac{1}{(1+y_{2})^{\tau}}\bigg{\\}},$

where $g$ is continuous and bounded on $\mathcal{I}^{2}$, and

$g_{\tau,\tau}(t,s)-g_{\tau,\tau}(y_{1},y_{2})=\frac{\mid
g(t,s)-g(y_{1},y_{2})\mid}{\mid t-y_{1}\mid^{\tau}\mid
s-y_{2}\mid^{\tau}};\quad(t,s),(y_{1},y_{2})\in\mathcal{I}^{2}.$ (7.1)

######  Theorem 7.2.

Let $g\in\mathcal{L}_{\tau,\tau}(E\times E),$ then for any
$\tau,\tau\in[0,1+p],q>0,$ there exists $M>0$ such that

$\displaystyle\mid
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(g;y_{1},y_{2})-g(y_{1},y_{2})\mid$
$\displaystyle\leq$ $\displaystyle
M\bigg{\\{}\bigg{(}\left(d(y_{1},E)\right)^{\tau}+\left(\delta_{n_{1},y_{1}}^{2}\right)^{\frac{\tau}{2}}\bigg{)}$
$\displaystyle\times$
$\displaystyle\bigg{(}\left(d(y_{2},E)\right)^{\tau}+\left(\delta_{n_{2},y_{2}}^{2}\right)^{\frac{\tau}{2}}\bigg{)}$
$\displaystyle+$
$\displaystyle\left(d(y_{1},E)\right)^{\tau}\left(d(y_{2},E)\right)^{\tau}\bigg{\\}},$

where $\delta_{n_{1},y_{1}}$ and $\delta_{n_{2},y_{2}}$ defined by Theorem
7.1.

###### Proof.

Take $\mid y_{1}-x_{0}\mid=d(y_{1},E)$ and $\mid y_{2}-y_{0}\mid=d(y_{2},E)$.
For any $(y_{1},y_{2})\in\mathcal{I}^{2}$, and $(x_{0},y_{0})\in E\times E$ we
let $d(y_{1},E)=\inf\\{\mid y_{1}-y_{2}\mid:y_{2}\in E\\}$. Thus we can write
here

$\mid g(t,s)-g(y_{1},y_{2})\mid\leq M\mid g(t,s)-g(x_{0},y_{0})\mid+\mid
g(x_{0},y_{0})-g(y_{1},y_{2})\mid.$ (7.2)

Apply $K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}$, we obtain

$\displaystyle\mid
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(g;y_{1},y_{2})$
$\displaystyle-$ $\displaystyle g(y_{1},y_{2})\mid$ $\displaystyle\leq$
$\displaystyle K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\mid
g(y_{1},y_{2})-g(x_{0},y_{0})\mid+\mid
g(x_{0},y_{0})-g(y_{1},y_{2})\mid\right)$ $\displaystyle\leq$ $\displaystyle
MK_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\mid
t-x_{0}\mid^{\tau}\mid s-y_{0}\mid^{\tau};y_{1},y_{2}\right)$ $\displaystyle+$
$\displaystyle M\mid y_{1}-x_{0}\mid^{\tau}\mid y_{2}-y_{0}\mid^{\tau}.$

For all $A,B\geq 0$ and $\tau\in[0,1+p],q>0$ we know inequality
$(A+B)^{\tau}\leq A^{\tau}+B^{\tau},$ thus

$\mid t-x_{0}\mid^{\tau}\leq\mid t-y_{1}\mid^{\tau}+\mid
y_{1}-x_{0}\mid^{\tau},$

$\mid s-y_{0}\mid^{\tau}\leq\mid s-y_{2}\mid^{\tau}+\mid
y_{2}-y_{0}\mid^{\tau}.$

Therefore

$\displaystyle\mid
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(g;y_{1},y_{2})-g(y_{1},y_{2})\mid$
$\displaystyle\leq$ $\displaystyle
MK_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\mid
t-y_{1}\mid^{\tau}\mid s-y_{2}\mid^{\tau};y_{1},y_{2}\right)$ $\displaystyle+$
$\displaystyle M\mid
y_{1}-x_{0}\mid^{\tau}K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\mid
s-y_{2}\mid^{\tau};y_{1},y_{2}\right)$ $\displaystyle+$ $\displaystyle M\mid
y_{2}-y_{0}\mid^{\tau}K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\mid
t-y_{1}\mid^{\tau};y_{1},y_{2}\right)$ $\displaystyle+$ $\displaystyle 2M\mid
y_{1}-x_{0}\mid^{\tau}\mid
y_{2}-y_{0}\mid^{\tau}K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\mu_{0,0};y_{1},y_{2}\right).$

On apply Hölder inequality on
$K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}$, we get

$\displaystyle K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}\left(\mid
t-y_{1}\mid^{\tau}\mid s-y_{2}\mid^{\tau};y_{1},y_{2}\right)$ $\displaystyle=$
$\displaystyle\mathcal{U}_{n_{1},k}^{\lambda_{1}}\left(\mid
t-y_{1}\mid^{\tau};y_{1},y_{2}\right)$ $\displaystyle\times$
$\displaystyle\mathcal{V}_{n_{2},l}^{\lambda_{2}}\left(\mid
s-y_{2}\mid^{\tau};y_{1},y_{2}\right)$ $\displaystyle\leq$
$\displaystyle\left(K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\mid
t-y_{1}\mid^{2};y_{1},y_{2})\right)^{\frac{\tau}{2}}$ $\displaystyle\times$
$\displaystyle\left(K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\mu_{0,0};y_{1},y_{2})\right)^{\frac{2-\tau}{2}}$
$\displaystyle\times$
$\displaystyle\left(K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\mid
s-y_{2}\mid^{2};y_{1},y_{2})\right)^{\frac{\tau}{2}}$ $\displaystyle\times$
$\displaystyle\left(K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(\mu_{0,0};y_{1},y_{2})\right)^{\frac{2-\tau}{2}}.$

Thus, we can obtain

$\displaystyle\mid
K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(g;y_{1},y_{2})-g(y_{1},y_{2})\mid$
$\displaystyle\leq$ $\displaystyle
M\left(\delta_{n_{1},y_{1}}^{2}\right)^{\frac{\tau}{2}}\left(\delta_{n_{2},y_{2}}^{2}\right)^{\frac{\tau}{2}}$
$\displaystyle+$ $\displaystyle
2M\left(d(y_{1},E)\right)^{\tau}\left(d(y_{2},E)\right)^{\tau}$
$\displaystyle+$ $\displaystyle
M\left(d(y_{1},E)\right)^{\tau}\left(\delta_{n_{2},y_{2}}^{2}\right)^{\frac{\tau}{2}}+L\left(d(y_{2},E)\right)^{\tau}\left(\delta_{n_{1},y_{1}}^{2}\right)^{\frac{\tau}{2}}.$

We have complete the proof. ∎

######  Example.

It is observed that in this example that for the following set of parameters
$q=5$, $\rho=0.9$ and $\lambda=0.5$, the operator
$K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(,;,)$ converges uniformly to
the function $f(y)=y_{1}^{3}y_{2}^{2}$ (Blue) as $m_{1}=m_{2}=10$ (Green) and
$m_{1}=m_{2}=20$ (Red) increases which is shown in the Figure 3.

Figure 3. $K_{m_{1},m_{2}}^{\rho,\lambda_{1},\lambda_{2}}(,;,)$ converges to
$f(x)=y_{1}^{3}y_{2}^{2}$

Acknowledgements
The third and the corresponding authors would like to acknowledge that this
research was partially supported by Ministry of Higher Education under the
Fundamental Research Grants Scheme (FRGS) with project number
FRGS/2/2014/SG04/UPM/01/1 and having vot number 5524674. Also, the
corresponding author is supported by the Jointly Awarded Research Degree
(JADD) program (by UPM-UON)

Data Availability
Data sharing is not applicable to this article as no data sets were generated
or analyzed during the current study.

Conflicts of Interest
The authors declare that they have no conflicts of interest.

Funding
Not applicable.

Authors Contributions
All authors have equal contributions.

## References

  * [1] S.N. Berntein, Démonstration du théorème de Weierstrass fondée sur le calcul de probabilités, Commun. Soc. Math. Kharkow. 13(2) (1912) 1–2.
  * [2] F. Schurer Linear Positive Operators in Approximation Theory, Math. Inst., Techn. Univ. Delf Report (1962).
  * [3] M. Mursaleen, F. Khan, A. Khan, A. Kiliçman, Some approximation results for generalized Kantorovich-type operators, J. Inequal. Appl. 2013 (2013) 585.
  * [4] M. Mursaleen, K.J. Ansari, A. Khan, Some approximation results by $(p,q)$-analogue of Bernstein–Stancu operators, Appl. Math. Comput. 264 (2015) 392–402.
  * [5] T. Acar, A. Aral, S.A. Mohiuddine, Approximation by bivariate $(p,q)$-Bernstein-Kantorovich operators, Iran. J. Sci. Technol. Trans. Sci. 42 (2018) 655–662.
  * [6] T. Acar, A. Kajla, Degree of Approximation for bivariate generalized Bernstein type operators, Results Math. 73 (2018) 79.
  * [7] S.A. Mohiuddine, T. Acar, A. Alotaibi Construction of a new family of Bernsteion Kantorovich operators, Math. Meth. Appl. Sci. 40 (2017) 7749-7759.
  * [8] A.M. Acu, T. Acar, V.A. Radu, Approximation by modified $U_{n}^{\tau}$ operators, Rev. R. Acad. Ciene. Exactas Fis. Nat. Ser. A Math. RACSAM. 113 (2019) 2715-2729.
  * [9] G. İçöz, B. Çekim, Dunkl generalization of Szász operators via $q$-calculus, J. Inequal. Appl. 2015 (2015) 284.
  * [10] G. İçöz, B. Çekim, Stancu-type generalization of Dunkl analogue of Szász-Kantorovich operators, Math. Met. Appl. Sci. 39(7) (2016) 1803–1810.
  * [11] A. Kajla, D. Micláus, Blending type approximation by GBS operators of generalized Bernstein-Durrmeyer type, Results Math. 73 (2018) 1.
  * [12] A. Kalja, T. Acar, Blending type approximation by generalized Bernstein Durrmeyer type operators, Misco. Math. Notes. 19(1) (2018) 319-336.
  * [13] X. Chen, J. Tan, Z. Liu, J. Xie, Approximation of functions by a new family of generalized Bernstein operators, J. Math. Anal. Appl. 450 (2017) 244-261.
  * [14] A. Aral, H. Erbay Parametric generalization of Baskakov operators, Math. Commun. 24 (2019) 119-131.
  * [15] F. Özger, H.M. Srivastava, S.A. Mohiuddine, Approximation of functions by a new class of generalized Bernstein–Schurer operators, Rev. Real Acad. Cienc. Exactas Fis. Nat. Ser. A-Mat. RACSAM. 114 (2020) 173.
  * [16] O. Shisha, B. Bond, B. The degree of convergence of linear positive operators, Proc. Nat. Acad. Sci. USA. 60 (1968) 1196-1200.
  * [17] A.D. Gadziev, Theorems of the type of P.P. Korovkin’s theorems, Mat. Zametki. 20(5) (1976) 781-786 (in Russian), Math. Notes 20(5-6) (1976) 995-998 (Engl. Trans.).
  * [18] A.D. Gadjiev, C. Orhan, Some approximation theorems via statistical convergence, Rocky Mountain J. Math. 32(1) (2007) 129-138.
  * [19] M.A. Özarslan, H. Aktuğlu, Local approximation for certain King type operators, FILOMAT. 27 (2013) 173-181.


# Classification of Distraction Levels Using Hybrid Deep Neural Networks From
EEG Signals 111††thanks: This research was supported by the Challengeable
Future Defense Technology Research and Development Program (912911601) of
Agency for Defense Development in 2020.

Dae-Hyeok Lee
Dept. Brain and Cognitive Engineering
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Sung-Jin Kim
Dept. Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Yeon-Woo Choi  Dept. Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

Non–invasive brain–computer interface technology has been developed for
detecting human mental states with high performances. Detection of the pilots’
mental states is particularly critical because their abnormal mental states
could cause catastrophic accidents. In this study, we presented the
feasibility of classifying distraction levels (namely, normal state, low
distraction, and high distraction) by applying the deep learning method. To
the best of our knowledge, this study is the first attempt to classify
distraction levels under a flight environment. We proposed a model for
classifying distraction levels. A total of ten pilots conducted the experiment
in a simulated flight environment. The grand–average accuracy was 0.8437
($\pm$0.0287) for classifying distraction levels across all subjects. Hence,
we believe that it will contribute significantly to autonomous driving or
flight based on artificial intelligence technology in the future.

Keywords–brain–computer interface, electroencephalogram, abnormal mental
states, flight environment;

##  I INTRODUCTION

Brain–computer interface (BCI) allows users to communicate between human and
external devices by recognizing human’s status and intention [1, 2, 3, 4, 5,
6, 7, 8]. In particular, the non–invasive BCI system has advantages in that it
does not require a surgical operation and costs less than other BCI systems.
Since electroencephalogram (EEG) can reflect the user’s intention and current
state, the BCI system using EEG has been developed [9, 10, 11, 12, 13]. In
addition, advanced BCI has been applied in various domains, including
diagnosis of Alzheimer’s disease [14] and controlling external devices such as
a robotic arm [15] and a speller [16].

One of the interesting issues in the BCI domain is the detection of human
abnormal mental states with robust performances [17, 18, 19, 20, 21, 22, 23].
In addition, the technology of autonomous driving or flight is required when
the driver’s or pilot’s mental state changes from a normal state to an
abnormal state. Since the driver’s or pilot’s mental state is directly related
to the safety of passengers, the technology to accurately detect their mental
state has been developed. Controlling an aircraft is a challenging task, and
it consumes a number of energy [24]. Among various abnormal mental states,
distraction occurs when pilots cannot concentrate on flight control for many
different variables. In addition, the CAA of New Zealand reported that pilots’
distraction was the major reason among the various reasons for flight
accidents [25].

EEG signals are important signals in the case of detecting human’s mental
states because they reflect the human’s mental states directly [26]. A few
research groups have studied detecting distraction using only EEG signals.
Sonnleitner et al. [23] described the impact of an auditory secondary task on
drivers’ mental states during a primary driving task. Provoked reaction time
to brake lights and EEG alpha spindles were analyzed to describe distracted
drivers. Brake reaction times and alpha spindle rate were significantly higher
in distraction. Wang et al. [27] assessed the differences in behavioral
performance and EEG activity when participants performed a lane–keeping
driving task and a mathematical problem–solving task. Their system achieved
84.6 $\pm$ 5.8 % and 86.2 $\pm$ 5.4 % classification accuracies in detecting
the participants’ attention on the math and driving tasks, respectively. Li et
al. [28] developed a novel deep learning–based distraction detection approach
based on both temporal and spatial information. The results showed that their
proposed approach achieved an overall binary (distraction vs. non–distraction)
classification accuracy of 0.92.

The main contributions are as follows. i) We acquired distraction–related EEG
signals under a flight environment with pilots. We measured EEG signals
corresponding to a normal state (NS), low distraction (LD), and high
distraction (HD) according to the difference in tasks. ii) We propose hybrid
deep neural networks for classifying distraction levels. We obtained the
highest classification performance using our proposed model compared to the
conventional models. To the best of our knowledge, this is the first attempt
to classify distraction levels based on the deep learning architecture
robustly.

The rest of this paper is organized as follows. In Section II, we introduce
our experimental design and the explanation of the proposed model. In Section
III, we present the experimental results. Finally, we conclude this paper in
Section IV.

##  II MATERIALS AND METHODS

###  II-A Subjects

Ten pilots (S1–S10, aged 25.6 ($\pm$0.52)) participated in our experiment. All
of them had flight experience (over 100 hr.) in the Taean Flight Education
Center. They had no history of psychiatric or neurological disorders. We
informed the entire experimental protocols to pilots. Our experimental
protocols and environment were approved by the Institutional Review Board of
Korea University [1040548–KU–IRB–18–92–A–2]. Before the experiment, they
consented according to the Declaration of Helsinki. For evaluating the
experimental paradigm, we instructed the pilots to fill out a questionnaire to
check their mental and physical conditions after finishing the experiment.

###  II-B Experimental Environment

The Cessna 172 (Garmin, Olathe, KS) was used in the flight simulator which
included the screen, the cockpit, a keypad, and the signal amplifier. A flight
yoke and other control panels were the components of the cockpit for
constructing a realistic flight environment. The subjects inputted the number
using a keypad. We attached a keypad to the flight yoke directly for
preventing the unnecessary movement when the subjects press the keypad. The
signal amplifier (BrainAmp, Brain Products GmBH, Germany) was used to measure
EEG and EOG signals. We set the sampling frequency of EEG and EOG signals as
1,000 Hz, and we used a 60 Hz notch filter for removing the power supply
noise. We placed the 30 EEG channels according to the international 10/20
system. Also, we attached the four EOG channels to the horizontal and vertical
lines around the eye. FCz and AFz channels were used to the reference and
ground electrodes, respectively. We set up the impedance of electrodes below
10 k$\Omega$ by injecting the conductive gel.

###  II-C Experimental Protocol and Paradigm

We designed the experimental paradigm for acquiring the distraction-based EEG
signals effectively, as shown in Fig. 1. The tasks given to the subjects were
not difficult for the pilot, as it is hard to obtain high–quality
distraction–based EEG signals if the tasks are difficult. The air traffic
control (ATC) message (length: 4–22 words) was presented to the subjects. The
subjects simultaneously counted the number of words contained in the ATC
message while performing the flight. We instructed the subject to count the
number of words without any body movement. We divided the distraction level
according to the length of the ATC message (level 1: 4–9 words, level 2: 10–14
words, level 3: 15–22 words). We defined level 1 as LD and level 2 and level 3
as HD.

Figure 1: Experimental paradigms for inducing distraction in a simulated
flight environment.

###  II-D Signal Preprocessing

We conducted the preprocessing of EEG signals using a BBCI toolbox in MATLAB
2019a [29]. We utilized the band—pass filter using a $2^{nd}$ order zero–phase
Butterworth filter between 1 and 50 Hz and down—sampled the signals from 1,000
to 100 Hz. We applied the independent component analysis [30] for removing the
contaminated components. Each trial was segmented into 1 sec. data without
overlap [26]. 1,860 samples ([Level 1] 10 samples$\times$40 trials + [Level 2]
10 samples$\times$40 trials + [Level 3] 10 samples$\times$40 trials + [Rest
after entering the numbers of words] 4 samples$\times$120 trials + [Rest
between $1^{st}$ and $2^{nd}$ sessions] 180 samples) were obtained for each
subject. Across all subjects, 18,600 samples were obtained. Since there is a
difference in the number of samples in the NS, LD, and HD, We solved the
problem of data unbalance according to the number of samples in the LD with
the smallest number of samples.

###  II-E Proposed Model

We proposed the model to classify distraction levels using EEG signals
accurately. EEG signals have various features such as spectral, spatial, and
temporal information. Hence, we designed the model with the principle of a
hybrid deep learning framework. The components of our proposed model were five
convolutional blocks and one LSTM block.

The five convolutional blocks were utilized for extracting the significant
spatial and spectral features from EEG signals. The $1^{st}$, $2^{nd}$, and
$3^{rd}$ convolutional blocks have two layers, 1 $\times$ 5 filters, 1
$\times$ 1 stride, and a batch normalization layer. $4^{th}$ and $5^{th}$
convolutional blocks have three layers, 5 $\times$ 1 and 3 $\times$ 1 filters,
respectively, 1 $\times$ 1 stride, and a batch normalization layer. In
addition, we applied the maximum– and average–pooling layers for avoiding the
overfitting problem. Also, we used the exponential linear unit as an
activation function.

LSTM network, which is one of the recurrent neural networks, is an effective
network for recognizing mental states. One LSTM block, which has two LSTM
layers with 256 and 128 hidden units, respectively, was utilized for
extracting the significant temporal features from EEG signals.

The last part of our proposed model is the classification block, which has
three fully connected layers and a softmax layer. The hidden units of $1^{st}$
and $2^{nd}$ fully connected layers were 128 and 64, respectively. Also, the
output of the $3^{rd}$ fully connected layer was fed to 3–way softmax.

TABLE I: Comparison of the classification performances with the statistical
analysis for classifying distraction levels among the conventional models and
the proposed model.

Subject |  PSD–SVM [31] |  DeepConvNet [32] | Proposed
---|---|---|---
S1 | 0.6831 | 0.7244 | 0.8128
S2 | 0.6806 | 0.7276 | 0.8489
S3 | 0.6184 | 0.7641 | 0.8403
S4 | 0.6999 | 0.7575 | 0.8374
S5 | 0.7305 | 0.7983 | 0.8401
S6 | 0.7177 | 0.8012 | 0.9088
S7 | 0.5804 | 0.7001 | 0.8235
S8 | 0.7198 | 0.7798 | 0.8554
S9 | 0.7087 | 0.7596 | 0.8612
S10 | 0.6582 | 0.7341 | 0.8082
Avg. | 0.6797 | 0.7547 | 0.8437
Std. | 0.0483 | 0.0331 | 0.0287
p–value | <0.05 | <0.05 | -

##  III RESULTS AND DISCUSSION

###  III-A Performance Evaluation

We applied the five–fold cross–validation method for evaluating performance
fairly. The dataset was randomly shuffled and divided into five parts. The
four parts were used as the training set and the one part was used as the test
set. Table I presented the comparison of the performances for classifying
distraction levels among the conventional models and the proposed model. The
conventional models used for performance comparison were the power spectral
density–SVM (PSD–SVM) [31] and DeepConvNet [32]. The PSD–SVM is a model that
uses PSD of the $\delta$– (1–4 Hz), $\theta$– (4–8 Hz), $\alpha$– (8–13 Hz),
and $\beta$–bands (13–30 Hz) as a feature and SVM as a classifier. The
DeepConvNet is one of the representative models for decoding EEG signals. It
consists of a total of four convolutional blocks. The first convolutional
block consists of consecutive temporal and spatial convolutions, and the
remaining convolutional blocks consist of temporal convolutions. Our model
showed the highest average accuracy of 0.8437 ($\pm$0.0287) compared to the
conventional models. In the case of our proposed model, S6 and S10 represented
the highest and the lowest performance, respectively, and the values were
0.9088 and 0.8082, respectively. In addition, our proposed model indicated the
lowest–standard deviation of 0.0287 compared to the conventional models. It
means the highest stability across all subjects.

Figure 2: Scalp topographies according to the spectral bands ($\delta$–,
$\theta$–, $\alpha$–, and $\beta$–bands) across the representative subject
(S6). The locations of channels with the statistical significance are
indicated as grey ‘$\ast$’ (*: p$<$0.05).

###  III-B Neurophysiological Analysis from EEG Signals

We divided the brain region into temporal, central, and parietal regions and
the frequency into $\delta$–, $\theta$–, $\alpha$–, and $\beta$–bands, as
shown in Fig. 2. We represented the scalp topographies using EEG signals of
the representative subject (S6). We used the grand–average band power to plot
the scalp topographies. All EEG channels and each frequency band were used for
calculating the amplitude. The amplitude was significantly different for each
brain region and each spectral band. When distraction level increased, the
amplitude of the $\theta$–band in the centro–parietal region increased, and
that of the $\alpha$–band in the central region increased. The grey $\ast$
indicates the locations of channels with the statistical significance
(p$<$0.05). However, we could not find any particular spatial tendencies in
the $\delta$– and $\beta$–bands.

##  IV CONCLUSION

The pilot’s abnormal mental states are the most important among the various
reasons for flight accidents. Since it is directly related to the safety of
passengers, detection of the pilots’ mental state with robust performance is
necessary. Detection of various mental states using EEG signals is one of the
important challenging issues in the BCI domain. Especially, accurate
classification of levels in–depth for mental states helps to prevent accidents
caused by human error. In this paper, we proposed a model for classifying
distraction levels using EEG signals. Our proposed model could classify
distraction levels with robust performance compared to other conventional
models. Since we acquired distraction–based EEG signals from highly
disciplined pilots, we believe that our study will help advance autonomous
driving or flight. To apply our study to the real–world environment, we will
develop our proposed model after acquiring more various abnormal mental
states-based EEG signals using our modified experimental paradigms.

## References

  * [1] H.-I. Suk, S. Fazli, J. Mehnert, K.-R. Müller, and S.-W. Lee, “Predicting BCI subject performance using probabilistic spatio–temporal filters,” _PLoS One_ , vol. 9, no. 2, p. e87056, 2014.
  * [2] S. Issa, Q. Peng, and X. You, “Emotion classification using EEG brain signals and the broad learning system,” _IEEE Trans. Syst. Man Cybern. Syst._ , vol. 51, no. 12, pp. 7382–7391, 2020.
  * [3] D.-O. Won, H.-J. Hwang, D.-M. Kim, K.-R. Müller, and S.-W. Lee, “Motion–based rapid serial visual presentation for gaze–independent brain–computer interfaces,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 26, no. 2, pp. 334–343, 2017.
  * [4] P. Chen _et al._ , “Multiattention adaptation network for motor imagery recognition,” _IEEE Trans. Syst. Man Cybern. Syst._ , 2021.
  * [5] N.-S. Kwak and S.-W. Lee, “Error correction regression framework for enhancing the decoding accuracies of ear–EEG brain–computer interfaces,” _IEEE Trans. Cybern._ , vol. 50, no. 8, pp. 3654–3667, 2019.
  * [6] X. Liu, K. Kasmarik, and H. Abbass, “Assessing player profiles of achievement, affiliation, and power motivation using electroencephalography,” _IEEE Trans. Syst. Man Cybern. Syst._ , vol. 52, no. 6, pp. 3648–3658, 2021.
  * [7] O.-Y. Kwon, M.-H. Lee, C. Guan, and S.-W. Lee, “Subject–independent brain–computer interfaces based on deep convolutional neural networks,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 31, no. 10, pp. 3839–3852, 2019.
  * [8] Y. Zhang, H. Zhang, X. Chen, S.-W. Lee, and D. Shen, “Hybrid high–order functional connectivity networks using resting–state functional MRI for mild cognitive impairment diagnosis,” _Sci. Rep._ , vol. 7, no. 1, pp. 1–15, 2017.
  * [9] B. Abibullaev and A. Zollanvari, “A systematic deep learning model selection for P300–based brain–computer interfaces,” _IEEE Trans. Syst. Man Cybern. Syst._ , vol. 52, no. 5, pp. 2744–2756, 2021.
  * [10] S.-J. Kim, D.-H. Lee, and S.-W. Lee, “Rethinking CNN Architecture for Enhancing Decoding Performance of Motor Imagery-Based EEG Signals,” _IEEE Access_ , vol. 10, pp. 96 984–96 996, 2022.
  * [11] M. Lee _et al._ , “Connectivity differences between consciousness and unconsciousness in non–rapid eye movement sleep: A TMS–EEG study,” _Sci. Rep._ , vol. 9, no. 1, pp. 1–9, 2019.
  * [12] A. Gupta _et al._ , “On the utility of power spectral techniques with feature selection techniques for effective mental task classification in noninvasive BCI,” _IEEE Trans. Syst. Man Cybern. Syst._ , vol. 51, no. 5, pp. 3080–3092, 2019.
  * [13] J.-H. Jeong, N.-S. Kwak, C. Guan, and S.-W. Lee, “Decoding movement–related cortical potentials based on subject–dependent and section–wise spectral filtering,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 28, no. 3, pp. 687–698, 2020.
  * [14] K.-H. Thung _et al._ , “Conversion and time–to–conversion predictions of mild cognitive impairment using low–rank affinity pursuit denoising and matrix completion,” _Med. Image Anal._ , vol. 45, pp. 68–82, 2018.
  * [15] J.-H. Jeong, K.-H. Shim, D.-J. Kim, and S.-W. Lee, “Brain–controlled robotic arm system based on multi–directional CNN–BiLSTM network using EEG signals,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 28, no. 5, pp. 1226–1238, 2020.
  * [16] M.-H. Lee, J. Williamson, D.-O. Won, S. Fazli, and S.-W. Lee, “A high performance spelling system based on EEG–EOG signals with visual feedback,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 26, no. 7, pp. 1443–1459, 2018.
  * [17] D.-H. Lee, J.-H. Jeong, K. Kim, B.-W. Yu, and S.-W. Lee, “Continuous EEG decoding of pilots’ mental states using multiple feature block–based convolutional neural network,” _IEEE Access_ , vol. 8, pp. 121 929–121 941, 2020.
  * [18] J. A. Hobson, “Sleep is of the brain, by the brain and for the brain,” _Nature_ , vol. 437, no. 7063, pp. 1254–1256, 2005.
  * [19] E. Q. Wu, X. Peng, C. Z. Zhang, J. Lin, and R. S. Sheng, “Pilots’ fatigue status recognition using deep contractive autoencoder network,” _IEEE Trans. Instrum. Meas._ , vol. 68, no. 10, pp. 3907–3919, 2019.
  * [20] L. G. Lim _et al._ , “A unified analytical framework with multiple fNIRS features for mental workload assessment in the prefrontal cortex,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 28, no. 11, pp. 2367–2376, 2020.
  * [21] G. Borghini, L. Astolfi, G. Vecchiato, D. Mattia, and F. Babiloni, “Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness,” _Neurosci. Biobehav. Rev._ , vol. 44, pp. 58–75, 2014.
  * [22] T.-T. N. Do, C.-H. Chuang, S.-J. Hsiao, C.-T. Lin, and Y.-K. Wang, “Neural comodulation of independent brain processes related to multitasking,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 27, no. 6, pp. 1160–1169, 2019.
  * [23] A. Sonnleitner _et al._ , “EEG alpha spindles and prolonged brake reaction times during auditory distraction in an on–road driving study,” _Accid. Anal. Prev._ , vol. 62, pp. 110–118, 2014.
  * [24] J.-R. Yen, C.-C. Hsu, H. Yang, and H. Ho, “An investigation of fatigue issues on different flight operations,” _J. Air Transp. Manag._ , vol. 15, no. 5, pp. 236–240, 2009.
  * [25] M. P. Çakır, M. Vural, S. Ö. Koç, and A. Toktaş, “Real–time monitoring of cognitive workload of airline pilots in a flight simulator with fNIR optical brain imaging technology,” in _Int. Conf. Hum.-Comput. Interact. (HCII)_ , 2016, pp. 147–158.
  * [26] J.-H. Jeong, B.-W. Yu, D.-H. Lee, and S.-W. Lee, “Classification of drowsiness levels based on a deep spatio–temporal convolutional bidirectional LSTM network using electroencephalography signals,” _Brain Sci._ , vol. 9, no. 12, p. 348, 2019.
  * [27] Y.-K. Wang, T.-P. Jung, and C.-T. Lin, “EEG–based attention tracking during distracted driving,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 23, no. 6, pp. 1085–1094, 2015.
  * [28] G. Li _et al._ , “A temporal–spatial deep learning approach for driver distraction detection based on EEG signals,” _IEEE Trans. Autom. Sci. Eng._ , 2021.
  * [29] B. Blankertz _et al._ , “The Berlin brain–computer interface: Non–medical uses of BCI technology,” _Front. Neurosci._ , vol. 4, p. 198, 2010.
  * [30] T.-P. Jung _et al._ , “Extended ICA removes artifacts from electroencephalographic recordings,” in _Adv. Neural Inf. Process. Syst. (NeurIPS)_ , 1998, pp. 894–900.
  * [31] X. Zhang _et al._ , “Design of a fatigue detection system for high–speed trains based on driver vigilance using a wireless wearable EEG,” _Sensors_ , vol. 17, no. 3, p. 486, 2017.
  * [32] R. T. Schirrmeister _et al._ , “Deep learning with convolutional neural networks for EEG decoding and visualization,” _Hum. Brain Mapp._ , vol. 38, no. 11, pp. 5391–5420, 2017.