# A Reinforcement Learning Environment for Mathematical Reasoning via Program
Synthesis

Joseph Palermo & Johnny Ye
Cash App Labs
{jpalermo<EMAIL_ADDRESS>
Alok Singh
Lawrence Berkeley National Laboratory
<EMAIL_ADDRESS>

###### Abstract

We convert the DeepMind Mathematics Dataset into a reinforcement learning
environment by interpreting it as a program synthesis problem. Each action
taken in the environment adds an operator or an input into a discrete compute
graph. Graphs which compute correct answers yield positive reward, enabling
the optimization of a policy to construct compute graphs conditioned on
problem statements. Baseline models are trained using Double DQN on various
subsets of problem types, demonstrating the capability to learn to correctly
construct graphs despite the challenges of combinatorial explosion and noisy
rewards.

##  1 Introduction

The DeepMind Mathematics Dataset [1] consists of synthetically generated math
problems. They cover a range of problem types including: Numbers, comparison,
measurement, arithmetic, algebra, polynomials, calculus, and probability.
These problem types are arranged into a collection of 56 modules each
containing different sub-types of problems. The dataset provides the problems
in the form of question-answer pairs represented as ASCII text.

The predominant algorithmic approach for learning to produce the answers
conditioned on the question statements has been to train seq2seq models
[1][2][3]. For some of the modules this approach yields very nearly 100%
accuracy, however for other modules this does extremely poorly. For example,
with this approach the best reported test accuracy for the module
”numbers__list_prime_factors” is less than 25% [3].

This poor performance is not surprising considering the nature of the problems
on which this occurs. For instance, the module ”numbers__is_prime” requires
discriminating primes from non-primes. Apart from memorizing answers, there
are only a few useful hacks that a learned model can easily pick up, for
example, the fact that any even number (except for 2) is not prime. Algorithms
to correctly test for primality require a sequence of divisions to be
performed very precisely, and before a perfectly correct algorithm is obtained
loss may not be much improved beyond random guessing. Thus, it’s unclear if
the gradient of loss obtained by comparing computed answers to correct answers
could be sufficient to learn such algorithms.

A further difficulty with such systems is that they are unlikely to be
sufficiently accurate or interpretable for real-world use. Models trained in
this manner are effectively black boxes which only return answers. Probability
estimates assigned to tokens sampled from the model can potentially provide
clues about the likelihood of correctness (assuming they are well calibrated).
However, the reasoning process by which the result was arrived at cannot be
easily inspected.

We think it would be desirable to train neural networks to make use of
existing programs for computing various mathematical operations [4][5]. Just
as human programmers use libraries when writing larger programs, learned
algorithms should be able to build upon existing operators. With predefined
operators available, the neural network wouldn’t have to rediscover for itself
algorithms and functions which are already well known, and could instead focus
on learning how to compose them. Essentially, we propose to treat the problem
from the point of view of program synthesis.

Program synthesis is the problem of how to automatically construct programs to
meet predefined specifications or constraints [6]. If the specification is
sufficiently precise, then one may be able to prove that a given program meets
it. However, the DeepMind Mathematics Dataset provides only question-answer
pairs. So, if a program was generated to compute the answer for a given
question, in general it would only be possible to confirm that the program
generated the correct answer for that specific question.

More formally, the learning problem is to learn a policy which maps a problem
statement represented as text, to a program. The program takes the form of a
compute graph composed of discrete operators. A policy which constructs these
compute graphs by conditioning on problem statements can be learned with
reinforcement learning because the question-answer pairs can be used to
provide a reward signal to adjust the parameters of the policy.

##  2 Framing the Problem using Reinforcement Learning

To frame the problem in terms of reinforcement learning we need to define an
environment. To explain how this is done we first need to explain how inputs
are extracted from problem statements and to specify how the operators are
defined.

###  2.1 Inputs

In problems in the DeepMind Mathematics Dataset there are often parts of the
text which are explicitly mathematical. We call these the inputs of the
problems.

For example, in the following problem:

⬇

Let h(t) = t**3 + t**2 + 1. Let v(d) = 6*d**3 + 24*d**2 + 4. Let w(j) = 4*h(j)
- v(j). What is the third derivative of w(x) wrt x?

The inputs are:

⬇

1) h(t) = t**3 + t**2 + 1

2) v(d) = 6*d**3 + 24*d**2 + 4

3) w(j) = 4*h(j) - v(j)

4) w(x)

5) x

It turns out that for many problem types, a parser can be written which
automatically extracts the inputs of the problem. By embedding such a parser
within the environment implementation, given any problem statement the
environment can determine which inputs are available. This will be used to
help define the action space in section 2.3.

###  2.2 Predefined Operators

To utilize program synthesis, a list of operators needs to be defined. Each of
them takes some fixed number of inputs and returns a single output. For
example, one operator called ’differentiate_wrt’ (short for “differentiate
with respect to” takes as input an expression and a variable, and it outputs
the derivative of the expression with respect to the variable.

A full list of operators is defined in Appendix A.

###  2.3 Defining the Reinforcement Learning Environment

Observations from the environment consist of a representation of the question
concatenated with a sequence of indices corresponding to actions taken
previously in the current episode.

The representation of the question can either be provided in encoded form
(i.e. as an array of indices into a vocabulary of tokens) or unencoded form
(i.e. raw text) depending on how the environment is configured. If the
environment is configured to return questions in encoded form, then it uses a
predefined byte pair encoding constructed from a corpus of questions. If the
representation is provided in encoded form then it is also padded up to a
fixed maximum length. Note that the raw text of the question is available at
every step regardless of how configurations are set (through the info object
returned by the ”step” method of the environment).

The action space is discrete and corresponds to the set of operators and
available inputs which can be introduced into the graph at any given time. The
graph is built up in breadth first order. Fixing the order of graph
construction simplifies the action space because otherwise the actions would
need to both specify which operator is being applied and where in the graph it
is being introduced.

The maximum number of available inputs is set as a parameter of the
environment. Thus if the max number of available inputs is set to n_inputs,
and there are n_ops operators, then the action space is represented by the set
of integers from 0 to n_ops + n_inputs - 1. The inputs are represented within
the action space in the order in which they appear in the problem. For
example, given the following problem:

⬇

Let h(t) = t**3 + t**2 + 1. Let v(d) = 6*d**3 + 24*d**2 + 4. Let w(j) = 4*h(j)
- v(j). What is the third derivative of w(x) wrt x?

The action space would be as follows:

⬇

0 : 1st operator

1 : 2nd operator

...

n_ops - 1 : nth operator

n_ops + 0 : h(t) = t**3 + t**2 + 1

n_ops + 1 : v(d) = 6*d**3 + 24*d**2 + 4

n_ops + 2 : w(j) = 4*h(j) - v(j)

n_ops + 3 : w(x)

n_ops + 4 : x

n_ops + 5 : None

...

n_ops + n_inputs - 1 : None

Note that if less than the maximum number of possible inputs is present in a
given problem, then the actions after the last available input are simply
defined as None and are masked (see section 3.1.1).

The reward has a value of 1 if the compute graph computes the right answer and
a value of 0 otherwise. Note that if the compute graph is incomplete then it
computes the value None which always yields a reward of 0.

Here is an example trajectory from the environment in which the environment
state is provided in unencoded format for legibility. Note that 5 is the
action index corresponding to the ”differentiate” operator and 14 is the
action index corresponding to the 1st input (which in this case corresponds to
6*k**2 - 101*k + 2548):

state t=0 : | What is the first derivative of 6*k**2 - 101*k + 2548?;
---|---
action t=0 : | 5
state t=1 : | What is the first derivative of 6*k**2 - 101*k + 2548?; 5
reward t=1 : | 0
action t=1 : | 14
state t=2 : | What is the first derivative of 6*k**2 - 101*k + 2548?; 5, 14
reward t=2 : | 1

Note that the environment will automatically terminate episodes with reward 0
if the configured maximum length is reached.

###  2.4 Modules

For the purposes of this research, we restricted consideration to a subset of
modules from the DeepMind Mathematics Dataset that have composed counterparts.
The reason for this is that the composed problems require much larger graphs
to reliably compute correct answers. This means that in a reinforcement
learning context, these problems require longer action sequences to deliver
optimal reward, and so constitute a more challenging reinforcement learning
environment.

Furthermore, it is the case that some of the modules without uncomposed
counterparts contain problems for which some of the relevant inputs are
represented simply as words and so are difficult to parse as inputs. For
example the module ’measurement__conversion’ contains questions such as: ”What
is seven halves of a day in minutes?”

A full list of supported modules is defined in Appendix A.

##  3 Challenges

###  3.1 Combinatorial Explosion

The experiments we present here use 15 operators and require up to 3 inputs,
which gives an action space of size 18. We also set a limit on graph size of
at most 7 nodes. Even with this restriction, there are $18^{7}\approx
6.12*10^{8}$ possible graphs. Given such a large state-space, naive search
would be unlikely to stumble upon correct graphs. Here we present strategies
for mitigating the impact of combinatorial explosion.

####  3.1.1 Masking Invalid Actions

To reduce the effective size of the search space we mask actions which are
guaranteed to produce invalid graphs. In particular we introduce a type
hierarchy and assign types to the parameter(s) and output of each operator.
Also, each input is automatically assigned a type by the environment. Taken
together, typed operators and typed inputs allow the application of type
constraints limit which actions are valid at any given time. To implement this
we define a method of the environment (called compute_mask) which generates a
boolean masking vector over the action space. This method can be called by an
agent before selecting an action to determine which actions are invalid,
although formally the environment does not force this constraint. So invalid
actions can still be taken, but they will result in graph that fails to
compute a meaningful result (i.e. output of the graph will be ”None”).

Recall that actions add nodes to the graph in breadth-first order. Thus, each
action after the first will correspond to a specific parameter of an operator
that has already been applied. The type of the parameter being filled-in by
the action determines the type constraint applied to that action. The form of
the constraint is that if the parameter requires type X, then only types at or
below X in the type hierarchy are permitted.

The type hierarchy consists of the following custom types: Equation (e.g. 2*x
+ y = 3), Expression (e.g. 2*x + 1), Function (e.g. f(x) = 2*x + 1), Value
(e.g. 2), Variable (e.g. x), Rational (e.g. 1/2). The hierarchy for the custom
types is provided in Figure 1. We also utilize the following types which are
built into Python: object, list, dict, and bool. Note that list, dict, bool
and all the custom types are subclasses of object, and thus fall below it in
the type hierarchy [7].

Figure 1: Type Hierarchy

In addition to type constraints, there are two other additional constraints
implemented by masking. The first action always corresponds to the root node
of the graph and has the additional constraint that it must correspond to an
operator (i.e. it cannot be an input). There is also the additional constraint
that any action defined as None is always masked. This occurs for problems
which have less than the maximum number of inputs in which case the action
space is padded up to a fixed size with actions defined as None.

####  3.1.2 Abstracting Subgraphs into Operators

Another strategy for addressing combinatorial explosion which can be applied
but was not implemented in our experiments is the process of abstracting
frequently rewarded subgraphs into new operators. The concept is to identify
subgraphs which frequently occur in graphs that yield reward, then to identify
the inputs and output of that subgraph such that it could be redefined as a
new operator. This is known more broadly in the computer science literature as
Frequent Subgraph Mining [8].

For example:

⬇

differentiate_wrt(differentiate_wrt(Expression(’-3*z**5 + 13*z**3 +
41*z**2’),Variable(’z’)),Variable(’z’)) reward = 1

This subgraph can be abstracted into a new operator as follows:

⬇

diff_wrt_2(p0, p1) = differentiate_wrt(differentiate_wrt(p0, p1), p1)

New operators defined in this way could be introduced into the action space
and would provide the possibility of finding shorter graphs to compute the
same functions. This is similar to the operator abstraction process in
Dreamcoder [9]. This would increase the size of the action space and hence the
number of possible graphs, however it would reduce the required length of
frequently rewarded graphs and hence would have a multiplicative effect in
reducing the effective size of the search space.

###  3.2 Noisy Reward

An additional challenge with this environment is that the rewards are noisy
with respect to graph correctness. In other words, a particular graph might
compute the right answer for a given question but do so in the wrong way. For
example:

⬇

Is 5340 a multiple of 10?; not(is_prime(Value(’10’))) = True, reward: 1

If the inputs of the problem were changed the same graph may no longer compute
the right answer because it is simply performing the wrong operations and so
cannot reliably generalize to different inputs. This effectively means that a
positive reward from the environment does not guarantee that the constructed
graph is correct.

##  4 Approach

We use Double DQN [10] to learn a value function that maps a state and an
action to the expected discounted sum of rewards. An off-policy learning
algorithm is used because it simplifies the maintenance of exploration which
is critical in the presence of noisy rewards. Epsilon-greedy exploration is
utilized with a step-wise linear annealing schedule on the value of epsilon.

We also apply prioritized experience replay [11] to improve the efficiency of
learning. We sample steps from replay memory with priority directly
proportional to the most recently computed TD-error. After every training
batch, we re-compute replay priorities for both the steps used to construct
the batch and an additional random sample of steps from the replay buffer. The
additional random sample is taken to encourage all replay priorities to remain
consistent with the current model parameters, even if they haven’t been used
to construct a training batch in some time.

We initialize the replay buffer with trajectories (containing up to 50k steps)
collected by a uniform random policy, where a one-to-one balance is kept
between trajectories with positive reward and trajectories with 0 reward. This
balancing is done to prevent a heavy skew towards trajectories with reward 0,
due to reward sparsity in the environment. After an initial period of training
on the replay buffer as initialized, new experience is continuously
incorporated into the replay buffer. However, a one-to-one balance between
trajectories with positive reward and 0 reward is maintained since otherwise
trajectories with 0 reward would overwhelm the replay buffer.

The model used in experiments is a transformer encoder [12] with 6 encoder
blocks followed by 2 dense layers. We use 4 attention heads and a hidden layer
size of 256. The dense blocks use a hidden layer size of 256. We apply dropout
[13] of 0.1 in the Transformer blocks and between the dense layers. We use a
learning rate of 5*10e-5 and a batch size of 512. The epsilon value for
epsilon-greedy exploration is initialized to 0.4 and is linearly annealed to
0.05 by an increment of 2.5*10e-5 per step. The full set of hyperparameters
used to produce the results reported here is provided in the code
(https://github.com/joepalermo/dm_math_solvers/blob/master/hparams-for-
paper.cfg).

##  5 Results

We conduct experiments on different subsets of modules to evaluate how it
affects model performance. For each experiment we run 5 trials with the same
hyperparameters and different random seeds (as recommended in Henderson et al.
2017 [14]). We sample 1.01 million examples from each module under
consideration with 800k/200k/10k across train/validation/test. The trials are
run for 50k steps each except for the trials in the ”Interference” experiment
(see below) which are run for 100k steps.

In Table 1 we report mean test reward for the median trial. We define the
median trial as the trial for which the mean test reward across modules is the
median across trials. In appendix B we report results from all trials and also
provide validation curves corresponding to the results reported in Table 1.

In our experiments we consider only the uncomposed modules. Composed modules
are distinguished by containing much longer problems (they are the modules
suffixed by ”_composed” or ”_compose”). Based on preliminary experiments, the
methods employed here do not perform well on the composed modules due to the
increased challenge of combinatorial explosion. Note also that in the
experiments we report here the ”calculus_differentiate” module is filtered to
remove multivariate problems as this reduces the maximum required graph size
(the option to remove this filter is provided as a hyperparameter of the
environment).

In these experiments we also limit the action space to contain only the 15
operators required to successfully compute correct graphs on the modules
selected. The list of operators used is provided in appendix B.2.

In our first experiment we train on all uncomposed modules simultaneously and
observe significant interference between modules reflected by lower final test
performance than in smaller subsets of modules. We hypothesize that by
including multiple modules in which answers to problems are similarly
expressed, interference between modules is magnified. For example, the modules
”numbers__is_factor” and ”numbers__is_prime” both have true/false answers. In
the case in which both modules are trained on simultaneously the operators
”divides” and ”is_prime” respectively will be frequently misused during
exploration (e.g. Is 5340 a multiple of 10? not(is_prime(Value(’10’))) = True,
which results in a positive reward).

To investigate this hypothesis we select two additional subsets of modules.
The first of these module subsets contains ”numbers__is_factor” and
”numbers__is_prime”, and we refer to it as the ”Inteference” experiment. The
second contains ”numbers__is_prime”, ”numbers__list_prime_factors”,
”numbers__calculus_differentiate”, ”numbers__div_remainder”, and
”numbers__gcd”, and we refer to it as the ”No Interference” experiment. The
modules in the ”Interference” experiment are selected because the form of the
answers in those modules are both true/false and so will result in the type of
collisions described above (i.e. noisy rewards). The modules in the ”No
Interference” experiment are selected because the form of the answers in those
modules is such that they are unlikely to result in the type of collisions we
describe above. However, notably the final results show that
”numbers__is_prime” had better performance in the ”Interference” experiment
which is evidence against the hypothesis.

Test Results
---
Module | Interference | No Interference | All Uncomposed Modules
numbers__is_factor | 0.7800 | - | 0.3669
numbers__is_prime | 1.000 | 0.7382 | 0.6567
numbers__list_prime_factors | - | 1.000 | 1.000
calculus__differentiate | - | 0.8511 | 0.3350
polynomials__evaluate | - | - | 0.9517
numbers__div_remainder | - | 1.000 | 0.8159
numbers__gcd | - | 1.000 | 0.9990
numbers__lcm | - | - | 1.000
algebra__linear_1d | - | - | 0.1124
algebra__polynomial_roots | - | - | 0.8253
algebra__linear_2d | - | - | 0.2660
Mean Reward across Modules | 0.8900 | 0.9179 | 0.6830

Table 1: Test reward per module for the median trial of each experiment.

##  6 Discussion

There are two main contributions of this paper. We have introduced a new
reinforcement learning environment by interpreting the DeepMind Mathematics
Dataset as a problem in program synthesis. We have also trained a baseline
model on several subsets of the uncomposed modules despite the dual challenge
of combinatorial explosion and noisy rewards.

From preliminary experiments it’s clear that performance on significantly
longer graphs is poor, however we believe that by implementing subgraph
abstraction (as described in section 3.1.2) performance on longer graphs could
be significantly improved. Furthermore we suspect that content based attention
(as in [15]) could be a useful architectural component to integrate into the
learned model due to the dynamic nature of the action space. We consider these
to be interesting topics for future research.

## Code

We provide a light-weight repository containing an implementation of the
reinforcement learning environment and setup instructions:

https://github.com/JohnnyYeeee/math_prog_synth_env

We also provide the full code used to produce the results reported here:

https://github.com/joepalermo/dm_math_solvers

## Acknowledgments

The authors would like to thank Alex Krizhevsky, Rayhane Mama, Hashiam Kadhim,
Marc Tyndel, and Ragavan Thurairatnam for helpful discussions.

## References

  * [1] David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing Mathematical Reasoning Abilities of Neural Models. In International Conference on Learning Representations, 2019.
  * [2] Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Juergen Schmidhuber, and Jianfeng Gao. Enhancing the Transformer With Explicit Relational Encoding for Math Problem Solving. arXiv preprint arXiv:1910.06611, 2019.
  * [3] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Autoregressive Generative Modeling, 2020. URL https://arxiv.org/abs/2010.14701.
  * [4] Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. 2019. Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension. arXiv preprint arXiv:1909.00109
  * [5] Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2020. Neural Module Networks for Reasoning over Text. In International Conference on Learning Representations.
  * [6] Sumit Gulwani, Alex Polozov and Rishabh Singh. Program Synthesis. Foundations and Trends in Programming Languages, vol. 4, no. 1-2, pp. 1–119, 2017. URL https://www.microsoft.com/en-us/research/publication/program-synthesis
  * [7] Guido Van Rossum, & Fred L. Drake, Jr. Python reference manual. 1995. Centrum voor Wiskunde en Informatica Amsterdam.
  * [8] Chuntao Jiang, Frans Coenen, and Michele Zito. A Survey of Frequent Subgraph Mining Algorithms. Knowledge Engineering Review, vol. 28, pp. 75-105, 2013. URL https://livrepository.liverpool.ac.uk/3005601/1/ker-jct-6-May-11.pdf
  * [9] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sabl´e-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning. arXiv:2006.08381, 2020.
  * [10] Hado van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-Learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016. URL http://arxiv.org/abs/1509.06461.
  * [11] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay, 2015. URL https://arxiv.org/abs/1511.05952.
  * [12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 6000–6010, 2017.
  * [13] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014.
  * [14] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep Reinforcement Learning that Matters. arXiv preprint arXiv:1709.06560, 2017.
  * [15] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473, 2014.

##  Appendix A Environment

###  A.1 Predefined Operators

We provide here a list of the predefined operators. The numbers associated to
them indicates their index in the action space. They are also provided along
with a type signature (e.g. f(x: type_of_x) -$>$ return_type).

  1. 0.

⬇

lookup_value(mapping: Dict[Variable: Value], key: Variable) -> object

Given a dictionary and a key to query it, it returns the corresponding value.

  2. 1.

⬇

solve_system(system: List[Equation]) -> Dict[Variable: Value]

Given a system of equations returns a dictionary mapping variables to values.

  3. 2.

⬇

append(system: List[Equation], equation: Equation) -> List[Equation]

Given a list of equations, appends a new equation to the end of that list.

  4. 3.

⬇

append_to_empty_list(equation: Equation) -> List[Equation]

Returns a new list containing only the given equation.

  5. 4.

⬇

factor(inpt: Expression) -> Expression

Converts a polynomial into irreducible factors over rational numbers.

  6. 5.

⬇

differentiate(expression: Expression) -> Expression

Returns the first derivative of a polynomial. (This function assumes
univariate)

  7. 6.

⬇

mod(numerator: Value, denominator: Value) -> Value

Returns the remainder of the numerator divided by the denominator

  8. 7.

⬇

gcd(x: Value, y: Value) -> Value

Returns the greatest common divisor of x and y

  9. 8.

⬇

divides(numerator: Value, denominator: Value) -> bool

Returns True if denominator is divisible by numerator

  10. 9.

⬇

is_prime(x: Value) -> bool

Returns True if x is prime and False otherwise

  11. 10.

⬇

lcm(x: Value, y: Value) -> Value

Returns the least common multiple of x and y

  12. 11.

⬇

lcd(x: Rational, y: Rational) -> Value

Given two rationals return the least common denominator

  13. 12.

⬇

prime_factors(n: Value) -> Set[Value]

Returns a set of all prime factors of n

  14. 13.

⬇

evaluate_function(function_definition: Function, function_argument:
Expression) -> Value

Evaluates a function by substituting the variable in function_definition by
the function_argument. function_argument can either look like ’2’ or ’f(2)’

  15. 14.

⬇

not_op(x: bool) -> bool

Returns the inverse of a boolean.

  16. 15.

⬇

differentiate_wrt(expression: Expression, variable: Variable) -> Expression

Returns the first derivative of an expression with respect to a given
variable.

  17. 16.

⬇

make_equation(expression1: Expression, expression2: Expression) -> Equation

Returns an equation where expression1 is set equal to expression2.

  18. 17.

⬇

simplify(inpt: object) -> object

Returns a simplification of inpt based on sympy heuristics.

  19. 18.

⬇

make_function(expression1: Expression, expression2: Expression) -> Function

Returns a function where expression1 is set to be equal to expression2.

  20. 19.

⬇

replace_arg(function: Function, var: Variable) -> Function

Replaces the argument in function with the given variable.

  21. 20.

⬇

lookup_value_equation(mapping: Dict[Variable: Value], key: Variable) ->
Equation

Given a dictionary and a key to query it, it returns an equation of ”key =
value”

  22. 21.

⬇

extract_isolated_variable(equation: Equation) -> Variable

Given an equation it returns the isolated variable.

  23. 22.

⬇

substitution_left_to_right(arb: object, eq: Equation) -> object

Returns the arb with all found instances of the equation’s left hand side
substituted by the equation’s right hand side.

###  A.2 Supported Modules

Here is the full list of supported modules:

⬇

numbers__is_factor

numbers__is_prime

numbers__list_prime_factors

calculus__differentiate

polynomials__evaluate

numbers__div_remainder

numbers__gcd

numbers__lcm

algebra__linear_1d

algebra__polynomial_roots

algebra__linear_2d

algebra__linear_1d_composed

algebra__linear_2d_composed

algebra__polynomial_roots_composed

calculus__differentiate_composed

numbers__div_remainder_composed

numbers__gcd_composed

numbers__is_factor_composed

numbers__is_prime_composed

numbers__lcm_composed

numbers__list_prime_factors_composed

polynomials__evaluate_composed

polynomials__compose

##  Appendix B Experiments

###  B.1 Full Test Results

Full Test Results for All Uncomposed Modules Module Run 1 Run 2 Run 3 Run 4
Run 5 numbers__is_factor 0.2907 0.5239 0.2798 0.4807 0.3669 numbers__is_prime
0.7294 1.000 0.4860 1.000 0.6567 numbers__list_prime_factors 1.000 1.000 1.000
1.000 1.000 calculus__differentiate 0.3238 0.3611 0.6095 0.3377 0.3350
polynomials__evaluate 0.9990 0.9950 0.0000 0.9942 0.9517
numbers__div_remainder 1.000 1.000 0.7944 1.000 1.000 numbers__gcd 0.9981
0.8132 0.9317 1.000 0.9990 numbers__lcm 1.000 1.000 1.000 1.000 1.000
algebra__linear_1d 0.7470 0.7470 1.000 1.000 0.1124 algebra__polynomial_roots
0.4443 0.7159 0.9457 0.9143 0.8253 algebra__linear_2d 0.8110 0.1400 0.9600
1.000 0.2660 Mean reward across modules 0.6973 0.6816 0.6629 0.8843 0.6830

Table 2: Test reward per module for runs on all uncomposed modules

Full Test Results for ”Interference” Experiment Module Run 1 Run 2 Run 3 Run 4
Run 5 numbers__is_factor 0.5155 0.7893 0.9383 0.9922 0.7800 numbers__is_prime
0.5229 0.9980 0.5258 1.000 1.000 Mean reward across modules 0.5192 0.8937
0.7321 0.9961 0.8900

Table 3: Test reward per module for ”Interference” experiment

Full Test Results for ”No Interference” Experiment Module Run 1 Run 2 Run 3
Run 4 Run 5 numbers__is_prime 0.5563 0.5400 0.7694 1.000 0.7382
numbers__list_prime_factors 1.000 1.000 1.000 1.000 1.000
calculus__differentiate 0.8112 0.9572 1.000 0.9653 0.8511
numbers__div_remainder 1.000 1.000 1.000 1.000 1.000 numbers__gcd 0.9990 1.000
1.000 1.000 1.000 Mean reward across modules 0.8733 0.8994 0.9539 0.9931
0.9179

Table 4: Test reward per module for ”No Interference” experiment

###  B.2 Operators Selected for Experiments

  1. 1.

lookup_value

  2. 2.

solve_system

  3. 3.

append

  4. 4.

append_to_empty_list

  5. 5.

factor

  6. 6.

differentiate

  7. 7.

mod

  8. 8.

gcd

  9. 9.

divides

  10. 10.

is_prime

  11. 11.

lcm

  12. 12.

lcd

  13. 13.

prime_factors

  14. 14.

evaluate_function

  15. 15.

not_op

###  B.3 Validation Curves

In the below graphs the dark centre line shows the median of the five trials
and the shaded area bounds the 10th and 90th percentiles based on linear
interpolation.

Figure 2: Reward for all uncomposed modules over 50000 steps. Figure 3: Reward
for ”Interference” experiment over 100000 steps Figure 4: Reward for ”No
Interference” experiment over 50000 steps.


# Machine Learning For Beamline Steering

Isaac Kante<EMAIL_ADDRESS>
###### Abstract

Beam steering is the process involving the calibration of the angle and the
position at which a particle accelerator’s electron beam is incident upon the
x‐ray target with respect to the rotation axis of the collimator. Beam
Steering is an essential task for light sources. In the case under study, the
LINAC To Undulator (LTU) section of the beamline is difficult to aim. Each use
of the accelerator requires re-calibration of the magnets in this section.
This involves a substantial amount of time and effort from human operators,
while reducing scientific throughput of the light source. We investigate the
use of deep neural networks to assist in this task. The deep learning models
are trained on archival data and then validated on simulation data. The
performance of the deep learning model is contrasted against that of trained
human operators.

###### keywords:

Particle Accelerators , Beam Steering , Machine Learning , Deep Learning

††journal: Physics Open

organization=Long Island University,addressline=1 University Plaza,
city=Brooklyn, postcode=11201, state=New York, country=USA

##  1 Introduction

Beam steering (Metcalfe et al., 2007; Gao et al., 2018) is a process involving
the calibration of the angle and the position at which a particle
accelerator’s electron beam is incident upon the x‐ray target with respect to
the rotation axis of the collimator. The shape of the profile of the dose is
extremely dependent upon accurate beam steering. This is essential to ensure
correct delivery of the radiotherapy treatment plan. Beamline steering is
essential in particle accelerator systems. These systems rely on beamline
steering because it ensures that the beam stays on it’s intended target and
trajectories, which is critical for experimental and operation results.
Beamline steering is important for several reasons like minimizing beam loss,
maintaining beam stability, reducing interactions and accelerator components,
error correction, and safety. It has a significant impact on the quality of
scientific research, the safety of accelerator operations, and the success of
various applications across multiple fields. Furthermore, beam steering is
critical in mitigating errors and imperfections in the accelerator’s magnetic
and radio frequency systems, allowing for optimal performance and the
successful execution of a variety of experiments and applications in fields
such as particle physics, medical therapy, and materials science.

There are various methods used for beamline steering. I discuss the primary
ones:
Steering magnets \- This method includes dipole and quadropole magnets that
are used to adjust the trajectory of the beam. These magnets generate magnetic
fields that deflect or focus the beams as needed. Precise control of the
magnetic field strength and alignment allows for accurate steering (Chung et
al., 2015).
Beam Position Monitors (BPMs) \- This methods includes placing bpms along the
beamline to detect and measure the position of the beam. The data from the
bpms are used to make adjustments (Forck et al. (2009)).
Orbit Correction Systems \- Orbit correction systems consist of a combination
of steering magnets and position monitors. They continuously monitor the
beam’s position and adjust the magnets’ settings to keep the beam on its
desired trajectory, making real-time corrections (Yu et al. (1989)).
Reducing Interactions with Accelerator Components \- When beams stray from
their planned routes, they might contact with accelerator components,
resulting in undesirable consequences such as heating, radiation, or damage.
Precision steering reduces these interactions, extending the life of
accelerator components.

Machine learning offers a good alternative for beamline steering due to it’s
ability to handle complex and dynamic real-time data. This allows for precise
adjustments to maintain beam stability and optimize trajectory. Machine
learning offers processing vast of information such as bpms, and learn
patterns that humans might overlook. Machine learning algorithms can assess
this data in real time, discover deviations from the expected trajectory, and
generate the ideal settings for steering magnets and other beamline components
on their own. They can adapt to changing conditions and respond to
disturbances more quickly, enhancing beam quality and lowering the chance of
beam loss.

Some applications of Machine Learning in Beamlines include anomaly detection,
system modeling, virtual diagnostics, data analysis, and beamline tuning.

Anomaly detection and machine protection \- Machine learning models may
identify abnormalities or unexpected behaviors in accelerator systems in real
time, such as beam instabilities or equipment faults (Fol, 2017). When
abnormalities are discovered, these systems can initiate safety measures or
shutdown processes to safeguard equipment and workers.

System Modeling \- Machine learning is used to create models that replicate
accelerator system behavior, assisting in the prediction and understanding of
how different parameters and components impact beam trajectories and
performance (Gupta et al. (2021); Mishra et al. (2021)). These models help in
the design and optimization of systems.

Virtual Instrumentation and Virtual Diagnostics\- Machine learning is used in
virtual instrumentation and diagnostics to construct virtual representations
of accelerator components and systems. These simulations can give insights on
beamline behavior, allowing for the testing of different circumstances without
the need for real tests, thereby assisting in system design and diagnostics
(Pang et al., 2015).

Tuning, Control, Rapid Switching Between Operating Conditions\- To ensure
optimal beam steering, machine learning techniques are used to automatically
tune and manage accelerator settings (Scheinker and Gessner, 2015). They can
adjust to changing conditions in real time and allow for quick switching
between operational modes to fulfill experimental or operational requirements.

Advanced Data Analysis \- To extract important insights from the massive
amounts of data generated by accelerator systems, machine learning and
sophisticated data analysis techniques are applied. They can spot patterns,
correlations, and trends in data to enhance beamline guidance, solve problems,
and boost overall performance.

These applications highlight how machine learning could improve the
efficiency, precision, and automation of beamline steering and accelerator
operations, resulting in better experimental findings, less downtime, and
increased safety. The area is evolving as a result of advances in machine
learning techniques and the incorporation of artificial intelligence into
accelerator facilities (Edelen et al. (2018)).

##  2 Mathematical, Dataset And Numerical Details

###  2.1 Deep Learning Based Modeling

Deep learning is a form of machine learning that uses neural networks with
numerous layers to uncover complicated patterns from difficult data. These
deep neural networks, also known as deep learning models, have shown to be
extremely useful in a variety of disciplines (Shlezinger et al. (2023)). Deep
learning excels at picture and audio recognition, natural language processing,
and other tasks (Deeplearning). It is based on an architecture inspired by the
biological neurons found in the brains of humans, that employs layers of
linked neurons to gradually improve and abstract data representations. This
layer structure enables deep learning models to automatically learn and adapt
to data attributes, resulting in cutting-edge outcomes in tasks such as
picture classification, object identification, and language translation.
Figure 2 is the deep learning model used for this project with seven fully
connected layers, each followed by a Rectified Linear Unit (ReLU) activation
function. It takes a certain size input, runs it through numerous hidden
layers, and returns predictions. The model can capture detailed patterns and
correlations in the data due to this architecture. The use of ReLU activation
promotes nonlinearity, which improves the model’s ability to learn complex
mappings. Because of its modular nature the deep learning model is adaptable
and flexible, making it suited for a variety of applications such as
regression or classification. In this study we use the PyTorch library (Paszke
et al. (2019)) to create, test and tune the deep learning models.

Figure 1: Neural Network Architecture used in this study. The Input is 73
dimensional, each of the hidden layers have 512 units, the output is 50
dimensional.

###  2.2 Dataset Details

Figure 2: Schematic of the Beamline, highlighting the LTU (LINAC To Undulator)
section. This section is difficult to correct for the operators. Each use of
the accelerator requires recalibrating the magnets in this section.

The Linac Coherent Light Source (LCLS) is a x ray Free Electron Laser (FEL)
based on a long, permanent magnet undulator constituted of numerous segments
that provide space for electron beam position monitors and quadrupole focusing
magnets (Emma, 1992). This is schematically shown in Figure 2. The 15 GeV
electron trajectory within the undulator must be completely straight to $5\mu
m$ over a gain length of 11 m so that the photon beam overlaps efficiently
with the electron beam. To train the model, we use archival data with the
required inputs and outputs for beam steering. The data being used for this
project are different Process Variables (PVs) arrays including XCOR, YCOR,
QUAD, BPMX, and BPMY.

The input features are X-Correcter (XCOR), Y-Corrector (YCOR), and Quadropole
magnet settings (QUAD).
XCOR and YCOR are horizontal and vertical beam correctors which are utilized
for shifting and adjusting the particle beam inside the beamline.
QUAD is the quadropole magnet that consist of groups of four magnets laid out,
which are used for focusing the particle beam. Bar magnets have 2 poles and
are di-polar. Quadrropole magnets have 4 poles. They can be used to focus and
transport charge particle beams, like x-rays, through long beamlines, in
linear colliders and circular machines like synchrotrons.
BPMX and BPMY are refered to as the ”Beam Position Monitors” in the horizontal
and vertical directions. Beam Position Monitors (BPM) are the non-destructive
diagnostics used most frequently at nearly all linacs, cyclotrons, and
synchrotrons. BPMs deliver the centre of mass of the beam and act as a monitor
for the longitudinal bunch shape. BPMs are devices that measure the location
of the charged particle beam as it goes through the beamline in particle
accelerators. They give real-time input on the location of the beam, allowing
for fine control and modifications, including trajectory correction. Each of
these fields represents a type of PV and contains an object with value arrays.

###  2.3 Hyperparameter Selection

In this section I want to highlight the selection of the hyperparameters and
the approach of metric being used. The deep learning model architecture being
used in this project was a feedforward neural network. This consisted of 7
linear layers with a loss function of mean squared error (MSE) to calculate
the loss for this regression task. The layers take in an input, hidden layers
and an output. A forward pass method than specifies the data through the
network with a Rectified Linear Unit (ReLU) activation in sequence Agarap
(2018). The training loop hyperparameters used for the model was a learning
rate set to 0.001, which is used for controlling the step size taken during
optimization Smith (2017). The optimization algorithm used was the Adam which
is short for Adaptive Moment Estimation Zhang (2018). The training loop was
set to an epoch of 300. This parameter is the amount of times the model would
see the data during training Sinha et al. (2010). The patience parameter is
set to 5 for early stopping if the validation loss does not improve for 5
consecutive epochs. This parameters helps with overfitting and saving training
time Prechelt (2002).

###  2.4 Simulation based data generation

The program used for generating simulation data is Bmad the software toolkit
for charged particles and x-ray simulations. According to Sagan (2006) ”Bmad
can be used to study both single and multi–particle beam dynamics as well as
X-rays”. Bmad is not a program itself and is used by programs. The program
used was Taothat was developed around bmad for this reason. ”Tao is a general
purpose simulation program, based upon Bmad. Tao can be used to view lattices,
do Twiss and orbit calculations, nonlinear optimization on lattices, etc Sagan
and Smith (2005).

Ensuring the simulation data corresponds to the same exact setting were
factored by certain steps. Keeping the data structure the same such as the
file and inputs that were used in model training, be the same when generating
the data. Keeping the same number of features as the training data and making
sure the targets are the corresponding targets as the training data. Applying
the same pre-processing steps as the training ensures the same settings as the
model. Evaluation metrics MSE and MAE stay the same for both the simulation
and real data. The amount of simulation data that was generated for testing
were 6 data sets.

###  2.5 Archival data

The archival data was derived from EPICS (Experimental Physics and Industrial
Control System) archiver. Specific timestamps from where we want the data are
included as well as the format we want the data in. Data retrieval involved
calling the method lcls archiver restore and supplies the parameters PVLIST
and epics timestamp. The function interacts with the EPICS archiver to get
historical data for the Process Variables (PVs) mentioned in PVLIST at the
supplied timestamp.

##  3 Hyperparameter Tuning

Tuning hyperparameters is an important part of developing machine learning
models since it has a direct influence on model performance and
generalization. Hyperparameters are settings that cannot be learnt from data
and must be specified beforehand. Learning rates, batch sizes, network
designs, regularization settings, and other factors are among them. Properly
calibrated hyperparameters can result in a model that not only fits the
training data well but also generalizes well to previously unknown data.
According to (Hutter et al., 2014) “Bayesian optimization found a better
instantiation of nine convolutional network hyperparameters than a domain
expert, thereby achieving the lowest error reported on the CIFAR-10 benchmark
at the time”. The goal of hyperparameter tuning is to optimize the
hyperparameters in a way that improves the performance of the machine learning
or deep learning model.

###  3.1 Automating Hyperparameter Tuning

Because manual exploration of hyperparameter settings is time-consuming as
well as inefficient, automating hyperparameter tweaking is essential. Grid
search, random search, and Bayesian optimization are examples of automated
algorithms that systematically examine the hyperparameter space to discover
the optimal combination for the given job. These approaches involve assessing
the model’s performance on validation data for various hyperparameter
settings, allowing optimal values to be selected without substantial user
involvement. Automated hyperparameter tweaking can save time and increase
model performance in complicated settings with many hyperparameters. It also
makes repeatability and the capacity to adjust models to changing data easier.
One of the automated hyperparameter frameworks I have used for this project is
Optuna. Optuna is a framework for automated hyperparameter optimization that
simplifies the process of choosing optimal hyperparameter settings for machine
learning models. It automatically explores through a set of hyperparameter
values, determined by optimization algorithms, in order to optimize a certain
objective function (Akiba et al., ). Optuna has 3 design criteria that are
made to outperform many major black-box optimization frameworks while being
easy to use and easy to setup invarious environments (Akiba et al., ). Define-
by-Run Programming: This means that you can dynamically create and change the
search space for hyperparameter optimization as your program runs. It allows
you to specify, modify, or extend the hyperparameters you want to optimize
without requiring a fixed, predefined configuration. Efficient Sampling and
Pruning: The framework offers efficient methods for trying out different
hyperparameter values (sampling) and allows you to customize how these values
are explored. It also supports a technique called pruning to stop unpromising
trials early, making the optimization process more efficient. Versatile and
Easy to Use: The framework is designed to be flexible and adaptable for
various tasks. It can be easily set up, from simple interactive experiments to
complex distributed computations, making it a versatile tool for optimizing
machine learning models. In FigureLABEL:optuna I demonstrate the use of Optuna
in optimizing a neural network model. The figure illustrates the model
architecture, showcasing its dynamic layers and the Optuna-driven objective
function for hyperparameter tuning. The optimization explores the search space
for hidden dimensions, dropout rates, and learning rates. This approach
efficiently refines the model’s performance through iterative tuning,
resulting in improved accuracy. The flexibility and adaptability of Optuna
facilitate a systematic exploration of hyperparameter configurations,
enhancing the overall effectiveness of the machine learning model.

##  4 Model Prediction

In this section I compare the the models predictions against the ground truth
of the data. This evaluation is essential to assessing the model’s accuracy
and dependability in real-world circumstances. We acquire vital information
about its capabilities and limits by comparing its performance against
established benchmarks. This study not only confirms the model’s reliability,
but also encourages prospective changes, resulting in a strong and trustworthy
prediction architecture. By basing our analysis on actual data, we improve the
application and relevance of our conclusions, thereby improving the overall
credibility of our prediction model.

###  4.1 Archival and Simulation Data

I use four datasets for evaluation: 1 archival and 3 simulation. The archival
dataset is made up of real-world, historical data that captures genuine system
activity. Meanwhile, the simulation dataset contains artificially generated
data that simulates various scenarios. The strength of archive data is its
authenticity, which reflects true system details. It may, however, be lacking
in diversity. However, simulation data assures a wide range of scenarios but
may not fully reflect real-world details. Balancing various datasets
eliminates any biases and improves the model’s flexibility to varied
conditions, resulting in a complete assessment technique. Juggling the
authenticity of archival data with the diversity of simulated scenarios for a
comprehensive evaluation is one of the challenges. We use key metrics to
evaluate the model’s prediction accuracy, such as Mean Squared Error (MSE),
Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE). MSE calculates
the average squared difference between expected and actual values, favoring
precise forecasts while punishing bigger mistakes. In the original scale,
RMSE, the square root of MSE, gives a more interpretable statistic. MAE
calculates the average absolute difference, which provides a reliable
measurement of prediction accuracy. These measures were chosen because of
their ability to capture many elements of model performance while addressing
the problem’s sensitivity to both tiny and big mistakes. RMSE and MAE provide
insight into prediction accuracy and help in determining the model’s
generalization across different targets, providing a thorough assessment
technique for our regression problem.

###  4.2 Visualization Prediction Assessment

I focused our investigation on 4 targets, assessing its predictions across
four datasets. These datasets included one from the archival data sources and
three from simulations. I hoped to gain a more detailed understanding of how
our model works on various data sources by isolating a certain goal.
Predictions using archive data provide insights on real-world applicability,
whilst simulated datasets allow examination of the model’s adaptability to
diverse settings. The comparison across datasets improves our understanding of
the model’s resilience and generalization. In Figure 3 the model shows its
predictive accuracy with the archival data. An abundunce of the model
predictions vs the ground truth are aligned closely as shown in the plot. The
model shows its potential for high efficiency prediction.

The scatter plots in Figure 4 show a significant difference between model
predictions and ground truth for simulated data. The scattered distribution of
points indicates difficulties in the model’s adaptability to the complexities
of simulated events. Disagreements may arise as a result of the small
properties presented in the simulation environment, influencing the model’s
generalization. This visualization serves as an important diagnostic tool,
identifying areas where the model may need to be improved which would be
beneficial. Addressing these disparities in simulated data is consistent with
ongoing work to improve the model’s resilience, assuring its efficacy across
various datasets and confirming its trustworthiness in real-world
applications.

Figure 3: Model prediction vs Truths on Archival Data

Figure 4: Model prediction vs Truths on Simulation Datasets

##  5 Performance

In this section, we examine our model’s performance across multiple datasets.
First, we evaluate its performance on archival data, gaining insights into its
accuracy and generalization. Following that, the model’s performance on
simulated datasets is examined, revealing information on its adaptability to
various settings. The verification procedure assures that the model is
reliable and consistent. I hope to give a full review of our model’s strengths
and limits in collecting patterns across both real-world and simulated
environments by thoroughly analyzing metrics and visualizations.

###  5.1 Model performance on Archival data

The evaluations provide important insights into the performance of our model
on archival data. As shown in Table 1 the Mean Absolute Error (MAE) of 0.159
shows that our model’s predictions differ from the actual values by 0.159
units on average. At 0.308, the Mean Squared Error (MSE) indicates the average
squared difference between forecasts and true values, emphasizing the
importance of greater mistakes. The Root Mean Squared Error (RMSE) of 0.555,
which is obtained from the square root of the MSE, provides a measure of the
normal amount of errors, providing a comprehensive picture of model accuracy.
The low values metrics for archival data suggest that our model performs
decently on the dataset. The closeness of predicted and actual values
indicates the model’s potency in capturing patterns in archival data. The
model’s robustness is demonstrated to be inadequate in the field of new
generalization to new examples. A careful review of these indicators ensures a
thorough grasp of our model’s capabilities on the archival dataset.

###  5.2 Model performance on simulation data

The notable difference in model performance between archival and simulation
datasets raises concerns. While the model performs admirably on archive data,
larger errors on Simulation Data 1, 2, and 3 in Figure 1 indicate difficulties
in adjusting to new settings. Among the simulated datasets, Simulation Data 1
has the largest Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root
Mean Squared Error (RMSE). This imbalance highlights potential differences
between the simulated and real-world data distributions. This discrepancy may
be caused by factors such as noise, parameter mismatch, model architecture,
model interpretability, and data quality. Simulation Data 2 provides a decent
MAE but a much lower RMSE, showing that outliers may be influencing the total
error. It requires deeper examination into the error distribution and the
model’s sensitivity to extreme values. Despite having a lower MAE, the model
on Simulation Data 3 has a larger RMSE than Archival Data. Differences in data
features, such as feature distributions and patterns, between archival and
simulated datasets present a substantial difficulty. Overfitting of the model
during training on more controlled archival data may contribute to its
inability to generalize to the complexities presented by the simulated
datasets.

###  5.3 Model verification

Model verification is an important step in ensuring the model’s dependability
and suitability for deployment. This procedure involves thorough testing and
comparing model outputs to ground truth across multiple datasets. In our
example, verification evaluates the model’s performance consistency across
archival and simulation data. Disagreements may signal the need for additional
refining. We can learn about the model’s resilience and accuracy by looking at
measures like MAE, MSE, and RMSE. Successful verification boosts confidence in
the model’s ability to generalize, increasing trust in its predictions across
a wide range of circumstances and datasets.

Table 1: Model Performance Metrics on Archival data and on data from Bmad simulations. Data | MAE | MSE | RMSE
---|---|---|---
Archival Data | 0.159 | 0.308 | 0.555
Simulation Data 1 | 0.606 | 0.816 | 0.903
Simulation Data 2 | 0.311 | 0.200 | 0.448
Simulation Data 3 | 0.340 | 0.324 | 0.569

##  6 Beamline Steering and Orbit Correction

In the context of beamline steering and orbit correction for accelerators, the
following principles are fundamental. The orbit response matrix, denoted as
$\mathbf{R}$, plays a crucial role in characterizing the relationship between
the desired changes to corrector magnets ($\boldsymbol{\theta}$) and the
resulting difference between the target and measured orbits
($\Delta\mathbf{x}$). These concepts, central to the beam-based correction and
optimization of accelerators, have been extensively explored by Huang in the
book ’Beam-based correction and optimization for accelerators’ (2020) (Huang,
2020).

###  6.1 SVD Approach

The Singular Value Decomposition (SVD) of the orbit response matrix provides a
key insight into the modes governing orbit changes. This method, as outlined
by Huang, expresses $\mathbf{R}$ as
$\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{T}$, where $\mathbf{U}$ and
$\mathbf{V}$ are orthogonal matrices, and $\boldsymbol{\Sigma}$ is a diagonal
matrix of singular values. The effectiveness of different modes in inducing
orbit changes is encapsulated in these singular values.

Throughout this discussion, we will delve into the application and
significance of these principles, drawing on the comprehensive work presented
by Huang (Huang, 2020).

The SVD (Singular Value Decomposition) method for beamline steering is a
powerful way to correct the path of particles in accelerators. In this method,
the goal is to minimize the difference between the measured beam orbit and the
target orbit in a least-square sense. The system involves multiple BPMs (Beam
Position Monitors) and correctors. The key equation for orbit correction with
SVD is:

$\theta=(R^{T}R)^{-1}R^{T}\Delta x$ (1)

Here, $\mathbf{R}$ is the orbit response matrix, $\boldsymbol{\theta}$ is the
vector of desired changes to the corrector magnets, and $\Delta\mathbf{x}$ is
the difference between the target orbit and the measured orbit. The SVD of the
orbit response matrix, as discussed in ’Beam-based correction and optimization
for accelerators’ (Huang, 2020), is expressed as

$\mathbf{R}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{T},$

where $\mathbf{U}$ and $\mathbf{V}$ are orthogonal matrices, and
$\boldsymbol{\Sigma}$ is a diagonal matrix of singular values. The singular
values represent the effectiveness of different modes in making orbit changes.
The SVD method provides a unique solution to the least-square problem even
when the system is over-constrained or under-constrained. The solution is
further refined by considering the pseudo-inverse of $\mathbf{R}$, especially
when the number of correctors is greater than the number of BPMs. The pseudo-
inverse is calculated using the singular values, ensuring a stable solution.
The SVD approach allows for weighting BPMs differently based on their
importance in the orbit control system. This flexibility is valuable in
situations where precise control is needed at specific locations along the
beamline. The method is demonstrated in practical scenarios, such as the
correction of orbit errors in accelerators like SPEAR3. The results show that
with a relatively small number of singular value modes, significant reduction
in orbit distortion can be achieved. The approach is also extended to include
weighting factors for BPMs, allowing for fine-tuning of orbit control based on
specific requirements at different locations.

###  6.2 SVD vs Deep Learning

The Singular Value Decomposition (SVD) and Deep Learning represent distinct
approaches with specific strengths and applications. SVD is well-suited for
linear problems where the underlying physics or mathematics is well
understood, making it effective for scenarios like beamline steering in
accelerators. It provides clear insights into contributing modes and
interpretable matrices. However, SVD’s applicability diminishes in highly
nonlinear systems.

Deep Learning excels in solving complex, nonlinear problems, offering accurate
predictions where traditional models fall short. It operates as a ”black box”
lacking the interpretability of SVD. While SVD demands a grasp of physics and
accurate data, Deep Learning thrives on large datasets, making it ideal for
intricate relationships.

In terms of flexibility, SVD may struggle with dynamic systems, assuming a
certain linearity, while Deep Learning adapts to diverse problems, suitable
for evolving patterns. SVD relies on established mathematical procedures,
eliminating the need for training, unlike Deep Learning, which requires
extensive training and computational resources.

Application-wise, SVD finds use in physics, engineering, and linear systems,
exemplified by beamline steering. Deep Learning dominates image recognition,
natural language processing, and other areas with complex, nontraditional
data. Both methods cater to specific niches, emphasizing the importance of
selecting the right tool for the nature of the problem at hand.

##  7 Conclusions And Future Work

Beamline steering is an essential task for light sources. Currently beamline
steering is carried out using trained operators. Faster, more reliable and
automated beamline steering would be beneficial to light sources in increasing
the capacity of experiments to be executed at the facility and the quality of
the results. Machine Learning models are fitting for this application as we
have a lot of data to learn this mapping. Deep learning based models with
their accuracy and flexibility are the prime choice. Deep learning models
sometimes give overconfident answers and are not immune to covariate shifts.
In the case under study we applied Deep Learning models for beamline steering.
On the archival data where the models were trained the models gave very
precise answers. But on simulation datasets their accuracy was off by a factor
of 2 or more. This highlights the susceptibility of deep learning models to
covariate shifts. In all the studies the deep learning models were almost an
order of magnitude worse than the Singular Value Decomposition approach being
used right now for beamline steering. In future work I recommend the use of
Recurrent Neural Network based models. These models are used to model
sequences where there is an order. On the beamline where different instruments
are positioned at different locations one after the other, this sequence
modeling should increase the quality of predictions by incorporating this
inductive bias.

## References

  * Agarap (2018) Agarap, A.F., 2018.  Deep learning using rectified linear units (relu).  arXiv preprint arXiv:1803.08375 .
  * (2) Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M., .  Optuna: Anext-generation hyperparameter optimization framework.  def 18, 19.
  * Chung et al. (2015) Chung, B., Kim, J., Kim, H., Kwon, H., Cho, Y., 2015.  Design, fabrication and installation of the steering magnet with high uniformity of magnetic field for komac beamline. .
  * Edelen et al. (2018) Edelen, A., Mayes, C., Bowring, D., Ratner, D., Adelmann, A., Ischebeck, R., Snuverink, J., Agapov, I., Kammering, R., Edelen, J., et al., 2018.  Opportunities in machine learning for particle accelerators.  arXiv preprint arXiv:1811.03172 .
  * Emma (1992) Emma, P., 1992.  Beam based alignment of sector-1 of the SLC Linac.  Technical Report. Stanford Linear Accelerator Center.
  * Fol (2017) Fol, E., 2017.  Evaluation of machine learning methods for LHC optics measurements and corrections software.  Ph.D. thesis. Hochschule, Eng. Econ., Karlsruhe.
  * Forck et al. (2009) Forck, P., Liakin, D., Kowina, P., 2009.  Beam position monitors .
  * Gao et al. (2018) Gao, S., Balter, P.A., Tran, B., Rose, M., Simon, W.E., 2018.  Quantification of beam steering with an ionization chamber array.  Journal of applied clinical medical physics 19, 168–176.
  * Gupta et al. (2021) Gupta, L., Edelen, A., Neveu, N., Mishra, A., Mayes, C., Kim, Y.K., 2021.  Improving surrogate model accuracy for the lcls-ii injector frontend using convolutional neural networks and transfer learning.  Machine Learning: Science and Technology 2, 045025.
  * Huang (2020) Huang, X., 2020.  Beam-based correction and optimization for accelerators.  Taylor & Francis.
  * Hutter et al. (2014) Hutter, F., Hoos, H., Leyton-Brown, K., 2014.  An efficient approach for assessing hyperparameter importance, in: International conference on machine learning, PMLR. pp. 754–762.
  * Metcalfe et al. (2007) Metcalfe, P.E., Kron, T., Hoban, P., 2007.  The physics of radiotherapy x-rays and electrons .
  * Mishra et al. (2021) Mishra, A.A., Edelen, A., Hanuka, A., Mayes, C., 2021.  Uncertainty quantification for deep learning in particle accelerator applications.  Physical Review Accelerators and Beams 24, 114601.
  * Pang et al. (2015) Pang, X., et al., 2015.  Advances in proton linac online modeling.  IPAC’15 , 2423–2427.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S., 2019.  Pytorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems 32. Curran Associates, Inc., pp. 8024–8035.
  * Prechelt (2002) Prechelt, L., 2002.  Early stopping-but when?, in: Neural Networks: Tricks of the trade. Springer, pp. 55–69.
  * Sagan (2006) Sagan, D., 2006.  Bmad: A relativistic charged particle simulation library.  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 558, 356–359.  doi:https://doi.org/10.1016/j.nima.2005.11.001. proceedings of the 8th International Computational Accelerator Physics Conference.
  * Sagan and Smith (2005) Sagan, D., Smith, J., 2005.  The tao accelerator simulation program , 4159–4161doi:10.1109/PAC.2005.1591750.
  * Scheinker and Gessner (2015) Scheinker, A., Gessner, S., 2015.  Adaptive method for electron bunch profile prediction.  Physical Review Special Topics-Accelerators and Beams 18, 102801.
  * Shlezinger et al. (2023) Shlezinger, N., Whang, J., Eldar, Y.C., Dimakis, A.G., 2023.  Model-based deep learning.  Proceedings of the IEEE 111, 465–499.  doi:10.1109/JPROC.2023.3247480.
  * Sinha et al. (2010) Sinha, S., Singh, T., Singh, V., Verma, A., 2010.  Epoch determination for neural network by self-organized map (som).  Computational Geosciences 14, 199–206.
  * Smith (2017) Smith, L.N., 2017.  Cyclical learning rates for training neural networks, in: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464–472.  doi:10.1109/WACV.2017.58.
  * Yu et al. (1989) Yu, L., Biscardi, R., Bittner, J., Bozoki, E., Galayda, J., Krinsky, S., Nawrocky, R., Singh, O., Vignola, G., 1989.  Real time closed orbit correction system, in: Proceedings of the 1989 IEEE Particle Accelerator Conference,.’Accelerator Science and Technology, IEEE. pp. 1792–1794.
  * Zhang (2018) Zhang, Z., 2018.  Improved adam optimizer for deep neural networks, in: 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), pp. 1–2.  doi:10.1109/IWQoS.2018.8624183.