11institutetext: Process and Data Science Chair, RWTH Aachen University,
Aachen, Germany 22institutetext: Fraunhofer FIT, Sankt Augustin, Germany
22email: {a.berti<EMAIL_ADDRESS><EMAIL_ADDRESS>
# PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks

Alessandro Berti 1122 0000-0002-3279-4795 Humam Kourani 2211
0000-0003-2375-2152 Wil M. P. van der Aalst 1122 0000-0002-0955-6940

###### Abstract

Large Language Models (LLMs) have the potential to semi-automate some process
mining (PM) analyses. While commercial models are already adequate for many
analytics tasks, the competitive level of open-source LLMs in PM tasks is
unknown. In this paper, we propose _PM-LLM-Benchmark_ , the first
comprehensive benchmark for PM focusing on domain knowledge (process-mining-
specific and process-specific) and on different implementation strategies. We
focus also on the challenges in creating such a benchmark, related to the
public availability of the data and on evaluation biases by the LLMs. Overall,
we observe that most of the considered LLMs can perform some process mining
tasks at a satisfactory level, but tiny models that would run on edge devices
are still inadequate. We also conclude that while the proposed benchmark is
useful for identifying LLMs that are adequate for process mining tasks,
further research is needed to overcome the evaluation biases and perform a
more thorough ranking of the “competitive” LLMs.

###### Keywords:

Process Mining Large Language Models Evaluation Strategies LLM Benchmarking

##  1 Introduction

Process mining (PM) is a branch of data science aiming to derive process-
related insights from the event data recorded during the execution of a
process. A wide set of automated PM techniques exist for process discovery
(the automated discovery of process models starting from the event data),
conformance checking (comparing event data and process models), and model
enhancement (annotating a process model with metrics derived from the event
data). PM could benefit significantly from the provision of domain knowledge
[7]. Modern Large Language Models (LLMs) have the capability to follow the
instructions contained in a given prompt and are trained on large sets of
generic knowledge, including process-related knowledge. The release of
OpenAI’s GPT-4 has been a milestone, as such LLM proved capable in different
PM tasks [4] including semantic anomaly detection and root cause analysis.
However, GPT-4 is a commercial LLM, and open-source LLMs were significantly
behind the quality of GPT-4. Recently, many companies released good-performing
open-source LLMs, which in general-purpose benchmarks approach GPT-4-level
quality111https://chat.lmsys.org/. For example, _Llama 3_
222https://llama.meta.com/llama3/ by Meta, _Mixtral 8x7B_ and _Mixtral 8x22B_
333https://mistral.ai/news/mixtral-of-experts/ by Mistral, and _WizardLM2_
444https://huggingface.co/WizardLM by Microsoft are all-rounder LLMs. Also
alternative commercial models have been proposed from Antrophic (_Claude AI_
555https://claude.ai/) and Google (_Gemini 666https://gemini.google.com/_),
which also approach GPT-4 levels of quality.

Given the large number of commercial and open-source LLMs, benchmarks are
essential to distinguish between good and bad-for-the-purpose LLMs. While many
general-purpose benchmarks exist for LLMs, there is a lack of comprehensive
benchmarks in process mining (PM). Several factors contribute to the
difficulty of proposing such a benchmark: multiple PM artifacts exist (e.g.,
traditional/object-centric event logs, procedural/declarative process models,
situation tables); multiple PM types exist (e.g., process discovery,
conformance checking, applications of machine learning such as anomaly
detection and root cause analysis, predictive analytics); multiple PM
practices exist, with different pathways followed during a process mining
analysis [27]; multiple PM code libraries and query languages exist, including
Python (e.g., pm4py), SQL, and non-relational languages; and the ability to
propose valuable answers depends on the ability of the analyst to propose
valuable inquiries/hypotheses [2].

In this paper, we propose three main contributions: i) a first _comprehensive
benchmark for process mining tasks executable by LLMs_ , focusing on two
implementation paradigms (direct provision of insights and code generation),
and including several categories of “static” prompts (stored in TXT files)
requiring process-mining-specific and process-specific domain knowledge; ii) a
_scalable evaluation strategy_ to assess the quality of the textual/coding
answers provided by LLMs; iii) the _results of the application of the
benchmark_ on several state-of-the-art LLMs. While the benchmark provides a
score useful to rank LLMs, some caveats discussed in Section 3 suggest
avoiding comparing the scores for highly-performing LLMs. In particular, the
role and limitations of LLMs as judges for PM tasks’ outputs need to be
discussed along with the need for ground truth in scoring the answers.
Moreover, advanced implementation paradigms (RAG, agents crew, multi-stage
hypothesis generation), discussed in Section 5, are not assessed by the
benchmark.

The rest of the paper is organized as follows. Section 2 describes the related
work connecting process mining and LLMs, and the evaluation of LLMs outputs.
Section 3 introduces the categories of prompts included in the benchmark and
the proposed evaluation strategy. Section 4 discusses the results of the
benchmark on state-of-the-art LLMs. Section 5 proposes some novel scenarios
requiring novel benchmarks. Finally, Section 6 concludes the paper.

##  2 Related Work

Connecting LLMs to PM: Several Business Process Management tasks have been
linked with LLMs [24]. For instance, process modeling exploits LLMs to create
process models starting from textual descriptions [11, 16, 15]. Process mining
tasks have been implemented on LLMs thanks to textual description of the
mainstream artifacts (event logs, process models) [4]. Three implementation
paradigms are identified [3]: i) _direct provision of insights_ , ii)
_generation of database queries_ (SQL), and iii) _autonomous hypotheses
generation_. The direct provision of insights requires the provision of the
necessary information to the LLM. Generating database queries [13] uses LLMs
to create (SQL) statements to be executed against the data source, mitigating
privacy risks but not exploiting the domain knowledge of LLMs. The autonomous
formulation of hypotheses combines the two methodologies allowing the LLM to
create database queries and interpret their output. In [2], the capabilities
required for process mining on LLMs are described: _acceptance of long
prompts_ , allowing for the provision of a significant amount of information
to the LLM; _acceptance of visual prompts_ , as visualizations allow us to
easily identify process-related patterns; _coding capabilities_ , including
the generation of scripts and SQL statements; and _factuality_ , being able to
cross-check the outputs against knowledge bases or search engines. Given the
absence of process-mining-specific benchmarks, [2] suggests using general-
purpose benchmarks (using traditional, domain-knowledge, visual, coding,
fairness, and hypotheses generation benchmarks).

Other Benchmarks of PM on LLMs: In [13], the authors assess the LLM-based
translation of process mining questions proposed in [1] to SQL statements.
They found that even advanced LLMs require the provision of database-specific
and process-mining-specific domain knowledge, that requires prompt injection.
Moreover, the questions that could be translated to SQL statements are
quantitative and do not assess the process-specific knowledge of the LLMs. In
[3], an initial comparison of two LLMs (GPT-4 and Google Bard) on different
process mining tasks and implementation paradigms is performed. The results
lay the foundations for this paper, which proposes a much more comprehensive
benchmark. In [10, 9], some benchmarks covering causal reasoning and
explaining decision points in business processes are proposed. In [21],
benchmarks for some semantics-aware process mining tasks, i.e. semantic
anomaly detection and the prediction of the next activity, are provided along
with strategies to fine-tune the LLMs to improve their ability to execute the
tasks. Moreover, [26] proposes a benchmark at the intersection between Robotic
Process Automation and Business Process Management, evaluating the ability to
exploit workflows recorded in user screenshots for Business Process Management
tasks.

LLMs-as-Judges: As the outputs of LLMs are mainly textual, evaluating them in
an automatic way is challenging. Some metrics have been proposed to evaluate
how well an answer matches a “ground truth” provided by an human
analyst777https://mlflow.org/docs/latest/llms/llm-evaluate/index.html.
However, their evaluation schema cannot be adapted to open-ended answers.
Another option is to let an LLM evaluate the answer (LLM-as-a-Judge) [28].
Using LLMs as judges, the evaluation can be tailored to the desired criteria
and consider open-ended answers. Some studies compared the scores given by
humans and LLMs to a given set of questions/answers, finding a good alignment
between humans and LLMs [28, 23]. However, some studies highlighted potential
weaknesses due to bias [20] and difficulty in following arbitrary evaluation
directives [8]. In [22], it is shown that LLMs (as also humans) suffer from
the Dunning-Kruger effect, underestimating/overestimating scores. LLMs-as-
Judges can be implemented with or without the provision of a ground truth [6].
If no ground truth is provided, it is necessary that the judge LLM would be
able to i) respond correctly to the given inquiry; ii) identify errors and
opportunities for improvement in the provided answer. In ranking different
LLMs without ground truth, there are other potential biases to consider. In
[25], it is highlighted how LLMs tend to prefer answers of similar LLMs. The
“egocentric bias” (LLMs preferring their own answers) is highlighted in [14].

##  3 Benchmark

In this section, we describe _PM-LLM-Benchmark_ , which is available at the
address https://github.com/fit-alessandro-berti/pm-llm-benchmark. First, in
Section 3.1, we describe the categories of prompts included in the benchmarks.
Then, in Section 3.2, we describe the evaluation strategy (LLM-as-a-Judge).

###  3.1 Categories of Tasks

Our benchmark measures how much an LLM is _knowledgeable_ and _capable_ in
process mining. The capability is measured in the correct interpretation and
production of different process mining artifacts (traditional and object-
centric event logs; procedural and declarative process models). We also
evaluate the ability of the LLM to autonomously formulate hypotheses over the
event data or the process model. Moreover, as the goal of process mining is to
assist data-driven decision-making, we aim to assess how much the LLM is able
to identify biases starting from the event data. We also want to assess the
ability of Large Vision Language Models (LVLMs; so LLMs supporting visual
prompts) to interpret popular visualizations and process mining diagrams.

Table 1: Prompts included in the benchmark.

| Prompt | Open | Requires DK | Task | Input Abstraction | Input Dataset
---|---|---|---|---|---|---
C1 | cat01_01_variants_bpic2020_rca | X | X | RCA | Variants | BPI2020 Domestic
C1 | cat01_02_variants_roadtraffic_anomalies | X | X | Semantic AD | Variants | Road Traffic
C1 | cat01_03_bpic2020_var_descr | X | X | Description | Variants | BPI2020 Domestic
C1 | cat01_04_roadtraffic_var_descr | X | X | Description | Variants | Road Traffic
C1 | cat01_05_bpic2020_dfg_descr | X | X | Description | DFG | BPI2020 Domestic
C1 | cat01_06_roadtraffic_dfg_descr | X | X | Description | DFG | Road Traffic
C1 | cat01_07_ocel_container_description | X | X | Description | OC-DFG | Logistics
C1 | cat01_08_ocel_order_description | X | X | Description | OC-DFG | Order Management
C1 | cat01_09_ocel_container_rca | X | X | RCA | OC-DFG | Logistics
C1 | cat01_10_ocel_order_rca | X | X | RCA | OC-DFG | Order Management
C2 | cat02_01_open_event_abstraction | X | X | Domain Knowledge |  |
C2 | cat02_02_open_process_cubes | X | X | Domain Knowledge |  |
C2 | cat02_03_open_decomposition_strategies | X | X | Domain Knowledge |  |
C2 | cat02_04_open_trace_clustering | X | X | Domain Knowledge |  |
C2 | cat02_05_open_rpa | X | X | Domain Knowledge |  |
C2 | cat02_06_open_anomaly_detection | X | X | Domain Knowledge |  |
C2 | cat02_07_open_process_enhancement | X | X | Domain Knowledge |  |
C2 | cat02_08_closed_process_mining |  | X | Domain Knowledge |  |
C2 | cat02_09_closed_petri_nets |  | X | Domain Knowledge |  |
C3 | cat03_01_temp_profile_generation | X | X | Process Modeling |  |
C3 | cat03_02_declare_generation | X | X | Process Modeling |  |
C3 | cat03_03_log_skeleton_generation | X | X | Process Modeling |  |
C3 | cat03_04_process_tree_generation | X | X | Process Modeling |  |
C3 | cat03_05_powl_generation | X | X | Process Modeling |  |
C3 | cat03_06_temp_profile_discovery | X | X | Process Discovery | Variants | Road Traffic
C3 | cat03_07_declare_discovery | X | X | Process Discovery | Variants | Road Traffic
C3 | cat03_08_log_skeleton_discovery | X | X | Process Discovery | Variants | Road Traffic
C4 | cat04_01_bpmn_xml_tasks | X | X | Task List | BPMN XML | Running Example
C4 | cat04_02_bpmn_json_description | X | X | Description | BPMN JSON | CCC19
C4 | cat04_03_bpmn_simp_xml_description | X | X | Description | BPMN XML (simple) | CCC19
C4 | cat04_04_declare_description | X | X | Description | DECLARE | BPI2020 Domestic
C4 | cat04_05_declare_anomalies | X | X | Semantic AD | DECLARE | BPI2020 Domestic
C4 | cat04_06_log_skeleton_description | X | X | Description | Log Skeleton | BPI2020 Domestic
C4 | cat04_07_log_skeleton_anomalies | X | X | Semantic AD | Log Skeleton | BPI2020 Domestic
C5 | cat05_01_hypothesis_bpic2020 | X | X | Hypothesis Generation | Variants | BPI2020 Domestic
C5 | cat05_02_hypothesis_roadtraffic | X | X | Hypothesis Generation | Variants | Road Traffic
C5 | cat05_03_hypothesis_bpmn_json | X | X | Hypothesis Generation | BPMN JSON | CCC19
C5 | cat05_04_hypothesis_bpmn_simpl_xml | X | X | Hypothesis Generation | BPMN XML (simple) | CCC19
C6 | cat06_01_renting_attributes |  | X | Discrimination Factors | Situation table | Renting (Fairness)
C6 | cat06_02_hiring_attributes |  | X | Discrimination Factors | Situation table | Hiring (Fairness)
C6 | cat06_03_lending_attributes |  | X | Discrimination Factors | Situation table | Lending (Fairness)
C6 | cat06_04_hospital_attributes |  | X | Discrimination Factors | Situation table | Hospital (Fairness)
C6 | cat06_05_renting_prot_comp | X | X | Comparison | Variants | Renting (Fairness)
C6 | cat06_06_hiring_prot_comp | X | X | Comparison | Variants | Hiring (Fairness)
C6 | cat06_07_lending_prot_comp | X | X | Comparison | Variants | Lending (Fairness)
C6 | cat06_08_hospital_prot_comp | X | X | Comparison | Variants | Hospital (Fairness)
C7 | cat07_01_dotted_chart | X | X | Description | Visual | Road Traffic
C7 | cat07_02_perf_spectrum | X | X | Description | Visual | Road Traffic
C7 | cat07_03_running-example | X | X | Description | Visual | Running Example
C7 | cat07_04_credit-score | X | X | Description | Visual | Credit Score
C7 | cat07_05_dfg_ru | X | X | Description | Visual | Running Example
C7 | cat07_06_process_tree_ru | X | X | Description | Visual | Running Example

The benchmark focuses on two implementation paradigms, i.e., the _direct
provision of insights_ and _code generation_. Moreover, specific focus is
given on process-mining-specific and process-specific _domain knowledge_ ,
which is required for the considered prompts. Other available lists of process
mining inquiries, such as the ones proposed in [1], are based on the
generation of SQL statements but do not require process-specific domain
knowledge.

Different categories of prompts (Table 1) are contained in the benchmark:

  1. C1

General-purpose qualitative tasks: The first category assesses the ability to
describe processes, detect anomalies, and analyze root causes using
DFG/variants abstractions of event logs. It also includes object-centric
process mining artifacts for testing object-centric comprehension.

  2. C2

Open/closed process mining domain knowledge questions: The second category
evaluates the process mining domain knowledge of the LLM, with open and closed
questions about process mining and Petri nets.

  3. C3

Process model generation: The third category tests the ability to generate
procedural (process trees, POWLs [17]) and declarative process models
(control-flow and temporal) for mainstream processes, and the ability to
propose constraints given some process data.

  4. C4

Process model understanding: The fourth category assesses the understanding of
proposed procedural (BPMN) and declarative (Log Skeleton and DECLARE [18])
process models.

  5. C5

Hypotheses generation: The fifth category evaluates the ability to generate
hypotheses over the proposed data and process models.

  6. C6

Fairness assessment: The sixth category tests the ability to identify event
log attributes sensitive for fairness and compare protected and non-protected
groups [19].

  7. C7

Visual prompts: The seventh category assesses the visual capabilities (if
supported) of the LLM/LVLM.

We tailored the benchmark to the level of comprehension and reasoning of
currently available state-of-the-art LLMs (in particular, _gpt-4o-20240513_
and _claude-3.5-sonnet_), which can perform the tasks satisfactorily. Some
mainstream tasks (for instance, applying the Alpha Miner algorithm, or
checking the soundness of a Petri nets) are still not supported effectively by
state-of-the-art LLMs. Therefore, they have not been included in the current
version of the benchmark, which is comprehensive but not complete. The prompts
of the benchmark are “static” (i.e., stored in TXT files). The benchmark could
be easily adapted in the future to contain more prompts and/or different
categories of tasks.

The size of the prompt does not exceed $8K$ characters, ensuring their
executability on any of the considered LLMs. Among the considered LLMs,
_Llama3 70B Instruct_ has the most restrictive context window (i.e., the
number of tokens that can be provided). Some state-of-the-art LLMs, such as
_Nemotron 340B_ or _Phi-3_ , only support a baseline context window of $4K$,
but we choose not to support that as it is too restrictive for process mining
tasks. For more difficult event logs or process models, a bigger context
window ($32K$, $64K$, or $128K$) is preferable as it allows to encode more
information in the prompt.

###  3.2 Evaluation Strategy

The challenge of evaluating textual outputs from LLMs in an objective manner
is significant. Traditional metrics that compare answers to a human-provided
ground truth are limited by their inability to be customized to specific
evaluation criteria and to consider open-ended answers. This leads to the
exploration of using LLMs as judges (LLMs-as-Judges), a method that offers the
potential for more comprehensive evaluations.

As the output of LLMs for the proposed prompts is textual, we propose to use
an advanced LLM (for instance, _gpt-4o-20240513_) as judge [28], assigning a
score from 1.0 (minimum) to 10.0 (maximum) to each answer.

In this benchmark, LLMs-as-Judges are utilized without ground truth for two
primary reasons:

  * •

The open-ended nature of many inquiries, in particular _process modeling_ (C3)
and _automated hypotheses generation_ (C5), which lack definitive answers.

  * •

The possibility that future LLM training sets could include the benchmark
inquiries and their ground truths, potentially compromising the integrity of
the benchmark by allowing “cheating” during the training phase of the LLM.

Potential limitations have been summarized in Section 2. In particular, LLMs-
as-Judges are more reliable when there is a significant performance gap in
favor of the judge LLM compared to the answering LLM. Also, scoring similar-
performing LLMs without ground truth is challenging and requires accounting
for potential “egocentric” bias. Moreover, high-performing LLMs tend to
produce verbose and detailed outputs, which can result in a bias against more
concise responses from lower-performing LLMs and vice-versa.

Table 2: Scores between 1.0 and 10.0 ($mean\pm stddev$) of various LLMs in the
proposed benchmark (using _gpt-4o-20240513_ as a judge).

Commercial LLMs | Big Open-Source LLMs | Small LLMs ($\leq$ 8GB) | Tiny LLMs ($\leq$ 4GB)
---|---|---|---
claude-3.5-sonnet | Qwen v2.0 72B (instruct, fp16) | Qwen v2.0 7B (instruct, Q6K) | Mistral 7B v0.3 (instruct, Q3KS)
$8.4\pm 0.7$ | $7.6\pm 1.4$ | $6.5\pm 2.2$ | $4.6\pm 2.2$
gpt-4o-20240513 (self) | WizardLM v2 8x22b (16b) | Mistral 7B v0.3 (instruct, Q6K) | Qwen v2.0 7B (instruct, Q2K)
$8.3\pm 1.0$ | $7.5\pm 1.7$ | $5.9\pm 2.5$ | $4.6\pm 2.1$
gpt-4-turbo-20240409 | Mixtral v0.1 8x22b (instruct, 16b) | Llama 3 8B (instruct, Q6K) | Gemma v1.0 2B (instruct, Q6K)
$8.1\pm 1.0$ | $7.5\pm 1.6$ | $5.9\pm 2.5$ | $4.0\pm 2.6$
claude-3-sonnet | Llama 3 70B (instruct, 16b) | WizardLM v2 7b (Q6K) | Qwen v2.0 1.5B (instruct, Q6K)
$7.8\pm 1.4$ | $7.4\pm 1.5$ | $5.9\pm 2.3$ | $3.8\pm 2.2$
Google Gemini (20240528) | Mixtral v0.1 8x7b (instruct, 16b) | Gemma v2.0 9B (instruct, Q6K) | Qwen v2.0 0.5B (instruct, Q6K)
$7.5\pm 1.7$ | $6.9\pm 1.7$ | $5.7\pm 2.9$ | $3.1\pm 1.7$
gpt-3.5-turbo-0125 | Llama 3 70B (instruct, Q4_0) | CodeGemma v1.5 7B (instruct, Q6K) | Qwen 4B v1.5 (text, Q6K)
$7.1\pm 1.8$ | $6.7\pm 2.4$ | $4.9\pm 2.6$ | $2.5\pm 2.2$
| Codestral 22B (Q6K) | Gemma v1.0 7B (instruct, Q6K) |
| $6.7\pm 2.1$ | $4.5\pm 2.6$ |
| Llama 3 8B (instruct, 16b) |  |
| $6.6\pm 2.0$ |  |
| OpenChat 3.6 8B (16b) |  |
| $6.5\pm 1.7$ |  |

Table 3: Scores between 1.0 and 10.0 ($mean\pm stddev$) for different model
categories and question categories (using _gpt-4o-20240513_ as a judge).

| Commercial LLMs | Big Open-Source LLMs | Small LLMs ($\leq$ 8GB) | Tiny LLMs ($\leq$ 4GB)
---|---|---|---|---
C1 | $7.6\pm 1.1$ | $7.0\pm 1.5$ | $5.5\pm 2.3$ | $3.2\pm 1.6$
C2 | $8.7\pm 0.6$ | $8.4\pm 0.9$ | $8.5\pm 0.9$ | $6.5\pm 2.2$
C3 | $7.1\pm 1.7$ | $5.7\pm 2.0$ | $4.7\pm 2.5$ | $2.4\pm 1.5$
C4 | $7.7\pm 1.5$ | $6.7\pm 1.9$ | $3.7\pm 2.0$ | $2.9\pm 1.5$
C5 | $7.8\pm 1.8$ | $7.3\pm 1.6$ | $6.0\pm 2.5$ | $4.6\pm 2.2$
C6 | $8.1\pm 1.2$ | $7.2\pm 2.0$ | $5.0\pm 2.2$ | $3.1\pm 1.6$
C7 | $8.2\pm 1.3$ | no supp. | no supp. | no supp.

For the evaluation, the following procedure is followed for every prompt:

  1. 1.

The prompt is provided to the LLM:

     * •

Reported as-is for all the textual prompts.

     * •

For visual prompts (if supported by the given model), upload the image
accompanied by the following textual prompt: _Can you describe the provided
visualization?_

  2. 2.

The LLM’s answer is persisted.

  3. 3.

An expert LLM (LLM-as-a-Judge) is used to evaluate the output. Template:

     * •

For textual prompts, _Given the following question: …How would you grade the
following answer from 1.0 (minimum) to 10.0 (maximum)?_.

     * •

For visual prompts, upload the image to the LVLM and ask _Given the attached
image, how would you grade the following answer from 1.0 (minimum) to 10.0
(maximum)?_.

Table 4: Scores between 1.0 and 10.0 ($mean\pm stddev$) for answering (rows)
and evaluating (columns) LLM pairs.

Answ./Eval. | gpt-4o-20240513 | Llama3 70B Instr. | Mixtral 8x22B | Mixtral 8x7B | Qwen2 72B Instr.
---|---|---|---|---|---
Llama3 70B Instr. | $7.4\pm 1.5$ | $\mathbf{8.8\pm 0.5}$ | $8.6\pm 1.4$ | $8.7\pm 1.3$ | $7.7\pm 2.4$
WizardLM-2-8x22B | $7.5\pm 1.7$ | $7.2\pm 3.0$ | $7.2\pm 2.7$ | $8.3\pm 1.3$ | $7.1\pm 2.9$
Mixtral 8x22B | $7.5\pm 1.6$ | $7.8\pm 2.3$ | $7.5\pm 3.1$ | $8.5\pm 1.3$ | $7.6\pm 2.4$
Mixtral 8x7B | $6.9\pm 1.7$ | $7.7\pm 2.3$ | $7.4\pm 2.9$ | $8.1\pm 1.7$ | $7.1\pm 2.8$
Qwen2 72B Instr. | $7.6\pm 1.4$ | $7.9\pm 2.3$ | $8.6\pm 1.3$ | $8.3\pm 1.9$ | $\mathbf{8.0\pm 2.5}$

Table 5: Scores between 1.0 and 10.0 ($mean\pm stddev$) for different model
categories and single prompts (using _gpt-4o-20240513_ as a judge).

Prompt | Comm. | Big OS | Small | Tiny
---|---|---|---|---
cat01_01_variants_bpic2020_rca | $8.6\pm 0.4$ | $6.9\pm 1.4$ | $5.7\pm 2.9$ | $3.2\pm 1.7$
cat01_02_variants_roadtraffic_anomalies | $6.5\pm 1.6$ | $6.1\pm 1.8$ | $4.3\pm 2.4$ | $2.2\pm 0.9$
cat01_03_bpic2020_var_descr | $7.8\pm 1.6$ | $7.5\pm 0.9$ | $7.0\pm 1.3$ | $3.4\pm 1.4$
cat01_04_roadtraffic_var_descr | $8.3\pm 0.6$ | $7.4\pm 0.9$ | $6.8\pm 1.4$ | $3.2\pm 1.6$
cat01_05_bpic2020_dfg_descr | $8.2\pm 0.6$ | $8.2\pm 0.7$ | $7.4\pm 1.8$ | $3.8\pm 2.1$
cat01_06_roadtraffic_dfg_descr | $7.8\pm 0.6$ | $6.4\pm 1.8$ | $5.1\pm 2.1$ | $3.9\pm 1.9$
cat01_07_ocel_container_description | $7.2\pm 0.9$ | $6.6\pm 1.5$ | $4.9\pm 1.2$ | $4.0\pm 1.3$
cat01_08_ocel_order_description | $7.7\pm 0.7$ | $6.8\pm 1.4$ | $4.8\pm 2.3$ | $3.5\pm 1.6$
cat01_09_ocel_container_rca | $7.2\pm 1.2$ | $7.5\pm 0.8$ | $5.4\pm 2.1$ | $2.6\pm 0.9$
cat01_10_ocel_order_rca | $7.2\pm 0.7$ | $6.7\pm 1.8$ | $3.5\pm 1.3$ | $2.5\pm 1.0$
cat02_01_open_event_abstraction | $8.7\pm 0.4$ | $8.4\pm 0.6$ | $8.7\pm 0.4$ | $6.6\pm 1.8$
cat02_02_open_process_cubes | $8.9\pm 0.2$ | $8.4\pm 0.5$ | $8.5\pm 0.5$ | $5.8\pm 1.8$
cat02_03_open_decomposition_strategies | $8.7\pm 0.9$ | $8.5\pm 0.3$ | $8.6\pm 0.3$ | $7.2\pm 1.2$
cat02_04_open_trace_clustering | $8.5\pm 0.7$ | $8.4\pm 0.5$ | $8.7\pm 0.7$ | $7.7\pm 0.9$
cat02_05_open_rpa | $9.1\pm 0.2$ | $8.8\pm 0.4$ | $9.0\pm 0.5$ | $8.2\pm 1.2$
cat02_06_open_anomaly_detection | $8.6\pm 0.5$ | $8.7\pm 0.4$ | $8.1\pm 1.0$ | $8.0\pm 1.4$
cat02_07_open_process_enhancement | $8.9\pm 0.3$ | $8.7\pm 0.4$ | $8.7\pm 0.7$ | $6.4\pm 1.0$
cat02_08_closed_process_mining | $8.8\pm 0.6$ | $8.0\pm 1.1$ | $8.8\pm 0.7$ | $6.0\pm 2.7$
cat02_09_closed_petri_nets | $8.1\pm 0.6$ | $7.2\pm 1.8$ | $6.9\pm 1.3$ | $2.9\pm 1.2$
cat03_01_temp_profile_generation | $9.0\pm 0.1$ | $7.8\pm 0.9$ | $7.8\pm 1.2$ | $3.4\pm 1.6$
cat03_02_declare_generation | $6.8\pm 1.5$ | $5.8\pm 1.2$ | $4.8\pm 1.4$ | $2.7\pm 0.9$
cat03_03_log_skeleton_generation | $8.2\pm 1.2$ | $7.3\pm 1.2$ | $5.8\pm 1.7$ | $3.2\pm 1.6$
cat03_04_process_tree_generation | $7.4\pm 1.2$ | $5.6\pm 2.0$ | $5.6\pm 1.6$ | $3.4\pm 1.4$
cat03_05_powl_generation | $6.6\pm 1.2$ | $6.5\pm 1.5$ | $6.9\pm 2.0$ | $2.5\pm 1.4$
cat03_06_temp_profile_discovery | $5.5\pm 1.8$ | $4.2\pm 1.7$ | $2.4\pm 0.7$ | $1.7\pm 0.7$
cat03_07_declare_discovery | $7.4\pm 1.3$ | $4.3\pm 1.4$ | $1.9\pm 1.0$ | $1.0\pm 0.0$
cat03_08_log_skeleton_discovery | $5.8\pm 1.2$ | $4.1\pm 1.4$ | $2.2\pm 1.1$ | $1.2\pm 0.6$
cat04_01_bpmn_xml_tasks | $9.2\pm 0.6$ | $7.8\pm 3.1$ | $1.3\pm 0.5$ | $1.7\pm 0.7$
cat04_02_bpmn_json_description | $7.9\pm 1.0$ | $6.3\pm 2.0$ | $2.9\pm 1.5$ | $2.8\pm 2.1$
cat04_03_bpmn_simp_xml_description | $8.2\pm 0.6$ | $6.7\pm 1.2$ | $3.6\pm 1.9$ | $2.6\pm 1.4$
cat04_04_declare_description | $7.3\pm 1.1$ | $7.1\pm 1.1$ | $5.7\pm 1.6$ | $3.6\pm 1.4$
cat04_05_declare_anomalies | $6.8\pm 2.0$ | $5.9\pm 1.6$ | $3.4\pm 0.9$ | $3.2\pm 1.3$
cat04_06_log_skeleton_description | $7.9\pm 1.3$ | $6.6\pm 1.4$ | $5.2\pm 1.9$ | $3.3\pm 1.7$
cat04_07_log_skeleton_anomalies | $6.5\pm 1.3$ | $6.5\pm 1.6$ | $4.0\pm 1.9$ | $2.9\pm 0.7$
cat05_01_hypothesis_bpic2020 | $8.2\pm 0.4$ | $7.4\pm 1.4$ | $7.6\pm 0.7$ | $4.6\pm 1.7$
cat05_02_hypothesis_roadtraffic | $8.1\pm 1.0$ | $6.6\pm 1.7$ | $6.8\pm 2.2$ | $4.7\pm 2.3$
cat05_03_hypothesis_bpmn_json | $7.5\pm 1.6$ | $7.7\pm 1.2$ | $5.2\pm 2.2$ | $4.5\pm 2.3$
cat05_04_hypothesis_bpmn_simpl_xml | $7.2\pm 2.9$ | $7.4\pm 1.9$ | $4.6\pm 2.9$ | $4.5\pm 2.4$
cat06_01_renting_attributes | $8.8\pm 0.6$ | $8.2\pm 1.4$ | $4.1\pm 2.0$ | $2.5\pm 1.1$
cat06_02_hiring_attributes | $8.9\pm 0.7$ | $8.6\pm 0.8$ | $6.1\pm 2.4$ | $5.0\pm 2.2$
cat06_03_lending_attributes | $8.3\pm 0.5$ | $8.3\pm 1.1$ | $7.1\pm 2.3$ | $2.8\pm 1.2$
cat06_04_hospital_attributes | $8.8\pm 0.5$ | $8.4\pm 0.7$ | $6.3\pm 1.8$ | $4.4\pm 2.1$
cat06_05_renting_prot_comp | $7.5\pm 1.2$ | $5.8\pm 1.8$ | $3.9\pm 1.1$ | $2.0\pm 0.0$
cat06_06_hiring_prot_comp | $7.2\pm 1.4$ | $6.1\pm 2.5$ | $4.4\pm 1.7$ | $2.9\pm 0.9$
cat06_07_lending_prot_comp | $8.3\pm 0.6$ | $6.5\pm 1.8$ | $3.6\pm 1.6$ | $2.3\pm 0.5$
cat06_08_hospital_prot_comp | $7.0\pm 1.6$ | $5.8\pm 1.8$ | $4.4\pm 1.6$ | $2.7\pm 0.9$
cat07_01_dotted_chart | $8.3\pm 0.7$ | - | - | -
cat07_02_perf_spectrum | $8.5\pm 0.8$ | - | - | -
cat07_03_running-example | $8.9\pm 0.5$ | - | - | -
cat07_04_credit-score | $8.6\pm 0.6$ | - | - | -
cat07_05_dfg_ru | $7.4\pm 2.3$ | - | - | -
cat07_06_process_tree_ru | $7.8\pm 1.0$ | - | - | -

###  3.3 Benchmarking Scripts

The provided benchmark can be executed manually (first, an LLM is used to
answer the questions, and its answers are evaluated by another LLM). We also
provide some scripts to automate the execution of the benchmark. In
particular, the Python script answer.py can be used to execute the prompts
against any LLM supporting the OpenAI APIs, while evaluation.py can be used to
evaluate the answers using an LLM as the judge. The configuration parameters
could be set up inside the two scripts.

##  4 Benchmark Results

We executed the proposed benchmark against current state-of-the-art LLMs. In
Table 2, we collect the evaluation results (using _gpt-4o-20240513_ as a
judge) for different LLMs of different sizes. In particular, we divide between
_commercial models_ (usually ranging hundreds of billions of parameters), _big
open-source LLMs_ , _small LLMs_ ($\leq$ 8 GB of RAM), and _tiny LLMs_ ($\leq$
4 GB of RAM).

We tested some LLMs with different levels of quantization. Quantization refers
to the process of reducing the precision of the model’s weights and
activations from higher bit-widths (such as 16-bit floating point) to lower
bit-widths (such as 8-bit or even lower). This technique aims to decrease the
model’s memory footprint and computational requirements, making it more
efficient to run on hardware with limited resources. While quantization can
lead to a slight loss in model accuracy, it often significantly improves the
model’s speed and reduces power consumption, enabling more practical and
scalable deployment of LLMs.

In general, we see that commercial and big open-source models can perform
process mining tasks adequately well. The winners among commercial models seem
to be _gpt-4o-20240513_ and _claude-3.5-sonnet_. The best of the open-source
models seem to be _Qwen2 72B Instruct_ (occupying $145GB$ of memory at full
quantization). Some small LLMs also report an overall sufficient score. For
instance, _Qwen2 7B Instruct_ at _Q6K_ quantization is process-mining-capable
while occupying just $6.3GB$ of memory. However, tiny LLMs are still
inadequate for process mining tasks.

Overall, we found that aggressive quantization has a severe impact on LLMs’
abilities in the proposed benchmark. While we do not have precise explanations
for this phenomenon, the benchmarks’ prompt requires significant attention to
some elements of the input prompt (e.g., the semantic anomalies), and
quantization impacts such ability.

In Table 3, we report the scores for every model category and question
category. In Table 5, the scores for every prompt of the benchmark are
averaged over a model category.

We notice that question category C2 (open/closed process mining domain
knowledge questions) is answered adequately by all the model categories. Also,
currently, only commercial models could answer to C7 (visual prompts).
Surprisingly, most of the considered LLMs could automatically generate
hypotheses over the provided process models and event data (C5). Categories C3
(process model generation) and C4 (process model understanding) have high
variance between the different model categories. While commercial and big
open-source LLMs can perform the tasks adequately, small and tiny models fail
to generate/understand the information provided in the prompt. Categories C1
(general-purpose qualitative tasks) and C6 (fairness assessment) have similar
outcomes. Commercial and big open-source LLMs successfully execute such tasks,
while small/tiny LLMs usually fail.

To further assess the validity of LLMs-as-Judges, we report in Table 4 a
cross-validation based on five open-source answering models and five
evaluation models (including _gpt-4o-20240513_). We see that only _Llama 3 70B
Instruct_ display the egocentric bias. The models tend to agree on the scores.
The smallest considered model (_Mixtral 8x7B_) reports lower scores than its
bigger counterpart (_Mixtral 8x22B_) and other state-of-the-art models (_Llama
3 70B Instruct_ and _Qwen2 72B Instruct_). Notably, _WizardLM2 8x22B_ achieves
lower overall scores than _Mixtral 8x7B_. This could be explained by the model
being a fine-tuned version of _Mixtral 8x22B_ favoring more verbose responses.
Overall, only _Qwen2 72B Instruct_ and _gpt-4o-20240513_ favor Qwen2 over
Llama3, while Qwen2 is an overall better-performing model in general-purpose
benchmarks. As the models and responses become more advanced, it becomes
complicated for lower-performing models to judge the differences between them.
Using the less advanced _Mixtral 8x7B_ , the “plateauing” of the scores
becomes evident, highlighting the importance of using advanced LLMs as judges.
Therefore, Table 4 justifies LLMs-as-Judges, but points to the importance of
choosing advanced LLMs for the task.

##  5 Future Benchmarking Strategies

Benchmarking Retrieval-Augmented Generation (RAG): while LLMs have been
trained on generic process-specific knowledge, the implementation of business
processes in real-life organizations could be different. Therefore, Retrieval-
Augmented Generation (RAG) techniques have been used to dynamically inject
process-specific knowledge to the prompts provided by process mining analysts
[5]. As the correct retrieval of the information is crucial for the quality of
the answers, benchmarks should also assess the capabilities of the RAG
pipeline.

Benchmarking LLM-Based Agent Crews: Our benchmark focuses on executing a
prompt, recording the answer, and evaluating it. The _agents crew_ paradigm
[12] involves workflows with specialized or role-specific LLMs performing
analytical tasks. For instance, estimating the level of discrimination in an
event log involves identifying the protected group and comparing differences
between groups. An agents crew could consist of one agent creating the SQL
statement to filter the protected group and another specializing in process
comparison. Each agent’s performance is crucial for accurate benchmarking, as
evaluating only the final output may be misleading.

Benchmarking Hypotheses Refinement: Hypotheses generation involves creating
and verifying hypotheses against event data. If a hypothesis is invalid, the
LLM refines it based on feedback. This process may lead to specific hypotheses
that perform poorly on imseem data. Thus, it is important to test the entire
hypothesis feedback and verification cycle, not just hypotheses generation.

Data Generation with Relevant Semantic Anomalies or Root Causes: Our benchmark
uses publicly available event data and process models. Including these
datasets in LLM training data, even without ground truth answers, can lead to
higher benchmark scores. Therefore, it is important to dynamically generate
novel event data for LLM benchmarking. Current simulation solutions lack the
semantic understanding needed to generate data suitable for process mining
assessment. LLMs should be considered for the goal of generating data with
various semantic anomalies and root causes.

##  6 Conclusion

We proposed _PM-LLM-Benchmark_ , a benchmark for process mining on LLMs,
utilizing LLMs-as-Judges for scalable evaluation. This benchmark does not
provide ground truth answers, relying instead on the ”judge” LLM’s
capabilities, addressing the open-ended nature of the inquiries. The benchmark
includes various task categories to assess LLMs’ abilities in process mining,
modeling, and comprehension. Applied to several commercial and open-source
LLMs, we found most models perform well in process mining tasks, with bigger
models achieving higher scores. Smaller models ($\leq$ 8 GB or $\leq$ 4 GB of
RAM) struggle with complex tasks. We also suggest improvements in benchmarking
strategies, including enhanced hypotheses generation, agent-crew-based
benchmarks, and dynamic dataset generation with semantic anomalies. To our
knowledge, this is the first general-purpose process mining benchmark focusing
on different implementation strategies. While it has limitations in evaluating
well-performing LLMs, it is a useful tool for assessing smaller LLMs and
evaluating the current state of PM-on-LLMs.

## References

  * [1] Barbieri, L., Madeira, E.R.M., Stroeh, K., van der Aalst, W.M.P.: A natural language querying interface for process mining. J. Intell. Inf. Syst. 61(1), 113–142 (2023)
  * [2] Berti, A., Kourani, H., Hafke, H., Yun-Li, C., Schuster, D.: Evaluating Large Language Models in Process Mining: Capabilities, Benchmarks, Evaluation Strategies, and Future Challenges. In: BPM-DS 2024 Working Conference. Springer (2024)
  * [3] Berti, A., Qafari, M.S.: Leveraging Large Language Models (LLMs) for Process Mining (Technical Report) (2023)
  * [4] Berti, A., Schuster, D., van der Aalst, W.M.P.: Abstractions, Scenarios, and Prompt Definitions for Process Mining with LLMs: a Case Study. In: BPM 2023\. vol. 492, pp. 427–439. Springer (2023)
  * [5] Chen, B., Zhang, Z., Langrené, N., Zhu, S.: Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023)
  * [6] Dhurandhar, A., Nair, R., Singh, M., Daly, E., Ramamurthy, K.N.: Ranking Large Language Models without Ground Truth (2024)
  * [7] Dixit, P.M., Buijs, J.C.A.M., van der Aalst, W.M.P., Hompes, B.F.A., Buurman, J.: Using Domain Knowledge to Enhance Process Mining Results. In: SIMPDA 2015\. vol. 244, pp. 76–104. Springer (2015)
  * [8] Dong, Y.R., Hu, T., Collier, N.: Can LLM be a Personalized Judge? arXiv preprint arXiv:2406.11657 (2024)
  * [9] Fahland, D., Fournier, F., Limonad, L., Skarbovsky, I., Swevels, A.J.E.: How well can large language models explain business processes? (2024)
  * [10] Fournier, F., Limonad, L., Skarbovsky, I.: Towards a Benchmark for Causal Business Process Reasoning with LLMs (2024)
  * [11] Grohs, M., Abb, L., Elsayed, N., Rehse, J.: Large Language Models Can Accomplish Business Process Management Tasks. In: BPM 2023 Workshops. vol. 492, pp. 453–465. Springer (2023)
  * [12] Guo, T., Chen, X., Wang, Y., et al., R.C.: Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024)
  * [13] Jessen, U., Sroka, M., Fahland, D.: Chit-Chat or Deep Talk: Prompt Engineering for Process Mining (2023)
  * [14] Koo, R., Lee, M., Raheja, V., Park, J.I., Kim, Z.M., Kang, D.: Benchmarking Cognitive Biases in Large Language Models as Evaluators (2023)
  * [15] Kourani, H., Berti, A., Schuster, D., van der Aalst, W.M.P.: Process Modeling With Large Language Models. In: EMMSAD 2024. Springer (2024)
  * [16] Kourani, H., Berti, A., Schuster, D., van der Aalst, W.M.P.: ProMoAI: Process Modeling with Generative AI. In: IJCAI 2024 Demo Track (2024)
  * [17] Kourani, H., van Zelst, S.J.: POWL: Partially Ordered Workflow Language. In: BPM 2023 Proceedings. vol. 14159, pp. 92–108. Springer (2023)
  * [18] Maggi, F.M.: Declarative Process Mining with the Declare Component of ProM. In: BPM Demos sessions 2013. vol. 1021. CEUR-WS.org (2013)
  * [19] Pohl, T., Berti, A., Qafari, M.S., van der Aalst, W.M.P.: A Collection of Simulated Event Logs for Fairness Assessment in Process Mining. In: BPM 2023 Resources. vol. 3469, pp. 87–91. CEUR-WS.org (2023)
  * [20] Raina, V., Liusie, A., Gales, M.J.F.: Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment (2024)
  * [21] Rebmann, A., Schmidt, F.D., Glavaš, G., van der Aa, H.: Evaluating the Ability of LLMs to Solve Semantics-Aware Process Mining Tasks. arXiv preprint arXiv:2407.02310 (2024)
  * [22] Singh, A.K., Devkota, S., Lamichhane, B., Dhakal, U., Dhakal, C.: The Confidence-Competence Gap in Large Language Models: A Cognitive Study (2023)
  * [23] Thakur, A.S., Choudhary, K., Ramayapally, V.S., Vaidyanathan, S., Hupkes, D.: Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges. arXiv preprint arXiv:2406.12624 (2024)
  * [24] Vidgof, M., Bachhofner, S., Mendling, J.: Large Language Models for Business Process Management: Opportunities and Challenges. In: BPM 2023 Forum. vol. 490, pp. 107–123. Springer (2023)
  * [25] Wang, P., Li, L., Chen, L., et al., D.Z.: Large Language Models are not Fair Evaluators (2023)
  * [26] Wornow, M., Narayan, A., et al., B.V.: Do Multimodal Foundation Models Understand Enterprise Workflows? A Benchmark for Business Process Management Tasks (2024)
  * [27] Zerbato, F., Soffer, P., Weber, B.: Initial Insights into Exploratory Process Mining Practices. In: BPM Forum 2021. vol. 427, pp. 145–161. Springer (2021)
  * [28] Zheng, L., Chiang, W., Sheng, Y., et al., S.Z.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In: NeurIPS 2023 (2023)


# PHOTONEUTRON REACTIONS ${}^{181}\rm{Ta}(\gamma,\textit{x}n;\textit{x}=1\div
8)^{181-\textit{x}}\rm{Ta}$ AT $E_{\rm{\gamma max}}$ = 80 $\div$ 95 MeV

A.N. Vodin,O.S. Deiev  I.S. Timchenko<EMAIL_ADDRESS>S.N. Olejnik,
M.I. Ayzatskiy, V.A. Kushnir, V.V. Mitrochenko, S.A. Perezhogin  National
Science Center ”Kharkov Institute of Physics and Technology”,
1 Akademicheskaya St., 61108 Kharkov, Ukraine

###### Abstract

The bremsstrahlung flux-averaged cross-sections $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle$ for the ${}^{181}\rm{Ta}(\gamma,\textit{x}n;\textit{x}=1\div
8)^{181-\textit{x}}\rm{Ta}$ photoneutron reactions have been measured at end-
point bremsstrahlung energies ranging from 80 MeV to 95 MeV. The measurements
were performed with the beam from the NSC KIPT electron linear accelerator
LUE-40 using the residual $\gamma$-activity method. The theoretical
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ values were computed using the
cross-sections $\sigma(E)$ from TALYS1.9 code. A comparison between the
measured cross-sections $\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ and the
theoretical values has demostrated their good agreement for the reactions with
escape of up to 6 neutrons, and substantial differences for the
$(\gamma,7\rm{n})$ and $(\gamma,8\rm{n})$ reactions. Isomeric ratios of the
average cross-sections $d(E_{\rm{\gamma max}})$ have been found for the
${}^{181}\rm{Ta}(\gamma,3n)^{178g,m}\rm{Ta}$ reaction products. The results
have been compared with the literature data and the computations based on
TALYS1.9 code.

###### keywords:

${}^{181}\rm{Ta}(\gamma,\textit{x}n;\textit{x}=1\div 8)^{181-\textit{x}}Ta$ ,
bremsstrahlung flux-averaged cross-section , isomeric ratio , end-point
bremsstrahlung energy of $80\div 95$ MeV , residual $\gamma$–activity method ,
TALYS1.9 , GEANT4.

###### PACS:

25.20.-x – Photonuclear reactions , 27.30.+t – $150\leqslant A\leqslant 189$

††journal:

##  1 Introduction

Most of the nuclear photodisintegration studies in a wide mass region have
been dedicated to investigation of the properties and nature of the giant
dipole resonance (GDR) with the use of bremsstrahlung/quasimonoenergetic
photon beams [1, 2]. At the same time, it is of interest to investigate
multiparticle photonucleon reactions on nuclei in the energy range above the
GDR and up to the pion production threshold ($E_{\rm{th}}\approx 145$ MeV).
This is linked with the change in the mechanism of photon interaction with
nuclei in this energy region. Thus, fundamental information can be obtained
about the competition between two mechanisms of nuclear photodisintegration,
namely, through the GDR excitation and the quasideuteron photoabsorption. The
observation of the mentioned reactions, which show relatively low cross-
section values, calls for intensive beams of incident $\gamma$-quanta that can
be provided by electron linear accelerators as the high-energy electrons pass
through the target-converter. Some results of investigations of multiparticle
nuclear reactions can be found, for example, in [3, 4].

The studies of photoneutron reactions on ${}^{181}\rm{Ta}$ are of interest,
because this nucleus is strongly deformed, with the quadrupole deformation
parameter being $\beta=0.26$ [5]. This strong deformation leads, for example,
to a complex energy-dependence structure of the
${}^{181}\rm{Ta}(\gamma,n)^{180}\rm{Ta}$ reaction cross-section, which shows
two maxima in the GDR region. Consequently, this may cause the change in the
form of the energy dependences of the reaction yield or the bremsstrahlung
flux-averaged cross-section. Besides, the ${}^{180\rm{m}}\rm{Ta}$ isotope
belongs to the so-called “by-passed” nuclei, the existence of which in nature
poses the problems of searching and investigating the processes of heavy
nuclei formation unrelated to neutron capture. The measurements of the
$(\gamma,\rm{n})$ reaction cross-section in the energy region above the GDR
may give a new insight into the properties of the high-energy part of the GDR
and improve the theoretical description of the nucleosynthesis rate
responsible for the $p$-production of ${}^{180\rm{m}}\rm{Ta}$ – the rarest
stable nucleus in nature.

The experiments on photodisintegration of ${}^{181}\rm{Ta}$ in the GDR region
were performed in a variety of works [6, 7, 8, 9, 10, 11, 12, 13] with the use
of bremsstrahlung/quasimonoenergetic photon beams. For example, the data on
photoabsorption cross-sections $\sigma(\gamma,\rm{abs})$ were obtained in ref.
[6] by the absorption method; the measurements in Livermore [7] and Saclay [8]
were carried out using quasimonochromatic photons; in refs. [9, 10, 11, 12,
13], to obtain the $\sigma(\gamma,s\rm{n})$ values, bremsstrahlung photon
beams were used. The cross-section $\sigma(\gamma,\rm{abs})$, evaluated as the
sum of cross-sections for all the investigated photonucleon reactions on
${}^{181}\rm{Ta}$, has been determined in ref. [14] from the analysis of
photonucleon reactions of different multiplicity.

The partial cross-sections for $(\gamma,\rm{n})$ and $(\gamma,2\rm{n})$
photoneutron reactions on ${}^{181}\rm{Ta}$ were obtained in ref. [7] by the
method of direct neutron registration in the GDR region. Besides, the studies
on the partial cross-sections for the ${}^{181}\rm{Ta}(\gamma,n)$,
$(\gamma,2\rm{n})$ and $(\gamma,3\rm{n})$ reactions, as well as for the
$(\gamma,4\rm{n})$ reaction up to an energy of 36 MeV, were carried out in
work [8]. The cross-section data obtained in those two works ([7] and [8])
diverge considerably from each other. For example, for the case of
$(\gamma,\rm{n})$ reaction the difference at the maximum of the cross-section
reaches $\sim\\!25\%$. This divergence may be due to the peculiarity of the
direct neutron registration method, in which it is difficult to separate
neutrons from the $(\gamma,\rm{n})$ and $(\gamma,2\rm{n})$ reactions.

The cross-sections for the photoneutron reactions on tantalum at energies
above the GDR were investigated in some experiments using the bremsstrahlung
beams. So, in ref. [15], at the end-point bremsstrahlung energy $E_{\rm{\gamma
max}}=55$ MeV the bremsstrahlung flux-averaged yields were obtained for the
reactions ${}^{181}\rm{Ta}(\gamma,3n)^{178g,m}\rm{Ta}$ and
${}^{181}\rm{Ta}(\gamma,p)^{180m}\rm{Hf}$. The data on relative yields were
found at $E_{\rm{\gamma max}}=67.7$ MeV in ref. [16] for the
${}^{181}\rm{Ta}(\gamma,\textit{x}n)^{181-\textit{x}}\rm{Ta}$ reactions with
escape of up to 6 neutrons, and also for the reactions with the charged
particle in the exit channel: ${}^{181}\rm{Ta}(\gamma,p)^{180m}\rm{Hf}$ and
${}^{181}\rm{Ta}(\gamma,pn)^{179m}\rm{Hf}$. In those studies, the comparison
was made between the experimental data and the theoretical computations with
the codes TALYS1.7 [17] and EMPIRE-3.2 [18], and also, with the use of the
combined model [19]. Note that the cross-sections for photoproton and
photoneutron reactions on the ${}^{181}\rm{Ta}$ nucleus with a larger
multiplicity $(\textit{x}>4)$ were not measured [20]. The isomeric ratio
values for the ${}^{181}\rm{Ta}(\gamma,3n)^{178g,m}\rm{Ta}$ reaction products
have been presented in papers [15, 16, 21, 22].

At the same time, it should be pointed out that the experiments on
bremsstrahlung beams considerably complicate the procedure of photonuclear
reaction cross-section determination. First of all, it is necessary to carry
out correct calculations of the $\gamma$-flux density that would correspond to
actual experimental conditions, using the modern computational techniques
(e.g., GEANT4). Aside from that, the said experiments involve the measurements
of integral characteristics of the reactions, and this calls for additional
mathematical treatment of the results. And yet, in spite of the encountered
difficulties, the bremsstrahlung beams remain an important tool in
photonuclear reaction studies.

The photonuclear reaction data obtained with the use of bremsstrahlung beams
by the residual $\gamma$-activity method can be represented in the forms of
the relative $Y(E_{\rm{\gamma max}})$ reaction yield [16], the bremsstrahlung
flux-averaged cross-section $\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ [23,
24] or the average cross-section per equivalent photon
$\langle{\sigma(E_{\rm{\gamma max}})_{\rm{Q}}}\rangle$ [25]. The analysis of
the relative $Y(E_{\rm{\gamma max}})$ data is restricted by their
representation as dimensionless normalized yields, generally normalized to the
$(\gamma,\rm{n})$ reaction yield. The cross-sections
$\langle{\sigma(E_{\rm{\gamma max}})_{\rm{Q}}}\rangle$ are convenient to use
in the case of the final nucleus-product formation in several reaction
channels [26]. For single-channel reactions, the representation of
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ permits a more detailed
investigation of the cross-section as a function of the bremsstrahlung energy,
since it is insensitive to the low-energy part of the bremsstrahlung spectrum.

The present work is concerned with the measurements of bremsstrahlung flux-
averaged cross-sections $\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ for
${}^{181}\rm{Ta}$ photoneutron reactions with escape of up to 8 neutrons, and
also, of isomeric ratios of average cross-sections $d(E_{\rm{\gamma max}})$
for the ${}^{181}\rm{Ta}(\gamma,3n)^{178g,m}\rm{Ta}$ reaction products at end-
point energies of bremsstrahlung spectra $E_{\rm{\gamma max}}=80\div 95$ MeV.

##  2 Experimental procedure

The experiments were performed on the bremsstrahlung beam from the NSC KIPT
electron linear accelerator LUE-40 [27, 28]. The experimental layout is shown
in Fig. 1. Electrons of initial energy $E_{e^{-}}$ were incident on the
target-converter, made from natural tantalum plate with cross dimensions of 20
mm×20 mm, and 1.05 mm in thickness. To remove the electrons from the
bremsstrahlung flux, a cylinder-shaped Al-absorber, 100 mm in diameter and 150
mm in length, was used.

Figure 1: Schematic block diagram of the experiment. The upper part shows the measuring room and the room for accelerator performance control. The lower part shows (from right to left) two sections of the accelerator LUE-40; Ta-converter; Al-absorder; bombardment chamber. Table 1:  Spectroscopic data [31] on the nuclei-products from the reactions ${}^{181}\rm{Ta}(\gamma,\textit{x}n;\textit{x}=1\div 8)^{181-\textit{x}}\rm{Ta}$ and the monitoring reaction ${}^{100}\rm{Mo}(\gamma,n)^{99}\rm{Mo}$ | Nuclear
---
reaction
$E_{\rm{th}}$, MeV |  |  $J^{\pi}$ of
---
nucleus-product
$T_{1/2}$ |  $E_{\gamma}$, keV |  $I_{\gamma}$, %
${}^{181}\rm{Ta}(\gamma,n)^{180g}\rm{Ta}$ | 7.58 | $1^{+}$ | 8.152 (6) h | 103.557 (7) | 0.81 (16)
${}^{181}\rm{Ta}(\gamma,n)^{180m}\rm{Ta}$ | 7.65 | $9^{-}$ |  $>1.2\cdot 10^{15}$ y |  |
${}^{181}\rm{Ta}(\gamma,2n)^{179}\rm{Ta}$ | 14.22 | $7/2^{+}$ | 1.82 (3) y | 54.611 | 13.6 (4)
|  |  |  | 55.790 | 23.7 (7)
${}^{181}\rm{Ta}(\gamma,3n)^{178g}\rm{Ta}$ | 22.21 | $1^{+}$ | 9.31 (3) min | 1350.68 (3) | 1.18 (3)
${}^{181}\rm{Ta}(\gamma,3n)^{178m}\rm{Ta}$ | 22.24 | $(7)^{-}$ | 2.36 (8) h | 426.383 (6) | 97.0 (13)
${}^{181}\rm{Ta}(\gamma,4n)^{177}\rm{Ta}$ | 29.01 | $7/2^{+}$ | 56.56 (6) h | 112.9498 (5) | 7.2 (8)
${}^{181}\rm{Ta}(\gamma,5n)^{176}\rm{Ta}$ | 37.44 | $(1)^{-}$ | 8.09 (5) h | 1159.28 (9) | 25 (?)
${}^{181}\rm{Ta}(\gamma,6n)^{175}\rm{Ta}$ | 44.46 | $7/2^{+}$ | 10.5 (2) h | 348.5 (5) | 12.0 (6)
${}^{181}\rm{Ta}(\gamma,7n)^{174}\rm{Ta}$ | 53.21 | $3^{+}$ | 1.05 (3) h | 1205.92 (4) | 4.9 (4)
${}^{181}\rm{Ta}(\gamma,8n)^{173}\rm{Ta}$ | 60.63 | $5/2^{-}$ | 3.14 (13) h | 172.2 (1) | 18 (?)
${}^{100}\rm{Mo}(\gamma,n)^{99}\rm{Mo}$ | 8.29 | $1/2^{+}$ | 65.94 (1) h | 739.50 (2) | 12.13 (12)

In the experiments, the method for measuring the residual $\gamma$-activity of
the irradiated sample has been used, which provided a simultaneous data
acquisition from different channels of photonuclear reactions. This method is
well known and has been described in a variety of papers dedicated to the
studies of multiparticle photonuclear reactions, e.g., on the 93Nb [23, 29].

The bremsstrahlung spectra were calculated by means of the open certified
programme code GEANT4 [30] with due regard to the real geometry of the
experiment, where spatial and energy distributions of the electron beam were
taken into account. In addition, the bremsstrahlung flux was estimated from
the yield of the ${}^{100}\rm{Mo}(\gamma,n)^{99}\rm{Mo}$ reaction. For this
purpose, the natural molybdenum target-witness, placed close by the target
under study, was exposed to radiation.

The natural tantalum/molybdenum targets of diameter 8 mm were arranged behind
the Al-absorber on the electron beam axis. For transporting the targets to the
place of irradiation and back for induced activity registration the pneumatic
tube transfer system was used. Experimentally, this enabled us to obtain the
data on the ${}^{181}\rm{Ta}(\gamma,3n)^{178g}\rm{Ta}$ reaction with a
relatively short half-life period (see Table 1).

In the experiments, six pairs of natural tantalum/ molybdenum samples were
exposed to radiation at different end-point bremsstrahlung energies
$E_{\rm{\gamma max}}$ in the range from 80 to 95 MeV. The masses of Ta target
and Mo target were, respectively, $\sim 43$ mg and $\sim 60$ mg. The time of
irradiation $t_{\rm{irr}}$ and the time of residual $\gamma$-activity spectrum
measurement $t_{\rm{meas}}$ were both 30 min. To exemplify, Fig. 2 shows a
fragment of the $\gamma$-spectrum from the tantalum target.

Figure 2: Fragment of $\gamma$-radiation spectrum from the tantalum target of
mass 43.665 mg after exposure to the bremsstrahlung flux. $E_{\rm{\gamma
max}}=85.6$ MeV, $t_{\rm{irr}}=30$ min.

For $\gamma$-radiation registration, a semiconductor HPGe detector (Canberra
GC-2018) was used with resolutions of 0.8 keV and 1.8 keV (FWHM) for the
energies $E_{\gamma}=122$ keV and 1332 keV, respectively. The absolute
registration efficiency of the detector was calibrated with a standard set of
$\gamma$-ray sources 22Na, 60Co, 133Ba, 137Cs, 152Eu and ${}^{241}\\!\rm{Am}$.

To investigate the reactions of interest, the residual activity
$\gamma$-spectrum of the irradiated target was analyzed, and the activities
$\triangle A$ (i.e., the number of counts of $\gamma$-quanta in the full
absorption peak) were determined for the $\gamma$-lines corresponding to the
isotopes produced. Relying on the data from ref. [31], Table 1 gives the
parameters of both the reactions under study and the monitoring reaction,
viz., the energy $E_{\gamma}$ and intensity $I_{\gamma}$ of the $\gamma$-lines
in use.

Based on the obtained activities $\triangle A$, we determined the experimental
values of the bremsstrahlung flux-averaged cross-sections by the expression:

$\begin{split}\langle{\sigma(E_{\rm{\gamma max}})}\rangle=\\\
\frac{\lambda\triangle A{\rm{\Phi}}^{-1}(E_{\rm{\gamma
max}})}{N_{x}I_{\gamma}\ \varepsilon(1-\exp(-\lambda
t_{\rm{irr}}))\exp(-\lambda t_{\rm{cool}})(1-\exp(-\lambda
t_{\rm{meas}}))},\end{split}$ (1)

where ${\rm{\Phi}}(E_{\rm{\gamma
max}})={\int\limits_{E_{\rm{th}}}^{E_{\rm{\gamma max}}}W(E,E_{\rm{\gamma
max}})dE}$ is the integrated bremsstrahlung flux $W(E,E_{\rm{\gamma max}})$ in
the energy range from the reaction threshold $E_{\rm{th}}$ of the
corresponding reaction up to the maximum energy of ${\gamma}$-quanta
$E_{\rm{\gamma max}}$, $N_{x}$ is the number of target atoms, $I_{\gamma}$ is
the absolute intensity of the analyzed $\gamma$-quanta, $\varepsilon$ is the
absolute detection efficiency for the analyzed $\gamma$-quanta energy,
$\lambda$ is the decay constant ($\rm{ln}2/\textit{T}_{1/2}$); $t_{\rm{irr}}$,
$t_{\rm{cool}}$ and $t_{\rm{meas}}$ are the irradiation time, cooling time and
measurement time, respectively.

The cross-sections $\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ obtained in
this way were compared with the theoretical values, which were based on the
reaction cross-sections $\sigma(E)$ from the TALYS1.9 code with the default
options: Constant temperature + Fermi gas model [32]. For this purpose, the
values of the $\sigma(E)$ were weighted over the bremsstrahlung flux
$W(E,E_{\rm{\gamma max}})$ from the reaction threshold $E_{\rm{th}}$ up to
$E_{\rm{\gamma max}}$. As a result of this procedure, the theretical
bremsstrahlung flux-averaged cross-sections $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle$ were obtained according to the formula:

$\langle{\sigma(E_{\rm{\gamma max}})}\rangle={\rm{\Phi}}^{-1}(E_{\rm{\gamma
max}})\int\limits_{E_{\rm{th}}}^{E_{\rm{\gamma max}}}\sigma(E)\cdot
W(E,E_{\rm{\gamma max}})dE.$ (2)

A more detailed description of all the procedures necessary for determining
experimental and theoretical values of $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle$ can be found in refs. [23, 29, 33].

If the nucleus-product has an isomeric state, the total average cross-section
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{tot}}$ for the reaction
under study (hereafter referred to as $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle$) is calculated as the sum of $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{g}}$ and $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{m}}$ average cross-sections for population of the ground
state and the isomeric state, respectively.

When calculating the values of the bremsstrahlung flux-averaged cross-
sections, it was assumed that all the radioisotopes were produced only as a
result of photonuclear reactions on ${}^{181}\rm{Ta}$, because the
${}^{180\rm{m}}\rm{Ta}$ isomer content in the natural tantalum mix is
negligibly small (0.012%). The self-absorption of $\gamma$-radiation of the
reaction products in the target was calculted with the GEANT4 code, and was
taken into account in calculations by eq. 1.

The bremsstrahlung flux monitoring by the
${}^{100}\rm{Mo}(\gamma,n)^{99}\rm{Mo}$ reaction yield was performed by
comparing the experimentally obtained average cross-section values with the
computation data. To determine the experimental $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{exp}}$ values by eq. 1, we have used the activity
$\triangle A$ for the $\gamma$-line of energy $E_{\gamma}=739.50$ keV and
absolute intensity $I_{\gamma}=12.13\%$ (see Table 1). The average cross-
section $\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{th}}$ values were
computed by eq. 2 with the cross-sections $\sigma(E)$ from the TALYS1.9 code.
The obtained normalization factor $k=\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{th}}/\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{exp}}$ represents the deviation of the GEANT4-computed
bremsstrahlung flux from the real $\gamma$-flux incident on the target. The
thus found $k$ values, which varied within $1.08\div 1.15$, were used for
normalization of the cross-sections for photoneutron reactions on the
${}^{181}\rm{Ta}$ nucleus. The details of the monitoring procedure can be
found in ref. [23, 29].

The Ta-converter and Al-absorber, used in the experiment, generates neutrons
that can cause the reaction ${}^{100}\rm{Mo}(n,2n)^{99}\rm{Mo}$. Calculations
were made of the energy neutrons spectrum and the fraction of neutrons with
energies above the threshold of this reaction. The contribution of the
${}^{100}\rm{Mo}(n,2n)^{99}\rm{Mo}$ reaction to the value of the induced
activity of the 99Mo nucleus has been estimated and it has been shown that
this contribution is negligible compared to the contribution of
${}^{100}\rm{Mo}(\gamma,n)^{99}\rm{Mo}$.

The uncertainty of measurements of the average cross-sections
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle$, $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{g}}$ and $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{m}}$ was determined as a squared sum of statistical and
systematic errors. The statistical error in the observed $\gamma$-activity is
mainly associated with the statistics calculation for the total-absorption
peak of the corresponding $\gamma$-line, which varies from 1 to 10%. The error
varies depending on the $\gamma$-line intensity and the background conditions
of the spectrum measurement. The $\gamma$-line intensity depends on the
registration efficiency, the half-life period and the absolute intensity
$I_{\gamma}$. The background is mainly contributed by the Compton scattering
of high-energy quanta.

The systematic errors stem from the uncertainties in:

  1. 1.

irradiation time – $0.25\div 0.5\%$;

  2. 2.

electron current – 0.5%;

  3. 3.

$\gamma$-radiation detection efficiency $\sim 2\div 3\%$, mainly due to the
measuring error of $\gamma$-radiation sources;

  4. 4.

half-life period $T_{1/2}$ of reaction products and the absolute intensity of
the analyzed $\gamma$-quanta $I_{\gamma}$ – $1.0\div 20\%$ (see Table 1);

  5. 5.

normalization of the experimental data to the yield of the monitoring reaction
${}^{100}\rm{Mo}(\gamma,n)^{99}\rm{Mo}$ – 2.5%;

  6. 6.

the GEANT4 computational error for the bremsstrahlung spectra $\sim 1.5\%$.

So, the statistical and systematic errors represent the variables and are
different for different ${}^{181}\rm{Ta}(\gamma,\textit{x}n;\textit{x}=1\div
8)^{181-\textit{x}}\rm{Ta}$ reactions. The overall uncertainty of the
experimental data is given in Figs. 3(a-h) and 5(a).

##  3 RESULTS AND DISCUSSION

###  3.1 The total bremsstrahlung flux-averaged cross-sections
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ for the reactions
${}^{181}\rm{Ta}(\gamma,\textit{x}n;\textit{x}=1\div
8)^{181-\textit{x}}\rm{Ta}$

The obtained experimental data on the total average cross-sections
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ for the
${}^{181}\rm{Ta}(\gamma,\textit{x}n;\textit{x}=1\div
8)^{181-\textit{x}}\rm{Ta}$ reactions in the end-point energy range of
bremsstrahlung $\gamma$-quanta $E_{\rm{\gamma max}}=80\div 95$ MeV are
presented in Fig. 3. The results of calculations by eq. 2 with the use of
cross-sections $\sigma(E)$ computed with the TALYS 1.9 code are shown in the
same figure.

Figure 3: Total bremsstrahlung flux-averaged cross-sections
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ for the
${}^{181}\rm{Ta}(\gamma,\textit{x}n;\textit{x}=1\div
8)^{181-\textit{x}}\rm{Ta}$ reactions. Solid circles show the present
experimental data. The calculated average cross-sections are shown by: the
curves – with the use of $\sigma(E)$ from the TALYS1.9 code; solid diamonds –
with the use of the partial cross-section data from ref. [7]; empty diamonds –
partial cross-sections from ref. [8].

In the ${}^{181}\rm{Ta}(\gamma,n)^{180}\rm{Ta}$ reaction case, one can measure
only $\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{g}}$ for occupation of
the ground state of the ${}^{180\rm{g}}\rm{Ta}$ nucleus. The total average
cross-section $\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ for this reaction
has been calculated with the use of the theoretical value of
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{m}}$ for occupation of the
isomeric state of the nucleus-product. According to the TALYS1.9 computations,
the ratio of the cross-sections $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{m}}$ and $\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ has
made 0.083 for our experimental conditions. This value is in satisfactory
agreement with the value 0.07 from ref. [16]. In the
${}^{181}\rm{Ta}(\gamma,3n)^{178}\rm{Ta}$ case, Fig. 3(c) shows the values of
the total cross-section $\langle{\sigma(E_{\rm{\gamma max}})}\rangle$, while
its components $\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{g}}$ and
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{m}}$ are presented in Fig.
5(a).

As is evident from Fig. 3, the obtained $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle$ values are in satisfactory agreement (within the experimental
uncertainties) with the calculations for the
${}^{181}\rm{Ta}(\gamma,\textit{x}n)^{181-\textit{x}}\rm{Ta}$ reactions with
the escape of 1 to 6 neutrons. With the increase in the number of escaping
neutrons up to 7 and 8 in the exit channel of the reactions, a significant
difference is observed between the experiment and the calculation. Note also
the tendency to satisfactory agreement between the experimental and calculated
data on the total cross-sections for photoneutron reactions on
${}^{181}\rm{Ta}$, in which the nuclei-products are produced with positive
parity $\pi$ in the ground state.

So far, no data can be found in the literature on the experimental average
cross-sections for tantalum photodisintegration. However, using eq. 2 and the
data on partial cross-sections $\sigma(E)$ for photonuclear reactions on the
${}^{181}\rm{Ta}$ nucleus with escape of up to 3 neutrons (refs. [7, 8]), it
appears possible to calculate the values of total average cross-sections
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ for the bremsstrahlung flux
consistent with the conditions of the given experiment. The calculated in this
way values for all three reactions exhibit significant deviations from both
our experimental results and the calculations performed with the use of the
TALYS1.9 code (see Figs. 3(a-c)).

The analysis of the experimental data from refs. [7, 8] shows their difference
for the $(\gamma,\rm{n})$ reaction, viz., at the maximum of the cross-section
the difference reaches $\sim~{}25\%$. This may be attributed to complexity of
neutron separation from the pairs of the $(\gamma,\rm{n})$ and
$(\gamma,2\rm{n})$ reactions in the method of direct neutron registration.
Another reason of the difference, according to ref. [16], may be probably due
to incorrect accounting of the bremsstrahlung part of the quasimonochromatic
photon spectrum. On this basis it can be assumed that the sum of total average
cross-sections $\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ for the reactions
$(\gamma,\rm{n})$, $(\gamma,2\rm{n})$ and $(\gamma,3\rm{n})$, which were
obtained from the data of ref. [8], should be close to a similar sum obtained
from our data. In the energy range $E_{\rm{\gamma max}}=80\div 95$ MeV, the
sums of three cross-sections
$(\gamma,\rm{n})+(\gamma,2\rm{n})+(\gamma,3\rm{n})$, calculated with the data
from ref. [8], smoothly decrease from $130\pm 6$ mb down to $123\pm 6$ mb with
energy increase. With the data found in the present work, the sums of the
three total average cross-sections vary from $132\pm 16$ mb to $122\pm 15$ mb.
It is apparent that there is close agreement with the values of the summed
total cross-sections (see Fig. 4). The similar summed cross-section computed
with $\sigma(E)$ from the TALYS1.9 code is in agreement with the results based
on the experimental data.

Figure 4: The sum of three total average cross-sections for the photoneutron
reactions on ${}^{181}\rm{Ta}$:
$(\gamma,\rm{n})$+$(\gamma,2\rm{n})$+$(\gamma,3\rm{n})$. Solid circles – the
present data, diamonds – with the use of partial cross-sections from ref. [8];
the curve shows the computation with $\sigma(E)$ from the TALYS1.9 code.

###  3.2 Isomeric ratio $d(E_{\rm{\gamma max}})$ of average cross-sections for
the products of the ${}^{181}\rm{Ta}(\gamma,3n)^{178g,m}\rm{Ta}$ reactions

The values of average cross-sections have been obtained experimentally for
occupation of both the ground $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{g}}$ and isomeric $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{m}}$ states of the nucleus-product in the
${}^{181}\rm{Ta}(\gamma,3n)$ reaction (see Fig. 5(a)). Comparison with the
calculation by eq. 2 for the ${}^{181}\rm{Ta}(\gamma,3n)^{178g}\rm{Ta}$
reaction shows that the experimental $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{g}}$ values slightly exceed the calculated data. On the
average, this is within the magnitude of the experimental accuracy. At the
same time, for $\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{m}}$ the
experiment gives the value substantially lower than the theoretical evaluation
throughout the energy range under study. However, as is seen from Fig. 3(c),
the total average cross-section values agree within the experimental accuracy
with the calculation.

On the basis of the obtained $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{m}}$ and $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{g}}$ values for the
${}^{181}\rm{Ta}(\gamma,3n)^{178g,m}\rm{Ta}$ reactions, we have determined the
isomeric ratio values for average cross-sections $d(E_{\rm{\gamma
max}})=\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{m}}/\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{g}}$
(see Fig. 5(b)). Within the limits of the experimental uncertainty, the
$d(E_{\rm{\gamma max}})$ values are constant and equal to $0.37\pm 0.02$ over
the investigated energy range $E_{\rm{\gamma max}}=80\div 95$ MeV. This
estimated value points to a significant (by a factor of $\sim 2.5$)
suppression of the reaction with occupation of the isomeric state of the
nucleus ${}^{178\rm{m}}\rm{Ta}$ ($J^{\pi}=(7)^{–}$) relative to the ground-
state occupation of ${}^{178\rm{g}}\rm{Ta}$ ($J^{\pi}=(1)^{+}$).

For comparison with theory, the $d(E_{\rm{\gamma max}})$ values were computed
with the use of the TALYS1.9 code (see Fig. 5(b)). The computed
$d(E_{\rm{\gamma max}})$ value has appeared to be nearly two-fold higher than
the experimental one. Most likely, the main reason for this difference, as it
follows from Fig. 5(a), lies in the TALYS1.9 overestimation of the
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{m}}$ cross-section values
for occupation of the isomeric state of the nucleus-product.

The earlier obtained values of the isomeric ratio $d(E_{\rm{\gamma max}})$ of
the ${}^{181}\rm{Ta}(\gamma,3n)^{178g,m}\rm{Ta}$ reaction yields can be found
in refs. [15, 16, 21, 22]. Note that in ref. [21] the $d(E_{\rm{\gamma max}})$
value was obtained in the range $E_{\rm{\gamma max}}=24\div 32$ MeV, and
according to the authors’ conclusions, the value $d(E_{\rm{\gamma
max}})=0.33\pm 0.07$ is constant in the energy region under study (in Fig.
5(b) the $d(E_{\rm{\gamma max}})$ was given only at $E_{\rm{\gamma max}}=32$
MeV). From the data of paper [16], we have estimated the $d(E_{\rm{\gamma
max}})$ value to be $0.28\pm 0.08$ at $E_{\rm{\gamma max}}=67.7$ MeV. The
experimental $d(E_{\rm{\gamma max}})$ values from refs. [15, 16, 21, 22] are
in agreement with our present data for the range $E_{\rm{\gamma max}}=80\div
95$ MeV, and also, similarly to the present results, are systematically below
the calculated values.

Figure 5(b) shows that the calculated value of the isomeric ratio
$d(E_{\rm{\gamma max}})$ increases with a decrease in the end-point
bremsstrahlung energies $E_{\rm{\gamma max}}$, however the available
experimental data do not support this dynamics. The experimental
$d(E_{\rm{\gamma max}})$ values in the energy range $E_{\rm{\gamma
max}}=24\div 95$ MeV are grouped around the constant value equal to $0.34\pm
0.02$.

Figure 5:  Reactions ${}^{181}\rm{Ta}(\gamma,3n)^{178g,m}\rm{Ta}$.
(a) The bremsstrahlung flux-averaged cross-sections for occupation of the
ground $\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{g}}$ and isomeric
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{m}}$ states of the nucleus-
product ${}^{178}\rm{Ta}$. Full squares represent
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{g}}$, empty squares –
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle_{\rm{m}}$ (experimental error is
smaller than the symbol size)), the lines show the corresponding computations
with TALYS1.9 code.
(b) Isomeric ratios $d(E_{\rm{\gamma max}})$ of average cross-sections. Full
diamonds show the present results, star – ref. [15], empty diamond – ref.
[16], square – ref. [21], triangle – ref. [22]. Solid line represents the
calculation, dashed line – experimental data fitting.

##  4 Conclusions

The present work has been concerned with investigation of the
${}^{181}\rm{Ta}$ photoneutron reactions with escape of up to 8 neutrons at
end-point bremsstrahlung energies $E_{\rm{\gamma max}}=80\div 95$ MeV with the
use of the residual $\gamma$–activity method. The obtained total
bremsstrahlung flux-averaged cross-section $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle$ values have been compared with the computations using
$\sigma(E)$ from the TALYS1.9 code with the default options. A satisfactory
agreement has been demonstrated for the
${}^{181}\rm{Ta}(\gamma,\textit{x}n)^{181-\textit{x}}\rm{Ta}$ reactions with
escape of 1 to 6 neutrons. With increase in the number of neutrons up to 7 and
8 in the exit channel of the reactions, a significant difference is observed
between the experiment and the calculation. The tendency has been traced to
satisfactory agreement between the experimental and calculated data on the
bremsstrahlung flux-averaged cross-sections for photoneutron reactions on
${}^{181}\rm{Ta}$, in which the nuclei-products are produced with positive
parity $\pi$ in the ground state.

The total bremsstrahlung flux-averaged cross-sections
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle$ measured in the present work for
the reactions $(\gamma,\rm{n})$, $(\gamma,2\rm{n})$ and $(\gamma,3\rm{n})$ are
different from the data obtained on the basis of partial cross-sections from
refs. [7, 8]. However, the value resulting from the summation of our total
bremsstrahlung flux-averaged cross-sections
($(\gamma,\rm{n})$+$(\gamma,2\rm{n})$+$(\gamma,3\rm{n})$) coincides with a
similar sum from ref. [8].

Isomeric ratios of average cross-sections $d(E_{\rm{\gamma max}})$ for the
${}^{181}\rm{Ta}(\gamma,3n)^{178g,m}\rm{Ta}$ reaction products have been
obtained in the energy range $E_{\rm{\gamma max}}=80\div 95$ MeV. The measured
values point to a significant (by a factor of $\sim~{}2.5$) suppression of the
occupation of the isomeric state of the nucleus ${}^{178\rm{m}}\rm{Ta}$ in
relation to the ground-state occupation of ${}^{178\rm{g}}\rm{Ta}$. The
comparison between the experimental results and the TALYS1.9 code-based
computed values has shown a two-fold difference between the data. The present
$d(E_{\rm{\gamma max}})$ values are in agreement with the literature data [15,
16, 21, 22]. All the available experimental $d(E_{\rm{\gamma max}})$ values in
the range $E_{\rm{\gamma max}}=24\div 95$ MeV are grouped around the constant
value $0.34\pm 0.02$.

The experimental data on the bremsstrahlung flux-averaged cross-sections
$\langle{\sigma(E_{\rm{\gamma max}})}\rangle$, $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{g}}$, $\langle{\sigma(E_{\rm{\gamma
max}})}\rangle_{\rm{m}}$ and the isomeric ratios of the average cross-sections
$d(E_{\rm{\gamma max}})$ at $E_{\rm{\gamma max}}$ energies ranging from 80 to
95 MeV have been obtained for the first time.

## References

  * [1] A.V. Varlamov, V.V. Varlamov, D.S. Rudenko, M.E. Stepanov. Atlas of Giant Dipole Resonances. Parameters and Graphs of Photonuclear Reaction Cross Sections. INDC(NDS)-394, IAEA NDS, Vienna, Austria, 1999.
  * [2] E.G. Fuller, H. Gerstenberg. Photonuclear Data - Abstracts Sheets 1955 – 1982. NBSIR 83-2742. U.S.A. National Bureau of Standards, 1986.
  * [3] O.M. Vodin, O.A. Bezshyyko, L.O. Golinka-Bezshyyko, et al., Probl. Atom. Sci. Tech.
  * [4] O.A. Bezshyyko, O.M. Vodin, L.O. Golinka-Bezshyyko, et al., Probl. Atom. Sci. Tech. 6, 144 (2019).
  * [5] Data Center of Photonuclear Experiments, http://cdfe.sinp.msu.ru/.
  * [6] G.M. Gurevich, L.E. Lazareva, V.M. Mazur, et al., Nucl. Phys. A 351, 257 (1981), doi.org/10.1016/0375-9474(81)90443-7.
  * [7] R.L. Bramblett, J.T. Caldwell, G.F. Auchampaugh, S.C. Fultz, Phys. Rev. 129, 2723 (1963), doi.org/10.1103/PhysRev.129.2723.
  * [8] R. Bergere, H. Beil, A. Veyssiere, Nucl. Phys. A 121, 463 (1968), doi.org/10.1016/0375-9474(68)90433-8.
  * [9] E.G. Fuller, M.S. Weiss, Phys. Rev. 112, 560 (1958), doi.org/10.1103/PhysRev.112.560.
  * [10] O.V. Bogdankevich, B.I. Goryachev, V.A. Zapevalov. Sov. Phys. JETP 42, 1502 (1962).
  * [11] G.P. Antropov, I.E. Mitrofanov, B.S. Russkikh. Bull. Acad. Sci. USSR. Phys. 31, 336 (1967).
  * [12] B.S. Ishkhanov, I.M. Kapitonov, E.V. Lazutin, et al., JETP Letters 54, 80 (1969).
  * [13] S.N. Belyaev, V.P. Sinichkin. in Proceedings of the 8th International Meeting on Beam dynamics and optimization. Saratov Univ. Publ., Saratov, 2002, 81.
  * [14] V.V. Varlamov, M.E. Stepanov, V.V. Chesnokov, Bull. Acad. Sci. Phys. 67, 656 (2003).
  * [15] V. Zheltonozhsky, M. Zheltonozhskaya, A. Savrasov, A. Chernyaev, Book of Abstracts ”LXX Int. Conf. ”NUCLEUS – 2020” online part, 12-17 October 2020, Saint Petersburg, 64.
  * [16] B.S. Ishkhanov, V.N. Orlin, S. Yu, Phys. At. Nucl. 75, 253 (2012).
  * [17] A.J. Koning, S. Hilaire, M.C. Duijvestijn, in Proceedings of the International Conference on Nuclear Data for Science and Technology, Nice, France, Apr. 22-27, 2007, Ed. by O. Bersillon, F. Gunsing, E. Bauge, et al. (EDP Sciences, 2008), 211.
  * [18] EMPIRE - Nuclear Reaction Model Code, https://www-nds.iaea.org/empire/.
  * [19] B.S. Ishkhanov, V.N. Orlin. Phys. At. Nucl. 74, 21 (2012).
  * [20] Experimental Nuclear Reaction Data (EXFOR). https://www-nds.iaea.org/exfor/.
  * [21] J.H. Carver, W. Turchinetz. Research School of Physical Sciences, Australian National University, Canberra, September 1957.
  * [22] H. Bartsch, K. Huber, U. Kneissl, H. Krieger. Nucl. Phys. A 256, 243 (1976).
  * [23] A.N. Vodin, O.S. Deiev, I.S. Timchenko, et al., Probl. Atom. Sci. Tech. 3, 148 (2020).
  * [24] H. Naik, G.N. Kim, R. Schwengner, et al., Nucl. Phys A 916, 168 (2013), dx.doi.org/10.1016/j.nuclphysa.2013.08.003.
  * [25] V.Di Napoli, A.M. Lacerenza, F. Salvetti, et al., Lett. Nuovo Cimento 1, 835 (1971).
  * [26] A.N. Vodin, O.S. Deiev, I.S. Timchenko, S.N. Olejnik, arXiv:2012.14475v1.
  * [27] A.N. Dovbnya, M.I. Aizatsky, V.N. Boriskin, et al., Probl. Atom. Sci. Tech. 2, 11 (2006).
  * [28] M.I. Aizatsky, V.I. Beloglasov, V.N. Boriskin, et al., Probl. Atom. Sci. Tech. 3, 60 (2014).
  * [29] A. N. Vodin, O. S. Deiev, V. Yu. Korda, et al., arXiv:2101.08614.
  * [30] Geant4 — a simulation toolkit, Nucl. Instrum. Methods Phys. A 506, 250 (2003), http://geant4.web.cern.ch/geant4/.
  * [31] S.Y.F. Chu, L.P. Ekstrom, R.B. Firestone, The Lund/LBNL, Nuclear Data Search, Version 2.0, February 1999, WWW Table of Radioactive Isotopes, http://nucleardata.nuclear.lu.se/toi/.
  * [32] A. Koning, D. Rochman, Nucl. Data Sheets 113 2841 (2012), TALYS – based evaluated nuclear data library, https://tendl.web.psi.ch/tendl 2019/tendl2019.html.
  * [33] O.M. Vodin, O.S. Deiev, S.M. Olejnik, Probl. Atom. Sci. Tech. 6, 122 (2019).