To appear in IEEE Transactions on Visualization and Computer Graphics.
xx.xxxx/TVCG.201x.xxxxxxx Research application/design study Heungseok Park is
with NAVER Labs. E-mail<EMAIL_ADDRESS>Aeree Cho is with
Georgia Tech. E-mail<EMAIL_ADDRESS>Hyojun Jeon, Hayoung Lee, Youngil
Yang, Sungjae Lee, and Heungsub Lee are with NAVER Cloud Corporation. E-mail:
hyojun.jeon, hayoung.lee, youngil.yang, sung-jae.lee,
<EMAIL_ADDRESS>Jaegul Choo is with KAIST. E-mail:
<EMAIL_ADDRESS>*Equally-contributing authors.

# HPC2lusterScape: Increasing Transparency and Efficiency of Shared High-
Performance Computing Clusters for Large-scale AI Models

Heungseok Park*0000-0001-9789-8547  Aeree Cho  Hyojun Jeon  Hayoung Lee
Youngil Yang  Sungjae Lee
Heungsub Lee  and Jaegul Choo

###### Abstract

The emergence of large-scale AI models, like GPT-4, has significantly impacted
academia and industry, driving the demand for high-performance computing (HPC)
to accelerate workloads. To address this, we present HPC2lusterScape, a
visualization system that enhances the efficiency and transparency of shared
HPC clusters for large-scale AI models. HPC2lusterScape provides a
comprehensive overview of system-level (e.g., partitions, hosts, and workload
status) and application-level (e.g., identification of experiments and
researchers) information, allowing HPC operators and machine learning
researchers to monitor resource utilization and identify issues through
customizable violation rules. The system includes diagnostic tools to
investigate workload imbalances and synchronization bottlenecks in large-scale
distributed deep learning experiments. Deployed in industrial-scale HPC
clusters, HPC2lusterScape incorporates user feedback and meets specific
requirements. This paper outlines the challenges and prerequisites for
efficient HPC operation, introduces the interactive visualization system, and
highlights its contributions in addressing pain points and optimizing resource
utilization in shared HPC clusters.

###### keywords:

Visualization for AI, High-performance computing, large-scale distributed
learning, unit visualizations

With HPC2lusterScape, users can monitor the real-time resource status of the
HPC cluster through the (A) Cluster Overview. GPU resources are represented as
square units and can be grouped by various data attributes to provide a
monitoring view that is tailored to individual needs. In this example, we have
grouped GPUs by DL workloads, authors, and projects. Users can define their
own conditions for the (B) Violation Rule, which tracks the statistics of
system metrics for the workloads, determines if the rule has been violated in
the past as well as at the current time, and highlights the corresponding
workload. Clicking on a specific workload enables the user to enter the (C)
Diagnostics View, which provides a node, GPU, and metric level analysis
environment that allows users to drill down into that workload.

Introduction The growing influence of large AI models, such as GPT, has led to
a surge in demand for high-performance computing (HPC) to accelerate deep
learning workloads in both industry and academia. However, effectively
utilizing these high-cost resources in HPC clusters with complex hardware and
network configurations requires collaborative efforts from two distinct
groups: cluster operators and ML researchers. Cluster operators play a vital
role in efficiently managing and operating HPC clusters. The complexity of
operating such clusters lies in the need for dynamic changes in operation
policies to minimize resource fragmentation and maximize training efficiency.
By carefully allocating resources and addressing challenges like GPU
communication bottlenecks, operators strive to optimize the utilization of HPC
clusters. On the other hand, ML researchers must select resources of
appropriate scale for their models and data sizes, monitor system metrics, and
optimize hyperparameters and source code to make efficient use of the
allocated GPUs. This is especially crucial for large-scale distributed
learning, which requires numerous GPUs and extensive training times, making
full resource utilization essential.

However, the dynamic nature of scheduling policies, coupled with varying
demands, makes wait time prediction difficult, leading to increased
uncertainty in resource allocation and frustration for both operators and
researchers. To address these challenges, providing transparent visibility and
flexible exploration of the cluster usage and scheduling status are essential
for both user groups. HPC operators need to understand how the configured
policies affect resource allocation and whether they contribute to or prevent
resource fragmentation. With a clear view of the scheduling state of the
cluster, operators can make informed decisions when introducing policies.
Similarly, by understanding the state of scheduling and potential bottlenecks,
ML researchers can plan DL workload submissions and reduce unnecessary delays.
Previous studies [35, 15, 5, 26, 31] primarily focused on using static views
to observe cluster resources as physical units, such as nodes or racks,
lacking the flexibility required for real-time exploration and decision-
making. To effectively address the challenges of scheduling in shared HPC
clusters, it is advantageous to provide a visualization system that offers
dynamic and flexible views, empowering both HPC operators and ML researchers
to monitor, analyze, and make informed decisions based on the real-time status
of the cluster and scheduling policies.

Additionally, large-scale distributed learning inherently faces instability
due to uncontrollable factors, including hardware and software failures, as
well as communication overheads between different nodes [34]. Even minor
issues can significantly impact the overall cluster efficiency given the
extensive resource utilization. Collaboration with skilled operators is
necessary for efficient troubleshooting and problem resolution, as ML
researchers typically lack expertise in HPC cluster hardware. Previous studies
have provided environments for detailed analysis of training-related metrics,
system resource metrics, and GPU operation execution times [34, 14, 42, 33,
28]. However, effectively identifying which aspects to analyze or where
potential problems may arise when dealing with large-scale experiments
involving numerous GPUs and extensive training times poses a challenge.

In this work, we propose a visualization system, HPC2lusterScape, designed to
assist two user groups in observing shared HPC clusters for efficiency
enhancement. HPC2lusterScape leverages unit visualization [8] to represent
various information of DL experiments and the shared HPC clusters, allowing
both user groups can explore and monitor cluster resources from multiple
perspectives along with their desired layout. HPC2lusterScape provides: 1) a
monitoring tool (HPC2lusterScape: Increasing Transparency and Efficiency of
Shared High-Performance Computing Clusters for Large-scale AI Models(A))
offering an integrated view of the application-level information of the ML
framework (e.g., identification of experiments and researchers) and the
system-level information of the HPC cluster (e.g., cluster partition, hosts,
machine types, workload status), facilitating users’ understanding of resource
usage and scheduling status; 2) a detection tool (HPC2lusterScape: Increasing
Transparency and Efficiency of Shared High-Performance Computing Clusters for
Large-scale AI Models(B)) that accumulates a diverse set of user-defined
utility-violation rules with visual cues, helping users to spot potential
issues of DL workloads; 3) a diagnostic tool (HPC2lusterScape: Increasing
Transparency and Efficiency of Shared High-Performance Computing Clusters for
Large-scale AI Models(C)) enabling detailed analysis of problematic
experiments, assisting users in identifying performance issues such as
workload imbalance or synchronization bottlenecks in large-scale distributed
learning. To demonstrate the utility and adaptability of the proposed system,
we deploy HPC2lusterScape on industrial-scale HPC clusters and conduct an
evaluation through in-depth interviews with professional users, focusing on
how the visualization helps two different user groups observe the HPC
clusters.

The main contributions of this work are as follows:

  * •

A summary of the pain points for operators and ML researchers of shared HPC
clusters, factors hindering HPC efficiency, and requirements for an
observational system to improve the efficiency and transparency of shared HPC
clusters.

  * •

An interactive observation system capable of flexibly representing both
system-level and application-level information from HPC clusters and ML
frameworks respectively.

  * •

Detecting and diagnostic tools for identifying potential problems in large-
scale distributed deep learning training.

  * •

A system deployed and utilized on industrial-scale HPC clusters, demonstrating
its adaptability and real-world applicability.

##  1 Related Work

###  1.1 Visualization for monitoring HPC clusters

There are numerous tools for monitoring HPC systems, such as Nagios [2] and
Ganglia [23]. However, these tools are primarily used to monitor the
performance metrics of individual experiments or nodes, it’s difficult to get
a comprehensive view of resource usage to gain insight into scheduling status
or to identify anomalous machines or experiments among a large number of
resources. We reviewed studies that attempted to visualize the overall
resource usage of an HPC system.

MELA [35] incorporates the node layout view, representing the physical compute
cards and racks of the HPC system to identify the distribution of hardware log
data. VALSE [15] employs a similar view to MELA, but with scalability through
zoom interaction. The spatial view uses semantic zooming and level-of-detail
rendering, ranging from compute cards to midplanes. HyperView [5] is a visual
analytics framework that monitors the health status of HPC systems. It
represents performance metrics of nodes as radar charts and visualizes the
overall distribution of nodes in a cluster as a heatmap. HyperJobViz [26] is
an analytical tool designed to visualize resource allocations in data centers
for jobs (i.e., workloads), users, and resource usage statistics. Waiting and
running times of jobs are represented using spiral graphs, while performance
metrics of nodes are displayed as radar charts. MTSAD [31] is a tool
specifically designed for analyzing high-dimensional time series data and
demonstrates its application in detecting anomalies in HPC system performance
metrics. It represents performance metrics of hundreds of nodes as radar
charts, and those charts that were periodically recorded are placed in a
timeline graph.

Some of the aforementioned studies provide an intuitive visualization of a
physical HPC cluster as a selector, along with common visualization components
for in-depth analysis of selected nodes [35, 15, 5]. Other studies visualize
performance metrics on a per-node or per-job basis and allow for the
comparison of multiple visualizations [26, 31]. The former enables users to
understand the physical structure of HPC systems easily, while the latter
facilitates quick detection of abnormal nodes or jobs. However, since both
approaches analyze HPC systems based on fixed units such as nodes or jobs,
they lack the ability to integrally visualize application-level information
and system-level information for HPC resource usage analysis.

HPC2lusterScape provides full observability of the physical HPC cluster using
unit visualization, and comprehensively presents system performance metrics,
scheduling information, and application-level data using a flexible layout,
color, and filter system. Furthermore, it employs user-defined violation rules
to detect anomalies in system metrics of during DL workload, highlighting
experiments and nodes where anomalies are detected and providing diagnostic
tools for debugging.

###  1.2 Visualization for analyzing computational performance of deep
learning

Numerous analytical tools have been developed to aid in profiling the
performance of DL workloads, often used in conjunction with existing
visualization systems. DeepProf[14] is a performance analysis tool that
summarizes GPU traces of TensorFlow models and generates performance analysis
graphs and reports. TensorFlow (TF) Profiler[13] is a standalone tool
integrated with TensorBoard[12], designed to monitor the performance of
TensorFlow models by capturing operation execution times and their
dependencies. PyTorch Profiler[33] provides similar functionalities for
PyTorch models, allowing users to collect performance data such as input
tensor shapes and stack traces. The collected data can be analyzed through the
TensorBoard plugin or the Chrome Trace Viewer. On the other hand, such
profiling tools have been proposed to provide deeper analysis related to the
execution of CUDA kernels [27, 1, 28, 29]. These existing tools contribute to
the detailed analysis of DL workload; however, they focus on low-level data
analysis, leaving the entire process up to human interpretation. Furthermore,
these tools lack consideration for users who may not possess expertise in
hardware.

Meanwhile, research efforts have been developed to offer practitioners greater
insight when analyzing DL workload performance by examining both low-level and
high-level information. SageMaker Debugger[34] offers a comprehensive
environment that enables practitioners to efficiently utilize resources and
troubleshoot problems by discussing the most common performance issues that
may occur in industry-scale DL training workflows and suggesting relevant
metrics to track for each performance issue. SageMaker Debugger provided a
framework to automatically collect performance-related data during training.
This includes high-level metrics at the DL framework level (e.g., data
loading, training, evaluation) and low-level system metrics (e.g., GPU
utilization, memory usage). Furthermore, with various visualization techniques
for system metrics [17] such as line graphs, flame graphs, and heatmaps, users
can comprehensively analyze hardware issues occurring during training by
cross-correlating the desired metrics. Skyline[42] is an interactive in-editor
tool that supports performance profiling, debugging, and visualization of DL
training. It provides an environment to diagnose the performance of training
by collecting and analyzing performance-related metrics such as training
throughput, operation run times, and memory allocation. Furthermore, it
proposes a predictive model for DL training performance based on batch size
hyperparameter and offers an interface to directly modify the batch size in
the in-editor environment.

The aforementioned studies contribute by suggesting metrics that impact DL
training performance and providing detailed analysis at the DL framework and
system resource levels. They offer valuable insights into DL training for ML
researchers without extensive hardware knowledge. However, in large-scale
distributed learning with numerous computations and occupied GPUs, training
times are prolonged. This leads to a vast amount of data to analyze, making it
difficult to visualize and analyze specific metrics like CUDA operations and
memory usage individually. Moreover, system metric patterns can vary based on
the learning’s synchronization strategy (e.g., data/model parallelism)[7],
which is challenging to formalize. As a result, previous approaches like
training performance prediction models have limitations.

HPC2lusterScape offers high-level performance information by visualizing the
system resource metric distribution and pattern of each GPU used in
distributed learning through small multiples. This assists users in
efficiently identifying abnormal GPUs with different metric distributions.
Furthermore, HPC2lusterScape proposes a methodology to quickly pinpoint GPUs
or nodes with varying distributions or patterns, even in cases where training
involves a vast number of GPUs, by clustering them based on the time series
metric data of each GPU.

##  2 Identification of Factors Hindering HPC Efficiency

To gain a thorough understanding of shared HPC cluster operations and usage,
we conducted interviews with expert users from NAVER111 NAVER Co., Ltd. is the
largest web search engine & portal in South Korea. Cloud company. Our goals
were to identify efficiency obstacles, explore challenges encountered during
operation and usage, and assess the potential for improved support through
visualization methods.

###  2.1 Background: shared HPC clusters for large-scale deep learning

Operating HPC clusters entails significant expenses in terms of acquisition,
operation, maintenance, and management. Consequently, it is common for
organizations or companies to share HPC clusters to optimize resource
utilization. Operators of shared HPC clusters are responsible for devising
scheduling systems and policies for resource allocation (i.e., DL workload
submissions) and managing resources, including detecting server failures.

Resource allocation for DL workloads in AI development-oriented HPC clusters
is particularly challenging. Shared HPC cluster operations often employ a mix
of scheduling strategies tailored to the task nature to minimize resource
waste[38, 25, 39, 10]. Unlike the FCFS (First Come, First Serve) scheduling,
allocation order in the pool is dynamically determined by the scheduling
policy. In the case of large-scale distributed learning, it is even more
complicated because resource allocation must take into account physical
factors to minimize bottlenecks in the I/O of various sections such as CPU-
GPU, GPU-GPU, and Host-Host. While virtualization and containerization
technologies are designed to make it easier to manage multiple workloads on
the same hardware, resource allocation for DL workloads with diverse sizes and
execution times exacerbates the problem of cluster resource fragmentation.

In summary, scheduling DL workloads of varying sizes and execution times,
minimizing communication bottlenecks between machines and avoiding cluster
resource fragmentation is a major challenge in shared HPC operations and
requires highly complex scheduling policies. These policies can change
dynamically based on demand, and often require heuristics for managing waiting
pools, especially in overdemand situations where spare resources are scarce.

Figure 1: Demonstration of (A) layout editing user interface and (B) example
monitoring views configured by various purposes.

###  2.2 Factors impeding HPC cluster efficiency

To understand how shared HPC clusters are commonly operated and used, as well
as the obstacles to efficiency and operational challenges, we conducted semi-
interviews with three HPC operators and five ML researchers from the same
cluster. Participants shared their views on cluster efficiency and pain points
during operation and usage. Based on these interviews, we identified key
factors that negatively impact cluster efficiency and pain points that can be
addressed through visualization.

  1. F1.

Uncertainty and low predictability in cluster operations

During the interview process, we noted that a majority of researchers
expressed frustration stemming from the uncertainty related to wait times for
resource allocation. The lack of a comprehensive overview of the waiting pool
and available resources presented difficulties in ascertaining the optimal
resource size (i.e., the number of GPUs) or machine type (e.g., NVIDIA V100 or
A100) for minimizing wait times when submitting DL workloads. Researchers
suspected unfair resource allocation and abuse among teams for longer-than-
expected wait times, and required transparency in resource operations.

Operators acknowledged the challenges inherent in predicting and providing
accurate allocation time due to intricate factors, such as workload size,
machine type, and the physical location of machines. This frustration was
further compounded by users’ repeated inquiries regarding wait time
information. Additionally, they highlighted the difficulty in detecting
unexpected bottlenecks within resource scheduling and in understanding the
impact of modifications to scheduling policies.

  2. F2.

Egocentric resource monitoring

We noticed that researchers prioritize the success of their own experiments
over improving the overall efficiency of the cluster. For instance, even when
a workload’s resource utilization is lower than the cluster average,
researchers often spend more time adjusting hyperparameters or refining the
model structure to enhance model performance, placing less importance on
optimizing resource efficiency. While optimizing hyperparameters or model
structure can contribute to efficiency by reducing unnecessary computations,
researchers typically prioritize monitoring model performance and training
stability, such as checking for memory overflow.

Operators were concerned about ensuring that GPUs allocated to workloads were
not being utilized (i.e., idle or inefficient workload), as it is important to
maximize the overall efficiency of HPC clusters. Due to the large number of
GPUs and DL workloads, they wanted to proactively identify workloads with
utilization below baseline or suspected hardware failures, and they were
considering how to differentiate resource allocation based on past utilization
metrics per user.

  3. F3.

Unstable large-scale training

Researchers who focus on large-scale training prioritize resource efficiency
to accelerate computation during distributed workloads. However, they face
challenges such as I/O bottlenecks and software/hardware failures on specific
GPUs, which cause workloads to stall or synchronization to be delayed,
resulting in poor training efficiency[22, 32, 16, 34]. Despite existing
strategies and studies on addressing failures in distributed learning[40, 37],
researchers often rely on operators for assistance due to limited hardware
expertise, particularly regarding GPU communication topology. Therefore, there
is a demand for tools that can help diagnose problems within DL workloads.

During the interview process, key pain points affecting shared HPC cluster
efficiency were identified: 1) Operators and researchers both face challenges
related to unpredictable resource allocation, due to complex policies and non-
deterministic workload durations. 2) Cluster operators focus on global
resource efficiency, while researchers prioritize model performance and
stability, causing differing tolerances for inefficiencies. 3) In large-scale
distributed learning, which consumes significant resources, both user groups
seek efficiency and require detailed analysis of synchronization bottlenecks
or hardware failures.

Previously proposed tools, such as monitoring of resource/scheduling status of
HPC clusters[23, 2, 35, 5] and analysis tools for DL training performance[34,
42], can individually be seen as partially supporting the factors hindering
HPC efficiency. However, there are limitations in adequately supporting both
operators and researchers, who are key user groups of HPC clusters. This
drives our motivation to develop a more comprehensive design that offers
transparent visibility into cluster and scheduling status, aiming to address
the identified pain points.

##  3 Design Goals

In this section, we highlight and formalize the primary pain points discussed
earlier in Section 2.2 with key design goals that HPC2lusterScape aims to
support. We label the three goals as G1 - G3.

G1. Transparency of cluster & scheduling status: Our goal is to provide a
transparent and comprehensive overview of resource operation status that can
be tailored to the needs of both user groups. To achieve this, we aim to
provide a unified visualization of system-level data and application-level
data. By visualizing both levels of data, researchers can monitor the
availability of GPUs and waiting pools, empowering them to plan their workload
submissions effectively. Additionally, cluster operators can gain real-time
insights into how the complex scheduling policy behaves. They can monitor
scheduling bottlenecks and make necessary adjustments to policies based on
demand. This level of transparency fosters informed decisions and helps
optimize resource utilization for both researchers and operators.

G2. Trace system metrics & highlight abnormality: We aim to identify DL
workloads where specific metrics deviate from predefined thresholds by
leveraging GPU machine metrics during workload execution. This approach allows
researchers and operators to promptly detect anomalies. Researchers can
identify unstable workloads by examining significant drops or peaks in
stability-related metrics, such as GPU utilization, memory usage, and
temperature, both in the past and in real-time. By analyzing these metrics,
researchers can pinpoint potential issues and take corrective actions.
Similarly, operators can monitor metrics related to utilization and hardware
failures over time to detect any abnormalities or underutilization. This helps
operators promptly identify and address issues related to resource allocation
or hardware failures, ensuring optimal system performance and efficiency.

G3. Diagnosis of DL workload status at scale: Our goal is to provide an
interface that allows users to efficiently and effectively analyze system
metrics for problematic experiments. With this interface, users can identify
and diagnose issues such as synchronization bottlenecks or workload/data
imbalances by visualizing system metric distributions for each GPU. By
comparing the distinct metric distributions among GPUs, users can detect
computational or memory imbalances in large-scale distributed learning.
Additionally, the interface provides analysis capabilities based on
timestamps, enabling users to identify suspicious ranges and abnormal points
in system metrics throughout the execution time of the workload. These
features empower users to gain insights into the performance of DL workloads
at scale and facilitate prompt identification and resolution of issues.

##  4 HPCClusterScape Design

In this section, we describe interfaces of HPC2lusterScape’s, a visualization
system that allows users to explore the status of shared HPC clusters and
analyze the performance of large-scale AI models.

###  4.1 HPC cluster overview

Figure 2: Representation of utility violation rules for allocated resources at
various granularities. (A) Users can tailor violation rules to suit their
specific DL workloads or objectives. (B) Within the cluster overview,
violation rules are represented by indicators and bar plots, determined by the
zoom level of the canvas. (C) In the diagnostic view for a selected DL
workload, each rule is highlighted at the node and GPU levels, allowing users
to examine the targeted units based on their individually configured
conditions. The metric-level analysis further enhances the examination of
selected units by tightly integrating with the small multiples of GPU units.

####  4.1.1 Multiple unit & group layers with layout operations

To offer multiple analytical perspectives to meet individual monitoring
objectives while simultaneously presenting an overview of the cluster resource
status (G1), we utilize unit visualization. Unit visualization preserves a
one-to-one correspondence between rows in a data table and their
representations, while still allowing for the mapping of visual attributes
(e.g., color or position) onto individual units[30, 8]. This enables the
representation of both overall patterns and individual outliers. Furthermore,
the visual attributes of units can be animated when transitioning from one
view to another, aiding users in perceiving patterns in the data[8]. These
benefits are particularly important given the design goal of HPC2lusterScape,
which aims to provide multiple views of a single data set that changes in
real-time.

The base layer of HPC2lusterScape is the canvas area (HPC2lusterScape:
Increasing Transparency and Efficiency of Shared High-Performance Computing
Clusters for Large-scale AI Models(A)) where units and groups are represented.
A unit represents one GPU and is encoded as a square. Units are grouped based
on certain data attributes and placed inside square containers, which are
referred to as groups. To mitigate the typical disadvantage of visual clutter
in unit visualization[9], HPC2lusterScape groups units into multiple layers
without overlapping and adjusts the margin, padding, and text size differently
for each layer to ensure visibility. Moreover, as shown in Fig. 1(A), the
layout editing UI is designed to reveal the hierarchical structure of the
group and allows for the addition and deletion of group layers via a drag-and-
drop interaction, facilitating users’ understanding of the layout
functionality.

To construct monitoring views tailored to specific objectives, it is crucial
to have the ability to flexibly arrange units and groups within each layer.
For instance, a simple view grouping GPUs by projects to compare resource
occupancy and a multi-layered view utilizing various hierarchical layers to
monitor scheduling bottlenecks would be challenging to observe with the same
layout. Users should be able to define different views by arranging units and
groups within each layer. To achieve this, we redefined and implemented the
layout operations proposed by Atom [30], dividing them into three categories:
Pack, Fill, and List. The "Pack" operator places units and groups in a fixed
aspect ratio by minimizing wasted space using a bin-packing algorithm. The
"Fill" operator packs units and groups along the X or Y axis according to a
fixed width or height. The "List" operator lists groups according to user-
defined sorting criteria and wraps them when they are off-screen. In addition,
each operation type can apply two sizes: "Uniform", which has the same size
among siblings, and "Count", which depends on the size of the child elements.
Users can adjust the layout operations for each layer in the Settings pop-up
UI of Fig. 1(A). Fig. 1(B1) is a layout configured to view a bar chart of how
many resources projects are using per partition. Groups of projects are filled
along the x-axis with the same size in a FillX-uniform layout. Projects are
grouped once again by cluster partition, and these groups are listed along the
X-axis in the ListX-Count layout. Along with resource occupancy by partition
and project, operators can monitor which projects are utilizing resources well
or poorly through color encoding.

####  4.1.2 Unit color system

Since each data attributes have different scales and distributions in their
values, the min, max, and binning of the color interpolation can be adjusted
to reveal meaningful information about the selected attributes. Therefore,
HPC2lusterScape provides color setting interface (HPC2lusterScape: Increasing
Transparency and Efficiency of Shared High-Performance Computing Clusters for
Large-scale AI Models(A1)) that includes customizable color scheme and scale
options such as logarithmic, quantile, and quantize. Users can also interact
with a remap checkbox to adjust colors to the domain of the data filtered,
which allows them to continue to leverage color for discrimination.

####  4.1.3 Unit & group interaction

Large-scale clusters often consist of thousands of GPU and CPU devices. To
enable simultaneous overview and detailed exploration, HPC2lusterScape
supports zooming and panning mouse interaction on the canvas area
(HPC2lusterScape: Increasing Transparency and Efficiency of Shared High-
Performance Computing Clusters for Large-scale AI Models(A)). It also
facilitates direct interaction with units, groups, and labels to provide on-
demand details on individual unit data as well as aggregated group data. When
users click the mouse, an information card UI(HPC2lusterScape: Increasing
Transparency and Efficiency of Shared High-Performance Computing Clusters for
Large-scale AI Models(A2)) is displayed, which includes details about the unit
or group, a summary of system metrics, and action buttons for further analysis
of the workload. Additionally, the filter tool (HPC2lusterScape: Increasing
Transparency and Efficiency of Shared High-Performance Computing Clusters for
Large-scale AI Models(A3)) allows users to create multiple filter conditions.
For categorical data attributes, a search tool is provided for convenience.

Figure 3: Diagnostics view for training status with a distributed training
using 64 GPUs. (A) This view displays training status summaries, including
metadata and primary metrics, while small multiples plots illustrate metric
distributions for each GPU, grouped by node. The color of the small multiples
plots represents the cluster’s ID, determined by the distance between
distributions, highlighting imbalances in the selected metric among the GPUs.
Additionally, red circles beside each plot indicate activated violation rules,
configured by users. (B) Line plots depict changes in system metrics over
time. (C) Bivariate analysis enables users to examine correlations between
selected metrics and detect outlier data points. Each view is interconnected,
and when outlier points in (C) are selected using the drag interaction, the
timestamp of the data points are activated on the x-axis of each line plot in
(B), emphasizing the abnormal range.

###  4.2 Highlighting abnormalities based on violation rules

By employing the unit visualization technique, the status of individual
resources, as well as the resource allocation status of HPC, can be
effectively represented using color encoding. This is particularly beneficial
for large-scale distributed learning, as it enables the rapid identification
of abnormal GPUs (see the 8 GPUs workload Clarey group’s in Fig. 2(B)).
However, a snapshot of the current resource status does not reveal critical
abnormalities that may have occurred in the past, potentially allowing
erroneous experiments to go undetected and overlooked (partially addressing
G2).

Deep learning training does not consistently utilize occupied resources;
instead, usage varies depending on the training strategy and source code. For
example, with a large batch size and extended epoch duration, GPU utilization
may remain high for an extended period before briefly dropping during the
validation phase, while in other cases, utilization may fluctuate frequently.
Moreover, distributed learning, which necessitates communication among
numerous GPUs and nodes, requires steps for data broadcasting and computation
result synchronization, the pattern of system usage can be much more intricate
and difficult to formalize compared to single-GPU experiments [34].
Consequently, it is crucial not only to obtain an instantaneous snapshot of
resource status but also to understand the extent of resource utilization from
the beginning of the experiment to the present. Detecting abnormal and
unstable DL workloads (G2) requires an understanding of whether there has been
critical wrong usage in the past (e.g., low GPU utilization, low GPU power
supply). Furthermore, since the pattern of system metrics varies across
experiments, anomaly detection based solely on built-in rules defined by
cluster operators does not guarantee accurate results. As such, users should
be able to establish their own metrics and thresholds for tracking, based on
their training strategies.

HPC2lusterScape offers an interface for detecting abnormalities by tracking
the history of key system metrics from the start of training to the current
moment. As a real-time monitoring tool, HPC2lusterScape leverages statistics
of selected system metrics (e.g., min, max, mean, and percentile over time)
instead of relying on computationally intensive machine learning or DNN
approaches [31]. The UI at the top of HPC2lusterScape allows users to define
various rules for detecting abnormalities (Fig. 2(A)). Each rule consists of a
set of conditions, with each condition comprising four fields: a system
metric, a statistic type for the metric, a relational operator, and a
comparison value. Multiple rules can be added simultaneously and assessed
independently. Additionally, multiple conditions can be combined to form a
single rule for detecting complex cases such as where GPU utilization is high
and power usage is low simultaneously.

Due to the substantial number of GPU servers operating in large-scale HPC
clusters and the numerous DL workloads allocating GPU resources,
HPC2lusterScape presents information granularity differently based on the
level of analysis, as demonstrated in Fig. 2. Initially, to promptly identify
whether a group corresponds to a violation rule in the HPC cluster overview
(Fig. 2(B)), an indicator is encoded in the parent group if any unit object
conforms to the rule, thereby highlighting the group. If the canvas zoom level
is high, the met rules are stacked as a bar plot at the top of each group,
displaying the number of violation rules and the proportion of units
corresponding to each rule [41]. Hovering over each bar highlights the units
associated with the rule and displays the rule’s properties at the top of the
screen, enabling rapid scanning for abnormal GPUs.

Subsequently, the diagnostics view (see Section 4.3) provides a more detailed
level of information for in-depth analysis (Fig. 2(C)). It exhibits node-level
and GPU-level information, arranging nodes and GPUs in a physical order using
small multiples and employing indicators to illustrate the correspondence
between GPU units and rules. The indicators are positioned according to the
order of the violation rule item at the top of the screen. When a rule is
satisfied, the indicator is mapped to its corresponding position in the
complete list of rules and their order. If no rules are in effect, the
indicator remains empty, indicating the absence of active rules.

Following the analysis at the node-level and GPU-level, metric-level analysis
is offered to assess the status changes of each GPU and node over time. Based
on the insights gained from the analysis, users can iteratively modify
existing violation rule settings or add new rules.

###  4.3 Diagnostics view for training status at scale

HPC2lusterScape offers an interface enabling users to perform in-depth
analysis of the system status of resources utilized by DL experiments Fig. 3.
The cluster overview, which presents the status of all resources, identifies
workloads suspected of violating rules at the cluster level, while the
detailed analysis view provides a node-, GPU-, and metric-level analysis
environment for thorough examination of the experiment (G3), as illustrated in
Fig. 2. Large-scale distributed workloads may involve numerous GPUs; thus, the
details are provided on demand for fine-grained analysis. Techniques such as
focus$+$context or pan&zoom can be employed to naturally link the overview and
detailed analysis view [41, 4, 3]; however, due to space constraints, a
detailed analysis of a large number of GPUs is presented in a separate view.
The diagnostic tool comprises three main components: (A) system metric
overview, (B) system metrics timeline, and (C) bivariate analysis view.

####  4.3.1 System metric overview

Identifying problematic GPUs from individual performance metrics of numerous
GPUs used in large-scale distributed training is challenging. Previous studies
have visualized each GPU metric assigned to the experiment in a simple line
graph [34, 17]. However, this approach has limitations, as it requires manual
inspection and interpretation of extensive datasets. To address this
challenge, we have developed an interface that leverages small multiples
techniques to analyze the distribution and patterns of metrics for each GPU.
These small multiples are arranged based on the physical layout of the GPU
servers (GPU index), global rank, and local rank (process ID) in distributed
DL workloads. This interface streamlines the debugging and detection process
of abnormal GPUs or nodes, facilitating more efficient analysis.

The small multiples in the interface represent time series data for system
metrics collected from each GPU, and users have the flexibility to switch
between different plot types to visualize metric distributions or changes over
time. For example, by examining the distribution of GPU utilization, users can
quickly identify workload imbalances, a critical issue in distributed learning
where computation may be biased towards specific GPUs or nodes [22].
Similarly, by analyzing the distribution of GPU memory usage, users can detect
data volume imbalances handled by each GPU. These imbalances are often
challenging to identify at the source code level[22, 16, 32], but become
easily discernible through the representation of system metrics as histograms
for each GPU.

However, When the scale of workload is immense, numerous small multiples plots
can make it challenging to discern metric patterns or distribution for
individual GPUs. To efficiently reveal distinct GPUs, clustering techniques
are employed. If the plot type for small multiples is a histogram, clustering
is computed by measuring the distance based on Jensen-Shannon divergence[11]
between the distributions on each GPU. If it is a timeline, clustering is
computed using metrics such as Euclidean distance or correlation coefficient.
Although various clustering methods can be considered, the well-known
agglomerative clustering algorithm was used to exclude randomness during
clustering recalculation [24]. The clustering results are mapped to unique
colors within each small multiples plot (Fig. 3(A)), allowing for rapid
identification of distinct distributions or time-series patterns, revealing
potential imbalances or abnormal patterns not evident through visualization.

When users click on a small multiple, the metric timeline and bivariate
analysis views are linked (Fig. 3(B,C)), enabling a detailed analysis of the
selected GPU’s metric changes over time and the correlations between metrics
of the GPU. The implementation of multiple selections allows users to compare
and analyze metrics among various GPUs and nodes. Through the effective use of
small multiples and analysis of time series data, it provides valuable
insights into the patterns of GPU metrics, aiding users in diagnosing critical
issues in large-scale distributed learning scenarios, such as workload and
data imbalances.

####  4.3.2 System metrics timeline

HPC2lusterScape offers a line plot analysis environment for in-depth
examination of system metric changes over time for the GPUs utilized in DL
workloads. Analyzing the changes in a metric over time is an effective way to
examine the performance of a machine learning model or identify problematic
areas, and is a common approach across many analytics tools [34, 12, 17]. The
system metrics timeline also serves as a means to filter data for specific
segments users wish to analyze. For DL training, which involves repetition of
data preprocessing, training, and evaluation, we provide a zoom & filter
interface to enable detailed analysis of abnormalities within specific
sections (coarse-to-fine). The filtered range is linked with the small
multiples in Fig. 3(A) to identify imbalances or outliers in the zoomed
window. Moreover, large-scale distributed learning typically involves lengthy
training times, making it challenging to visualize all metrics simultaneously.
HPC2lusterScape is implemented to sample data points and render them quickly
for long training periods, and to display denser data as the time span
expands.

####  4.3.3 Bivariate analysis between system metrics

HPC2lusterScape provides an interface for analyzing correlations between
system metrics and identifying outlier data points that significantly deviate
from the distribution of time series data across GPUs. Data points for two
selected metrics are represented in a scatterplot, with outlier scores
measured and highlighted for each point, as shown in Fig. 3(C). The scoring
method is based on the Mahalanobis distance [6, 21], and users can adjust the
chi-squared significance level to control the outlier cutoff according to the
features or distribution of the metrics. Highlighted outliers can be selected
using the mouse through the brush interaction, and the timestamp of the
corresponding data point is activated on the x-axis of the system metric
timeline, as shown in Fig. 3(B). This feature enables users to identify when
an abnormal data point was recorded during the training process.

##  5 Deployment to Industrial-scale Environment

###  5.1 Deployment details

We deployed HPC2lusterScape on the machine learning platform by NAVER Cloud
called NSML [18, 36]. The HPC clusters supported by this platform are mainly
composed of NVIDIA A100 and V100 Tensor Core GPU servers. Each cluster
consists of hundreds of nodes, and A100 device-type clusters are specifically
designed for distributed deep learning by minimizing communication bottlenecks
between GPU servers via InfiniBand and NVLink configurations.

On the other hand, each cluster is used by numerous researchers and multiple
projects, and there is a need to train models of various sizes based on their
specific domains. As the number of researchers surpasses the cluster size, the
clusters are often in high demand. To prevent cluster fragmentation among GPU
servers and maximize efficiency, operators employ various scheduling policies,
such as pre-demand surveys, workload size formalization, and gang scheduling.

###  5.2 Implementation details

The server side of HPC2lusterScape comprises three modules: one for gathering
GPU server status and system metrics data, one for gathering application layer
data (including scheduling data), and one for aggregating the collected data
and delivering it to the REST API. We developed a TSDB using
VictoriaMetrics222Open source Time Series Database:
https://victoriametrics.com to collect data on GPU server status and system
metrics via the dcgm-exporter333NVIDIA GPU metrics exporter:
https://github.com/NVIDIA/dcgm-exporter provided by NVIDIA.

The client-side implementation of HPC2lusterScape was developed using
React444React JavaScript framework: https://reactjs.org. The HPC cluster
overview was implemented with Deck.gl555Deck.gl WebGL-powered framework:
https://deck.gl, a WebGL framework to draw thousands of GPU units, and other
modules for layout operations, filtering, and color encoding were implemented
with JavaScript and D3666D3 JavaScript library: https://d3js.org.
HPC2lusterScape ingests Atom’s visualization grammar which defines as JSON
objects. Visual interfaces in the diagnostics view were implemented using D3
and SVG/Canvas elements. The clustering for imbalance analysis was created
from SciPy777SciPy Python library: https://scipy.org’s hierarchical clustering
implementation.

##  6 Case Studies

To demonstrate how HPC2lusterScape can help users understand the scheduling
status of shared HPC clusters and analyze issues in distributed training, we
conducted case studies in collaboration with a HPC cluster operator (denoted
as P1) and n ML researcher who utilizes the ML platform where HPC2lusterScape
is deployed (denoted as P2).

###  6.1 Adjusting scheduling policies

On the platform the cluster is running on, users should select a type of
predefined computing resource (i.e., resource type) when submitting DL
workloads, which is determined by the machine type and the number of GPUs. For
example, the A100-8 type uses a full host which has eight A100 GPUs. To
prevent resource fragmentation, the cluster is divided into partitions to
which only workloads using 8 or fewer GPUs can be assigned, and partitions to
which larger workloads can be assigned.

P1 is responsible for investigating real-time resource demand to adjust
resource supply per cluster partition or resolve scheduling bottlenecks. To do
this, he configured the cluster overview of HPC2lusterScape that shows the
number of workloads waiting to be allocated for each resource type (Fig.
1(B2)). P1 noticed a steady increase in the number of waiting workloads of the
A100-8 resource type, and discovered through color encoding that these
workloads generally have longer waiting times than those submitted with other
resource types.

To further investigate the cause, P1 changed the layout to check the resource
allocation status of the physical cluster partitions (Fig. 1(B3)). Looking at
the partition for A100 small workloads ($\leq$ 8 GPUs), P1 observed that
workloads using 1, 2, and 4 GPUs were spread across all hosts, preventing the
allocation of A100-8 type workloads, which required all GPUs on a single host.
To tackle this problem, P1 configured the MostAllocated888 Kubernetes
scheduling plugin: https://kubernetes.io/docs/reference/scheduling scheduling
policy, giving priority to hosts with more resources allocated. P1
continuously monitored host occupancy status in real-time, ensuring that the
modified policy was working as intended, mitigating resource fragmentation.

###  6.2 Detecting workload imbalance of large-scale distributed training

ML researcher P2 is developing a Large Language Model (LLM) and train models
with 64 A100 GPUs provided by the HPC cluster. P2 organized a distributed
learning experiment with each node applying data parallel and the GPUs within
the nodes applying tensor parallel techniques to train the model. P2
established two violation rules based on GPU utilization metrics to assess not
only the experiment’s current state but also the even and efficient
utilization of GPUs during distributed learning (R1: GPU utilization 95th
percentile $<$ 80%, R2: GPU utilization median $<$ 50%). Upon noticing his
experiment was marked as a rule violation in the cluster overview, P2
proceeded to the diagnostics view.

Within the diagnostics view’s metric overview, small multiples and clustering
results visualized the utilization distribution per GPU, revealing that the
master GPU of all nodes belonged to the same cluster group and differed from
other GPUs’ utilization distributions, as shown in Fig. 3(A). Distributions
for non-master GPUs were left-skewed, and violation rules set by P2 were
activated for all of them, implying that a workload imbalance occurred during
the training process.

To determine where the imbalance occurred, P2 employed a bivariate analysis
view, selecting data points using brush interactions that were distant in the
distribution across metrics (Fig. 3(C1)). The selection resulted in outlier
data points being activated on the system metrics timeline, and P2 zoomed in
on that region (Fig. 3(B1)). P2 noticed that the GPUs in the range had zero
utilization values, except for the master GPU on all nodes. P2 suspected that
there was a bottleneck in the data aggregation process across the GPUs. The
updated histogram of small multiples retained the same clustering, and when
the plot type was changed to timeline, P2 saw the same clustering result,
confirming the imbalance occurred at that point. P2 then did more GPU
debugging and found that the workload was biased towards the master GPU during
collective communication process.

##  7 Use Cases

In this section, we describe various use cases of HPC2lusterScape by the
representative users and discuss its utility through in-depth interviews.

###  7.1 Participants and study protocol

To understand how HPC2lusterScape supports the pain points of different user
groups in shared HPC clusters described in Section 2.2, We conducted
interviews with HPC operators and ML researchers who operate and use shared
HPC clusters where HPC2lusterScape is deployed.

User A is an ML researcher focused on developing speech recognition models.
Their task is exploring optimized models by tuning numerous experiments with
different hyperparameter sets.

User B is an ML researcher working on developing language models, with large-
scale distributed learning experiments.

User C is the operator of shared HPC clusters, focusing on scheduling policy
development to prevent cluster fragmentation.

Each of the three participants had a 60-minute session. The initial 15 minutes
focused on their primary tasks and typical workflows in monitoring using
HPC2lusterScape. The following 45 minutes involved reviewing their interaction
sequences and customized layout combinations. The interviewer aims to
understand their perspectives on observing experiments and clusters and asks
questions related to their intentions behind the interactions while thinking
aloud.

###  7.2 Key findings and insights

We summarized the main findings and feedback from the interviews into the
following criteria, by highlighting how the HPC2lusterScape helps to monitor
shared HPC clusters and analyze their training status, achieving our design
goals.
Understanding resource usage and scheduling bottlenecks: Both User A and B
reported using the monitoring view to observe resource availability when
submitting DL workloads. Through the visualization, User A filtered their
frequently used small-scale experiment partition and monitored the running
time of allocated experiments and the order of waiting experiments in real
time. This enabled them to make predictions about allocation times and
effectively plan their work. On the other hand, User B expressed that when
they noticed their submitted experiments consistently being pushed back in the
waiting pool, the visualization helped them quickly understand that
experiments with the higher scheduling priority scores had been submitted,
which resolved their doubts and provided clarity. These examples highlight the
positive feedback on HPC2lusterScape’s ability to assist users in
understanding scheduling situations, rather than solely waiting for resource
allocations. In addition, user B described their experience with a scheduling
bottleneck, a situation where experiments in the waiting pool were not being
allocated despite having available resources. They took a screenshot of the
cluster overview and shared it with the operators, which allowed them to
identify the bug more quickly and take appropriate action. They emphasized
that visualization helped the communication process. User C, a cluster
operator, said that after introducing new scheduling policies, they were able
to see side effects in real time that were not present in the simulation. For
example, after deploying a scheduling policy that assigns higher priority
scores to larger workloads, they observed waiting pools of each partition via
the cluster overview and noticed that small workloads were constantly being
pushed back and not allocated.
Collaborative effect for workload optimization: User A, who was initially
uninterested in utilization optimization, became motivated upon seeing their
resource utilization significantly lower than the average in overall
experiments. In addition, A, B, and C all mentioned detecting experiments with
consistently low utilization records using violation rules. They informed each
other about their efforts to optimize their experiments and instances where
colleagues’ experiment statuses seemed unstable or inefficient.
Detecting stalled training by iteratively stacking violation rules: User B,
whose main task is large-scale learning experiments, mentioned that stacking
violation rules could catch many cases of GPU idling due to stalled training.
Initially, they defined a simple violation rule condition to detect cases
where some GPUs exhibited low utilization values during training (R1: GPU
utilization 5th percentile $<$ 15%). For many of the experiments detected by
the rules, they could identify a common pattern in the case of GPU idling
through bivariate analysis view and system metrics timeline, that training
does not progress when utilization is high but power usage is low. After
gaining the empirical observation, they added an additional rule (R2: GPU
utilization 95th percentile $>$ 90%, R2: GPU power usage median $<$ 70%) to
capture stalled workloads characterized by consistently high utilization and
low power usage. However, they noted that identifying the root cause required
the use of other debugging applications.
Intuitively detecting imbalance in distributed training: User B provided
positive feedback on the ability of small multiples plots to intuitively
reveal workload and data imbalances that may arise during large-scale
training. They mentioned that while workload imbalances are difficult to track
at the code level, it was easy to identify which GPU or node exhibited
computational bias through the small multiples. Additionally, they appreciated
being able to grasp local imbalances for specific intervals while expanding a
particular time interval. They mainly identified the workload imbalance
comparing the utilization and NVLink bandwidth usage of each GPU and assumed
the data collection process between GPUs, represented by the NCCL library, as
the primary cause. This assumption was confirmed by further GPU profiling in
collaboration with user C, who has expertise in the hardware system. In
particular, it was discovered that the load was biased towards the master node
during the data collection process. To address this issue, they considered
implementing techniques that could reduce the amount of data communication
during training and modifying the network topology of the NCCL library.

##  8 Discussion & Future Work

Scalability of violation rules representation: Violation rules can be scaled
to create composite rules by combining various conditions within each rule,
but the number of configurable rules in HPC2lusterScape is limited to five.
This limitation arises from the restricted space between each group and group
label in the cluster overview, as well as the limited space available for
small multiples in the diagnostics view. We considered various indicating and
piling approaches previously proposed [20, 19], but concluded that an
excessive number of rules would hinder usability.
Identifying the root cause of problems in DL workload: The user study showed
that the diagnostics view is effective in detecting workload imbalance and
idling GPUs, but it also showed limitations in identifying the root cause of
each problem. Providing analytics of kernel-level metrics required for root
cause detection may be out of scope, but it gives a crucial research direction
for future studies.
Evaluation of cluster efficiency: Based on user studies, we found that many of
pain points were addressed through transparency in shared HPC clusters,
indicating that our work contributes to cluster efficiency. However,
quantitative evaluation of the effectiveness of the visualization approach
remains a limitation as it is challenging to control time-dynamic variables
such as the scheduler policy of the actual running cluster and the demand for
resources.

##  9 Conclusion

In this paper, we presents HPC2lusterScape, a visualization system designed to
improve the efficiency and transparency of shared high-performance computing
(HPC) clusters for AI model development. By offering an integrated monitoring
tool and a set of diagnostic features, HPC2lusterScape helps HPC operators and
ML researchers to effectively manage resources, identify potential issues, and
optimize their DL workloads. The system has been deployed and tested in an
industrial-scale HPC cluster, demonstrating its adaptability and real-world
applicability. By addressing the challenges and requirements of efficient HPC
operation in shared environment, HPC2lusterScape paves the way for more
effective resource allocation and management, ultimately facilitating the
advancement of large-scale AI models.

###### Acknowledgements.

The authors wish to thank MLOps HPC team members, including Chanwoong Kim,
Youngkwan Kim, and Adrian Kim.

## References

  * [1] A. Agrawal and M. Kolodziej.  PyProf – PyTorch Profiling Tool github.com/NVIDIA/PyProf.  NVIDIA, Inc., 2021.  Last accessed 2023.
  * [2] W. Barth.  Nagios: System and network monitoring.  No Starch Press, 2008.
  * [3] P. Baudisch, N. Good, V. Bellotti, and P. Schraedley.  Keeping things in context: a comparative evaluation of focus plus context screens, overviews, and zooming.  In Proc. the ACM SIGCHI International Conference on Human Factors in Computing Systems (CHI), pp. 259–266, 2002.
  * [4] A. Cockburn, A. Karlson, and B. B. Bederson.  A review of overview+ detail, zooming, and focus+ context interfaces.  ACM Computing Surveys (CSUR), 41(1):1–31, 2009.
  * [5] T. Dang, N. Nguyen, and Y. Chen.  Hiperview: real-time monitoring of dynamic behaviors of high-performance computing centers.  The Journal of Supercomputing, 77:11807–11826, 2021.
  * [6] R. De Maesschalck, D. Jouan-Rimbaud, and D. L. Massart.  The mahalanobis distance.  Chemometrics and intelligent laboratory systems, 50(1):1–18, 2000\.
  * [7] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. a. Ranzato, A. Senior, P. Tucker, K. Yang, Q. Le, and A. Ng.  Large scale distributed deep networks.  In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, eds., Advances in Neural Information Processing Systems, vol. 25. Curran Associates, Inc., 2012.
  * [8] S. Drucker and R. Fernandez.  A unifying framework for animated and interactive unit visualizations.  Microsoft Research, 2015.
  * [9] G. Ellis and A. Dix.  A taxonomy of clutter reduction for information visualisation.  IEEE transactions on visualization and computer graphics, 13(6):1216–1223, 2007.
  * [10] Y. Fan.  Job scheduling in high performance computing.  arXiv preprint arXiv:2109.09269, 2021.
  * [11] B. Fuglede and F. Topsoe.  Jensen-shannon divergence and hilbert space embedding.  In International Symposium on Information Theory, 2004. ISIT 2004\. Proceedings., 2004. doi: 10 . 1109/ISIT . 2004 . 1365067
  * [12] Google, Inc.  TensorBoard: TensorFlow’s visualization toolkit tensorflow.org/tensorboard.  Last accessed 2023.
  * [13] Google, Inc.  TensorFlow Profiler: A profiling and performance analysis tool for TensorFlow tensorflow.org/guide/profiler.  Last accessed 2023.
  * [14] J. Gu, H. Liu, Y. Zhou, and X. Wang.  Deepprof: Performance analysis for deep learning applications via mining gpu execution patterns.  arXiv preprint arXiv:1707.03750, 2017.
  * [15] H. Guo, S. Di, R. Gupta, T. Peterka, and F. Cappello.  La valse: Scalable log visualization for fault characterization in supercomputers.  In EGPGV@ EuroVis, pp. 91–100, 2018.
  * [16] T. Hoefler, T. Schneider, and A. Lumsdaine.  The effect of network noise on large-scale collective communications.  Parallel processing letters, 19(04):573–593, 2009.
  * [17] K. E. Isaacs, A. Gimenez, I. Jusufi, T. Gamblin, A. Bhatele, M. Schulz, B. Hamann, and P.-T. Bremer.  State of the art of performance visualization.  EuroVis (STARs), 2014.
  * [18] H. Kim, M. Kim, D. Seo, J. Kim, H. Park, S. Park, H. Jo, K. Kim, Y. Yang, Y. Kim, et al.  NSML: Meet the mlaas platform with a real-world case study.  arXiv preprint arXiv:1810.09957, 2018.
  * [19] F. Lekschas, B. Bach, P. Kerpedjiev, N. Gehlenborg, and H. Pfister.  Hipiler: visual exploration of large genome interaction matrices with interactive small multiples.  IEEE transactions on visualization and computer graphics, 24(1):522–531, 2017.
  * [20] F. Lekschas, X. Zhou, W. Chen, N. Gehlenborg, B. Bach, and H. Pfister.  A generic framework and library for exploration of small multiples through interactive piling.  IEEE Transactions on Visualization and Computer Graphics, 27(2):358–368, 2020.
  * [21] C. Leys, O. Klein, Y. Dominicy, and C. Ley.  Detecting multivariate outliers: Use a robust variant of the mahalanobis distance.  Journal of experimental social psychology, 74:150–156, 2018.
  * [22] S. Li, T. Ben-Nun, S. D. Girolamo, D. Alistarh, and T. Hoefler.  Taming unbalanced training workloads in deep learning with partial collective operations.  In Proceedings of the 25th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pp. 45–61, 2020.
  * [23] M. L. Massie, B. N. Chun, and D. E. Culler.  The ganglia distributed monitoring system: design, implementation, and experience.  Parallel Computing, 30(7):817–840, 2004.
  * [24] F. Murtagh.  A survey of recent advances in hierarchical clustering algorithms.  The computer journal, 26(4):354–359, 1983.
  * [25] D. Narayanan, K. Santhanam, F. Kazhamiaka, A. Phanishayee, and M. Zaharia.  Heterogeneity-aware cluster scheduling policies for deep learning workloads.  In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation, pp. 481–498, 2020.
  * [26] N. Nguyen, T. Dang, J. Hass, and Y. Chen.  Hiperjobviz: Visualizing resource allocations in high-performance computing center via multivariate health-status data.  In 2019 IEEE/ACM Industry/University Joint International Workshop on Data-center Automation, Analytics, and Control (DAAC), pp. 19–24. IEEE, 2019.
  * [27] NVIDIA, Inc.  DLProf docs.nvidia.com/deeplearning/frameworks/dlprof-user-guide, 2023\.  Last accessed 2023.
  * [28] NVIDIA, Inc.  Nsight Systems developer.nvidia.com/nsight-systems, 2023\.  Last accessed 2023.
  * [29] NVIDIA, Inc.  NVProf docs.nvidia.com/cuda/profiler-users-guide, 2023\.  Last accessed 2023.
  * [30] D. Park, S. M. Drucker, R. Fernandez, and N. Elmqvist.  Atom: A grammar for unit visualizations.  IEEE transactions on visualization and computer graphics, 24(12):3032–3043, 2017.
  * [31] V. Pham, N. Nguyen, J. Li, J. Hass, Y. Chen, and T. Dang.  Mtsad: Multivariate time series abnormality detection and visualization.  In 2019 IEEE International Conference on Big Data (Big Data), pp. 3267–3276, 2019. doi: 10 . 1109/BigData47090 . 2019 . 9006559
  * [32] S. Pumma, D. Buono, F. Checconi, X. Que, and W.-c. Feng.  Alleviating load imbalance in data processing for large-scale deep learning.  In 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID), pp. 262–271. IEEE, 2020.
  * [33] The PyTorch Team.  Pytorch Profiler: Profiling and optimizing pytorch applications.  Last accessed 2023.
  * [34] N. Rauschmayr, S. Kama, M. Kim, M. Choi, and K. Kenthapadi.  Profiling deep learning workloads at scale using amazon sagemaker.  In Proc. the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 3801–3809, 2022.
  * [35] F. Shilpika, B. Lusch, M. Emani, V. Vishwanath, M. E. Papka, and K.-L. Ma.  Mela: A visual analytics tool for studying multifidelity hpc system logs.  In Proc. the IEEE/ACM Industry/University Joint International Workshop on Data-center Automation, Analytics, and Control (DAAC), pp. 13–18, 2019.
  * [36] N. Sung, M. Kim, H. Jo, Y. Yang, J. Kim, L. Lausen, Y. Kim, G. Lee, D. Kwak, and J.-W. Ha.  NSML: A machine learning platform that enables you to focus on your models.  arXiv preprint arXiv:1712.05902, 2017.
  * [37] S. Teerapittayanon, B. McDanel, and H. Kung.  Distributed deep neural networks over the cloud, the edge and end devices.  In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pp. 328–339, 2017. doi: 10 . 1109/ICDCS . 2017 . 226
  * [38] W. Xiao, R. Bhardwaj, R. Ramjee, M. Sivathanu, N. Kwatra, Z. Han, P. Patel, X. Peng, H. Zhao, Q. Zhang, et al.  Gandiva: Introspective cluster scheduling for deep learning.  In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 595–610, 2018.
  * [39] W. Xiao, S. Ren, Y. Li, Y. Zhang, P. Hou, Z. Li, Y. Feng, W. Lin, and Y. Jia.  Antman: Dynamic scaling on gpu clusters for deep learning.  In OSDI, pp. 533–548, 2020.
  * [40] E. P. Xing, Q. Ho, W. Dai, J.-K. Kim, J. Wei, S. Lee, X. Zheng, P. Xie, A. Kumar, and Y. Yu.  Petuum: A new platform for distributed machine learning on big data.  In Proc. the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), KDD ’15, p. 1335–1344. Association for Computing Machinery, New York, NY, USA, 2015. doi: 10 . 1145/2783258 . 2783323
  * [41] Y. Yang, W. Xia, F. Lekschas, C. Nobre, R. Krueger, and H. Pfister.  The pattern is in the details: An evaluation of interaction techniques for locating, searching, and contextualizing details in multivariate matrix visualizations.  Proc. the ACM SIGCHI International Conference on Human Factors in Computing Systems (CHI), 2022.
  * [42] G. X. Yu, T. Grossman, and G. Pekhimenko.  Skyline: Interactive in-editor computational performance profiling for deep neural network training.  In Proc. the ACM Symposium on User Interface Software and Technology (UIST), pp. 126–139, 2020.


# Test of a modified BCS theory performance in the Picket Fence Model

V.Yu. Ponomarev  and A.I. Vdovin  Institut für Kernphysik, Technische
Universität Darmstadt, D–64289 Darmstadt, Germany  Bogoliubov Laboratory of
Theoretical Physics, Joint Institute for Nuclear Research, 141980 Dubna,
Russia

###### Abstract

Analyses of a modified BCS (MBCS) theory performance at finite temperatures in
the Picket Fence Model (PFM) for light and heavy systems is presented. Both
symmetric, $\Omega=N$ ($N$ particles on $\Omega$ twice-degenerate levels), and
asymmetric, $\Omega\neq N$, versions of the PFM are considered. The quantities
known exactly from particle-hole symmetry of the $\Omega=N$ PFM are
calculated. Starting from very low temperatures, these quantities are found in
dramatic deviation from the exact values in MBCS results. Consequences of the
MBCS prediction that heating generates a thermal constituent of the pairing
gap, are discussed. Thermodynamical inconsistency of the MBCS is also
addressed.

PACS: 21.60.-n, 24.10.Pa, 24.60.-k, 24.60.Ky

##  1 Introduction

A modified BCS (MBCS) theory for treating pairing correlations in atomic
nuclei at finite temperatures [1, 2] has been recently tested [3] in the
Picket Fence Model (PFM) in which $N$ particles are distributed over $\Omega$
twice-degenerate levels. The PFM with $N=\Omega$ is usually considered in the
literature.

The MBCS predicts a smooth decreasing behavior for a pairing gap as
temperature $T$ increases up to some $T_{M}$ when the theory suddenly breaks
down. It was reported [3, 4] that adding one extra level ($\Omega=N+1$)
extends the MBCS applicability to much higher temperatures. As is already
established [5], the $\Omega=N+1$ case of the PFM is the only example of an
exceptional MBCS pairing gap behavior and thus, it could be hardly considered
as a typical one. Accordingly, we find it necessary to provide the reader with
other $\Omega\neq N$ PFM examples not available in [3]. This is done in Sec.
2.

In Sec. 3, the MBCS prediction that not only the pairing force but also the
heating itself generates the pairing gap, is discussed.

In Sec. 4, we address the key question of the article: whether the MBCS is a
reliable theory in the temperature domain where its pairing gap looks
reasonable at first glance. For that, we examine some quantities which are
known exactly in the $N=\Omega$ PFM because of symmetry. We show that in the
MBCS predictions, these quantities dramatically deviate from their exact
values starting from very low temperatures.

Thermodynamical inconsistency of the MBCS is discussed in Sec. 5.

##  2 MBCS pairing gap in PFM systems with $\Omega=N$ and $\Omega\neq N$

The PFM or Richardson model is widely used as a test model for the pairing
problem. It is the pairing Hamiltonian applied to a system of $N$ fermions
distributed over $\Omega$ equidistant levels. All levels are twice degenerate
for the spin up and down. The levels below (above) the Fermi surface will be
referred to as holes (particles) and labeled by “$-i$” (“$i$”). Their single
particle energies are $\varepsilon_{-i}=(0.5-i)$ MeV for holes and
$\varepsilon_{i}=(-0.5+i)$ MeV for particles, where $i=1,~{}2,~{},\ldots$
(i.e. $\varepsilon_{i}=-\varepsilon_{-i}$). In all calculations presented
below (except for the ones in Fig. 6a), the pairing strength parameter $G$ is
adjusted so that the pairing gap $\Delta$ equals 1 MeV at zero temperature.

In addition to the fact that exact solution of the PFM is possible if $N$ and
$\Omega$ are not big, the PFM with $N=\Omega$ (to be referred to as the
conventional PFM) possesses internal particle-hole symmetry. This means that
at any temperature:

a) the energy of the Fermi surface $E_{F}$ equals exactly 0 MeV:

$E_{F}\equiv 0~{};$ (1)

b) the quasiparticle energies $E_{i}$ for particles and holes should be
degenerate:

$E_{i}=\sqrt{(\varepsilon_{i}-E_{F})^{2}+\Delta^{2}}\equiv\sqrt{(\varepsilon_{-i}-E_{F})^{2}+\Delta^{2}}=E_{-i}$
(2)

because of (1);

c) the particle occupation probabilities (Bogoliubov $u_{i}$ and $v_{i}$
coefficients) are related as:

$u_{i}=\sqrt{\frac{1}{2}\left(1+\frac{\varepsilon_{i}-E_{F}}{E_{i}}\right)}\equiv\sqrt{\frac{1}{2}\left(1-\frac{\varepsilon_{-i}-E_{F}}{E_{-i}}\right)}=v_{-i}$
(3)

because of (1,2);

d) the thermal quasiparticle occupation numbers $n_{i}=1/(1+\exp{(E_{i}/T)})$
and quasiparticle-number fluctuations $\delta N_{i}=\sqrt{n_{i}(1-n_{i})}$
should be equal for particles and holes with the same $i$:

$n_{i}\equiv n_{-i}~{}~{}~{}~{}~{}{\rm and}~{}~{}~{}~{}~{}\delta
N_{i}\equiv\delta N_{-i}~{}$ (4)

because of (2).

Of course, an asymmetric version of the PFM with $\Omega=N+k$ (where
$k=-N/2+1,\ldots,-1,$ $1,\ldots,$ $\infty$) may be considered as well but Eqs.
(1-4) are not valid for it.

The first test of the MBCS performance in the conventional PFM with
$N=\Omega=10$ revealed that at $T\approx 1.75$ MeV the system undertakes a
phase transition which manifests itself as a sharp simultaneous increase in
the pairing gap, a sharp decrease of the system energy and a discontinuity in
the specific heat $C_{V}$ (this phase transition was defined as a superfluid –
super-superfluid phase transition) [6]. This critical temperature was denoted
as $T_{M}$ [3, 4]. It was found that $T_{M}$ linearly increases with the
number of particles $N$ in the conventional PFM.

It was also reported that enlarging the space by one more level, $\Omega=N+1$,
restores the MBCS applicability to much higher temperatures even for $N\leq
14$ systems [3, 4]. Fig. 1 plots the MBCS pairing gap for $N=14$ particles and
$\Omega=N+k$ levels with $k$ changing from $-4$ to 50 (solid curves). Indeed,
the MBCS gap above $T_{c}$ is small and almost constant up to $T_{M}=7.2$ MeV
when suggested example of extended configuration spaces, $k=1$, is considered.
However, the MBCS theory breaks down again at rather modest temperature
$T_{M}=2.15$ MeV with adding one more extra level, $k=2$, etc. Thus, the
extension of the configuration space beyond $k=1$ makes the MBCS results
inappropriate again.

Figure 1: MBCS pairing gap $\Delta$ for the system of $N=14$ particles on
$\Omega=N+k$ levels (solid curves). The BCS pairing gap is plotted by the
dashed curve. The $k$ values are indicated in an oval at each curve. The
conventional critical temperature $T_{c}$ is shown in this and all other
figures by the vertical arrow.

One immediately notices from Fig. 1 that the MBCS theory predicts two typical
scenarios for the system evolution with heating. As temperature increases, the
system undertakes either

a) a superfluid – super-superfluid phase transition (examples with $k\leq 0$
in Fig. 1)

or

b) a phase transition from a superfluid phase with a positive gap to another
superfluid phase but with a negative gap (examples with $k>1$ in Fig. 1),

instead of a superfluid – normal phase transition of the conventional BCS.
Corresponding BCS pairing gap behavior in the same systems is shown in Fig. 1
by the dashed curve. It is impossible to visually distinguish the results when
$k$ value changes from $-4$ to 50.

We have performed additional calculations for the asymmetric PFM with $N$
changing from 6 to 100. The results look very similar to the ones in Fig. 1.
In all these examples, there exists only a single case $k=1$ with abnormally
large $T_{M}$ which grows almost linearly with $N$. For all other $k$ values,
the phase transitions of unknown types take place at much lower $T_{M}$ which
becomes smaller and smaller as $k$ differs from $1$ larger and larger.

However, only the case $\Omega=N+1$ has been selected for presentation in [3,
4] as an example of an extended configuration space ($\Omega>N$).

##  3 MBCS pairing gap: quantal and thermal constituents

Although the MBCS performance has been already discussed in several articles,
a clear answer to the question what is responsible for phase transitions of
unknown types predicted by this theory is not yet established. According to
the MBCS founders, there exists a criterion of the MBCS applicability,
according to which the line shape of the quasiparticle-number fluctuations
$\delta N_{i}$ should be symmetric [3, 4]. And the theory reaches the limit of
its applicability at $T_{M}$. The physical origin of this criterion is
unclear. The physical spectra are never symmetric, at least in nuclear
physics. On the other hand, the single particle spectrum of the conventional
PFM is ideally symmetric with respect to a chemical potential, see Eqs. (2,4),
but the MBCS itself breaks down this symmetry (see below).

The reader of [3] is suggested to notice from Fig. 2 that the line shape
becomes rather asymmetric approaching $T_{M}$, thus violating the above-
mentioned criterion of the MBCS applicability. We find it difficult to judge
of the symmetry of lines in Figs. 2(a-e). On the other hand, it is very easy
to quantify the results in this figure by plotting the $\delta N_{i}/\delta
N_{-i}$ ratios as a function of temperature for different $i$. This is done in
our Fig. 2 for different systems.

Figure 2: MBCS predictions for the ratios of quasiparticle-number fluctuations
$\delta N_{i}/\delta N_{-i}$ in different $(N,\Omega)$ systems. Exact result
$\delta N_{i}/\delta N_{-i}\equiv 1$ in Figs. 2(a,c,d) is shown by the dotted
line.

Indeed, the asymmetry increases above $T_{M}$ in the $N=\Omega=10$ system,
Fig. 2(a), while this does not in more heavy symmetric $N=\Omega$, Figs.
2(c,d), or selected asymmetric $\Omega=N+1$, Figs. 2(b), systems. However, it
is difficult to miss that this effect is marginal compared to asymmetry in all
systems (symmetric, asymmetric, light, and heavy) at much lower $T$ with the
maximum around $T_{c}$. In other words, the most severe violation of the
suggested criterion of the MBCS applicability takes place around the critical
temperature of the conventional BCS and not at $T_{M}$ where the MBCS breaks
down, as stated in [3, 4]. This conclusion does not depend on details of the
PFM system being considered.

Since the suggested criterion does not help to understand what causes the
theory breaking down, we continue our analysis. Let us read once again the
paragraph containing Eq. (26) in Ref. [3] that an important feature of the
MBCS theory is that the MBCS gap is the sum of a quantal part (which looks the
same as in the conventional BCS) and a thermal part $\delta\Delta$:

$\displaystyle\delta\Delta_{h}$ $\displaystyle=G\sum_{i}^{\rm
holes}(v_{i}^{2}-u_{i}^{2})\delta N_{i}$ $\displaystyle~{}~{}~{}{\rm
for~{}holes}$ (5) $\displaystyle\delta\Delta_{p}$
$\displaystyle=G\sum_{i}^{\rm particles}(v_{i}^{2}-u_{i}^{2})\delta N_{i}$
$\displaystyle~{}~{}~{}{\rm for~{}particles}~{}.$ (6)

Fig. 3 presents the MBCS gap for (a) light and (b) heavy PFM systems by the
thick solid curve. The quantal part of it is shown by the thin solid curve, it
becomes very small soon above $T_{c}$. The thermal part for holes (particles)
is plotted by the dashed (dot-dashed) curve. The quantity $\delta\Delta_{h}$
is always positive ($v_{i}>u_{i}$) and the quantity $\delta\Delta_{p}$ is
always negative ($u_{i}>v_{i}$); their absolute values increase with
temperature.

Figure 3: MBCS pairing gap (thick solid curve) for the PFM systems with (a)
$N=14$, $\Omega=15$ and (b) $N=\Omega=100$ and in calculation with a realistic
single particle spectrum for 120Sn: (c) neutron and (d) proton systems. This
gap is the sum of a quantal (particles plus holes – thin solid curve) and
thermal (holes – dashed curve, particles – dot-dashed curve) parts.

The physics of pairing suggested by the conventional BCS is very simple and
transparent. Pairing is generated by the pairing force. As temperature
increases the thermal scattering of nucleons becomes stronger and stronger and
finally destroys the pairing at $T_{c}$.

The MBCS suggests another physics, according to which the heating itself
generates a thermal constituent of the pairing gap: positive for holes and
negative for particles. The heavier is the system, the stronger thermal
pairing gap may be generated. A similar phenomenon takes place in calculation
with realistic single particle spectra (see Figs. 3(c) and (d) where the
pairing gap behavior in 120Sn is presented for neutrons and protons,
respectively).

Notice, when the pairing strength is weak to generate pairing at zero
temperature, the MBCS predicts that the heating develops the pairing gap at
finite $T$, as is obvious from Eqs. (5-6). An example of the pairing induced
by heating alone in a magic nuclear system is shown in Fig. 3(d). One should
not be surprised that the resulting pairing gap is negative in this example.
Fig. 3 clearly shows that as temperature increases, the MBCS gap receives main
contribution from $\delta\Delta_{h}$ and $\delta\Delta_{p}$ terms and their
sum can be equally positive (less particle levels, $k<0$ in Fig. 1) or
negative (more particle levels, $k>1$ in Fig. 1).

The main goal of the MBCS theory is to mimic the thermal behavior of the
pairing gap of a macroscopic treatment [9] that the normal – superfluid phase
transition is washed out and soon above $T_{c}$ the gap remains rather small
but positively finite. However, Fig. 3 shows that even technically, it is very
difficult to achieve the desirable goal with the MBCS equations: it is
necessary that two almost linearly growing functions $\delta\Delta_{h}$ and
$-\delta\Delta_{p}$ almost cancel each other with a high accuracy in a large
temperature interval111Independently from the physical content of the thermal
gap in the MBCS, one finds from Eqs. (3-6) that
$\delta\Delta_{h}\equiv-\delta\Delta_{p}$ for the conventional PFM and the
cancellation between these two terms should be exact at any $T$. This does not
happen in MBCS calculations because the MBCS theory violates Eqs. (3,4), see
below.. Of course, the final result is very sensitive to tiny details of a
single particle spectrum employed. If a spectrum with desirable properties is
occasionally found, the theory breaks down anyway with a step aside from it.
Fig. 1 clearly demonstrates it.

##  4 On range of the MBCS validity

As is established, the MBCS theory is capable of generating the pairing gap
behavior which looks reasonable at first glance below $T_{M}$. It is also
established that $T_{M}$ grows linearly with the mass of the system and it is
possible to find a single exceptional case of abnormally large $T_{M}$. Does
this mean that the MBCS theory should be considered as reliable for $T<T_{M}$
and recommended for applications?

To answer this question we return to the conventional PFM. Due to its internal
particle - hole symmetry, the accuracy of the MBCS predictions for the theory
variables can be easily examined even without having exact solutions.

Verification of Eq. (4) for the $\delta N_{i}$ quantities in Figs. 2(a,c,d)
reveals that this property of the system is enormously violated in the MBCS
predictions. The behavior of the ratios $n_{i}/n_{-i}$ is very similar to the
one of $\delta N_{i}/\delta N_{-i}$ in Fig. 2. Deviation from the exact result
$n_{i}/n_{-i}\equiv 1$ reaches a few thousand per cent near $T_{c}$.

The accuracy of the MBCS predictions for the quasiparticle energies, Eq. (2),
and for the particle occupation probability, Eq. (3), with $i=1$ is examined
in Fig. 4 and Fig. 5(a), respectively. Fig. 5(b) shows that the MBCS theory
does not keep the energy of the Fermi surface at zero energy, as it should be,
Eq. (1).

Figure 4: MBCS predictions for the ratios of quasiparticle energies
$(E_{-i}/E_{i})$ in different $N=\Omega$ systems. Exact results
$(E_{-i}/E_{i})\equiv 1$ are shown by the dotted line. Figure 5: MBCS
predictions for (a) ratios $(v_{-1}/u_{1})$ and (b) the energy of the Fermi
level $E_{F}$ in $N=\Omega=$ 10, 30, and 100 systems. Exact results (a)
$(v_{-1}/u_{1})\equiv 1$ and (b) $E_{F}\equiv 0$ for these systems are shown
by the dotted line.

The results in Figs. 2(a,c,d), 4, and 5 obviously demonstrate that as soon as
heating starts to play a role, the MBCS predictions are irrelevant to a system
which the theory aims to describe, because genuine properties of the system
are severely violated in these predictions. It does not matter whether a light
or heavy systems is considered, the results are almost independent on $N$.

All-in-all, and this is the main conclusion of the article: although the MBCS
theory generates a pairing gap which looks reasonable in some temperature
interval $[0,T_{M}]$, it is clear from the analysis of other quantities that
the theory predictions have nothing to do with the properties of a system
under discussion even in this temperature interval.

##  5 Thermodynamic inconsistency of the MBCS theory

We remind the reader that expressions for the MBCS theory were copied from
zero-temperature BCS expressions by straightforward replacing the BCS
$\\{u_{j},v_{j}\\}$ coefficients with the MBCS $\\{\bar{u}_{j},\bar{v}_{j}\\}$
coefficients. It has been already pointed out [6] that a method of mechanical
copying of equations from one theory into another one has absolutely no
grounds because the properties of $\\{u_{j},v_{j}\\}$ and
$\\{\bar{u}_{j},\bar{v}_{j}\\}$ coefficients are simply different. E.g., the
thermal part of the pairing gap $\delta\Delta$, discussed in Sec. 3, has
appeared in the MBCS theory as a result of such an uncontrolled copying.

It is very natural to verify whether expressions for thermodynamical
observables obtained by such a method are thermodynamically consistent in the
MBCS theory. Fig. 6 compares the system entropy which is calculated according
its thermodynamical definition (dashed lines):

$S_{\rm th}=\int_{0}^{T}\,\frac{1}{\tau}\,\cdot\,\frac{\partial
E_{tot}}{\partial\tau}\,d\tau$ (7)

and within a statistical approach (solid lines):

$S_{\rm sp}=-\sum_{j}\,(2j+1)\,\left[\,n_{j}\,{\rm ln}\,n_{j}+(1-n_{j})\,{\rm
ln}\,(1-n_{j})\right]~{}~{}.$ (8)

The latter one is also called the single particle entropy in Refs. [10, 11] or
the quasiparticle entropy in Ref. [8].

Figure 6: MBCS predictions for the system entropy calculated as $S_{\rm th}$,
Eq. (7), and $S_{\rm sp}$, Eq. (8) in different light and heavy, symmetric and
asymmetric PFM systems. Notice the logarithmic $y$ scale.

Fig. 6 shows that $S_{\rm th}$ and $S_{\rm sp}$ being considered separately
are amazingly close in different PFM systems for $T\leq 1$ MeV but they are
dramatically different one from another in the MBCS predictions. The latter
fact has been already addressed [5] when $S_{\rm th}^{\rm MBCS}$ and $S_{\rm
sp}^{\rm MBCS}$ for the neutron system of 120Sn have been found different by
two orders of magnitude.

Different entropy-like quantities in nuclear physics have been considered in
[10, 11]. It has been found that “the thermodynamic entropy $\ldots$, the
information entropy $\ldots$ and the single-particle entropy $\ldots$, all
coincide for strong enough interaction but only in the presence of a mean
field” [11]. It is not possible to accept interpretation of these results in
Ref. [8] as that $S_{\rm th}$ and $S_{\rm sp}$ “are nearly the same only for
noninteracting particles”. Notice, correspondence between $S_{\rm th}$ and
$S_{\rm sp}$ in a mean field plus residual forces (Figs. 56(IIa, IIc) in [10])
is definitely not worse than in an almost pure mean field approach (Figs.
56(Ia, Ic) in [10]) .

Fig. 6(a) also presents the quantities $S_{\rm th}^{\rm BCS}$ and $S_{\rm
sp}^{\rm BCS}$ calculated within the conventional BCS theory and $S_{\rm
th}^{\rm exact}$ obtained from the exact solution of the PFM for the
$N=\Omega=10$ system. One notices that for $T>100$ keV:

$S_{\rm th}^{\rm exact}\approx S_{\rm th}^{\rm BCS}\approx S_{\rm sp}^{\rm
BCS}\approx S_{\rm sp}^{\rm MBCS}>>S_{\rm th}^{\rm MBCS}~{}~{}.$ (9)

An attempt to compare $S_{\rm th}^{\rm MBCS}$ and $S_{\rm sp}^{\rm MBCS}$ with
the entropy calculated from the exact solution of the PFM has been made in
Fig. 6 of Ref. [8] for the $N=10$, $\Omega=11$ system. First of all, the
calculation of $S_{\rm sp}^{\rm exact}$ (thick solid line in Fig. 6(a) of Ref.
[8]) is definitely not correct. The author of [8] claims that $S_{\rm sp}^{\rm
exact}$ does not vanish at $T=0$ because occupation probabilities $f_{h}<1$
and $f_{p}>0$. The problem with the third law of thermodynamics222“at absolute
zero, any part of the body must be in a definite quantum state - namely the
ground state $\ldots$ the entropy of the body - the logarithm of its
statistical weight - is equal to zero” (see $\S$23, page 66 in [12]). in this
calculation is caused by confusion of interacting particles with
noninteracting “quasiparticles”: particle levels are not eigen states of the
pairing Hamiltonian, their occupation numbers do not obey Fermi-Dirac
distribution and because of that, they cannot be used in Eq. (8) which
represents the free Fermi-gas combinatorics. Incorrectly calculated $S_{\rm
sp}^{\rm exact}$ is also published in Fig. 8 of [3].

One notices that Fig. 6(b) of Ref. [8] lacks comparison between $S_{\rm
th}^{\rm MBCS}$ and $S_{\rm th}^{\rm exact}$ quantities333What is plotted by
thick solid line in this figure and makes impression of agreement with $S_{\rm
th}^{\rm MBCS}$ is not specified.. The latter quantity can be easily
calculated from $E_{tot}^{\rm exact}$ (thick solid line in Fig. 4(a) of Ref.
[8]). A level of expected disagreement between $S_{\rm th}^{\rm MBCS}$ and
$S_{\rm th}^{\rm exact}$ quantities can be seen from our Fig. 6(a) (compare
dashed curve with circles).

Analysis of the MBCS predictions for the system entropy clearly indicate a
problem with the expression for the system energy which enters in Eq. (7). How
this expression was obtained, is mentioned in the beginning of the section. In
Fig. 9 of Ref. [6] the system energy calculated from this expression has been
straightforwardly compared to the system energy calculated as
${\rm<H>=Tr(HD)}$ where $D$ is a density operator and $<\ldots>$ means
averaging over the grand canonical ensemble. Dramatic disagreement in two
quantities representing the same physical observable has been obtained
indicating the same problem.

##  6 Conclusions

In this article we continue discussion on the validity of the MBCS theory. The
theory performance is examined within the PFM. Our present goal is to allow
the reader, who is not familiar with the previous discussion on the subject in
Refs. [4, 5, 6, 7], to better understand the results and conclusions in Refs.
[3, 8].

We confirm that there exists a single example of the PFM (with the number of
levels $\Omega$ equal to the number of particles $N$ plus one) in which the
MBCS produces the thermal behavior of the pairing gap similar to the one of a
macroscopic theory up to a rather high temperature. In our opinion, this fact
alone is not sufficient to conclude on the theory performance without noticing
that in all other examples of the PFM with $\Omega\neq N$ the theory predicts
phase transitions of unknown types at a much lower temperature. These other
examples are neither shown nor discussed in Ref. [3].

Although the MBCS formally yields a visually acceptable pairing gap behavior
in some temperature domain, the physical content of its pairing gap is very
dubious. The MBCS predicts that heating generates a thermal constituent of the
pairing gap which becomes stronger and stronger with temperature in
contradiction with generally accepted understanding of the pairing phenomenon
in nuclei as a result of a specific particle-particle interaction. We point
out that this unphysical constituent of the MBCS pairing gap is responsible
for strong sensitivity of the theory predictions to tiny details of single
particle spectra.

The conventional PFM with $\Omega=N$ possesses internal particle-hole
symmetry. Due to it, some quantities in the model are known exactly without
any calculations. We have demonstrated that the MBCS predictions deviate from
exact results by a few hundred per cent starting from very low temperatures.

We also point out that the MBCS is a thermodynamically inconsistent theory.

Performing a systematic analysis of the MBCS performance in the PFM we have
failed to find a model system and/or temperature range where the MBCS
predictions are not questionable.

## Acknowledgments

The work was partially supported by the Deutsche Forschungsgemeinschaft (SFB
634).

## References

  * [1] N.D. Dang, V. Zelevinsky, Phys. Rev. C 64 (2001) 064319.
  * [2] N.D. Dang, A. Arima, Phys. Rev. C 67 (2003) 014304.
  * [3] N.D. Dang, Nucl. Phys. A 784 (2007) 147.
  * [4] N.D Dang, A. Arima, Phys. Rev. C 74 (2006) 059801.
  * [5] V.Yu. Ponomarev, A.I. Vdovin, Phys. Rev. C 74 (2006) 059802.
  * [6] V.Yu. Ponomarev, A.I. Vdovin, Phys. Rev. C 72 (2005) 034309.
  * [7] N.D. Dang, A. Arima, Phys. Rev. C 68 (2003) 014318.
  * [8] N.D. Dang, Phys. Rev. C 76 (2007) 064320.
  * [9] L.G. Moretto, Phys. Lett. B 40 (1972) 1.
  * [10] V. Zelevinsky, B.A. Brown, N. Frazier, and M. Horoi, Phys. Rep. 276 (1996) 85.
  * [11] V.K. Kota and R. Sahu, Phys. Rev. E 66 (2002) 037103.
  * [12] L.D. Landau and E.M. Lifshitz, Statistical Physics (Pergamon Press, London-Paris, 1958).