# Design and implementation of the AMIGA embedded system for data acquisition

The Pierre Auger Collaboration et al

###### Abstract

The Auger Muon Infill Ground Array (AMIGA) is part of the AugerPrime upgrade
of the Pierre Auger Observatory. It consists of particle counters buried
$2.3\text{\,}\mathrm{m}$ underground next to the water-Cherenkov stations that
form the $23.5\text{\,}{\mathrm{km}}^{2}$ large infilled array. The reduced
distance between detectors in this denser area allows the lowering of the
energy threshold for primary cosmic ray reconstruction down to about
${10}^{17}\text{\,}\mathrm{e}\mathrm{V}$. At the depth of
$2.3\text{\,}\mathrm{m}$ the electromagnetic component of cosmic ray showers
is almost entirely absorbed so that the buried scintillators provide an
independent and direct measurement of the air showers muon content. This work
describes the design and implementation of the AMIGA embedded system, which
provides centralized control, data acquisition and environment monitoring to
its detectors. The presented system was firstly tested in the engineering
array phase ended in 2017, and lately selected as the final design to be
installed in all new detectors of the production phase. The system was proven
to be robust and reliable and has worked in a stable manner since its first
deployment.

##  1 Introduction

The Pierre Auger Observatory [1] was originally designed to study ultra-high-
energy cosmic rays (UHECR) with primary particle energy above
$3\text{\times}{10}^{18}\text{\,}\mathrm{eV}$. It is located in the Southern
Hemisphere near the Andes mountains in the south-west part of the province of
Mendoza, Argentina. It uses a hybrid detection technique employing both a
surface detector array (SD) and a fluorescence detector (FD).

The SD of Auger is composed of an extensive array of about 1600 water
Cherenkov detectors (WCD), separated by a $1500\text{\,}\mathrm{m}$ spacing
and covering an area of $3000\text{\,}{\mathrm{km}}^{2}$. As above
$5\text{\times}{10}^{19}\text{\,}\mathrm{eV}$, the flux is of around 1
particle per $\text{\,}{\mathrm{km}}^{2}$ per century, the large area of the
observatory allows to observe more than 30 high energy cosmic rays per year
[1]. A detailed description of the SD of the Auger Observatory can be found in
[2].

Each of the 1600 surface detector stations includes a $3.6\text{\,}\mathrm{m}$
diameter water tank containing a sealed liner with a reflective inner surface.
The liner contains $12\,000\text{\,}\mathrm{l}$itres of purified water.
Cherenkov light produced by the passage of charged particles through the water
is collected by three nine-inch photo-multiplier tubes (PMTs) that are
symmetrically distributed at a distance of $1.20\text{\,}\mathrm{m}$ from the
center of the tank and look downwards through windows of clear polyethylene
into the water. The surface detector station is self-contained. A photovoltaic
system provides an average of $10\text{\,}\mathrm{W}$ for the PMTs and the
electronics system consisting of a processor, GPS receiver, radio transceiver
and power controller.

The surface detector electronics records the PMT signals, makes local
triggering decisions, sends time-stamps to the central data acquisition system
for the global trigger building, and stores event data for retrieval when a
global trigger condition is fulfilled. Due to the low available bandwith of
$150\text{\,}\mathrm{B}\mathrm{y}\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{/}\mathrm{s}$
per station, they must operate semi-autonomously, performing calibrations and
taking action in response to alarm conditions at the station level. The
electronic system was designed 15 years ago using the technology available at
that time.

Figure 1: Map of the Pierre Auger Surface detector (right), located in
Malargüe, in the southern region of the Mendoza province as shown in the lower
right panel, with the AMIGA infill area marked with dotted lines and zoomed in
(left). In the blow-up the positions of the engineering array are named
“Unitary cell” and enclosed by a dotted line. The central communication tower
is located about $7\text{\,}\mathrm{km}$ to the west (distance not to scale).
The denser $433\text{\,}\mathrm{m}$ infill is not shown in this figure for
simplicity, but can be observed in figure 10

AMIGA (Auger Muon Infill for the Ground Array) [3] is an enhancement of the
Pierre Auger Observatory designed to satisfy two different objectives. The
first objective is to lower the minimum detectable cosmic ray energy threshold
to $1\text{\times}{10}^{17}\text{\,}\mathrm{eV}$ by installing 61 SD stations
with $750\text{\,}\mathrm{m}$ spacing in an infill array covering an area of
$23.5\text{\,}{\mathrm{km}}^{2}$, which was completed in 2012. A more recent,
and even denser array with $433\text{\,}\mathrm{m}$ spacing over
$1.9\text{\,}{\mathrm{km}}^{2}$ was also included and completed in 2019.
Figure 1 shows a representation of AMIGA. The second objective is to enhance
the capability of Auger to study ray composition by a direct measurement of
the muonic shower component. The muon detection is achieved by
$30\text{\,}{\mathrm{m}}^{2}$ scintillator detectors buried
$2.3\text{\,}\mathrm{m}$ underground alongside all surface stations. The
$540\text{\,}\mathrm{g}\mathrm{/}\mathrm{c}\mathrm{m}\mathrm{{}^{2}}$ of
overburden determined by the local soil density completely shields the
electromagnetic component of the showers and only muons of energy greater than
$1\text{\times}{10}^{9}\text{\,}\mathrm{eV}$ are capable of hitting the
detectors.

The main function for the AMIGA Data Acquisition System is to manage the data
transfer from the muon counters in a modular, flexible, easily configurable
and scalable fashion compatible with the Pierre Auger Observatory data
trigger. These characteristics led to specifications on how to design the
firmware, software, hardware, network structure and protocol, and finally the
monitoring system. For each of these important aspects of the data acquisition
system, we dedicate a section on this paper, where we describe in more detail
the motivation and the chosen solutions.

The AMIGA buried stations work in sync with the SD, receiving the local
triggers by their accompanying WCD. The energy and geometry of the showers are
reconstructed by the SD while extra information on the particle composition is
provided by the muon-related observables such as the timing and size of the
signal provided by buried scintillators.

In this paper we describe the features of the embedded system design for the
AMIGA buried scintillator detectors, dubbed Underground Muon Detector (UMD).
This description includes the electronics, as well as the synchronization with
the associated SD station. As an illustration of the functionality of the
whole acquisition chain, we conclude showing an event measured by the seven
stations that compose the engineering array.

The main contribution of this work to distributed embedded systems in general
is the following: We present in this paper a flexible scalable massive system
of multiple distributed muon detectors that can be organized in a remote
fashion, using existing tools for a very specific problem: the detection of
the muon component of cosmic showers. In this paper we describe how we
organize the detection system using the preexisting trigger conditions of the
surface detectors in the Pierre Auger Observatory and how we can apply
reliable existing technology that can be adapted and easily applied as a
solution to this problem. The electronics system of the front-end is described
in a companion paper [4] where the details of the front-end electronics of the
UMDs, its design and performance are described. In summary, we must mention
that the UMDs work as muon counters, using two types of detection: 1 bit
multichannel digitalization, one for each of the 64 channels, and a
$14\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ Analog to Digital Converter with
two independent channels that measure the charge of the signal from the 64
channels for a muon count that exceeds 64 muons.

##  2 Underground Muon Detector overview

As the main objective of the UMD is to count the number of muons in a cosmic
shower produced by a cosmic ray, the original design of AMIGA was based on
particle detectors that would be able to count individual muons in a radiation
detector. This detector was implemented as a segmented scintillator, divided
into three modules of 64 channels each, with a total area of detection
determined via simulations [5]. These simulations were also used to obtain the
optimum detector segmentation ($192\text{\,}$ scintillator bars). The total
area needed of $30\text{\,}{\mathrm{m}}^{2}$ was therefore covered by three
modules of $10\text{\,}{\mathrm{m}}^{2}$ each.

All SD stations in the infilled array have been installed and are operating
since 2008 (61 stations) [6]. The SD stations in the infilled
$750\text{\,}\mathrm{m}$ array have been installed since 2008 (42 additional
stations) [7] and the complete grid is operative since 2012. More recently,
the SD 433m array was also completed. The companion UMD had an engineering
array (EA) phase of seven stations to validate and optimize the detector
design. This prototype phase ended in 2017 with two major changes: the
replacement of the optical device from PMTs to SiPMs and improved electronics,
which is presented in this work. Immediately after the EA phase, the
production phase started and the UMD array completion is foreseen by the end
of 2022 [7].

AMIGA uses a telecommunications system that consists of a point-to-multipoint
802.11n WiFi radio link, with redundant coordinators located at the Coihueco
fluorescence detector building (about
$6\text{\,}\mathrm{km}10\text{\,}\mathrm{km}$ link distance). That link
provides TCP/IP remote access to the LAN of each station [8]. The system is
also designed to be energy efficient (as power from the photovoltaic system is
limited) and has the capacity to transmit a high amount of data from those
remote areas. All AMIGA stations are expected to work at least 10 years in
remote (and possibly hard to reach) areas. Under these conditions, the systems
need to have a long service life and to allow remote access for diagnostics
and early fault detection for high quality of data.

The electronics were designed as embedded systems to ease eventual hardware
upgrades. They run Linux as open source operating system to be free from any
particular commercial software and be able to implement different hardware
choices used on the detectors.

###  2.1 Electronics Overview

The electronics for the UMD was designed to provide a reliable data transfer
from the buried scintillator modules to the main data storage server in
Malargüe. In order to do this, each UMD is connected to an SD to use the
Pierre Auger Observatory trigger. In order to provide a safe data transfer and
not to overload the existing SD telecommunications and power system, each
station was equipped with a new telecommunications system [8], a new solar
power system [9], a trigger relay system (the _distributor_ , see section
2.5), three buried modules that comprise the UMD itself, and a synchronization
system that connects the UMD data acquisition system with its corresponding SD
data acquisition system. The motivation for the telecommunications system is
stated in the mentioned reference, as a summary, an 802.11n WiFi system
running a TCP/IP network is used. The motivation for the solar panel system
design is stated in the mentioned reference. In summary, a
$24\text{\,}\mathrm{V}$ power system was implemented to provide the power
needed via two deep discharge $12\text{\,}\mathrm{V}$ batteries connected in
series that satisfy the power requirements of the UMDs and their
telecommunications system, a total of $\sim 46\text{\,}\mathrm{W}$ in the
engineering array for at least one day without solar energy (For production
the power of the buried electronics of the UMDs was decreased to $\sim
3.6\text{\,}\mathrm{W}$ as measured in the field, reducing the total power
budget to $\sim 19\text{\,}\mathrm{W}$ per station in the main array, allowing
the photovoltaic system to have an extended autonomy of at least three days
without solar energy). The trigger distribution and synchronization mentioned
was motivated by the need to provide muon counting data stored in the UMDs
each time a trigger request from the main data processing center in Malargüe
is received. Therefore, the UMDs must be synchronized with the SDs and the
trigger lines must be distributed to each of the three buried muon counters in
each station.

A drawing that illustrates the design of the SD and UMD combined detectors at
a position in the infill is shown in figure 2 with its main components. Three
$10\text{\,}{\mathrm{m}}^{2}$ buried modules are used at each surface detector
station for the final design [5].

Figure 2: General overview of an infill station. The figure shows a counter
and surface detector. Three modules are shown buried $2.3\text{\,}\mathrm{m}$
deep, with a detection area of $10\text{\,}{\mathrm{m}}^{2}$. Module
electronics are accessed for field installation and service via a plastic
tube. In addition, the solar panel and the additional battery box are shown.
Figure 3: Block diagram of an AMIGA station electronics. The Ethernet switch
conforms the center of the star of the Ethernet LAN. The distributor
replicates the fast trigger signal from the SD station and injects power from
the photovoltaic system to the Ethernet cables that connect the buried
modules.

The block diagram of the electronics is shown in figure 3, which describes the
components of one combined station and its interconnection as detailed in the
following subsections.

###  2.2 UMD scintillator module

In this section, we present a summary of the underground scintillator module
and its electronics functionality. A thorough description of the scintillator
module and the buried electronics has already been published in [10], [11] and
[4]. In summary, the motivation for the design of this electronics was to
provide a ring buffer that serves as a temporary storage big enough to save
muon data of a full cosmic shower for the time the Auger Observatory trigger
takes to send a request for that data. In this section we introduce the
scintillator module, the hardware of the underground electronics, the
structure of the stored data and the requirements for its implementation in
the digital back-end of the underground electronics. We also review the
hardware structure of the buried electronics interconnected with the
scintillator bars using optical fibers.

Every module comprises 64 scintillation bars, each of $41\text{\,}\mathrm{mm}$
width, $10\text{\,}\mathrm{mm}$ thickness and $4\text{\,}\mathrm{m}$ length
for a detection area of $10\text{\,}{\mathrm{m}}^{2}$. The light produced in
each scintillator bar is collected and propagated along a wavelength-shifting
(WLS) optical fiber of $1.2\text{\,}\mathrm{mm}$ diameter glued into a
lengthwise groove of the bar. All fibers connect to an optical coupler (also
called “cookie”) located in front of the light sensor. The $64\text{\,}$
scintillators and optical fibers are enclosed within a PVC (Polyvinyl
Chloride) casing and they form together with the electronics the detector
module. A detailed description of the AMIGA detector is found in [10].

Two groups of $32\text{\,}$ scintillator bars are mounted in each module at
opposite sides of a central dome that contains the light sensor and
electronics. For the final design we selected the Hamamatsu S13361 Silicon
Photomultiplier (SiPM) [11]. An integrated electronics acquires the analog
signals from this SiPM, processes them and provides control, monitoring and
communication functionality to the module. The system records event data
synchronized with the associated SD station (at about
$100\text{\,}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$)
and stores them for roughly the same time as the SD station. Each recorded
event is composed of $2048\text{\,}$ samples of
$64\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ measured at
$320\text{\,}\mathrm{M}\mathrm{s}\mathrm{p}\mathrm{s}$ and $1024\text{\,}$
samples of two $14\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ ADCs (high and low
gain) measured at $160\text{\,}\mathrm{M}\mathrm{s}\mathrm{p}\mathrm{s}$ [4].
The former and the latter are dubbed the binary and the integrator acquisition
modes respectively. Then, an event with a temporal length of
$6.4\text{\,}\mathrm{\SIUnitSymbolMicro s}$ requires:
$64\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ x
$2048\text{\,}\mathrm{+}$ 2 x
$16\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ x 1024 =
$163\text{\,}\mathrm{K}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$. Notice that
the $14\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ traces are sent in
two Bytes, hence the $16\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$
value in the second term. Added to this trace are a header of
$336\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$, a trace of 2 x
$2048\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ that correspond to the
OR gates in the two CITROCs of the front-end [4] and
$16\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ x
$2048\text{\,}\mathrm{t}$hat are the identifiers of each the bins in the
$64\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ trace. All this adds up
to a total number for the trace of:
$163\text{\,}\mathrm{K}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ \+
$336\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ \+ 2 x
$2048\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ \+
$16\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ x
$2048\text{\,}\mathrm{=}$
$201\text{\,}\mathrm{K}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$. In order to
store the data of a candidate event for about $20\text{\,}\mathrm{s}$ at a
rate of
$100\text{\,}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$,
the minimum memory needed is
$402\text{\,}\mathrm{M}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$.

The electronics for each module consists of the front-end board and the
Acquisition and Control board. The front-end processes the 64 light signals by
two Application Specific Integrated Circuit (ASIC) [11] and two
$160\text{\,}\mathrm{M}\mathrm{s}\mathrm{p}\mathrm{s}$
$14\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ ADCs (Analog Devices ADS4246)
digitizing the sum of all SiPM channels (one with a low-gain amplifier and one
with a high-gain amplifier). Each ASIC channel provides a pre-amplifier with
programmable gain and a fast shaper with $15\text{\,}\mathrm{n}\mathrm{s}$
peak time. Finally, a discriminator digitizes the signal into one bit. The
discriminator threshold has a coarse setting via a
$10\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ DAC (common for the 32 channels)
and a per-channel fine setting using individual
$4\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ DACs. The Acquisition and Control
Board (also called in this paper the _back-end_) records the digital
information of the front-end board and stores the data in a ring buffer when a
trigger condition is found. Details are given in section 3.

###  2.3 Synchronization

In order to comply with the trigger structure of the Pierre Auger Observatory,
we need to establish how the synchronization of the UMD will be carried out
whenever a cosmic shower is detected by the SDs. The detailed description of
the trigger structure of the observatory is already thoroughly described in
[12] and [1]. In this section we outline the main restrictions our Data
Acquisition System has to abide by in order to comply with these trigger
requirements. The specifications described in this section serve as a
reference for the trigger interconnection between the surface and buried
electronics, the memory usage in the ring buffer and the data handling that
has to be performed in the SD in order to successfully transfer muon data from
the UMD to the main server in Malargüe that is effectively synchronized with
the detection of a cosmic shower.

The Auger surface detector data acquisition and storage system is comprised of
a central storage server (or “CDAS” located in the city of Malargüe) and
several client stations; the muon counter replicates this structure, merging
the muon counter data with the surface detector data in a post-processing step
offline.

The trigger system of SD is hierarchical: The first level of trigger (T1) is
generated independently by each station based on local analysis of the signals
produced by its PMT. At calibrated SD stations the threshold is set for an
average T1 trigger rate of about
$100\text{\,}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{s}$.
These events are stored in a circular buffer with a capacity of
$3000\text{\,}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}$.
Events are identified by a 64-bit time-stamp (GTS) obtained from the GPS
receiver with a latency of about $200\text{\,}\mathrm{\SIUnitSymbolMicro s}$
after T1 as measured in the laboratory. The station then applies predefined
quality cuts to generate a second level trigger T2 with an average rate of
$20\text{\,}\mathrm{H}\mathrm{z}$. A list of T2 time-stamps is sent to CDAS
once every second. CDAS searches through the list of T2 triggers for a
coincidence in vicinity and time. When a coincidence is found, an array
trigger (T3) is sent back to participating stations and its neighbors. Upon
receiving a T3 trigger request, the station replies to the CDAS with the event
data, if it is present in the circular buffer [12], [1].

UMD buried modules were designed to rely on an external trigger in order to
acquire event data during a possible particle shower. To provide
synchronization with the underground particle counter modules, additional
electronics were installed in the surface station. These complementary
electronics were designed to trigger and label the particle counters data when
the surface detector encounters a T1 trigger condition, and to forward T3 data
requests from CDAS [13] to the buried electronics afterwards.

####  2.3.1 Trigger output from Auger Surface Detectors

The time that precedes a T1 event strongly depends on the delay between the
issuing of the trigger signal and the arrival of the identifier. To address
this issue a buffer that stores the T1 data must take this delay into account.
Also a faster additional time-stamp was implemented, capable of arriving at
the buried modules within a few microseconds after the SD station detects the
trigger condition. This time-stamp, called Local Time-Stamp (LTS) consists of
a $24\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ number generated by the surface
station. Taking into account the average T1 rate, each LTS value is guaranteed
to be unique for at least $26.8\text{\,}\mathrm{s}$, enough time to ensure
that requested events will not have time-stamp collision (and therefore data
collision) with newer events [14]. The time it takes between the detection of
an event condition (T1) and the sending of the signal is about
$760\text{\,}\mathrm{ns}$. However, using this time-stamping mechanism
requires a way of converting between the time-stamp used by the surface
detector and the LTS. This conversion is provided by electronics named Trigger
and Data Request (TDR).

Figure 4: T1 circuit (LTS) between the surface detector electronics and the
modules. The simplified circuit for a line is shown in order to have a better
understanding. The implemented circuit distributes the signals in the
distributor, after the LVDS receiver.

Both trigger signals (to the buried electronics and the information sent to
the TDR) are sent from an auxiliar board retrofitted in every SD station with
muon counter detectors installed (“Trigger-TX”). The minimum required data
transmission rate is $10\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$
($100\text{\,}\mathrm{ns}$ per bin). This board has two low-voltage
differential signalling (LVDS) drivers. Each LVDS line uses a point-to-point
configuration, using drivers and receivers from Fairchild Semiconductors
(FIN1001 and FIN1002 respectively) supporting data rates higher than
$600\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$.

The typical power consumption of the transmitter is $23\text{\,}\mathrm{mW}$
and of the receiver $13\text{\,}\mathrm{mW}$, thus each data line requires
$36\text{\,}\mathrm{mW}$. This system uses two of the available four pairs of
lines to send signals (data and clock), which gives a power consumption per
link of $72\text{\,}\mathrm{mW}$.

In order to avoid ground loops, the fast trigger signal is isolated before it
reaches the LVDS driver using an Digital Isolator from Texas Instruments
(ADuM1400), as shown in figure 4. The transmission line is terminated by a
$100\text{\,}\mathrm{\SIUnitSymbolOhm}$ resistor placed very close to the
receiver for best impedance match. An Unshielded Twisted Pair (UTP) Cat 5e
cable with RJ45 connectors crimped at both ends (following TIA/EIA norm 586-A)
is used to transmit the signal.

The delays (calculated from timing data in each component’s datasheet) are
shown in table 1.

Component | Delay | Unit
---|---|---
Isolator | $43\text{\,}$ | $\text{\,}\mathrm{ns}$
Driver LVDS | $5\text{\,}$ | $\text{\,}\mathrm{ns}$
Cable UTP | $6\text{\,}$ | $\text{\,}\mathrm{ns}\text{\,}{\mathrm{m}}^{-1}$
LVDS receiver | $5\text{\,}$ | $\text{\,}\mathrm{ns}$
Table 1: Summary of delays for components involved in signal transmission.

Figure 4 shows that the signal crosses two isolators, two LVDS transmitter /
receiver pairs and two sections of UTP cables. With a $6\text{\,}\mathrm{m}$
cable between SD electronics and distributor and a $30\text{\,}\mathrm{m}$
cable to the UMD electronics the latency for signal transmission is
$322\text{\,}\mathrm{ns}$.

To calculate the total delay, the time needed for the acquisition system to
recognize the trigger condition must be added to the signal latency calculated
below. This time is about $500\text{\,}\mathrm{ns}$ therefore, the trigger of
the AMIGA electronics will arrive about $1600\text{\,}\mathrm{ns}$ after the
trigger signal has been generated in the Auger station electronics.

####  2.3.2 Trigger and Data Request

In the beginning of the engineering array phase the functionality of the TDR
was realized by a Single Board Computer (SBC) with a buffered SPI receiver on
the basis of an Altera MAX-II complex programmable logic device (CPLD). Since
the production phase, an AMIGA Acquisition board is used for this purpose.
With the installation of the new SD electronics called “Upgraded Unified
Board” (UUB) [15] as part of the Auger upgrade, the TDR functions are
available in this board and no auxiliary electronics are needed.

The TDR receives the GTS+LTS time-stamp from the Auger surface electronics via
a $10\text{\,}\mathrm{M}\mathrm{b}\mathrm{p}\mathrm{s}$ SPI channel and sends
it to the SBC, which stores these time-stamps in a circular buffer with a
depth of $2048\text{\,}$ values. The TDR wiretaps the T3 requests to the
station by a RS-232 interface at the moment. The UUB has all the hardware
ports, software (T3 trigger to the buried electronics and distributor
commanding/monitoring), and FPGA code needed to replace the TDR. Therefore, on
stations with a UUB installed, the TDR and SPI line are removed, and the
distributor control cable is connected to the UUB directly.

When a T3 is found, the TDR looks for the received GTS in the GTS+LTS table.
If the GTS is found, its corresponding LTS is broadcasted to all the modules
via UDP; otherwise, a “LTS not found” message is sent. The TDR also provides
access to the SD serial console via SSH or TELNET protocols. Note that this
trigger scheme has no dependency on the buried module electronics, and can be
reused by any project designed to trigger with Auger surface detector
stations.

###  2.4 Communications

The telecommunications system connects the AMIGA stations to the central
storage server, and it comprises of an Ethernet LAN network (connecting the
UMDs and the SD electronics) and a WLAN telecommunications system for the data
transmission to CDAS. The WLAN is implemented as a point-to-multipoint WiFi
radio link attached to the original Auger inter-FD microwave links and it is
provided by a Mikrotik RB493 router, which includes an Ethernet switch and a
802.11n WiFi radio. These characteristics are mentioned in [8], where we can
also find that the network structure for the UMDs was implemented over the
$2.4\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ WiFi band. The motivation for
this decision was to have a modular system that is higly reliable to work
outdoors and that can handle the throughput required for the network of
detectors over the $2.4\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ Wi-Fi band,
for which the Pierre Auger Observatory has a licence for exclusive use in the
local region granted by the _Comisión Nacional de Comunicaciones_ of
Argentina. The details of the design, the protocol as well as the tests
performed to measure its reliability are described in the mentioned reference.
In this section we expand the description of the network to the higher layers
and how it is organized in subnetworks within each detector.

####  2.4.1 Network

The network addressing scheme for communications uses one /24 network (Class C
network, 253 available IP addresses) for each station LAN, with all radios
connected on the same WLAN network. (172.16.0.0/16). A station with an
internal LAN network 10.a.b.0/24 will have the IP 172.16.a.b in the WLAN
network. In the internal detector LANs, the IP 10.a.b.1 is reserved for the
gateway (WiFi radio), 10.a.b.2 for the TDR, and 10.a.b.(10 + x) for the module
x of that detector (when x $<$ 90).

###  2.5 Power and Trigger distribution

This section explains the distribution of the Pierre Auger Observatory
triggers and their interconnection at a physical level with the UMDs. It is
worth mentioning that the trigger and power distribution between a SD and the
three buried UMDs are intimately related since they share the same lines,
therefore we also include a brief description of the power and monitoring
system used. As mentioned before, the main piece of hardware that is in charge
of handling the trigger and power lines to the buried UMDs is called the
_distributor_ , and in this section we introduce its functionality.

The power for each station is supplied by independent photovoltaic systems
since all installation sites are far away from any electric power grid [9].
The distributor receives power from the solar panels and distributes it to the
four modules via Passive Power-over-Ethernet (PPoE) lines. It uses the same
cable to distribute the fast trigger signal sent to the buried modules.

The distributor converts the LVDS signals from the Auger surface electronics
to TTL-CMOS logic using LVDS receivers. The TTL fast trigger is isolated in
order to avoid a possible ground loop and is split into four lines. Each line
is independently converted to LVDS signals, available at the distributor
output ports. Each PPoE output uses the pin-out of 802.3af mode
B111$24\text{\,}\mathrm{V}\mathrm{D}\mathrm{C}$ on pins 4 and 5 and GND on
pins 7 and 8 and is connected to the power supply through a relay that allows
switching the power on and off. The relays are commanded by the TDR using
RS-232 messages transmitted via UTP cable, allowing up to four distributors in
a daisy-chain (required when more than four counters are present in a
station).

##  3 Acquisition and Control System

In this section we present the functional structure of the Data Acquisition
and Control (DAQ) system of the UMDs. We describe the implementation at the
hardware level on a Cyclone IV FPGA, the timing issues and memory requirements
for the tasks performed by the DAQ system, including the firmware of each UMD
and the soft microprocessor platform used to implement every function required
by the UMD. We also describe the power system distribution and consumption of
the electronics, the interface required and finally a short description of the
printed circuit board layout and how it was designed. As we will see in this
section the constraints of speed and memory are significant, therefore we
emphasize the need for a soft microprocessor platform in order to have easy
access to upgrades or future hardware changes that may be required due to
spares availability. In this way we have a flexible and scalable embedded
system that can be used for any kind of particle or radiation detector that
works synchronously. The Acquisition and Control System is composed by two
distinct parts: A soft microprocessor and custom acquisition code (both
detailed below). A soft microprocessor was chosen to simplify the electronics,
reducing the component count and improving the electronics future-proofing.

###  3.1 Hardware

In this section we describe the back-end digital hardware used for data
acquisition on each UMD. We introduce the basic structure of the main system,
comprised of an FPGA and a memory that functions as a ring buffer storing data
according to a trigger provided by the SD. The main CPU controls the interface
and data transfer to the surface, as well as monitoring several key hardware
parameters that indicate its good performance. As the main motivation for the
design, we should mention the need for a compact flexible embedded system that
can easily be reprogrammed remotely, as we will see in the following sections.
As mentioned, the CPU and firmware were implemented using an FPGA connected to
two physically independent LPDDR1 memories. This board has an expansion port
with $150\text{\,}$ pins using a $1.8\text{\,}\mathrm{V}$ single ended voltage
standard for fast speed digital signals. Additionally, one RJ-45 connector
receives the fast trigger (T1) and the clock line with LVDS receivers, another
one uses a LVDS transmitter/receiver pair for serial communication. These LVDS
drivers can be fed by power from the connected cable, or by power from the
Acquisition board, selected by a jumper. All input/output signals from LVDS
transmitters/receivers are electrically isolated. In the following subsections
we describe the hardware used with its specifications in terms of timing,
resources and memory usage, the printed circuit board design and the interface
with the SD and the telecommunications system.

Figure 5 shows a picture of the top and bottom side of the assembled board. It
depicts the layout of the board and indicates the locations of the main
connectors and components.

Figure 5: Acquisition board, showing the main connectors and components of the
top side (left), and the front-end connector located at the bottom (right).

####  3.1.1 FPGA and memory

The board is built around an Altera Cyclone IV FPGA (model EP4CE40U19I7) which
provides $39\,600\text{\,}$ logic cells and a memory of
$1.16\text{\,}\mathrm{M}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$. Table 2
summarizes the usage of these resources for the custom logic (Firmware) and
the LEON3 softcore CPU. While only $18\text{\,}\%$ of the logic cells are not
used, there is nearly half of the memory cells ($48\text{\,}\%$) left for
future improvements.

The chosen industrial package with $329\text{\,}$ I/O pins (of which
$257\text{\,}$ are used) works in a wide temperature range
($-40\leavevmode\nobreak\ ^{\circ}$C to $+100\leavevmode\nobreak\ ^{\circ}$C),
which is adequate for the operating conditions of the electronics (buried
underground). The FPGA series is optimized for low power consumption with a
low core voltage of $1.2\text{\,}\mathrm{V}$. The firmware incorporates the
Altera Remote Update system [16] with a remote recovery option, which ensures
additional safety in case of an upgrade failure.

Component | Logic Cells | Memory bits
---|---|---
Firmware | 12156 ($30\text{\,}\%$) | 480211 ($41\text{\,}\%$)
LEON3 CPU | 20371 ($51\text{\,}\%$) | 133872 ($11\text{\,}\%$)
Total | 32527 ($82\text{\,}\%$) | 614083 ($52\text{\,}\%$)
Available | 39600 ($100\text{\,}\%$) | 1161216 ($100\text{\,}\%$)
Table 2: Resource usage of the FPGA image, split into LEON3 and Firmware.
These numbers were obtained from logs generated by the the Quartus II
compiler.

The board has two physically separated memories, one for the soft-core CPU and
one for the acquisition firmware. This design avoids logic elements for the
synchronized access to the two RAMs, thus speeding-up the acquisition firmware
and saving resources. Both memories are Micron LPDDR (low power DDR, model
MT46H32M16LFBF-5), with $1\text{\,}\mathrm{G}\mathrm{B}$ of capacity and a
working frequency of up to $200\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ (Both
memories are used at $100\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$).

Figure 6: Close-up of the PCB Layout focusing on the FPGA (“U1”) and LPDDR
memories (“U4” and “U5”).

####  3.1.2 Power sources

Based on the estimates for the required ratings of the different devices, we
designed a cascaded high-efficient power system as shown in figure 7.

Figure 7: Power Supply diagram. Indicated are the maximum power and current supported for each component.  Voltage [V] | Current [mA] | Power [W]
---|---|---
24 | 75 | 1.800
3.3 | 210 | 0.693
1.8 | 44 | 0.079
2.5 | 46 | 0.115
1.2 | 236 | 0.283
Table 3: Power consumption for each of the outputs shown in figure 7.

The board uses a $3\text{\,}\mathrm{W}$ isolated DC/DC-converter RS3-2405S to
avoid ground loops. The device converts the nominal $24\text{\,}\mathrm{V}$
input into a $5\text{\,}\mathrm{V}$ output (max $0.6\text{\,}\mathrm{A}$).
This output is converted by the LM46002 DC/DC-regulator to a stable
$3.3\text{\,}\mathrm{V}$ supply (max $2\text{\,}\mathrm{A}$). With this
$3.3\text{\,}\mathrm{V}$ as input, further step-down DC/DC regulators
TPS62150, TPS62230 and TPS62231 generate $1.2\text{\,}\mathrm{V}$ (max
$1\text{\,}\mathrm{A}$), $2.5\text{\,}\mathrm{V}$ (max 0.5A) and
$1.8\text{\,}\mathrm{V}$ (max $0.5\text{\,}\mathrm{A}$) with an efficiency of
typically $95\text{\,}\%$, respectively. The $3.3\text{\,}\mathrm{V}$ and
$1.8\text{\,}\mathrm{V}$ power supplies are connected to the FPGA I/O blocks,
providing power and reference voltage. Table 3 show the nominal values of
current and power consumption as measured in the laboratory for each of the
outputs in figure 7.

####  3.1.3 Interface and Communications

The board provides two expansion connectors: one for data/logic signals and
one for power supplies. The data expansion connector is a single high-speed
connector, Samtec QSH-090-01-LDAK-TR, capable of transmitting signals with a
maximum frequency of $9\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$. It also has a
central ground bar that ensures low resistance for the return currents through
the connector. The connector has 180 lines, with a pitch between pins of
$0.5\text{\,}\mathrm{m}\mathrm{m}$.

The power expansion connector provides the $24\text{\,}\mathrm{V}$ input power
(directly from the PPoE connector) and the $3.3\text{\,}\mathrm{V}$ power
supply. Additionally, two control lines provide the functionality to switch on
or off the connected expansion board.

The Ethernet PHY driver is a DP83848IV from Texas Instruments, which is
supported by the Leon 3 built-in Ethernet MAC and the Linux operating system.
The MAC address of each board is divided in two parts: The most significant 11
nibbles (44 bits) are a firmware compile-time constant, and the lowest nibble
(4 bits) can be set using four switches in the board.

For the monitoring of the system voltages an ADS7828 ADC from Texas
Instruments was chosen. It has 8 multiplexed input channels, uses an internal
reference voltage, and supports I2C communications for transferring monitoring
data to the soft-core.

####  3.1.4 Printed circuit board

Based on the maximum working frequency within the board ($\sim
200\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$) it was decided that the
appropriate material for this design is FR4, which can accommodate working
frequencies of up to $500\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$. A 12-layer
design was made, following the recommendations of the FPGA and RAM
manufacturers. The FPGA recommendations specify tracks with a minimum width of
$5\text{\,}\mathrm{m}\mathrm{i}\mathrm{l}\mathrm{s}$ and a characteristic
impedance of $50\text{\,}\mathrm{\SIUnitSymbolOhm}$. The impedance of the
lines depends on the materials (dielectrics and conductors) and their
geometry, that is, track width, thickness of the dielectric, among others. A
standard thickness ($1.6\text{\,}\mathrm{m}\mathrm{m}$) FR4 PCB with 12 layers
does not allow $50\text{\,}\mathrm{\SIUnitSymbolOhm}$ lines, therefore a
thicker ($2.4\text{\,}\mathrm{m}\mathrm{m}$ ) PCB was used.

Following the application notes of Altera, one LPDDR memory was connected to a
whole FPGA I/O bank. The other memory was connected using part of a bank for
the address pins and part of another for the data bus, as indicated by the
FPGA manual [17]. Once this was defined, the routing of the tracks from the
FPGA to the memories was done.

The tracks between RAM and FPGA must be striplines of controlled impedance and
length to conserve signal integrity and the timing constraints imposed by the
RAM. The track width and distance between the signal and reference layers
needed for a specific impedance was determined by specialized programs taking
into account the properties of the PCB material. They must also have equalised
lengths to ensure that the propagation times of the signals are within the
values recommended by the RAM manufacturer.

###  3.2 Soft microprocessor LEON3

The back-end design went through several iterations before arriving to the
final design presented in this paper. Through this debugging process we
concluded that the flexibility of the hardware was essential, as well as the
remote control and configuration. After two prototypes were built, we chose to
use a LEON3 soft microprocessor running at $50\text{\,}\mathrm{MHz}$ that has
an open source architecture. It allows for the system to be fully programmable
remotely from the Observatory campus in Malargüe, which eased its debugging on
the field and allows for upgrades or changes in the firmware to accommodate
future needs such as changes in the trigger system, calibration, monitoring
and disabling malfunctioning parts of the hardware if necessary. LEON3 is a
32-bit CPU microprocessor core, based on the Scalable Processor Architecture
Version 8 (SPARC-V8) instruction set architecture with an AMBA AHB system bus
[18]. It was originally designed by the European Space Research and Technology
Centre (ESTEC), part of the European Space Agency (ESA), and after that by
Cobham Gaisler Research. It is described in synthesizable VHDL.

This Soft microprocessor has numerous advantages, particularly for a such
long-running project: The base code (CPU without Floating Point support and
most peripherals) is available under a free-software license (GPLv2), it is
FPGA-independent (upstream code supports Altera, Xilinx and Microsemi FPGAs),
comes with a complete library of peripherals, and allows a wide array of
configuration options (both in adding and removing peripherals and selecting
or removing CPU features to prioritize performance or resource usage). The
peripherals used are: SPI Memory controller (used for early boot and remote
Firmware upgrade), I2C (used for environmental sensors), RS-232 serial ports
(used for High Voltage power supply and acquisition control/data transfer),
GPIO ports (used for the front-end power supply control, reboot/power-off and
watchdog), and 10/100 Ethernet MAC (used for communications).

###  3.3 UMD firmware

In this section we introduce the details of the firmware functionality for a
UMD back-end. It handles the data flow and secondary tasks required of the UMD
hardware. We describe each part of the firmware in a block diagram divided
into three parts: the CPU core, the back-end main task of muon data storage
and the interface with the front-end and the peripherals. The current FPGA
implementation is divided into six main blocks, as shown in figure 8. The EMC
block drives $128\text{\,}\mathrm{M}\mathrm{i}\mathrm{B}$ of external LPDDR1
memory for temporary event data storage. This data has a fixed size
($2048\text{\,}$ anode samples for each SiPM, plus data from both ADCs and the
aux digital outputs from both ASICs, equivalent to
$6.4\text{\,}\mathrm{\SIUnitSymbolMicro s}$ per event). The trigger block
receives the T1 signal, immediately notifies the incoming event condition to
the data block and forwards the LTS value corresponding to the incoming event
once it has been fully received. The read-out stage processes the
$64\text{\,}$ digital inputs at $320\text{\,}\mathrm{MHz}$ and packs four
samples into a single $256\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ word. To
reduce the number of resources needed, in particularly interconnects, all
firmware except for the readout process runs at $80\text{\,}\mathrm{MHz}$. The
data acquisition block stores
$2048\text{\,}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{s}$
of these $256\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ words in two alternating
circular (internal) buffers. When a trigger arrives, the actual buffer is
filled up and then switched to the alternative buffer. While the alternate
buffer records the data, the recorded data is saved in the external RAM. The
LTS table process manages a look-up table with 2048 entries. When addressed
with an LTS of an event, the memory address of the event data block is
returned, or an ‘event not found’ error is raised [14].

The ASIC programming block receives the ASIC configuration from the embedded
system and then uses it to program both ASICs. The micro-controller interface
block handles communications with the embedded system. This is implemented
using serial ports. Finally, the coordinator block manages the rest of the
blocks.

Figure 8: Block diagram of the UMD electronics. We can observe in the
acquisition board: in green the CPU blocks, in blue the back-end blocks and in
red the peripheral blocks.

###  3.4 Remote upgrade

In order to have a flexible distributed system we need to provide a robust
firmware upgrade system. Therefore we included in the main firmware a remote
upgrade that can be performed by an operator and provide a failsafe if there
is any problem during the upgrade. In this way we can remotely enhance and
develop the main functions of the UMD. Altera Remote Upgrade works as follows:
The hardware stores two FPGA images in a persistent memory, named “Factory”
and “Application” in Altera literature. On FPGA power-on, the Factory image is
programmed, which starts a watchdog and immediately programs the Application
image. If the watchdog isn’t periodically reset (at least once every 60
seconds), the Factory image is reprogrammed in the FPGA and takes over. Both
firmwares can be identified by software via a GPIO pin. The Application image
contains both a LEON3 and AMIGA code, while the Factory image contains only a
LEON3. The watchdog is controlled by software, and the routines to reset it
start late in the boot process. If the module boots in Factory mode, the
software tries to re-enter Application mode ten minutes after booting has
completed.

Firmware upgrades are done by writing a specially processed FPGA image into
the SPI Flash. To prevent an overwriting of deployed software images by
mistake, all Flash regions except the one containing the FPGA Application
image are set write-protected.

##  4 Software overview

This section describes how the software in the UMD is implemented, the tasks
provided to the UMD and the operating system used. The software is responsible
for sending event data and monitoring data via a network to a remote location.
To simplify and speed-up the development phase, this software runs over an
operating system which also provides system tools. The system was designed to
minimise the use of persistent storage in the module electronics, improving
resiliency, easing electronics installation and replacement, and reducing the
possibility of “bricking” the electronics on future software updates (the only
persistent data on every module electronic is LEON3, Firmware, boot-loader and
Ethernet MAC address). A description of the software, the boot process and the
SDK follows.

###  4.1 Operating System

Following the open source philosophy adopted for the soft microprocessor, the
operating system used is also open source. The underground scintillator
detector modules and the TDRs use Linux [19] as the operating system. It was
chosen for user familiarity, relative ease of use, and because it has full
hardware support for LEON3 (CPU and required peripherals). Therefore the
motivations for implementing the software in this manner is to facilitate
remote upgrades and configuration, with a complete access to the hardware
functionality of the UMD electronics deployed in the field. The current
version used at the time of the writing of this paper is 4.9, with patches
(provided by Cobham Gaisler) for hardware support forward-ported to 4.9.
Additionally, the kernel was patched to use the built-in GPIO power-off
driver, which has been-setup to instruct the FPGA to program the Application
image. The system runs in RAM from an initcpio (embedded in the kernel image
due to bootloader limitations). Loadable module support is disabled (all
modules are compiled into the kernel image) because no out-of-tree kernel
modules are used, the connected hardware is known in advance, and disabling
module support results in smaller kernel and initcpio binaries.

####  4.1.1 Boot

Upon board power-on/CPU Reset, a boot-loader (uBoot 1.16 with Cobham Gaisler
patches) starts running. This boot-loader brings up the Ethernet port, obtains
an IP address via DHCP, and loads the full operating system image (specified
by the DHCP reply) from a TFTP server located in the Mikrotik RB493 router in
each station. The DHCP server runs on the central data storage server, with
DHCP relays running on each station LAN. The TFTP servers are located in each
station LAN to preserve WLAN bandwidth. This system allows for easy image
upgrades: Simply overwrite the image on the respective TFTP servers and reboot
the affected modules. Modules are assigned a fixed IP and hostname based on
their MAC address. This address has the upper 44 bits set by the software
image (common for all stations) and the lower 4 bits configurable by hardware-
switches, admitting at most 16 module electronics per station. Since each
station LAN is physically isolated and has its own network address, no MAC
address conflicts arise as long as MAC addresses are not repeated within the
same station LAN.

####  4.1.2 System Applications

All basic system applications (init, shell, etc) are provided by Busybox
(currently v1.33.1) [20]. The most notable applications are: Telnet daemon,
watchdog user-space application, NTP client, DHCP client and SPI flash. The
DHCP client has been configured to obtain the client hostname published by the
DHCP server (required by AMIGA software to identify the module), the station
LSID (used by the monitoring software to identify the station), and the NTP
server used to synchronize the internal time.

####  4.1.3 Software Development Kit (SDK)

The tool-chain used for software development (compiler, libc, linker and
related utilities) was custom-built. It consists of a GCC 8.1 C compiler
targeting by default the Leon3 architecture with software floating point
support, using binutils 2.30 and uCLibc-ng 1.30 C library. The tool-chain was
built for Ubuntu 18.04 running on Intel x86_64 compatible CPUs.

###  4.2 Software

In this section we describe in more detail the software structure and how it
performs the tasks needed for the UMD to store data and transfer it to the
CDAS. We organize this section into three main subsections that describe the
most important routines and their functionality: Module configuration, data
transfer, and calibration. The software was written to control the module and
to manage event and monitoring data acquisition. It consists of five pairs of
client-server applications. With the single exception of the Calibration
software (MdCalib), all clients run automatically after booting in Application
mode on each buried module electronics and all servers run automatically on
boot on a server in CDAS, with the clients keeping a persistent connection to
the server (unless specified). Additionally (and with the same exception), all
programs use the host-name to identify each individual module, allowing the
system to work behind a Network address translation (NAT). A brief description
of each pair of programs, with the names of each program client/server side,
follows.

Figure 9: Software communications diagram. All communications with the central
server go through TCP sockets, maximizing data integrity and minimizing data
loss.

####  4.2.1 Module configuration - MdCfg/MdCfgd

This software configures the front-end (CITIROCs, ADCs, and HV), the firmware
parameters and provides configuration/status information (ex: acquisition
mode, masked channels, etc) to other programs using a Unix socket. On start-
up, the client connects to the server and requests the initial module
configuration (stored in the central server in INI files). Once the
configuration data has been sent and applied, this daemon disconnects from the
server and keeps running in background, handling configuration and status
requests from all other programs (these requests will be specified in each
program’s section).

####  4.2.2 Data transfer - MdSend/MdRcvd

This pair of programs handle the event data transfer and storage. When idle,
the client periodically consults with MdCfg the current acquisition mode and
the UDP broadcast port for data requests (only if in acquisition modes sending
of data is requested). Upon a data request, the software instructs the
firmware to search for the specified event, receiving either the event data or
an error code, and sends the result to the server. The server keeps in RAM a
dictionary of events indexed by the 16-bit internal CDAS Event ID. For each
Event ID, the data for all participating counters is stored in a dictionary
indexed by the counter ID. The server has a TTL counter for each event, which
is decreased every 30 seconds and increased every time data for a new module
arrives. Once this counter reaches zero, the event is written to disk and
discarded from RAM (if more data for the same event arrives, it is treated as
a new event and merged as part of the standard offline post-processing of
AMIGA data). The event data is stored in JSON format, using a single file
which is periodically restored and compressed (in gzip format) once per day.
This format allows easy parsing using libraries available in most programming
languages, and has a very good compression ratio when using gzip [21].

####  4.2.3 Calibration - MdCalib

These programs run the pre-defined Calibration routines on each module as
described in [11]. On the module, MdCalib starts on boot and listens to
calibration requests. Once a calibration request is received, the program
acquires the file-lock for background data, sets the acquisition mode to
‘Calibration’ (all data requests are sent to the server indicating that the
sent data is invalid due to the module being calibrated) and runs the
requested calibration routine, setting the needed configuration parameters via
MdCfg and sending the resulting background data to the requester. Once the
calibration finishes, the module is returned to its previous state and the
file-lock is released. On a remote machine, a dedicated program communicates
with the calibration software running on one or more modules (identified by IP
address) and receives the calibration data. Currently the request for
calibration date must be issued manually by an operator, but full automation
(via scheduled jobs) can be used.

###  4.3 Monitoring system

To guarantee the validity of the acquired data, the status of the detector, as
well as the status of its measured data, have to be monitored. In this section
we include a description of the monitoring system as implemented in the UMD
and the main server. The motivation for the monitoring system is described
below, in summary it is designed to perform two very important tasks:
Background signal monitoring, where the record of the signal counts by the UMD
is constantly taken, and environmental monitoring, where the hardware status
(including power sources voltages, currents and temperature) is handled. The
monitoring acquisition system was designed as a series of processes (one for
each kind of monitoring) using client/server architectures, where each of the
clients runs in a MC electronic and establishes a TCP connection to its
corresponding server. The resources needed by the monitoring processes are
extremely low, particularly from the client side: they use
$1\text{\,}\mathrm{M}\mathrm{B}$ of RAM that is activated every 5 minutes to
send data to the server at the CDAS. The data volume sent is
$1\text{\,}\mathrm{K}\mathrm{B}\mathrm{y}\mathrm{t}\mathrm{e}$ every
$30\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{s}$.
A description of these monitoring processes follows.

####  4.3.1 Background signal monitoring - backpulse_client/backpulse_server

These programs read and store the background pulse counters from the firmware.
Background pulses are all the current pulses produced by the SiPM (not
necessarily belonging to a possible event). The rate of these pulses for a
given SiPM is expected to be relatively constant throughout the operating
lifetime of the detector; variations on this rate over time may indicate aging
or damage of any of the optical parts. The client periodically orders the
firmware to snapshot the current counter values, which are then sent to the
server and saved in space-separated files (one file per day and module),
formatted with the module time-stamp and the 64 counter values. Since the
background data client shares resources with the calibration daemon, this
program tries to acquire a lock-file in a predetermined location before
reading the counter data, and immediately closes this file after reading.

####  4.3.2 Environmental monitoring -
monitoring_client/monitoring_server/data_handler

This trio of programs handle the monitoring data readout and storage. The
client periodically reads the data directly from the monitoring ADCs and data
from the HV power supply via MdCfg, and sends it to the monitoring server upon
request. Monitoring data is taken every
$5\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{s}$.
The server spawns a new process for each connection request from a client
which periodically requests its monitoring data. Every
$30\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{s}$
up to $432\text{\,}\mathrm{B}\mathrm{y}\mathrm{t}\mathrm{e}\mathrm{s}$ of
monitoring data plus a header of
$35\text{\,}\mathrm{B}\mathrm{y}\mathrm{t}\mathrm{e}\mathrm{s}$ is sent per
station. This data is then immediately sent to the data_handler process via a
unix socket. The data_handler process stores all the monitoring data it
receives from its unix socket to a MySQL database and, as a backup, to a .csv
file (rotated every
$12\text{\,}\mathrm{h}\mathrm{o}\mathrm{u}\mathrm{r}\mathrm{s}$) that can be
imported manually into the database.

###  4.4 Online-Monitoring

At the server side, the monitoring data is received and stored in a database.
In this section we describe the monitoring data handling from all the UMDs
deployed in the field and how it is presented to the user in order to
visualize it and make decisions accordingly.

Figure 10: Screen capture of the monitoring system (integrated within the
Auger monitoring system) displaying UMD stations. Green ticks indicate
stations with buried modules in normal operating status. The red hexagon
circumscribes the $750\text{\,}\mathrm{m}$ infill and the blue hexagon
circumscribes the $433\text{\,}\mathrm{m}$ infill. Displayed are the stations
deployed at the time of publication of this paper. The station in yellow that
shows a warning is displayed in detail in figure 11.

The monitoring system allows access to the most relevant information of the
different components of the MC via a web-interface. It also enables the
visualisation and analysis of the environmental and trigger monitoring
parameters over time, which can be very useful to find anomalies that could
indicate the presence of failures in some component of the MC.

Figure 11: Screen capture of the monitoring system, displaying installation
and monitoring data for an UMD. As can be observed, module 103 shows a warning
as three of its channels (5, 7 and 27) are not operational.

The main page of the monitoring system shows a Map option with the position of
each SD station. Figure 10 shows an example, where the coloured marker depends
on the alarm state of all the scintillator modules for each SD station. In the
figure, green marks represent a station working in normal state, and gray
marks represent stations that have yet to be installed.

Selecting a SD station shows the deploy view associated to this station and
the monitoring data from the database for a particular date. An example of the
monitoring system displaying a typical AMIGA station after deployment is shown
in figure 11. Selecting a module shows the monitoring information and alarm
state for each variable of the chosen module. This view also gives the option
to mask alarms of any monitoring parameters and visualise the full monitoring
history of the corresponding module.

##  5 AMIGA sample event

As an example of the acquisition process, this section presents data collected
by the AMIGA UMDs for an UHECR of
$4\text{\times}{10}^{18}\text{\,}\mathrm{e}\mathrm{V}$ impinging with a
$20\text{\,}$∘ zenith angle222Values taken from the official Pierre Auger
Observatory event reconstruction, that hit the border of the muon counters.
This event was recorded on May 9, 2018 at 14:44:56 UTC. The temperature
measured in the SiPMs for the four modules belonging to station 1764 (which
participated in the event) is shown in figure 12, taken form the monitoring
system. Event data for each AMIGA station is shown in this section to
illustrate the reconstruction procedure applied to the binary acquisition mode
only333As the time of writing this paper, the reconstruction with the
integrator mode is still under development.

Figure 12: Time plot of the SiPM Temperature Sensor (in
$\text{\,}\mathrm{\SIUnitSymbolCelsius}$) for four AMIGA modules of station
with ID 1764. The 12-day time period includes the moment of the presented
event (at the solid vertical line).

###  5.1 Status of the AMIGA Array at the time of the event recording

As seven stations of the $750\text{\,}\mathrm{m}$ array were part of the
prototyping phase (the so called Unitary Cell), the first operative hexagon
for physics analysis has stations with higher segmentation (256 bars instead
of 192) for the projected $30\text{\,}{\mathrm{m}}^{2}$ detection area as
shown in figure 13. Two of these stations have four buried modules (two
$10\text{\,}{\mathrm{m}}^{2}$ modules and two $5\text{\,}{\mathrm{m}}^{2}$
modules). One of these stations has eight modules (four
$10\text{\,}{\mathrm{m}}^{2}$ modules and four $5\text{\,}{\mathrm{m}}^{2}$
modules) installed symmetrically with respect to the Auger Surface Detector;
also known as a “Twin” (for validation purposes and to assess the resolution
of the reconstruction procedure as described in [22]). One station has six
modules (four $10\text{\,}{\mathrm{m}}^{2}$ modules and two
$5\text{\,}{\mathrm{m}}^{2}$ modules), with the southern side of the station
installed like the others, and the northern side with two
$10\text{\,}{\mathrm{m}}^{2}$ modules. The rest of the stations have one or
two $10\text{\,}{\mathrm{m}}^{2}$ modules.

Figure 13: Status of the AMIGA Engineering Array at March 2018. Each station
is identified by a number (“Station ID”). For each station the size and
relative orientation of the AMIGA muon counter is sketched. The size of the
modules represent the the detection area ($10\text{\,}{\mathrm{m}}^{2}$ or
$5\text{\,}{\mathrm{m}}^{2}$). Drawing is not to scale.

###  5.2 UMD module binary data

AMIGA event data for each buried scintillator module is stored as lists of bin
number/anode sample, only for samples where at least one channel has detected
light (all other samples are not stored), and a list of bin number/sampled
value for each ADC. The anode bin number indicates a time-stamp in
$3.125\text{\,}\mathrm{n}\mathrm{s}$ increments. These traces have a length of
2048 bins ($6.4\text{\,}\mathrm{\SIUnitSymbolMicro s}$). The length of the
pre-trigger / post-trigger window is a configurable parameter of the
electronics. In this example, the pre-trigger and post-trigger windows are
about $4.4\text{\,}\mathrm{\SIUnitSymbolMicro
s}$/$2\text{\,}\mathrm{\SIUnitSymbolMicro s}$ long, thus the trigger arrived
at bin $1400\text{\,}$.

Muon counts are obtained from these raw traces (shown as illustration in top
panels of figure 14) in an offline analysis, searching for patterns in the
signals of each channel using the technique described in [11]. A counting
strategy must be applied to avoid undercounting from pile-up effects [5].

Figure 14: Example of acquisition for two modules in two different stations.
(left) Plots of acquired event data for the presented event on a non-saturated
AMIGA buried muon counter module (1622 south M1) with 29 pulses. (right) Plots
of acquired event data for the presented event on a saturated AMIGA buried
moun counter module (1570 south M1) with 64 pulses. The top graphic shows a
black rectangle on each channel that had signal over threshold in a given time
bin. The bottom graphic shows the sum of all channels with signal over
threshold for a given time bin. The red line indicates the level when all 64
channels are active. A station is considered to be saturated based on the
amount of active channels for a time bin.

As an example of the reconstruction chain applied to extract a number of muons
from the raw data, figure 14 shows the traces collected by two AMIGA modules
at different distances from the shower core for the same event. It is apparent
how the particle density and therefore the triggered raw traces decreases as
the distance increases. In particular, the figure shows an extreme case for a
very close-to-core detector at $145\text{\,}\mathrm{m}$ distance. In this
case, the UMD is saturated444In the binary mode as all 64 channels in three of
its modules are simultaneously triggered. To identify the number of active
channels, i.e. the number of channels with at least four positives samples in
the raw traces in a time window of 18 bins, is the first step in the
reconstruction procedure [11]. For the station 1570, only one of the larger
modules (M1) of $10\text{\,}{\mathrm{m}}^{2}$ is not saturated and measures
471 particles. On the other hand, for the $10\text{\,}{\mathrm{m}}^{2}$ M1
(south) buried module of the detector 1622 (at about $600\text{\,}\mathrm{m}$
of the shower core) the active channels are 29. Applying the pile-up
correction, this number of active channels translate into 33.7 reconstructed
muons. This correction is applied to the number of active channels, and makes
this number differ from the reconstructed muons. The method relies on a
probabilistic model to estimate the number of particles, reporting the mean
number of muons for a given number of input signals (or active channels).

As a final remark, it is important to stress that the detected particles
arrived in a time window of about $200\text{\,}\mathrm{ns}$ and that there are
no identified particles before or after this window.

The reconstructed number of muons for all AMIGA modules triggered in this
sample event is presented in table 4. These counts were obtained using the
Auger Offline software [23], which applies the already described counting
strategy and the pile-up correction described in [5].

Station |  | Distance
---
to axis (m)
$\mu$ density $[\frac{\mu}{$\text{\,}\mathrm{m}^{2}$}]$ |  | M1
---
($10\text{\,}{\mathrm{m}}^{2}$)
| M2
---
($10\text{\,}{\mathrm{m}}^{2}$)
| M3
---
($5\text{\,}{\mathrm{m}}^{2}$)
| M4
---
($5\text{\,}{\mathrm{m}}^{2}$)
1570 S | 145 | 44.86 | 471.1 | - | - | -
1622 S | 603 | 2.69 | 33.7 | 29.4 | 10.1 | 11.3
1622 N | 603 | 2.84 | 34.7 | 25.0 | - | -
688 S | 633 | 3.17 | 41.8 | 24.8 | - | -
1574 S | 669 | 2.10 | 19.7 | 25.0 | 9.2 | 12.2
1764 S | 1114 | 0.41 | 4.0 | 5.0 | 3.0 | 1.0
1764 N | 1114 | 0.44 | 6.1 | 3.0 | 1.0 | 4.0
93 N | 1170 | 0.22 | 1.0 | 2.0 | 1.0 | 3.0
1773 S | 1344 | 0.00 | 0.0 | - | - | -
Table 4: Muon counts calculated for the presented event, separated by station,
and analysing twin detectors separately.  Figure 15: MLDF (Muon Lateral
Distribution Function) of the studied event, including a fit (dashed blue
line) of the calculated data. This event has data points for distances to the
core smaller and greater than $450\text{\,}\mathrm{m}$, which results in a
better quality fit.

After the muon densities as a function of the distance are reconstructed, the
observations are fit to obtain the estimator $\rho_{\mu}(450)$, the muon
density at $450\text{\,}\mathrm{m}$ from the shower core. This observable is
sensitive to the chemical composition of the primary cosmic ray entering the
Earth atmosphere and is the one used for higher-level physical analysis.
Combining this parameter with others from the shower reconstruction (as the
energy, X${}_{\textrm{max}}$, etc), gives valuable information regarding the
primary particle mass. Figure 15 shows the fit of the muon lateral
distribution function made for the event studied following the procedure in
[24].

##  6 Conclusions

The AMIGA front-end electronics was designed as an embedded system around the
Cylone FPGA family. A soft-core LEON3 processor runs Linux as its operating
system for high flexibility, short development time and other benefits. A
front-end compatible with this board was designed and built for AMIGA buried
muon counters using SiPMs as the sensing device and two CITIROC ASICs for
signal conditioning. The system is capable of 64 SiPM channels sampled at
frequencies of $320\text{\,}\mathrm{M}\mathrm{s}\mathrm{p}\mathrm{s}$ and two
$14\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ ADCs (high and low gain) measured
at $160\text{\,}\mathrm{M}\mathrm{s}\mathrm{p}\mathrm{s}$.

The first eight prototypes of this system were installed in October 2016.
Afterwards, 53 electronics were built and are acquiring data since February
2018. After several tests, this design will be produced to complete the AMIGA
muon counters, by installing three $10\text{\,}{\mathrm{m}}^{2}$ modules in 61
stations.

## Acknowledgments

The successful installation, commissioning, and operation of the Pierre Auger
Observatory would not have been possible without the strong commitment and
effort from the technical and administrative staff in Malargüe. We are very
grateful to the following agencies and organizations for financial support:

Comisión Nacional de Energía Atómica, Agencia Nacional de Promoción Científica
y Tecnológica (ANPCyT), Consejo Nacional de Investigaciones Científicas y
Técnicas (CONICET), Gobierno de la Provincia de Mendoza, Municipalidad de
Malargüe, NDM Holdings and Valle Las Leñas, in gratitude for their continuing
cooperation over land access, Argentina; the Australian Research Council;
Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq),
Financiadora de Estudos e Projetos (FINEP), Fundação de Amparo à Pesquisa do
Estado de Rio de Janeiro (FAPERJ), São Paulo Research Foundation (FAPESP)
Grants No. 2010/07359-6 and No. 1999/05404-3, Ministério de Ciência e
Tecnologia (MCT), Brazil; Grant No. MSMT CR LG15014, LO1305 and LM2015038 and
the Czech Science Foundation Grant No. 14-17501S, Czech Republic; Centre de
Calcul IN2P3/CNRS, Centre National de la Recherche Scientifique (CNRS),
Conseil Régional Ile-de-France, Département Physique Nucléaire et
Corpusculaire (PNC-IN2P3/CNRS), Département Sciences de l’Univers (SDU-
INSU/CNRS), Institut Lagrange de Paris (ILP) Grant No. LABEX ANR-10-LABX-63,
within the Investissements d’Avenir Programme Grant No. ANR-11-IDEX-0004-02,
France; Bundesministerium für Bildung und Forschung (BMBF), Deutsche
Forschungsgemeinschaft (DFG), Finanzministerium Baden-Württemberg, Helmholtz
Alliance for Astroparticle Physics (HAP), Helmholtz-Gemeinschaft Deutscher
Forschungszentren (HGF), Ministerium für Wissenschaft und Forschung, Nordrhein
Westfalen, Ministerium für Wissenschaft, Forschung und Kunst, Baden-
Württemberg, Germany; Istituto Nazionale di Fisica Nucleare (INFN),Istituto
Nazionale di Astrofisica (INAF), Ministero dell’Istruzione, dell’Universitá e
della Ricerca (MIUR), Gran Sasso Center for Astroparticle Physics (CFA),
CETEMPS Center of Excellence, Ministero degli Affari Esteri (MAE), Italy;
Consejo Nacional de Ciencia y Tecnología (CONACYT) No. 167733, Mexico;
Universidad Nacional Autónoma de México (UNAM), PAPIIT DGAPA-UNAM, Mexico;
Ministerie van Onderwijs, Cultuur en Wetenschap, Nederlandse Organisatie voor
Wetenschappelijk Onderzoek (NWO), Stichting voor Fundamenteel Onderzoek der
Materie (FOM), Netherlands; National Centre for Research and Development,
Grants No. ERA-NET-ASPERA/01/11 and No. ERA-NET-ASPERA/02/11, National Science
Centre, Grants No. 2013/08/M/ST9/00322, No. 2013/08/M/ST9/00728 and No.
HARMONIA 5 – 2013/10/M/ST9/00062, Poland; Portuguese national funds and FEDER
funds within Programa Operacional Factores de Competitividade through Fundação
para a Ciência e a Tecnologia (COMPETE), Portugal; Romanian Authority for
Scientific Research ANCS, CNDI-UEFISCDI partnership projects Grants No.
20/2012 and No.194/2012 and PN 16 42 01 02; Slovenian Research Agency,
Slovenia; Comunidad de Madrid, Fondo Europeo de Desarrollo Regional (FEDER)
funds, Ministerio de Economía y Competitividad, Xunta de Galicia, European
Community 7th Framework Program, Grant No. FP7-PEOPLE-2012-IEF-328826, Spain;
Science and Technology Facilities Council, United Kingdom; Department of
Energy, Contracts No. DE-AC02-07CH11359, No. DE-FR02-04ER41300, No. DE-
FG02-99ER41107 and No. DE-SC0011689, National Science Foundation, Grant No.
0450696, The Grainger Foundation, USA; NAFOSTED, Vietnam; Marie Curie-
IRSES/EPLANET, European Particle Physics Latin American Network, European
Union 7th Framework Program, Grant No. PIRSES-2009-GA-246806; and UNESCO.

## References

  * [1] Pierre Auger collaboration, _The Pierre Auger Cosmic Ray Observatory_ , _Nucl. Instrum. Methods Phys. Res._ 798 (2015) 172.
  * [2] I. Allekotte, A. Barbosa, P. Bauleo, C. Bonifazi, B. Civit, C. Escobar et al., _The surface detector system of the Pierre Auger Observatory_ , _Nucl. Instrum. Methods Phys. Res._ 586 (2008) 409.
  * [3] A. Etchegoyen, P. Bauleo, X. Bertou, C. Bonifazi, A. Filevich, M. Medina et al., _Muon-track studies in a water Cherenkov detector_ , _Nucl. Instrum. Methods Phys. Res._ 545 (2005) 602.
  * [4] Pierre Auger collaboration, _Design, upgrade and characterization of the silicon photomultiplier front-end for the AMIGA detector at the Pierre Auger Observatory_ , _Journal of Instrumentation_ 16 (2021) P01026 [2011.06633].
  * [5] A. Supanitsky, A. Etchegoyen, G. Medina-Tanco, I. Allekotte, M. Gómez Berisso and M. Medina, _Underground muon counters as a tool for composition analyses_ , _Astroparticle Physics_ 29 (2008) 461.
  * [6] Pierre Auger collaboration, _The AMIGA infill detector of the Pierre Auger Observatory: Performance and first data_ , in _Proceedings of the 32nd International Cosmic Ray Conference_ , vol. 1, pp. 267–270, 1, 2011, DOI.
  * [7] Pierre Auger collaboration, _The AMIGA muon detectors of the Pierre Auger Observatory: overview and status_ , in _Proceedings of the 33rd International Cosmic Ray Conference_ , pp. 712–715, 2013, 1307.5059.
  * [8] M. Platino, M. R. Hampel, P. Fiszelew, A. Almela, A. Sedoski, G. D. L. Vega et al., _AMIGA at the Auger Observatory: the telecommunications system_ , _Journal of Instrumentation_ 8 (2013) P12014.
  * [9] A. Cancio Montbrun, A. Mancilla, J. Maya, B. García, A. Almela, B. Andrada et al., _Photovoltaic monitoring system for Auger Muons and Infill for the Ground Array_ , _Energy Science & Engineering_ 6 (2018) 289.
  * [10] Pierre Auger collaboration, _Prototype muon detectors for the AMIGA component of the Pierre Auger Observatory_ , _Journal of Instrumentation_ 11 (2016) P02012 [1605.01625].
  * [11] Pierre Auger collaboration, _Muon counting using silicon photomultipliers in the AMIGA detector of the Pierre Auger Observatory_ , _Journal of Instrumentation_ 12 (2017) P03002.
  * [12] Pierre Auger collaboration, _Trigger and aperture of the surface detector array of the Pierre Auger Observatory_ , _Nucl. Instrum. Methods Phys. Res. A_ 613 (2010) 29 [1111.6764].
  * [13] Pierre Auger collaboration, _The Pierre Auger Observatory upgrade - Preliminary design report_ , 2016.
  * [14] O. Wainberg, A. Almela, M. Platino, F. Sanchez, F. Suarez, A. Lucero et al., _Digital electronics for the Pierre Auger Observatory amiga muon counters_ , _Journal of Instrumentation_ 9 (2014) T04003 [1312.7131].
  * [15] Pierre Auger collaboration, _New electronics for the surface detectors of the Pierre Auger Observatory_ , _Nucl. Instrum. Methods Phys. Res. A_ 824 (2016) 302.
  * [16] Altera Corporation, _Configuration and Remote System Upgrades in Cyclone IV Devices_.  https://www.altera.com/en_US/pdfs/literature/hb/cyclone-iv/cyiv-51008.pdf.
  * [17] Intel Corporation, _External memory interfaces in Cyclone IV devices_.  https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/hb/cyclone-iv/cyiv-51007.pdf.
  * [18] Cobham Gaisler, _LEON3 Processor_.  https://www.gaisler.com/index.php/products/processors/leon3.
  * [19] _Linux_.  https://www.kernel.org/.
  * [20] _Busybox_.  https://busybox.net/.
  * [21] ECMA International, _ECMA-404 The JSON data interchange standard. 2nd edition_.  http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf, 2017\.
  * [22] Pierre Auger collaboration, _Direct measurement of the muonic content of extensive air showers between $\mathbf{2\times 10^{17}}$ and $\mathbf{2\times 10^{18}}\leavevmode\nobreak\ $eV at the Pierre Auger Observatory_, _Eur. Phys. J. C_ 80 (2020) 751.
  * [23] S. Argiro, S. L. C. Barroso, J. Gonzalez, L. Nellen, T. C. Paul, T. A. Porter et al., _The offline software framework of the Pierre Auger Observatory_ , _Nucl. Instrum. Meth._ A580 (2007) 1485 [0707.1652].
  * [24] D. Ravignani and A. Supanitsky, _A new method for reconstructing the muon lateral distribution with an array of segmented counters_ , _Astroparticle Physics_ 65 (2015) 1.

$\bullet$

## The Pierre Auger Collaboration

A. Aab80, P. Abreu72, M. Aglietta52,50, J.M. Albury12, I. Allekotte1, A.
Almela8,11, J. Alvarez-Muñiz79, R. Alves Batista80, G.A. Anastasi61,50, L.
Anchordoqui87, B. Andrada8, S. Andringa72, C. Aramo48, P.R. Araújo Ferreira40,
J. C. Arteaga Velázquez66, H. Asorey8, P. Assis72, G. Avila10, A.M. Badescu75,
A. Bakalova30, A. Balaceanu73, F. Barbato43,44, R.J. Barreira Luz72, K.H.
Becker36, J.A. Bellido12,68, C. Berat34, M.E. Bertaina61,50, X. Bertou1, P.L.
Biermannb, V. Binet6, T. Bister40, J. Biteau35, J. Blazek30, C. Bleve34, M.
Boháčová30, D. Boncioli55,44, C. Bonifazi24, L. Bonneau Arbeletche19, N.
Borodai69, A.M. Botti8, J. Brackd, T. Bretz40, P.G. Brichetto Orchera8, F.L.
Briechle40, P. Buchholz42, A. Bueno78, S. Buitink14, M. Buscemi45, K.S.
Caballero-Mora65, L. Caccianiga57,47, F. Canfora80,82, I. Caracas36, J.M.
Carceller78, R. Caruso56,45, A. Castellina52,50, F. Catalani17, G. Cataldi46,
L. Cazon72, M. Cerda9, J.A. Chinellato20, K. Choi13, J. Chudoba30, L.
Chytka31, R.W. Clay12, A.C. Cobos Cerutti7, R. Colalillo58,48, A. Coleman93,
M.R. Coluccia46, R. Conceição72, A. Condorelli43,44, G. Consolati47,53, F.
Contreras10, F. Convenga54,46, D. Correia dos Santos26, C.E. Covault85, S.
Dasso5,3, K. Daumiller39, B.R. Dawson12, J.A. Day12, R.M. de Almeida26, J. de
Jesús8,39, S.J. de Jong80,82, G. De Mauro80,82, J.R.T. de Mello Neto24,25, I.
De Mitri43,44, J. de Oliveira26, D. de Oliveira Franco20, F. de Palma54,46, V.
de Souza18, E. De Vito54,46, M. del Río10, O. Deligny32, A. Di Matteo50, C.
Dobrigkeit20, J.C. D’Olivo67, R.C. dos Anjos23, M.T. Dova4, J. Ebr30, R.
Engel37,39, I. Epicoco54,46, M. Erdmann40, C.O. Escobara, A. Etchegoyen8,11,
H. Falcke80,83,82, J. Farmer92, G. Farrar90, A.C. Fauth20, N. Fazzinia, F.
Feldbusch38, F. Fenu52,50, B. Fick89, J.M. Figueira8, A. Filipčič77,76, T.
Fodran80, M.M. Freire6, T. Fujii92,e, A. Fuster8,11, C. Galea80, C.
Galelli57,47, B. García7, A.L. Garcia Vegas40, H. Gemmeke38, F. Gesualdi8,39,
A. Gherghel-Lascu73, P.L. Ghia32, U. Giaccari80, M. Giammarchi47, M. Giller70,
J. Glombitza40, F. Gobbi9, F. Gollan8, G. Golup1, M. Gómez Berisso1, P.F.
Gómez Vitale10, J.P. Gongora10, J.M. González1, N. González13, I. Goos1,39, D.
Góra69, A. Gorgi52,50, M. Gottowik36, T.D. Grubb12, F. Guarino58,48, G.P.
Guedes21, E. Guido50,61, S. Hahn39,8, P. Hamal30, M.R. Hampel8, P. Hansen4, D.
Harari1, V.M. Harvey12, A. Haungs39, T. Hebbeker40, D. Heck39, G.C. Hill12, C.
Hojvata, J.R. Hörandel80,82, P. Horvath31, M. Hrabovský31, T. Huege39,14, J.
Hulsman8,39, A. Insolia56,45, P.G. Isar74, P. Janecek30, J.A. Johnsen86, J.
Jurysek30, A. Kääpä36, K.H. Kampert36, B. Keilhauer39, J. Kemp40, H.O.
Klages39, M. Kleifges38, J. Kleinfeller9, M. Köpke37, N. Kunka38, B.L. Lago16,
R.G. Lang18, N. Langner40, M.A. Leigui de Oliveira22, V. Lenok39, A.
Letessier-Selvon33, I. Lhenry-Yvon32, D. Lo Presti56,45, L. Lopes72, R.
López62, L. Lu94, Q. Luce37, J.P. Lundquist76, A. Machado Payeras20, G.
Mancarella54,46, D. Mandat30, B.C. Manning12, J. Manshanden41, P. Mantscha, S.
Marafico32, A.G. Mariazzi4, I.C. Mariş13, G. Marsella59,45, D. Martello54,46,
H. Martinez18, O. Martínez Bravo62, M. Mastrodicasa55,44, H.J. Mathes39, J.
Matthews88, G. Matthiae60,49, E. Mayotte36, P.O. Mazura, G. Medina-Tanco67, D.
Melo8, A. Menshikov38, K.-D. Merenda86, S. Michal31, M.I. Micheletti6, L.
Miramonti57,47, S. Mollerach1, F. Montanet34, C. Morello52,50, M. Mostafá91,
A.L. Müller8, M.A. Muller20, K. Mulrey14, R. Mussa50, M. Muzio90, W.M.
Namasaka36, A. Nasr-Esfahani36, L. Nellen67, M. Niculescu-Oglinzanu73, M.
Niechciol42, D. Nitz89, D. Nosek29, V. Novotny29, L. Nožka31, A Nucita54,46,
L.A. Núñez28, M. Palatka30, J. Pallotta2, P. Papenbreer36, G. Parente79, A.
Parra62, M. Pech30, F. Pedreira79, J. Pȩkala69, R. Pelayo64, J. Peña-
Rodriguez28, E.E. Pereira Martins37,8, J. Perez Armand19, C. Pérez
Bertolli8,39, M. Perlin8,39, L. Perrone54,46, S. Petrera43,44, T. Pierog39, M.
Pimenta72, V. Pirronello56,45, M. Platino8, B. Pont80, M. Pothast82,80, P.
Privitera92, M. Prouza30, A. Puyleart89, S. Querchfeld36, J. Rautenberg36, D.
Ravignani8, M. Reininghaus39,8, J. Ridky30, F. Riehn72, M. Risse42, V.
Rizi55,44, W. Rodrigues de Carvalho19, J. Rodriguez Rojo10, M.J. Roncoroni8,
M. Roth39, E. Roulet1, A.C. Rovero5, P. Ruehl42, S.J. Saffi12, A. Saftoiu73,
F. Salamida55,44, H. Salazar62, G. Salina49, J.D. Sanabria Gomez28, F.
Sánchez8, E.M. Santos19, E. Santos30, F. Sarazin86, R. Sarmento72, C.
Sarmiento-Cano8, R. Sato10, P. Savina54,46,32, C.M. Schäfer39, V. Scherini46,
H. Schieler39, M. Schimassek37,8, M. Schimp36, F. Schlüter39,8, D. Schmidt37,
O. Scholten81,14, P. Schovánek30, F.G. Schröder93,39, S. Schröder36, J.
Schulte40, S.J. Sciutto4, M. Scornavacche8,39, A. Segreto51,45, S. Sehgal36,
R.C. Shellard15, G. Sigl41, G. Silli8,39, O. Sima73,f, R. Šmída92, P.
Sommers91, J.F. Soriano87, J. Souchard34, R. Squartini9, M. Stadelmaier39,8,
D. Stanca73, S. Stanič76, J. Stasielak69, P. Stassi34, A. Streich37,8, M.
Suárez-Durán28, T. Sudholz12, T. Suomijärvi35, A.D. Supanitsky8, J. Šupík31,
Z. Szadkowski71, A. Tapia27, C. Taricco61,50, C. Timmermans82,80, O.
Tkachenko39, P. Tobiska30, C.J. Todero Peixoto17, B. Tomé72, A. Travaini9, P.
Travnicek30, C. Trimarelli55,44, M. Trini76, M. Tueros4, R. Ulrich39, M.
Unger39, L. Vaclavek31, M. Vacula31, J.F. Valdés Galicia67, L. Valore58,48, E.
Varela62, V. Varma K.C.8,39, A. Vásquez-Ramírez28, D. Veberič39, C. Ventura25,
I.D. Vergara Quispe4, V. Verzi49, J. Vicha30, J. Vink84, S. Vorobiov76, H.
Wahlberg4, C. Watanabe24, A.A. Watsonc, M. Weber38, A. Weindl39, L. Wiencke86,
H. Wilczyński69, M. Wirtz40, D. Wittkowski36, B. Wundheiler8, A. Yushkov30, O.
Zapparrata13, E. Zas79, D. Zavrtanik76,77, M. Zavrtanik77,76, L. Zehrer76, A.
Zepeda63 and N. del Castillo8, G. de Innocenti8, L. Ferreyro8, S. Garavano8,
D. Gorbeña8, N. Leal8,9, G. Ríos8,9, M. Paramidani8,11, G. Pierri8,11, C.
Reyes8, A. Riello8, J.M. Salum8,11, A.P.J. Sedoski Croce8, D. Silva8, C.
Varela8

1

Centro Atómico Bariloche and Instituto Balseiro (CNEA-UNCuyo-CONICET), San
Carlos de Bariloche, Argentina

2

Centro de Investigaciones en Láseres y Aplicaciones, CITEDEF and CONICET,
Villa Martelli, Argentina

3

Departamento de Física and Departamento de Ciencias de la Atmósfera y los
Océanos, FCEyN, Universidad de Buenos Aires and CONICET, Buenos Aires,
Argentina

4

IFLP, Universidad Nacional de La Plata and CONICET, La Plata, Argentina

5

Instituto de Astronomía y Física del Espacio (IAFE, CONICET-UBA), Buenos
Aires, Argentina

6

Instituto de Física de Rosario (IFIR) – CONICET/U.N.R. and Facultad de
Ciencias Bioquímicas y Farmacéuticas U.N.R., Rosario, Argentina

7

Instituto de Tecnologías en Detección y Astropartículas (CNEA, CONICET,
UNSAM), and Universidad Tecnológica Nacional – Facultad Regional Mendoza
(CONICET/CNEA), Mendoza, Argentina

8

Instituto de Tecnologías en Detección y Astropartículas (CNEA, CONICET,
UNSAM), Buenos Aires, Argentina

9

Observatorio Pierre Auger, Malargüe, Argentina

10

Observatorio Pierre Auger and Comisión Nacional de Energía Atómica, Malargüe,
Argentina

11

Universidad Tecnológica Nacional – Facultad Regional Buenos Aires, Buenos
Aires, Argentina

12

University of Adelaide, Adelaide, S.A., Australia

13

Université Libre de Bruxelles (ULB), Brussels, Belgium

14

Vrije Universiteit Brussels, Brussels, Belgium

15

Centro Brasileiro de Pesquisas Fisicas, Rio de Janeiro, RJ, Brazil

16

Centro Federal de Educação Tecnológica Celso Suckow da Fonseca, Nova Friburgo,
Brazil

17

Universidade de São Paulo, Escola de Engenharia de Lorena, Lorena, SP, Brazil

18

Universidade de São Paulo, Instituto de Física de São Carlos, São Carlos, SP,
Brazil

19

Universidade de São Paulo, Instituto de Física, São Paulo, SP, Brazil

20

Universidade Estadual de Campinas, IFGW, Campinas, SP, Brazil

21

Universidade Estadual de Feira de Santana, Feira de Santana, Brazil

22

Universidade Federal do ABC, Santo André, SP, Brazil

23

Universidade Federal do Paraná, Setor Palotina, Palotina, Brazil

24

Universidade Federal do Rio de Janeiro, Instituto de Física, Rio de Janeiro,
RJ, Brazil

25

Universidade Federal do Rio de Janeiro (UFRJ), Observatório do Valongo, Rio de
Janeiro, RJ, Brazil

26

Universidade Federal Fluminense, EEIMVR, Volta Redonda, RJ, Brazil

27

Universidad de Medellín, Medellín, Colombia

28

Universidad Industrial de Santander, Bucaramanga, Colombia

29

Charles University, Faculty of Mathematics and Physics, Institute of Particle
and Nuclear Physics, Prague, Czech Republic

30

Institute of Physics of the Czech Academy of Sciences, Prague, Czech Republic

31

Palacky University, RCPTM, Olomouc, Czech Republic

32

CNRS/IN2P3, IJCLab, Université Paris-Saclay, Orsay, France

33

Laboratoire de Physique Nucléaire et de Hautes Energies (LPNHE), Sorbonne
Université, Université de Paris, CNRS-IN2P3, Paris, France

34

Univ. Grenoble Alpes, CNRS, Grenoble Institute of Engineering Univ. Grenoble
Alpes, LPSC-IN2P3, 38000 Grenoble, France

35

Université Paris-Saclay, CNRS/IN2P3, IJCLab, Orsay, France

36

Bergische Universität Wuppertal, Department of Physics, Wuppertal, Germany

37

Karlsruhe Institute of Technology (KIT), Institute for Experimental Particle
Physics, Karlsruhe, Germany

38

Karlsruhe Institute of Technology (KIT), Institut für Prozessdatenverarbeitung
und Elektronik, Karlsruhe, Germany

39

Karlsruhe Institute of Technology (KIT), Institute for Astroparticle Physics,
Karlsruhe, Germany

40

RWTH Aachen University, III. Physikalisches Institut A, Aachen, Germany

41

Universität Hamburg, II. Institut für Theoretische Physik, Hamburg, Germany

42

Universität Siegen, Department Physik – Experimentelle Teilchenphysik, Siegen,
Germany

43

Gran Sasso Science Institute, L’Aquila, Italy

44

INFN Laboratori Nazionali del Gran Sasso, Assergi (L’Aquila), Italy

45

INFN, Sezione di Catania, Catania, Italy

46

INFN, Sezione di Lecce, Lecce, Italy

47

INFN, Sezione di Milano, Milano, Italy

48

INFN, Sezione di Napoli, Napoli, Italy

49

INFN, Sezione di Roma “Tor Vergata”, Roma, Italy

50

INFN, Sezione di Torino, Torino, Italy

51

Istituto di Astrofisica Spaziale e Fisica Cosmica di Palermo (INAF), Palermo,
Italy

52

Osservatorio Astrofisico di Torino (INAF), Torino, Italy

53

Politecnico di Milano, Dipartimento di Scienze e Tecnologie Aerospaziali ,
Milano, Italy

54

Università del Salento, Dipartimento di Matematica e Fisica “E. De Giorgi”,
Lecce, Italy

55

Università dell’Aquila, Dipartimento di Scienze Fisiche e Chimiche, L’Aquila,
Italy

56

Università di Catania, Dipartimento di Fisica e Astronomia, Catania, Italy

57

Università di Milano, Dipartimento di Fisica, Milano, Italy

58

Università di Napoli “Federico II”, Dipartimento di Fisica “Ettore Pancini”,
Napoli, Italy

59

Università di Palermo, Dipartimento di Fisica e Chimica ”E. Segrè”, Palermo,
Italy

60

Università di Roma “Tor Vergata”, Dipartimento di Fisica, Roma, Italy

61

Università Torino, Dipartimento di Fisica, Torino, Italy

62

Benemérita Universidad Autónoma de Puebla, Puebla, México

63

Centro de Investigación y de Estudios Avanzados del IPN (CINVESTAV), México,
D.F., México

64

Unidad Profesional Interdisciplinaria en Ingeniería y Tecnologías Avanzadas
del Instituto Politécnico Nacional (UPIITA-IPN), México, D.F., México

65

Universidad Autónoma de Chiapas, Tuxtla Gutiérrez, Chiapas, México

66

Universidad Michoacana de San Nicolás de Hidalgo, Morelia, Michoacán, México

67

Universidad Nacional Autónoma de México, México, D.F., México

68

Universidad Nacional de San Agustin de Arequipa, Facultad de Ciencias
Naturales y Formales, Arequipa, Peru

69

Institute of Nuclear Physics PAN, Krakow, Poland

70

University of Łódź, Faculty of Astrophysics, Łódź, Poland

71

University of Łódź, Faculty of High-Energy Astrophysics,Łódź, Poland

72

Laboratório de Instrumentação e Física Experimental de Partículas – LIP and
Instituto Superior Técnico – IST, Universidade de Lisboa – UL, Lisboa,
Portugal

73

“Horia Hulubei” National Institute for Physics and Nuclear Engineering,
Bucharest-Magurele, Romania

74

Institute of Space Science, Bucharest-Magurele, Romania

75

University Politehnica of Bucharest, Bucharest, Romania

76

Center for Astrophysics and Cosmology (CAC), University of Nova Gorica, Nova
Gorica, Slovenia

77

Experimental Particle Physics Department, J. Stefan Institute, Ljubljana,
Slovenia

78

Universidad de Granada and C.A.F.P.E., Granada, Spain

79

Instituto Galego de Física de Altas Enerxías (IGFAE), Universidade de Santiago
de Compostela, Santiago de Compostela, Spain

80

IMAPP, Radboud University Nijmegen, Nijmegen, The Netherlands

81

KVI – Center for Advanced Radiation Technology, University of Groningen,
Groningen, The Netherlands

82

Nationaal Instituut voor Kernfysica en Hoge Energie Fysica (NIKHEF), Science
Park, Amsterdam, The Netherlands

83

Stichting Astronomisch Onderzoek in Nederland (ASTRON), Dwingeloo, The
Netherlands

84

Universiteit van Amsterdam, Faculty of Science, Amsterdam, The Netherlands

85

Case Western Reserve University, Cleveland, OH, USA

86

Colorado School of Mines, Golden, CO, USA

87

Department of Physics and Astronomy, Lehman College, City University of New
York, Bronx, NY, USA

88

Louisiana State University, Baton Rouge, LA, USA

89

Michigan Technological University, Houghton, MI, USA

90

New York University, New York, NY, USA

91

Pennsylvania State University, University Park, PA, USA

92

University of Chicago, Enrico Fermi Institute, Chicago, IL, USA

93

University of Delaware, Department of Physics and Astronomy, Bartol Research
Institute, Newark, DE, USA

94

University of Wisconsin-Madison, Department of Physics and WIPAC, Madison, WI,
USA

—–

a

Fermi National Accelerator Laboratory, Fermilab, Batavia, IL, USA

b

Max-Planck-Institut für Radioastronomie, Bonn, Germany

c

School of Physics and Astronomy, University of Leeds, Leeds, United Kingdom

d

Colorado State University, Fort Collins, CO, USA

e

now at Hakubi Center for Advanced Research and Graduate School of Science,
Kyoto University, Kyoto, Japan

f

also at University of Bucharest, Physics Department, Bucharest, Romania