# Introducing the Partitioned Equivalence Test: Artificial Intelligence in
Automatic Passenger Counting Validation

David Ellenberger1 Michael Siebert1

(Received: date / Accepted: date)

###### Abstract

> Automatic passenger counting (APC) in public transport has been introduced
> in the 1970s and has been rapidly emerging in recent years. APC systems,
> like all other measurement devices, are susceptible to error, which is
> treated as random noise and is required to not exceed certain bounds. The
> demand for very low errors is especially fueld by applications like revenue
> sharing, which is in the billions, annually. As a result, both the
> requirements as well as the costs heavily increased. In this work, we
> address the latter problem and present a solution to increase the efficiency
> of initial or recurrent (e.g. yearly or more frequent) APC validation. Our
> new approach, the _partitioned equivalence test_ , is an extension to this
> widely used statistic hypothesis test and guarantees the same bounded, low
> user risk while reducing effort. This can be used to either cut costs or to
> extend validation without cost increase. It involves a pre-classification
> step, which itsself can be arbitrary, so we evaluated several use cases:
> entirely manual and algorithmic, artificial intelligence assisted workflows.
> For former, by restructuring the evaluation of manual counts, our new
> statistical test can be used as a drop-in replacement for existing test
> procedures. The largest savings, however, result from latter algorithmic use
> cases: Due to the user risk being as bounded as in the original equivalence
> test, no additional requirements are introduced. Algorithms are allowed to
> be failable and thus, our test does not require the availability of general
> artificial intelligence. All in all, automatic passenger counting as well as
> the equivalence test itself can both benefit from our new extension.
>
> Keywords: automatic passenger counting APC validation APC accuracy revenue
> sharing equivalence testing certainty classification cost reduction

##  1 Introduction

Assessment of passenger counts is of paramount importance for public transport
agencies in order to plan, manage and evaluate their transit service. Over the
past three decades, automatic passenger counting (APC) systems have played an
increasingly important role in determining the number of passengers in local
public transport. They are used in the daily monitoring of operations, in
long-term demand planning, as well as in revenue sharing within transport
associations around the world. For more details and an overview of APC
development and current practice, see Siebert and Ellenberger (2019). Revenue
magnitudes in the billions are common in public transport (Armstrong and
Meissner, 2010), e.g. in the year 2018, total ticket revenues in Germany alone
have been 12.95 billion euros (Wagner, 2019), while APC systems are deployed
worldwide. In many cases, a passenger count is the de facto standard for
public tenders or the acquisition of subsidies. The counting quality of
existing APC systems on the market has been continuously improved by technical
developments in combination with increased requirements. Nowadays, APC systems
are expected to have a maximal systematic error or bias of 1%. This aspect of
APC validation is referred to as an accuracy of 99%. For transport
associations in Germany, but also internationally, validation is typically
regulated by the VDV, recommendation 457 (Köhler et al, 2018): In 2018, a
criterion based on the t-test was replaced by an equivalence test, which takes
the user error into account and limits it to $5\%$. This tightening of the
requirements compared to the previous test criterion has led to up to four
times larger sample sizes in the testing of the measurement accuracy, which
also quadruples the costs that arise primarily from the manual inspection of
the counting situations by comparison counting personnel. The already high
cost pressure on the comparison counting continues to increase and the need
for solutions to reduce costs increases alongside. Since the hourly wages of
the reference counters remain the same or increase in perspective, technical
and regulatory solutions are necessary. In order to increase validation
efficiency and reduce costs, we have identified the following levels during
our recent years of research (compare Table 1 for real world numbers):

Efficiency Level 0:

Manual ride checkers that stay in the vehicle during its entire journey and
count boarding and alighting passengers (and other count objects) at doors.

Efficiency Level 1:

Perform all manual counting on recorded (and automatically cut) (3D-)videos,
which modern APC systems can acquire directly from the sensors, compare Table
1. Having a pool of (possibly unseen) videos available is a requirement for
Efficiency Level 2 and 3.

Efficiency Level 2:

Increase efficiency in the evaluation of video material, e.g. by using
application-specific software that integrates comparison counting and video
viewing, compare Figure 1.

Efficiency Level 3:

Reduction of the video volume to be evaluated manually with verifiably equal
validation quality (manufacturer and user risk) as the current method based on
the equivalence test.

For latter, Efficiency Level 3, the reduction of the manually evaluated video
volume, an adapted mathematical-statistical formalism is required, which is
the subject of this manuscript: We introduce and discuss the concept of this
so-called _Partitioned Equivalence Test_ in the second and present a
mathematical formalization third section. We show how to perform a sample size
calculation for our new method, analyse and optimize costs in the fourth
section, evaluate real world data in the fifth section and close with some
concluding remarks and future prospects in the last section. In the following
we assume that a single _(3D-)video_ corresponds to a _door opening phase
(DOP)_ , i.e. it shows a view from inside the vehicle that allows to see an
entire indoor door area (compare Figure 1) and Figure 2) from door opening to
door closing:

  1. 1.

Door opening: initially, the door is (almost) completetly closed, so that no
passenger (or countable object) might pass.

  2. 2.

All the events that happen while the door is open, e.g. boarding and alighting
passengers.

  3. 3.

Door closing: at the end of the video, the door is (almost) completely closed
again so that no passenger (or countable object) might pass.

A commonly used unit for (automatic) passenger counting so far has been the
_Stop Door Event_ (SDE), which corresponds to a video of an entire door during
an entire stop: e.g. if a door (e.g. door 2) opens and closes again 3 times
during a stop, 3 DOP and one SDE is generated. However, nowadays APC systems
allow to record DOP, e.g. by accessing the corresponding door opening signal,
which reduces the amount of video data to be manually evaluated considerably
and upfront, compare Table 1. From an analytical point of view, the standard
deviation can be estimated more reliably, since door opening phases are the
smaller statistical unit than stop door events. In the following we therefore
use door opening phases. The sample size calculation may still be carried out
on stop door events e.g. for comparison purposes, see Section 4.

Figure 1: Screenshot of VisualCount: a specialized video software to perform
manual counts, which has been used to obtain our count data, compare Section
5. The height information from 3D videos can be used to distinguish between
children and adults, e.g. by using a critical height of 1.20 meters. To theses
standards, the small person in the center of the image above would be
considered as an adult, while the one to the left as a child. VisualCount
yields a 70% speedup over the use of a standard video player with a
spreadsheet software (like Microsoft Excel), which reduces the additional
costs for video evaluation from $300\%$ (the quadrupling of costs by
accounting for the user risk in the VDV 457 v2.1 vs. the old version v2.0) to
approximately $33\%$. It runs entirely in the browser, thus can operate
without a server by using videos from the users filesystem, which increases
data privacy. A cloud-based operation would be possible as well. Performance-
critial parts are implemented in C/C++ and compiled to JavaScript or
WebAssembly with Emscripten (Zakai, 2011), with filesystem-call passthrough.
This allows to determine durations, frame counts as well as single-frame
random access to hundreds of gigabytes of video data not only using browser-
native (like H.264 or AV1), but also general standard (e.g. MJPEG via FFmpeg)
or specialized, custom file formats (e.g. multiple, losslessly compressed 3D
video formats) without noticeable delay, eliminating almost all wait times. To
ensure a seamless integration, we reverse engineered a proprietary video file
format from one of our suppliers, who provided the full documentation after a
technical demonstration. A gamecontroller is used to navigate through the
video with playback or rewind speeds corresponding to the pressure applied to
the analog buttons to obtain both slow motion for crowds as well as high
speeds (10x and more) to skip through long idle timespans. Rates of 60 frames
per second can be reached within the browser so that even in fastest fast
forward mode, no boarding or alighting passenger is accidently skipped. A
specialized video software (such as VisualCount) is required to reach
Efficiency Level 2 and above.

Figure 2: Mounted with industrial velcro (first left) above doors, the Depth Sensing Unit DSU (second left) a portable, battery-powered 3D video recording device. For demo image material from VisualCount, see right image with colorscale. |  Efficiency Level 0 | Efficiency Level 1 |  Efficiency Level 2 | Efficiency Level 3
---|---|---|---|---
|  Ride Checkers in Vehicle |  Vehicle Stop |  Stop Door Event |  Door Opening Phase |  Specialized Video Software |  Manual Partitioned Equiv. Test |  Algorithmic Partitioned Equiv. Test
Duration |  38:43:26 |  15:52:20 |  05:45:11 |  02:48:15 |  00:50:29 |  00:40:23 |  00:25:14
Manual Partitioned Equiv. Test |  |  |  |  37,50%
Specialized Video Software |  |  |  20,00% |  50,00%
Door Opening Phases |  |  |  |  70,00% |  76,00% |  85,00%
Stop Door Events |  |  |  51,26% |  85,38% |  88,30% |  92,69%
Vehicle Stop |  |  63,75% |  82,33% |  94,70% |  95,76% |  97,35%
Ride Checkers |  59,01% |  85,14% |  92,76% |  97,83% |  98,26% |  98,91%
Table 1: Savings w.r.t. different video material granularity for an urban
railway transport system. The original video data (38:43h) has been recorded
with a GoPro connected to an external battery and represents the original ride
checker approach. From the actual timetable, real arrival and departure times
were known, so the original video could automatically be cut into smaller
pieces accordingly. From an APC system installed in the vehicle, the door
opening phases were known as well and utilized in a similar way. Stop door
events is a granularity in between vehicle stops and door opening phases: all
videos with no door opening have been removed from the vehicle stop set, which
is around 50%, since in most cases only one side of the vehicle doors open at
stops. Specialized video software (e.g. VisualCount, compare Figure 1) savings
are 70% according to our studies for the reason that users fast-forward
through idle sequences. Manual and algorithmic partitioned equivalence tests
have been accounted for with 20% resp. 50% savings. All in all, Efficiency
Level 3 reduces the manual effort by around 99% over Efficiency Level 0.

##  2 Partitioned Equivalence Test Concept

As mentioned in Section 1, in order to achieve a reduction in certification
costs beyond Efficiency Level 2 (the use of more integrated hard- and software
solutions) a further reduction in video volume to be evaluated manually is
necessary.

But how can this reduction be achieved methodically? Here, we initially had
the idea of using an additional algorithm to pre-classify the videos according
to the certainty or _safeness_ of a correct count by the APC system, so that a
manual count is only necessary on material with a considered highly uncertain
or _unsafe_. The challenge would have been to formulate rules for the general
approval of such a classification system and to develop an algorithm that is
powerful enough to meet the formulated requirements. However, as it turned out
in the course of the project, this approach has unmanageable implications: it
either requires the construction of a kind of superior, infallible algorithm,
which can, with unlimited certainty, identify the incorrectly counted videos.
Alternatively, if that algorithm had been fallible, the _ground truth_ would
have had to be redefined: currently, a count value is considered correct if it
was generated by (at least) two manual counters and verified a third manual
counter – the supervisor – in case the first two counters differ (Köhler et
al, 2018). An attempt to redefine this current ground truth would have raised
both ethical as well as technical questions (Lake et al, 2017) and would
definitely have lead to unfruitful, never ending discussions in the
foreseeable future. Facing these challenges, we finally changed our approach
to include a sample in the material that the algorithm has classified as _not
necessary to be manually viewed_ or _safe_. Here the question arises whether
savings can be achieved in this way at all, but such a procedure is at least
technically feasible by today’s standards and also methodically sound. We have
therefore continued our investigations.

For the implementation of Efficiency Level 3 (reduction of the video volume to
be evaluated manually with verifiably equal validation quality, compare
Section 1), a mathematical-statistical formalism is required which satisfies
the following requirements:

  1. 1.

Automatic passenger counting and the VDV 457 influence the worldwide
distribution of revenue in public transport, which means that all changes,
especially in the statistical inference, are critical. This results in the
following fundamental requirements:

    1. (a)

The cost savings should be relevant enough to make the adaptation of an
existing validation procedure legitimate.

    2. (b)

For a well-founded decision, an analytical derivation of the new statistical
method must be possible.

    3. (c)

The cost savings should be achievable without changing the definition of the
ground truth.

  2. 2.

Compared to the equivalence test, the new test should not place any additional
requirements on the statistical distribution of counting errors.

  3. 3.

If the parameters of the new test are selected so that the entire sample is
counted, the new test should correspond to the previous test, i.e. the
equivalence test.

  4. 4.

As in the equivalence test, it should always be ensured that the user risk is
not greater than a specified limit (currently max. $5\%$) and a specialized
sample size estimation should lead to a controllable adaptation of the
manufacturer risk.

  5. 5.

It would be beneficial if no special software is required to determine the
result of the test, i.e. an evaluation itsself should be possible with
commonly used spreadsheet software.

We have created a test procedure that statisfied all the above mentioned
requirements, the _Partitioned Equivalence Test_. The idea is as follows:
First, divide (or _partition_) the video material to be evaluated into two
parts: one part comprises the unsafe door opening phases, i.e. all videos
where a miscount of the APC is suspected. This entire so-called _unsafe
partition_ is counted manually according to the current procedure, i.e. by (at
least) two persons and a supervisor count in case the first two differ. In the
other part, the so-called _safe partition_ , there is only a _relatively
small_ manual sample count. The _partitioned_ equivalence test is now able to
merge the comparison counts of the two partitions again and to create a common
confidence interval, on which a regular equivalence test can be carried out.
The partitioned equivalence test is performed in four steps, compare Figure 3:

Figure 3: Schematic representation of the partitioned equivalence test.

  1. 1.

Acquisition of the counting material (e.g. 3D videos)

  2. 2.

Creation of a _partition_ by classification into safe and unsafe door opening
phases

  3. 3.

Selection of a random sample in the safe partition

  4. 4.

Carrying out the comparison count on the reduced material

##  3 Statistical Model

As a starting point we take the model as in (Siebert and Ellenberger, 2019):
Let $\Omega_{0}=\\{\omega_{i}\\},i=1,\ldots,\infty$ be the statistical
population of _door opening phases_ (_DOP_), which are used to summarize all
boarding and alighting passengers at a single vehicle (bus, tram, train) door
during a door opening of that door. Further, let
$\Omega=\\{\omega_{i_{j}}\\},i_{j}\in\\{1,\ldots,\infty\\},\
j\in\\{1,\ldots,n\\}$ be a sample, which consists of $n$ either randomly or
structurally selected door opening phases (e.g. by a given sampling plan). We
use the notation $n=n_{e}$ when the sample size was planned according to the
equivalence test. Let $M_{i}$, $i\in\\{i,\ldots,n\\}$ be the _manual count_ ,
$K_{i}$, $i\in\\{i,\ldots,n\\}$ be the _automatic count_ of boarding
passengers made by the APC system and $X_{i}=K_{i}-M_{i}$,
$i\in\\{i,\ldots,n\\}$ the differences among them. For the partitioned version
of the equivalence test yet to be introduced, it does not suffice to obtain
the manual count by ride checkers, (3D-)videos of door opening phases are
mandatory: We require a supply of (possibly unseen) footage, which can
subsequently be used for counting through multiple manual sightings of single
videos, favourably 3D depth data or 2D videos with additional, lower
resolution 3D information. 3D depth data allows to estimate the persons
heights, which is used to distinguish between adults and children and which is
especially relevant for revenue sharing, since tickets for children are
typically sold at a reduced price. To create the ground truth counts, only
manual counts may be used: at least two with an additional supervisor count in
case of differences. Alighting passengers (or other objects) are counted as
well and results apply analogously, but without loss of generality we only
consider the boarding passengers. Let
$\overline{M}=\frac{1}{n}\sum_{i=1}^{n}M_{i}$ be the average manual boarding
passenger count. We consider the random variables

$\displaystyle D_{i}:=\frac{K_{i}-M_{i}}{\overline{M}}\ \ ,$ (1)

which we call _relative differences_ being the differences of the
automatically and manually counted boarding passengers relative to the average
of the manually counted boarding passengers. The average
${\overline{D}}:=\frac{1}{n}\sum_{i=1}^{n}D_{i}$ is the statistic of interest
which is used in the equivalence test. The expected value
$\mu_{\Omega}:=E({\overline{D}})=\frac{1}{n}\kern-1.0pt\sum_{i=1}^{n}\kern-2.0ptE(D_{i})$
is the _actual systematic error_ of an APC system, since it can systematically
discriminate participants of the revenue sharing system. It could also be
referred to as _bias_ of the measurement device (see e.g. Nielsen et al, 2014)
or as _statistical distortion_ (see e.g. Köhler et al, 2018). Let
$\nu_{i}^{2}:=E(D_{i}\kern-2.0pt-\kern-2.0pt\mu_{\Omega})^{2}$ be the
quadratic error of the $D_{i}$ relative to the expected value $\mu_{\Omega}$.
It corresponds to the _variance_ of $D_{i}$ in the case of
$E(D_{i})\\!=\\!\mu_{\Omega}$ and
$\nu^{2}:=\frac{1}{n}\sum_{i=1}^{n}\nu_{i}^{2}$, which is the mean square
error of $D_{i}$. The square root $\nu=\sqrt{\nu^{2}}$ corresponds to the
definition of the _standard deviation_ , while
${\widehat{\nu}}^{2}:=\\!\frac{1}{n-1}\kern-1.0pt\sum_{i=1}^{n}(D_{i}\kern-1.0pt-\kern-1.0pt{\overline{D}})^{2}$
is the empirical variance estimator for $\nu^{2}$.

To test for _equivalence_ , one wants to show that observed differences are
within certain bounds, as opposed to complete equality. We here use the
hypotheses and error types as commonly defined for equivalence testing, also
sometimes referred to as the _two one-sided tests_ (TOSTs) procedure
(Schuirmann, 1987). As an alternative approach, the equivalence test can be
derived directly from the two-tailed t-test under certain simple assumptions,
i.e., that the parameters are induced (or simply exchanged) from those of the
t-test (Siebert and Ellenberger, 2019).

Thus, the hypotheses are (Julious, 2004)

$\displaystyle H_{0}$ $\displaystyle:\ \text{There is a (relevant) systematic
APC measurement error}(|\mu_{\Omega}|\geq\Delta)$ (2) $\displaystyle H_{1}$
$\displaystyle:\ \text{There is no (relevant) systematic APC measurement error
}(|\mu_{\Omega}|<\Delta)\ \ .$ (3)

We define $\Delta$ to be the equivalence margin and the relevant errors for
the equivalence test with $\alpha$ referring to (half) the risk of the user
and $\beta$ to the risk of the device manufacturer. We will consider two-sided
$1-2\alpha$ confidence intervals where $\alpha$ is commonly chosen to be
$2.5\%$. The test criterion to be evaluated is

$\displaystyle|{\overline{D}}|\leq\Delta-z_{1-\alpha}\
\frac{{\widehat{\nu}}}{\sqrt{n}}\ \ .$ (4)

Sample size estimation for an equivalence test defined this way is given by
Julious (2004) as:

$\displaystyle n=\bigl{(}z_{1-\beta/2}+z_{1-\alpha}\bigr{)}^{2}\
\frac{\nu^{2}}{\Delta^{2}}\ \ .$ (5)

To develop the partitioned equivalence test as outlined, one uses an already
existing classification into safe and unsafe door opening phases (DOP) on the
total sample $n_{\text{rec}}$ of recored videos ($n_{e}\leq n_{\text{rec}}$),
which results in two partitions. This classifcation is given by
$W_{i}\,,\;i=1,..,n$ into _safe_ ($W_{i}\\!=\\!\text{{s}}$) and _unsafe_
($W_{i}\\!=\\!\text{{u}}$) door opening phases (DOP). To improve readability,
we use the character placeholders s and u but any indicator will suffice.
Further, let
$p_{\text{{s}}}\,,\;p_{\text{{u}}}\kern-2.0pt=\kern-2.0pt1\kern-1.0pt\text{{-}}\kern
1.0ptp_{\text{{s}}}$ be the likelihood $p_{\text{{s}}}$ of a DOP to be
classified as _safe_ and the (counter) probability $p_{\text{{u}}}$ of a DOP
to be classified as _unsafe_. Then $W_{i}$ is accordingly Bernoulli
distributed:
$W_{i}\sim{\operatorname{P}\kern-1.0pt}^{W}\kern-5.0pt=\\!\operatorname{Bin}(1,\kern
1.0ptp_{\text{{s}}})$. Let
$N_{\text{{s}}}\,,\;N_{\text{{u}}}\\!=\\!N\kern-3.0pt-\kern-3.0ptN_{\text{{s}}}$
be the total number of safe and unsafe DOP and
${\widehat{p}}_{\text{{s}}}\kern-1.0pt=\kern-1.0ptN_{\text{{s}}}\kern-1.0pt/n$
be the frequency of safe DOP. Analogously, let
$\mu_{\text{{s}}}\,,\;\nu_{\text{{s}}}\,,\;\mu_{\text{{u}}}\,,\;\nu_{\text{{u}}}$
be the corresponding parameters for $\mu$ and $\nu$ on the respective
partitions of the safe DOP and the unsafe DOP, explicitly
$\mu_{\text{{s}}}={\textstyle\frac{1}{N_{\text{{s}}}}\kern-3.0pt\sum_{k=1}^{N_{\text{{s}}}}}\mu_{\text{{s}}k}$,
$\mu_{\text{{u}}}={\textstyle\frac{1}{N_{\text{{u}}}}\kern-3.0pt\sum_{k=1}^{N_{\text{{u}}}}}\mu_{\text{{u}}k}$
and
$\nu^{2}_{\text{{s}}}={\textstyle\frac{1}{N_{\text{{s}}}}\kern-3.0pt\sum_{k=1}^{N_{\text{{s}}}}}\nu^{2}_{\text{{s}}k}$
and
$\nu^{2}_{\text{{u}}}={\textstyle\frac{1}{N_{\text{{u}}}}\kern-3.0pt\sum_{k=1}^{N_{\text{{u}}}}}\nu^{2}_{\text{{u}}k}$.
Let $q$ be the so-called _counted quota_ of the safe partition and $1-q$ the
fraction not to be counted, i.e. skipped.

Different procedures, the so-called _use cases_ , to obtain such partitions
are described when discussing application cases in Section 5. In this context,
useful additional information, such as video data and stop characteristics,
can be used in a meaningful way to achieve a classification $W_{i}$,
$i=1,..,n$. This implies that, in the following, conditional distributions
(and parameters) on _safe_ DOP and on _unsafe_ DOP are to be considered, e.g.
for $M_{i}\,{\kern 0.0pt\rvert\kern-1.0pt}W_{i}$. A critical part of the use
cases is the cost control, which is introduced in Section 4.3: taking into
account basic costs
$c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}$, as
well as manual counting costs
$c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}$ for
safe DOP and the combined costs for unsafe DOP $c_{\text{{u}}}$.

For a schematic representation, compare Figure 4 of the current as well as the
newly introduced parameters. The alighting passengers are not explicitly
mentioned, as they are handled analogously to the boarding passengers.

Figure 4: Schematic representation of the parameters of the partitioned
equivalence test.

Our general statistical model is thus based on two strata that are defined by
the two partitions. Since the classification into a safe and an unsafe
partition can be entirely arbitrary, our model used stems from a general
mixture model. A two dimensional mixture model can described as a hierarchical
model consisting of the random classification as mixture component, see e.g.
McLachlan and Basford (1988) or for Gaussian mixture models (Reynolds, 1993).
Methods described in the statistical literature usually consider estimating
the parameters of the mixture components. In our case, the classification is
known and estimation of this component is thus not required. The estimation
here is only the weighted recombination of both partitions. The case of
combining data from different partitions (i.e. sources) resembles a fixed-
effects meta analysis, but is different in two aspects. First, weights are not
determined solely by sample size or standard error but rather by an artificial
weighting scheme that is induced to avoid any overrepresentation that might be
introduced by any (free choice) $q$. Secondly, the weights are not fixed but
are dependent on the randomness of the classification, i.e. to be considered a
random variable. In the following two sections we will introduce methods that
address both aspects.

###  3.1 Estimation of the expected manual count

As a first step for all further calculations with the partitioned equivalence
test, it is necessary to (indirectly) estimate the mean number of boarding
passengers in the total sample ${\overline{M}}$ because in the safe partition
the ground truth is only determined, i.e. manually counted, for a proportion
$q$. Thus, in addition to the information of the measurement error $X_{i}$,
the non-comparison counted data also lacks the information of the actual
$M_{i}$. The average boarding number is needed for the definition of $D_{i}$
and has to be estimated, since the mean value ${\overline{M}}$ is not
available. We here use the leave-q-out estimator (see Appendix A for details).
With

$\displaystyle\overline{M}_{\text{{u}}}:={\textstyle\frac{1}{N_{\text{{u}}}}\kern-3.0pt\sum_{k=1}^{N_{\text{{u}}}}}M_{\text{{u}}k}\qquad\qquad{\widehat{M}}_{\text{{s}}q}:={\textstyle\frac{1}{N_{\text{{s}}}}\kern-3.0pt\sum_{k=1}^{N_{\text{{s}}}}}M_{\text{{s}}k}\cdot
Z_{k}\kern-1.0pt/\kern-1.0ptq$ (6)

we obtain the estimator

$\displaystyle{\widehat{M}}_{q}$
$\displaystyle:={\widehat{p}}_{\text{{s}}}\\!\cdot\\!{\widehat{M}}_{\text{{s}}q}+{\widehat{p}}_{\text{{u}}}\\!\cdot\\!{\overline{M}}_{\text{{u}}}$
(7) $\displaystyle\kern
3.0pt=\textstyle\frac{1}{n}\\!\\!\left(\sum_{k=1}^{N_{\text{{u}}}}M_{\text{{u}}k}+\sum_{k=1}^{N_{\text{{s}}}}M_{\text{{s}}k}\\!\cdot\\!Z_{k}\kern-1.0pt/\kern-1.0ptq\right)$
(8)

for the average number of boarding passengers. This value can now be used
instead of ${\overline{M}}$ when calculating $D_{i}$:

$\displaystyle D_{i}:=\frac{X_{i}}{{\widehat{M}}_{q}}$ (9)

and in the case of a full manual count $q=100\%$, the definition of $D_{i}$ is
identical to the definition of the regular equivalence test because
${\widehat{M}}_{q=100\%}\\!=\\!{\overline{M}}$.

###  3.2 Estimation of APC bias

Analogously to the estimation of the expected manual counts and with the the
expected values of safe $\mu_{\text{{s}}}$ and unsafe $\mu_{\text{{u}}}$ DOP,
a similar result regarding the expected value holds (see Appendix A for
details with the random variables $X_{\text{{u}}k}$ being the relative
differences $D_{\text{{u}}k}$):

$\displaystyle\operatorname{E}({\overline{D}})=p_{\text{{s}}}\\!\cdot\\!\mu_{\text{{s}}}+p_{\text{{u}}}\\!\cdot\\!\mu_{\text{{u}}}$
(10)

Methods are now needed to obtain a range-preserving confidence interval if
only a portion specified in advance (for example $20\%$) of the safe DOP is to
be counted. A random selection of $q\\!\cdot\\!N_{\text{{s}}}$ safe DOP is
formed which are to be included in the final evaluation – i.e. for which the
actual ground truth must be determined. The mean value serves as the estimator
for $\mu_{\text{{u}}}$.

$\displaystyle\overline{D}_{\text{{u}}}:={\textstyle\frac{1}{N_{\text{{u}}}}\kern-3.0pt\sum_{k=1}^{N_{\text{{u}}}}}\,D_{\text{{u}}k}$
(11) and for $\mu_{\text{{s}}}$ a similar estimator can be obtained, which,
however, only uses the quota $q$. The $Z_{j}$ indicate herby, if the DOP was
randomly selected ($\kern 0.0pt\text{{=}}\kern 0.0pt1)$ or not ($\kern
0.0pt\text{{=}}\kern 0.0pt0)$:
$\displaystyle{\widehat{D}}_{\text{{s}}q}:={\textstyle\frac{1}{N_{\text{{s}}}}\kern-3.0pt\sum_{k=1}^{N_{\text{{s}}}}}\,D_{\text{{s}}k}\cdot
Z_{k}\kern-1.0pt/\kern-1.0ptq$ (12) From the different estimators one can now
generate some kind of composite estimator for $\mu$:
$\displaystyle{\widehat{D}}_{q}:={\widehat{p}}_{\text{{s}}}\\!\cdot\\!{\widehat{D}}_{\text{{s}}q}+{\widehat{p}}_{\text{{u}}}\\!\cdot\\!{\overline{D}}_{\text{{u}}}$
(13) $\displaystyle\kern
17.0pt=\textstyle\frac{1}{n}\\!\left(\sum_{k=1}^{N_{\text{{u}}}}D_{\text{{u}}k}+\sum_{k=1}^{N_{\text{{s}}}}D_{\text{{s}}k}\\!\cdot\\!Z_{k}\kern-1.0pt/\kern-1.0ptq\right)$
(14)

All values $D_{\text{{u}}j}$ where $Z_{j}$ is zero are therefore no longer
needed for calculations. Then

$\displaystyle\operatorname{E}({\widehat{D}}_{q})$
$\displaystyle=\textstyle\frac{1}{n}\cdot\operatorname{E}\\!\left(\kern-1.0pt\frac{N_{\text{{s}}}}{N_{\text{{s}}}}\sum_{k=1}^{N_{\text{{s}}}}D_{\text{{s}}k}\cdot
Z_{k}\kern-1.0pt/\kern-1.0ptq\,+\,\frac{N_{\text{{u}}}}{N_{\text{{u}}}}\sum_{k=1}^{N_{\text{{u}}}}D_{\text{{u}}k}\\!\right)$
(15)
$\displaystyle=\textstyle\frac{1}{n}\cdot\left(p_{\text{{s}}}\\!\cdot\\!n\cdot\mu_{\text{{s}}}\\!\cdot\\!\operatorname{E}(Z_{1})\kern-1.0pt/\kern-1.0ptq+p_{\text{{u}}}\\!\cdot\\!n\cdot\mu_{\text{{u}}}\right)$
(16)
$\displaystyle=p_{\text{{s}}}\\!\cdot\\!\mu_{\text{{s}}}+p_{\text{{u}}}\\!\cdot\\!\mu_{\text{{u}}}=\mu$
(17)

and thus ${\widehat{D}}_{q}$ is an unbiased estimator which can be calculated
on a reduced dataset. Using the results of Appendix A,

$\displaystyle\operatorname{Var}({\widehat{D}}_{q})$
$\displaystyle=\textstyle\frac{1}{n}\bigl{(}\,p_{\text{{s}}}\kern-1.0pt\\!\cdot\\!\kern-1.0pt\nu^{2}_{\text{{s}}}\kern-1.0pt/\kern-1.0ptq\,+(1\kern-1.0pt\text{{-}}\kern
1.0ptp_{\text{{s}}})\\!\cdot\\!\nu^{2}_{\text{{u}}}\;+\;(\mu_{\text{{s}}}\kern
1.0pt\text{{-}}\kern
1.0pt\mu_{\text{{u}}}\kern-1.0pt)^{2}\kern-1.0pt\\!\cdot\\!p_{\text{{s}}}\kern-2.0pt\\!\cdot\\!\kern-1.0pt(\kern-1.0pt1\kern-1.0pt\text{{-}}\kern
1.0ptp_{\text{{s}}})\bigr{)}$ (18)

holds for the variance of the estimator. It should be noted that the
variability of the uncertainty classification (as a random variable)
$(\mu_{\text{{s}}}\kern 1.0pt\text{{-}}\kern
1.0pt\mu_{\text{{u}}}\kern-1.0pt)^{2}\kern-1.0pt\\!\cdot\\!p_{\text{{s}}}\kern-1.0pt\\!\cdot\\!\kern-1.0pt(\kern-1.0pt1\kern-1.0pt\text{{-}}\kern
1.0ptp_{\text{{s}}}\kern-1.0pt)$ may account for a substantial part of the
variance if the expected values $\mu_{\text{{s}}}$ and $\mu_{\text{{u}}}$
should be very different. If the uncertainty is assumed to be fixed, this part
would not be taken into account, which would lead to too narrow confidence
intervals. This yields the (asymptotic) confidence interval:

$\displaystyle\left[{\widehat{D}}_{q}\pm
z_{1-\alpha/2}\\!\cdot\\!\operatorname{Var}({\widehat{D}}_{q})\right]$ (19)

where the unknown parameters $p_{\text{{s}}}$, $\mu_{\text{{s}}}$,
$\mu_{\text{{u}}}$, $\nu^{2}_{\text{{s}}}$, $\nu^{2}_{\text{{u}}}$ can be
replaced by the empirical variance estimators as plugin estimators. Details on
variance estimation are described in the following section.

##  4 Sample size calculation

For the sample size calculation, we introduce a type II error adjustment, a
minimal (relative) standard deviation and consider costs.

###  4.1 Type II error adjustments

The required sample size calculation for the equivalence test in the case
without partitioning can be obtained from Siebert and Ellenberger (2019). With
the partitioned equivalence test, however, the challenge arises that a
reduction of the sample size (initially) increases the manufacturer risk. In
order to compensate for this increased risk, an adjustment of the sample size
is necessary, which we call the _recorded size_. The sample size calculation
of the partitioned equivalence test can be directly derived from that of the
previous conventional equivalence test, with the previous standard error
$\nu\\!/\\!\sqrt{n}$ replaced by the one determined in equation (18). This
results in the following formula:

$\displaystyle n$ $\displaystyle=(z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\alpha\kern-1.0pt/\kern-1.0pt2}\kern-1.5pt+\kern-1.0ptz_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\beta\kern-1.0pt/\kern-1.0pt2}\kern-1.0pt)^{2}\cdot\frac{p_{\text{{s}}}\kern-1.0pt\\!\cdot\\!\nu_{\text{{s}}}^{2}/\kern-1.0ptq\,+\;p_{\text{{u}}}\kern-1.0pt\\!\cdot\\!\nu_{\text{{u}}}^{2}\;+\;p_{\text{{s}}}\kern-1.5pt\\!\cdot\\!\kern-0.5ptp_{\text{{u}}}\kern-1.5pt\\!\cdot\\!\kern-1.0pt(\mu_{\text{{s}}}\kern
1.0pt\text{{-}}\kern 1.0pt\mu_{\text{{u}}}\kern-1.0pt)^{2}}{\Delta^{2}}\ \ .$
(20)

The following applies to the variance on the total sample in case of
partitioning (see Equation 18 summed over the full sample in the case $q=1$)

$\displaystyle\nu^{2}$
$\displaystyle=p_{\text{{s}}}\kern-1.5pt\\!\cdot\\!\nu_{\text{{s}}}^{2}\kern
2.0pt\texttt{+}\kern
2.0ptp_{\text{{u}}}\kern-1.5pt\\!\cdot\\!\nu_{\text{{u}}}^{2}\kern
2.0pt\texttt{+}\kern
2.0ptp_{\text{{s}}}\kern-1.5pt\\!\cdot\\!p_{\text{{u}}}\kern-1.5pt\\!\cdot\\!\kern-0.5pt(\mu_{\text{{s}}}\kern-1.0pt\texttt{-}\mu_{\text{{u}}}\kern-1.0pt)^{2}\
\ ,$ (21)

which allows the following

$\displaystyle n$ $\displaystyle=(z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\alpha\kern-1.0pt/\kern-1.0pt2}\kern-1.5pt+\kern-1.0ptz_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\beta\kern-1.0pt/\kern-1.0pt2}\kern-1.0pt)^{2}\cdot\frac{1}{\Delta^{2}}\biggl{[}p_{\text{{s}}}\nu_{\text{{s}}}^{2}\Bigl{(}\frac{1}{q}-1\Bigr{)}+\nu^{2}\biggr{]}\
\ $ (22)

simplified representation. Compared with the calculations for the previous
equivalence test, this results in

$\displaystyle
n=n_{\text{rec}}=n_{\text{e}}\kern-1.0pt\cdot\biggl{[}\frac{p_{\text{{s}}}\nu_{\text{{s}}}^{2}}{\nu^{2}}\Bigl{(}\frac{1}{q}-1\Bigr{)}+1\biggr{]}\qquad\text{with}\qquad
n_{\text{e}}=(z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\alpha\kern-1.0pt/\kern-1.0pt2}+z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\beta\kern-1.0pt/\kern-1.0pt2})^{2}\frac{\nu^{2}}{\Delta^{2}}\
\ ,$ (23)

i.e. since $0<q\leq 1$ and thus $(1/q-1)\geq 0$ and $p_{\text{{s}}}\geq 0$ it
can always be viewed as multiplication of $n_{\text{e}}$ by a factor $\geq 1$
and thus is an (apparent) enlargement of the sample. In fact, the partitioned
equivalence test initially only increases the _recorded sample_ , i.e. more
comparative video footage, of which only a part is manually counted. The added
value of first recording more and then omitting material again during the
evaluation is that a higher proportion of unsafe events can be sighted
compared to the original equivalence test. This property enables the
partitioned equivalence test, after optimizing the costs, to make a more
precise statement about the systematic error (or bias) of the counting error
with less overall effort than the original equivalence test.

###  4.2 Minimal Standard Deviation

At very small sample sizes, it is difficult to reliably estimate any
parameters, including the standard deviation, which is a problem known for
example as _small-sample bias_ (Hummel et al, 2005). This affects the
equivalence test in general, but special attention must be paid to the
partitioned equivalence test in particular, since sample sizes in the safe
partition can be very small and errors very rare. To better understand the
implications for practice, we ran simulations with real world APC system count
errors (for more details, see Appendix C) and in some scenarios the user risk
$\leq\alpha$ indeed cannot be ensured for inappropriately small sample sizes.
However, this is more of a hypothetical problem, since for very small sample
sizes the overall chance to pass the test is below 15%, independent of the
actual error ${\overline{D}}$ of the APC system (even for ${\overline{D}}=0$)
and this can therefore not be made a sustainable business model for any APC
manufacturer. From an authority’s perspective, this is still undesirable
because the chance of approving an APC system due to a poorly designed
validation is greater than the user risk $\alpha$ implies. For any
(partitioned) equivalence test, a _minimal (relative) standard deviation_
$\nu_{\text{min}}$ can be introduced as a restriction, which solves this
problem: if $\hat{\nu}<\nu_{\text{min}}$ is encountered anywhere it is
replaced by $\nu_{\text{min}}$ (in our case
$\hat{\nu}_{\text{{s}}q}<\nu_{\text{min}}$ or
$\hat{\nu}_{\text{{u}}}<\nu_{\text{min}}$). Surprisingly, this allows the
partitioned equivalence test to operate even more theory-compliant than the
original equivalence test. For the latter, introducing a low
$\nu_{\text{min}}$ like $\nu_{\text{min}}=3\%$ has almost no effect, since
there is no separtation into safe and unsafe videos and thus
${\widehat{\nu}}\leq\nu_{\text{min}}$ only very rarely occurs.

###  4.3 Cost Management

To minimize costs (see also Appendix B), we introduce
$c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}$, the
_basic costs_ of a safe DOP. These include, for example, marginal costs of
video recording, marginal costs of data preparation and execution of the
additional algorithms, as well as costs due to time delays in carrying out
further comparison procedures. Further, let
$c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}$ be the
costs incurred for a manual comparative counting of safe DOP so that it can be
considered as a ground truth. The video data must be viewed by at least two
human counters and by a supervisor in case of conflicts. These so-called
counting costs are mainly composed of the personnel costs of the counters and
the costs due to time delays in the testing process. Finally, $c_{\text{{u}}}$
is the (average) combined cost of the unsafe DOP, i.e. consisting out of to
basic costs and the counting costs, since both are always carried out here.
With the definition of the above partial costs, the total project cost of a
validation process can be approximated by the following function:

$\displaystyle\operatorname{Cost}(n,q)=n_{\text{rec}}\cdot\big{(}p_{\text{{u}}}\\!\\!\cdot\\!c_{\text{{u}}}+p_{\text{{s}}}\\!\cdot\\!(c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}\\!+q\\!\cdot\\!c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}})\\!\big{)}\
\ .$ (24)

This approximation takes into account taking into multiplicity in which the
partial costs occur in dependence of the total recorded sample
$n_{\text{rec}}$ and the proportions of the subgroups. If $n_{\text{rec}}$ is
already determined (for some reason) and greater than $n$ from Equation (22),
the optimal quota $q_{0}$ can be determined by solving that equation:

$\displaystyle
q_{0}:=1\Big{/}\\!\left\\{\frac{1}{p_{\text{{s}}}\nu_{\text{{s}}}^{2}}\left(\frac{n_{\text{rec}}\cdot\Delta^{2}}{(z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\alpha\kern-1.0pt/\kern-1.0pt2}\kern-1.5pt+\kern-1.0ptz_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\beta\kern-1.0pt/\kern-1.0pt2}\kern-1.0pt)^{2}}-\nu^{2}\right)+1\right\\}\
\ .$ (25)

In practice, however, $n_{\text{rec}}$ is to be determined and depends on
$q_{0}$. This can be done numerically by running a simple loop over possible
$q$ and checking the cost function. However, it can also be done analytically:
With the model from Section 3 and using the sample size formula (22) we can
optimise the total costs in relation to $q$:

$\displaystyle q_{0}$
$\displaystyle:=\operatorname{Opt}(q)=\operatorname{argmin}_{q}\\!\\!\left\\{\operatorname{Cost}(n(q),q)\right\\}$
(26)
$\displaystyle=\operatorname{argmin}_{q}\\!\\!\left\\{n(q)\cdot\big{(}p_{\text{{u}}}\\!\\!\cdot\\!c_{\text{{u}}}+p_{\text{{s}}}\\!\\!\cdot\\!\\!(c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}\\!+q\\!\cdot\\!c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}})\\!\big{)}\right\\}$
(27)
$\displaystyle=\operatorname{argmin}_{q}\\!\\!\left\\{\\!\\!\Big{(}\\!z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\alpha\kern-1.0pt/\kern-1.0pt2}\kern-3.0pt+\kern-2.5ptz_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\beta\kern-1.0pt/\kern-1.0pt2}\kern-1.0pt\\!\Big{)}^{\\!2}\\!\\!\cdot\\!\frac{p_{\text{{s}}}\kern-1.0pt\\!\cdot\\!\nu_{\text{{s}}}^{2}(1/\kern-1.0ptq\,\text{{-}}\,1)\\!+\\!\nu^{2}}{\Delta^{2}}\\!\cdot\\!\left(\\!\frac{p_{\text{{u}}}c_{\text{{u}}}}{p_{\text{{s}}}c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}}+\frac{c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}}{c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}}+q\\!\right)\\!\cdot\\!(p_{\text{{s}}}c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}})\\!\\!\right\\}$
(28)
$\displaystyle=\operatorname{argmin}_{q}\\!\\!\left\\{\\!\\!\frac{(z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\alpha\kern-1.0pt/\kern-1.0pt2}\kern-3.0pt+\kern-2.5ptz_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\beta\kern-1.0pt/\kern-1.0pt2}\kern-1.0pt)^{2}p_{\text{{s}}}\nu_{\text{{s}}}^{2}}{\Delta^{2}}\\!\\!\cdot\\!\left(\frac{1}{q}\,\text{{-}}\,\frac{p_{\text{{s}}}\nu_{\text{{s}}}^{2}}{p_{\text{{s}}}\nu_{\text{{s}}}^{2}}+\frac{\nu^{2}}{p_{\text{{s}}}\nu_{\text{{s}}}^{2}}\right)\\!\cdot\\!\left(q+\\!\frac{p_{\text{{u}}}c_{\text{{u}}}}{p_{\text{{s}}}c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}}+\frac{c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}}{c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}}\\!\right)\\!\\!\right\\}$
(29)
$\displaystyle=\operatorname{argmin}_{q}\\!\\!\Big{\\{}\\!\operatorname{const}\cdot\Big{(}\frac{1}{q}+\underbrace{\frac{\nu^{2}\,\text{{-}}\,p_{\text{{s}}}\nu_{\text{{s}}}^{2}}{p_{\text{{s}}}\nu_{\text{{s}}}^{2}}}_{=:b}\Big{)}\\!\cdot\\!\Big{(}q+\\!\underbrace{\frac{p_{\text{{u}}}c_{\text{{u}}}}{p_{\text{{s}}}c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}}+\frac{c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}}{c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}}}_{=:a}\\!\Big{)}\\!\\!\Big{\\}}$
(30) The derivative of the function $f(q)=(q+a)\\!\cdot\\!(1/q+b)=b\cdot
q+a/q+\operatorname{const}\;$ to the variable $q$ results in
$f^{\prime}(q)=b+a\cdot\kern-3.0pt(-1/q^{2})$ and with
$f^{\prime}(q_{\text{min}})\overset{!}{=}0$ it follows, that for all $q\geq
0\%$ and $q\leq 100\%$ all non-marginal minima must hold
$q_{\text{min}}=+\sqrt{a/b}$. Since costs diverge to infinity for $q$ towards
0, the minimum of $f(q)$ is
$\operatorname{min}\\!\\!\big{(}\\!\sqrt{a/b},100\%\\!\big{)}$ and it follows
$\displaystyle q_{0}$
$\displaystyle=\operatorname{min}\\!\left(\sqrt{\left(\frac{p_{\text{{u}}}c_{\text{{u}}}}{p_{\text{{s}}}c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}}+\frac{c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}}{c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}}\right)/\left(\frac{\nu^{2}\,\text{{-}}\,p_{\text{{s}}}\nu_{\text{{s}}}^{2}}{p_{\text{{s}}}\nu_{\text{{s}}}^{2}}\right)},\,100\%\right)\
\ .$ (31)

##  5 Application

The equivalence test as currently in use is summarized in Procedure 1 and our
new partitioned equivalence test in Procedure 2.

###  5.1 Use Cases

Since the classification function itself can be completely arbitrary and due
to the large amount of possible partitions (e.g. in the case of a recommended
sample size of 6147 there are $2^{6141}=2.7\cdot 10^{1850}$ possible
partitions), we introduce _use cases_ :

  1. 1.

The original equivalence test: either all videos are classified as safe or all
videos are classified as unsafe. In both cases, the partitioned equivalence
test reduces to the original equivalence test.

  2. 2.

A simple classification, a so-called _rule of thumb_ : this use case can e.g.
depend on the automatic counts only, like sorting videos by their passengers
per minute count and creating the partitions according to whether a certain
threshold has been surpassed or not. By this method, door opening phases more
prone to overcrowding are considered to be less safe.

  3. 3.

Classification by using the first manual count: since in VDV 457 two manual
counts plus a supervisor as a tie-breaker is required, the first manual count
can already be used to classify whether a video is safe. Like in the second
use case, videos are sorted according to their difference in manual and
automatic count and split into safe and unsafe partitions using a certain
threshold.

  4. 4.

Classification by algorithms/artificial intelligence: the videos are processed
by an additional algorithm, which produces an estimate of how difficult or
unsafe it considers the video to be, which are handled like in the use cases
before.

    1. (a)

Only use the safeness estimate and ignore the count of the APC system
entirely. We take a look at this use case to determine whether there is an
_algorithm independent intrinsic video or scene difficulty_. In case the
additional algorithm and the APC system’s algorithm are related, we expect
greater savings.

    2. (b)

The additional algorithm is capable to create a count as well. Use the
difference to the APC system’s count and the safeness estimate as a tie-
breaker if the count delta is zero.

  5. 5.

Combined Classification: the methods above can be combined to yield better
savings than the individual use cases themselves.

    1. (a)

Use cases 2 and 3: first, classify by a rule of thumb, then do a first manual
count according to that classification. This approach is still entirely
manual.

    2. (b)

Use cases 3 and 4b: as the case before, but with an algorithm instead of a
rule of thumb. This approach has higher requirements, but may yield greater
savings as well.

For an evaluation of the use cases performance, see Figure 5.

Procedure 1 Current Equivalence Test w.r.t. VDV 457 v2.1

  1. 1.

Parameter specification: $\alpha=\beta=5\%$, $\Delta=1\%$, $\nu=20\%$
$\nu$ is APC system manufacturer dependent, modern systems can achieve
$\nu\leq 15\%$

  2. 2.

Sample size estimation: $n=n_{\text{e}}=(z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\alpha\kern-1.0pt/\kern-1.0pt2}+z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\beta\kern-1.0pt/\kern-1.0pt2})^{2}\frac{\nu^{2}}{\Delta^{2}}$

  3. 3.

Sample size buffer (includes an increase of the sample size by 15%)

  4. 4.

Perform the actual, manual comparison count

  5. 5.

Evaluation of the $(1\\!-\\!\alpha)(=95$%$)$–confidence interval
$\left[{\overline{D}}\pm
z_{1-\alpha/2}\cdot\frac{{\widehat{\nu}}}{\sqrt{n}}\right]$

  6. 6.

Check, if confidence interval is contained entirely within
$[-\Delta,+\Delta]$:

    1. (i)

if yes: equivalence test successfully passed

    2. (ii)

if no: possibly increase sample size and reevaluate equivalence test or
consider equivalence test as failed

Procedure 2 Partitioned Equivalence Test

  1. 1.

Test parameter specification: $\alpha=\beta=5\%$, $\Delta=1\%$, $\nu=15\%$
$\nu$ is APC system manufacturer dependent, modern systems can achieve
$\nu\leq 15\%$
To protect user risk $\leq\alpha$ against inappropriately low sample sizes,
e.g. $\nu_{\text{min}}=3\%$ can be introduced

  2. 2.

Partition parameter specification: $0\leq p_{\text{{s}}}\leq 1$,
$\nu_{\text{{s}}}/\nu$ and $0\leq q\leq 1$,
e.g. $p_{\text{{s}}}=90\%$, $\nu_{\text{{s}}}/\nu=35\%$, $q=17.5\%$ for
algorithmic and $q=35\%$ for purely manual use cases

(Actual values can be specified by the classification method provider.
User risk $\leq\alpha$ is guaranteed like in the non-partitioned equivalence
test.)

  3. 3.

Sample- resp. record size estimation:

$\displaystyle
n=n_{\text{rec}}=n_{\text{e}}\kern-1.0pt\cdot\biggl{[}p_{\text{{s}}}\frac{\nu_{\text{{s}}}^{2}}{\nu^{2}}\Bigl{(}\frac{1}{q}-1\Bigr{)}+1\biggr{]}\qquad\text{with}\qquad
n_{\text{e}}=(z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\alpha\kern-1.0pt/\kern-1.0pt2}+z_{1\kern-1.0pt\text{{-}}\kern
1.0pt\kern-1.0pt\beta\kern-1.0pt/\kern-1.0pt2})^{2}\frac{\nu^{2}}{\Delta^{2}}$

  4. 4.

Sample size buffer (includes an increase of the sample size by 15%)

  5. 5.

Record the counting material (e.g. 3D-videos)

  6. 6.

Use classification method to partition the counting material

  7. 7.

Perform the manual count w.r.t. the partition

Count all $N_{\text{{u}}}$ door opening phases in the unsafe partition

Count $N_{\text{{s}}}\cdot q$ random door stop events in the safe partition

  8. 8.

Evaluation of the $(1\\!-\\!\alpha)(=95$%$)$–confidence interval
$\left[{\overline{D}}\pm
z_{1-\alpha/2}\cdot\frac{{\widehat{\nu}}}{\sqrt{n}}\right]$ with

$\displaystyle\textstyle{\widehat{M}}_{q}$
$\displaystyle={\widehat{p}}_{\text{{s}}}\\!\cdot\\!{\widehat{M}}_{\text{{s}}q}+{\widehat{p}}_{\text{{u}}}\\!\cdot\\!{\overline{M}}_{\text{{u}}}\
\ ,\qquad
D_{i}:=(K_{i}-M_{i})/{\widehat{M}}_{q}\quad\text{für}\quad{\overline{D}}_{\text{{s}}q},\
{\overline{D}}_{\text{{u}}},\ {\widehat{\nu}}_{\text{{s}}q},\
{\widehat{\nu}}_{\text{{u}}}$ $\displaystyle{\overline{D}}$
$\displaystyle={\widehat{D}}_{q}=\frac{N_{\text{{s}}}}{n}\\!\cdot\\!{\overline{D}}_{\text{{s}}q}+\frac{N_{\text{{u}}}}{n}\\!\cdot\\!{\overline{D}}_{\text{{u}}}$
$\displaystyle{\widehat{\nu}}^{2}$
$\displaystyle={\widehat{\nu}}_{q}^{2}=\frac{N_{\text{{s}}}}{n}\\!\cdot\\!\frac{\max({\widehat{\nu}}_{\text{{s}}q},\nu_{\text{min}})^{2}}{q}+\frac{N_{\text{{u}}}}{n}\\!\cdot\\!\max({\widehat{\nu}}_{\text{{u}}},\nu_{\text{min}})^{2}+\frac{N_{\text{{s}}}\\!\cdot\\!N_{\text{{u}}}}{n^{2}}\\!\cdot\\!\bigl{(}{\overline{D}}_{\text{{s}}q}-{\overline{D}}_{\text{{u}}}\bigr{)}^{2}$

  9. 9.

Check, if confidence interval is contained entirely within
$[-\Delta,+\Delta]$:

    1. (i)

if yes: partitioned equivalence test successfully passed

    2. (ii)

if no: possibly increase sample size and reevaluate partitioned equivalence
test or consider partitioned equivalence test as failed

Figure 5: Possible savings for different use cases and APC systems. Red bars
indicate an entirely manual workflow with only sensor counts necessary (e.g.
from the APC system vendor) while blue bars indicate that the videos from the
door opening phases were processed by an additional (possibly non APC system
vendor) algorithm which yields both additional counts as well as a self-
estimated confidence of correctness. In our case, this is an ensemble of 9
Neural APC (Jahn, 2019) agents. For the manual workflows, the savings are
between 15% and 25% with the first manual count (use cases 3 and 5a) being
slightly more effective than the rule of thumb (use case 2) solely. Still,
costs can already be significantly reduced by reorganizing the current
validation workflow and using the partitioned equivalence test. Using only a
second algorithm (use case 4a) and interpreting its confidence estimation as a
general scene difficulty shows mixed results: when the algorithm and the
estimator share a common background, which is the case for the Neural APC
20190826_102009_16280 and the ensemble of the remaining agents, the savings
are around 35%, while otherwise, they are typically below 20%, which is a
little less than purely manual obtainable savings (red bars). However, when
combining the sensor counts with the second algorithm and confidence, around
50% savings can be reached in average (use case 4b and use case 5b). For the
simulation, passenger counts on a combined (2D and 3D) dataset of 11530 videos
have been used, with at least three manual counts per video (six-eyes
principle). The costs are stated in euro cent, compare Appendix B.

###  5.2 Discussion and Suggested Values

As can be seen in Figure 5, basically two use cases remain: the entirely
manual use case 5a and use case 4b. Use case 2, the rule of thumb, is not
effective enough and use case 3, the first manual count can often
significantly profit from a pre-classification, turning it into use case 5a.
For the algorithmic partitioned equivalence test, combined cases do not yield
a lot of improvement, yet complicating the process.

Overall, as optimal parameters, $p_{s}=90\%$ is a common finding and
$\nu_{s}/\nu=35\%$ can be assumed. For $q$, in purely manual methods, $q=35\%$
seems suiteable, while $q=17.5\%$ proved viable for the algorithmically
assisted use cases (compare figures in Appendix C).

##  6 Conclusion

Our investigations have shown that a seamless connection of the partitioned
equivalence test to current requirements from VDV 457 (and the equivalence
test itsself) is possible. Here the standard benefits considerably from the
completed changeover from the t-test to the equivalence test. The partitioned
equivalence test allows statistically robust samples to be realized at
significantly reduced costs when compared to the classic equivalence test. Not
only can costs be reduced, but also more manual tests can be carried out with
the same monetary investment, thus increasing quality. Measured by the volume
of funds to be distributed by revenue sharing, shortcomings in passenger
counting validation are not justifiable in economic terms anyway. The
partitioned equivalence test is not exclusively limited to the use of
algorithms, as shown by the classification by the first manual counting, a
rule of thumb or a combination of both. Basically, any information available
about the counting behavior of the sensors can be used to reduce costs – in a
statistically robust way and thus without increased risk for the user. There
is also no vendor lock-in, as all relevant calculations can be performed with
simple spreadsheet formulas without specialized software. In the medium to
long term, the partitioned equivalence test lays the foundation for a deeper
integration between manual and automatic counting to raise count quality to a
new level. This will not only benefit revenue distribution, but also
forecasting and real-time in-vehicle passenger counts. The new test will be an
incentive for transport companies to invest in the corresponding IT
infrastructure. In other industries, e.g. the tech industry, hybrid systems,
in which people and algorithms work together, are already common, e.g. in
fraud detection. We hope that the partitioned equivalence test will not only
help automatic passenger counting catch up with state-of-the-art technologies,
but will even make it a technological pioneer other fields can profit from.

## Acknowledgements

This research is financially supported by the European Regional Development
Fund.

## Authors’ contribution

D Ellenberger: Statistics Lead, Partitioned Equivalence Test Formalization &
Formal Proofs, Literature Search and Review, Illustration and Code Prototypes,
Data Processing & Analysis, Manuscript Writing and Editing.
M Siebert: Research Lead, Partitioned Equivalence Test Concept, Illustrations
and Code, Data Processing & Analysis, Simulations, Creation of VisualCount &
Depth Sensing Unit, Manuscript Writing and Editing.

## Conflict of Interest

David Ellenberger has been employed by Interautomation Deutschland GmbH during
the time of research and manuscript preparation. None resulted in a conflict
of interest. Michael Siebert is an employee of Interautomation Deutschland
GmbH. The submitted work does not pose a conflict of interest.

## References

  * Armstrong and Meissner (2010) Armstrong A, Meissner J (2010) Railway Revenue Management: Overview and Models. Lancaster University Management School
  * Hummel et al (2005) Hummel R, Banga S, Hettmansperger TP (2005) Better confidence intervals for the variance in a random sample. Tech. rep., Department of Statistics, The Pennsylvania State University
  * Jahn (2019) Jahn N (2019) Counting people on image sequences using recurrent neural networks
  * Julious (2004) Julious SA (2004) Sample sizes for clinical trials with normal data. Statistics in Medicine 23(12):1921–1986
  * Köhler et al (2018) Köhler S, Bobinger S, Branick R, Cerfontaine B, Krogull S, Luther A, Ritschel E, Brunner D, Ellenberger M, Siebert M, Schulze M, Starck M, Bruns W (2018) Recommendations for the Appliction of APCSs within Public Transport and Regional Rail Transport, Version 2.1. VDV-Recommendations 457, Verband Deutscher Verkehrsunternehmen (VDV), Köln, URL https://www.beka-verlag.de/ebook-vdv-schrift-457-automatic-passenger-counting-systems-v2-1.html
  * Lake et al (2017) Lake BM, Ullman TD, Tenenbaum JB, Gershman SJ (2017) Building machines that learn and think like people. Behavioral and Brain Sciences 40:e253, DOI 10.1017/S0140525X16001837
  * McLachlan and Basford (1988) McLachlan G, Basford K (1988) Mixture models marcel. Dekker, New York
  * Nielsen et al (2014) Nielsen BF, Frølich L, Nielsen OA, Filges D (2014) Estimating passenger numbers in trains using existing weighing capabilities. Transportmetrica A: Transport Science 10(6):502–517
  * Reynolds (1993) Reynolds DA (1993) A gaussian mixture modeling approach to text-independent speaker identification
  * Schuirmann (1987) Schuirmann DJ (1987) A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Pharmacodynamics 15(6):657–680
  * Siebert and Ellenberger (2019) Siebert M, Ellenberger D (2019) Validation of automatic passenger counting: introducing the t-test-induced equivalence test. Transportation DOI 10.1007/s11116-019-09991-9, URL https://doi.org/10.1007/s11116-019-09991-9
  * Wagner (2019) Wagner L (2019) ÖPNV-Bilanz 2018: erneuter Fahrgastrekord im Nahverkehr. https://www.vdv.de/190213-pm-bilanz-2018.pdfx, [Online; accessed 14-April-2021]
  * Zakai (2011) Zakai A (2011) Emscripten: An llvm-to-javascript compiler. In: Proceedings of the ACM International Conference Companion on Object Oriented Programming Systems Languages and Applications Companion, Association for Computing Machinery, New York, NY, USA, OOPSLA ’11, p 301–312, DOI 10.1145/2048147.2048224, URL https://doi.org/10.1145/2048147.2048224

##  Appendix A Confidence intervals for sums with random weights

In the following we use the notation of Section 2.2. Let
$\mathds{X}=(X_{i})_{i=1,..,n}$ the counting errors of an APC system,
independent, identically distributed with probability measure
${\operatorname{P}\kern-1.0pt}^{X}$, $\operatorname{E}(X_{1})=\mu$ and
$\operatorname{Var}(X_{1})=\nu$. The indicator $W_{i}$ represents the (random)
classification into safe and unsafe door opening phases (DOP). Then, $W_{i}$
is Bernoulli($p_{s}$)-distributed, in the sense that the outcome is _safe_
with likelihood $p_{s}$ and _unsafe_ with likelihood $p_{\text{{u}}}\kern
0.0pt\text{{=}}\kern 0.0pt1\kern-1.0pt\text{{-}}\kern 1.0ptp_{\text{{s}}}$
($W_{i}\sim{\operatorname{P}\kern-1.0pt}^{W}\sim\operatorname{Bin}(1,\kern
1.0ptp_{s})$). The total number of safe DOP is $N_{\text{{s}}}=\sum
W_{i}\left(=\sum\mathds{1}_{\left\\{\kern-1.0ptW_{i}\kern 0.0pt\text{{=}}\kern
0.0pt\text{{s}}\kern-1.0pt\right\\}}\right)$ which is the sum of all $W_{i}$
and ${\widehat{p}}_{\text{{s}}}=N_{\text{{s}}}/n$. We now consider the random
variables $X_{i}{\kern 0.0pt\rvert\kern-1.0pt}W_{i}$, i.e. the distribution of
errors in the case that a DOP is safe resp. unsafe. The associated probability
measure of the conditional distribution $X{\kern 0.0pt\rvert\kern-1.0pt}W$
always exists, since $W$ is integer and further the assumption
${\operatorname{P}\kern-1.0pt}(W_{i}\kern 0.0pt\text{{=}}\kern
0.0pt\text{{s}})\kern-3.0pt>\kern-3.0pt0$ holds, since otherwise it would be
trivial. Note that $X_{i}$ and $W_{i}$ are not needed to be stochastically
independent. Let $\operatorname{E}(X_{i}\,{\kern
0.0pt\rvert\kern-1.0pt}W_{i}\kern 0.0pt\text{{=}}\kern
0.0pt\text{{u}})=\mu_{\text{{u}}}$ and $\operatorname{E}(X_{i}\,{\kern
0.0pt\rvert\kern-1.0pt}W_{i}\kern 0.0pt\text{{=}}\kern
0.0pt\text{{s}})=\mu_{\text{{s}}}$. Then for the expected value the following
holds:

$\displaystyle\operatorname{E}(X_{1})$ $\displaystyle=\int\kern-9.0pt\int
x\;{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{X{\kern
0.0pt\rvert\kern-1.0pt}W}\\!\\!(\kern-1.0ptx\kern-1.0pt)\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}$
(32) $\displaystyle=P(W_{1}\kern 0.0pt\text{{=}}\kern
0.0pt\text{{s}})\\!\cdot\\!\\!\\!\int
x\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{X{\kern
0.0pt\rvert\kern-1.0pt}W\\!\kern 0.0pt\text{{=}}\kern
0.0pt\text{{s}}}\,+\,{\operatorname{P}\kern-1.0pt}(W_{1}\kern
0.0pt\text{{=}}\kern 0.0pt\text{{u}})\\!\cdot\\!\\!\\!\int
x\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{X{\kern
0.0pt\rvert\kern-1.0pt}W\\!\kern 0.0pt\text{{=}}\kern 0.0pt\text{{u}}}$ (33)
$\displaystyle=p_{\text{{s}}}\\!\cdot\\!\mu_{\text{{s}}}+p_{\text{{u}}}\\!\cdot\\!\mu_{\text{{u}}}$
(34)

Further, let now be $i_{\text{{s}}}(k)$ for $k\kern 0.0pt\text{{=}}\kern
0.0pt1,..,N_{\text{{s}}}$ the assignment in ascending order to the (random)
variables of the safe DOP, such that for $i=i_{\text{{s}}}(k)$ the notation
$X_{\text{{s}}k}:=X_{i}$, $W_{\text{{s}}k}:=W_{i}$, etc. is well defined and
$i_{\text{{u}}}(k)$ for $k\kern 0.0pt\text{{=}}\kern 0.0pt1,..,N_{\text{{u}}}$
the analogous assignment to the variables of the unsafe DOP
($X_{\text{{u}}k}$,$W_{\text{{u}}k}$, etc.). Methods are now required to
construct a range-preserving confidence interval if only a previously
specified proportion $q_{0}$ (e.g. $50\%$) of the safe DOP to be manually
counted. First an adjustment (upwards) of the pre-specified $q_{0}$ takes
place, such that the number of safe DOP to be counted is integer $q:=\lceil
q_{0}\\!\cdot\\!N_{\text{{s}}}\rceil/N_{\text{{s}}}$.

A random sample of $q\\!\cdot\\!N_{\text{{s}}}$ safe DOP can be implemented
with $\mathds{Z}\kern 0.0pt\text{{=}}\kern
0.0pt(Z_{k})_{1,..,N_{\text{{s}}}}\\!\\!\in\\!\left\\{0,1\right\\}$, such that
$\sum_{k=1}^{N_{\text{{s}}}}Z_{k}=q\\!\cdot\\!N_{\text{{s}}}$ holds. The
resulting $(Z_{k})_{1,..,N_{\text{{s}}}}$ are by design independent of all
measured variables. For $i=1,..,n$ let with $i=i_{\text{{s}}}(k)$ the weights
be defined $C_{i}=Z_{k}\kern-1.0pt/\kern-1.0ptq$ for the safe DOP and with
$i=i_{\text{{u}}}(k)$ the corresponding $C_{i}=1$ for the unsafe DOP.

The estimator for $\mu_{\text{{u}}}$ will be the mean value

$\displaystyle\overline{X}_{\text{{u}}}=\frac{1}{N_{\text{{u}}}}\sum_{k=1}^{N_{\text{{u}}}}X_{\text{{u}}k}$
(35) and for $\mu_{\text{{s}}}$ one can state a similar estimator, which we
will refer to as leave $(1\kern-1.0pt\text{{-}}\kern 1.0ptq)$ out estimator:
$\displaystyle{\widehat{X}}_{\text{{s}}q}=\frac{1}{N_{\text{{s}}}}\sum_{k=1}^{N_{\text{{s}}}}X_{\text{{s}}k}\cdot
Z_{k}\kern-1.0pt/\kern-1.0ptq$ (36) An estimator for $\mu$ can now be
constructed from the individual estimators as follows:
$\displaystyle{\widehat{X}}_{q}={\widehat{p}}_{\text{{s}}}\\!\cdot\\!{\widehat{X}}_{\text{{s}}q}+{\widehat{p}}_{\text{{u}}}\\!\cdot\\!{\overline{X}}_{\text{{u}}}$
(37)

All values $X_{\text{{u}}k}$ where $Z_{k}$ is zero are not needed. These are
the values that are not needed for the calculation, i.e. can be discarded.
Then,

$\displaystyle\operatorname{E}({\widehat{X}}_{q})$
$\displaystyle=\frac{1}{n}\cdot\operatorname{E}\left(\frac{N_{\text{{s}}}}{N_{\text{{s}}}}\sum_{k=1}^{N_{\text{{s}}}}X_{\text{{s}}k}\cdot
Z_{k}\kern-1.0pt/\kern-1.0ptq\,+\,\frac{N_{\text{{u}}}}{N_{\text{{u}}}}\sum_{k=1}^{N_{\text{{u}}}}X_{\text{{u}}k}\right)$
(38)
$\displaystyle=\frac{1}{n}\cdot\left(p_{\text{{s}}}\kern-1.0pt\\!\cdot\\!n\cdot\mu_{\text{{s}}}\\!\\!\cdot\\!\operatorname{E}(Z_{k})\kern-1.0pt/\kern-1.0ptq+p_{\text{{u}}}\kern-1.0pt\\!\cdot\\!n\cdot\mu_{\text{{u}}}\right)$
(39)
$\displaystyle=p_{\text{{s}}}\\!\cdot\\!\mu_{\text{{s}}}+p_{\text{{u}}}\\!\cdot\\!\mu_{\text{{u}}}=\mu$
(40)

and thus ${\widehat{X}}_{q}$ is an unbiased estimator which can be calculated
on a reduced data set. For the variance of the estimator the following holds:

$\displaystyle\operatorname{Var}({\widehat{X}}_{q})$
$\displaystyle=\operatorname{E}({\widehat{X}}_{q}\\!-\mu)^{2}=\int\kern-9.0pt\int({\widehat{X}}_{q}\\!-\mu)^{2}\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{\mathds{X}\,{\kern
0.0pt\rvert\kern-1.0pt}W}\\!\\!\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}$
$\displaystyle=\frac{1}{n^{2}}\int\kern-9.0pt\int\left(N_{\text{{s}}}\kern-1.0pt\\!\cdot\\!{\widehat{X}}_{\text{{s}}q}+N_{\text{{u}}}\kern-1.0pt\\!\cdot\\!{\overline{X}}_{\text{{u}}}\;-n\\!\cdot\\!\mu\right)^{\\!\\!2}\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{\mathds{X}\,{\kern
0.0pt\rvert\kern-1.0pt}W}\\!\\!\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}$
$\displaystyle=\frac{1}{n^{2}}\int\kern-9.0pt\int\left(\sum_{k=1}^{N_{\text{{s}}}}(X_{\text{{s}}k}\cdot
Z_{k}\kern-1.0pt/\kern-1.0ptq-\mu)+\sum_{k^{\prime}=1}^{N_{\text{{u}}}}(X_{\text{{u}}k^{\prime}}-\mu)\right)^{\\!\\!\\!2}\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{\mathds{X}\,{\kern
0.0pt\rvert\kern-1.0pt}W}\\!\\!\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}$
$\displaystyle=\frac{1}{n^{2}}\\!\int\kern-9.0pt\int\\!\left(\sum_{k=1}^{N_{\text{{s}}}}(X_{\text{{s}}k}\\!\cdot\\!Z_{k}\kern-1.0pt/\kern-1.0ptq\text{{-}}\kern
1.0pt\mu_{\text{{s}}}\text{{+}}(p_{\text{{u}}}\\!\\!\cdot\\!\mu_{\text{{s}}}\kern-1.0pt\text{{-}}\kern
1.0ptp_{\text{{u}}}\\!\\!\cdot\\!\mu_{\text{{u}}}))+\sum_{k^{\prime}=1}^{N_{\text{{u}}}}(X_{\text{{u}}k^{\prime}}\text{{-}}\kern
1.0pt\mu_{\text{{u}}}\text{{+}}p_{\text{{s}}}\\!\\!\cdot\\!(\mu_{\text{{u}}}\kern-1.0pt\text{{-}}\kern
1.0pt\mu_{\text{{s}}}))\\!\right)^{\\!\\!\\!2}\\!{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{\mathds{X}\,{\kern
0.0pt\rvert\kern-1.0pt}W}\\!\\!\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}$
with $\delta:=\mu_{\text{{s}}}\kern 1.0pt\text{{-}}\kern
1.0pt\mu_{\text{{u}}}$ und
$\sum_{k=1}^{N_{\text{{s}}}}Z_{k}\kern-1.0pt/\kern-1.0ptq\\!\cdot\\!\mu_{\text{{s}}}=N_{\text{{s}}}\\!\cdot\\!\mu_{\text{{s}}}$
it follows:
$\displaystyle=\frac{1}{n^{2}}\\!\int\kern-9.0pt\int\\!\left(\sum_{k=1}^{N_{\text{{s}}}}Z_{k}\kern-1.0pt/\kern-1.0ptq\\!\cdot\\!(X_{\text{{s}}k}\text{{-}}\kern
1.0pt\mu_{\text{{s}}})\\!+\\!\sum_{k^{\prime}=1}^{N_{\text{{u}}}}(X_{\text{{u}}k^{\prime}}\text{{-}}\kern
1.0pt\mu_{\text{{u}}})+N_{\text{{s}}}\\!\\!\cdot\\!p_{\text{{u}}}\\!\cdot\\!\delta\kern
1.0pt\text{{-}}\kern
1.0ptN_{\text{{u}}}\\!\\!\cdot\\!p_{\text{{s}}}\\!\cdot\\!\delta\right)^{\\!\\!\\!2}{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{\mathds{X}\,{\kern
0.0pt\rvert\kern-1.0pt}W}\\!\\!\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}$
with $N_{\text{{s}}}\\!\\!\cdot\\!p_{\text{{u}}}\\!\cdot\\!\delta\kern
1.0pt\text{{-}}\kern
1.0ptN_{\text{{u}}}\\!\\!\cdot\\!p_{\text{{s}}}\\!\cdot\\!\delta=(N_{\text{{s}}}\\!\cdot\\!(1\kern-1.0pt\text{{-}}\kern
1.0ptp_{\text{{s}}}\kern-1.0pt)\kern 0.0pt\text{{-}}\kern 1.0pt(n\kern
1.0pt\text{{-}}\kern
1.0ptN_{\text{{s}}}\kern-1.0pt)\\!\cdot\\!p_{\text{{s}}})\\!\cdot\\!\delta=(N_{\text{{s}}}\text{{-}}\kern
1.0ptn\\!\cdot\\!p_{\text{{s}}})\\!\cdot\\!\delta$ and union of the sums of
the safe and unsafe DOP it follows:
$\displaystyle=\frac{1}{n^{2}}\\!\int\kern-9.0pt\int\\!\Biggl{(}\sum_{i=1}^{n}C_{i}\\!\cdot\\!\Bigl{(}X_{i}\,{\kern
0.0pt\rvert\kern-1.0pt}W_{i}-\underbrace{\operatorname{E}(X_{i}\,{\kern
0.0pt\rvert\kern-1.0pt}W_{i})}_{=:\,\mu_{i}^{\text{{s}}\\!/\\!\text{{u}}}}\Bigl{)}\quad+\quad(N_{\text{{s}}}\kern
0.0pt\text{{-}}\kern
1.0ptn\\!\cdot\\!p_{\text{{s}}})\\!\cdot\\!\delta\Biggl{)}^{\\!\\!\\!2}{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{\mathds{X}\,{\kern
0.0pt\rvert\kern-1.0pt}W}\\!\\!\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}$
$\displaystyle=\frac{1}{n^{2}}\\!\int\kern-9.0pt\int\\!\Bigl{(}\underbrace{\sum_{i=1}^{n}C_{i}\\!\cdot\\!(X_{i}\,{\kern
0.0pt\rvert\kern-1.0pt}W_{i}-\mu_{i}^{\text{{s}}\\!/\\!\text{{u}}})}_{=:\,A}\Bigl{)}^{\\!\\!2}\quad+\quad\underbrace{\vphantom{\sum_{1}^{n}}2\\!\cdot\\!A\\!\cdot\\!B}_{\rightarrow
0}\quad+\quad\Bigl{(}\underbrace{\vphantom{\sum_{1}^{n}}\\!(N_{\text{{s}}}\kern
0.0pt\text{{-}}\kern
1.0ptn\\!\cdot\\!p_{\text{{s}}})\\!\cdot\\!\delta\\!}_{=:\,B}\Bigl{)}^{\\!\\!2}{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{\mathds{X}\,{\kern
0.0pt\rvert\kern-1.0pt}W}\\!\\!\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}$
$\displaystyle=\frac{1}{n^{2}}\\!\int\kern-9.0pt\int\\!\sum_{i=1}^{n}\bigl{[}C_{i}\\!\cdot\\!\bigl{(}X_{i}\,{\kern
0.0pt\rvert\kern-1.0pt}W_{i}-\mu_{i}^{\text{{s}}\\!/\\!\text{{u}}}\bigl{)}\bigl{]}^{2}+\sum_{i\neq
i^{\prime}}^{n}C_{i}\\!\cdot\\!C_{i^{\prime}}\\!\cdot\\!\underbrace{\bigl{(}X_{i}{\kern
0.0pt\rvert\kern-1.0pt}W_{i}\\!-\\!\mu_{i}^{\text{{s}}\\!/\\!\text{{u}}}\bigl{)}\\!\cdot\\!\bigl{(}X_{i^{\prime}}{\kern
0.0pt\rvert\kern-1.0pt}W_{i^{\prime}}\\!-\\!\mu_{i^{\prime}}^{\text{{s}}\\!/\\!\text{{u}}}\bigl{)}}_{\rightarrow
0}+B^{2}\,{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{\mathds{X}\,{\kern
0.0pt\rvert\kern-1.0pt}W}\\!\\!{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}$
$\displaystyle=\frac{1}{n^{2}}\left(\int
n\\!\cdot\\!\operatorname{Var}\bigl{(}C_{1}\\!\cdot\\!(X_{1}\,{\kern
0.0pt\rvert\kern-1.0pt}W\\!-\\!\mu_{i}^{\text{{s}}\\!/\\!\text{{u}}})\bigl{)}{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}+\delta^{2}\\!\\!\cdot\\!\\!\\!\int(N_{\text{{s}}}\kern
0.0pt\text{{-}}\kern
1.0ptn\\!\cdot\\!p_{\text{{s}}})^{2}{d\kern-2.0pt\operatorname{P}\kern-1.0pt}^{W}\right)$
$\displaystyle=\frac{1}{n}\;\;\Bigl{(}\;\operatorname{Var}\bigl{(}Z_{1}\kern-1.0pt/\kern-1.0ptq\,\\!\cdot\\!(X_{\text{{s}}1}\text{{-}}\kern
1.0pt\mu_{\text{{s}}})\bigr{)}\\!\cdot\\!p_{\text{{s}}}+\operatorname{Var}\bigl{(}X_{\text{{u}}1}\kern
0.0pt\text{{-}}\kern
1.0pt\mu_{\text{{u}}}\bigr{)}\\!\cdot\\!p_{\text{{u}}}\quad+\quad\delta^{2}\\!\cdot\\!p_{\text{{s}}}\\!\\!\cdot\\!p_{\text{{u}}}\Bigr{)}$

Furthermore, for the variance of a product of independent random variables $C$
and $X$ the identity
$\operatorname{Var}(C\\!\cdot\\!X)=\operatorname{E}(C)^{2}\\!\cdot\\!\operatorname{Var}(X)+\operatorname{E}(X)^{2}\\!\cdot\\!\operatorname{Var}(C)+\operatorname{Var}(C)\\!\cdot\\!\operatorname{Var}(X)$
holds and thus

$\displaystyle\operatorname{Var}(Z_{1}\kern-1.0pt/\kern-1.0ptq\\!\cdot\\!(X_{\text{{s}}1}\text{{-}}\kern
1.0pt\mu_{\text{{s}}}))$
$\displaystyle=\underbrace{\operatorname{E}(Z_{1}\kern-1.0pt/\kern-1.0ptq)}_{=1}\kern-2.0pt^{2}\\!\cdot\\!\operatorname{Var}(X_{\text{{s}}1})\\!+\\!\underbrace{\operatorname{E}(X_{\text{{s}}1}\kern
0.0pt\text{{-}}\mu_{\text{{s}}})}_{=0}\kern-2.0pt^{2}\\!\cdot\\!\operatorname{Var}(Z_{1}\kern-1.0pt/\kern-1.0ptq)\\!+\\!\operatorname{Var}(Z_{1}\kern-1.0pt/\kern-1.0ptq)\\!\cdot\\!\operatorname{Var}(X_{\text{{s}}1})$
$\displaystyle=\operatorname{Var}(X_{\text{{s}}1})\kern-1.0pt/\kern-1.0ptq\quad\text{,}$

because $Z_{1}$ has a hypergeometric distribution and thus
$\operatorname{Var}(Z_{1}\kern-1.0pt/\kern-1.0ptq)=1\kern-1.0pt/\kern-1.0ptq\kern
1.0pt\text{{-}}1$ holds. Thus,

$\displaystyle\operatorname{Var}({\widehat{X}}_{q})$
$\displaystyle=\frac{1}{n}\Bigl{(}\,p_{\text{{s}}}\kern-1.0pt\\!\cdot\\!\kern-1.0pt\operatorname{Var}\bigl{(}X_{\text{{s}}1}\bigr{)}\kern-1.0pt/\kern-1.0ptq\,+(1\kern-1.0pt\text{{-}}\kern
1.0ptp_{\text{{s}}})\\!\cdot\\!\operatorname{Var}\bigl{(}X_{\text{{u}}1}\bigr{)}\;+\;(\mu_{\text{{s}}}\kern
1.0pt\text{{-}}\kern
1.0pt\mu_{\text{{u}}}\kern-1.0pt)^{2}\kern-1.0pt\\!\cdot\\!p_{\text{{s}}}\kern-2.0pt\\!\cdot\\!\kern-1.0pt(\kern-1.0pt1\kern-1.0pt\text{{-}}\kern
1.0ptp_{\text{{s}}})\Bigr{)}$ (41)

.

##  Appendix B Practical guidance on parameters for cost functions

For calculations regarding the planning of an APC validation, a plausible cost
function must be assumed:

  1. (a)

Video recording time $t_{\\{\text{video},\,i\\}}$ in hours

  2. (b)

The acceleration factor
$r_{\text{A}\kern-1.0pt/\kern-0.5pt\text{V}}=t_{\text{labor}}/t_{\text{video}}$
of the video corresponds to the ratio of working time $t_{\text{labor}}$ to
recording time $t_{\text{video}}$, i.e.
$r_{\text{A}\kern-1.0pt/\kern-0.5pt\text{V}}<1$ corresponds to an
acceleration, $r_{\text{A}\kern-1.0pt/\kern-0.5pt\text{V}}=1$, that viewing
the footage takes just as long as its duration and
$r_{\text{A}\kern-1.0pt/\kern-0.5pt\text{V}}>1$ corresponds to a slowdown
compared to the video duration.

  3. (c)

The labor cost $c_{\text{labor}}$, which is usually equal to the hourly wage
of the manual comparison counters.

  4. (d)

Additional costs due to the second manual count and the supervisor:
$r_{\text{S}}$

Then the following basic costs are derived from this:

$\displaystyle c_{(\text{Z},i)}:=t_{\\{\text{video},\,i\\}}\cdot
r_{\text{A}\kern-1.0pt/\kern-0.5pt\text{V}}\cdot c_{\text{labor}}$ (42)

Empirical results with manual counts, which were performed with untrained
personnel and with software optimized for the workflow, allow the following
approximate values:

$\displaystyle r_{\text{A}\kern-1.0pt/\kern-0.5pt\text{V}}=0.7\qquad\qquad
c_{\text{labor}}=20\text{€}\qquad\qquad r_{\text{S}}=1.2\,.$ (43)

This results in average counting costs of

$\displaystyle\textstyle\frac{1}{n}\\!\sum_{i=1}^{n}c_{(\text{Z},i)}=0.164\,\text{€}$
(44)

per door opening phase (DOP). If the recording costs are assumed to be zero,
the total costs of the uncertain DOP are calculated as follows:

$\displaystyle c_{\text{{u}}}$
$\displaystyle=(1\\!+\\!r_{\text{S}})\cdot\textstyle\frac{1}{N_{\text{{u}}}}\\!\sum_{i=1}^{N_{\text{{u}}}}c_{(\text{Z},\text{{u}}i)}\
\ ,$ (45) while two cases must be distinguished here for the safe DOP. The
first case represents the classification without manual counting:
$\displaystyle
c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}$
$\displaystyle=0$ (46) $\displaystyle
c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}$
$\displaystyle=(1\\!+\\!r_{\text{S}})\cdot\textstyle\frac{1}{N_{\text{{s}}}}\\!\sum_{i=1}^{N_{\text{{s}}}}c_{(\text{Z},\text{{s}}i)}$
(47) The second case when the first manual count is included: $\displaystyle
c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}$
$\displaystyle=\textstyle\frac{1}{N_{\text{{s}}}}\\!\sum_{i=1}^{N_{\text{{s}}}}c_{(\text{Z},\text{{s}}i)}$
(48) $\displaystyle
c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}$
$\displaystyle=r_{\text{S}}\cdot\textstyle\frac{1}{N_{\text{{s}}}}\\!\sum_{i=1}^{N_{\text{{s}}}}c_{(\text{Z},\text{{s}}i)}\
\ .$ (49)

Cases in which only a partial count is performed are calculated analogously as
partial totals over the respective DOP. For the use cases described in Section
5.1, different classification options result in safe and unsafe DOP. The
better a classification succeeds, with simultaneously low costs, the greater
is the overall savings potential of the process. A special approach here is
the _combined classification_ , e.g. consisting of cases 2 and 3: a rule of
thumb and the first manual count. With such a combined classification, the
costs incurred can also be determined in a simple manner. This is done by
considering a more detailed partitioning: the _safe DOP_ are those that have
already been classified as safe using the first classification rule (e.g. rule
of thumb) and those that were initially classified as unsafe, but were checked
using the second classification rule (here the manual count) and were
reclassified as _safe_ due to lack of discrepancies in the counts. The
quantity of reclassified DOP

$\displaystyle\mathbb{W}_{\text{reclass.},\text{{s}}i}=\Bigg{\\{}\begin{array}[]{ll}1&\text{if
DOP $i$ is reclassified as safe after the first classification}\\\ 0&\text{if
DOP $i$ is initially classified as safe}\end{array}$ (52)

is important for appropriate allocation of costs. The counting costs incurred
by the reclassified DOP for the initial manual (comparative) count are now
added to the basic costs, since they may have been incurred without the ground
truth for that DOP actually being finally determined. In contrast, this
attribution does not happen for _unsafe_ DOP, where both classification rules
have provided an unsafe classification, since for these the ground truth must
be determined in any case. Since full counting costs are therefore always
incurred here, the sequence is irrelevant. In summary, the costs of the safe
DOP are as follows:

$\displaystyle
c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{0}$}}$
$\displaystyle=\textstyle\frac{1}{N_{\text{{s}}}}\\!\sum_{i=1}^{N_{\text{{s}}}}\mathbb{W}_{\text{reclass.},\text{{s}}i}\cdot
c_{(\text{Z},\text{{s}}i)}$ (53) $\displaystyle
c_{\text{{s}}\kern-1.0pt\leavevmode\resizebox{4.0pt}{}{$\texttt{Z}$}}$
$\displaystyle=\textstyle\frac{1}{N_{\text{{s}}}}\\!\sum_{i=1}^{N_{\text{{s}}}}(1-\mathbb{W}_{\text{reclass.},\text{{s}}i}+r_{\text{S}})\cdot
c_{(\text{Z},\text{{s}}i)}\ \ .$ (54)

An application of this approach is shown in Figure 5.

##  Appendix C Small-sample problems in variance estimation

This section consists of various test success simulations, compare Figures 6,
7, 8 and 9.

Figure 6: Equivalence Test Success Theory and Simulation with a normal
distribution. The overlap is almost perfect: at $\nu=7.5\%$, the success
chance is supposed to be zero analytically, from which the simulation deviates
slightly. For a comparison with actual passenger counting system errors, see
Figure 7, Figure 8 and Figure 9.

Figure 7: Test Success Simulation, comparing the equivalence test with the
partitioned equivalence test, no $\nu_{\text{min}}$ resp.
$\nu_{\text{min}}=0\%$. The difference of ${\overline{D}}$ to a perfect normal
distribution (compare Figure 6) is that it has a slightly higher chance of
test success than it should have for low sample sizes. Even though success
chances of around $10\%$ to $15\%$ across all ${\overline{D}}$ will not allow
any APC manufacturer any kind of sustainable business, in order to ensure the
bounded user risk promise of success chance $\leq\alpha/2=2.5\%$ for
$|{\overline{D}}|\geq\Delta=1\%$, the introduction of $\nu_{\text{min}}>0$ can
close the theoretical gap as can be seen in Figure 8 and Figure 9.

Figure 8: Test Success Simulation, comparing the equivalence test with the
partitioned equivalence test, $\nu_{\text{min}}=3\%$. Compared to Figure 7,
even for low sample sizes, test success $\leq\alpha/2=2.5\%$ for
$|{\overline{D}}|\leq\Delta=1\%$ is ensured. To our surprise, the partitioned
equivalence test can even be closer to the theoretical success probability
than the original equivalence test: we use $\nu_{\text{min}}$ for both tests,
but only in the non-partitioned test, the threshold is never triggered, since
safe and unsafe videos are mixed up with $\nu$ being around $10\%$. Also, the
partitioned equivalence test has a larger $n_{\text{rec}}$ to draw videos
from.

Figure 9: Test Success Simulation, comparing the equivalence test with the
partitioned equivalence test, $\nu_{\text{min}}=3\%$. As in Figure 8, test
success $\leq\alpha/2=2.5\%$ for $|{\overline{D}}|\leq\Delta=1\%$ is ensured.
However, $\nu_{\text{min}}=5\%$ penalizes the partitioned equivalence test
much stronger than $\nu_{\text{min}}=3\%$, increasing costs or increasing the
manufacturers risk. Therefore, we suggest $\nu_{\text{min}}=3\%$.