# Whole-examination AI estimation of fetal biometrics from 20-week ultrasound
scans

Lorenzo Venturini1 , Samuel Budd1, Alfonso Farruggia1, Robert Wright1,
Jacqueline Matthew1, Thomas G. Day12, Bernhard Kainz13, Reza Razavi12, Jo V.
Hajnal1
1
2
3  Department of Imaging Sciences, Faculty of Life Sciences and Medicine,
King’s College London  Guy’s and St Thomas’ NHS Foundation Trust, London
Department of Computing, Faculty of Engineering, Imperial College London,
London

###### Abstract

The current approach to fetal anomaly screening is based on biometric
measurements derived from individually selected ultrasound images. In this
paper, we introduce a paradigm shift that attains human-level performance in
biometric measurement by aggregating automatically extracted biometrics from
every frame across an entire scan, with no need for operator intervention. We
use a convolutional neural network to classify each frame of an ultrasound
video recording. We then measure fetal biometrics in every frame where
appropriate anatomy is visible. We use a Bayesian method to estimate the true
value of each biometric from a large number of measurements and
probabilistically reject outliers. We performed a retrospective experiment on
1457 recordings (comprising 48 million frames) of 20-week ultrasound scans,
estimated fetal biometrics in those scans and compared our estimates to the
measurements sonographers took during the scan. Our method achieves human-
level performance in estimating fetal biometrics and estimates well-calibrated
credible intervals in which the true biometric value is expected to lie.

###### Index Terms:

Ultrasound; Fetal imaging; Machine learning; Bayesian estimation; Biometric
measurement

##  I Introduction

Ultrasound (US) imaging is routinely used in many countries during pregnancy
to screen for fetal abnormalities, often at ~18-22 weeks [1]. Procedures
typically involve imaging of standard planes of fetal anatomy and measurement
of several fetal biometrics. In the UK, requirements for screening
examinations are published in the Fetal Anatomy Screening Programme (FASP)
standard [2].

Fetal biometrics are conventionally measured in single images: the operator
pauses the US image stream on a specified view, then manipulates calipers to
measure the anatomy. Some guidelines recommend that this whole procedure is
repeated up to three times to ensure reliability [3]. Traditionally, these
measurements have been done manually. Manual measurement of fetal biometrics
displays significant expected-value bias [4]. Selection bias may also
influence the planes used for measurement. Furthermore, these manual processes
display significant inter-observer variability, ranging from 4.9% to 11.1%
depending on the measurement [5].

Recent research has proposed methods for automating the detection of standard
anatomical planes in during live scanning live scanning [6, 7, 8]and biometric
measurements [9]. US machine manufacturers are increasingly integrating these
tools in their equipment [10]. However, fetal biometry generally still follows
existing workflows that rely on operator selection of individual images. This
does not take full advantage of the real-time nature of US, where a video
stream of tens of frames per second is acquired. A fully automatic system
could perform plane classification on every frame of the US stream and
biometric measurement using all available data.

Fully automation would reduce human selection bias from plane selection and
expected-value bias during measurement, potentially improving veracity,
reproducibility and reducing operator dependence. Automating some key operator
tasks could also reduce cognitive load, allowing the sonographer to focus more
on the patient and on identifying signs of anomalies.

Evaluating every frame in a fully automatic system generates a very large
number of measurements. A key challenge in making such a system useful for
clinical practice, therefore, is to generate a single estimate from the vast
resulting amount of information. There has been some work in developing
methods to calculate fetal biometry from a video feed. Płotka et al [11]
propose a system to select the best standard planes to extract biometry, and
Lee et al [12] average convolutional neural network (CNN) outputs across a
scan to estimate gestational age. Matthew et al [13] have proposed a system to
automatically classify every frame in a US scan and extract fetal biometrics,
but an operator still needs to select appropriate frames to report biometrics.

No work has, to our knowledge, sought to obtain an expected value of the
biometry and a credible interval by combining all frame measurements while
acknowledging potential outliers. This would represent a substantial departure
from current clinical practice, which is fully focused on a selection and
measurement of single images.

###  I-A Contributions

We propose a real-time system that can identify standard planes and estimate
fetal biometrics achieving and reporting progressively reducing uncertainties.
The system is designed to seamlessly link into clinical practice during
20-week US screening scanning. We build upon Sononet [6] for standard plane
detection, and pair it with automatic biometric estimation per frame,
aggregating these to generate a progressive global estimate using a Bayesian
framework. When used during live scanning, this results in progressively more
reliable central estimates and credible intervals for each biometric. The
proposed method has the potential to improve clinical practice by providing
robust measurements free from expectation bias, while reducing cognitive load
on the sonographer.

##  II Methods

###  II-A Datasets

Data fold | Train | Validation | Test | lowest p-value
---|---|---|---|---
Total scans | 4395 | 1457 | 1457 | N/A
Abnormalities | 528 (12.0%) | 155 (10.6%) | 163 (11.2%) | 0.360
Live birth | 4363 (99.2%) | 1442 (98.9%) | 1450 (99.5%) | 0.375
Avg. GA | 20w 3d | 20w 3d | 20w 3d | 0.653
Nonwhite | 1352 (30.8%) | 458 (31.4%) | 418 (28.7%) | 0.166
Avg. BMI (kg / m2) | 24.7 | 24.9 | 24.9 | 0.448
Table I: Patient characteristics and metadata across the data folds used for
model training. None of the patient characteristics, including gestational age
(GA), maternal body mass index (BMI), or percentage of abnormal scans were
significantly different across the chosen folds.

The data for this work is taken from the iFIND project on fetal US, which
recruited mothers attending a routine 20-week anomaly screening clinic in
London, UK. 10000 volunteers gave consent to have their full 20-week US
examinations (gestational age 18-22 weeks) recorded111www.ifindproject.com.The
examinations were conducted on identical US machines (GE Voluson E8) by 145
professional sonographers following FASP protocols between 2015 and 2020. The
sonographer identified, labelled, and saved standard plane images and manually
measured biometrics. To best reflect the screening population, all scans were
included regardless of whether the scan was reported as normal or abnormal.

Although the same machines were used throughout this study, software updates
during this period changed the video resolution and the interface presented by
the machine. Recordings were split across three resolutions depending on the
date on which they were acquired: $678\times 576$, $980\times 784$, and
$1280\times 1024$.

Due to operator error, technical glitches, and patient withdrawal of consent,
not all of these scans could be used in the dataset: 7309 video recordings of
prenatal US scans were used for this study.

To maintain patient anonymity, personal information was removed from the scan
recordings and each scan was labelled with a numerical iFIND ID, numbered
sequentially. We created training/validation/test splits across the dataset
assigned subjects using the trailing digit of their iFIND ID. Scans with an
iFIND ID with a trailing digit between 0-5 were used to train our models, 6-7
were used for validation, and 8-9 were held aside for testing and not used
during training of any of our models. The demographic characteristics of
patients across these folds, none of which showed statistically significant
differences, are shown in Table I.

####  II-A1 Validation dataset

Some of our methods were also validated using a small number of paired scans
from a smaller study [13]. 23 volunteers were recruited after their 20-week
anomaly scan for a follow-up scan. These ranged in gestational age from 20+0
to 24+0 gestational weeks (median 23GW), and had all screened as normal at
their anomaly scan. These subjects had paired scans (encompassing the FASP
planes and biometrics) conducted by separate sonographers on the same day,
using a Philips EpiQ 7 US machine. These scans were not used to train our
models.

###  II-B Data labelling

Standard plane | Description
---|---
Head:
Brain-CB | Transcerebellar view
Brain-TV | Transventricular view
Heart:
3VT | 3-vessel + trachea
LVOT | left ventricular outflow tract
RVOT | left ventricular outflow tract
4CH | 4-chamber view
Other:
Abdominal |
Femur |
Lips |
Kidneys | axial view
Profile | sagittal view
Spine-cor | coronal view
Spine-sag | sagittal view
Background
Table II: List of standard planes recommended under FASP. We added a
‘Background’ label for frames in none of these views.

The iFIND dataset includes full recordings of examinations that were labelled
during scanning with standard planes and biometrics according to the FASP
standard as shown in Table II (note there is an additional background class
for all other planes). These labels are present in the recording and were
extracted automatically. Typically, a sonographer freezes the video stream
once she identifies a suitable standard plane. Text labels and biometric
measurements are then overlaid on the frame, which is saved for the
sonographer’s report. Example frames with these overlays are shown in Figure
1.

####  II-B1 Extraction of standard planes

(a) (b)

Figure 1: Example frames with annotations and calipers. Shown are (a) a Brain-
TV image, with annotations for head circumference, biparietal diameter and
posterior ventricle, (b) a femur image, with the femur length measured and
annotated. These frames were acquired and saved by a sonographer.

To find labelled planes in the recorded examinations, we automatically
detected any pauses and freezes in the recordings, then used OCR
software222For this project, we used the open-source Tesseract OCR package
[14] to read text labels. to extract any added text labels. The text labels,
including biometric labels (such as the Brain-TV view in Figure 1a) was then
associated with standard planes and manually checked for consistency.

In the experiments performed in this paper, we removed segments where the
sonographer froze the frame and added annotations. We did this by finding
frames where over 95% of pixels did not change relative to the previous frame.
This eliminates the sonographer’s own annotations from the recordings we used
to test our methods.

####  II-B2 Biometrics

Biometric | Type | No. of labels
---|---|---
Brain-TV: |
HC | Head circumference | Ellipse | 8162
BPD | Biparietal diameter | Ellipse axis | 8162
Brain-CB: |
TCD | Transcerebellar diameter | Linear | 6906
Abdominal: |
AC | Abdominal circumference | Ellipse | 6073
Femur: |
FL | Femur length | Linear | 5718
Table III: Biometrics measured during the 20-week scan according to FASP
guidelines, and the number of such labels present in the iFIND1 dataset.

In the UK, the FASP standard mandates a minimum of 5 biometric measurements
per scan [2]333Other biometrics, such as the nuchal fold (NF), must be
measured in certain circumstances but are not required for every scan.. These
are measured across a range of standard planes, as shown in Table III.

(a) (b)

Figure 2: CaliperNet example heatmap outputs for (a) a femur image, and (b) a
brain-CB image with labels for the cerebellum, cisterna magna and nuchal fold.

The calipers identifying a biometric are generally connected by a dotted line,
with a text label next to them to identify the calipers (see Figure 2). To
extract these biometrics, we trained a simple U-Net based neural network which
we called CaliperNet (see Appendix A for details). We manually labelled 300
images of each biometric with the location of the calipers and trained a CNN
to find calipers, with very good accuracy.

###  II-C Pixel size estimation

(a) (b) (c)

Figure 3: (a) Part of a scale bar, cropped from an image frame. Two lines are
overlaid upon it. (b) The pixel values along those two scan lines for the
entire frame. (c) The autocorrelation of the pixel values of the blue line
after it has been processed with a high-pass filter.

The method described in Section III can find biometric endpoints on an image,
but the measurements are given in pixels rather than millimetres. Moreover,
pixel size can be changed during scanning by varying probe settings and
machine zoom levels.

However, almost every frame shows a standard scale bar with prominent ticks at
intervals $d_{\mathrm{bar}}=50$mm, with smaller ticks every 10mm and 5mm [15].
These ticks are at predictable positions on the screen: they can be seen
clearly by looking at pixel values $X_{i}$ along the correct scan line (see
Figure 3). Ticks can sometimes blend into bright structures shown on screen.

A simple, reliable and computationally efficient solution is to remove likely
background signals from $X_{i}$ by subtracting a 1D Gaussian filter,
$G_{x},$with standard deviation $\sigma$ empirically selected to be 3px and
keeping only positive values:

$X_{i}^{{}^{\prime}}=\max\left(X_{i}-X_{i}\ast G_{x}(\sigma),0\right),$ (1)

We then computed the autocorrelation $R_{XX}$ of the resulting filtered pixel
values $X_{i}^{{}^{\prime}}$ along the scan lines corresponding to the scale
bar (see Figure 3):

$R_{XX}(n)=\sum_{i}X_{i}^{{}^{\prime}}X_{i+n}^{{}^{\prime}}.$ (2)

The first peak of the autocorrelation function corresponds to the spacing
between axis ticks (in pixels). The size of individual pixels in millimetres
$L_{x}$ is given by

$L_{x}=\frac{d_{\mathrm{bar}}}{\max(R_{XX}(n))}.$ (3)

To minimise quantisation noise, we estimate pixel size from the bars with the
highest spacing.

###  II-D Biometric measurement

(a) (b)

Figure 4: Images used for biometric training showing (a) femur, and (b) head
views. The second row shows the training labels and the third row shows the
CNN output overlaid on the image. For the head image, the HC output is shown
in green and the BPD output in pink.

To measure biometrics, we trained a the U-Net segmentation network using
binary cross-entropy loss [9] to predict heatmaps constructed from the
sonographer’s point annotations, extracted by CaliperNet, convolved with a
Gaussian kernel. Figure 4 shows examples of training labels constructed this
way. A separate CNN was trained for each standard plane, using the same U-Net
architecture for each network and differing only in training planes and
labels.

Biometrics were extracted from unseen images directly from the predicted
heatmaps. For linear biometrics (FL and TCD), the Euclidean distance between
the coordinates of the two highest maxima was used; For elliptical biometrics
(HC and AC) we fit an ellipse to the heatmap using the least-squares
criterion. The ellipse perimeter was then estimated using the formula
recommended by the British Medical Ultrasound Society444The exact calculation
of the perimeter of an ellipse is an infinite series. The cited guideline
suggests using the first five terms of this series, which has a relative error
of $<0.01\%$ for most biologically plausible ellipse shapes. [16].

Since BPD is mathematically equivalent to the minor axis of the ellipse used
to segment the head, we calculated it directly from the head circumference
label.

###  II-E Whole-scan biometric estimation

In clinical practice, only a few frames are labelled by the sonographer.
Generally, no more than three measurements are taken of each biometric [2] and
the best subjective measurement is reported.

Figure 5: The pipeline which each frame in an US scan feed is processed by to
estimate biometrics.

The automated methods described in Section II-D can extract measurements in
all identified frames in which a given biometric is visible, resulting in
hundreds or thousands of measurements per biometric. These form sample
populations from which we wish to infer a true value and to estimate
confidence intervals.

However, for each biometric there may be erroneous values, caused by
misclassification of the target anatomical planes or by failure modes of the
biometric CNN. Any estimation needs to reject or at least minimise the effect
of these out-of-distribution measurements to reliably converge on the true
biometric value.

Figure 5 shows our processing pipeline. Every frame from the US real time
image stream is processed by Sononet to identify and label standard planes.
Following Baumgartner et al [6]:, each frame is resized to a resolution of
$288\times 224$ and converted to greyscale.

Any frame identified with $>95\%$ confidence as a standard plane with a FASP
biometric is then processed by our CNNs to obtain a biometric measurement. The
native frame is subsampled to a $384\times 288$ resolution, with interface
elements removed, and converted to greyscale and processed as described in
Section II-D.

A quick check is then applied: if the proposed measurement is anatomically
implausible (lower than the 3rd centile of the biometric at $\mathrm{GA}-3$
weeks or higher than the 97th centile at $\mathrm{GA}+3$ weeks), it is
rejected. Furthermore, where the CNN’s output does not allow an ellipse to be
fitted or endpoints to be returned (for instance, where one distinct maximum
is found, making it impossible to find two endpoints) the output is discarded.

These two filters (of plane-classification confidence and biometric
measurements) may be undesirable for single-frame comparisons, as they discard
some data. However, when estimating biometrics across a whole scan several
hundred measurements of each biometric are often recorded: if some fraction of
these are discarded, this only has a small impact on data availability.
Appendix B details the proportion of biometric measurements discarded.

####  II-E1 Distribution of measurements

Figure 6: Idealised distribution of measurements for a biometric with true
value 30mm. The observed distribution of measurements is a sum of a
measurement distribution and a nuisance distribution of outliers.

This section describes how measurements of individual frames are aggregated
and outliers are rejected using a Bayesian framework.

Repeated measurements of the same biometric in different frames should cluster
around a mean value. We modelled this using a normal distribution
$\mathcal{N}\left(\mu,\sigma^{2}\right)$, with mean $\mu$ (if unbiased, this
should be the true biometric value) and variance $\sigma^{2}$(this depends on
the precision of the biometric measurements).

Misclassified planes also return a biometric measurement and the biometric
measurement process itself may fail even in a correctly identified plane.
Since we have no information with which to model these failures we treat them
as random and use a uniform distribution $U\left(a,b\right)$, over the full
biological limits $a$ and $b,$ outside of which a measurement is rejected.

Thus, the observed distribution $D_{o}$ of measurements will be a weighted sum
of the measurement distribution $\mathcal{N}\left(\mu,\sigma^{2}\right)$ (for
correct classifications) and a nuisance distribution $U\left(a,b\right)$ (for
misclassifications). This can be represented by

$D_{o}=P_{t}\mathcal{N}\left(\mu,\sigma^{2}\right)+(1-P_{t})U\left(a,b\right),$
(4)

where $P_{t}$ represents the proportion of valid measurements.

This equation contains several unknowns $P_{t},\mu,\sigma^{2}$ which can be
estimated iteratively using a Bayesian approach, but each must be initialised.

####  II-E2 Estimating truth probability and distribution parameters

To perform a Bayesian estimation, $P_{t}$needs to be initialised to a prior
probability (in our approach, we set $P_{t_{0}}=0.75$ for all biometrics, but
it should converge on the true probability from any starting point other than
0 or 1), along with a prior weight $W_{0}$555The estimate should converge on
the true value of $P_{t}$ regardless of what $W_{0}$ is set to, but setting it
too high can cause it to converge slowly, while setting it too low can cause
the estimate of $P_{t}$ to oscillate inappropriately.. Prior estimates of the
distribution parameters $\mu_{0}$ and $\sigma_{0}^{2}$ need to be set, along
with appropriate prior weights $W_{\mu,0}$ and $W_{\sigma^{2},0}$.

For each new measurement $x_{i}$, we estimate the likelihood of it being a
true measurement $T$ or a nuisance sample $\overline{T}$ using Bayes’ rule:

$P(T|x_{i})=\frac{P(x_{i}|T)P_{t}}{P(x_{i}|\overline{T})(1-P_{t})}$ (5)

where $P(x_{i}|T)$ is the value of $\mathcal{N}\left(\mu,\sigma^{2}\right)$ at
$x_{i}$, and $P(x_{i}|\overline{T})$ is the value of $U\left(a,b\right)$ at
value $x_{i}$. This returns $P(T|x_{i}),$ the probability that this
measurement was sampled from the true measurement distribution.

Finally, $P_{t}$ and the distribution parameters $\mu,\sigma^{2}$ can be
updated for this biometric using a weighted cumulative average:

$\displaystyle P_{t_{i}}$ $\displaystyle=$
$\displaystyle\frac{P(T|x_{i})+W_{i-1}P_{t_{i-1}}}{W_{i-1}+1}$ (6)
$\displaystyle\mu_{i}$ $\displaystyle=$
$\displaystyle\frac{x_{i}P(T|x_{i})+W_{\mu,i-1}\mu_{i-1}}{P(T|x_{i})+W_{\mu,i-1}}$
(7) $\displaystyle\sigma_{i}^{2}$ $\displaystyle=$
$\displaystyle\frac{(x_{i}-\mu_{i-1})^{2}P(T|x_{i})+W_{\sigma^{2},i-1}\sigma_{i-1}^{2}}{P(T|x_{i})+W_{\sigma^{2},i-1}}$
(8)

and their weighting factors can be updated using

$\displaystyle W_{i}$ $\displaystyle=$ $\displaystyle W_{i-1}+1$ (9)
$\displaystyle W_{\mu,i}$ $\displaystyle=$ $\displaystyle
W_{\mu,i-1}+P(T|x_{i})$ (10) $\displaystyle W_{\sigma^{2},i}$ $\displaystyle=$
$\displaystyle W_{\sigma^{2},i-1}+P(T|x_{i}).$ (11)

As the noise distribution is taken to be uniform, there is no need to update
estimates of its terms.

The estimates for $\mu$ and $\sigma^{2}$ are updated in real time and
independently. At any given moment, the estimate of $\mu$ is the best estimate
of the true value of the biometric of interest. Meanwhile, the estimate of
$\sigma^{2}$ can be used to calculate the standard error ,$\hat{\sigma}_{i}$,
of that estimate to find credible intervals for the biometric, given by

$\hat{\sigma}_{i}=\frac{\sigma_{i}}{\sqrt{W_{\sigma^{2},i}}}.$ (12)

The 95% credible interval is given by the range $\mu_{i}\pm
2\hat{\sigma}_{i}$.

##  III Experiments

The variability in biometric measurements between two sonographers arises from
two components: image selection for measurement and caliper placement within
each image. Previous work has shown that 50-80% of variability between humans
can be explained by caliper placement [5], leaving 20-50% of variability
accounted for by image selection. The proposed whole-scan method eliminates
this second source of noise by processing every frame in a scan with no manual
intervention and seeks to reduce the effect of random error in caliper
placement by aggregating measurements from many individual frames.

We performed three experiments to quantify the reliability of our biometric
estimates:

  1. 1.

Using our biometric CNNs to independently measure the same frames that
sonographers labelled. This tests the biometric measurement performance of our
CNNs and compares it to published estimates of inter-rater differences in
caliper placement on a single image. Note this is still a restricted problem,
requiring human intervention to perform frame selection. For these single-
image experiments, biometrics were estimated automatically in the same frames
that sonographers chose to make their manual annotations. We trained one CNN
per standard plane (from subjects in the training set) and tested it using the
subjects in the test set using the unannotated copies previously extracted for
all frames in the iFIND test set: 1516 in the brain-TV standard plane, 1360
for brain-CB, 1205 for abdominal, and 1124 for femur.

  2. 2.

Aggregating biometric measurements across a whole recorded scan to obtain
global estimates for the scan and comparing to sonographer manual
measurements. This experiment considers a more unconstrained problem with no
human interaction. It does not control for individual frame selection, so our
results can be compared to inter-rater variability in biometric measurement
from different scans. All 1457 recordings in the test set were used. The final
estimate of each biometric, along with credible intervals, was reported.

  3. 3.

Conducting a test-retest experiment on the paired scan data described in
Section II-A1. Finally, we conducted a test-retest experiment on the paired
scan data described in Section II-A1. We ran the full pipeline on paired scans
of the same subject acquired on the same day using a different scanner from
the training dataset. We then compared biometrics from each scan and measured
test-retest variability of our algorithm. The scans here were performed on an
US machine by a different manufacturer from the training data, which allowed
us to examine domain shift.

All experiments were conducted on a single Nvidia GeForce RTX 3080 GPU.

###  III-A Validation of distribution choice

Section II-E models the distribution of measurements of each biometric with a
mixture of a uniform distribution and a Gaussian distribution

$D=P_{t}\mathcal{N}\left(\mu,\sigma^{2}\right)+(1-P_{t})U\left(a,b\right).$
(13)

This has three parameters $P_{t},\mu$ and $\sigma$ which are tuned iteratively
to incoming data. This model assumes that incoming measurements follow a
Gaussian distribution (if correctly classified) or a uniform distribution (if
noise), which can be tested. We therefore examined the output biometric
measurements to check the appropriateness of this model.

##  IV Results

###  IV-A Single-frame biometric estimation

Structure | Bias (mm) | MSD (mm) | Human MSD
---|---|---|---
HC | +0.11 mm (+0.07%) | 4.34mm (2.68%) | 2.70mm (1.8%)
BPD | +0.06mm (+0.17%) | 1.28mm (3.20%) | N/A
AC | +1.51mm (+0.98%) | 5.03mm (3.26%) | 4.03mm (2.8%)
FL | 0.00mm (0.00%) | 1.53mm (4.67%) | 1.02mm (2.9%)
TCD | -0.16mm (-0.75%) | 2.1mm (10.1%) | N/A
Table IV: Comparison of the biometric measurements obtained on the test set of
the iFIND dataset. ‘Bias’ refers to the average difference in measurement
between our method and human sonographers. ‘MSD’ is the mean-squared
difference between pairs of measurements. The ‘Human MSD’ column is populated
with values of inter-observer variability reported by Sarris et al [5].

(a) (b)

Figure 7: Two failure cases of biometric estimation in (a) HC, and (b) FL. The pink annotation shows the ground truth measurement made by a sonographer, while the green annotation is the automatic one made by our CNNs. Biometric | Bias | MSD | MAD (mm) | std. err (mm) | within human 95%
---|---|---|---|---|---
HC | -0.30mm (-0.18%) | 3.23mm (1.99%) | 2.50mm (1.42%) | 3.39mm | 94.7%
BPD | +0.90mm (+1.89%) | 1.86mm (4.65%) | 1.11mm (2.31%) | 0.95mm | N/A
AC | -0.49mm (-0.31%) | 5.55mm (3.60%) | 4.23mm (2.74%) | 3.26mm | 96.6%
FL | -0.51mm (-1.59%) | 1.63mm (4.99%) | 1.19mm (3.64%) | 1.12mm | 93.9%
TCD | +0.27mm (+1.32%) | 0.91mm (4.37%) | 0.68mm (3.28%) | 0.47mm | N/A
Table V: Comparison of human measurements to machine estimates taken across a
whole scan. ‘MAD’ refers to mean absolute difference between human and machine
measurements. The ‘within human 95%’ column shows the percentage of biometrics
in the test set where the human-machine difference is within 95% of human-
human inter-observer differences (taken from Sarris2012 [5])

A valid biometric estimate (following the constraints outlined in Section
II-E) was obtained in approximately 90% of cases across all biometrics. Table
IV shows performance on the unlabelled copies of the frames labelled by
sonographers in the test dataset.

Using the sonographer measures as the reference point, the automated single
frame performance showed bias of less than 1% and mean square difference (MSD)
of less than 5% except for TCD which showed a MSD of just over 10%. Where
inter-observer data is available for manual measurements, the automated
approach could be seen to have an average of 41% lower agreement with the
human rater.

###  IV-B Whole-scan processing

Figure 8: Evolution over time of the HC estimate in an individual subject,
overlaid with credible intervals and individual frame measurements. Figure 9:
A still image from a video showing real-time measurement of biometrics and
updating of the biometric estimate. In three seconds of real-time scanning,
our system acquired 83 FL measurements and updated its FL estimate from 34mm
(the prior, based on the average FL for this gestational age) to 36mm. The
full video clip can be found at http://www.ifindproject.com/wp-
content/uploads/sites/79/2023/12/realtime.gif

Figure 10: Bland-Altman plots showing the differences between the
sonographer’s measurements and our own across all complete recordings in the
test dataset.

Figure 8 shows how a biometric estimate (in this case HC) changes over time
during one exemplar scan, overlaid with measurements from individual frames.
Initially the prior dominates, estimating the 50th percentile for the age with
wide error bars. As the number of frames contributing measurement values
increases the overall estimate becomes more stable and resistant to outliers.
Although individual frame measurements show significant noise, including
extreme values (which are probabilistically discounted by our system), the
credible interval progressively shrinks with increasing data. Figure 9 shows
this happening in real time over 3 seconds of scanning.

Figure 10 shows Bland-Altman plots for human and machine measurement on the
same subjects for three biometrics, with 95th percentile intervals overlaid
for three biometrics. The difference between human measurements666Human
measurements were taken from sonographers’ reports after each scan. and
machine estimates can be compared to inter-rater disagreement measured by
Sarris et al [5]. Our 95th percentile intervals are overall very similar to
human inter-rater variability for HC, AC, and FL. 95.1% of machine-human
measurement differences (across these three biometrics) lie within the 95%
range of human differences.

###  IV-C Paired scan data

Biometric | MSD | Standard deviation | Human s.d. (full scan) | Number of pairs
---|---|---|---|---
HC | 2.60mm (1.25%) | 1.23% | 2.75% | 15
BPD | 0.79mm (1.36%) | 1.17% | N/A | 15
AC | 4.61mm (2.51%) | 2.50% | 5.05% | 17
FL | 0.97mm (2.44%) | 2.44% | 5.56% | 10
TCD | 0.57mm (2.33%) | 1.62% | N/A | 8
Table VI: Test-retest reliability of our model outputs from paired scans of
the same subject. The ‘Human s.d.’ column is populated with results by Sarris
et al [5].

We also conducted a test-retest experiment to measure reliability across
repeated scans of the same subject.

Table VI shows the mean-square difference (MSD) and the standard deviation of
the difference in the final biometric estimates from the test-retest
experiment using data from 20 subjects scanned twice on the same day by
different sonographers, as described in Section II-A1777We measured percent
standard deviation in this table to ensure comparability with the measurements
reported by Sarris et al [5], who reported human standard deviation. They
examined a slightly different gestational age range, so we report percent
standard deviation to normalise for that.. Not all pairs of scans had
sufficient frames visible of each biometric in both scans: the number of pairs
available for analysis is shown in the table.

###  IV-D Measurement distributions

(a) (b)

Figure 11: (a) The histogram of all femur length measurements made in
individual frames in the test set. (b) The cumulative distribution function of
all femur length measurements overlaid with the equivalent distribution for a
fitted model using a mixed Gaussian + uniform distribution.

Figure 11(a) shows the distribution of all femur length measurements across
the test dataset. The values were demeaned and then aggregated across all
scans.

Figure 11(b) shows the cumulative distribution function (cdf) of all
measurements obtained that way, as well as the best-fit theoretical model
distribution for this data. The global best-fit parameters for the femur
length were $P_{t}=0.79$, $\mu=0$ and $\sigma=1.8$mm.

Similar curves were found for the other biometrics examined with comparable
goodness of fit.

##  V Discussion

Our experiments show that the proposed workflow can improve the accuracy and
repeatability of biometric estimation. In single frames, our models generally
produce more variable biometric estimates than humans. However, when
estimating biometrics across an entire examination, our estimates are in line
with those made by sonographers (machine-human MSD ~ human-human MSD). Paired
scan data also shows that our results are highly repeatable, with little
difference across scans of the same subject. This latter test is a direct
analogue of inter-rater testing by human observers and results were in closer
agreement than for manual measurement.

###  V-A Single-frame biometric estimation

Section IV-A compares our model’s performance in individual frames to that of
a human. We generally find good agreement with sonographer measurements.
Sarris et al [5] report inter-observer differences when two sonographers
manually place calipers on the same US frame for three biometrics. When
referenced to the corresponding human measurements, the machine measurements
in this experiment across all biometrics show more divergence than humans do
in a single frame. This appears to be random error: there is little observed
bias.

Part of this difference can be explained by failure modes of our networks.
Figure 7 shows some frames for which the models give an inaccurate biometric
estimate. This typically occurs when structures of interest are poorly
visualised in the US image, or in some edge cases such as when both femurs are
visible in a frame. These failure modes are uncommon (divergences of over 20%
as shown in Figure 7 are seen in only 1.5% of segmentations) but they are not
mistakes that a human is likely to make.

There is significant variation in human-machine difference across biometrics.
TCD shows a very large MSD of over 10%, while other biometrics are
significantly more closely aligned to the human measurement. Human inter-
observer variability in TCD measurements has been reported as being between
3-5% [17], so the machine method appears to show significantly greater
variability.

###  V-B Whole-scan biometric estimation

The clinical standard for biometric measurement in fetal US relies on operator
selection and annotation of a small number of individual frames. We take a
different approach: we analyse every frame in which the biometry is visible,
and use all available measurements to generate a global estimate of the
biometric. This avoids operator dependence associated with selection of
standard planes, which can be a substantial contributor to inter-rater
variability. Furthermore, it reduces the impact of random error from caliper
placement in any one plane, which is also a significant source of variability.

The overall estimates of biometrics across a scan show better agreement with
manual measurements than those from a single image. For instance, the TCD
estimates across a scan show an average MSD with the manual measurement of
4.37%, compared to 10.1% for individual frames. This is despite the fact that
the frames selected by the sonographers were removed: aggregating measurements
from a diverse and unselected range of views of the anatomy results in a
similar level of agreement with the sonographer. This value is also comparable
with human inter-observer variability estimates found in the literature [17].

There are also few extreme outliers, demonstrating that our system achieves
human-level performance. This is despite the fact that performance of our CNNs
on individual frames is lower than that achieved by humans: our whole-scan
estimation method can compensate for the errors introduced by our single-frame
estimation CNNs. We expect that our whole-scan algorithm could therefore
achieve superhuman performance in biometric estimation if the single-frame
estimator can be improved to human level. The test-retest experiment we
performed in Section IV-C provides support for this, showing the variability
in biometric estimates across scans is much lower than that between humans.

Furthermore, our method allows us to estimate credible intervals in which a
given biometric can be expected to lie. This is based on the distribution of
measurements and the number of observations, as described in Section II-E. On
average, this is smaller than the MSD by a factor of 1.4. However, the manual
measurement itself displays variability a: assuming Gaussianity, MSD should be
larger than the standard error by a factor of $\sqrt{2}\approx 1.4$.
Therefore, the calculated standard error is in line with the observed
variability in our estimates. Our credible intervals therefore provide a
quantitative measure of the uncertainty in biometric estimation.

The system discussed in this paper treats single-frame measurements in a US
scan as following a Gaussian distribution (when they are not due to
misclassifications). Section IV-D shows that this is globally an appropriate
assumption, with measurements following the expected distribution. The
distribution of measurements within each scan may be variable, depending on
the operator’s choice of scanning planes, but the Bayesian estimation method
ensures that an optimal estimate is always achieved given the available data.

Another advantage of this system, when applied in real time, is that it
requires no sonographer interaction to obtain biometrics. Much of a prenatal
US scan is spent acquiring biometrics, and removing the need for this
measurement can reduce scan time by up to a third [13]. A prospective trial of
this system would be needed to estimate and quantify a time saving.

###  V-C Whole-scan performance and test-retest reliability

Section IV-C shows results of our models on paired data from the same scan.
Despite a small number of pairs, across all biometrics our models are more
repeatable and consistent than humans. The standard deviation of the
difference between measurements of the same subject was approximately half of
that between human sonographers. Much of the remaining difference can likely
be explained by the different views of the relevant biometrics acquired in
each scan determined by sonographer technique and fetal lie. This experiment
was conducted in a sample scanned with a different US scanner and at a higher
gestational age range than the training dataset, yet the model outputs
remained robust to this domain shift.

##  VI Conclusion

This paper has demonstrated a novel method using machine learning to estimate
fetal biometrics at the 20-week scan. The proposed method does not rely on
measurements performed on individual manually-selected standard planes, as is
the norm in both manual measurements and for many commercial machine assisted
systems, but estimates biometrics across the entire scan. This avoids the
biases exhibited by humans in plane selection and biometric measurement. We
further present evidence that estimates produced using the proposed methods
may be more consistent and repeatable than human measurements. This approach
can also present other benefits that have not been quantified in this paper:
by removing the need for sonographers to freeze the US stream and perform
measurements, the system can improve sonographer focus and reduce the time
needed to complete a FASP scan. A prospective trial of this system is in
progress to establish whether this can realise those benefits and improve
detection of fetal abnormalities.

## Data availability statement

The US scan recordings used to train the CNNs used in this paper were
collected with an ethical requirement for patient data to remain confidential.
As such, the scan data cannot be made publicly available. The raw and
processed CNN outputs used to construct our results are available from the
corresponding author on reasonable request.

## Acknowledgements

This work was supported by the Wellcome Trust/EPSRC iFIND grant (IEH award
102431), by core funding from the Wellcome/EPSRC Centre for Medical
Engineering [WT203148/Z/16/Z] and by the National Institute for Health
Research (NIHR) Clinical Research Facility based at Guy’s and St Thomas’ NHS
Foundation Trust and King’s College London.

## Competing interests

The authors declare the following competing interests:

All authors are co-inventors on a patent filing related to the core methods
described in this work filed by King’s College London (patent application
number P333GB).

All authors are co-founders and hold equity ownership in Fraiya Ltd, an entity
that is commercialising this technology.

## References

  * [1] L. J. Salomon, Z. Alfirevic, F. D. S. Costa, R. L. Deter, F. Figueras, T. Ghi, P. Glanc, A. Khalil, W. Lee, R. Napolitano, A. Papageorghiou, A. Sotiradis, J. Stirnemann, A. Toi, and G. Yeo, “Isuog practice guidelines: ultrasound assessment of fetal biometry and growth,” Ultrasound in Obstetrics and Gynecology, vol. 53, pp. 715–723, 6 2019.
  * [2] N. H. S. England, “Fetal anomaly screening programme handbook,” 2021.
  * [3] L. J. Salomon, Z. Alfirevic, V. Berghella, C. M. Bilardo, G. E. Chalouhi, F. D. S. Costa, E. Hernandez-Andrade, G. Malinger, H. Munoz, D. Paladini, F. Prefumo, A. Sotiriadis, A. Toi, and W. Lee, “Isuog practice guidelines (updated): performance of the routine mid-trimester fetal ultrasound scan,” Ultrasound in Obstetrics and Gynecology, vol. 59, pp. 840–856, 6 2022.
  * [4] L. Drukker, R. Droste, P. Chatelain, J. A. Noble, and A. T. Papageorghiou, “Expected-value bias in routine third-trimester growth scans,” Ultrasound in Obstetrics and Gynecology, vol. 55, pp. 375–382, 3 2020.
  * [5] I. Sarris, C. Ioannou, P. Chamberlain, E. Ohuma, F. Roseman, L. Hoch, D. G. Altman, and A. T. Papageorghiou, “Intra- and interobserver variability in fetal ultrasound measurements,” Ultrasound in Obstetrics and Gynecology, vol. 39, pp. 266–273, 2012.
  * [6] C. F. Baumgartner, K. Kamnitsas, J. Matthew, T. P. Fletcher, S. Smith, L. M. Koch, B. Kainz, and D. Rueckert, “Sononet: Real-time detection and localisation of fetal standard scan planes in freehand ultrasound,” IEEE Transactions on Medical Imaging, vol. 36, 2017.
  * [7] H. Chen, Q. Dou, D. Ni, J. Z. Cheng, J. Qin, S. Li, and P. A. Heng, “Automatic fetal ultrasound standard plane detection using knowledge transferred recurrent neural networks,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9349, pp. 507–514, 2015.
  * [8] X. P. Burgos-Artizzu, D. Coronado-Gutierrez, B. Valenzuela-Alcaraz, E. Bonet-Carne, E. Eixarch, F. Crispi, and E. Gratacos, “Evaluation of deep convolutional neural networks for automatic classification of common maternal fetal ultrasound planes,” Scientific Reports 2020 10:1, vol. 10, pp. 1–12, 6 2020.
  * [9] M. Sinclair, C. F. Baumgartner, J. Matthew, W. Bai, J. C. Martinez, Y. Li, S. Smith, C. L. Knight, B. Kainz, J. Hajnal, A. P. King, and D. Rueckert, “Human-level performance on automatic head biometrics in fetal ultrasound using fully convolutional neural networks,” Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, vol. 2018-July, pp. 714–717, 10 2018.
  * [10] M. Yaqub, N. Sleep, S. Syme, Z. Chen, H. Ryou, S. Walton, J. A. Noble, and A. T. Papageorghiou, “491 scannav Â® audit: an ai-powered screening assistant for fetal anatomical ultrasound,” American Journal of Obstetrics and Gynecology, vol. 224, p. S312, 2 2021.
  * [11] S. Plotka, T. Wlodarczyk, A. Klasa, M. Lipa, A. Sitek, and T. Trzcinski, “Fetalnet: Multi-task deep learning framework for fetal ultrasound biometric measurements,” Communications in Computer and Information Science, vol. 1517 CCIS, pp. 257–265, 2021.
  * [12] C. Lee, A. Willis, C. Chen, M. Sieniek, A. Watters, B. Stetson, A. Uddin, J. Wong, R. Pilgrim, K. Chou, D. Tse, S. Shetty, and R. G. Gomes, “Development of a machine learning model for sonographic assessment of gestational age,” JAMA Network Open, vol. 6, pp. e2248685–e2248685, 1 2023\.
  * [13] J. Matthew, E. Skelton, T. G. Day, V. A. Zimmer, A. Gomez, G. Wheeler, N. Toussaint, T. Liu, S. Budd, K. Lloyd, R. Wright, S. Deng, N. Ghavami, M. Sinclair, Q. Meng, B. Kainz, J. A. Schnabel, D. Rueckert, R. Razavi, J. Simpson, and J. Hajnal, “Exploring a new paradigm for the fetal anomaly ultrasound scan: Artificial intelligence in real time,” Prenatal Diagnosis, vol. 42, pp. 49–59, 1 2022.
  * [14] R. Smith, “An overview of the tesseract ocr engine,” Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, vol. 2, pp. 629–633, 2007.
  * [15] Voluson E8/E8 Expert Basic User Manual.  GE Healthcare, 2012.
  * [16] B. 3rd Trimester Special Interest Group, “Professional guidance for fetal growth scans performed after 23 weeks of gestation,” 2022.
  * [17] M. R. Chavez, C. V. Ananth, J. C. Smulian, S. Lashley, E. V. Kontopoulos, and A. M. Vintzileos, “Fetal transcerebellar diameter nomogram in singleton gestations with special emphasis in the third trimester: A comparison with previously published nomograms,” American Journal of Obstetrics and Gynecology, vol. 189, pp. 1021–1025, 10 2003.
  * [18] “Labelbox | data-centric ai platform for building intelligent applications.”
  * [19] S. Constantine, A. Kiermeier, and P. Anderson, “The normal fetal cephalic index in the second and third trimesters of pregnancy,” Ultrasound Quarterly, vol. 36, pp. 255–262, 2020.

##  Appendix A CNN for caliper localisation

###  A-A Introduction

Figure 12: Zoomed examples of two calipers in the training set. These are at
different scales, with different backgrounds and levels of overlap with other
labels: it is difficult to design a machine method that can extract both
reliably.

For this project, we used the sonographer’s annotations taken during their
ultrasound scans as a source of data labels. These are readily visible in the
scan and trivial for a human to detect: the calipers places by sonographers
are prominent in their shape and colour. However, there were too many calipers
to feasibly extract manually: approximately 40,000 labelled images were used
in this paper.

Similarly, it is difficult to use a machine algorithm to extract these
calipers automatically. Though all ultrasound scans were performed on
identical hardware, there were several software updates over the course of the
iFIND study, which changed the resolution and rendering of calipers. These
weren’t always consistently rendered on the machine, as they appeared to be
interpolated and therefore displayed differently. Furthermore, calipers often
overlap with text annotations or other calipers, making it difficult to
reliably distinguish them. Figure 12 shows two very different caliper shapes
on different backgrounds.

We used a simple ML tool, which we called CaliperNet, to extract these labels
automatically.

###  A-B Labels and training

Figure 13: The interface of Labelbox which we used to annotate calipers, with
an example annotation of a transcerebellar image with three manual
annotations.

We used a sample of 300 images per standard plane to be manually annotated
with caliper locations. We obtained these labels from a single human annotator
on Labelbox [18]. Caliper locations are exact, so we expect very little
labelling noise.

Labelbox does not have a native ellipse annotation facility, so to trace
ellipse labels we asked the annotator to pinpoint five points along the
perimeter of the ellipse: five points are sufficient to fully constrain an
ellipse’s parameters. Training labels were generated in the same way as in the
main paper: for length biometrics, the endpoints were convolved with a
Gaussian kernel to generate a heatmap. For ellipse biometrics, the outline of
the ellipse was convolved with a Gaussian kernel.

We used the same U-Net architecture as for our full biometric models to train
CaliperNet. The only difference was the retention of image colour channels:
calipers in the GE interface are coloured yellow, while the rest of an
ultrasound image is usually in greyscale (though the operator can change the
colour map), so important information is carried by the colour channel.

We trained three separate models: one to extract ellipse annotations (such as
HC and AC), one to extract caliper locations in the femur image, and one to
extract TCD locations in transcerebellar brain images. The TCD model had to be
trained separately as for most images in that plane, the sonographer took
three annotations and placed three sets of calipers: the TCD, the cisterna
magna (CM), and the nuchal fold (NF)888Of these, the TCD measurement is in the
FASP standard and is required to always be measured. Nuchal fold must be
measured only if it is abnormally thick, while cisterna magna was in a
previous version of the FASP standard but is no longer required.. Figure 13
shows an example image with three sets of calipers. For TCD calipers, we
generated multi-channel labels, with one channel per annotated structure. Not
all images in this standard plane had all three biometrics labelled: if
calipers were missing, the relevant channel was left blank.

Training was performed using a 75:10:15 train:validation:test split for
images. We used the same hardware to train CaliperNet models as the models
described in the main paper.

###  A-C Results

Biometric | Bias (%) | MSE (%) | Dice coefficient
---|---|---|---
HC | -0.01% | 0.30% | 0.821
AC | 0.00% | 0.34% | 0.819
FL | +0.09% | 0.66% | 0.861
TCD | +0.07% | 0.95% | 0.946
Table VII: Results for each CaliperNet trained on one biometric.

(a) (b)

Figure 14: Annotations with the largest % errors in the test sets: green is
the ground-truth annotation and pink are the CaliperNet outputs. Shown in (a)
femur (length error of about 1%), and (b) abdomen (about 2%).

Table VII shows the performance of CaliperNet on our test dataset across the
structures of interest. There is very little error, showing that this is a
fairly straightforward task for these networks. Often, the pixel localisations
are exactly the same: the errors are usually on the order of $\pm 1$px in
specific pixel locations. The ‘Dice coefficient’ column measures how well the
network output maps replicate the training labels. While it is high, it is not
perfect: this may be because the limited training data makes it difficult for
the CNN to learn to replicate the Gaussian kernels consistently. Nevertheless,
the regressed endpoints and ellipses remain reliable.

There were no failure cases across our test dataset. Figure 14 shows the
images with the largest errors (-1% and +2%) in biometric between the manual
annotation of the calipers and the CaliperNet output. In both cases, the
difference was of 1px in endpoint localisation \- within labelling noise. This
reflects the fact that caliper localisation is a trivial task for humans, and
is not a challenging task for CNNs. Therefore, we can have confidence in the
labels generated by CaliperNet and can use them to generate training labels
for our biometric CNNs.

##  Appendix B Additional performance metrics for biometric CNNs

###  B-A Introduction

Two additional metrics were used to measure the performance of our biometric
CNNs and to filter for failure cases at test time. These could give
substantial indicators of failure cases, making them useful to reject failure
cases and reduce noise in our whole-video estimates. One metric, the Dice
similarity coefficient of a reconstructed heatmap, was used to measure the
quality of fit of derived ellipses and points from the CNN’s heatmap: where
this was too low, the fit was deemed of insufficient quality to accept the
resulting measurement. The other was the eccentricity of the output ellipses
for head and abdominal circumference measurements: where this was anatomically
implausible, the measurements were rejected.

###  B-B Dice similarity coefficient

$\mathrm{DSC}=0.805$ $\mathrm{DSC}=0.583$

(a) (b)

Figure 15: Two comparisons of the output heatmap from a CNN and the resulting
‘reconstructed’ ellipse. (a) a success case with $\mathrm{DSC}=0.805$, (b) a
failure case with $\mathrm{DSC}=0.583$.

The Dice similarity coefficient (DSC) is a measure of overlap between two sets
defined as

$\mathrm{DSC}(X,Y)=\frac{2\left|X\cap
Y\right|}{\left|X\right|+\left|Y\right|}$

and is often used in image segmentation tasks to measure how closely a
segmentation follows the ground truth.

Our biometric CNNs output heatmaps based on endpoints and ellipses, as
described in Section II-D. These are then processed to return a biometric
measurement: for linear measurements, endpoints are found from local maxima of
the heatmap; for ellipses, an ellipse is fit using least-squares to the
heatmap.

This can then be checked by generating a heatmap from the resulting
annotation, using the same process described in Section II-D. The two heatmaps
should match exactly in the case of a perfect output. In reality, there is
always some deviation, but when the reconstructed ellipse faithfully follows
the output heatmap, the resulting DSC is high.

We found that a $\mathrm{DSC}<0.6$ between the output heatmap and the
reconstruction indicated a poor reconstruction of the CNN output. Therefore,
we discarded any proposed reconstructions that fell below that threshold, as
the output would be very noisy and inaccurate in following anatomical
boundaries.

Figure 15 shows two example cases of reconstructed ellipses based on an output
heatmap. (a) shows a success case, where the reconstructed ellipse closely
follows the output heatmap: this would be accepted. (b) shows a case where the
output heatmap is quite noisy and results in poor fit of the reconstructed
ellipse. This has low $\mathrm{DSC}$ and therefore is rejected.

###  B-C Ellipse eccentricity

$e=0.242$ $e=0.670$

(a) (b)

Figure 16: Biometric failure cases which violate the eccentricity constraint
in (a) the head, and (b) the abdomen.

For ellipse biometrics (HC and AC), another relevant metric to consider at
test time is eccentricity.

$e=\sqrt{1-\frac{a^{2}}{b^{2}}}$

where $a$ is the minor axis of the ellipse and $b$ is the major axis. For the
head, $a$ is the BPD measure and $b$ is the OFD (occipito-frontal diameter).
While this is a commonly used mathematical measure, in obstetrics the more
common measure for head measurements is the cephalic index, defined as

$CI=\frac{a}{b}\times 100.$

In the case of an ellipse, there is a 1:1 correspondence between the two
measures, where

$e=\sqrt{1-\left(\frac{CI}{100}\right)^{2}}.$

In the 17-22 week range of gestational age, the mean cephalic index has been
reported as 75.9, with a standard deviation of 3.7 [19]. Therefore, an
anatomically plausible range of cephalic indices is $65-90$, which corresponds
to an eccentricity range of $0.43-0.76$. Any ellipse with an eccentricity
outside of this range is likely to be a failure mode of the biometric CNN.

In the final product using this CNN, we relaxed this constraint further to an
eccentricity range of $0.25-0.8$. This was to restrict rejections to clear
failures, allowing a large margin for biological variation.

The fetal abdomen has somewhat different morphology. It is typically round,
with a very low eccentricity. We constrained it to have $e<0.6$ for the
purposes of our software - any abdominal ellipse with a higher eccentricity
was considered biologically implausible and rejected.

Figure 16 shows two examples of drawn ellipses which violate this eccentricity
constraint. Both are clear failures, where the ellipse does not follow
anatomical boundaries. The addition of our constraint rejects the measurements
from these ellipses as noise, even if the circumferences appear biologically
plausible.

###  B-D Plane classification confidence

Standard plane | Frames >95% confidence (%)
---|---
Brain-TV | 83.5%
Brain-CB | 82.1%
Abdominal | 73.2%
Femur | 66.4%
Table VIII: Proportion of frames in the four biometric standard planes which
are classified with >95% softmax confidence.

Section II-E describes one filtering method: only frames identified with
$>95\%$ confidence as a standard plane with a FASP biometric are processed by
biometric networks. This somewhat reduces the number of frames passed to
biometric networks and therefore the number of samples of each biometric.

Table VIII shows the proportion of frames classified as belonging to each of
the four biometric standard planes with sufficient confidence for a
measurement. In all cases, only a minority of frames are discarded.


Yooyoung Lee1, Craig Greenberg1, Eliot Godard1,∗, Asad A. Butt1,∗, Elliot
Singer2, Trang Nguyen2, Lisa Mason3, Douglas Reynolds3 ††thanks: ∗NIST
Associates

# The 2022 NIST Language Recognition Evaluation

###### Abstract

In 2022, the U.S. National Institute of Standards and Technology (NIST)
conducted the latest Language Recognition Evaluation (LRE) in an ongoing
series administered by NIST since 1996 to foster research in language
recognition and to measure state-of-the-art technology. Similar to previous
LREs, LRE22 focused on conversational telephone speech (CTS) and broadcast
narrowband speech (BNBS) data. LRE22 also introduced new evaluation features,
such as an emphasis on African languages, including low resource languages,
and a test set consisting of segments containing between 3s and 35s of speech
randomly sampled and extracted from longer recordings. A total of 21 research
organizations, forming 16 teams, participated in this 3-month long evaluation
and made a total of 65 valid system submissions to be evaluated. This paper
presents an overview of LRE22 and an analysis of system performance over
different evaluation conditions. The evaluation results suggest that Oromo and
Tigrinya are easier to detect while Xhosa and Zulu are more challenging. A
greater confusability is seen for some language pairs. When speech duration
increased, system performance significantly increased up to a certain
duration, and then a diminishing return on system performance is observed
afterward.

Index Terms: human language technology, LRE, language recognition, language
detection, speech technology performance evaluation

##  1 Introduction

The 2022 NIST Language Recognition Evaluation (LRE), held in fall of 2022, was
the latest in an ongoing series of language recognition evaluations conducted
by NIST since 1996 [1]. The primary objectives of the LRE series are to: 1)
advance language recognition technologies with innovative ideas, 2) facilitate
the development of language recognition technology by providing data and
research direction, and 3) measure the performance of the current state-of-
the-art technology. Figure 1 shows the number of target languages and
participants (based on sites) for all NIST LREs.

LRE22 was conducted entirely online using a web-based platform like LRE15 [2]
and LRE17 [3, 4]. The updated LRE22 web-platform111https://lre.nist.gov
supported a variety of evaluation activities, such as registration, data
license submission, data distribution, system output submission and
validation/scoring, and system description/presentation uploads. A total of 16
teams from 21 organizations in 13 different countries made submissions for
LRE22. Figure 2 displays a world map with heatmap representing the number of
participating sites per country. Since two teams did not submit valid system
descriptions, analysis considering only 14 teams in presented this paper. It
should be noted that all participant information, including country, was self-
reported.

Figure 1: Language and participant count for the NIST LREs Figure 2: Heatmap
of the world showing the number of LRE22 participating sites per country.

##  2 Task

The general task in the NIST LREs is language detection, i.e. to automatically
determine whether a particular target language was spoken in a given test
segment of speech. Since LRE11 [5], the focus of the language detection task
had turned to distinguishing between closely related, and sometimes mutually
intelligible, languages. However LRE22 introduced a new emphasis on
distinguishing between African languages, including low resource languages.
Table 1 shows the 14 target languages included in LRE22. Similar to LRE17,
LRE22 participants were required to provide a 14-dimensional vector of log-
likelihood scores corresponding to the languages in Table 1. Unlike LRE17,
language clusters were not considered in this evaluation; a language cluster
is a group of two or more consonant sounds with those from the same speech
community [6].

Like LRE17, there were two training conditions in LRE22: fixed and open. For
the fixed training condition, participants were restricted to use only a
limited pre-specified set of data for system training and target model
development. For the open training condition, participants were allowed to
utilize unlimited amounts of publicly available and/or proprietary data for
their system training and target model development. To facilitate more
meaningful cross-system comparisons, LRE22 participants were required to
provide submissions to the fixed condition while participation in the optional
open condition was strongly encouraged to understand the impacts that larger
amounts of training and development data have on system performance. In order
to encourage participation in the open training condition, the deadline for
this condition was made one week later than the required fixed training
condition submission deadline. A total of 65 valid submissions were received,
40 for the fixed training condition and 25 for the open condition. LRE
participants were required to specify one submission as primary for each
training condition they took part in, while all other systems submitted were
considered alternate.

Table 1: LRE22 target languages Language |  Code |  Language |  Code
---|---|---|---
Afrikaans |  afr-afr |  Ndebele |  nbl-nbl
Tunisian Arabic |  ara-aeb |  Oromo |  orm-orm
Algerian Arabic |  ara-arq |  Tigrinya |  tir-tir
Libyan Arabic |  ara-ayl |  Tsonga |  tso-tso
South African English |  eng-ens |  Venda |  ven-ven
Indian-accent South African English |  eng-iaf |  Xhosa |  xho-xho
North African French |  fra-ntf |  Zulu |  zul-zul

##  3 Data

This section provides a brief description of data used in LRE22 for training,
development (dev), and evaluation (test) sets, along with the associated
metadata.

###  3.1 Training set

As mentioned in Section 2, there were two training conditions in LRE22. The
fixed condition limited the system training and development data to the
following specific data sets provided to participants by the Linguistic Data
Consortium (LDC): 2017 NIST LRE dev set and previous NIST LRE training data
(LDC2022E16), 2017 NIST LRE test set (LDC2022E17), 2022 NIST LRE dev set
(LDC2022E14). The VoxLingua107 data set [7] was also permitted for use in the
fixed condition. The open training condition removed the limitations of the
fixed condition. In addition to the data listed in the fixed condition,
participants could use any additional data to train and develop their system,
including proprietary data and data that are not publicly available. LDC also
made selected data from the IARPA Babel Program [8] available to participants
to be used in the open training condition.

###  3.2 Development and test sets

The development (dev) set is normally used to build/optimize a system model
during the development process while the evaluation (test) set is used to
evaluate the performance of the system model. The speech segments in the LRE22
dev and test sets were selected from data sets collected by the Linguistic
Data Consortium (LDC) to support LR technology evaluations; namely the
Maghrebi Language identification (MAGLIC), Speech Archive of South African
Languages (SASAL), and Low Resource African Languages (LRAL) corpora. The
MAGLIC corpus was a CTS-only collection based in Tunisa and includes four
regional language varieties spoken in North Africa: Algerian Arabic, Libyan
Arabic, Tunisian Arabic, and North African French. The SASAL corpus was a CTS
and BNBS collection located in South Africa and contains several African
language varieties, a subset of which were included in LRE22: Afrikaans,
Ndebele, Tsonga, Venda, Xhosa, and Zulu, as well as South African English and
Indian-accented South African English. The LRAL corpus was a BNBS collection
based in Ethiopia, and, of the languages in LRAL, two were selected for
inclusion in LRE22: Oromo and Tigrinya.

Figure 3: Distribution of speech segments per target language for both dev and
test sets

All audio data provided was sampled at 8 kHz, a-law encoded, and formatted as
SPHERE [9] files. When the source audio recordings were higher bandwidth or
encoded differently, they were downsampled and transcoded to 8-kHz a-law.
Unlike in previous LREs, the amount of speech in the LRE22 segments was
uniformly sampled between approximately 3 and 35 seconds, as determined by an
automatic speech activity detector. Figure 3 shows a stacked histogram for the
dev and test sets. The dev set consisted of 300 segments per target language
while the test set contained a total of 26,473 segments ranging from 383 to
2,769 segments across the target languages.

Figure 4: System performance (actual and minimum costs) on primary submissions
under the fixed training condition Figure 5: A performance comparison of the
fixed and open training conditions

###  3.3 Metadata

The metadata collected by LDC can be categorized into audio- and audit-related
metadata. The audio metadata indicates information related to the audio
recording or segment, such as speech duration, data source type (i.e., either
CTS or BNBS), and source file (i.e., the original recording from which the
audio segment was extracted). The audit metadata reflects a human auditor’s
judgement of the speech, having listened to an audio recording, such as
whether the recording contained a single speaker, if the person speaking was a
native speaker, the speech clarity, the speaker sex, or if the recording took
place in a noisy environment. In this paper, we limit our analyses on data
source type and speech duration.

##  4 Performance Measure

As stated in the Section 2, LRE22 participants were required to provide a
14-dimensional vector of log-likelihood scores for the 14 target languages
(see Table 1 for the LRE22 target languages). Unlike LRE17, language clusters
were not considered in this evaluation. Pair-wise performance was computed for
all target/non-target language pairs. A decision threshold derived from log-
likelihood ratios was used to determine the number of missed detections and
false alarms, computed separately for each target language. The missed
detections (Misses) indicate the segments that are the target language, but
are not predicted to be, while the false alarms (FAs) indicate the segments
that are falsely identified as the target language. The probabilities of
missed detections ($P_{Miss}$) and false alarms ($P_{FA}$) are then combined
using a linear cost function [10]:

$\displaystyle C(L_{T},L_{N})=$ $\displaystyle C_{Miss}\times P_{Target}\times
P_{Miss}(L_{T})+$ $\displaystyle C_{FA}\times(1-P_{Target})\times
P_{FA}(L_{T},L_{N})$ (1)

where $L_{T}$ and $L_{N}$ are target and non-target languages, respectively.
Here, $C_{Miss}$ (cost of a missed detection), $C_{FA}$ (cost of a false
alarm), and $P_{Target}$ (the a priori probability of the specified target
language) are application-motivated cost model parameters. Two sets of cost-
function parameters were used in LRE22: the first set of parameters provides
equal weighting to the costs of errors ($C_{Miss}=C_{FA}=1$) and a target
probability of 0.5, while the second set of parameters changed the target
probability to 0.1. The final metric, $C_{Primary}$, consisted of the mean
value of the costs using the two different cost function parameters,
normalized by dividing by the cost of a ``no information'' system. Costs using
thresholds that minimize the Bayes risk, $actC_{Primary}$, as well as using
thresholds that minimize the empirical cost, $minC_{Primary}$, were computed.
We refer readers to the LRE22 evaluation plan [10] for details of the
performance measures.

##  5 Results and Analyses

A total of 14 teams from academic and industrial sectors successfully
completed LRE22. For both the fixed and open training conditions, the teams
were allowed to have one primary submission and one or more alternate
submissions. In this section, we present a summary of results and key findings
on the primary submissions using the performance metrics defined in Section 4.

Figure 4 illustrates system performance for all the primary submissions under
the fixed training condition. The x-axis are anonymized team names and the
y-axis are $C_{Primary}$ values for both the actual and minimum costs (N.B., a
lower $C_{Primary}$ value indicates better performance). The orange dashed-
line indicates an actual cost, $actC_{Primary}$, and the blue is a minimum
cost, $minC_{Primary}$, for a reference system; we used an off-the-shelf
algorithm as a reference to validate the LRE22 data construction and
evaluation process. The reference system was trained and fine-tuned only on
VoxLingua107 and the LRE22 development set. The shaded color on each team’s
bar indicates the difference between $actC_{Primary}$ and $minC_{Primary}$,
which indicicates a calibration error. In Figure 4, we observe that, given the
primary submissions under the fixed condition, the $C_{Primary}$ values range
from 0.11 to 0.73 across all the teams. It is observed that the top-performing
systems (e.g., T1-T4) have small calibration errors (i.e., the absolute
difference between the actual and minimum costs is relatively small) while a
few teams (e.g., T5, T7, T11 and T12) are less well-calibrated.

Figure 6: A language-level performance on primary submissions under the fixed
training condition

As described in Section 2, the fixed training condition is required while open
is optional; 7 out of the 14 teams submitted their system outputs to the open
training condition. Figure 5 illustrates a performance comparison of training
conditions (fixed vs open) for the seven teams only (ordered by open system
performance). The result shows that system performance from the open condition
generally outperforms the fixed condition submission across the teams (except
T9), and a calibration error is observed in team T7 under the open training
condition.

To understand variability of language-level system performance and language
detection difficulty, Figure 6 illustrates a box plot of the primary
submission performance under the fixed training condition. The x-axis is a
team name (ordered by median), the y-axis is the actual cost
($actC_{Primary}$), and each point represents a target language. The black
line within a box is the median, the box edges represent the lower quartile
and upper quartile, and the whiskers extending from the box indicate
variability outside the upper and lower quartiles. We observe a high
dispersion of language performance for a few teams such as T4, T5, and T9.
Overall, the Oromo (orm-orm) and Tigrinya (tir-tir) points marked in blue are
located in the bottom side of Figure 6 (easier to detect) while Xhosa (xho-
xho) and Zulu (zul-zul) are in the top (harder to detect); a similar trend is
observed across the teams.

Figure 7: Language confusability of the leading systems Figure 8: A data
source type distribution and effect on systems

(a)

(b)

Figure 9: SAD duration effect on system performance (a) SAD duration
distribution for the dev and test set (b) T1 system performance vs. SAD
duration.

To examine language-pair confusability, we conducted data analysis using
heatmap confusion matrices as shown in Figure 7. The axes are language codes.
The diagonal values from upper-left to bottom-right are $P_{Miss}$ (false
reject rates) and the off-diagonal values are $P_{FA}$ (false alarm rates). A
higher false alarm probability implies a potential confusability for that
language pair. For simplicity, results of $P_{Target}=0.5$ for the four
leading systems are demonstrated using heatmap confusion matrices. Given the
test set and systems, a higher confusability is observed for three clusters of
language pairs as follows: 1) among Arabic languages (ara-aeb, ara-arq, ara-
ayl), 2) between South African English (eng-ens) and Indian-accented South
African English (eng-iaf), and 3) Ndebele (nbl-nbl), Tsonga (tso-tso), Venda
(ven-ven), Xhosa (xho-xho) and Zulu (zul-zul).

To gain insight on how metadata variables (i.e., factors) affect system
performance, we conducted experiments given the metadata listed in Section
3.3. For simplicity, the following analyses are demonstrated using data source
type and speech duration only. The LRE22 data was collected in two primary
genres, namely, conversational telephone speech (CTS) and broadcast narrowband
speech (BNBS) which we call data source type. Figure 8 shows system
performance ($actC_{Primary}$) partitioned by data source type (CTS vs BNBS)
for all the primary submissions under the fixed training condition. The top-
left pie chart is a distribution of CTS and BNBS on the test set, which is
imbalanced. The bar plot shows a performance comparison between CTS (blue) and
BNBS (orange) across all the teams. The results indicates that, given the
imbalanced distribution, CTS is more challenging and that data source type has
a strong effect on system performance; a similar trend is observed across the
systems.

Durations of test set segments varied between 3s and 35s of speech that have
been randomly sampled and extracted from longer recordings as determined by an
automatic Speech Activity Detector (SAD) which we call SAD duration. Figure 9a
shows a distribution of SAD duration for the test set and Figures 9b shows the
performance of a top-performing system by SAD duration. Given the test set and
systems, it is seen that when SAD duration increases, $actC_{Primary}$
significantly decreases up to a certain duration (between 15s and 20s). After
that, a diminishing return on system performance improvement is observed
across the systems.

##  6 Conclusions

We presented a summary of the 2022 NIST Language Recognition Evaluation with
an emphasis on low resource languages and random duration of speech segments.

The results showed that almost no calibration error was observed for the top-
performing systems for both the fixed and open training condition. Overall,
the submissions under the open training condition had better performance
compared to the fixed condition submissions, with only one exception. Given
the test set and primary systems under the fixed training condition, we found
that Oromo and Tigrinya were easier to detect while Xhosa and Zulu were harder
to detect. A greater confusability was observed for the language pairs 1)
among Zulu, Xhosa, Ndebele, Tsonga, and Venda, 2) between South African and
Indian-accent South African English, and 3) among Tunisian, Algerian, and
Libyan Arabic languages. Some of the metadata, such as data source type and
SAD duration, had a significant effect on system performance for all systems.
In terms of SAD duration, when speech duration increased, system performance
significantly increased up to a certain duration, and then we observed a
diminishing return on system performance afterward.

##  7 Disclaimer

These results presented in this paper are not to be construed or represented
as endorsements of any participant's system, methods, or commercial product,
or as official findings on the part of NIST or the U.S. Government.

The work of MIT Lincoln Laboratory (MITLL) is sponsored by the Department of
Defense under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings,
conclusions or recommendations expressed in this material are those of the
author(s) and do not necessarily reflect the views of the U.S. Air Force.

## References

  * [1] NIST, ``NIST language recognition evaluation overview,'' 1996-2022, [Online; accessed 17-February-2023].
  * [2] H. Zhao, D. Bansé, G. Doddington, C. Greenberg, J. Hernández-Cordero, J. Howard, L. Mason, A. Martin, D. Reynolds, E. Singer, and A. Tong, ``Results of the 2015 NIST language recognition evaluation,'' in _Interspeech 2016_ , San Francisco, USA, September 2016, pp. 3206–3210.
  * [3] S. O. Sadjadi, T. Kheyrkhah, A. Tong, C. S. Greenberg, D. A. Reynolds, E. Singer, L. P. Mason, and Hernandez-Cordero, ``The 2017 nist language recognition evaluation.'' in _Odyssey_ , 2018, pp. 82–89.
  * [4] S. O. Sadjadi, T. Kheyrkhah, C. Greenberg, E. Singer, D. Reynolds, L. Mason, and J. Hernandez-Cordero, ``Performance Analysis of the 2017 NIST Language Recognition Evaluation,'' in _Proc. Interspeech 2018_ , 2018, pp. 1798–1802.
  * [5] A. F. Martin, C. S. Greenberg, J. M. Howard, G. R. Doddington, and J. J. Godfrey, ``NIST language recognition evaluation - past and future,'' in _Odyssey 2014_ , Joensuu, Finland, June 2014, pp. 145–151.
  * [6] F. Ahmad and G. Widén, ``Language clustering and knowledge sharing in multilingual organizations: A social perspective on language,'' _Journal of Information Science_ , vol. 41, no. 4, pp. 430–443, 2015.
  * [7] J. Valk and T. Alumäe, ``Voxlingua107: A dataset for spoken language recognition,'' in _2021 IEEE Spoken Language Technology Workshop (SLT)_ , 2021, pp. 652–658.
  * [8] M. P. Harper, ``Data resources to support the Babel program,'' https://goo.gl/9aq958, [Online; accessed 17-February-2023].
  * [9] NIST, ``Speech file manipulation software (SPHERE) package version 2.7,'' ftp://jaguar.ncsl.nist.gov/pub/sphere-2.7-20120312-1513.tar.bz2, 2012, [Online; accessed 01-March-2018].
  * [10] NIST, ``The 2022 NIST Language Recognition Evaluation Plan,'' https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=935161, 2022, [Online; accessed 01-Feb-2023].