# End to End Lip Synchronization with a Temporal AutoEncoder

Yoav Shalev Lior Wolf
School of Computer Science
Tel Aviv University

###### Abstract

We study the problem of syncing the lip movement in a video with the audio
stream. Our solution finds an optimal alignment using a dual-domain recurrent
neural network that is trained on synthetic data we generate by dropping and
duplicating video frames. Once the alignment is found, we modify the video in
order to sync the two sources. Our method is shown to greatly outperform the
literature methods on a variety of existing and new benchmarks. As an
application, we demonstrate our ability to robustly align text-to-speech
generated audio with an existing video stream. Our code and samples are
available at https://github.com/itsyoavshalev/End-to-End-Lip-Synchronization-
with-a-Temporal-AutoEncoder.

##  1 Introduction

With the advancement of video technology, one may assume that lip syncing
would become a thing of the past. However, with the ongoing shift form
captured to generated media, coupled with a surge in user generated content
apps, the need for effective methods is rapidly growing.

Consider, for example, the scenario of a generated video. It could involve the
head of one actor, extracted from an old footage, the lips of another actor,
added to match the script, and the voice of a Text to Speech (TTS) robot.
Syncing the different sources, and especially the lip motion to the audio, to
which viewers are very sensitive, poses a challenge.

As another example, consider the trending lip syncing apps. Users try their
best to align their lips with a song they choose. In many cases, this
alignment is only partly successful and the acquisition process needs to
repeat itself.

Despite the well-defined task and the clear application need, the literature
is relatively limited. Many of the existing solutions perform a global shift,
which cannot address many of the modern use cases. Other methods rely on
unrealistic assumptions and are only partly trained. In addition, while
modifying the video without causing artifacts is much easier than modifying
the audio, existing methods choose the latter, creating metallic voices and
other artifacts.

In this work, we employ an end to end temporal autoencoder to perform the
alignment. The method projects simple video and audio features to a joint
space, where their mutual distances are considered. An LSTM autoencoder, with
an added attention layer is then used to predict the correct alignment for
temporal windows of up to one second. A straightforward matching method is
then used to align sequences of unlimited length.

In an extensive series of experiments, we compare our method to the literature
algorithms and show a sizable improvement in accuracy. Our method is robust
enough to work in real-world scenarios and the vast majority of users cannot
tell the difference between a video that was modified and an untouched video,
even when matching to the audio track of a different speaker.

##  2 Related Work

Dynamic time warping (DTW) [1], [17] uses a dynamic programming to align
multiple time-series. DTW measures the similarity and finds an optimal match
by inserting frames. Anguera et al. [3] use a predefined similarity between
graphemes and phonemes. Tapaswi et al. [30] introduce a similarity between
visual scenes and sentences based on appearance of same characters to align TV
shows and plot synopses. Other dynamic programming approaches have also been
used for multi modal alignment of text to speech [14] and video [34]. DTW was
extended using canonical correlation analysis (CCA) for learning a mapping
[26], [32], [33]. CCA based DTW models aren’t able to model nonlinear
relationships. This has been addressed by Deep Canonical Time Warping (DCTW)
[29], which learns non-linear, correlated and temporally aligned
representations of multiple time-series.

The earliest methods in the field of lip motion synchronization, employ
phoneme recognition for the audio stream [19, 22] and align them with the
mouth positions [19] or with visemes [22], which are face templates that
correspond to a specific configuration of the face during speech production.
Zoric and Igor classified the MFFC representation of the audio signal into
visemes and then aligned them with a parametric face model [35].

Other methods perform the alignment without employing visemes or phonemes.
Argones et al. used co-inertia analysis and a coupled hidden Markov model [4].
Sargin et al. used canonical correlation analysis for speech and lip texture
features [24].

Within the domain of deep learning methods, [21] used a network that operates
on predefined audio and visual features, in contrast to our work where we
predict a shift for each frame and learn the visual features directly. Chung
et al. used a CNN model called SyncNet to learn visual features directly from
the video stream, and audio features from the MFCC representation of the audio
[9]. They then find the optimal global shift by computing the contrastive loss
over all frames. In contrast, we employ an alignment network and are not
limited to a global shift.

In contrast to the metric learning approach of SyncNet, Chung et al. learned
audio-visual embeddings as a multiclass classification task [10]. Their method
classifies one visual frame into one multiple possibly matching audio frames
obtained at $N$ different locations. The decisions are integrated in order to
obtain a global shift.

Realizing that in many cases a single global shift is suboptimal, Halperin et
al. have shown that a network that is similar in structure to SyncNet can be
leveraged to achieve a dynamic temporal alignment [12]. This is done by using
the Dijkstra algorithm as a post processing step, in order to find the optimal
path in the graph of pairwise distances.

A related task is that of reconstructing a 3D mesh animation, or generating a
video of a talking face, which is aligned to an input audio signal. Karras et
al. predict 3D vertex coordinates of a target face model, given a waveform of
any source speaker, and aim to control the emotional states of the generated
face [15]. ObamaNet [18] uses a recurrent neural network (RNN) in order to
predict a PCA representation of normalized mouth key-points, given a raw
waveform created by a text to speech module. The key-points are then used to
condition a generator that synthesizes a video of a target identity.
Suwajanakorn et al. trained an RNN on many hours of Obama’s weekly address
footage to map from the MFCC representation of the input audio stream into
mouth shapes [28]. Then, they synthesized a talking face video of Obama. Son
Chung et al. proposed an encoder-decoder model that learns the joint
embeddings of a given audio and a target face images, and then generates a
lip-synced talking face video [27].

##  3 Proposed Method

An overview of our method is shown in Fig. 1. The inputs to our model are
unsynchronized audio and visual streams. The visual stream and the audio
stream are both encoded to represent the local neighborhoods as tensors. A
temporal encoding is obtained by considering pairwise distances between the
video and the audio tensors. The temporal encoding then goes through an
attention modulated LSTM decoder. At every time step, the decoder attends the
encoder’s outputs and predicts an index of the input’s video stream.

Figure 1: Overview of our method. For every audio feature a vector of
Euclidean distances is computed with all video features. The vectors are fed
into an LSTM encoder followed by LSTM decoder, in order to predict a sequence
of indices from the visual stream.

###  3.1 Inputs

As a preprocessing step for the video stream, facial key-points and rotations
are extracted using Openface [6, 31]. We then extract the mouth area, align it
to the vertical axis, and normalize its size to $120\times 120$ pixels. Each
video input is a temporal stack of five consecutive video frames, and the
stride of consecutive video inputs is one frame, see Fig. 2(a).

The alignment network has a temporal receptive field of three seconds at 25
FPS, i.e., 75 frames. Each frame is represented by a tensor of size $5\times
120\times 120$, where the five channels correspond to the five consecutive
frames above. This redundant representation, in which the same cropped mouth
frames appear in multiple tensors, follows the early fusion model suggested by
Chung and Zisserman [8].

|
---|---
(a) | (b)
Figure 2: Input preprocessing. (a) Each video input is a temporal stack of
five consecutive video frames of the mouth area. (b) Each audio input consists
of 20 time steps of 13 Mel-frequency bands MFCC.

The input audio is sampled at 16kHz. Each single audio input (“audio frame”)
consists of 20 time steps of MFCC features which are corresponding to a 0.2
second input signal. Specifically, 13 Mel-frequency bands are used for the
MFFC features, with a window length of 25 ms and a window step of 10ms. The
stride between consecutive audio inputs is 40ms, which is the same as one
video frame at 25 FPS. An example of a single audio input frame is shown in
Fig. 2(b).

The network has a receptive field of one second, i.e., 25 audio frames, where
each frame is represented by a matrix of size $13\times 20$, corresponding to
the 20 time steps of MFCC features detailed above.

Note that in our method, video and audio are not treated in a symmetric way,
and we match a second of audio to three seconds of video. The underlying
reason is that we want to allow for each audio frame a search window of 25
video frames on both sides.

###  3.2 Method and Architecture

For given audio and visual tensors of lengths $n=25$ and $m=75$, respectively,
each audio and visual input is encoded using the audio or visual encoder into
feature vectors of size 1024:

$\begin{split}\psi_{a_{i}}=E^{a}(a_{i})&\quad i\in\\{1,...,n\\}\\\
\psi_{v_{j}}=E^{v}(v_{j})&\quad j\in\\{1,...,m\\}\end{split}$ (1)

Where $\psi_{a_{i}}$ ($\psi_{v_{j}}$) are the embeddings of audio (video)
input at time step i (j). In order to emphasize our method’s advantage over
the literature, we employ the same architecture for the audio and video
encoders ($E^{a}$ and $E^{v}$) as the one used in SyncNet [9]. This
architecture consists of 3D convolutions (operating also on the temporal
domain) with max-pooling layers, ending with fully connected layers. Batch
normalization layers are added after each convolutional layer.

Next, for each $\psi_{a_{i}}$ we calculate the Euclidean distance with all
$\psi_{v_{j}}$ and obtain the Euclidean distances features vectors of length
$m$:

$\begin{split}\vec{\rho_{i}}=[d_{i,1},...,d_{i,m}]\\\ \end{split}$ (2)

where $d_{i,j}=\|\psi_{a_{i}}-\psi_{v_{j}}\|$.

The sequence of vectors $\rho=[\vec{\rho_{1}}...\vec{\rho_{n}}]$ is then
encoded using a recurrent neural encoder that produces outputs $o^{e}$ and
hidden states vectors $h^{e}$:

$\begin{split}o^{e}_{i},h^{e}_{i}=\operatorname{RNN_{e}}\left(\vec{\rho_{i}},h^{e}_{i-1}\right)\\\
\end{split}$ (3)

and then decoded using a recurrent neural decoder followed by a multi-layer
perceptron (MLP). As suggested by Bahdanau et al. [5], in order to allow the
decoder to attend the entire encoded sequence at every output step, we are
using an attention mechanism. At every output step $k$, the decoder produces
state vectors $h^{d}_{k}$ and output vectors $o^{d}_{k}$, based on the
previous step context vector $c_{k-1}$, the decoder state vector $h^{d}_{k-1}$
and the predicted visual frame index $y_{k-1}$. The latter is obtained by an
MLP that is applied to the output vector $o^{d}_{k-1}$ and the context vector
$c_{k-1}$.

Written explicitly, the RNN decoder has the following input-output structure:

$\begin{split}o_{k}^{d},h_{k}^{d}=\operatorname{RNN_{d}}\left(h_{k-1}^{d},y_{k-1},c_{k-1}\right),\end{split}$
(4)

where, the next step context vector $c_{k}$ is computed using the attention
model:

$\displaystyle e_{k,i}=U^{T}\tanh\left(Wh_{k}^{d}+Vo_{i}^{e}+b\right)$ (5)
$\displaystyle\alpha_{k,i}=\operatorname{Softmax}\left(e_{k,\cdot}\right)=\frac{\exp\left(e_{k,i}\right)}{\sum_{i^{\prime}=1}^{n}\exp\left(e_{k,i^{\prime}}\right)}$
(6) $\displaystyle c_{k}=\sum_{i}\alpha_{k,i}{o}^{e}_{i}$ (7)

where $U$, $W$, $V$ and $b$ are learned weights, and $\alpha_{k,i}$ are the
attention weights, which are obtained by a softmax operator applied to the
pre-weights $e_{k,i}$.

The context vector $c_{k}$ is concatenated with the output vector $o^{d}_{k}$
and fed into the MLP, which outputs a vector of size $1\times m$. Then the
predicted index of the visual frame that matches audio frame $k$ is given by
applying a softmax layer:

$\displaystyle
p=P\left(y_{k}|v,a,y_{k-1}\right)=\operatorname{Softmax}\left(\operatorname{MLP}\left(o_{k}^{d},c_{k}\right)\right)$
(8) $\displaystyle y_{k}=\operatorname{argmax}_{i}p_{i}$ (9)

At $k=1$, the context vector $c_{0}$ is set to $0$, the input $y_{0}$ is set
to a special symbol that indicates a start of a sequence, and the decoder
state $h^{d}_{0}$ is set to the last encoder’s hidden state $h^{e}_{n}$.

The architecture of the RNN encoder and decoder consists of three LSTM layers
with hidden sizes of 512. The MLP consists of two linear layers with 512 and
256 hidden units respectively, followed by a ReLU activation and another
linear layer that outputs a vector of size 75.

###  3.3 Loss Function

By predicting sequences of indices of the visual stream, we are treating the
problem as a multiclass classification problem. Therefore we use the cross
entropy loss:

$\mathcal{L}(p,t)=-\log\left(\frac{\exp(p_{t})}{\sum_{j}\exp(p_{j})}\right),$
(10)

where $p$ is the vector of assignment probabilities, given in Eq. 8 and $t$ is
the ground truth index of the video frame that matches the current audio
frame. This loss is aggregated across all audio frames in all sequences of the
training set.

###  3.4 Training Details

In order to train the model for predicting the local frames shifts, we created
a synthetic training dataset out of the LRS2-BBC dataset [2] as described in
Sec. 5.

We employ a two phase training. In the first, we employ a version of the
LRS2-BBC in which only a single shift is performed, and pretrain for 20 epochs
the input encoders $E^{a}$ and $E^{v}$. The full model, including all
networks, is then trained for 30 epochs, with a batch size of four, using the
full-blown synthetic data set.

The LSTMs are initialized using semi-orthogonal matrices as suggested by Saxe
et al. [25], the convolutional and linear layers were initialized using Xavier
Normal [11] and the batch normalization layers were initialized with a normal
distribution ($1\pm 0.02$). We use an Adam optimizer [16] with a learning rate
of $10^{-6}$ and beta coefficients of 0.5 and 0.999. At train time, we apply
dropout with 0.1 probability on the LSTM decoder’s previous-step output-
embeddings.

In order to improve performance and reduce overfitting, we applied data
augmentations for the visual input at train time. We multiplied the tensor of
the visual information by a random value and randomly flipped, scaled, rotated
and translated all images in the tensor.

##  4 Finding Optimal Alignment at Inference

We assume that at the starting point of the sequence, the optimal alignment
distance of the video and the audio streams is up to two seconds. If the shift
is larger, a global alignment step is first required.

The learned method is applied to a second of audio and three seconds of video.
At inference time we are required to align, in a coherent manner, long
sequences of visual frames. In addition, a direct inference may be suboptimal,
even for short sequence. Recall that at each time step, the decoder outputs a
probability for each of multiple indices of the video stream. A greedy
approach would predict for each audio frame the most probable visual frame
index. This index, due to the decoder’s structure, is conditioned on the past
frames, but not on the future frames.

In order to overcome these challenges, we perform the following steps. First,
we apply the method multiple times with a stride of ten frames. At each
application, we employ a three second temporal window in the video domain that
is centered around the one second window in the audio domain. As a result, for
each audio frame we obtain up to three votes for a prediction of a video frame
index.

We treat these votes as equal and consider for each audio frame, as possible
matching candidates, only the set of indices with the maximal number of votes.
For example, an audio frame can be associated twice with index $i_{1}$ and
once with index $i_{2}$. In this case, we only consider the index $i_{1}$ as a
potential match. In other cases, three different indices may be potential
matches.

We then perform dynamic programming in order to find the longest sequence of
matches with monotonically increasing indices. In other words, the search
assigns matches to one audio frame at a time, going sequentially over all
audio frames. It only considers future matches that do not go back (multiple
votes to the same video frame are possible), and stops the sequence, if no
such further matches are possible. The length of such monotonic sequences is
optimized.

At the end of the obtained sequence, there is at least one frame for which a
match cannot be found without going back. We skip such frames and continue
with a longest sequence search when an audio frame has a possible match that
does not break monotonicity.

In order to smooth the matching results and fill-in the missing matches, we
apply an adaptive smoothing procedure as suggested by Halperin et al. [12].
The procedure employs a Laplacian filter with a parameter $\sigma$ and a
smoothness criterion. It searches for the lowest $\sigma$ such that the
criterion is met. This is done with one such parameter for the entire video
sequence. Predictions that are shorter than the audio stream are padded with
the last visual predicted frame.

##  5 Datasets

Our network is trained on a dataset that is derived from the LRS2-BBC dataset
[2]. Another split of this dataset is used for evaluation. We also perform
evaluation by switching the audio between two TCD-TIMIT [13] speakers uttering
the same test.

###  5.1 LRS2-BBC

LRS2-BBC consists of thousands of spoken sentences from BBC television. Videos
are of frame size of $160\times 160$, with a frame rate and audio rate of 25
FPS and 16kHz respectively. For training, we use the pre-train set, which
consists of 96,318 utterances. The test set, which consists of 1,243
utterances, is used for evaluation in our experiments.

The dataset itself is well-synchronized and we adapt it to our purposes by
augmenting the data. In the video stream, frames are randomly dropped and
duplicated, while maintaining a maximal distance of 25 frames from the frame’s
original position, which bounds the number of drops and duplications. We also
restrict the augmetnation such that the maximal number of occurrences for a
single frame is four.

The target prediction is set such that if a visual frame is dropped, the
corresponding audio frame is mapped to the closest remaining frame, and if a
visual frame is duplicated, the corresponding audio frame is mapped to the
last occurrence of that frame. The audio stream is also augmented, by adding a
global random shift inside the three second visual stream window. The size of
the shift is [-25…25] frames.

An illustration of the process is shown in Fig. 3.

Figure 3: The creation process of a training audio-video pair, along with its
ground trough labels. For a given audio-video pair (1, 2), a sequence of
random actions is taken (3) on the visual stream, to create an unsynchronized
video stream (4) along with the ground truth labels (5). The resulted training
pair consists of the input audio (1) and the modified video stream (4) and the
task is to predict the ground truth labels (5). Note that the frames at the
end of the modified video stream are removed since they fall after the audio
boundaries.

###  5.2 TCD-TIMIT

The TCD-TIMIT dataset, developed for speaker and speech recognition purposes,
is used for evaluation only. The dataset consists of $1,920\times 1,080$
videos of 62 speakers, both males and females, saying 6,913 sentences. Videos
are down-sampled by us into a frame rate and audio rate of 25 FPS and 16kHz
respectively.

The sentences vary between speakers. However, at the start of the session, all
speakers say the same two sentences – sa1 and sa2. For evaluation, we took
sentences sa1 and sa2 of the first five speakers which are three males and two
females. We then consider the various combinations of video from one speaker
and speech audio from another.

##  6 Experiments

We compare the performance of the proposed method to existing methods for
visual speech synchronization. We evaluate both in the case in which
synchronization is obtained by applying a single shift for the entire video,
and in the case where the shift is different for each frame. For the local
shift case, we perform experiments where either the audio or the video are
modified. Out of these options, we strongly advocate for a local form of
synchronization and for modifying the video. Local alignment allows for a
greater flexibility and generalizes the global shift. Modifying the video
results in changes that are far less noticeable, while modifying the audio
often results in metallic sounds and other artifacts, which are not easy to
overcome. The reason that we evaluate our method with the other options is to
match the literature methods, in the same domain in which these methods
operate.

A summary of the literature baselines is given in Tab. 1. Since no other
method in the literature, as far as we know, performs local shifts and warps
the video, we created a suitable baseline method by modifying the Dynamic
Temporal Alignment (DTA) method of Halperin et el. [12]. The original method
is local and warps the audio. We have implemented their method for video
augmentation by inverting the search of the Dijkstra matching to be from audio
to video.

Method |  Global/local alignment |  Augments video/audio
---|---|---
Ours |  both |  both
SyncNet |  global |  same
DTA |  both |  audio
Our modified DTA |  both |  video
Chung et al.[10] |  global |  same
Table 1: The literature baselines. The baseline below the ruler were not
available for evaluation. For global methods, aligning the video is performed
similarly to aligning the audio.

###  6.1 Single Shift Synchronization For LRS2-BBC

In order to create the evaluation set for this experiment, we uniformly
shifted the audio of LRS2’s test set, by a random shift of up to one second.
We compare our method to SyncNet and the modified DTA method (matching is
performed in the other direction). Despite some effort, we could not make the
original DTA method work well on this dataset. SyncNet is run in the accurate
mode, in which it is applied multiple times at different strides followed by a
voting procedure. Our method is run only once.

The results are reported in Tab. 2. As can be seen, our method outperforms the
other methods by a significant margin. In 88% of 1,200 videos, our prediction
was exact. Moreover, the error for our method was never more than one frame.
The results of our method without the attention mechanism are added to the
table in order to study the importance of this module. As can be seen, the
performance without the attention mechanism is not competitive.

Method |  Mean shift error (frames) |  Max shift error (frames) |  Top-1 accuracy per video
---|---|---|---
SyncNet |  0.45 |  36 |  60%
Modified DTA |  0.25 |  27 |  83%
Ours |  0.12 |  1 |  88%
Ours (no Att.) |  0.61 |  13 |  45%
Table 2: Single shift performance evaluation for LRS2. The top-1 accuracy per
video indicates the percentage of videos where the global shift was predicted
exactly.

###  6.2 Dynamic Per-Frame Shifts For LRS2-BBC

As detailed in Sec. 5, using the same pipeline that was used for creating the
training set, an evaluation set for LRS2-BBC was created.

The results for the modified DTA method as well as for our method are reported
in Tab. 3. The original DTA method, which is the only other local alignment
method, cannot be applied here since the video is modified and the audio
remains unchanged. As can be seen, our method greatly outperforms the baseline
method in both the average shift error and in the percent of frames that were
assigned the correct alignment. The table also contains the results of our
method without the attention mechanism. The results for this method are
considerably worse than those of our complete method.

Method | Mean shift error (frames) | Top-1 accuracy
---|---|---
Modified DTA | 1.94 | 25%
Ours | 0.85 | 60%
Ours (no Att.) | 1.60 | 26%
Table 3: Results on the LRS2-BBC benchmark, where shifts vary locally. We have
measured the average shift error in frames and the percentage of frames where
the local shift was predicted exactly (Accuracy).

###  6.3 Shift Per-Frame Synchronization For TCD-TIMIT

For the TCD-TIMIT benchmark, the videos for sentences sa1 and sa2 of the first
5 speakers were used. All permutations between source and destination
speakers, for each of the two sentences, were evaluated, resulting in 40
different experiments.

In each experiment, we have the video of speaker $a$ aligned to the audio of
speaker $b$. In order to evaluate performance, we can compare the modified
video of speaker $a$ to the video of speaker $b$. We perform this comparison
by considering the correlation of facial keypoints in those videos.

The OpenFace [6] software is used to extract these keypoints, and we calculate
the Pearson correlation in the mouth height (measured as the vertical distance
between key points 63 and 67) and in the mouth width (horizontal distance
between facial key points 61 and 65). A high correlation indicates better
alignment.

We compare the correlations along both the $x$ and $y$ axes with no alignment,
as well as the modified DTA method (since the video is modified) and our
method. The results, listed in Tab. 4, indicate that the proposed method
outperforms the modified DTA method.

In addition to measuring the correlation between the aligned video and the
original video that is associated with the voice, the quality of the generated
video also needs to be evaluated. When a video is generated by duplicating or
deleting frames, it is desirable to duplicate and delete as few frames as
possible. In addition, we would prefer situations in which consecutive frames
in the original video remain consecutive in the modified version of it.
Lastly, we would like to cover as many of the original frames, i.e., have as
many unique frames as possible present in the modified video. As Tab. 5 shows,
our method has a sizable advantage in all four criteria (number of duplicates,
number of deletions, number of consecutive frames copied sequentially, and
number of unique frames in the modified video).

Method | corr x | corr y
---|---|---
No alignment | 0.31 | 0.46
Modified DTA | 0.40 | 0.47
Ours | 0.42 | 0.50
Table 4: Average Pearson correlation for mouth height (correlation y) and width (correlation x) on the TCD-TIMIT cross speaker benchmark.  Method | dup$\downarrow$ | del$\downarrow$ | conseq$\uparrow$ | unique$\uparrow$
---|---|---|---|---
Modified DTA | 67.10 | 80.25 | 20.48 | 40.85
Ours | 38.45 | 44.85 | 58.95 | 76.25
Table 5: Number of deleted and duplicated frames on the TCD-TIMIT benchmark,
as well as the number of frames in which the matching frame is the consecutive
frame of the match of the previous frame and the number of unique frames.

In addition, we have also repeated the experiment such that the audio is
aligned to the video, as is done in the unmodified DTA method. For our method,
this required modifying it to perform the alignment in the other direction.
For this purpose, we replace the roles of audio and video and let each
distance vector $\vec{\rho_{k}}$ represent the distances of 75 audio frames to
a single video frame, out of 25 such frames. We do this without retraining the
network, assuming symmetry in the alignment process. Additional performance
may be gained by training specifically for this scenario.

For evaluation, we need to compare the warped audio of speaker $b$ to the
original audio of speaker $a$. This is done using the Mel Cepstral Distortion
(MCD) measure. The closer the sequences are, the lower the MCD score. A second
variant of this distortion (MCD-DTW), employs Dynamic Time Warping to minimize
the MCD score. In the context of alignment methods, MCD-DTW measures the
extent to which content is maintained during the alignment process. In other
words, one can obtain a relatively low MCD score, while removing part of the
original audio, instead of mapping it. The MCD-DTW score performs its own
alignment and is more sensitive to such neglect.

For a fair comparison, we normalized the audio of the reference and warped
inputs for all methods into -30 dBFS. The Mel generalized cepstral analysis is
done with a frame length of 64 ms and hop length of 5 ms, in order to extract
60 length features for the warped and reference audio streams. The obtained
scores are reported in Table 6. Our method outperforms DTA with respect to
both MCD and MCD-DTW.

Method | MCD | MCD-DTW
---|---|---
DTA | 10.37 $\pm$ 1.84 | 5.64 $\pm$ 1.19
Ours (audio warp version) | 9.13 $\pm$ 2.11 | 3.92 $\pm$ 1.48
Table 6: MCD and MCD DTW scores for TCD-TIMIT (Mean $\pm$ SD; lower is better)

###  6.4 User Study

In order to quantitatively evaluate the performance of our method, we
conducted a user study. TCD-TIMIT dataset was used in order to synthesize
eight different videos, where the audio of a source speaker was synchronized
with the video of a different target speaker. All pairs are both either males
or females and the videos consist of seven different speakers, four males and
three females, saying sentences sa1 and sa2.

The participants were presented with the videos and were asked to tag each
video with a real or a modified label. The participants were told that we may
present any combination of real and modified videos, while, in reality, we
always presented two real and two modified videos that were randomly selected
and ordered.

The participants were given ample time, were able to watch the videos an
unlimited number of times, and were challenged to do their best. The videos
were watched on a 27 inch computer monitor, and were displayed at a resolution
of $1920\times 1080$. Ten males and ten females between the ages of 20–50
participated in the study. The results are shown in Tab. 7. As can be seen,
the results clearly demonstrate that most of the time participants are not
able to distinguish between real and modified videos, which indicates the
artifact-free nature of the modified video and that these videos are extremely
well-synchronized.

Answer | Real video | Modified video
---|---|---
“The video is real” | $86\%$ | $83\%$
“The video is fake” | $14\%$ | $17\%$
Table 7: User study results. The 20 users were asked to specify for a set of
four videos, which ones are real and which are fake.

###  6.5 Application to TTS Voices

Our method allows synchronizing the audio and the video streams of two
different speakers, where the tempo of the two streams is completely
different. One potential usage is for same-language dubbing, which is a
standard industry practice known as looping. Same-language dubbing is required
for cases where filming special effects shots or in the presence of a
background noise. Another common case is with shows aimed at preschoolers
where dubbing is done in order to ensure the usage of a ”right accent”, as
they learn to speak their native tongue. Another use case are animation films
and musicals, where the actor’s singing ability may not be good enough, and
may be dubbed by a professional singer.

In order to illustrate the capabilities of our method for same-language
dubbing, we use a text to speech module in order to dub variety of speakers.
We use free online services as the text to speech system, and a variety of
YouTube videos, including hosting shows, Obama’s weekly addresses, etc. as the
video stream. The results are very convincing and are available as part of the
supplementary videos.

###  6.6 Runtime

The proposed method and the modified DTA baseline are both running
significantly faster than real-time. Our method is $1.7$ times faster than the
modified DTA baseline, once the input tensors have been prepared. However, the
run time of both is negligible in comparison to the time it takes to extract
the facial keypoints.

##  7 Conclusions and Future Work

We present a novel method for lip syncing. The method obtains a sizable gap in
accuracy over the literature on multiple benchmarks. Applying the method, even
in challenging scenarios where the audio and the video do not perfectly match,
results in an artifact-free, well-aligned video. As we demonstrate, users find
it hard to distinguish, based on either quality or the accuracy of the audio-
video alignment, between the modified video and the original unmodified video.
While we have demonstrated compelling results, further improvements may be
considered. In order to eliminate the post-processing step, a monotonic
attention mechanism [23], [7], [20], may be utilized, in order to force the
decoder to produce a monotonically increasing alignment. In addition,
monotonic constraints could be added to the loss function, as well as using an
Euclidean or an ordinal loss function, in order to weight the magnitude of the
prediction error. Our code and scripts for training and running the method are
attached as supplementary and would be released as open source.

##  8 Acknowledgements

This project has received funding from the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme
(grant ERC CoG 725974).

## References

  * [1] Dynamic Time Warping, pages 69–84.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2007.
  * [2] T. Afouras, J. Son Chung, A. Senior, O. Vinyals, and A. Zisserman.  Deep Audio-Visual Speech Recognition.  arXiv e-prints, page arXiv:1809.02108, Sep 2018.
  * [3] X. Anguera, J. Luque, and C. Gracia.  Audio-to-text alignment for speech recognition with very limited resources.  In Fifteenth Annual Conference of the International Speech Communication Association, 2014.
  * [4] E. Argones Rúa, H. Bredin, C. García-Mateo, G. Chollet, and D. Jiménez.  Audio-visual speech asynchrony detection using co-inertia analysis and coupled hidden markov models.  Pattern Anal. Appl., 12:271–284, 09 2009.
  * [5] D. Bahdanau, K. Cho, and Y. Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [6] T. Baltrusaitis, A. B. Zadeh, Y. C. Lim, and L.-P. Morency.  Openface 2.0: Facial behavior analysis toolkit.  2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 59–66, 2018.
  * [7] C.-C. Chiu and C. Raffel.  Monotonic Chunkwise Attention.  arXiv e-prints, page arXiv:1712.05382, Dec 2017.
  * [8] J. S. Chung and A. Zisserman.  Lip reading in the wild.  In ACCV, 2016.
  * [9] J. S. Chung and A. Zisserman.  Out of time: Automated lip sync in the wild.  In ACCV Workshops, 2016.
  * [10] S.-W. Chung, J. Son Chung, and H.-G. Kang.  Perfect match: Improved cross-modal embeddings for audio-visual synchronisation.  arXiv e-prints, page arXiv:1809.08001, Sep 2018.
  * [11] X. Glorot and Y. Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Y. W. Teh and M. Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249–256, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.
  * [12] T. Halperin, A. Ephrat, and S. Peleg.  Dynamic Temporal Alignment of Speech to Lips.  arXiv e-prints, page arXiv:1808.06250, Aug 2018.
  * [13] N. Harte and E. Gillen.  Tcd-timit: An audio-visual corpus of continuous speech.  IEEE Transactions on Multimedia, 17(5):603–615, May 2015.
  * [14] A. Haubold and J. R. Kender.  Alignment of speech to highly imperfect text transcriptions.  In 2007 IEEE International Conference on Multimedia and Expo, pages 224–227. IEEE, 2007.
  * [15] T. Karras, T. Aila, S. Laine, A. Herva, and J. Lehtinen.  Audio-driven facial animation by joint end-to-end learning of pose and emotion.  ACM Transactions on Graphics, 36:1–12, 07 2017.
  * [16] D. P. Kingma and J. Ba.  Adam: A Method for Stochastic Optimization.  arXiv e-prints, page arXiv:1412.6980, Dec 2014.
  * [17] J. B. Kruskal.  An overview of sequence comparison: Time warps, string edits, and macromolecules.  SIAM Review, 25(2):201–237, 1983.
  * [18] R. Kumar, J. Sotelo, K. Kumar, A. de Brebisson, and Y. Bengio.  ObamaNet: Photo-realistic lip-sync from text.  arXiv e-prints, page arXiv:1801.01442, Dec 2017.
  * [19] J. Lewis.  Automated lip-sync: Background and techniques.  The Journal of Visualization and Computer Animation, 2(4):118–122, 1991.
  * [20] X. Ma, J. Pino, J. Cross, L. Puzon, and J. Gu.  Monotonic Multihead Attention.  arXiv e-prints, page arXiv:1909.12406, Sep 2019.
  * [21] E. Marcheret, G. Potamianos, J. Vopicka, and V. Goel.  Detecting audio-visual synchrony using deep neural networks.  In INTERSPEECH, 2015.
  * [22] S. Morishima, S. Ogata, K. Murai, and S. Nakamura.  Audio-visual speech translation with automatic lip syncqronization and face tracking based on 3-d head model.  In 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 2, pages II–2117–II–2120, May 2002.
  * [23] C. Raffel, M.-T. Luong, P. J. Liu, R. J. Weiss, and D. Eck.  Online and Linear-Time Attention by Enforcing Monotonic Alignments.  arXiv e-prints, page arXiv:1704.00784, Apr 2017.
  * [24] M. E. Sargin, Y. Yemez, E. Erzin, and A. M. Tekalp.  Audiovisual synchronization and fusion using canonical correlation analysis.  IEEE Transactions on Multimedia, 9(7):1396–1403, Nov 2007.
  * [25] A. M. Saxe, J. L. McClelland, and S. Ganguli.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  arXiv e-prints, page arXiv:1312.6120, Dec 2013.
  * [26] S. Shariat and V. Pavlovic.  Isotonic cca for sequence alignment and activity recognition.  In 2011 International Conference on Computer Vision, pages 2572–2578, Nov 2011.
  * [27] J. Son Chung, A. Jamaludin, and A. Zisserman.  You said that?  arXiv e-prints, page arXiv:1705.02966, May 2017.
  * [28] S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman.  Synthesizing obama: Learning lip sync from audio.  ACM Trans. Graph., 36(4):95:1–95:13, July 2017.
  * [29] G. Trigeorgis, M. A. Nicolaou, S. Zafeiriou, and B. W. Schuller.  Deep canonical time warping.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5110–5118, 2016.
  * [30] R. Vedantam, C. L. Zitnick, and D. Parikh.  CIDEr: Consensus-based Image Description Evaluation.  arXiv e-prints, page arXiv:1411.5726, Nov 2014.
  * [31] A. Zadeh, T. Baltrušaitis, and L.-P. Morency.  Convolutional Experts Constrained Local Model for Facial Landmark Detection.  arXiv e-prints, page arXiv:1611.08657, Nov 2016.
  * [32] F. Zhou and F. De la Torre.  Generalized time warping for multi-modal alignment of human motion.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1282–1289. IEEE, 2012.
  * [33] F. Zhou and F. Torre.  Canonical time warping for alignment of human behavior.  In Advances in neural information processing systems, pages 2286–2294, 2009.
  * [34] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In Proceedings of the IEEE international conference on computer vision, pages 19–27, 2015.
  * [35] G. Zoric and S. Igor.  A real-time lip sync system using a genetic algorithm for automatic neural network configuration.  pages 1366–1369, 01 2005.


# Tetrahedra with congruent face pairs

Daniel A. Klain  D. Klain, Department of Mathematical Sciences, University of
Massachusetts Lowell, Lowell, MA 01854, USA<EMAIL_ADDRESS>
###### Abstract.

If the four triangular facets of a tetrahedron can be partitioned into pairs
having the same area, then the triangles in each pair must be congruent to one
another. A Heron-style formula is then derived for the volume of a tetrahedron
having this kind of symmetry.
Mathematics Subject Classification: 52B10, 52B12, 52B15, 52A38.

From elementary geometry we learn that two triangles are congruent if their
edges have the same three lengths. In particular, there is only one congruence
class of equilateral triangles having a given edge length. Said differently,
any pair of equilateral triangles in the Euclidean plane are similar,
differing at most by an isometry and a dilation. Meanwhile, triangles that are
symmetric under a single reflection have two congruent sides and are said to
be isosceles.

The situation is more complicated in higher dimensions. Indeed, an analogous
characterization of $3$-dimensional tetrahedra already leads to $25$ different
symmetry classes [22]. These tetrahedral symmetry classes are of special
interest in organic chemistry [8, 9, 21], and conditions for tetrahedral
symmetry based on the measures of dihedral angles have also been explored
[23].

A tetrahedron in $\mathbb{R}^{3}$ is equilateral or regular if all of its
edges have the same length. More generally, a tetrahedron is said to be
isosceles if all four triangular facets are congruent to one another, or,
equivalently, if opposing (non-incident) edges have the same length. Isosceles
tetrahedra are also known as disphenoids [4, p. 15]. It has been shown that if
all four facets of a tetrahedron $T$ have the same area, then $T$ must be
isosceles [10, p. 94][11, 16].

Consider the following more general symmetry class of tetrahedra: A
tetrahedron $T$ will be called reversible if its facets are congruent in
pairs; that is, if the facets of $T$ can be labelled
$f_{1},f_{2},f_{3},f_{4}$, where $f_{1}\cong f_{2}$ and $f_{3}\cong f_{4}$.

$a$$a$$b$$b$$d$$c$ Figure 1. An reversible tetrahedron with edge lengths
$a,a,b,b,c,d$.

In this note we show that, as in the isosceles case, reversible tetrahedra are
characterized by the areas of their facets: if the four triangular facets of
$T$ can be partitioned into pairs with the same area, then those pairs consist
of congruent facets.

In the final section we give an intuitive method for deriving a Heron-style
factorization of the volume of a reversible tetrahedron in terms of its edge
lengths.

##  1\. Facets normals and areas determine tetrahedra

The following proposition will allow us to exploit symmetries more easily.

######  Proposition 1.1.

Suppose that $u_{0},u_{1},u_{2},u_{3}$ are unit vectors that span
$\mathbb{R}^{3}$, and suppose that
$\alpha_{0},\alpha_{1},\alpha_{2},\alpha_{3}>0$. Then there exists a
tetrahedron $T$, having outward facet unit normals $u_{i}$, and corresponding
facet areas $\alpha_{i}$, if and only if

(1)
$\displaystyle\alpha_{0}u_{0}+\alpha_{1}u_{1}+\alpha_{2}u_{2}+\alpha_{3}u_{3}=0.$

Moreover, this tetrahedron is unique up to translation.

This proposition is a very special case of the Minkowski Existence Theorem,
which plays a central role in the Brunn-Minkowski theory of convex bodies, and
is somewhat difficult to prove [3, 18]. However, this special case for
tetrahedra is a simple consequence of linear algebra.

$u_{3}$$u_{1}$$u_{2}$$u_{0}$$v_{0}$$v_{3}$$v_{1}$$v_{2}$ Figure 2. A
tetrahedron with outward unit normals $u_{i}$.

###### Proof.

Let $T$ be a tetrahedron with vertices at
$v_{0},v_{1},v_{2},v_{3}\in\mathbb{R}^{3}$, where $v_{0}=o$, the origin. Let
us assume the vertices are labelled so that $v_{1},v_{2},v_{3}$ have a
positive (“right-handed”) orientation.

Denote by $u_{0},u_{1},u_{2},u_{3}$ the outward unit normal vectors of the
facets of $T$, where $u_{i}$ is associated with the facet opposite to the
vertex $v_{i}$, as in Figure 2. Let $\alpha_{i}$ denote the area of that same
$i$th facet. Since $v_{0}=o$, we have

$\displaystyle v_{2}\times v_{3}$ $\displaystyle=-2\alpha_{1}u_{1}$ (2)
$\displaystyle v_{3}\times v_{1}$ $\displaystyle=-2\alpha_{2}u_{2}$
$\displaystyle v_{1}\times v_{2}$ $\displaystyle=-2\alpha_{3}u_{3}$
$\displaystyle(v_{3}-v_{1})\times(v_{2}-v_{1})$
$\displaystyle=-2\alpha_{0}u_{0}.$

After summing both sides of these equations the identity (1) now follows.

Next, let $A$ denote a $3\times 3$ matrix whose columns are given by the
vectors $v_{1},v_{2},v_{3}$. Recall that these vectors are ordered so that $A$
has positive determinant. In this instance $\det(A)=6V(T)$, where $V(T)$
denotes the volume of $T$.

Let $c(A)$ denote the cofactor matrix of $A$. Cramer’s Rule asserts that

(3) $c(A)^{t}A=\det(A)I,$

where $I$ is the $3\times 3$ identity matrix. (See [2, p. 30], for example, or
any traditional linear algebra text.)

Let $z_{i}$ denote the $i$th column of the matrix $c(A)$. The identity (3)
asserts that $z_{i}\perp v_{j}$ for $j\neq i$. It follows from (2) that
$z_{i}$ is parallel to the facet normal $u_{i}$, and that
$z_{i}=-|z_{i}|u_{i}$, since $z_{i}\cdot v_{i}=\det(A)>0$, while $u_{i}$
points out of the tetrahedron (away from the vertex $v_{i}$). Meanwhile, (3)
also asserts that

$z_{i}\cdot v_{i}\;=\;\det(A)\;=\;6V(T).$

It follows that

$-|z_{i}|(u_{i}\cdot v_{i})\;=\;z_{i}\cdot
v_{i}\;=\;6V(T)\;=\;2\alpha_{i}(-u_{i}\cdot v_{i}),$

where the final identity follows from the base-height formula for the volume
of a cone, using the $i$th facet of $T$ as the base. Hence,
$|z_{i}|=2\alpha_{i}$ and $z_{i}=-2\alpha_{i}u_{i}.$ In other words, the facet
normals $u_{1},u_{2},u_{3}$ and corresponding facet areas $\alpha_{i}$ are
determined by the columns $z_{i}$ of the cofactor matrix $c(A)$. The remaining
facet normal $u_{0}$ and area $\alpha_{0}$ is then determined by the identity
(1). This encoding of facet data into the cofactor matrix allows a simple
proof of both existence and uniqueness for the tetrahedron $T$ given the data
$\\{u_{i}\\}$ and $\\{\alpha_{i}\\}$.

To prove the uniqueness of $T$, note that $c(A)=\det(A)A^{-t}$, by Cramer’s
Rule (3). It follows that $\det(c(A))=\det(A)^{2}$ and that

$A\;=\;\det(A)c(A)^{-t}\;=\;\det(c(A))^{\frac{1}{2}}c(A)^{-t}.$

In other words, if two matrices $A$ and $B$ with positive determinant have the
same cofactor matrix $c(A)=c(B)$, then $A=B$. It follows that if two
tetrahedra $T_{1}$ and $T_{2}$ each have the origin as a vertex and share the
same facet normals and corresponding facet areas, then $T_{1}$ and $T_{2}$
must have the same vertices, so that $T_{1}=T_{2}$.

More generally, if two tetrahedra have the same facet normals and
corresponding facet areas, then they must be translates of one another.

To prove the existence of a tetrahedron having the given facet data, Let $C$
denote the matrix having columns $-2\alpha_{i}u_{i}$ for $i>0$, ordered so
that $C$ has positive determinant. The matrix

$A=\det(C)^{\frac{1}{2}}C^{-t}$

has cofactor matrix $C$. The columns of $A$, along with the origin, yield the
vertices of a tetrahedron having facet normals $u_{i}$ and corresponding facet
areas $\alpha_{i}$. ∎

##  2\. Equal areas imply congruent faces

We now prove that the areas of the facets alone will determine if a
tetrahedron is reversible.

######  Theorem 2.1.

Suppose that $T$ is a tetrahedron in $\mathbb{R}^{3}$, and denote by
$f_{1},f_{2},f_{3},f_{4}$ the triangular facets of $T$. If the facets of $T$
satisfy the conditions

$Area(f_{1})=Area(f_{2})\;\;\;\hbox{ and }\;\;\;Area(f_{3})=Area(f_{4})$

then $f_{1}\cong f_{2}$ and $f_{3}\cong f_{4}$.

The proof of Theorem 2.1 uses the method given by McMullen in [16] to verify
the special case in which all four facets have the same area (as in Corollary
2.2 below).

###### Proof.

Denote by $u_{i}$ the outward unit normal vector to the facet $f_{i}$ of $T$.
Suppose that $Area(f_{1})=Area(f_{2})=\alpha$ and
$Area(f_{3})=Area(f_{4})=\beta$, where $\alpha,\beta>0$. The identity (1)
asserts that

$\alpha u_{1}+\alpha u_{2}+\beta u_{3}+\beta u_{4}=0.$

Denote

$w=\alpha u_{1}+\alpha u_{2}=-\beta u_{3}-\beta u_{4}.$

Let $\psi$ denote the rotation of $\mathbb{R}^{3}$ by the angle $\pi$ around
the the axis through $w$. Since the vectors $\alpha u_{1}$ and $\alpha u_{2}$
have the same length, the points $o,\alpha u_{1},\alpha u_{2},w$ are the
vertices of a rhombus. The rotation $\psi$ rotates this rhombus onto itself,
exchanging the vectors $\alpha u_{1}$ and $\alpha u_{2}$. The points $o,\beta
u_{3},\beta u_{4},-w$ form a rhombus through the same axis, so that $\psi$
also exchanges the vectors $\beta u_{3}$ and $\beta u_{4}$. Since $\psi$ is a
rotation, it preserves orthogonality. It follows that $P$ and $\psi P$ have
the same normal vectors and the same corresponding facet areas. Proposition
1.1 then implies that $P$ and $\psi P$ are congruent by a translation. In
particular, the facets $f_{1}$ and $f_{2}$ are congruent, as are $f_{3}$ and
$f_{4}$. ∎

The case of isosceles tetrahedra described in the introduction follows as an
immediate corollary to Theorem 2.1.

######  Corollary 2.2.

Suppose that $T$ is a tetrahedron in $\mathbb{R}^{3}$. If the faces $f_{i}$ of
$T$ satisfy the condition

$Area(f_{1})\;=\;Area(f_{2})\;=\;Area(f_{3})\;=\;Area(f_{4})$

then $f_{1}\cong f_{2}\cong f_{3}\cong f_{4}$.

In other words, if a tetrahedron $T$ is equiareal, then $T$ is also isosceles.
For alternative proofs and variants of Corollary 2.2, see [10, 11, 15, 16].

Remark: Corollary 2.2 has long been known to have an analogue in which area is
replaced by perimeter. The proof is very simple: If all of the facets of $T$
have the same perimeter, the resulting system of linear equations (in the six
edge lengths of $T$) implies that opposing edges must have the same length, so
that $T$ is isosceles. A similar argument shows that if the facets of $T$ can
be partitioned into pairs having the same perimeter then $T$ is reversible.

##  3\. Factoring the volume

Suppose that $T\subseteq\mathbb{R}^{3}$ is a tetrahedron with vertices at
$v_{0},v_{1},v_{2},v_{3}\in\mathbb{R}^{3}$, where $v_{0}=o$, the origin. As
before, let $A$ denote the matrix whose columns are given by the vectors
$v_{i}$, and suppose that the $v_{i}$ are ordered so that $A$ has positive
determinant. The volume of $T$ is then given by $\det(A)=6V(T),$ so that

$V(T)^{2}=\frac{1}{36}\,\det(A^{t}A).$

The entries of the matrix $A^{t}A$ are dot products of the form $v_{i}\cdot
v_{j}$. From the identity,

(4) $2v_{i}\cdot v_{j}\;=\;|v_{i}|^{2}+|v_{j}|^{2}-|v_{i}-v_{j}|^{2}$

it then follows that the value of $V(T)^{2}$ is a polynomial in the squares of
the edge lengths of $T$. Said differently, if $T$ has edge lengths $a_{ij}$
(the distance between vertices $v_{i}$ and $v_{j}$), then $V(T)^{2}$ is a
polynomial in the variables $b_{ij}=a_{ij}^{2}$, as well as the variables
$a_{ij}$ themselves. This polynomial is sometimes formulated in terms of
linear algebraic expressions such as Cayley-Menger determinants [19, p. 125].
While the Cayley-Menger heuristic outlined above applies in arbitrary
dimension, the $3$-dimensional case has been known at least as far back as
Piero della Francesca [17].111Piero della Francesca (1415-1492), an Italian
painter and geometer of the early Renaissance period.

In certain instances, the polynomial $V(T)^{2}$ admits factorization into
linear or quadratic irreducible factors. For the 2-dimensional case, the area
$A(\Delta)$ of a triangle $\Delta$ having edge lengths $a,b,c$ is given by

$A(\Delta)^{2}=\frac{1}{16}(a+b+c)(-a+b+c)(a-b+c)(a+b-c),$

a factorization known as Heron’s formula [5, p. 58]. Although the
3-dimensional case is more complicated [7], there exist non-trivial
factorizations of $V(T)^{2}$ when the tetrahedron $T$ satisfies the symmetry
properties examined in the previous section.

For example, if $T$ is an isosceles tetrahedron, having edge lengths $a,b,c$
(each repeated twice in pairs of opposing edges), then

(5)
$V(T)^{2}=\frac{1}{72}\big{(}a^{2}+b^{2}-c^{2}\big{)}\big{(}a^{2}-b^{2}+c^{2}\big{)}\big{(}-a^{2}+b^{2}+c^{2}\big{)}.$

A synthetic proof of (5) can be found in [20, p. 101]. Instead we will give an
algebraic proof of the following more general result, using a technique
outlined in [13].

The edges of a reversible tetrahedron $T$ come in (at most) 4 lengths. To see
this, label the edge lengths of $T$ so that the triangular facets $f_{1}\cong
f_{2}$ have edge lengths $a,b,c$, with common edge of length $c$. Since
$f_{3}\cong f_{4}$, they must have edge lengths $a,b,d$. The six edges of $T$
then have lengths $a,a,b,b,c,d$, as in Figure 1.

######  Theorem 3.1 (Volume Formula).

Suppose that $T$ is a reversible tetrahedron having edge lengths
$a,a,b,b,c,d$. Then

(6)
$V(T)^{2}=\frac{1}{72}\bigg{(}c^{2}d^{2}-(a^{2}-b^{2})^{2}\bigg{)}\bigg{(}a^{2}+b^{2}-\frac{c^{2}+d^{2}}{2}\bigg{)}.$

The first polynomial factor in the formula (6) is a difference of two squares,
so that (6) can be reformulated as

(7)
$V(T)^{2}=\frac{1}{72}\big{(}cd+a^{2}-b^{2}\big{)}\big{(}cd-a^{2}+b^{2}\big{)}\bigg{(}a^{2}+b^{2}-\frac{c^{2}+d^{2}}{2}\bigg{)}.$

In the special case where $c=d$, the tetrahedron $T$ is isosceles, and the
formula (7) reduces to (5).

The proof of (6) will make use of two identities from plane geometry. The
well-known parallelogram law asserts that if edges of a parallelogram in
$\mathbb{R}^{2}$ are labelled as in Figure 1, then
$2a^{2}+2b^{2}=c^{2}+d^{2}$.

The less well-known trapezoid law asserts that, if the edges of a convex
isosceles trapezoid are labelled as in Figure 3, then $b^{2}-a^{2}=cd$.

$c$$d$$a$$a$$b$$b$${\bf o}$${\bf w}$${\bf v}$${\bf u}$ Figure 3. The trapezoid
law: $b^{2}-a^{2}=cd$.

To see why, observe that

$\displaystyle b^{2}-a^{2}$ $\displaystyle=|{\bf u}-{\bf w}|^{2}-|{\bf v}-{\bf
w}|^{2}$ $\displaystyle={\bf u}\cdot{\bf u}-2{\bf u}\cdot{\bf w}+{\bf
w}\cdot{\bf w}-{\bf v}\cdot{\bf v}+2{\bf v}\cdot{\bf w}-{\bf w}\cdot{\bf w}$
$\displaystyle=|{\bf u}|^{2}-|{\bf v}|^{2}+2{\bf w}\cdot({\bf v}-{\bf u})$
$\displaystyle=a^{2}-b^{2}+2cd,$

where the last step follows from the parallelism of ${\bf w}$ and ${\bf
v}-{\bf u}$. The trapezoid law now follows.

###### Proof of The Volume Formula 6.

Let $f(a,b,c,d)$ denote the polynomial $V(T)^{2}$. The factors of $f$ can be
determined by considering the cases in which the volume of $T$ is zero,
namely, when the tetrahedron $T$ is flat or otherwise degenerate. If $T$ is
reversible, this can occur in two ways.

In one case, $T$ may flatten to a parallelogram, having edges of length
$a,b,a,b$ and diagonals of length $c,d$. In this instance, the parallelogram
law for the standard inner product implies that $2a^{2}+2b^{2}=c^{2}+d^{2}$.

In the second case, $T$ may flatten to a trapezoid, having non-parallel edges
of length $a,a$, parallel edges of length $c,d$, and diagonals of length
$b,b$. In this instance, the trapezoid law implies that
$(b^{2}-a^{2})^{2}=c^{2}d^{2}$,

These cases suggest both $2a^{2}+2b^{2}-c^{2}-d^{2}$ and
$c^{2}d^{2}-(b^{2}-a^{2})^{2}$ as possible factors of the polynomial $f$.

Denote $A=a^{2}$, $B=b^{2}$, $C=c^{2}$ and $D=d^{2}$. We observed following
(4) above that $f$ is a polynomial in the squared values
$a^{2},b^{2},c^{2},d^{2}$, so that $f=f(A,B,C,D)\in\mathbb{R}[A,B,C,D]$. Since
volume $V$ is homogeneous of degree 3 with respect to length, the polynomial
$f=V^{2}$ is homogeneous of degree 6 with respect to the variables $a,b,c,d,$
and is therefore homogeneous of degree 3 with respect to the variables
$A,B,C,D$; that is, a homogeneous cubic polynomial in $\mathbb{R}[A,B,C,D]$.

To verify that $2a^{2}+2b^{2}-c^{2}-d^{2}$ is indeed a factor of $f(a,b,c,d)$,
use division with remainder in $\mathbb{R}[A,B,C,D]$ to obtain

$f(A,B,C,D)=(2A+2B-C-D)g(A,B,C,D)+r(B,C,D),$

for some $g\in\mathbb{R}[A,B,C,D]$ and $r\in\mathbb{R}[B,C,D]$. Here division
with remainder in $\mathbb{R}[A,B,C,D]$ is performed here using
lexicographical order on the variables $A,B,C,D$. (See, for example, [6, p.
54].)

Note that $A$ does not appear in the polynomial expression for $r$. Suppose
that $C>D>0$. By the triangle inequality, each $B$ such that

$\sqrt{C}-\sqrt{D}<2\sqrt{B}<\sqrt{C}+\sqrt{D}$

gives rise to a parallelogram as in Figure 4, yielding $A\geq 0$ so that
$2A+2B-C-D=0$. This degenerate reversible tetrahedron $T$ has volume zero, so
that $f(A,B,C,D)=V^{2}=0$. It follows that $r(B,C,D)=0$ on a non-empty open
set. Since $r$ is a polynomial, it follows that $r$ is identically zero, so
that

$f(A,B,C,D)=(2A+2B-C-D)g(A,B,C,D).$

In other words, $2A+2B-C-D$ divides $f$ in $\mathbb{R}[A,B,C,D]$.

$\sqrt{A}$$\sqrt{B}$$\sqrt{C}/2$$\sqrt{D}/2$ Figure 4. This parallelogram
exists iff
$\frac{1}{2}\sqrt{C}-\frac{1}{2}\sqrt{D}\leq\sqrt{B}\leq\frac{1}{2}\sqrt{C}+\frac{1}{2}\sqrt{D}$.

For the trapezoidal factors, view $f$ as polynomial in $\mathbb{R}[A,B,c,d]$,
and write

$f(A,B,c,d)=(cd-B+A)\tilde{g}(A,B,c,d)+\tilde{r}(B,c,d),$

using division with remainder in $\mathbb{R}[A,B,c,d]$ under lexicographical
order on the variables $A,B,c,d$. Once again the remainder $\tilde{r}$ is
independent of the variable $A$, while a trapezoidal degenerate (zero volume)
tetrahedron can be constructed for an open set of values $(B,c,d)$, so that
$\tilde{r}$ is also identically zero. Therefore, $(cd-B+A)$ is also a factor
$f$.

Finally, a symmetrical argument (reversing the roles of $A$ and $B$) yields a
factor of $(cd-A+B)$.

Since $\mathbb{R}[A,B,c,d]$ is a unique factorization domain [2, p. 371][6, p.
149], the irreducible factors $(cd-B+A)$, and $(cd-A+B)$ are prime, so that

$(cd-B+A)(cd-A+B)\;=\;c^{2}d^{2}-(B-A)^{2}\;=\;CD-(B-A)^{2}$

divides $f$.

Similarly, since $\mathbb{R}[A,B,C,D]$ is a unique factorization domain, the
two irreducible factors $CD-(B-A)^{2}$, and $2A+2B-C-D$ are prime in
$\mathbb{R}[A,B,C,D]$, so that

(8) $\displaystyle V^{2}\;=\;f\;=\;(2A+2B-C-D)(CD-(B-A)^{2})k.$

Because $f$ is a homogeneous cubic polynomial in $\mathbb{R}[A,B,C,D]$, the
factor $k$ must be a constant, independent of the parameters $A,B,C,D$.

To compute the constant $k$, recall that the volume of the regular
(equilateral) tetrahedron of unit edge length $A=B=C=D=1$ is $\sqrt{2}/12$. It
follows that

$\frac{1}{72}=\left(\frac{\sqrt{2}}{12}\right)^{2}=V^{2}=f(1,1,1,1)=2k.$

Hence, $k=1/144$, and (8) becomes (6). ∎

I. Izmestiev has pointed out that applying the Regge symmetry [1] to a
reversible tetrahedron gives a new reversible tetrahedron having the same
volume, and for which the factors of the Cayley-Menger polynomial (7) are
permuted [12].

##  4\. Generalizations

A convex polytope $P$ in $\mathbb{R}^{n}$ will be called reversible if there
is an affine plane $\xi$ of co-dimension $2$ such that $P$ is symmetric under
the $180^{\circ}$ rotation of the $2$-plane $\xi^{\perp}$ that fixes $\xi$.

If a tetrahedron $T$ is $\mathbb{R}^{3}$ is symmetric under a $180^{\circ}$
rotation around a line $\ell$, then this rotation must map facets to facets
and facet normals to facet normals. In view of Proposition 1.1, the only way
this can occur is when $\ell$ passes through the midpoints of two non-adjacent
edges of $T$, so that $T$ must have pairs of congruent facets, as in the
examples addressed earlier. It follows that this more general definition of a
reversible polytope is consistent with the definition given earlier for
tetrahedra in $\mathbb{R}^{3}$. However, naive analogues of the theorems of
this paper do not follow, because this level of symmetry admits many more
variations in structure for dimensions $n\geq 4$. Indeed, there exist
$4$-dimensional simplices in which all $5$ facets have the same volume in
spite of not being mutually congruent. For an extensive treatment of this
subject, see [16].

In addition to admitting the Heron-type formula (5) for volume, isosceles
tetrahedra satisfy many other characteristic properties (see, for example,
[10, p. 90-97][14]). It would be interesting to consider what parallels these
other properties may have in the more general context of reversible
tetrahedra.

## References

  * [1] A. Akopyan and I. Izmestiev, The Regge symmetry, confocal conics, and the Schläfli formula, Bull. London Math. Soc., 51 (2019), 765–775.
  * [2] M. Artin, _Algebra, 2nd ed._ , Prentice-Hall, Upper Saddle River, NJ, 2010.
  * [3] T. Bonnesen and W. Fenchel, _Theory of Convex Bodies_ , BCS Associates, Moscow, Idaho, 1987.
  * [4] H. Coxeter, _Regular Polytopes_ , Dover, New York, 1973.
  * [5] H. Coxeter and S. Greitzer, _Geometry Revisited_ , MAA, Washington, D.C., 1967.
  * [6] D. Cox, J. Little, and D. O’Shea, _Ideals, Varieties, and Algorithms, 2nd ed._ , Springer, New York, 1996.
  * [7] C. D’Andrea and M. Sombra, The Cayley-Menger determinant is irreducible for $n\geq 3$, Sib. Math. J., 46 (2005), 71–76.
  * [8] P. Fowler and A. Rasset, Is There a “Most Chiral Tetrahedron”?, Chem. Eur. J., 10 (2004), 6575–6580.
  * [9] P. Fowler and A. Rasset, A classification scheme for chiral tetrahedra, C. R. Chimie, 9 (2006), 1203–1208.
  * [10] R. Honsberger, _Mathematical Gems II_ , MAA, Washington, D.C., 1976.
  * [11] J. Horváth, A property of tetrahedra with equal faces in spaces of constant curvature (Hungarian, German summary), Mat. Lapok, 20 (1969), 257–263.
  * [12] I. Izmestiev. Private communication, (2022).
  * [13] D. Klain, An intuitive derivation of Heron’s formula, Amer. Math. Monthly, 111 (2004), no. 8, 709–712.
  * [14] J. Leech, Some properties of the isosceles tetrahedron, Math. Gaz., 34 (1950), no. 310, 269–271.
  * [15] H. Martini, Regular simplices in spaces of constant curvature, Amer. Math. Monthly, 100 (1993), no. 2, 169–171.
  * [16] P. McMullen, Simplices with equiareal faces, Discrete Comput. Geom., 24 (2000), 397–411.
  * [17] M. Peterson, The geometry of Piero della Francesca, Math. Intell., 19 (1997), no. 3, 33–40.
  * [18] R. Schneider, _Convex Bodies: The Brunn-Minkowski Theory, 2nd ed._ , Cambridge University Press, New York, 2014.
  * [19] D. Sommerville, _An Introduction to the Geometry of n Dimensions_ , Dover, New York, 1958.
  * [20] H. Steinhaus, _One Hundred Problems in Elementary Mathematics_ , Dover, New York, 1979.
  * [21] K. Wirth and A. Dreiding, Edge lengths determining tetrahedrons, Elem. Math., 64 (2009), 160–170.
  * [22] K. Wirth and A. Dreiding, Tetrahedron classes based on edge lengths, Elem. Math., 68 (2013), 56–64.
  * [23] K. Wirth and A. Dreiding, Relations between edge lengths, dihedral, and solid angles in tetrahedra, J. Math. Chem., 52 (2014), 1624 –1638.