# Speex: A Free Codec For Free Speech

Jean-Marc Valin
CSIRO ICT Centre, Cnr Vimiera & Pembroke Roads, Marsfield NSW 2122, Australia
Xiph.Org Foundation
<EMAIL_ADDRESS>
###### Abstract

The Speex project has been started in 2002 to address the need for a free,
open-source speech codec. Speex is based on the Code Excited Linear Prediction
(CELP) algorithm and, unlike the previously existing Vorbis codec, is
optimised for transmitting speech for low latency communication over an
unreliable packet network. This paper presents an overview of Speex, the
technology involved in it and how it can be used in applications. The most
recent developments in Speex, such as the fixed-point port, acoustic echo
cancellation and noise suppression are also addressed.

###### Index Terms:

speech coding, voice over IP

##  I Introduction

When I started implementing Speex in 2002, there was no good free codec that
could be used for speech communication. The situation was greatly harming the
development of voice over IP (VoIP) on Linux and other free operating systems.
Unlike the Vorbis111http://www.vorbis.com/ codec that already existed at that
time, the Speex codec is optimised for speech and is designed for low latency
communication over an unreliable packet network.

Speex is based on the popular Code Excited Linear Prediction (CELP) algorithm
but because of patent restrictions, some techniques like the use of algebraic
codes (ACELP) could not be used. Nevertheless, Speex is able to achieve a
quality comparable to proprietary codecs at the same bitrate, while supporting
features not found in other speech codecs. These features include variable
bitrate, which makes better use of the bits when encoding to a file, and
embedded coding, which provides an easy way to interface wideband (16 kHz)
channels with legacy narrowband (8 kHz) telephony equipment.

Speex is now evolving into a complete toolkit for voice over IP (VoIP)
development, including algorithms for noise cancellation, acoustic echo
cancellation and adaptive jitter buffering. This allows a developer without
any signal processing knowledge to implement a VoIP client. In the mean time,
Speex is being ported to architectures without a floating-point unit, allowing
Speex to be used in embedded devices equipped with a fixed-point CPU or DSP.

The paper is organised as follows. Section II presents overview of Speex.
Then, an explanation of the underlying coding algorithm (CELP) is provided in
Section III. Section IV provides information and tips about using the Speex
library in applications. Finally, Section V discusses recent developments in
Speex and Section VI concludes this paper.

##  II Overview

The Speex project was stated in February 2002 because there was a need for a
speech codec that was open-source and free from software patents. These are
essential conditions for being used by any open-source software. At that time,
the only options for compressing speech were the G.711 codec (also known as
$\mu$-law and A-law) and the open-source implementation of the aging GSM-FR
codec222http://kbs.cs.tu-berlin.de/~jutta/toast.html, although the patent
status of the latter is unclear. The Vorbis codec was also available for
general audio compression, but it was not really suitable for speech.

Unlike many other speech codecs, Speex is not targeted at cell phones but
rather at voice over IP (VoIP) and file-based compression. Because any codec
suitable for VoIP is also suitable for file-based compression, only the VoIP
requirements are taken into account. Version 1.0 was released in March 2003
and the bitstream has been frozen since that time, which means that all
versions since then are compatible with version 1.0.

###  II-A Design Decisions

The use of Speex for VoIP imposes the following requirements:

  * •

The frame size and algorithmic delay must be small

  * •

Both the encoder and decoder must run in real-time with limited resources

  * •

The effect of lost packets during transmission must be minimised

  * •

The codec must support both narrowband and wideband

  * •

Multiple bit-rates and quality settings must be supported to take into account
different connection speeds

  * •

Good compression must be achieved while avoiding known speech coding patents

The CELP algorithm [1] was chosen because of its proven track record from low
bitrates (e.g. DoD CELP at 4.8 kbps) to high bitrates (e.g. G.728 LD-CELP at
16 kbps) and because the patents on the base algorithm have now expired.
Unfortunately, newer variants such as the use of arithmetic codebooks in ACELP
could not be used in Speex because of patent issues. It was decided to encode
in chucks of 20 ms (frame size) and only use 10 ms of additional buffering
(look-ahead).

Because Speex is designed for VoIP instead of cell phones (like many other
speech codecs) it must be robust to lost packets, but not to corrupted ones –
UDP/RTP packets either arrive unaltered or they don’t arrive at all. For this
reason, we have chosen to restrict the amount of inter-frame dependency to
pitch prediction only, without incurring the overhead of coding frames
independently as done in the iLBC [2] codec.

As much as we would like it to be true, Speex is not the most technologically
advanced speech codec available and it was never meant to be. The original
goal was to achieve the same quality as state-of-the-art codecs with
comparable or slightly higher bit-rate and this goal has been achieved.

Speex also has the following features, some of which are not found in other
speech codecs:

  * •

Integration of narrowband and wideband using an embedded bit-stream

  * •

Wide range of bit-rates available (from 2 kbps to 44 kbps)

  * •

Dynamic bit-rate switching and Variable Bit-Rate (VBR)

  * •

Voice Activity Detection (VAD, integrated with VBR)

  * •

Variable complexity

  * •

Ultra-wideband mode at 32 kHz

  * •

Intensity stereo encoding option

###  II-B Legal Status

One of the most frequently asked question about Speex is: “But how can you be
sure it is not patented?”. The short answer is that we don’t know for sure.
The longer answer is that first, we (Xiph.Org, me) obviously did not patent
Speex. During development, we were careful not to use techniques known to be
patented and we searched through the USPTO database to see if anything we were
doing was covered.

That being said, we cannot offer any absolute warranty regarding patents. No
company does that, simply because of the mess the patent system is. Even if
you pay for a codec, you cannot have the guaranty that nobody else owns a
(previously unknown) patent on that codec. Even if someone comes forward and
claims to have a patent on a certain piece of software, it takes years of
legal procedures to sort out. This has been happening recently with the JPEG
standard, which has always been thought to be free of patents. It is not yet
clear what will happen in this particular case. Sadly, this is how things are
when it comes to patents – at least in the United States because every country
has different patent laws, which makes the matter even more complicated.

In any program written that is longer than "Hello World" (and even that!),
there might be a patent issue, generally a trivial patent (e.g. 1-click
shopping) nobody thought about. There are probably patents out there claiming
rights on "using a computer for doing useful things" or something like that.
What this means for Speex is that we did our best to avoid known patents and
minimise the risk. This is about the best that can be done when writing
software at the moment, regardless of whether the software is Free or
proprietary.

##  III Speex and CELP

Speex is based on CELP, which stands for Code Excited Linear Prediction. The
CELP technique is based on three ideas:

  1. 1.

Using a linear prediction (LP) model to model the vocal tract;

  2. 2.

Using of an adaptive and a fixed codebook as the input (excitation) of the LP
model;

  3. 3.

Performing a search in closed-loop in a “perceptually weighted domain”.

The description of the Speex encoder and decoder in this section has been
greatly simplified for clarity purposes. Nonetheless, speech processing
knowledge is required to understand some parts.

###  III-A Source-Filter Speech Model

Before going into Speex and CELP in details, let us introduce the source-
filter model of speech. This model assumes that the vocal cords are the source
of spectrally flat sound (the “excitation signal”), and that the vocal tract
acts as a filter to spectrally shape the various sounds of speech. While still
an approximation, the model is widely used in speech coding because of its
simplicity. Its use is also the reason why most speech codecs (Speex included)
perform badly on music signals.

The different phonemes can be distinguished by their excitation (source) and
spectral shape (filter). Voiced sounds (e.g. vowels) have an excitation signal
that is periodic and that can be approximated by an impulse train in the time
domain or by regularly-spaced harmonics in the frequency domain. On the other
hand, fricatives (such as the “s”, “sh” and “f” sounds) have an excitation
signal that is similar to white Gaussian noise. So called “voice fricatives”
(such as “z” and “v”) have excitation signal composed of an harmonic part and
a noisy part.

###  III-B Speex Decoder

Before exploring the complex encoding process of Speex we introduce the Speex
decoder here. For the sake of simplicity only the narrowband decoder is
presented. Figure 1 describes a generic CELP decoder. The excitation is
produced by summing the contributions from an adaptive (_aka_ pitch) codebook
and a fixed (_aka_ innovation) codebook:

$e[n]=e_{a}[n]+e_{f}[n]$ (1)

where $e_{a}[n]$ is the adaptive codebook contribution and $e_{f}[n]$ is the
fixed codebook contribution.

The filter that shapes the excitation has an all-pole (infinite impulse
response) model of the form $1/A(z)$, where $A(z)$ is called the prediction
filter and is obtained using linear prediction (Levinson-Durbin algorithm). An
all-pole filter is used because it is a good representation of the human vocal
tract and because it is easy to compute.

Figure 1: A generic CELP decoder.

In Speex, frames are 20 ms long, which is 160 samples for narrowband.
Furthermore, each frame is divided into 4 sub-frames of 40 samples that are
encoded sequentially. In most modes, only the synthesis filter and the global
excitation gain (see below) is encoded on a frame basis, the other parameters
are encoded on a sub-frame basis.

Speex includes three main differences compared to most recent CELP codecs.
First, while most recent CELP codecs make use of fractional pitch estimation
[3] with a single gain, Speex uses an integer to encode the pitch period, but
uses a 3-tap predictor [4] (3 gains). The adaptive codebook contribution can
thus be expressed as:

$e_{a}[n]=g_{0}e[n-T-1]+g_{1}e[n-T]+g_{2}e[n-T+1]$ (2)

where $g_{0}$, $g_{1}$ and $g_{2}$ are the pitch gains and are jointly
quantised (VQ).

Many current CELP codecs use moving average (MA) prediction to encode the
fixed codebook gain. This provides slightly better coding at the expense of
introducing some extra dependency with past frames. Speex encodes the fixed
codebook gain as a global excitation gain for the frame as well as a set of
sub-frame gain corrections.

Speex uses sub-vector quantisation of the innovation (fixed codebook) signal.
Unfortunately, this is not a very efficient quantisation method, but it was
the best that could be used given patent restrictions. Each sub-frame is
divided into sub-vectors of length ranging between 5 and 20 samples. Each sub-
vector is chosen from a bitrate-dependent codebook and all sub-vectors are
then concatenated. As an example, the 3.95 kbps mode uses a sub-vector size of
20 samples with 32 entries in the codebook (5 bits). This means that the
innovation is encoded with 10 bits per sub-frame, or 2000 bps. On the other
hand, the 18.2 kbps mode uses a sub-vector size of 5 samples with 256 entries
in the codebook (8 bits), so the innovation uses 64 bits per sub-frame, or
12800 bps.

###  III-C Speex Encoder

The main principle behind CELP is called Analysis-by-Synthesis (AbS) and means
that the encoding (analysis) is performed by perceptually optimising the
decoded (synthesis) signal in a closed loop. In theory, the best CELP stream
would be produced by trying all possible bit combinations and selecting the
one that produces the best-sounding decoded signal. This is obviously not
possible in practice for two reasons: the required complexity is beyond any
currently available hardware and the “best sounding” selection criterion
implies a human listener.

In order to achieve real-time encoding using limited computing resources, the
CELP optimisation is broken down into smaller, more manageable, sequential
searches using a simple perceptual weighting function. In the case of Speex,
the optimisation is performed in four steps

  1. 1.

Linear prediction analysis is used to determine the synthesis filter, which is
converted to Line Spectral Pair (LSP) coefficients and vector-quantised.

  2. 2.

The adaptive codebook entry and gain are jointly searched for the best pitch-
gain combination using AbS.

  3. 3.

The fixed codebook gain is determined in an “open-loop” manner only the energy
of the excitation signal.

  4. 4.

The fixed codebook is searched for the best entry using AbS.

Optimisation for steps 2) and 4) are performed in the so called “perceptually
weighted domain”, meaning that we try to minimise the perceptual difference
with the original (as opposed to simply maximising the signal-to-noise ratio).
In order to do that, the following weighting filter is applied on the input
signal:

$W(z)=\frac{A(z/\gamma_{1})}{A(z/\gamma_{2})}$ (3)

where $A(z)$ is the linear prediction filter and $\gamma_{1}$ and $\gamma_{2}$
control the shape of the filter by moving the poles and zeros toward the
centre of the _z_ -transform unit circle (Speex uses $\gamma_{1}=0.9$ and
$\gamma_{2}=0.6$). The filter defined in 3 is in fact a very, very rough
approximation for what is known as the “masking curve” in audio codecs such as
Vorbis. The overall effect of the weighting filter $W(z)$ is that the encoder
is allowed to introduce more noise at frequencies where the power level is
high and less noise at frequencies where the power level is low.

###  III-D Wideband CELP (aka SB-CELP)

The wideband mode in Speex differs from most wideband speech codecs as it
separates the signal into two sub-bands (SB-CELP). The lower band is encoded
using the narrowband encode. This has the advantage of making it easy to
interoperate with “legacy” systems (e.g. PSTN) operating in narrowband. The
high-band is encoded using a CELP encoder similar to the narrowband encoder
with the difference that only a fixed codebook is used, with no adaptive
codebook. The high-band information is encoded after then narrowband (low-
band) data, so simply discarding the last part of a wideband frame produces a
valid narrowband frame. Alternatively, a wideband decoder is able to identify
that the high-band information is missing and thus is able to decode a
narrowband frame.

One particularity of the wideband mode made possible by the SB-CELP scheme is
that for low-bitrate, only the spectral shape (envelope) of the high-band is
encoded. The high-band excitation is obtained simply by “folding” the
narrowband excitation. The technique is similar to the low bit-rate encoding
of the high frequencies described in [5].

##  IV Implementing Speex Support

Unlike most speech codecs, Speex supports several sampling rates and several
bit-rates for each sampling rate. The thing one must determine before choosing
a codec (or in this case, which mode of the codec) is:

  * •

What is the bandwidth available?

  * •

What is the quality requirement?

  * •

What is the latency requirement?

When designing a Voice over IP application, the first aspect one must consider
is the fact that the combination of IP, UDP and RTP incurs an overhead of 40
bytes per packet transmitted. When transmitting one packet every 20 ms as is
commonly done, the overhead is equal to 16 kbps! Considering that, it is
obvious that in most situations, there is little incentive to use very low
bit-rate.

###  IV-A Choosing the Right Mode

The first choice one must make when using Speex is the sampling rate used. For
new applications, it is recommended to use wideband (16 kHz sampling rate)
audio whenever possible. Wideband speech sounds much clearer than narrowband
speech and is also easier to understand. In many case, for example when
implementing a VoIP application, it is a good idea to also support narrowband
– either directly or through the wideband encoder/decoder. In this context,
narrowband is useful for both compatibility reasons (with clients that may not
support wideband) and because at very low bit-rates (<10 kbps), it sometimes
provides better quality than wideband.

Once the sampling rate is chosen, one must decide on the bitrate. It is
generally recommended to use constant bitrate (CBR) in voice over IP
applications because of the fixed channel (especially if using a modem)
capacity. On the other hand, VBR is usually more efficient for file
compression, because bits can be allocated where needed. When calculating for
a target bit-rate, it is important to also consider the overhead. The VoIP
overhead when using RTP and one frame per packet is 16 kbps, so when using a
56 kbps modem, this leaves very little bandwidth for the codec itself. For
file encoding in Ogg, the overhead is approximately 1 byte per frame, so it is
usually not problematic. For very low bit-rates, the _–nframes_ option to
_speexenc_ can be useful as it packs multiple frames in an Ogg packet.

###  IV-B Libspeex API Overview

The reference implementation of Speex uses a C API. Only an overview of the
codec API is provided here. When using Speex, one must first create an
instance of an encoder and/or a decoder. If more than one channel is used,
then an instance is required for each channel (Speex is not stateless).

In order to encode speech using Speex, the following header must be included:

  * #include <speex/speex.h>

Then, a Speex bit-packing struct and a encoder state are necessary:

  * SpeexBits bits;

void *enc;

These are initialized by:

  * speex_bits_init(&bits);

enc = speex_encoder_init(&speex_nb_mode);

For wideband coding, speex_nb_mode will be replaced by speex_wb_mode. In most
cases, you will need to know the frame size used by the mode you are using.
The value can be obtained in the frame_size variable using:

  * speex_encoder_ctl(enc,

SPEEX_GET_FRAME_SIZE,&frame_size);

In practice, frame_size will correspond to 20 ms when using 8, 16, or 32 kHz
sampling rate.

Once the initialization is complete, frames can be encoded using:

  * speex_bits_reset(&bits);

speex_encode_int(enc_state, input_frame,

&bits);

nbBytes = speex_bits_write(&bits, byte_ptr,

MAX_NB_BYTES);

where input_frame is a _(int16_t *)_ pointing to the beginning of a speech
frame, byte_ptr is a _(char *)_ where the encoded frame will be written,
MAX_NB_BYTES is the maximum number of bytes that can be written to byte_ptr
without causing an overflow and nbBytes is the number of bytes actually
written to byte_ptr (the encoded size in bytes). Before calling
speex_bits_write, it is possible to find the number of bytes that need to be
written by calling speex_bits_nbytes(&bits), which returns a number of bytes.

After you’re done with the encoding, all resources can be freed using:

  * speex_bits_destroy(&bits);

speex_encoder_destroy(enc_state);

The decoder API can be used in a similar manner. For more details, refer the
Speex manual333http://www.speex.org/docs.html.

###  IV-C Common Layer 8 Problems

There are problems that frequently arise when implementing support for Speex.
While some of them are due to programming errors, most are by sending or
receiving the wrong signals. The most important aspect for the input signal is
the amplitude. Speex can encode any signal in the $[-32768,32767]$ range.
However, it is best to make use of the full dynamic range without being too
close to saturation because that could cause the decoded speech to saturate
even if the encoded speech does not. In practice, it is best to scale the
input signal so the maximum amplitude is between 5000 and 20000.

Another important characteristics of the signals is the frequency content. DC
offsets should be avoided at all cost as it causes the Speex encoder to
produce bad sounding files (DC is never needed for audio). A DC offsets is
sometimes caused by a cheap soundcard and means that the signal is not centred
around the zero (zero-mean). In the most simple case, a DC offset can easily
be removed using a notch filter of the form:

$N(z)=\frac{1-z^{-1}}{1-\alpha z^{-1}}$ (4)

where a value of $\alpha=0.98$ is usually appropriate for both narrowband and
wideband. The filter described in (4) can be implemented as:

  * #define ALPHA .98f

static float mem=0;

for (i=0;i<frame_size;i++)

{

mem = ALPHA*mem + (1-ALPHA)*input[i];

output[i] = output[i] - mem;

}

While the effect is less obvious, sometimes excessive amount of low
frequencies can cause sub-optimal encoding with Speex. As a general rule,
narrowband audio should be high-pass filtered at 300 Hz and wideband audio
should be filtered at 50 Hz. It is one of the planned features for Speex to do
the filtering automatically, this has not been done yet (as of November 2005).

##  V Recent Developments

Improvements to Speex have continued since the Speex bitstream was frozen with
the release of version 1.0. These improvements are included in the 1.1.x
unstable branch, which will lead to version 1.2. These are improvements to the
actual implementation as well as extra features outside of the codec itself.
It should be emphasised that the bitstream of version 1.2 will be exactly the
same as for version 1.0.

###  V-A Fixed-Point Port

Since November 2003 (version 1.1.1), work has been going on to convert Speex
to fixed-point. This effort has been pushing Speex into embedded devices.

While the port is not yet complete (as of November 2005), a large part of the
code has now been converted. Because the fixed-point code is implemented using
arithmetic operator abstraction (macros), using Speex on a fixed-point
architecture is as simple as configuring with the –enable-fixed-point option
or defining the FIXED_POINT macro. As of November 2005, the status of the
different components of Speex regarding fixed-point are as follows:

  * •

Narrowband: Completely converted for all constant bit-rates ranging from 3.95
kbps to 18.2 kbps (inclusively). Other bit-rate and narrowband options have
only a few float operations left (float emulation is realistic)

  * •

Wideband: Mostly converted to fixed-point, but some float operations are left
(float emulation is realistic on a fast chip).

  * •

Preprocessor: Not converted to fixed-point yet.

  * •

Echo canceller: Fixed-point port planned in the near future.

Using the fixed-point port, Speex is known to run on the following fixed-point
architectures:

  * •

Blackfin ($\mu$Clinux)

Speex now includes architecture-dependent optimisations for the Analog Devices
Inc. Blackfin DSP. It is possible to develop with Speex on Blackfin using
$\mu$Clinux444http://blackfin.uclinux.org/, GCC and
Linphone555http://www.linphone.org/ on the GPL-licensed STAMP
board666http://blackfin.uclinux.org/projects/stamp/.

  * •

ARM (Linux, Symbian)

Speex includes architecture-dependent optimisations for the ARM v4 and ARM v5E
architectures. The GCC compiler can be used to compile Speex on ARM.

  * •

TI C55x, C54x, C6x.

Speex has been adapted to the “non-conventional” architecture of these TI DSP
families. There are unfortunately no architecture-dependent optimisations yet
and (to the author’s knowledge) no Free compiler available.

Other fixed-point architectures may also be able to run Speex simply by
enabling fixed-point compilation.

###  V-B Preprocessor

A preprocessor module was introduced early in the 1.1.x branch. It is still
under development, but many developers have already used it successfully. The
preprocessor is designed to be used on the audio _before_ running the encoder
and provides three functionalities:

  * •

noise suppression (denoiser)

  * •

automatic gain control (AGC)

  * •

voice activity detection (VAD)

The denoiser can be used to reduce the amount of background noise present in
the input signal. This provides higher quality speech whether or not the
denoised signal is encoded with Speex (or at all). However, when using the
denoised signal with the codec, there is an additional benefit. Speech codecs
in general (Speex included) tend to perform poorly on noisy input and tend to
amplify the noise. The denoiser greatly reduces this effect.

Automatic gain control (AGC) is a feature that deals with the fact that the
recording volume may vary by a large amount between different hardware and
software setups. The AGC provides a way to adjust a signal to a reference
volume. This is useful for voice over IP because it removes the need for
manual adjustment of the microphone gain. A secondary advantage is that by
setting the microphone gain to a conservative (low) level, it is easier to
avoid clipping.

The voice activity detector (VAD) provided by the preprocessor is more
advanced than the one directly provided in the codec. Unfortunately, it cannot
directly be used by the encoder yet.

###  V-C Acoustic Echo Cancellation

The Speex library now includes an echo cancellation algorithm suitable for
Acoustic Echo Cancellation (AEC). AEC is especially useful in applications
(VoIP or others) where one records from a microphone while speakers are being
used. Speex uses an implementation of the multidelay block frequency (MDF)
algorithm [6]. Again, the algorithm is still under development, but has
already been successfully used in some applications.

Echo cancellation algorithms are generally very dependent on the quality and
characteristics of the processed signals. The Speex AEC is no exception and
there are several things that may prevent the echo canceller from working
properly, some of which are non-obvious to people with no signal processing
background. While it is always possible to encounter a bug in the Speex AEC,
there are many other options that should be considered before.

First, using a different soundcard to do the capture and playback will not
work, regardless of what some people think. The only exception to that is if
the two cards can be made to have their sampling clock “locked” on the same
source. However, this feature is only supported on professional soundcards.

The delay between the record and playback signals must be minimal. Any signal
played has to “appear” on the playback (far end) signal slightly before the
echo canceller “sees” it in the near end signal. However, excessive delay
means that part of the filter length is wasted. In the worst-case situation,
the delay is such that it is longer than the filter length, in which case, no
echo can be cancelled.

When it comes to echo tail length (filter length), longer is not necessarily
better. The longer the tail length, the longer it takes for the filter to
adapt. Of course, a tail length that is too short will not cancel enough echo,
but the most common problem seen is that people set a very long tail length
and then wonder why no echo is being cancelled (or why adaptation is very
slow).

Non-linear distortion cannot (by definition) be modelled by the linear
adaptive filter used in the echo canceller and thus cannot be cancelled. It is
thus best to use good audio hardware and avoid saturation/clipping at all
cost. Also, the AEC should be the first algorithm to process the microphone
signal and the last algorithm to see the speaker (far end) signal. A common
mistake when implementing echo cancellation is to use noise suppression or
automatic gain control before the AEC. Any processing, especially if it is
non-linear, should occur only after the echo cancellation algorithm.

##  VI Conclusion

In this paper, the origin and design goals of the Speex speech codec were
presented. Also, a description of the CELP algorithm and its implementation in
Speex was given. Some guidelines for programming with libspeex and choosing
the right encoding options were provided in order to help developers make
better use of Speex in applications.

Even though the bitstream itself has been frozen for more than two years, work
on Speex is continuing. Recent improvements focus on bringing support for
fixed-point CPU and DSP architectures as well as providing additional
functionality, such as echo cancellation and noise suppression, that are
useful for VoIP applications.

## Resources

Speex: http://www.speex.org

Speex mailing list<EMAIL_ADDRESS>
Xiph.Org Foundation: http://www.xiph.org

## Application support

Speex is supported in the following applications777Only open-source software
is listed here, in no particular order. This is not a complete list.:

  * •

OpenH323 (http://www.openh323.org/): Free implementation of the H323 protocol

  * •

Linphone (http://www.linphone.org/): SIP-based VoIP application, one of the
first application to support Speex

  * •

Gnomemeeting (http://www.gnomemeeting.org/): VoIP application based on
OpenH323 (with SIP support coming)

  * •

Sweep (http://www.metadecks.org/software/sweep/): Sound editor

  * •

Asterisk (http://www.asteriskpbx.org/): PBX Software

  * •

Annodex (http://www.annodex.net/): Open standard for annotating media

  * •

Ogg DirectShow (http://www.illiminable.com/ogg/): Windows DirectShow filters
for Speex, Vorbis, FLAC and Theora

  * •

JSpeex (http://jspeex.sourceforge.net/): Java port of Speex

  * •

Libfishsound (http://www.annodex.net/software/libfishsound/): Simple library
for encoding and decoding Ogg audio files

  * •

GStreamer (http://gstreamer.freedesktop.org/): Multimedia framework library

  * •

MPlayer (http://www.mplayerhq.hu/): Media player

  * •

Xine (http://xinehq.de/): Media player

  * •

VLC (http://http://www.videolan.org/vlc/): Media player and streaming server

  * •

SIP Express Media Server (http://www.iptel.org/sems/): Extensible media server
for voice services

## References

  * [1] M. Schroeder and B. Atal, “Code-excited linear prediction(celp): High-quality speech at very low bit rates,” in Proceedings IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 937–940, 1984.
  * [2] S. Andersen, W. Kleijn, R. Hagen, J. Linden, M. Murthi, and J. Skoglund, “iLBC-a linear predictive coder with robustness to packet losses,” in Proc. IEEE Speech Coding Workshop, pp. 23–25, 2002.
  * [3] P. Kroon and B. Atal, “Pitch predictors with high temporal resolution,” in Proceedings IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 661–664, 1990.
  * [4] J.-H. Chen, “Toll-quality 16 kb/s celp speech coding with very low complexity,” in Proceedings IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 9–12, 1995.
  * [5] J.-M. Valin and R. Lefebvre, “Bandwidth extension of narrowband speech for low bit-rate wideband coding,” in Proceedings IEEE Speech Coding Workshop, pp. 130–132, 2000.
  * [6] J.-S. Soo and K. Pang, “Multidelay block frequency domain adaptive filter,” IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 38, no. 2, pp. 373–376, 1990.


11institutetext:  Salzburg University of Applied Sciences 11email:
<EMAIL_ADDRESS>22institutetext: University of Salzburg
33institutetext: MedPhoton GmbH

# Multimodal Learning With Intraoperative CBCT & Variably Aligned Preoperative
CT Data To Improve Segmentation

Maximilian E. Tschuchnig 1122 0000-0002-1441-4752 Philipp Steininger 33
Michael Gadermayr 11 0000-0003-1450-9222

###### Abstract

Cone-beam computed tomography (CBCT), is an important tool facilitating
computer aided interventions, despite often suffering from artifacts that pose
challenges for accurate interpretation. While the degraded image quality can
affect downstream segmentation, the availability of high quality, preoperative
scans represents potential for improvements. Here we consider a setting where
preoperative CT and intraoperative CBCT scans are available, however, the
alignment (registration) between the scans is imperfect. We propose a
multimodal learning method that fuses roughly aligned CBCT and CT scans and
investigate the effect of CBCT quality and misalignment on the final
segmentation performance. For that purpose, we make use of a synthetically
generated data set containing real CT and synthetic CBCT volumes. As an
application scenario, we focus on liver and liver tumor segmentation. We show
that the fusion of preoperative CT and simulated, intraoperative CBCT mostly
improves segmentation performance (compared to using intraoperative CBCT only)
and that even clearly misaligned preoperative data has the potential to
improve segmentation performance.

###### Keywords:

Multimodal Learning Intraoperative Segmentation Radiology

##  1 Introduction

To establish computer-assisted interventions, precise and reliable imaging,
especially intraoperative imaging, is crucial. Mobile robotic medical imaging
systems, like cone-beam computed tomography (CBCT) [12], enable intraoperative
medical imaging with real time capabilities. CBCT is an imaging method that
utilizes a cone-shaped X-ray beam and a flat-panel detector to capture
detailed, three-dimensional images of a patient’s anatomy using a mobile
system [9]. However, this type of intraoperative imaging often comes with the
disadvantage suffering from more artifacts than preoperative CT imaging,
affecting the performance of downstream tasks like segmentation [15].

While this degraded image quality can affect downstream segmentation, the
availability of high quality preoperative scans represents a high potential
for improvements based on the idea of multimodal learning [17, 11, 16].
Multimodal learning is an approach that involves fusing images from multiple
domains to improve machine learning models for a downstream task like
segmentation. In medical imaging, a common approach is to enrich computed
tomograpy (CT) data, focusing on bone structures, with magnetic resonance
imaging data for soft tissue analysis [17, 11]. Multimodal learning is
typically separated into three fusion strategies [17, 16]: early, late and
hybrid. The most common multimodal fusion, early-fusion, combines images of
different modalities before being processed by a downstream model. Typically,
the two domains are fused along a dimension additional to the spacial volume
dimensions and processed jointly [17, 13]. Another form of early fusion
processes the volumes of different modalities in separate feature extraction
stages, finally fusing the extracted features. Late-fusion is performed before
the final layer of the downstream task. In a segmentation case, late-fusion
merges the features extracted from multiple independent encoder-decoder
networks before the final layer (facilitating segmentation). Hybrid-fusion,
combines aspects of both early and late fusion for enhanced performance.

Typically, multimodal learning assumes that the fused images are aligned,
utilizing affine or even elastic registration. However, registration of 3D
samples, if performed accurately on a high resolution, is computationally
expensive, especially if non-linear deformations need to be compensated. Deep
learning-based approaches [2, 4] have shown promising performance while also
reducing computational complexity, compared to classical, optimization based
approaches. Podobnik et al. [11] integrated registration within their hybrid
multimodal segmentation approach. Integrated registration was achieved by
merging features from different modality branches using the affine Spatial
Transformer Networks (STN) localization net [8], along with a grid generator
and sampler.

Here we consider the setting where preoperative CT scans and intraoperative
CBCT scans are available, however, the alignment (registration) between the
scans is imperfect. We hypothesised that, by adding high quality, preoperative
CT to intraoperative CBCT scans, segmentation performance will increase. We
assume that, the more accurate the alignment between the modalities is, the
more pronounced the performance increase will be.

Contribution:  We propose a method that combines roughly aligned CBCT and CT
scans (early-fusion) and investigate the effect of CBCT quality and
misalignment (based on affine and elastic transformations) on segmentation
performance.

In detail, we synthesised a collection of synthetic CBCT data, focusing on the
segmentation of liver and liver tumors based on the LiTS CT dataset [3].
Beyond varying the amount of digitally reconstructed radiographs (DRR) used
for CBCT synthesis, resulting in $5$ different imaging qualities, we generated
$9$ variably misaligned versions based on linear and non-linear models and $1$
baseline model resulting in overall $50$ sub data sets. We evaluated all $45$
combinations based on a unet architecture and compared the performances to the
$5$ unimodal baseline settings.

##  2 Methodology

We propose to fuse intraoperative CBCT with roughly aligned, high quality,
preoperative CT to investigate if this kind of additional information improves
model training for computer aided intervention systems. To accomplish this
study we apply multimodal learning in medical imaging for the downstream task
of liver and liver tumor segmentation using CT and CBCT data based on the LiTS
dataset [3]111For the CBCT version of LiTS see the Kaggle dataset CBCT Liver
and Liver Tumor Segmentation Train Data. Similar to Ren et al. [13] we used
early-fusion for the multimodal setup. Furthermore, we investigated two
parameters, $\alpha_{np}$ as the number of DRRs used to simulate the current
CBCT (representing undersampling), and $\alpha_{a}$ as the current alignment
factor between the preoperative CT and the intraoperative CBCT. As a baseline,
segmentation was performed using only the intraoperative CBCT scans, which was
evaluated for all settings of $\alpha_{np}$. Applied to the task of liver and
liver segmentation, this setup results in $100$ different experiments.
Misalignment was performed using affine transformations during training and
validation to decrease computational complexity in the dataloader. During
validation, which is only performed once for each sample, affine misalignment
was followed by elastic misalignment to evaluate the models ability to
generalize to non-linear deformations.

Figure 1: Multimodal model configuration. After fusing the intraoperative CBCT
and preoperative CT (early fusion), the data was processed by the shown unet,
segmenting liver and liver tumors.

We used a holistic, 3D unet introduced by Çiçek et al. [5] as the basis of our
segmentation model and our baseline for all investigated settings. The 3D unet
was updated to process the multimodal data by adding a paired and variably
misaligned, preoperative CT as a second channel, resulting in a 4d data
structure. The 3D unet used for segmentation, shown in Fig. 1, consisted of an
encoder with 3 double convolution layers and $3\times 3\times 3$ convolutional
kernels, connected by 3D max pooling. The latent space was constructed using
one double convolution block followed by the unet decoder, mirroring the
encoder. As is typical for unet, each double convolutional output in the
encoder was also connected to the decoder double convolutional block of the
same order. Additionally, one 3D convolutional layer was added to the decoder
with a filter size of $1\times 1\times 1$ and the number of filters set to the
number of segmentation classes (in the case of liver and liver tumor
segmentation this value was set to $2$). The number of feature maps were set
to $\\{32,64,128,256\\}$ as shown in Fig. 1. Batch norm was applied after each
layer in the double convolutional blocks. The model was trained utilizing a
sum of binary cross-entropy and Dice similarity. For our baseline, a unimodal
unet was used, with only the CBCT as input. Our multimodal approach added the
high quality, misaligned and preoperative CT as a second channel to the CBCT.

###  2.1 Experimental Details

All models were trained on an Ubuntu server using NVIDIA RTX A6000 graphics
cards. Due to the large size of the data and memory restrictions (48 GB VRAM),
the volumes were downscaled (isotropic) by the factor of two. To binarize the
masks, a threshold of $0.5$ was applied to each channel of the unet output. As
an additional baseline, CBCT volumes with perfectly aligned CT volumes,
$\alpha_{a}=0$, was also investigated. All experiments were trained and
evaluated $4$ times to facilitate stable results with the same random splits
as well as the same random CT misaligned for comparable result. Adam was used
as an optimizer with a learning rate of $0.005$. The source code for the
experiments can be found on https://github.com/blind/blind. Since annotations
were not available for the LiTS test dataset, the test dataset was disregarded
for this publication and the (processed) LiTS training set was separated into
training-validation-testing data [1, 6]. The separation was performed using
the ratios $0.7$ (training), $0.2$ (validation), $0.1$ (testing).

###  2.2 Dataset Generation

For evaluation of the multimodal learning approach, paired CT and CBCT volumes
with variable degrees of misalignment had to be generated.

Figure 2:  Data generation process: after centering the original CT volumes
around the liver (using the liver segmentations), $\alpha_{np}$ projections
were simulated. Finally, CBCT were simulated and aligned with the original CT
and masks, in order to fit them to the CBCT field-of-view.

To generate CBCT/CT pairs we simulated DRRs from the CT volumes, with variable
undersampling (with $\alpha_{np}\in\\{32,64,128,256,490\\}$ representing the
predefined number of DRRs). We then used these DRRs to simulate CBCT with
varying visual quality. Higher $\alpha_{np}$ corresponded to better image
quality, with $\alpha_{np}=490$ serving as the CBCT quality similar to
preoperative CT and $\alpha_{np}=32$ as the lowest visual quality with
significant artifacts [14]. Fig. 2 shows these steps to convert the CT LiTS
dataset into synthetic CBCT scans.

The LiTS dataset was chosen to perform the experiments [3]. LiTS consists of
$131$ abdominal CT scans in the training set and $70$ test volumes. The $131$
training volumes include segmentations of 1) the liver and 2) liver tumors.
The dataset contains data from $7$ different institutions with a diverse set
of liver tumor diseases. The CT scans were acquired using different CT
scanners and acquisition protocols. For further information about the dataset
we refer to Bilic et al. [3]222To download the LiTS dataset follow the link:
https://competitions.codalab.org/competitions/17094.

Affine misaligned was performed using random (non-isotropic) scaling, with the
scaling parameter sampled from
$\mathcal{U}(1-0.5\cdot\alpha_{a},1+0.5\cdot\alpha_{a})$, rotation, parameter
sampled from $\mathcal{U}(-22.5\cdot\alpha_{a},22.5\cdot\alpha_{a})$, and
translation, with the parameter sampled from
$\mathcal{U}(0,0.5\cdot\alpha_{a})$ with tri-linear interpolation. Elastic
misalignment was applied with a maximum displacement sampled from
$\mathcal{U}(0,20\cdot\alpha_{a})$ with $7$ control points and no locked
borders. To reduce the number of parameters controlling misalignment
parameters to one, the alignment factor
$\alpha_{a}\in\\{0,0.125,0.25,0.5,1\\}$ was introduced, controlling the
misalignmnet between pairs of scans. Augmentation was performed using TorchIO
RandomAffine and RandomElasticDeformation.

Fig 3 shows examples of affine misalignment with the
$\alpha_{a}\in\\{0,0.125,0.25,0.5,1\\}$ and $4$ different volumes. Fig 4 shows
the same volumes, with applied random elastic deformation. Here we display the
elastic transformation in isolation from affine transformations for improved
visibility.

Figure 3: Results of the random affine augmentation with differing
augmentation factor $\alpha_{a}\in\\{0,0.125,0.25,0.5,1\\}$ of $4$ different
volumes. Figure 4: Results of the random elastic augmentation with differing
augmentation factor $\alpha_{a}\in\\{0,0.125,0.25,0.5,1\\}$ of $4$ different
volumes.

##  3 Results

Experimental results are shown in Table 1, denoted as mean Dice values. If the
test data was transformed using the affine transformations, rows are marked as
affine-s. Affine, followed by the elastic transformed settings are marked as
elastic-s. Two baselines are also shown: unimodal CBCT with no preoperative CT
(base CBCT) and CBCT with perfectly aligned CT (no misalignment). Blue values
show improvements compared to the baseline (base CBCT), while red values show
decreased scores. If the values are bold there is at least a $5\%$
increase/decrease. The $+,-$ corresponds to improvements/decreases to the
previous, less accurately aligned experiments. If $+,-$ are colored, the
increase/decrease compared to the previous less accurately aligned example is
larger than $5\%$. The different undersampled $\alpha_{np}$ are displayed
horizontally and $\alpha_{a}$ vertically for both liver and liver tumor
segmentation. In the affine setting, the scores increased compared to the
baseline in all liver segmentation and in $11$ out of $20$ liver tumor
segmentation cases. The biggest improvement for liver segmentation was
achieved with the parameters $\alpha_{np}=32$ and $\alpha_{a}=0$ with an
improvement from $0.784$ to $0.932$ resulting in an increase of $0.148$. For
liver tumor segmentation, the biggest improvement was also achieved with the
parameters $\alpha_{np}=32$ and $\alpha_{a}=0$ from $0.029$ to $0.297$
resulting in an increase of $0.268$. There was also an observable trend of
increasing Dice scores from $\alpha_{a}\in\\{0.5,0.25,0.125,0\\}$. This trend
was stronger, the lower $\alpha_{np}$ was. Adding elastic transformation on
top of affine transformations lead to slightly lower scores in the cases of
$\alpha_{a}\in{0.5,0.25}$ with $\alpha_{a=0.25}$ suffering most from adding
elastic transformations with an average Dice decrease of $0.023$ for liver and
$0.032$ for liver tumor segmentation. The elastic transformation effects
regarding $\alpha_{np}$ show a decrease of average Dice proportional to image
quality. The most significant decrease in Dice, attributable to elastic
transformation was reached with $\alpha_{np}=32$ with $0.016$ for liver and
$0.021$ for liver tumor segmentation.

Table 1: Experimental results (affine and elastic misalignment) showing the
mean Dice values. Blue values denote improvements compared to the baseline
(base CBCT), while red values show decreased scores. If the values are in bold
there is at least a $5\%$ increase/decrease. The $+,-$ corresponds to
increases/decreases compared to the previous, less accurately aligned setting.
If $+,-$ are colored, this increase/decrease is larger than $5\%$.

| Liver Segmentation | Liver Tumor Segmentation
---|---|---
| 490 |  | 256 |  | 128 |  | 64 |  | 32 |  | 490 |  | 256 |  | 128 |  | 64 |  | 32 |
base CBCT | 0.884 |  | 0.884 |  | 0.859 |  | 0.817 |  | 0.784 |  | 0.165 |  | 0.162 |  | 0.093 |  | 0.061 |  | 0.029 |
no misalignment | 0.933 | $\bm{+}$ | 0.931 | $\bm{+}$ | 0.931 | $\bm{+}$ | 0.933 | $\bm{+}$ | 0.932 | $\bm{+}$ | 0.330 | $\color[rgb]{0,0,1}\bm{+}$ | 0.322 | $\color[rgb]{0,0,1}\bm{+}$ | 0.298 | $\color[rgb]{0,0,1}\bm{+}$ | 0.325 | $\color[rgb]{0,0,1}\bm{+}$ | 0.297 | $\color[rgb]{0,0,1}\bm{+}$
affine-s1 | 0.906 | $\bm{+}$ | 0.897 | $\bm{+}$ | 0.879 | $\bm{+}$ | 0.857 | $\bm{+}$ | 0.806 | $\bm{+}$ | 0.154 | $\bm{-}$ | 0.124 | $\bm{-}$ | 0.057 | $\bm{-}$ | 0.023 | $\bm{-}$ | 0.051 | $\bm{+}$
affine-s0.5 | 0.895 | $\bm{-}$ | 0.891 | $\bm{-}$ | 0.875 | $\bm{-}$ | 0.863 | $\bm{+}$ | 0.851 | $\bm{+}$ | 0.096 | $\color[rgb]{1,0,0}\bm{-}$ | 0.098 | $\bm{-}$ | 0.077 | $\bm{+}$ | 0.056 | $\bm{+}$ | 0.107 | $\color[rgb]{0,0,1}\bm{+}$
affine-s0.25 | 0.904 | $\bm{+}$ | 0.906 | $\bm{+}$ | 0.893 | $\bm{+}$ | 0.883 | $\bm{+}$ | 0.883 | $\bm{+}$ | 0.181 | $\color[rgb]{0,0,1}\bm{+}$ | 0.160 | $\color[rgb]{0,0,1}\bm{+}$ | 0.163 | $\color[rgb]{0,0,1}\bm{+}$ | 0.169 | $\color[rgb]{0,0,1}\bm{+}$ | 0.171 | $\color[rgb]{0,0,1}\bm{+}$
affine-s0.125 | 0.908 | $\bm{+}$ | 0.900 | $\bm{-}$ | 0.889 | $\bm{-}$ | 0.884 | $\bm{+}$ | 0.887 | $\bm{+}$ | 0.187 | $\bm{+}$ | 0.166 | $\bm{+}$ | 0.166 | $\bm{+}$ | 0.152 | $\bm{-}$ | 0.185 | $\bm{+}$
elastic-s1 | 0.907 | $\bm{+}$ | 0.897 | $\bm{+}$ | 0.880 | $\bm{+}$ | 0.857 | $\bm{+}$ | 0.081 | $\bm{+}$ | 0.153 | $\bm{-}$ | 0.122 | $\bm{-}$ | 0.057 | $\bm{-}$ | 0.022 | $\bm{-}$ | 0.049 | $\bm{+}$
elastic-s0.5 | 0.895 | $\bm{-}$ | 0.891 | $\bm{-}$ | 0.870 | $\bm{-}$ | 0.845 | $\bm{-}$ | 0.800 | $\bm{+}$ | 0.098 | $\color[rgb]{1,0,0}\bm{-}$ | 0.103 | $\bm{-}$ | 0.085 | $\bm{+}$ | 0.029 | $\bm{+}$ | 0.049 | $\bm{+}$
elastic-s0.25 | 0.893 | $\bm{-}$ | 0.898 | $\bm{+}$ | 0.869 | $\bm{-}$ | 0.853 | $\bm{+}$ | 0.846 | $\bm{+}$ | 0.154 | $\bm{+}$ | 0.142 | $\color[rgb]{0,0,1}\bm{+}$ | 0.136 | $\color[rgb]{0,0,1}\bm{+}$ | 0.129 | $\color[rgb]{0,0,1}\bm{+}$ | 0.122 | $\color[rgb]{0,0,1}\bm{+}$
elastic-s0.125 | 0.910 | $\bm{+}$ | 0.901 | $\bm{+}$ | 0.891 | $\bm{+}$ | 0.885 | $\bm{+}$ | 0.887 | $\bm{+}$ | 0.186 | $\bm{+}$ | 0.168 | $\bm{+}$ | 0.163 | $\bm{+}$ | 0.151 | $\bm{+}$ | 0.168 | $\bm{+}$

##  4 Discussion

The results confirmed the hypothesis that enriching intraoperative CBCT with
roughly aligned preoperative CT can improve downstream tasks like
segmentation. Most multimodal setups improved downstream performance with the
only outliers observable in liver tumor segmentation with
$\alpha_{a}\in\\{0.5,1\\}$ and $\alpha_{np}\in\\{490,256,128,64\\}$ and
$(\alpha_{a},\alpha_{np})=(0.25,256)$ in the affine misaligned cases. If data
was additionally misaligned through elastic transformations, the settings
$(\alpha_{a},\alpha_{np})=(0.25,490)$ also lead to a decrease in average Dice.

Several trends are observable. For one, the worse the CBCT quality, the more
can be gained by adding high quality CT, leading to an increase of Dice from
$0.784\rightarrow 0.932$ for liver and $0.029\rightarrow 0.297$ for liver
tumor segmentation. There is a particularly pronounced effect in the case of
adding preoperative information to low-quality interoperative CBCT. This
effect is especially pronounced in the case of liver segmentation, where
baseline scores can be reached using $\alpha_{np}=32$.

We also observe a trend regarding alignment of preoperative CT and
intraoperative CBCT. At first, increased alignment lead to mostly worse
average Dice for $\alpha_{a}\in\\{1,0.5\\}$ followed by a notable increase in
average Dice for $\alpha_{a}\in\\{0.25,0.125,0\\}$. This trend was only
observed for liver tumor segmentation. We hypothesise that this is the case
due to heavily misaligned data interfering with the training process. This is
especially the case for complex structures like tumors. Here, the model can
easily learn to ignore the higher quality but heavily misaligned CT but
overfit to better aligned CT data. However, further experiments to determine
the reason for this should be performed. For now our findings suggest that if
there is significant misalignment and the target is difficult, such as liver
tumor segmentation, preregistration is crucial for successful multimodal
learning.

The addition of elastic misalignment lead to a proportional decrease in
downstream segmentation performance depending on the undersampling
$\alpha_{np}$. Therefore, the same elastic misalignment had more pronounced
degrading effects on segmentation performance the worse the image quality got,
hinting at a possible implicit registration effect that diminishes with image
quality. Investigating the factor $\alpha_{a}$, elastic misalignment had no
effect in the cases of $\alpha_{a}\in\\{1,0.125\\}$ with the average change
$<0.005$. However, in the cases of $\alpha_{a}\in\\{0.5,0.25\\}$ the effect of
adding elastic misalignment was on average a decrease of $0.021$. These
insights hint at the deformation of $\alpha_{a}=1$ already being almost
unfeasible to handle in the affine case, leaving the addition of elastic
transform in these cases without effect. Since we also observed almost no
effect in the cases of $\alpha_{a}=0.125$ we suspected that in these cases the
elastic transform was negligible, in comparison to the affine misalignment.
Interestingly, these performance gains show that the unet used for
segmentation implicitly learned limited registration.

Although we only evaluated our results on unet, this form of multimodal
learning is theoretically applicable to segmentation models based on similar
building blocks, most importantly convolutional filters and pooling. Further
experiments are needed to investigate this assumption for other architectures,
such as the Segment Anything Model [10] or UNETR [7].

##  5 Conclusion

This study highlights the effectiveness of multimodal learning for downstream
tasks, combining roughly aligned intraoperative CBCT with high quality,
preoperative CT data. It further investigates how the different factors of
volume quality and volume alignment influence the performance of a specific
multimodal learning based model. Using the multimodal learning setup,
improvements in segmentation accuracy, especially when CBCT volume quality was
suboptimal, were reported suggesting that high-quality preoperative CT data
can compensate for intraoperative CBCT limitations, as long as the data is
roughly aligned. Therefore, using this approach, clinicians could be supplied
with more reliable information for surgical decision-making, particularly in
near real-time settings of computer-assisted intervention. Assuming there is
an efficient way for rough preregistration, this underscores the practical
applicability of the approach. We further showed that a simple 3D unet was
able to learn limited, implicit registration.

Since evaluation was based on synthetic data with relatively simple
misalignment and without preregistration useful next steps would be to
evaluate using real, paired preoperative and intraoperative data as well as
incorporating (pre)registration into the multimodal model. Furthermore,
continued exploration into multimodal approaches, including late and hybrid
fusion, are promising future research directions.

####  5.0.1 Acknowledgements

This project was partly funded by the Austrian Research Promotion Agency (FFG)
under the bridge project "CIRCUIT: Towards Comprehensive CBCT Imaging
Pipelines for Real-time Acquisition, Analysis, Interaction and Visualization"
(CIRCUIT), no. 41545455 and by the county of Salzburg under the project AIBIA.

####  5.0.2

The authors have no competing interests to declare that are relevant to the
content of this article.

## References

  * [1] Araújo, J.D.L., da Cruz, L.B., Diniz, J.O.B., Ferreira, J.L., Silva, A.C., de Paiva, A.C., Gattass, M.: Liver segmentation from computed tomography images using cascade deep learning. Computers in Biology and Medicine 140, 105095 (2022)
  * [2] Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: Voxelmorph: a learning framework for deformable medical image registration. IEEE transactions on medical imaging 38(8), 1788–1800 (2019)
  * [3] Bilic, P., Christ, P., Li, H., Vorontsov, E., Ben-Cohen, A., Kaissis, G., others, Menze, B.: The liver tumor segmentation benchmark (lits). Medical Image Analysis 84 (Feb 2023). https://doi.org/10.1016/j.media.2022.102680
  * [4] Chen, J., Frey, E.C., He, Y., Segars, W.P., Li, Y., Du, Y.: Transmorph: Transformer for unsupervised medical image registration. Medical image analysis 82, 102615 (2022)
  * [5] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19. pp. 424–432 (2016)
  * [6] Han, K., Liu, L., Song, Y., Liu, Y., Qiu, C., Tang, Y., Teng, Q., Liu, Z.: An effective semi-supervised approach for liver ct image segmentation. IEEE Journal of Biomedical and Health Informatics 26(8), 3999–4007 (2022)
  * [7] Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H.R., Xu, D.: Unetr: Transformers for 3d medical image segmentation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 574–584 (2022)
  * [8] Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. Advances in neural information processing systems 28 (2015)
  * [9] Jaffray, D.A., Siewerdsen, J.H., Wong, J.W., Martinez, A.A.: Flat-panel cone-beam computed tomography for image-guided radiation therapy. International Journal of Radiation Oncology* Biology* Physics 53(5), 1337–1349 (2002)
  * [10] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023)
  * [11] Podobnik, G., Strojan, P., Peterlin, P., Ibragimov, B., Vrtovec, T.: Multimodal ct and mr segmentation of head and neck organs-at-risk. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 745–755 (2023)
  * [12] Rafferty, M.A., Siewerdsen, J.H., Chan, Y., Daly, M.J., Moseley, D.J., Jaffray, D.A., Irish, J.C.: Intraoperative cone-beam ct for guidance of temporal bone surgery. Otolaryngology—Head and Neck Surgery 134(5), 801–808 (2006)
  * [13] Ren, J., Eriksen, J.G., Nijkamp, J., Korreman, S.S.: Comparing different ct, pet and mri multi-modality image combinations for deep learning-based head and neck tumor segmentation. Acta Oncologica 60(11), 1399–1406 (2021)
  * [14] Tschuchnig, M.E., Coste-Marin, J., Steininger, P., Gadermayr, M.: Multi-task learning to improve semantic segmentation of cbct scans using image reconstruction. In: BVM Workshop. pp. 243–248 (2024)
  * [15] Wei, C., Albrecht, J., Rit, S., Laurendeau, M., Thummerer, A., Corradini, S., others, Landry, G.: Reduction of cone-beam ct artifacts in a robotic cbct device using saddle trajectories with integrated infrared tracking. Medical Physics (2024)
  * [16] Zhang, Y., Yang, J., Tian, J., Shi, Z., Zhong, C., Zhang, Y., He, Z.: Modality-aware mutual learning for multi-modal medical image segmentation. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24. pp. 589–599 (2021)
  * [17] Zhang, Y., Sidibé, D., Morel, O., Mériaudeau, F.: Deep multimodal fusion for semantic image segmentation: A survey. Image and Vision Computing 105, 104042 (2021)