a4paper, textwidth=18cm, textheight=26cm, heightrounded, hratio=1:1,
vratio=2:3,

Conceptualization; Methodology; Resources; Investigation; Writing original
draft; Writing, review, and editing

Conceptualization; Methodology; Resources; Investigation; Writing original
draft; Writing, review, and editing

[1]The first is the corresponding author.

# Advanced Artificial Intelligence Algorithms in Cochlear Implants: Review of
Healthcare Strategies, Challenges, and Perspectives

Billel Essaid<EMAIL_ADDRESS>Hamza Kheddar<EMAIL_ADDRESS>Noureddine Batel<EMAIL_ADDRESS>Abderrahmane Lakas<EMAIL_ADDRESS>Muhammad E. H. Chowdhury<EMAIL_ADDRESS>LSEA Laboratory, Department of
Electrical Engineering, University of Medea, 26000, Algeria  College of
Information Technology, United Arab Emirates University, Al Ain P.O. Box
17551, United Arab Emirates  Department of Electrical Engineering, Qatar
University, Doha 2713, Qatar

###### Abstract

Automatic speech recognition (ASR) plays a pivotal role in our daily lives,
offering utility not only for interacting with machines but also for
facilitating communication for individuals with either partial or profound
hearing impairments. The process involves receiving the speech signal in
analogue form, followed by various signal processing algorithms to make it
compatible with devices of limited capacity, such as cochlear implants (CIs).
Unfortunately, these implants, equipped with a finite number of electrodes,
often result in speech distortion during synthesis. Despite efforts by
researchers to enhance received speech quality using various state-of-the-art
signal processing techniques, challenges persist, especially in scenarios
involving multiple sources of speech, environmental noise, and other
circumstances. The advent of new artificial intelligence (AI) methods has
ushered in cutting-edge strategies to address the limitations and difficulties
associated with traditional signal processing techniques dedicated to CIs.
This review aims to comprehensively review advancements in CI-based ASR and
speech enhancement, among other related aspects. The primary objective is to
provide a thorough overview of metrics and datasets, exploring the
capabilities of AI algorithms in this biomedical field, summarizing and
commenting on the best results obtained. Additionally, the review will delve
into potential applications and suggest future directions to bridge existing
research gaps in this domain.

###### keywords:

Profound hearing loss Postoperative cochlear implant Speech therapy
rehabilitation Cochlear implant Automatic speech recognition Machine learning

## Acronyms and Abbreviations

CNN

    convolutional neural network
AI

    artificial intelligent
DL

    deep learning
CI

    cochlear implant
ML

    machine learning
FCN

    fully convolutional neural networks
STOI

    short time objective intelligibility
MMSE

    minimum mean square error
DDAE

    deep denoising auto-encoder
SNR

    signal to noise ratio
EAS

    electric and acoustic stimulation
CGRU

     convolutional recurrent neural network with gated recurrent units
CWT

     continuous wavelet transform
GRU

    gated recurrent units
IGCIP

    image guided cochlear implant programming
CT

    computed tomography
OCT

    optical coherence tomography
MEE

    mean endpoint error
TMHINT

    taiwan mandarin hearing in noise test
Acc

    Accuracy
Sen

    Sensitivity
Spe

    Specificity
Pre

    Precision
Rec

    Recall
F1

    F1 score
NCM

    normalized covariance measure
FFT

    fast Fourier transform
ACE

    advanced combination encoder
LGF

    loudness growth function
MMS

    min-max similarity
BF

    boundary F1
ROC

     receiver operating characteristic curve
NCM

    normalized covariance measure
DCS

    dice coefficient similarity
ASD

    average surface distance
AVD

    average volume difference
JI

    jaccard index
IoU

    intersection over union
MAE

    mean absolute error
HD

    hausdorff distance
VAE

    variational autoencoders
CAE

    convolutional autoencoder
AE

    autoencoder
SOTA

    state-of-the-art
MFCC

    mel-frequency cepstral coefficient
GFCC

    gammatone frequency cepstral coefficient
AMR

    adaptive multi-rate
LSTM

    long short term memory
ASR

    automatic speech recognition
DNN

    deep neural networks
CS

    channel selection
RNN

    recurrent neural network
GAN

    generative adversarial network
PESQ

     perceptual estimation of speech quality
KLT

    karhunen-loéve transform
logMMSE

    log minimum mean squared error
DAE

    deep autoencoder
ICA

    intra cochlear anatomy
ST

    scala tympani
SV

    scala vestibul
AR

    active region
ASM

    active shape model
NR

    noise reduction
Res-CA

    residual channel attention
GCPFE

    global context-aware pyramid feature extraction
ACE-Loss

    active contour with elastic loss
DS

    deep supervision
UHR

    ultra-high-resolution
ESCSO

    enhanced swarm-based crow search optimization
cGAN

    conditional generative adversarial networks
MAR

    metal artifacts reduction
P2PE

    point to point error
ASE

    average surface error
MARGAN

    metal artifact reduction based generative adversarial networks
ESCSO

     enhanced swarm based crow search optimization
EEG

    electroencephalography
ERP

    event-related potential
ANN

    artificial neural network
RBNN

    radial basis functions neural networks
KNN

    K-nearest neighbours
RF

    random forests
DRNN

    deep recurrent neural networks
MLP

    multilayer perceptrons
NMF

    non-negative matrix factorization
DCAE

    deep convolutional auto-encoders
ILD

    interaural level difference
RTF

    relative transfer function
SDR

    source-to-distortion ratio
SAR

    source-to-artifact ratio
SIR

    source-to-interference ratio
dB

    decibel
MIMO

    multiple-input multiple-output
EVD

    eigenvector decomposition
ReLU

    rectified linear unit
RNN

    recurrent neural network
MICE

    multiple imputation by chained equations
ECochG

    intracochlear electrocochleography
CM

    cochlear microphonic
FFR

    frequency following responses
$f_{0}$

    fundamental frequency
TFSC

    temporal fine structure cues
RMSE

    root mean square error
BM

    basilar-membrane
ROI

    regions of interest
DRL

    deep reinforcement learning
RL

    reinforcement learning
ABR

    auditory brainstem responses
DBS

    deep brain stimulation
FOX

    fitting to outcomes expert
DSP

    digital signal processor
MHINT

    mandarin of hearing in noise test
BCP

    bern cocktail party
EST

    effective stimulation threshold
ECAP

    electrically evoked compound action potential
MRI

    magnetic resonance imaging
BSS

    blind source separation
AEP

    auditory evoked potential
CCA

    canonical correlation analysis
ASE

    average surface error
STOI

    short-time objective intelligibility
TL

    transfer learning
CIS

    continuous interleaved sampling
DTL

    deep transfer learning
FL

    federated learning
CTC

    connectionist temporal classification
BERT

    bidirectional encoder representations from transformer

##  1 Introduction

In the symphony of modern technology, automatic speech recognition (ASR)
emerges as a master, orchestrating a seamless interaction between humans and
machines. This transformative technology has quietly become an integral part
of our daily lives, influencing how we communicate, access information, and
even navigate the intricacies of healthcare. The significance of ASR extends
beyond its role in facilitating human-computer interaction; it permeates
diverse applications such as voice assistants and virtual agents, speech-to-
text conversion, identity verification, and holds particular promise in the
realm of biomedical research [1, 2]. ASR bridges the gap between spoken
language and digital communication, enabling the conversion of spoken words
into written text with remarkable accuracy. The pervasiveness of ASR
technology is evident in the devices we use daily—smartphones, smart speakers,
and voice-activated virtual assistants—all seamlessly responding to our spoken
commands and queries. The convenience it brings to our lives is undeniable,
offering a hands-free and efficient mode of interaction that has become second
nature.

ASR is pivotal in authentication systems, safeguarding the security and
privacy of sensitive information. The integrity of audio speech can be
verified through techniques-based ASR such as adversarial attacks detections
[3], steganalysis [4, 5, 6], speech biometric [7] and more. Beyond the realm
of communication, ASR finds itself at the heart of various applications, each
playing a unique role in different domains. Speaker recognition, a facet of
ASR, is not merely confined to enhancing security measures. It has evolved
into a versatile tool employed in healthcare, where the identification of
individuals through their unique vocal signatures holds promise for
personalized patient care. This is particularly relevant in scenarios where
quick and secure authentication is crucial, such as accessing medical records
or authorizing medical procedures. Event recognition, another dimension of
ASR, is a game-changer in sectors ranging from security to healthcare. In the
former, ASR algorithms analyze audio data to automatically detect and
categorize specific events, reinforcing surveillance capabilities. In
healthcare, event recognition becomes a powerful tool for monitoring and early
detection of health-related events. For instance, in the context of cardiac
health, ASR can aid in identifying anomalies in heart sounds, potentially
enabling early intervention and preventive measures [8]. Source separation,
the ability to discern and isolate individual sound sources from complex audio
signals, is a boon in fields like entertainment and music production. However,
its significance extends into the realm of biomedical research, where ASR
plays a pivotal role in decoding the intricate language of physiological
signals. In the context of cochlear implants, source separation becomes a
critical component in enhancing the auditory experience for individuals with
hearing impairments.

The CI are designed to restore hearing in individuals with severe hearing loss
or deafness, rely on ASR for optimizing their functionality. ASR contributes
significantly to the improvement of speech perception in CI users by enhancing
the processing and interpretation of auditory signals. CI work by converting
sound waves into electrical signals that stimulate the auditory nerve,
bypassing damaged parts of the inner ear. ASR complements this process by
aiding in the recognition and translation of spoken language. The technology
plays a crucial role in optimizing speech understanding for CI users by
refining the interpretation of varied speech patterns, tones, and nuances.
Moreover, ASR in the context of CI extends beyond basic speech recognition. It
contributes to the recognition of environmental sounds, facilitating a more
immersive auditory experience for individuals with hearing impairments. This
is particularly significant in enhancing the quality of life for CI
recipients, allowing them to navigate and engage with their surroundings more
effectively.

###  1.1 Related work

Many reviews have been written in the context of CI. For example, [9]
discussed the advantages offered by machine learning (ML) to cochlear
implantation, such as analyzing data to personalize treatment strategies. It
enhances accuracy in speech processing optimization, surgical anatomy location
prediction, and electrode placement discrimination. Besides, it delves into
its applications, including optimizing cochlear implant fitting, predicting
patient threshold levels, and automating image-guided CI surgery. The review
discusses some open novel opportunities for research, emphasizing the need for
high-quality data inputs and addressing concerns about algorithm transparency
in clinical decision-making for improved patient care. Similarly, the review
written by Manero et al. [10] details some benefits of employing artificial
intelligent (AI) in enhancing CI technology, involving adaptive sound
processing, acoustic scene classification, and auditory scene analysis. The
authors discuss AI-driven advancements aiming to optimize sound signals, adapt
to diverse environments, and improve speech perception for individuals with
hearing loss, ultimately enhancing their overall quality of life. Also, the
review [11] explores three main topics: direct-speech neuroprosthesis, which
involves decoding speech from the sensorimotor cortex using AI, including the
synthesis of produced speech from brain activity. The second topic is a top-
down exploration of pediatric cochlear implantation using ML, delving into the
applications of ML in pediatric cochlear implantation. Lastly, the paper
considers the potential of AI to solve the hearing-in-noise problem, examining
its capabilities in addressing challenges related to hearing in noisy
environments. Moreover, the review [12] critically examines the current
landscape of tele-audiology practices, highlighting both their constraints and
potential opportunities. Specifically, it explores intervention and
rehabilitation efforts for CIs, focusing on remote programming and the concept
of self-fitting CIs. Recently, a review written by Henry et al. in 2023 [13]
conducts a comprehensive review of noise reduction algorithms employed in CIs.
Maintaining a general classification based on the number of microphones
used—single or multiple channels—the analysis extends to incorporate recent
studies showcasing a growing interest in ML techniques. The review culminates
with an exploration of potential research avenues that hold promise for future
advancements in the field. Table 1 offers a comparative analysis of the
proposed review in contrast to other discussed AI-based CI reviews and
surveys.

Table 1: Comparison of this review against other existing AI-based CI reviews
and surveys. Tick marks (✓) indicate that a particular field has been
considered, while cross marks (✗) indicate that a field has been left
unaddressed. The symbol (★) indicates that the most critical concerns of a
field have not been addressed.

Ref. |  Year |  Description | Backg. | TCI | Metrics | datasets | MLT | DLT | Apps | RGCs | FDs
---|---|---|---|---|---|---|---|---|---|---|---
[9] |  2019 |  The intersection of ML and CIs |  ✗ |  ✗ |  ✗ |  ✗ | ✓ |  ✗ | ✓ | ★ | ★
[10] |  2022 |  Explores materials and devices designed for medical care |  ✗ |  ✗ |  ✗ | ★ |  ✗ |  ✗ | ✓ |  ✗ |  ✗
[11] |  2022 |  AI in otolaryngology and communication sciences | ✓ |  ✗ |  ✗ |  ✗ | ✓ |  ✗ | ★ |  ✗ |  ✗
[12] |  2022 |  Remote audiology services |  ✗ |  ✗ |  ✗ |  ✗ |  ✗ |  ✗ | ✓ |  ✗ |  ✗
[13] |  2023 |  Signal processing in CI for noise reduction | ✓ |  ✗ |  ✗ |  ✗ | ✓ |  ✗ | ★ |  ✗ | ★
Our |  2024 |  Advanced AI algorithms for CIs | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

Abbreviations: Taxonomy in CI (TCI), ML-based techniques (MLT), DL-based
techniques (DLT), Applications (apps), Research gaps and challenges (RGCs) ,
Future directions (FDs)

###  1.2 Statistics on investigated papers

Recently, there has been a surge in publications related to AI-based CI. The
review methodology entails defining the search strategy and study selection
criteria. Criteria for inclusion, like keyword relevance and impact, shape the
quality assessment protocol. A comprehensive search was conducted on databases
such as Scopus and Web of Science. Keywords were extracted for theme
clustering, resulting in a formulated query to gather advanced AI-based CI
studies. The research query fetches references from papers containing the
keywords ”Cochlear implant” or ”Hearing loss” and ”Artificial intelligence” in
their abstracts, titles, or authors’ keywords. It subsequently refines these
papers, focusing on those that also include ”Machine learning,” ”Deep
learning,” or ”Reinforcement learning.” Figure 1 illustrates the most
frequently used keywords by the authors in the titles, abstracts, and keywords
of the selected papers.

Figure 1: Top keywords in AI-based CI research field.

Figure 2 illustrates the distribution of these papers by type and publication
year, encompassing articles, conference papers, and reviews. Additionally, it
showcases the percentage of publications from before 2015 to 2023, with a
total count of 99 papers. Notably, the peak in publications within this field
is observed in 2023 (Figure 2 (a)), underscoring the heightened interest among
researchers. Furthermore, the predominant publication type is research
articles, followed by conference papers (Figure 2 (b)).

Figure 2: Bibliometrics analysis of the papers included in this review. (a)
Papers distribution over the last years. (b) Percentage breakdown of paper
types included in this review.

###  1.3 Motivation and contribution

The motivation behind conducting a comprehensive review on CIs stems from the
imperative to critically assess and consolidate the current state of AI
applications in this crucial field. CIs have revolutionized auditory
rehabilitation for individuals with hearing impairment, and integrating ML and
deep learning (DL) techniques holds immense potential for further
advancements. This review seeks to fill a significant gap in the existing
literature by providing a detailed analysis of recent AI-based CI frameworks.

The primary objective is to present a nuanced understanding of the landscape,
categorizing frameworks based on ML and DL methodologies, available datasets,
and key metrics. By addressing this gap, the review aims to offer valuable
insights for researchers, clinicians, and technologists involved in the
development and improvement of CI technologies. Furthermore, the exploration
of advanced DL algorithms, such as transformers and reinforcement learning
(RL), in the context of CIs, underscores the potential for transformative
breakthroughs. Ultimately, this research review aspires to contribute to the
enhancement of CI technologies, fostering innovation and improving the quality
of life for individuals with hearing impairment. The principal contributions
of this paper can be succinctly outlined as follows:

  * •

Detailing the assessment metrics associated with AI and CIs, and elucidating
the extensively utilized datasets, whether publicly accessible or generated,
employed to validate AI-based ASR for CI methodologies.

  * •

The implementation of CI, along with a comprehensive elucidation of the
taxonomy encompassing ML and DL-based CI, is thoroughly expounded upon.
Additionally, recommended frameworks for AI-based CI are thoroughly discussed
and succinctly summarized in tables for enhanced clarity.

  * •

Providing detailed insights into the applications of ML and DL within the
domain of CI, encompassing functions such as denoising and speech enhancement,
segmentation, thresholding, imaging, as well as CI localization, along with
various other functionalities.

  * •

Delve into the existing gaps in AI-driven CI, offer insights, and propose
novel ideas to address these gaps. Additionally, explore potential avenues for
future research to deepen comprehension and provide valuable guidance for
subsequent investigations.

The subsequent sections are organized as follows: Section 2 delves into the
background in speech processing for, outlining datasets and metrics. Section 3
discusses the methodology employed for CI based on AI. Section 4 presents the
medical applications and impact of applying AI on CI. Section 5 offers a
comprehensive discussion research gaps, future directions, and perspectives.
Finally, Section 6 concludes the paper with implications and future research
directions.

##  2 Background in speech processing for CIs

###  2.1 Speech processing for CIs

CIs are electronic devices that can be implanted in one or both ears to
restore some level of hearing for individuals with partial or severe deafness.
CIs comprise an external part with a microphone and speech processor and an
internal part with a receiver-stimulator and electrode array as shown in
Figure 3. They convert sounds into electrical signals, stimulating the
auditory nerve to enable sound perception in individuals with profound hearing
loss [14].

Figure 3: Illustration of a CI depicting the components situated externally
and internally within the device [14]

The incoming sound is divided into multiple frequency channels using bandpass
filters and then processed by envelope detectors. Non-linear compressors
adjust the dynamic range of the envelope for each patient. The compressed
envelope amplitudes are then utilized to modulate a fixed-rate biphasic
carrier signal. A current source converts voltage into pulse trains of
current, which are delivered to electrodes placed along the cochlea in a non-
overlapping manner. This stimulation method is called continuous interleaved
sampling (CIS). Another coding strategy, known as advanced combination encoder
(ACE), uses a greater number of channels and dynamically selects the ”n-of-m”
bands with the largest envelope amplitudes (prior to compression). Only the
corresponding ”n” electrodes are stimulated. A popular device widely used for
CIs, such as the cochlear nucleus, typically has 22 channels. The sound
processor, which usually contains a microphone, battery, and other components,
can be worn either behind-the-ear (BTE) or off-the-ear (OTE). A headpiece
holds a transmitter coil, positioned externally above the ear, while
internally, a receiver coil, stimulator, and electrode array are implanted.
The SP includes a digital signal processor (DSP) with memory units (maps) that
store patient-specific information. An audiologist configures these maps
during the fitting process, adjusting thresholds for each electrode, including
T-Levels (the softest current levels audible to the CI user) and C/M-Levels
(current levels perceived as comfortably loud), as well as the stimulation
rate or programming strategy. Data (pulse amplitude, pulse duration, pulse
gap, etc.) and power are sent through the skull via an radio frequency signal
from the transmitter coil to the receiver coil. The stimulator decodes the
received bitstream and converts it into electric currents to be delivered to
the cochlear electrodes. High-frequency signals stimulate electrodes near the
base of the cochlea, while low-frequency signals stimulate electrodes near the
apex. The CI stimulates the auditory nerve afferents, which connect to the
central auditory pathways. However, compared to individuals with normal
hearing (NH), CI users face more difficulties in speech perception,
particularly in noisy environments. Hearing loss can be caused by various
factors, including natural aging, genetic predisposition, exposure to loud
sounds, and medical treatments. Damage to the hair cells in the inner ear
often leads to a reduced dynamic range of hearing, as well as decreased
frequency selectivity and discriminative ability in speech processing. To
evaluate the effectiveness of CIs in speech perception amid noise, listening
tests involving both normal hearing individuals and CI users are commonly
conducted. These tests typically employ a combination of speech utterances
from a recognized speech corpus and background noises like speech-weighted
noise and babble. Alternatively, vocoder simulations can be utilized alongside
speech intelligibility metrics. While sentence-based tests are frequently
employed, other stimuli such as vowels, consonants, and phonemes are also
used. As a result, noise reduction techniques are increasingly employed to
enhance the performance of CIs in challenging environments [13].

###  2.2 Datasets

Researchers have utilized numerous datasets to validate their proposed
schemes, comprising both widely recognized publicly available sets and locally
generated ones. These datasets fall into two categories: speech or images.
Table 2 provides a summary of these datasets, detailing their characteristics,
citing studies that have utilized them, and indicating their availability
through links or references.

Table 2: List of publicly available datasets used for CIs applications

.  Dataset NoS NoC Characteristics Used by Source MHINT 2560
8 It is a test resource that was developed in two versions: MHINT-M for use in
Mainland China and MHINT-T for use in Taiwan. The development of MHINT took
into consideration the tonal nature of Mandarin, recognizing the importance of
lexical tone in designing the test. [15],[16],[17] [18] PhonDat 1 21587 201 It
is from the Bavarian Archive For Speech Signals, contains 21.4 hours of speech
data, and it is orthographically transcribed and phonemically annotated.
[19],[20] Link111https://www.phonetik.uni-muenchen.de/Bas/BasPD1eng.html TIMIT
6300 630 The dataset consists of phonemically and lexically transcribed speech
from American English speakers belonging to diverse demographics and dialects.
It provides comprehensive information with time-aligned orthographic,
phonetic, and word transcriptions. Additionally, each utterance is accompanied
by its corresponding 16-bit, 16kHz speech waveform file, ensuring a complete
and detailed dataset for analysis and experimentation in ASR and acoustic-
phonetic studies. [21],[22],[23] Link222https://www.kaggle.com/mfekadu/darpa-
timit-acousticphonetic-continuous-speech GSC 18 Hours 30 The dataset was
gathered using crowd-sourcing. It consists of 65,000 recordings, each lasting
one second, and contains 30 brief words. Among these words, 20 commonly used
ones were spoken five times by the majority of participants, while 10 other
words (considered unfamiliar) were spoken only once. [24]
Link333https://research.googleblog.com/2017/08/launching-speech-commands-
dataset.html WSJ0-2mix - - This dataset is a collection of mixed speech
recordings that are used for speech recognition. It utilizes utterances from
the Wall Street Journal (WSJ0) corpus to create these mixtures. [25] Link
444https://www.merl.com/research/highlights/deep-clustering LibriVox 547 Hours
- The LibriVox dataset consists of sentence-aligned German audio, text, and
English translations from audio books. [22] Link 555https://librivox.org/ HSM
20 30 The development of the German HSM Sentence Test aimed to provide an
ample amount of test sentences for the repeated assessment of speech
comprehension among CI users. [22],[26] [27] CEC1 6000 24 The dataset features
simulated living rooms with static sources, including a single target speaker,
interferer (competing talker or noise), and a large target speech database of
English sentences produced by 40 British English speakers. [26]
Link666https://claritychallenge.org/clarity_CEC1_doc/docs/cec1_data DEMAND 560
6 The dataset consists of 15 recordings capturing acoustic noise in various
environments. These recordings were made using a 16-channel array, with
microphone distances ranging from 5 cm to 21.8 cm. [22]
Link777https://www.kaggle.com/datasets/chrisfilo/demand THCHS-30 35 Hours 50
The dataset is a free Chinese speech corpus accompanied by resources such as
lexicon and language models. [28] Link888https://www.openslr.org/18/ BCP 55938
20 The bern cocktail party (BCP) dataset contains Cocktail Party scenarios
with individuals wearing CI audio processors and a head and torso simulator.
Recorded in an acoustic chamber, it includes multi-channel audio, image
recordings, and digitized microphone positions for each participant. [29] [30]
ikala 252-30s 206 The dataset comprises audio recordings consisting of vocal
and backing track music with a sampling rate of 44100 Hz. Each music track is
a stereo recording, where one channel contains the singing voice and the other
channel contains the background music. All tracks were performed by
professional musicians and featured a group of six singers, evenly split
between three females and three males. [31]
Link999https://zenodo.org/records/3532214 MUSDB 150 4 The dataset is a
collection of music tracks specifically designed for music source separation
research. It consists of professionally mixed songs across various genres,
with individual tracks isolated for vocals, drums, bass, and other
accompaniment. [31] Link101010https://sigsep.github.io/datasets/musdb.html
CQ500 491 - The dataset includes anonymized dicoms files, along with the
interpretations provided by radiologists. The interpretations were conducted
by three radiologists who have 8, 12, and 20 years of experience in
interpreting cranial CT scans, respectively. [32]
Link111111http://headctstudy.qure.ai/dataset ECochG 21 - The dataset captures
inner ear potentials in response to acoustic stimulation, specifically
focusing on cochlear implant recipients. [33],[34],[35]
Link121212https://zenodo.org/records/7092661

Abbreviations: Number of samples (NoS), Number of classes (NoC)

###  2.3 Metrics

Multiple evaluation metrics are utilized during the training and validation of
any DL models, including AI-based CI. These metrics, integral to the confusion
matrix, are widely known and applicable across various data types such as
speech or image. They include Accuracy (Acc), Sensitivity (Sen), Recall (Rec),
Specificity (Spe), Precision (Pre), F1 score (F1), and  receiver operating
characteristic curve (ROC). Moreover, different metrics play roles in
prediction task. For instance, intersection over union (IoU) assesses overlap,
and mean absolute error (MAE) quantifies absolute differences.For a
comprehensive understanding of the metrics discussed, including their
equations, refer to the details provided in [5, 36]. Other metrics that are
widely used for CI are summarized in Table 3.

Table 3: An overview of the metrics employed for evaluating CI methods. |  |
---|---|---
Metric |  Formula |  Description
DCS |  $\displaystyle\frac{{2\cdot|A\cap B|}}{{|A|+|B|}}$ |  The metric dice coefficient similarity (DCS) is used to evaluate the performance of the vestibule segmentation network. The Dice coefficient is a widely used similarity metric in image segmentation tasks. It measures the overlap between the predicted segmentation mask and the ground truth mask.
ASD |  $\displaystyle\frac{\sum_{a\in S(A)}{\min_{b\in S(B)}(a-b)}+\sum_{b\in S(B)}{\min_{a\in S(A)}(b-a)}}{|S(A)|+|S(B)|}$ |  ASD is a commonly used evaluation measure in medical image segmentation tasks. It quantifies the average distance between the surfaces of two segmented objects, typically the predicted segmentation and the ground truth.
AVD |  $\displaystyle\max(\text{d}(A,B),\text{d}(B,A))$ |  AVD quantifies the average difference in volume between a predicted segmentation and a reference or ground truth segmentation..
SNR |  $\displaystyle\mathrm{10\log_{10}\Big{(}\frac{Signal}{Noise}\Big{)}_{dB}}$ |  SNR is a measure of the quality of the speech signal. It is commonly used to evaluate the quality of the stego-speech (the speech signal after the hidden information has been embedded). A lower SNR indicates that the steganography technique has introduced more distortion to the speech signal.
ASE |  $\displaystyle\frac{\sum E_{ij}}{N}$ |  Is the average surface error (ASE). The distances between corresponding points on the measured surface and the reference surface are computed using $\displaystyle E_{ij}=|I_{1}(i,j)-I_{2}(k,l)|$, known as point to point error (P2PE). $N$ is the total number of correspondence points. Normalize average error by dividing by the imaging system’s dynamic range. $(i,j)$, and $(k,l)$ represent pixel values in the first and second images, respectively.
ILD |  $20\log_{10}\left(\frac{L}{R}\right)$ |  ILD a psychoacoustic metric, measuring sound level differences between left (L) and right (R) ears, reflects cues essential for sound localization, enhancing spatial awareness in auditory perception for directional sound source identification.
MEE |  $\displaystyle\frac{1}{N}\sum_{i=1}^{N}|L_{i}-\hat{L}_{i}|$ |  MEE measures the average absolute difference between the true $L_{i}$ and estimated $\hat{L}_{i}$ endpoint locations across multiple utterances $N$.
MMSE |  Estimator $\displaystyle\hat{x}_{MMSE}=E[X|Y]$ |  MMSE is a statistical estimation technique used in speech enhancement to minimize the mean square error between the estimated $Y$ and true $X$ clean speech signals
STOI |  $\displaystyle\frac{\sum_{t=1}^{T}g(t)\cdot\text{STOI}_{\text{frame}}(t)}{\sum_{t=1}^{T}g(t)}$ |  STOI is a metric used to assess the intelligibility of time-frequency weighted noisy speech. It is based on the idea that human speech perception relies on the availability of important acoustic features in short time frames [37].
NCM |  |  NCM
SDR, SAR, and SIR |  $\displaystyle 10log_{10}\frac{{\left\|s_{target}\right\|}^{2}}{{\left\|e_{total}\right\|}^{2}},\hskip 8.5359pt10log_{10}\frac{{\left\|s_{target}+e_{inter}+e_{noise}\right\|}^{2}}{{\left\|e_{artifacts}\right\|}^{2}},\newline \hskip 8.5359pt10log_{10}\frac{{\left\|s_{target}\right\|}^{2}}{{\left\|e_{inter}\right\|}^{2}}$ |  SDR, source-to-artifact ratio (SAR), and source-to-interference ratio (SIR) are metrics objectively assess and compare speech source-separation algorithms based on accuracy and minimization of distortions and interference. source-to-distortion ratio (SDR) gauges source separation quality by comparing true source power to introduced distortion. SAR evaluates source separation from artifacts or noise, while SIR measures the ratio of true source power to interference after separation.

##  3 Taxonomy of CI-based AI techniques

Several artificial intelligence techniques have been employed to enhance the
efficacy of CIs. While some rely on 1D data, others process information in a
2D image format. Figure 4 summarizes all AI algorithms utilized, alongside the
features employed and hybrid AI methodologies. Additionally, Table 4 provides
a summary of DL-based techniques utilized in CI hearing devices.

Figure 4: Taxonomy of the employed AI techniques for CI.

###  3.1 CI-based AI Implementation

CI programming involves adjusting device settings to optimize sound perception
for individual users. This includes setting stimulation levels, electrode
configurations, and signal processing parameters to enhance speech
understanding and auditory experiences based on patient feedback and objective
measures. In 2010, govaerts et al. [38] describes the development of an
intelligent agent, called fitting to outcomes expert (FOX), for optimizing CI
programming as described in Figure 6. The agent analyzes map settings and
psycho-acoustic test results to recommend and execute modifications to improve
outcomes. The tool focuses on an outcome-driven approach, reducing fitting
time and improving the quality of fitting. It introduces principles of AI in
the CI fitting process. The study proposed objective measures and group
electrode settings as strategies to reduce fitting time.

Figure 5:  The fundamental operating concept of the FOX involves inputting an
initial program and multiple psychoacoustic test outcomes. FOX processes this
information and generates fitting suggestions as its output. When integrated
with proprietary outcome and CI fitting software, the shaded boxes represent
its functionality, while the unfilled boxes represent its standalone
capability [38]. Audiqueen is a dataset with A and E (A&E) phoneme
discrimination.

Similarly in [39, 40, 41, 42, 43], all have employed FOX for programming CI.
Vaerenberg et al. [39] discusses the use of FOX for programming CI sound
processors in new users. FOX modifies maps based on specific outcome measures
using heuristic logic and deterministic rules. The study showed positive
results and optimized performance after three months of programming, with good
speech audiometry and loudness scaling outcomes. The paper highlights the
importance of individualized programming parameters and the need for outcome-
based adjustments rather than relying solely on comfort. In [40], computer-
assisted CI fitting using FOX assessed its impact on speech understanding.
Results from 25 recipients showed that 84% benefited from suggested map
changes, significantly improving speech understanding thanks to the learning
capacity of FOX. This approach offers standardized, systematic CI fitting,
enhancing auditory performance.

The COCH gene, referred to as the Cochlin gene, is responsible for encoding
the cochlin protein situated on chromosome 14 in humans, primarily expressed
in the inner ear. Cochlin predominantly functions within the cochlea, a
spiral-shaped structure involved in the process of hearing, contributing to
its structural integrity and proper operation. Wathour et al. in [41]
discusses the use of AI in CI fitting through two case studies. The first case
involves a 75-year-old lady who received a left ear implant due to gradual and
severe hearing loss in both ears without a clear cause. In the second case, a
72-year-old man with a COCH gene mutation causing profound hearing loss in
both ears underwent a right ear implant, to assess whether CI programming
using the AI software FOX application, could improve CI performance. The
results showed that CI-assisted fitting led to improvements in auditory
outcomes for adult CI recipients who had previously undergone manual fitting.
The AI suggestions helped improve word recognition scores and loudness scaling
curves. Likewise, Waltzman et al. [42] incorporate AI in programming CIs,
aimed to assess, on fifty-five adult’s CI , the performance and
standardization of AI-based programming. The results showed that the AI-based
FOX for some patients performed better, while others had similar results, yet
he majority preferred the FOX system.

###  3.2 ML-based methods

ML is a subfield of AI that focuses on developing algorithms and statistical
models that enable computer systems to improve their performance in a specific
task by learning features from input data. The research in [44] focuses on
algorithm-based hearing and speech therapy rehabilitation after cochlear
implantation, particularly for older individuals. They propose the development
of an ML-based application that offers personalized hearing therapy tailored
to the patient’s needs, such as select exercises, adjust difficulty levels,
and analyze patient difficulties. It operates independently, reducing the
reliance on local speech therapists, cost-effective, and accessible
alternative to traditional therapy, improving outcomes and quality of life for
CI recipients. In addition, Torresen et al. [45] discusses the use of ML
techniques to streamline the adjustment process for CIs. The goal is to
predict optimal adjustment values for new patients based on data from previous
patients. By analyzing data from 158 former patients, the study shows that
while fully automatic adjustments are not possible, ML can provide a good
starting point for manual adjustment. The research also identifies the most
important electrodes to measure for predicting levels of other electrodes.
This approach has the potential to reduce programming time, benefit patients,
and improve speech recognition scores, particularly for young children and
patients with post-lingual deafness. Henry et al. in their [46] investigates
the importance of acoustic features in optimizing intelligibility for CIs in
noisy environments. The study employs ML algorithms and extracts acoustic
features from speech and noise mixtures to train a deep neural networks (DNN).
The results, using various metrics, reveal that frequency domain features,
particularly Gammatone features, perform best for normal hearing, while Mel
spectrogram features exhibit the best overall performance for hearing
impairment. The study suggests a stronger correlation between short-time
objective intelligibility (STOI) and normalized covariance measure (NCM) in
predicting intelligibility for hearing-impaired listeners. The findings can
aid in designing adaptive intelligibility enhancement systems for CIs based on
noise characteristics.

Moreover, the research in [47] focuses on imputing missing values in CI
candidate audiometric data. This study assessed the performance of various
imputation algorithms using a dataset of 7,451 audiograms from CI patients.
The results showed that the quantity of missing data affected the imputation
performance, with greater amounts leading to poorer results. The distribution
of sparsity in the audiometric data was found to be non-uniform, with inter-
octave frequencies being less commonly tested. The multiple imputation by
chained equations (MICE) method, safely imputed up to six missing data points
in an 11-frequency audiogram, consistently outperformed other models. This
study highlights the importance of imputation techniques in maximizing
datasets in hearing healthcare research. Xu et al. in [48] explores the
objective discrimination of bimodal speech using frequency following
responsess. The study investigates the neural encoding of fundamental
frequency ($f_{0}$), called also pitch [49], and temporal fine structure cues
(TFSC) in simulated bimodal speech conditions. The results show that
increasing acoustic bandwidth enhances the neural representation of $f_{0}$
and TFSC components in the non-implanted ear. Moreover, ML algorithms
successfully classify and discriminate FFRs based on spectral differences
between vowels. The findings suggest that the enhancement of $f_{0}$ and TFSC
neural encoding with increasing bandwidth is predictive of perceptual bimodal
benefit in speech-in-noise tasks. FFRs may serve as a useful tool for
objectively assessing individual variability in bimodal hearing. The research
conducted by Crowson et al. [50] aimed to predict postoperative CI performance
using supervised ML. The authors used neural networks and decision tree-based
ensemble algorithms on a dataset of 1,604 adults who received CIs. They
included 282 text and numerical variables related to demographics, audiometric
data, and patient-reported outcomes. The results showed that the neural
network model achieved a 1-year postoperative performance prediction root mean
square error (RMSE) of 0.57 and classification accuracy of 95.4%. When both
text and numerical variables were used, the RMSE was 25.0% and classification
accuracy was 73.3%. The study identified influential variables such as
preoperative sentence-test performance, age at surgery, and specific
questionnaire responses. The findings suggest that supervised ML can predict
CI performance and provide insights into factors affecting outcomes. In the
same context of prediction, Mikulskis et al. [51] focuse on predicting the
attachment of broad-spectrum pathogens to coating materials for biomedical
devices as illustrated in Figure 6. The authors employ ML methods to generate
quantitative predictions for pathogen attachment to a large library of
polymers. This approach aims to accelerate the discovery of materials that
resist bacterial biofilm formation, reducing the rate of infections associated
with medical devices. The study highlights the need for new materials that
prevent bacterial colonization and biofilm development, particularly in the
context of antibiotic resistance. The results demonstrate the potential of ML
in designing polymers with low pathogen attachment, offering promising
candidate materials for implantable and indwelling medical devices. Similarly,
Alohali et al. [52] focuses on using ML algorithms to predict the post-
operative electrode impedances in CI patients. The study used a dataset of 80
pediatric patients and considered factors such as patient age and
intraoperative electrode impedance. The results showed that the best algorithm
varied by channel, with Bayesian linear regression and neural networks
providing the best results for 75% of the channels. The accuracy level ranged
between 83% and 100% in half of the channels one year after surgery.
Additionally, the patient’s age alone showed good prediction results for 50%
of the channels at six months or one year after surgery, suggesting it could
be a predictor of electrode impedance.

Figure 6:  Diagram illustrating the procedures utilized in the production of
microarrays, analysis of pathogens, data modeling, and forecasting the
attachment of pathogens to novel polymers [51].

###  3.3 CNN-based methods

CNN are a class of DL algorithms widely used in computer vision tasks. Their
architecture includes convolutional layers that automatically learn
hierarchical features from input data. The core convolution for 2D data
operation is defined by the equation:

$S(i,j)=(I*K)(i,j)=\sum_{m}\sum_{n}I(m,n)\cdot K(i-m,j-n)$ (1)

Here, $I$ represents the input 2D data, $K$ is the convolutional kernel, and
$S$ is the output feature map. CNN excel at recognizing spatial patterns,
making them essential in image recognition, object detection, and other visual
tasks. Additionally, there exist 1D convolutional neural networks, which are
effective for sequential data analysis, such as in natural language processing
or time series applications. CNN is widely used for the interdisciplinary
nature of CI, which involves aspects of neurobiology, signal processing, and
medical technology. For example, the proposed work [53] introduces a novel
pathological voice identification system using signal processing and DL. It
employs CI models with bandpass and optimized gammatone filters to mimic human
cochlear vibration patterns. The system processes speech samples and utilizes
a CNN for final pathological voice identification. Results show discrimination
of pathological voices with F1 scores of 77.6% (bandpass) and 78.7%
(gammatone). The paper addresses voice pathology causes, compares filter
models, and proposes a non-invasive, objective assessment system. It
contributes to the field with a comprehensive performance analysis, achieving
high accuracy and demonstrating effectiveness compared to related works.
Addtionally, in the scheme proposed by Wang [15], the fully convolutional
neural networks (FCN) model is evaluated for enhancing speech intelligibility
in mismatched training and testing conditions. Using 2,560 Mandarin utterances
and 100 noise types, the study compares FCN with traditional minimum mean
square error (MMSE) and deep denoising auto-encoder (DDAE) models. Two sets of
experiments are conducted for normal and vocoded speech. The FCN model
demonstrates superior performance, maintaining clearer speech structures,
especially in mid-low frequency regions crucial for intelligibility. Objective
evaluations using STOI scores and a listening test confirm FCN’s effectiveness
under challenging signal to noise ratio (SNR) conditions, outperforming MMSE
and DDAE. The study suggests FCN as a promising choice for electric and
acoustic stimulation (EAS) speech processors.

Moving on, the research paper in [54] presents a novel approach to optimize
stimulus energy for CIs. A CNN was developed as a surrogate model for a
biophysical auditory nerve fiber model, significantly reducing simulation time
while maintaining high accuracy. The CNN was then used in conjunction with an
evolutionary algorithm [55] to optimize the shape of the stimulus waveform,
resulting in energy-efficient waveforms and potential improvements in CI
technology. Traditional computational models of the cochlea, which represent
it as a transmission line, are computationally expensive due to their cascaded
architecture and the inclusion of non-linearities. As a result, they are not
suitable for real-time applications such as hearing aids, robotics, and ASR.
For the aforementioned conditions, the study in [56] presents a hybrid
approach, called
CoNNear131313https://github.com/HearingTechnology/CoNNear_periphery, which
combines CNNs, capable of performing end-to-end waveform predictions in real-
time, with computational neuroscience to create a real-time model of human
cochlear mechanics and filter tuning. The CNN filter weights were trained
using simulated basilar-membrane (BM) displacements from cochlear channels,
and the model’s performance was evaluated using basic acoustic stimuli. The
CoNNear model is designed to capture the tuning, level-dependence, and
longitudinal coupling characteristics of human cochlear processing. It
converts acoustic speech stimuli into BM displacement waveforms across 201
cochlear filters. Its computational efficiency and ability to capture human
cochlear characteristics make it suitable for developing human-like machine-
hearing applications.

The research paper in [24] explores the utilization of a CNN in simulating
speech processing with CIs. The study investigates the effect of channel
interaction, a phenomenon that degrades spectral resolution in CI delivered
speech, on learning in neural networks. By modifying speech spectrograms to
approximate CI delivered signals, the CNN is trained to classify them. The
findings suggest that early in training, the presence of channel interaction
negatively impacts performance. This indicates that the spectral degradation
caused by channel interaction conflicts with perceptual expectations acquired
from high-resolution speech. The study highlights the potential for reducing
channel interaction to enhance learning and improve speech processing in CI
users, particularly those who have adapted to high-resolution speech.

Schuerch et al. [33] focus on the objectification of intracochlear
electrocochleography (ECochG) using AlexNet, CNN architecture, to automate and
standardize the assessment and analysis of cochlear microphonic (CM) signals
in ECochG recordings for clinical practice and research. The authors compared
three different methods: correlation analysis, Hotelling’s T2 test, and DL, to
detect CM signals. The DL algorithm performed the best, followed closely by
Hotelling’s T2 test, while the correlation method slightly underperformed. The
automated methods achieved excellent discrimination performance in detecting
CM signals with an accuracy up to 92%, providing fast, accurate, and examiner-
independent evaluation of ECochG measurements.

Moreover, Arias et al. [19] presents a methodology for speech processing using
CNNs. The study aims to improve the representation learning capabilities of
CNNs by combining multiple time-frequency representations of speech signals.
The proposed approach involves generating multi-channel spectrograms by
combining continuous wavelet transform, Mel-spectrograms, and Gammatone
spectrograms. These spectrograms are utilized as input data for the CNN
models. The effectiveness of the approach is evaluated in two applications:
automatic detection of speech deficits in CI users and phoneme class
recognition. The results demonstrate the advantages of using multi-channel
spectrograms with CNNs, showcasing improved performance in speech analysis
tasks. The  convolutional recurrent neural network with gated recurrent units
(CGRU) architecture is utilized, as illustrated in Figure 7. The input
sequences consist of 3D-channel inputs created by combining Mel-spectrograms,
Cochleagrams, and  continuous wavelet transform (CWT) with Morlet wavelets.
Convolution is applied solely on the frequency axis in order to preserve the
time information. The resulting feature maps are subsequently fed into a
2-stacked bidirectional gated recurrent units (GRU). A softmax function is
employed to predict the phoneme label for each speech segment in the input
signal.

Figure 7:  CGRU architecture, focusing on input sequences composed of
3D-channel data generated from Mel-spectrograms, Cochleagrams, and CWT using
Morlet wavelets [19].

This paper [20] introduces a novel method for automatically detecting speech
disorders in CI users using a multi-channel CNN. The model processes 2-channel
input comprising Mel-scaled and Gammatone filter bank spectrograms derived
from speech signals. Testing on 107 CI users and 94 healthy controls
demonstrates improved performance with 2-channel spectrograms. The study
addresses a gap in acoustic analysis of CI user speech, proposing a DL
approach with potential applications beyond CI users. Experimental results
indicate the effectiveness of the proposed CNN-based method, offering promise
for speech disorder detection and potential extensions to other pathologies or
paralinguistic aspects that employ mel-frequency cepstral coefficients and
gammatone frequency cepstral coefficient s features.

For 2D CNN, the following work [57] introduces image guided cochlear implant
programming  (IGCIP), enhancing CI outcomes using image processing. IGCIP
segments intra-cochlear anatomy in computed tomography (CT) images, aiding
electrode localization for programming. The scheme addresses challenges in
automating this process due to varied image acquisition protocols. The
proposed solution employs a DL-based approach, utilizing CNNs to detect the
presence and location of inner ears in head CT volumes. The CNNs is trained on
a dataset with 95.97% classification accuracy. Results indicate potential for
automatic labeling of CT images, with a focus on further 3D algorithm
development. However, in [54] presents a machine-learning approach to optimize
stimulus energy for CIs. A CNN was developed as a surrogate model for a
biophysical auditory nerve fiber model, significantly reducing simulation time
while maintaining high accuracy. The CNN was then used in conjunction with an
evolutionary algorithm to optimize the shape of the stimulus waveform,
resulting in energy-efficient waveforms. The proposed surrogate model offers
an efficient replacement for the original model, allowing for larger-scale
experiments and potential improvements in CI technology.

The work proposed by [58] introduces sliding window based CNN (SlideCNN), a
novel DL approach for auditory spatial scene recognition with limited
annotated data. The proposed method converts auditory spatial scenes into
spectrogram images and utilizes a SlideCNN for image classification. Compared
to existing models, SlideCNN achieves a significant improvement in prediction
accuracy, with a 12% increase. By leveraging limited annotated samples,
SlideCNN demonstrates an 85% accuracy in detecting real-life indoor and
outdoor scenes. The results have practical implications for analyzing auditory
scenes with limited annotated data, benefiting individuals with hearing aids
and CIs.

This paper [59] focuses on advancing laser bone ablation in microsurgery using
4D optical coherence tomography (OCT). The challenge lies in automatic control
without external tracking systems. The paper introduces a 2.5D scene flow
estimation method using CNN for OCT images, enhancing laser ablation control.
A two-stage approach involves lateral scene flow computation followed by depth
flow estimation. Training is semi-supervised, combining ground truth error and
reconstruction error. The method achieves a mean endpoint error (MEE) of (4.7
$\pm$ 3.5) voxel, enabling markerless tracking for image guidance and
automated laser ablation control in minimally invasive cochlear implantation.

Table 4: Summary of some proposed methods based on differents DL techniques.
When comparing the work with numerous existing schemes, only the best-
performing one will be highlighted.

Ref. |  DL method |  Domain |  Dataset |  Explanation |  CTODLM |  Best performance |  Improvement
---|---|---|---|---|---|---|---
[15] |  FCN |  Speech |  MHINT |  Experiment the effectiveness of FCN in restoring clean speech signals from noisy counterparts and enhancing the intelligibility of speech in EAS devices. |  DDAE |  STOI=0.85 (SNR=9dB) |  STOI=0.03
[19] |  CGRU |  Speech |  PhonDat 1 |  Using of onset and offset transitions in speech analysis and The input sequences are 3D-channel inputs |  CGRU |  Pre=0.68 $\pm\ 0.22$
Rec=0.69 $\pm\ 0.21$
F1=0.69 $\pm\ 0.19$ |  —
[59] |  CNN |  Image |  Created |  Multiple stages and follows a pyramidal network architecture inspired by SPyNet, which is designed for optical flow estimation. |  — |  MEE=4.7$\pm\ 3.5$ |  —
[20] |  multi-channel CNN |  Speech |  PhonDat 1 |  Developing a methodology for the automatic detection of speech disorders in CI users using a 2D-channel CNN. |  SVM |  Acc=83.4,
Sen=91.7,
Spe=78.7 |  Acc=+0.6,
Sen=-1.8,
Spe=+2.7
[16] |  DDAE |  Speech |  MHINT |  The architecture of the DDAE model consisting of five layers with 500 neurons each. Objective evaluations were conducted using STOI and NCM, and a listening test (PESQ) |  KLT |  NCM=0.25 (SNR=9dB )
STOI=0.85 (SNR=12dB) |  STOI=+0.07
[60] |  Res-CA |  Image |  Created |  Introduces Res-CA block, GCPFE, ACE loss, and DS mechanism to enhance the DAE’s performance and robustness for compressing CI stimulation patterns. |  3D-DSD |  DCS=89.91$\pm 1.77$
average surface distance (ASD)=0.13
average volume difference (AVD)=0.14 |  DCS=+0.24
ASD=+0.05
AVD=-0.04
[61] |  MMS141414https://github.com/AngeLouCN/Min_Max_Similarity |  Image |  Created |  The classifiers extract features to create negative pairs, while projectors transform unlabeled predictions into high-dimensional features to measure pixel-wise consistency. |  TransUNet |  DCS=0.920
IoU=0.861
MAE=0.021
F1=0.925 |  DCS=+0.032
IoU=+0.021
MAE=-0.01
F1=+0.012
[62] |  MARGAN |  Image |  Created |  MARGAN is a 3D generative adversarial network (GAN)-based approach that uses simulated training data to reduce metal artifacts in CT images. |  marLI |  RMSE =0.12
PSNR= 18.31
SSIM= 0.64 |  RMSE=-0.03
PSNR=+1.78
SSIM=+0.08
[63] |  cGAN |  Image |  Created |  Active shape model-based method used for segmenting intra-cochlear anatomical structures. |  — |  ASE=0.173 |  ASE=-50%
[25] |  MIMO-TasNet using RNN |  Speech |  WSJ0-2mix,
ITA, DEMAND |  A speech separation framework using TasNet and recurrent neural network (RNN)- eigenvector decomposition (EVD) to preserve spatial cues for CI users. |  MIMO-TasNet |  $\Delta$ interaural level difference (ILD)=0.38 decibel (dB) |  $\Delta$ILD=-0.69 dB
[31] |  DNN |  Speech |  iKala, MUSDB |  A DNN algorithm enhances music perception for CI users, addressing pitch and timbre limitations. Objective measurements and listener experiments reveal preferences for enhanced vocals. |  — |  SDR=8dB (with iKala)
SDR=5.5dB (with MUSDB) |  —
[64] |  Hybrid DNN |  Speech |  Created |  Hearing training system consists of a speech training database, automatic lip-reading using a hybrid neural network, comparison of lip shapes, and providing a standard lip-reading sequence. |  — |  Acc=98 (gesture recognition) Acc=87 (lip-reading recognition) |  —
[65] |  ESCSO-based LSTM |  ERP of EEG |  Created |  LSTM-ESCSO is a method that combines long short term memory (LSTM) network with  enhanced swarm based crow search optimization (ESCSO) to improve the performance of predicting CI scores. |  LSTM |  Acc=96.03 |  Acc=+7.14
[66] |  FOX |  Speech |  Created |  The study investigates the utilization of AI in remote CI fitting, demonstrates the feasibility of AI-based fitting with remote supervision, highlighting its potential for streamlining the fitting process. |  Manual fitting |  66% performed better with FOX |  —
[43] |  FOX |  Speech |  Created |  A study comparing manual and computer-assisted CI fitting. |  Manual fitting |  82% performed better with FOX |  —
[67] |  FOX |  Speech |  Created |  Examined the impact of an AI-based CI programming tool on experienced CI patients. The AI-generated CI map led to improved auditory outcomes. |  Manual fitting |  89% preferring FOX’s map |  —

Abbreviations: Project link (PL), Not available (NA), Compared to other DL
methods (CTODLM).

###  3.4 GAN-based methods

A GAN is a type of AI model consisting of two neural networks, a generator,
and a discriminator, engaged in a competitive learning process as presented in
Figure 8.

Figure 8:  Illustrative depiction of a standard GAN.

The generator aims to create realistic data, such as images, while the
discriminator tries to differentiate between real and generated samples. This
adversarial training dynamic leads to the refinement of the generator’s
output, generating increasingly authentic data. The objective is for the
generator to produce data that is indistinguishable from real samples. The
training process is represented by the minimax game framework, with the GAN
objective function given by:

$\min_{G}\max_{D}V(D,G)=\mathbb{E}_{x\sim p_{\text{data}}(x)}[\log
D(x)]+\mathbb{E}_{z\sim p_{z}(z)}[\log(1-D(G(z)))]$ (2)

In the GAN objective function, $\mathbb{E}_{x\sim p_{\text{data}}(x)}$ and
$\mathbb{E}_{z\sim p_{z}(z)}$ indicate the expected values over real data
samples $x$ and noise samples $z$, respectively. $G$ generates samples, $D$
discriminates between real and generated samples, $p_{\text{data}}$ and
$p_{z}$ are the distributions of real data and noise, respectively. Using GAN,
the research in [68] proposes a DL-based method for reducing metal artifacts
in post-operative CT imaging. The method utilizes a 3D-GAN trained on a large
number of pre-operative images with simulated metal artifacts. The GAN
generates artifact-free images by reducing the metal artifacts. The
effectiveness of the method is evaluated quantitatively and qualitatively,
showing promising results compared to classical artifact reduction algorithms.
The approach overcomes the challenges of post-operative assessment of cochlear
implantation caused by metal artifacts, and it does not require registration
of pre and post-operative images. The 3D-GAN improves spatial consistency and
is applicable to various types of artifacts. In addition, Wang et al. in
theirs paper [62], proposes a 3D metal artifact reduction algorithm for post-
operative high-resolution CT imaging. The algorithm is based on a GAN that
uses simulated physically realistic CT metal artifacts created by CI
electrodes. The generated images are used to train the network for artifact
reduction. The metal artifact reduction-GAN based method as described in [62],
utilizes a three-step process for reducing metal artifacts. Firstly, a
simulation is performed to replicate CI positioning. Secondly, a physical
simulation of CI metal artifacts is conducted. Lastly, a 3D GAN is trained
using both simulated and preoperative datasets. The generator component of the
GAN generates an image that has reduced metal artifacts, while the
discriminator network is responsible for determining whether the input image
contains metal artifacts or not. The method was evaluated on clinical CT
images of CI postoperative cases and outperformed other general metal artifact
reduction approaches. The paper introduces a novel approach that combines the
physical simulation of metal artifacts with 3D-GAN, providing a promising
solution for improving the visual assessment of post-operative imaging in CT.

Similarly, for CI metal artifacts reduction also, a conditional generative
adversarial networks (cGAN) were proposed by Wang et al. [63]. The approach
involves training a cGAN to learn mapping from artifact-affected CTs to
artifact-free CTs. During inference, the cGAN generated CT images with removed
artifacts. Additionally, a band-wise normalization method was proposed as a
preprocessing step to improve the performance of cGAN. The method was
evaluated on post-implantation CTs recipients, and the quality of the
artifact-corrected images was quantitatively assessed using P2PE. The results
demonstrate promising artifact reduction, outperforming the previously
proposed techniques. The authors evaluates the quality of artifact-corrected
images using a quantitative metric based on segmentations of intracochlear
anatomical structures. Specifically, the segmentation results obtained from a
previously published method were compared between real preimplantation CTs and
artifact-corrected CTs generated by the proposed method. The ASE was used as a
metric to assess the accuracy of the segmentation. The paper reports that the
proposed method achieves an ASE of 0.18 mm, which is approximately half of the
error obtained with a previously proposed technique.

###  3.5 RNN/LSTM-based methods

RNN are a class of artificial neural networks designed for sequential data
processing. They maintain hidden state information that is updated at each
time step, allowing them to capture temporal dependencies. The hidden state at
time $t$, denoted as $h_{t}$, is computed based on the input $x_{t}$, the
previous hidden state $h_{t-1}$, and model parameters $W$ and $U$, with $b$
representing the bias term. The equations governing the hidden state update
are given by:

$h_{t}=\sigma(Wx_{t}+Uh_{t-1}+b)$ (3)

where, $\sigma$ is an activation function, typically the hyperbolic tangent or
rectified linear unit (ReLU). Several speech processing techniques that
utilize CI are based on RNNs.

CI users struggle with music perception, and many studies have shown that
enhancing music vocals improves their enjoyment. The study described by
Gajęcki. et al. [69] explores source separation algorithms to remix pop songs
by emphasizing the lead-singing voice. DCAE, deep recurrent neural networks
(DRNN), multilayer perceptrons (MLP), and non-negative matrix factorization
(NMF) were evaluated through perceptual experiments involving CI recipients
and normal hearing subjects. The results show that a MLP and DRNN perform
well, providing minimal distortions and artifacts that are not perceived by CI
users. The paper also highlights the benefits of the implementation of a MLP
for real-time audio source separation to enhance music for CI users due to
their reduced computation time. In addition, The study described in [25]
proposes a speech separation framework for CI users using TasNet and RNN-EVD.
TasNet, a non-causal multiple-input multiple-output (MIMO)-based method, is
employed as the speech separation module. RNN-EVD, which combines RNNs with
EVD, is utilized to preserve spatial cues. The framework aims to effectively
separate speech and reduce ILD errors. The RNN-EVD network is trained using
$\Delta$ILD as the objective, and an additional SNR term is added to the loss
function for convergence. The experimental results demonstrate the
effectiveness of the proposed framework in preserving ILD cues for CI users in
various hearing scenarios.

The long short term memory (LSTM), an enhanced version of the RNN, addresses
limitations observed in RNNs under specific conditions [5, 70]. Unlike RNN,
LSTM excels in preserving past information, making it suitable for tasks with
long-term dependencies. Comprising LSTM units forming layers, each unit
regulates information flow through input, output, and forget gates, allowing
for prolonged retention of crucial information. The forward pass equations (4)
illustrate this process [5]. The symbols $L_{i}$ and $L_{j}$ denote input and
output, while $A_{f}$, $A_{i}$, and $A_{j}$ represent activation vectors for
forget, input, and output gates. $V_{c}$ is the cell state vector, and
$\sigma$ for the sigmoid activation function and $\odot$ for element-wise
multiplication. This LSTM structure with weight matrices $W$ and $U$ and bias
vector $b$ is outlined by [71].

$\displaystyle\begin{split}A_{f}&=\sigma(W_{f}*L_{i}+U_{f}*L_{j-1}+b_{f}),\quad
A_{i}=\sigma(W_{i}*L_{i}+U_{i}*L_{j-1}+b_{i}),\\\
L_{j}&=A_{j}\odot\tanh(V_{c}),\\\
A_{j}&=\sigma(W_{0}*L_{i}+U_{0}*L_{j-1}+b_{j}),\\\
V_{c}&=A_{f}*V_{c-1}+A_{i}*\text{cell\\_state}(W_{c}*L_{i}+U_{c}*L_{j-1}+b_{c}).\end{split}$
(4)

Recently, several schemes for CI and utilizing LSTM have been proposed in the
literature. The study described by Lu. et al. in 2020 [64] introduces a speech
training system designed for individuals with hearing impairments, such as
those with CIs, as well as individuals with dysphonia, utilizing automated
lip-reading recognition. The system combines CNN and RNN to compare mouth
shapes and train speech skills. It includes a speech training database,
automatic lip-reading using a hybrid neural network, matching lip shapes with
sign language vocabulary, and drawing comparison data. The system enables
hearing-impaired individuals to analyze and improve their vocal lip shapes
independently. It also supports the use of medical devices for correct
pronunciation. Experimental results demonstrate the system’s effectiveness in
correcting lip shape and enhancing speech ability. The proposed model utilizes
ResNet50, MobileNet, and LSTM networks for accurate lip-reading recognition.
Later on, the scientific paper published by Chu et al. in 2021 [72] proposes a
causal DL framework for classifying phonemes in CIs to enhance speech
intelligibility. The authors trained LSTM networks using features extracted at
the time-frequency resolution of a CI processor. They compared CI-inspired
features (log STFT power spectrum, log ACE power spectrum, and log-mel-
filterbank) with traditional ASR features. The results showed that CI-inspired
features outperformed traditional features, providing slightly higher levels
of performance. The author claimed that, this study is the first to introduce
a classification framework with the potential to categorize phonetic units in
real-time in a CI, offering possibilities for improving speech recognition in
reverberant environments for CI users. In 2023, Huang et al. proposed in [73],
a DL-based sound coding strategy for CIs, called ElectrodeNet. By leveraging
DNN, CNN, and LSTM, ElectrodeNet replaces conventional envelope detection in
the ACE strategy. Objective evaluations using measures like STOI and NCM
demonstrate strong correlations between ElectrodeNet and ACE. Additionally,
subjective tests with normal-hearing listeners confirm the effectiveness of
ElectrodeNet in sentence recognition for vocoded Mandarin speech. The study
extends ElectrodeNet with ElectrodeNet-CS, incorporating channel selection
(CS) through a modified DNN network. ElectrodeNet-CS produces N-of-M
compatible electrode patterns and performs comparably or slightly better than
ACE in terms of STOI and sentence recognition. This research showcases the
feasibility and potential of deep learning in CI coding strategies, paving the
way for future advancements in AI-powered CI systems. Similarly, the research
presented by Jeyalakshmi et al. [65] focuses on predicting CI scores for
children aged 5 to 10 using a reconfigured LSTM network as illustrated in
Figure 9. The proposed architecture aims to enhance language development
skills in children with auditory deprivation, this could be achieved by
guiding CI programming through the analysis of cross-modal data obtained from
previously programmed patients. The research utilizes visual cross-modal
plasticity and visual evoked potential to discover patterns in the data that
can predict outcomes for future patients. The proposed methodology involves
the use of LSTM network and ESCSO to identify optimal weights. The results
demonstrate the superiority of the ESCSO-based LSTM technique over other
methods.

Figure 9:  The design of the LSTM architecture suggested by Jeyalakshmi et al.
[65].

In Figure 9, ”oz,” ”cz,” ”t5,” and ”t6” refer to specific electrode placements
or positions on the scalp in the international 10-20 system for
electroencephalography (EEG) or event-related potential (ERP) recordings.
These positions represent specific areas on the scalp where electrodes are
attached to measure electrical activity in the brain. The amplitude represents
the intensity or strength of the electrical signal detected at a particular
point on the scalp, reflecting the neural activity in the corresponding brain
region. The parameters ”N75,” ”P100,” and ”N145” refer to specific components
or peaks of ERPs obtained from EEG recordings. ERPs are electrical responses
recorded from the brain in response to specific stimuli or events, and they
reflect the neural processing associated with those stimuli. Besides, $I/P$
represent inputs, $O/P$ for output, and $W$ represent Weights.

###  3.6 AE-based methods

An autoencoder (AE) is a type of neural network designed for unsupervised
learning, tasked with encoding input data into a compressed representation and
decoding it back to the original form. Examples include variational
autoencoderss, which balance data compression with generative modeling,
convolutional autoencoders, which employ convolutional layers for efficient
feature learning and reconstruction, and sparse AEs, which induce sparsity,
promoting selectivity in feature representation, among others. The encoding
equation typically involves a mapping function, such as $h=f(x)$, where $h$ is
the encoded representation and $x$ is the input. The decoding equation is the
reconstruction of the input, often expressed as $r=g(h)$, where $r$ is the
reconstructed output and $g$ is the decoding function. AEs find applications
in data compression, denoising, feature learning, and more. Recently, many
research papers for CI have been proposed that are based on AE.

As a point of the case, the scientific paper [16] delves into the pivotal
objective of enhancing speech perception for CI users in noisy conditions,
recognizing the critical role of noise reduction (NR) in this pursuit. The
proposed method, named DDAE-NR, has been proven effective in restoring clean
speech. The study focuses on evaluating the DDAE-based NR using envelope-based
vocoded speech, mimicking CI devices. The procedure of DDAE-based NR can be
split into two main stages as shown in Figure 10: training and testing. During
the training phase, a collection of pairs of noisy and clean speech signals is
prepared. These signals are initially transformed into the frequency domain
using an fast Fourier transform (FFT). The logarithmic amplitudes of the noisy
and clean speech spectra are then used as inputs and outputs, respectively,
for the DDAE model.

Key findings underscore the superior intelligibility of DDAE-based NR in
vocoded speech compared to state-of-the-art (SOTA) conventional methods,
indicating its potential implementation in CI speech processors. However, the
study acknowledges the use of noise-vocoded speech simulation for evaluation
and emphasizes the need for further validation with real CI recipients in
clinical settings, addressing potential inconsistencies in the transition to
actual CI devices.

Figure 10:  The configuration of a DDAE-based NR system for enhancing speech
perception [16].

A zero-delay deep autoencoder (DAE) is proposed in [21] for compressing and
transmitting electrical stimulation patterns generated by CIs. The goal is to
conserve battery power in wireless transmission while maintaining low latency,
which is crucial for speech perception in CI users. The DAE architecture is
optimized using Bayesian optimization and the STOI. The results show that the
proposed DAE achieves equal or superior speech understanding compared to audio
codecs, with reference vocoder STOI scores at 13.5 kbit/s. This approach
offers a promising solution for efficient and real-time compression of CI
stimulation patterns, addressing the constraints of low latency and battery
power consumption. Moreover, The research in [60] focuses on achieving
accurate segmentation of the vestibule in CT images, a crucial step for
clinical diagnosis of congenital ear malformations and CIs. The challenges
addressed include the small size and irregular shape of the vestibule, making
segmentation difficult, and the limited availability of labelled samples due
to high labour costs. To overcome these challenges, the proposed method
introduces a vestibule segmentation network within a basic encoder-decoder
framework. Key innovations include the incorporation of a residual channel
attention (Res-CA) block for channel attention, a global context-aware pyramid
feature extraction (GCPFE) module for global context information, an , active
contour with elastic loss (ACE-Loss) function for detailed boundary learning,
and a deep supervision (DS) mechanism to enhance network robustness. The
network architecture utilizes ResNet34 as the backbone with skip connections
for multi-level feature fusion. Results showcases a high performance, and are
supported by comprehensive comparisons, ablation studies, and visualized
segmentation outcomes. The study also acknowledges limitations, such as
reliance on professional annotations.

In addition, the study presented in [74] aims to enhance the accuracy and
robustness of intra cochlear anatomy (ICA) segmentation, a vital component in
preoperative decisions, insertion planning, and postoperative adjustments for
CI procedures. The ICA includes structures such as scala tympani (ST), scala
vestibul (SV), and the active region (AR). The researchers employed two
segmentation methods, active shape model (ASM) and DL based on 3D U-Net AE,
and combined them to achieve improved accuracy and robustness. A two-level
training strategy involved pretraining on clinical CTs using ASM and fine-
tuning on specimens’ CTs with ground truth. Results demonstrated that DL
methods outperformed ASM in accuracy. While a trade-off between accuracy and
robustness was observed, the combined DL and ASM approach showed improvements
in both aspects. The study concludes that the proposed DL and ASM method
effectively balances accuracy and robustness for ICA segmentation,
highlighting the potential of DL-based methods, especially when integrated
with ASM, to enhance CI procedures.

The proposed min-max similarity (MMS) methodology in [61] represents a
groundbreaking approach to semi-supervised segmentation networks, particularly
in the context of medical applications such as endoscopy surgical tool
segmentation and CI surgery. MMS is introduced through dual-view training with
contrastive learning, utilizing classifiers and projectors to create negative,
positive, and negative pairs. The inclusion of pixel-wise contrastive loss
ensures the consistency of unlabeled predictions. In the evaluation phase, MMS
was tested on four public endoscopy surgical tool segmentation datasets and a
manually annotated CI surgery dataset. The results demonstrate its superiority
over state-of-the-art semi-supervised and fully supervised segmentation
algorithms, both quantitatively and qualitatively. Notably, MMS exhibited
successful recognition of unknown surgical tools, providing reliable
predictions, and achieved real-time video segmentation with an impressive
inference speed of about 40 frames per second. This signifies the potential of
MMS as a highly effective and efficient tool in medical image segmentation,
showcasing its applicability in real-world surgical scenarios.

Similarly, The primary aim of the study in [75] is to devise an automated
method for the segmentation and measurement of the human cochlea in ultra-
high-resolution (UHR) CT-images. The objective is to explore variations in
cochlear size to enhance outcomes in cochlear surgery through personalized
implant planning. Initially, the input scans undergo a two-step process using
a detection module and a pixel-wise classification module for cochlea
localization and segmentation, respectively using an AE as illustrated in
Figure 11. The detection module reduces the search area for the classification
module, improving algorithm speed and reducing false positives. Both modules
are trained on image patches, allowing for a larger training set size by
generating multiple examples from each scan. The segmented cochlear structure
then proceeds to a final module that combines DL and thinning algorithms to
extract patient-specific anatomical measurements. DL is employed in each step
to leverage its ability to learn directly from input data, providing automatic
results without the need for user-adjustable parameters during testing.

Figure 11:  Encoder-decoder network used in the pixel-wise classification
model [75].

###  3.7 RL-based methods

DRL is a subfield of ML that enable agents to learn and make decisions in
complex environments. It involves training an agent to interact with an
environment, learn from the outcomes of its actions, and optimize its behavior
over time [76]. In traditional RL, agents learn by trial and error, receiving
feedback in the form of rewards or penalties for their actions. However, deep
reinforcement learning (DRL) incorporates DNN, make it capable of learning
complex patterns and representations from raw data. This allows DRL agents to
handle high-dimensional input spaces, such as images or sensor data, and make
more sophisticated decisions. Figure 12 illustrate the priciple of RL.

Figure 12: RL principle.

The paper presented by radutoiu et al. [32] presents a novel method for
accurately localizing regions of interests in the inner ear using DRL. The
proposed method addresses the challenges of robust ROI extraction in full head
CT scans, which is crucial for CI surgery. The approach utilizes communicative
multi-agent RL and landmarks specifically designed to extract orientation
parameters. The method achieves an average estimated error of 1.07 mm for
landmark localization. The extracted ROIs demonstrate an IoU of 0.84 and a
dice similarity coefficient of 0.91, conducted over 140 full head CT scans,
showing promising results for automatic ROI extraction in medical imaging. In
addition, Lopez et al. presents in [77] a pipeline for characterizing facial
and cochlear nerves in CT scans using DRL. Key landmarks around these nerves
are located using a communicative multi-agent DRL model. The pipeline includes
automated measurement of the cochlear nerve canal diameter, extraction and
segmentation of the cochlear nerve cross-section, and path selection for the
facial nerve characterization. The pipeline was developed and evaluated using
119 clinical CT images. The results show accurate characterizations of the
nerves in the cochlear region, providing reliable measurements for computer-
aided diagnosis and surgery planning. The proposed approach demonstrates the
potential of DRL for landmark detection in challenging medical imaging tasks.

##  4 Applications of DL-based medical CI

This section explores the application of deep learning in the field of
cochlear implants, encompassing tasks such as speech denoising and
enhancement, segmentation for precise identification and analysis of cochlear
structures, thresholding, imaging, localization of CI, and more. Figure 13
provides a comprehensive overview of AI-based applications for CIs and theirs
associated benefits. Furthermore, Table 5 summarizes various applications
based on AI techniques, highlighting their performance, and pros and cons.

Figure 13: Taxonomy of AI-based applications for CIs and their benefits.

###  4.1 Speech denoising and enhancement

The integration of ML, and DL has proven invaluable in the field of CIs.
Researchers have harnessed these technologies to tackle numerous challenges
and enhance speech perception for individuals with hearing impairments. The
work [17, 78] employed DDAE approach to reduce unwanted background noise in
speech signals. However, Lai et al. in [78] devised a NR system that employed
a noise classifier and DDAE, specifically tailored for Mandarin-speaking CI
recipients. The proposed schemes [22, 26, 79] aim to perform end-to-end speech
denoising, with the goal of enhancing speech intelligibility in noisy
environments. Gajecki et al. [22, 26] employed DNN to develop the Deep ACE
method, while Healy et al. in [79] utilized DNN to separate speech from
background noise. These examples underscore the broad spectrum of applications
of AI, particularly DL techniques, in addressing challenges related to noise
reduction and enhancing speech intelligibility in CIs applications.

Moreover, Kang et al. [28] used DL-based speech enhancement algorithms to
optimize speech perception for CI recipients. Their approach achieved a
balance between noise suppression and speech distortion by experimenting with
different loss functions. Hu et al. [80] developed environment-specific noise
suppression algorithms for CIs using ML techniques. They improved the
processed sound by classifying and selecting envelope amplitudes based on the
SNR in each channel. Banerjee et al. [81] employed online unsupervised
algorithms to learn features from the speech of individuals with severe-to-
profound hearing loss, aiming to enhance the audibility of speech through
modified signal processing. Li et al. [82] developed an improved NR system for
CIs using DL, specifically DDAE, and knowledge transfer technology. Their goal
was to enhance speech intelligibility in noisy conditions. Fischer et al. [29]
utilized DL-based virtual sensing of head-mounted microphones to improve
speech signals in cocktail party scenarios for individuals with hearing loss,
resulting in enhanced speech quality and intelligibility, particularly in
noisy environments. These studies exemplify the versatility of AI and DL in
addressing various challenges associated with CIs, including NR, speech
enhancement, and improved speech perception. Furthermore, the paper by Chu et
al. [23] explores the application of ML algorithms to mitigate the effects of
reverberation and noise in CIs, to improve speech intelligibility for
individuals with severe hearing loss.

Table 5: Summary of the performance and limitations of specific DL
applications dedicated to CIs. In cases where multiple tests are conducted,
only the best performance is reported.

Ref. |  Year |  DLM |  Description |  BP (%) |  Limitations |  PLA
---|---|---|---|---|---|---
[82] |  2021 |  NC and DDAE_T |  The DL model comprises Siren noise at 6dB, a classifier, and the DDAE. The transfer learning (TL) is incorporated to help reduce the number of parameters in the model. |  PESQ= 3.037
STOI= 0.843 |  The performance of the DL model can be degraded when there is a mismatch between the training and testing data. |  No
[22] |  2023 |  DeepACE |  Takes the raw audio signal as input and generates a denoised electrodogram, which represents the electrical stimulation patterns applied to the electrodes over time. |  STOI =0.807 |  Heavy data reliance impedes performance with insufficient or biased data, posing generalization challenges. Computational complexity, interpretability issues, and vulnerability are concerns. |  Yes151515https://github.com/APGDHZ/DeepACE2.0
[29] |  2021 |  U-Net |  The proposed approach involves using deep virtual sensing to estimate microphone signals, improving speech quality in cocktail party scenarios for hearing aid and CI users. |  PESQ=2, STOI=0.53, SI-SDR=-28.91dB |  Challenges include long delay impracticality for hearing aids, binaural cues omission, insufficient network optimization, and exploring robustness in high-reverberation environments. |  No
[83] |  2022 |  CNN |  A web-based research platform called Nautilus, which utilizes DL techniques for automated image processing in cochlear implantation-related studies. |  DCS= 80 $\pm 3$%, ASD= 0.19 $\pm 0.04$mm |  Dependence on large amounts of annotated training data poses a challenge. Ensuring generalizability across various imaging conditions and populations is crucial. |<EMAIL_ADDRESS>available upon reasonable request
[60] |  2023 |  AE+Res-CA |  Is a vestibule segmentation network for CT images. It is based on the basic encoder-decoder framework and incorporates Res-CA Block, GCPFE and ACE-Loss Function |  DCS=94.77 $\pm 2.45$%, ASD=0.06 mm |  Limited training data, small object handling, generalization capability, and computational resource constraints are areas of concern. |  No
[84] |  2019 |  cGAN |  Is a DL-based method for accurately localizing CI electrode contacts in CT images, facilitating customization of CI settings. |  success rate of localization =96.7% |  coarse resolution, metal artifacts, difficulty isolating contacts from cortical bones, site-dependent image quality, and variability among electrode array manufacturers. |  No
[32] |  2022 |  DRL |  Is a novel method for accurately extracting ROIs in the inner ear from CT scans. The approach achieves high precision and demonstrates promising results for surgical planning in cochlear implantation. |  IoU=0.84
DSC=0.91 |  Small dataset, image quality variability, and computational requirements not specified. |  No
[75] |  2020 |  U-Net |  Is a multi-scale DL framework for automatic segmentation and measurement of the human cochlea in clinical ultra-high-resolution CT images, with potential applications in personalized CI surgery. |  DSC=0.90 $\pm 0.03$
BF=0.95$\pm 0.03$ |  The work include a small dataset, potential variability in image quality, lack of external validation, and limited assessment of clinical utility and computational requirements. |  No
[85] |  2023 |  UNETR |  Explores the feasibility of using a DL method based on the UNETR model for automatic segmentation of the cochlea in temporal bone CT images. |  DSC=0.92 |  The small dataset used, the variability in image quality, and the absence of specifications regarding computational requirements. |  No
[86] |  2019 |  3D U-NET |  Is a two-level training approach using a DL method to accurately segment the intra-cochlear anatomy in head CT scans. The method combines an active shape model-based method and a 3D U-Net model |  DSC=0.87 |  Limited dataset, variable image quality, lack of external validation, limited assessment of clinical utility, and no specifications on computational requirements. |  No
[34] |  2023 |  AlexNet |  The study assesses repeatability, thresholds, and tonotopic patterns using a DL-based algorithm, providing insights into inner ear function and potential clinical applications. |  Acc=83.8% |  Potential dependence on the quality of the input data, limited generalizability to different patient populations or implant systems, and the need for further external validation and comparison with expert visual inspection. |  No
|  |  |  |  |  |

Abbreviations: Deep learning model (DLM); Best performance (BP); Project link
availability (PLA).

###  4.2 Imaging

DL methods have revolutionized CI applications by leveraging imaging data for
enhanced analysis and optimization. Hussain et al. [87] employed image
analysis tools, such as the oticon medical nautilus software, to automatically
detect landmarks and extract clinically relevant parameters from cochlear CT
images. This approach provides valuable insights into cochlear morphology,
facilitating the development of less traumatic electrode arrays for cochlear
implantation. Zhang et al. [57] focused on automatically detecting the
presence and location of inner ears in head CT images, aiming to assist in
image-guided CI programming for patients with profound hearing loss. Regodic
et al. [88] introduced an algorithm that utilizes a CNN for automatic fiducial
marker detection and localization in CT images, enhancing registration
accuracy, reducing human errors, and shortening intervention time in computer-
assisted surgeries. Margeta et al. [83] presented Nautilus, a web-based
research platform that employs AI and image processing techniques for
automated cochlear image analysis. This platform enables accurate delineation
of cochlear structures, detection of electrode locations, and personalized
pre- and post-operative metrics, facilitating clinical exploration in cochlear
implantation studies. Li et al. [89] proposed the integration of DL techniques
into a clinical $\mu$CT system to optimize imaging performance, improve
reconstruction accuracy, and enhance diagnostic capabilities in temporal bone
imaging and other clinical applications. Wang et al. [68] addressed the
reduction of metal artifacts in post-operative CI CT imaging using a 3D GAN,
enabling better analysis of electrode positions and assessment of CI
insertion. These advancements highlight the significant role of DL, ML, and AI
in leveraging imaging data for improved CI analysis, design, and surgical
procedures.

In addition to the previous advancements, DL and AI have been applied to
various aspects of CI applications using imaging data. Chen et al. [60]
utilize AI for accurate vestibule segmentation in CT images, which plays a
crucial role in the clinical diagnosis of congenital ear malformations and CI
procedures. Kugler et al. [90] employ AI techniques to accurately estimate
instrument pose from X-ray images in temporal bone surgery, enabling high-
precision navigation and facilitating minimally invasive procedures. Waldeck
et al. [91] develop an ultra-fast algorithm that utilizes automated cochlear
image registration to detect misalignment in CIs, significantly reducing the
time required for diagnosis compared to traditional multiplanar reconstruction
analysis. Finally, Chen et al. [92] focus on creating a three-dimensional
finite element model of the brain based on magnetic resonance imaging (MRI)
data to analyze and optimize the current flow path induced by CIs. This
application of AI contributes to the improvement of implant design in the
future. These innovative approaches demonstrate the diverse applications of
DL, ML, and AI in CI research, ranging from scene understanding to precise
segmentation, instrument pose estimation, misalignment detection, and implant
design optimization.

###  4.3 Segmentation

DL, ML, and AI have revolutionized CI segmentation, enabling precise
identification and analysis of cochlear structures in various imaging
modalities. Li et al. [85] applied a UNETR model to automatically segment
cochlear structures in temporal bone CT images, enhancing surgical planning
and cochlear implantation outcomes. Reda et al. [93] developed an automatic
segmentation method for intra-cochlear anatomy in post-implantation CT scans,
facilitating the customization of sound processing strategies for individual
CI recipients. Moudgalya et al. [94] employed a modified V-Net CNN to segment
cochlear compartments in $\mu$CT images, enabling precise quantification of
local drug delivery for potential treatment of sensorineural hearing loss.
Wang et al. [95] focused on metal artifact reduction and intra-cochlear
anatomy segmentation in CT images using a multi-resolution multi-task deep
network, benefiting CI recipients. Heutink et al. [75] developed a DL
framework for the automatic segmentation and analysis of cochlear structures
in ultra-high-resolution CT images, providing accurate measurements for
personalized implant planning in cochlear surgery. Zhang et al. [86] utilized
a 3D U-Net DL method to achieve accurate segmentation of intra-cochlear
anatomy in head CT images, facilitating optimal programming of CIs and
improving hearing outcomes. These studies highlight the significant impact of
DL, ML, and AI in advancing CI segmentation, ultimately leading to improved
patient care and treatment outcomes.

###  4.4 Thresholding

DL, ML, and AI have been instrumental in the field of CIs, particularly in
thresholding applications. Kuczapski et al. [96] developed a software tool
that utilizes AI to estimate and monitor the effective stimulation threshold
(EST) levels in CI recipients. By leveraging patient data, audiograms, and
fitting settings, this tool aids in the fitting process and predicts changes
in hearing levels, enhancing personalized care. Botros et al. [97] introduced
AutoNRT, an automated system that combines ML and pattern recognition to
measure electrically evoked compound action potential (ECAP) thresholds with
the Nucleus Freedom CI. This objective fitting system streamlines clinical
procedures and ensures precise and efficient threshold measurements.
Furthermore, Schuerch et al. [34] utilized a DL-based algorithm to objectively
evaluate and analyze ECochG signals. This algorithm enables the assessment of
ECochG measurement repeatability, comparison with audiometric thresholds, and
identification of signal patterns and tonotopic behavior in CI recipients.
Through the integration of DL, machine ML, and AI, these studies have
significantly advanced thresholding techniques in CI applications, leading to
improved fitting accuracy, streamlined procedures, and objective evaluation of
signal responses.

###  4.5 Localization of CI

DL methods have been instrumental in CI localization applications, providing
accurate and automated solutions. Chi et al. [84] proposed a DL-based method
for precise localization of electrode contacts in CT images. Their approach
utilized cGANs to generate likelihood maps, which were then processed to
estimate the exact location of each contact. Radutoiu et al. [32] focused on
the automatic extraction of ROIs in full head CT scans of the inner ear. By
leveraging AI, they achieved high precision in ROI localization, facilitating
accurate surgical planning for insertion. Noble et al. [98] and Zhao et al.
[99, 100], developed AI-based systems to automatically identify and position
electrode arrays in CT images. These technologies enable large-scale analyses
of the relationship between electrode placement and hearing outcomes, leading
to potential advancements in implant design and surgical techniques. Heutink
et al. [75] employed DL for the automatic segmentation and localization of the
cochlea in ultra-high-resolution CT images. This approach allows for precise
measurements that can be used in personalized planning, reducing the risk of
intracochlear trauma and optimizing surgical outcomes. These studies showcase
the significant contributions of DL and AI in localization applications,
enabling accurate and efficient identification, positioning, and analysis of
electrode arrays and facilitating improved surgical planning and outcomes.

###  4.6 Other

DL techniques have been employed in various CI applications, showcasing their
potential to enhance hearing outcomes and improve device performance. Bermejo
et al. [101] introduced a decision support system using a novel probabilistic
graphical model to optimize CI parameters based on audiological tests and the
current device status, aiming to optimize the user’s hearing ability.
Castaneda et al. [102] focused on the use of blind source separation (BSS) and
independent component analysis (ICA) to identify auditory evoked potentials
and isolate artifacts in children with CIs, enabling improved assessment of
auditory function. Incerti et al. [103] investigated the impact of varying
cross-over frequency settings for EAS on binaural speech perception,
localization, and functional performance in adults with CIs and residual
hearing, providing valuable insights for personalized device programming.
Katthi et al. [104] developed a DL framework based on canonical correlation
analysis (CCA) to decode the auditory brain, establishing a strong correlation
between audio input and brain activity measured through EEG recordings. This
research has implications for decoding human auditory attention and improving
CIs by leveraging the power of DL.

##  5 Open issues and future directions

While significant strides have been achieved in integrating AI into CIs,
numerous research lacunae persist, offering avenues for further advancements
in the field. Here are several potential realms warranting exploration in
future studies:

Real-time signal processing and personalized design:  Investigating real-time
adaptive signal processing methods employing AI algorithms has the potential
to enhance sound processing for CI recipients, yielding enhanced speech
intelligibility outcomes. Enhancements in adaptability to dynamic acoustic
environments and real-time optimization of stimulation parameters have the
capacity to substantially enhance CI performance. The authors have observed a
gap in the study and implementation of AI models tailored for CI on real-time
platforms like field programmable gate arrays (FPGAs). Further research in
this burgeoning area holds promise for adapting a variety of existing AI
models to enhance real-time capabilities.

Besides, tailoring CIs to meet the unique needs of individual users poses a
significant challenge. Investigating AI-driven methodologies leveraging
personal data to personalize device configurations based on factors like
physiological, auditory, and neural feedback during mobility can enhance both
individual outcomes and overall satisfaction.

Predicting the long-term effects: Gaining insight into the enduring CIs is
vital for enhancing patient selection, counseling, and device advancement.
Utilizing AI methods to shift through extensive datasets can pinpoint the
predictive elements influencing sustained success. These factors may encompass
pre-implantation attributes, surgical approaches, and auditory rehabilitation.
Constructing predictive models using AI algorithms can furnish valuable
perspectives on long-term consequences, thereby informing clinical judgments.

Incorporating multiple sensory and modalities: CIs traditionally prioritize
the reinstatement of auditory experiences. Yet, enriching the perception and
comprehension of sound can be achieved by integrating additional sensory
dimensions like vision and touch, resulting in a multi-modal approach.
Investigating AI-driven techniques that amalgamate inputs from various senses
to enhance speech recognition, spatial sound perception, and overall auditory
understanding presents a promising direction for future exploration. Besides,
CIs in both ears, when paired with AI algorithms, can enhance speech
comprehension. By analyzing sound patterns from both implants, AI adjusts
settings to optimize signal processing, improving overall accuracy and clarity
of speech perception for users with bilateral implants, and enhancing their
auditory experience and communication abilities, as investigated in [105].
However, extensive research possibilities are required to tailor solutions
with CI hardware capabilities, by taking into account computation cost, and AI
model complexity.

Empowering AI-based CI using DTL: DTL is a highly efficient DL technique
enabling the transfer of knowledge from pre-trained models, trained on
millions of speech corpora and/or images, to train smaller models with limited
data availability. This approach offers significant advantages in producing
lightweight AI models suitable for devices with limited computational
resources, such as CIs. Only a limited number of studies have explored the
impact of deep transfer learning (DTL) on CI, as demonstrated in [82], which
has received relatively little attention from researchers. We anticipate
further exploration of this promising technique, particularly through the
utilization of various DTL sub-techniques, such as domain adaptation,
transductive methods like cross-lingual transfer, cross-corpus transfer, zero-
shot learning, fine-tuning, among others [2].

Ensuring data privacy through FL: FL facilitates collaborative model training
across decentralized devices by aggregating local updates rather than
centralizing data. This preserves user privacy and enhances model performance,
particularly beneficial in healthcare applications [106]. Gathering
comprehensive datasets is challenging due to rare anomaly cases and privacy
concerns. federated learning (FL) addresses this by training models on
distributed, encrypted data from multiple sources, ensuring privacy while
maintaining efficacy. Researchers have yet to fully explore FL-based model
building for CI, neglecting the potential to construct efficient AI models
capable of accommodating diverse classes. Further investigation into this
promising technique is warranted, with potential for significant advancements
in model robustness and versatility. Moreover, this approach could lead to the
development of a pretrained model utilizing FL, which could be seamlessly
integrated with DTL.

Transformers-based CI techniques:  Transformer-based techniques, such as
connectionist temporal classification (CTC), bidirectional encoder
representations from transformer (BERT), and others, proved in the literature
that have the potential to greatly enhance the functioning of ASR [107, 108].
By leveraging the self-attention mechanism, transformers can improve speech
intelligibility by effectively suppressing background noise and modeling long-
range dependencies. They can also aid in acoustic scene analysis, separating
and prioritizing important auditory information in complex environments.
Transformers can build language models that enhance ASR systems, improving
speech comprehension for users. Additionally, transformers enable personalized
sound processing by adapting stimulation patterns and processing parameters
based on user-specific preferences. They facilitate multi-modal integration,
combining audio and visual inputs to enhance speech recognition and sound
localization. Furthermore, transformers support long-term learning and
adaptation, continually optimizing CI performance over time. These
advancements offer promising prospects for improving auditory experiences and
overall quality of life for CI users.

Exploring chat-bots-based CI capabilities:  Chat-bot techniques offer several
opportunities to enhance the functioning of CIs. They can provide real-time
support, troubleshooting, and personalized rehabilitation programs for users,
empowering them to address common issues and improve their auditory skills.
Chat-bots enable remote monitoring, allowing users to share data and receive
adjustments to their device settings without in-person appointments. They also
offer emotional and psychological support, fostering a sense of community and
well-being. Chat-bots contribute to data collection for research and
development, aiding in the improvement of CI technology and rehabilitation
protocols. Additionally, chat-bots employML to continuously learn from user
interactions, improving their responses and understanding over time. These
techniques have the potential to enhance the overall user experience,
outcomes, and accessibility of CI services.

##  6 Conclusion

This review has provided a comprehensive overview of the advancements in AI
algorithms for CIs applications and their impact on ASR and speech
enhancement. The integration of AI methods has brought cutting-edge strategies
to address the limitations and challenges faced by traditional signal
processing techniques in the context of CIs. Moreover, the application of AI
in CI has led to the emergence of new datasets and evaluation metrics,
offering alternative methods for validating proposed schemes without the need
for human surgical intervention and traditional tests. The review highlighted
the role of ASR in optimizing speech perception and understanding for CI
users, contributing to the improvement of their quality of life. ASR not only
enhances basic speech recognition but also aids in the recognition of
environmental sounds, enabling a more immersive auditory experience.
Furthermore, ASR finds applications in authentication systems, event
recognition, source separation, and speaker recognition, extending its reach
beyond communication. Various AI algorithms, belong to ML and DL, have been
explored in the context of CIs, demonstrating promising results in speech
synthesis and noise reduction. These algorithms have shown the potential to
overcome challenges associated with multiple sources of speech, environmental
noise, and other complex scenarios. The review has summarized and commented on
the best results obtained, providing valuable insights into the capabilities
of AI algorithms in this biomedical field. Moving forward, the review suggests
future directions to bridge existing research gaps in the domain of AI
algorithms for CIs. It emphasizes the need for high-quality data inputs,
algorithm transparency, and collaboration between researchers, clinicians, and
industry experts. Addressing these aspects will facilitate the development of
more accurate and efficient AI algorithms for CI, ultimately benefiting
individuals with hearing impairments. The integration of advanced AI
algorithms has the potential to revolutionize the field of CIs, providing
individuals with hearing impairments to better communicate and engage with the
world around them. Continued research and development in this area hold great
promise for the future of CI technology.

## Acknowledgement

The second author acknowledges that the study was partially funded by the
Algerian Ministry of Higher Education and Scientific Research (Grant No.
PRFU–A25N01UN260120230001).

## References

  * [1] M. B. Er, E. Isik, I. Isik, Parkinson’s detection based on combined cnn and lstm using enhanced speech signals with variational mode decomposition, Biomedical Signal Processing and Control 70 (2021) 103006.
  * [2] H. Kheddar, Y. Himeur, S. Al-Maadeed, A. Amira, F. Bensaali, Deep transfer learning for automatic speech recognition: Towards better generalization, Knowledge-Based Systems 277 (2023) 110851.  doi:10.1016/j.knosys.2023.110851.
  * [3] S. Hu, X. Shang, Z. Qin, M. Li, Q. Wang, C. Wang, Adversarial examples for automatic speech recognition: Attacks and countermeasures, IEEE Communications Magazine 57 (10) (2019) 120–126.
  * [4] H. Kheddar, D. Megías, High capacity speech steganography for the g723. 1 coder based on quantised line spectral pairs interpolation and cnn auto-encoding, Applied Intelligence 52 (8) (2022) 9441–9459.
  * [5] H. Kheddar, M. Hemis, Y. Himeur, D. Megías, A. Amira, Deep learning for steganalysis of diverse data types: A review of methods, taxonomy, challenges and future directions, Neurocomputing (2024) 127528doi:https://doi.org/10.1016/j.neucom.2024.127528.
  * [6] H. Kheddar, D. Megias, M. Bouzid, Fourier magnitude-based steganography for hiding 2.4 kbps melp secret speech, in: 2018 International Conference on Applied Smart Systems (ICASS), IEEE, 2018, pp. 1–5.
  * [7] N. Singh, A. Agrawal, R. Khan, Voice biometric: A technology for voice based authentication, Advanced Science, Engineering and Medicine 10 (7-8) (2018) 754–759.
  * [8] T.-E. Chen, S.-I. Yang, L.-T. Ho, K.-H. Tsai, Y.-H. Chen, Y.-F. Chang, Y.-H. Lai, S.-S. Wang, Y. Tsao, C.-C. Wu, S1 and s2 heart sound recognition using deep neural networks, IEEE Transactions on Biomedical Engineering 64 (2) (2016) 372–380.
  * [9] M. G. Crowson, V. Lin, J. M. Chen, T. C. Chan, Machine learning and cochlear implantation—a structured review of opportunities and challenges, Otology & Neurotology 41 (1) (2020) e36–e45.
  * [10] A. Manero, K. E. Crawford, H. Prock-Gibbs, N. Shah, D. Gandhi, M. J. Coathup, Improving disease prevention, diagnosis, and treatment using novel bionic technologies, Bioengineering & Translational Medicine 8 (1) (2023) e10359.
  * [11] B. S. Wilson, D. L. Tucci, D. A. Moses, E. F. Chang, N. M. Young, F.-G. Zeng, N. A. Lesica, A. M. Bur, H. Kavookjian, C. Mussatto, et al., Harnessing the power of artificial intelligence in otolaryngology and the communication sciences, Journal of the Association for Research in Otolaryngology 23 (3) (2022) 319–349.
  * [12] K. L. D’Onofrio, F.-G. Zeng, Tele-audiology: Current state and future directions, Frontiers in Digital Health 3 (2022) 788103.
  * [13] F. Henry, M. Glavin, E. Jones, Noise reduction in cochlear implant signal processing: A review and recent developments, IEEE reviews in biomedical engineering (2021).
  * [14] O. Macherey, R. P. Carlyon, Cochlear implants, Current Biology 24 (18) (2014) R878–R884.
  * [15] N. Y.-H. Wang, H.-L. S. Wang, T.-W. Wang, S.-W. Fu, X. Lu, H.-M. Wang, Y. Tsao, Improving the intelligibility of speech for simulated electric and acoustic stimulation using fully convolutional neural networks, IEEE Transactions on Neural Systems and Rehabilitation Engineering 29 (2020) 184–195.
  * [16] Y.-H. Lai, F. Chen, S.-S. Wang, X. Lu, Y. Tsao, C.-H. Lee, A deep denoising autoencoder approach to improving the intelligibility of vocoded speech in cochlear implant simulation, IEEE Transactions on Biomedical Engineering 64 (7) (2016) 1568–1578.
  * [17] S.-S. Wang, Y. Tsao, H.-L. S. Wang, Y.-H. Lai, L. P.-H. Li, A deep learning based noise reduction approach to improve speech intelligibility for cochlear implant recipients in the presence of competing speech noise, in: 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), IEEE, 2017, pp. 808–812.
  * [18] L. L. Wong, S. D. Soli, S. Liu, N. Han, M.-W. Huang, Development of the mandarin hearing in noise test (mhint), Ear and hearing 28 (2) (2007) 70S–74S.
  * [19] T. Arias-Vergara, P. Klumpp, J. C. Vasquez-Correa, E. Nöth, J. R. Orozco-Arroyave, M. Schuster, Multi-channel spectrograms for speech processing applications using deep learning methods, Pattern Analysis and Applications 24 (2021) 423–431.
  * [20] T. Arias-Vergara, J. C. Vasquez-Correa, S. Gollwitzer, J. R. Orozco-Arroyave, M. Schuster, E. Nöth, Multi-channel convolutional neural networks for automatic detection of speech deficits in cochlear implant users, in: Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 24th Iberoamerican Congress, CIARP 2019, Havana, Cuba, October 28-31, 2019, Proceedings 24, Springer, 2019, pp. 679–687.
  * [21] R. Hinrichs, F. Ortmann, J. Ostermann, Vector-quantized zero-delay deep autoencoders for the compression of electrical stimulation patterns of cochlear implants using stoi, in: 2022 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), IEEE, 2022, pp. 165–170.
  * [22] T. Gajecki, Y. Zhang, W. Nogueira, A deep denoising sound coding strategy for cochlear implants, IEEE Transactions on Biomedical Engineering (2023).
  * [23] K. Chu, C. Throckmorton, L. Collins, B. Mainsah, Using machine learning to mitigate the effects of reverberation and noise in cochlear implants, in: Proceedings of Meetings on Acoustics, Vol. 33, AIP Publishing, 2018.
  * [24] R. Grimm, M. Pettinato, S. Gillis, W. Daelemans, Simulating speech processing with cochlear implants: How does channel interaction affect learning in neural networks?, Plos one 14 (2) (2019) e0212134.
  * [25] Z. Feng, Y. Tsao, F. Chen, Preservation of interaural level difference cue in a deep learning-based speech separation system for bilateral and bimodal cochlear implants users, in: 2022 International Workshop on Acoustic Signal Enhancement (IWAENC), IEEE, 2022, pp. 1–5.
  * [26] T. Gajecki, W. Nogueira, An end-to-end deep learning speech coding and denoising strategy for cochlear implants, in: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2022, pp. 3109–3113.
  * [27] I. Hochmair-Desoyer, E. Schulz, L. Moser, M. Schmidt, The hsm sentence test as a tool for evaluating the speech understanding in noise of cochlear implant users., The American journal of otology 18 (6 Suppl) (1997) S83–S83.
  * [28] Y. Kang, N. Zheng, Q. Meng, Deep learning-based speech enhancement with a loss trading off the speech distortion and the noise residue for cochlear implants, Frontiers in Medicine 8 (2021) 740123.
  * [29] T. Fischer, M. Caversaccio, W. Wimmer, Speech signal enhancement in cocktail party scenarios by deep learning based virtual sensing of head-mounted microphones, Hearing research 408 (2021) 108294.
  * [30] T. Fischer, M. Caversaccio, W. Wimmer, Multichannel acoustic source and image dataset for the cocktail party effect in hearing aid and implant users, Scientific data 7 (1) (2020) 440.
  * [31] S. Tahmasebi, T. Gajecki, W. Nogueira, Design and evaluation of a real-time audio source separation algorithm to remix music for cochlear implant users, Frontiers in Neuroscience 14 (2020) 434.
  * [32] A.-T. Radutoiu, F. Patou, J. Margeta, R. R. Paulsen, P. López Diez, Accurate localization of inner ear regions of interests using deep reinforcement learning, in: International Workshop on Machine Learning in Medical Imaging, Springer, 2022, pp. 416–424.
  * [33] K. Schuerch, W. Wimmer, A. Dalbert, C. Rummel, M. Caversaccio, G. Mantokoudis, S. Weder, Objectification of intracochlear electrocochleography using machine learning, Frontiers in neurology 13 (2022) 943816.
  * [34] K. Schuerch, W. Wimmer, C. Rummel, M. D. Caversaccio, S. Weder, Objective evaluation of intracochlear electrocochleography: repeatability, thresholds, and tonotopic patterns, Frontiers in neurology 14 (2023).
  * [35] K. Schuerch, W. Wimmer, A. Dalbert, C. Rummel, M. Caversaccio, G. Mantokoudis, T. Gawliczek, S. Weder, An intracochlear electrocochleography dataset-from raw data to objective analysis using deep learning, Scientific data 10 (1) (2023) 157.
  * [36] Y. Habchi, Y. Himeur, H. Kheddar, A. Boukabou, S. Atalla, A. Chouchane, A. Ouamane, W. Mansoor, Ai in thyroid cancer diagnosis: Techniques, trends, and future directions, Systems 11 (10) (2023) 519.
  * [37] C. H. Taal, R. C. Hendriks, R. Heusdens, J. Jensen, A short-time objective intelligibility measure for time-frequency weighted noisy speech, in: 2010 IEEE international conference on acoustics, speech and signal processing, IEEE, 2010, pp. 4214–4217.
  * [38] P. J. Govaerts, B. Vaerenberg, G. De Ceulaer, K. Daemers, C. De Beukelaer, K. Schauwers, Development of a software tool using deterministic logic for the optimization of cochlear implant processor programming, Otology & Neurotology 31 (6) (2010) 908–918.
  * [39] B. Vaerenberg, P. J. Govaerts, G. De Ceulaer, K. Daemers, K. Schauwers, Experiences of the use of fox, an intelligent agent, for programming cochlear implant sound processors in new users, International Journal of Audiology 50 (1) (2011) 50–58.
  * [40] M. Meeuws, D. Pascoal, I. Bermejo, M. Artaso, G. De Ceulaer, P. J. Govaerts, Computer-assisted ci fitting: Is the learning capacity of the intelligent agent fox beneficial for speech understanding?, Cochlear Implants International 18 (4) (2017) 198–206.
  * [41] J. Wathour, P. J. Govaerts, N. Deggouj, From manual to artificial intelligence fitting: two cochlear implant case studies, Cochlear implants international 21 (5) (2020) 299–305.
  * [42] S. B. Waltzman, D. C. Kelsall, The use of artificial intelligence to program cochlear implants, Otology & Neurotology 41 (4) (2020) 452–457.
  * [43] J. Wathour, P. J. Govaerts, L. Derue, S. Vanderbemden, H. Huaux, E. Lacroix, N. Deggouj, Prospective comparison between manual and computer-assisted (fox) cochlear implant fitting in newly implanted patients (2023).
  * [44] T. Eichler, W. Rötz, C. Kayser, F. Bröhl, M. Römer, A. H. Witteborg, F. Kummert, T. Sandmeier, C. Schulte, P. Stolz, et al., Algorithm-based hearing and speech therapy rehabilitation after cochlear implantation, Brain Sciences 12 (5) (2022) 580.
  * [45] J. Torresen, A. H. Iversen, R. Greisiger, Data from past patients used to streamline adjustment of levels for cochlear implant for new patients, in: 2016 IEEE Symposium Series on Computational Intelligence (SSCI), IEEE, 2016, pp. 1–7.
  * [46] F. Henry, A. Parsi, M. Glavin, E. Jones, Experimental investigation of acoustic features to optimize intelligibility in cochlear implants, Sensors 23 (17) (2023) 7553.
  * [47] C. Pavelchek, A. P. Michelson, A. Walia, A. Ortmann, J. Herzog, C. A. Buchman, M. A. Shew, Imputation of missing values for cochlear implant candidate audiometric data and potential applications, Plos one 18 (2) (2023) e0281337.
  * [48] C. Xu, F.-Y. Cheng, S. Medina, E. Eng, R. Gifford, S. Smith, Objective discrimination of bimodal speech using frequency following responses, Hearing research 437 (2023) 108853.
  * [49] H. Kheddar, M. Bouzid, D. Megías, Pitch and fourier magnitude based steganography for hiding 2.4 kbps melp bitstream, IET Signal Processing 13 (3) (2019) 396–407.
  * [50] M. G. Crowson, P. Dixon, R. Mahmood, J. W. Lee, D. Shipp, T. Le, V. Lin, J. Chen, T. C. Chan, Predicting postoperative cochlear implant performance using supervised machine learning, Otology & Neurotology 41 (8) (2020) e1013–e1023.
  * [51] P. Mikulskis, A. Hook, A. A. Dundas, D. Irvine, O. Sanni, D. Anderson, R. Langer, M. R. Alexander, P. Williams, D. A. Winkler, Prediction of broad-spectrum pathogen attachment to coating materials for biomedical devices, ACS applied materials & interfaces 10 (1) (2018) 139–149.
  * [52] Y. A. Alohali, M. S. Fayed, Y. Abdelsamad, F. Almuhawas, A. Alahmadi, T. Mesallam, A. Hagr, Machine learning and cochlear implantation: Predicting the post-operative electrode impedances, Electronics 12 (12) (2023) 2720.
  * [53] R. Islam, E. Abdel-Raheem, M. Tarique, A novel pathological voice identification technique through simulated cochlear implant processing systems, Applied Sciences 12 (5) (2022) 2398.
  * [54] J. de Nobel, A. V. Kononova, J. J. Briaire, J. H. Frijns, T. H. Bäck, Optimizing stimulus energy for cochlear implants with a machine learning model of the auditory nerve, Hearing Research 432 (2023) 108741.
  * [55] T. Bäck, H.-P. Schwefel, An overview of evolutionary algorithms for parameter optimization, Evolutionary computation 1 (1) (1993) 1–23.
  * [56] D. Baby, A. Van Den Broucke, S. Verhulst, A convolutional neural-network model of human cochlear mechanics and filter tuning for real-time applications, Nature machine intelligence 3 (2) (2021) 134–143.
  * [57] D. Zhang, J. H. Noble, B. M. Dawant, Automatic detection of the inner ears in head ct images using deep convolutional neural networks, in: Medical Imaging 2018: Image Processing, Vol. 10574, SPIE, 2018, pp. 565–572.
  * [58] W. Li, T. Gueuret, B. Lin, Poster: Slidecnn: Deep learning for auditory spatial scenes with limited annotated data, in: 2022 IEEE/ACM 7th Symposium on Edge Computing (SEC), IEEE, 2022, pp. 316–317.
  * [59] M.-H. Laves, S. Ihler, L. A. Kahrs, T. Ortmaier, Deep-learning-based 2.5 d flow field estimation for maximum intensity projections of 4d optical coherence tomography, in: Medical Imaging 2019: Image-Guided Procedures, Robotic Interventions, and Modeling, Vol. 10951, SPIE, 2019, pp. 189–195.
  * [60] M. Chen, L. Zhuo, Z. Zhu, H. Yin, X. Li, Z. Wang, Deeply supervised vestibule segmentation network for ct images with global context-aware pyramid feature extraction, IET Image Processing 17 (4) (2023) 1267–1279.
  * [61] A. Lou, K. Tawfik, X. Yao, Z. Liu, J. Noble, Min-max similarity: A contrastive semi-supervised deep learning network for surgical tools segmentation, IEEE Transactions on Medical Imaging (2023).
  * [62] Z. Wang, C. Vandersteen, T. Demarcy, D. Gnansia, C. Raffaelli, N. Guevara, H. Delingette, Inner-ear augmented metal artifact reduction with simulation-based 3d generative adversarial networks, Computerized Medical Imaging and Graphics 93 (2021) 101990.
  * [63] J. Wang, Y. Zhao, J. H. Noble, B. M. Dawant, Conditional generative adversarial networks for metal artifact reduction in ct images of the ear, in: Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I, Springer, 2018, pp. 3–11.
  * [64] Y. Lu, S. Yang, Z. Xu, J. Wang, Speech training system for hearing impaired individuals based on automatic lip-reading recognition, in: Advances in Human Factors and Systems Interaction: Proceedings of the AHFE 2020 Virtual Conference on Human Factors and Systems Interaction, July 16-20, 2020, USA, Springer, 2020, pp. 250–258.
  * [65] M. Jeyalakshmi, C. R. Robin, D. Doreen, Predicting cochlear implants score with the aid of reconfigured long short-term memory, Multimedia Tools and Applications 82 (8) (2023) 12537–12556.
  * [66] M. Meeuws, D. Pascoal, S. Janssens de Varebeke, G. De Ceulaer, P. J. Govaerts, Cochlear implant telemedicine: remote fitting based on psychoacoustic self-tests and artificial intelligence, Cochlear Implants International 21 (5) (2020) 260–268.
  * [67] J. Wathour, P. J. Govaerts, E. Lacroix, D. Naïma, Effect of a ci programming fitting tool with artificial intelligence in experienced cochlear implant patients, Otology & Neurotology 44 (3) (2023) 209–215.
  * [68] Z. Wang, C. Vandersteen, T. Demarcy, D. Gnansia, C. Raffaelli, N. Guevara, H. Delingette, Deep learning based metal artifacts reduction in post-operative cochlear implant ct imaging, in: Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part VI 22, Springer, 2019, pp. 121–129.
  * [69] T. Gajecki, W. Nogueira, Deep learning models to remix music for cochlear implant users, The Journal of the Acoustical Society of America 143 (6) (2018) 3602–3615.
  * [70] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (8) (1997) 1735–1780.
  * [71] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, J. Schmidhuber, LSTM: A search space odyssey, IEEE transactions on neural networks and learning systems 28 (10) (2016) 2222–2232.
  * [72] K. Chu, L. Collins, B. Mainsah, A causal deep learning framework for classifying phonemes in cochlear implants, in: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021, pp. 6498–6502.
  * [73] E. H.-H. Huang, R. Chao, Y. Tsao, Electrodenet–a deep learning based sound coding strategy for cochlear implants, IEEE Transactions on Cognitive and Developmental Systems (2023).
  * [74] Y. Fan, D. Zhang, R. Banalagay, J. Wang, J. H. Noble, B. M. Dawant, Hybrid active shape and deep learning method for the accurate and robust segmentation of the intracochlear anatomy in clinical head ct and cbct images, Journal of Medical Imaging 8 (6) (2021) 064002–064002.
  * [75] F. Heutink, V. Koch, B. Verbist, W. J. van der Woude, E. Mylanus, W. Huinck, I. Sechopoulos, M. Caballo, Multi-scale deep learning framework for cochlea localization, segmentation and analysis on clinical ultra-high-resolution ct images, Computer methods and programs in biomedicine 191 (2020) 105387.
  * [76] A. Gueriani, H. Kheddar, A. C. Mazari, Deep reinforcement learning for intrusion detection in iot: A survey, in: 2023 2nd International Conference on Electronics, Energy and Measurement (IC2EM), Vol. 1, 2023, pp. 1–7.  doi:10.1109/IC2EM59347.2023.10419560.
  * [77] P. López Diez, J. V. Sundgaard, F. Patou, J. Margeta, R. R. Paulsen, Facial and cochlear nerves characterization using deep reinforcement learning for landmark detection, in: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part IV 24, Springer, 2021, pp. 519–528.
  * [78] Y.-H. Lai, Y. Tsao, X. Lu, F. Chen, Y.-T. Su, K.-C. Chen, Y.-H. Chen, L.-C. Chen, L. P.-H. Li, C.-H. Lee, Deep learning–based noise reduction approach to improve speech intelligibility for cochlear implant recipients, Ear and hearing 39 (4) (2018) 795–809.
  * [79] E. W. Healy, S. E. Yoho, J. Chen, Y. Wang, D. Wang, An algorithm to increase speech intelligibility for hearing-impaired listeners in novel segments of the same noise type, The Journal of the Acoustical Society of America 138 (3) (2015) 1660–1669.
  * [80] Y. Hu, P. C. Loizou, Environment-specific noise suppression for improved speech intelligibility by cochlear implant users, The Journal of the Acoustical Society of America 127 (6) (2010) 3689–3695.
  * [81] B. Banerjee, L. L. Mendel, J. K. Dutta, H. Shabani, S. Najnin, Identifying hearing deficiencies from statistically learned speech features for personalized tuning of cochlear implants, in: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
  * [82] L. P.-H. Li, J.-Y. Han, W.-Z. Zheng, R.-J. Huang, Y.-H. Lai, Improved environment-aware–based noise reduction system for cochlear implant users based on a knowledge transfer approach: Development and usability study, Journal of medical Internet research 23 (10) (2021) e25460.