2021

[2]Molly M. Stevens

[4]Mauricio Barahona

1]Department of Computing, Imperial College London, United Kingdom

2]Department of Materials, Department of Bioengineering & Institute of
Biomedical Engineering, Imperial College London, United Kingdom

3]Present address: University of Southern Denmark, Odense, Denmark

4]Department of Mathematics, Imperial College London, United Kingdom

#  _RamanSPy_ : An open-source Python package for integrative Raman
spectroscopy data analysis

Dimitar Georgiev  Simon Vilms Pedersen  Ruoxiao Xie  Álvaro Fernández-Galiana
<EMAIL_ADDRESS><EMAIL_ADDRESS>[  [  [  [

###### Abstract

Raman spectroscopy is a non-destructive and label-free chemical analysis
technique, which plays a key role in the analysis and discovery cycle of
various branches of science. Nonetheless, progress in Raman spectroscopic
analysis is still impeded by the lack of software, methodological and data
standardisation, and the ensuing fragmentation and lack of reproducibility of
analysis workflows thereof. To address these issues, we introduce _RamanSPy_ ,
an open-source Python package for Raman spectroscopic research and analysis.
_RamanSPy_ provides a comprehensive library of ready-to-use tools for
spectroscopic analysis, which streamlines day-to-day tasks, integrative
analyses, as well as novel research and algorithmic development. _RamanSPy_ is
modular and open source, not tied to a particular technology or data format,
and can be readily interfaced with the burgeoning ecosystem for data science,
statistical analysis and machine learning in Python.

###### keywords:

Raman spectroscopy, spectral analysis, chemometrics, preprocessing pipeline,
artificial intelligence, machine learning, Python package

Raman spectroscopy (RS) is a powerful sensing modality based on inelastic
light scattering, which provides qualitative and quantitative chemical
analysis with high sensitivity and specificity colthup2012introduction . RS
yields a characterisation of the vibrational profile of molecules, which can
help elucidate the composition of chemical compounds, biological specimens and
materials mccreery2005raman ; shipp2017raman ; Fernandez-
Galiana:2023:10.1002/adma.202210807 . In contrast to most conventional
technologies for (bio)chemical characterisation (e.g., staining, different
omics, fluorescence microscopy and mass spectrometry), RS is both label-free
and non-destructive, thereby allowing the acquisition of rich biological and
chemical information without compromising the structural and functional
integrity of the probed samples. This advantage has enabled a broad range of
applications of RS in biomedical and pharmaceutical research butler2016using ;
movasaghi2007raman ; vankeirsbilck2002applications , including in the imaging
of cells and tissues smith2016raman ; kallepitis2017quantitative ; Lalone:2023
; pedersen2023spectral , the chemical analysis of drug compounds
wang2018research ; VANKEIRSBILCK2002869 , and the detection of disease
auner2018applications ; mahadevan1996raman ; kong2015raman ; penders2021single
.

An area of topical interest is the frontier of Raman spectroscopy,
chemometrics and artificial intelligence (AI), with its promise of more
autonomous, flexible and data-driven RS analytics pan2022review ; luo2022deep
; lussier2020deep . There has been a recent surge in the adoption of AI
methods in Raman-based research Fernandez-Galiana:2023:10.1002/adma.202210807
, with applications to RS now spanning domains as broad as the identification
of pathogens and other microbes bacteria ; lu2020combination ; yan2021raman ;
wang2021applications ; the characterisation of chemicals, including minerals
carey2015machine , pesticides zhu2021rapid  and other analytes han2020bayesian
; akpolat2020high ; the development of novel diagnostic platforms
ralbovsky2020towards ; talari2019advancing ; heng2021advances ; zhang2022raman
; as well as the application of techniques from computer vision for denoising
and super-resolution in Raman imaging deeper .

As new hardware, software and data acquisition RS technologies continue to
emerge qi2023recent ; zhu2014technical , there is a pressing need for an
integrated RS data analysis environment, which facilitates the development of
pipelines, methods and applications, and bolsters the use of RS in biomedical
research. Yet, the full deployment of RS and its capabilities is still
hindered by practical factors stemming from the restrictive, functionally
disparate, and highly encapsulated nature of current commercial software for
RS data analysis. RS data analysis often operates within proprietary software
environments and data formats, which have induced methodological
inconsistencies and reduced cross-platform and benchmarking efforts, with
growing concerns around reproducibility. These restrictions have also hampered
the adoption of new AI technologies into the field byrne2016spectral ;
tanwar2021advancing ; barton2022chemometrics ; moller2017robust ;
ntziouni2022review . As a consequence, researchers increasingly resort to
developing in-house scripts for RS analysis in Python van1995python , further
adding to methodological fragmentation and lack of standardisation
guo2021chemometric .

In response to these challenges, we have developed _RamanSPy_ \- a modular,
open-source framework for integrated Raman Spectroscopy analytics in Python.
_RamanSPy_ is designed to systematise day-to-day workflows, enhance
algorithmic development and validation, and accelerate the adoption of novel
AI technologies into the RS field. Firstly, _RamanSPy_ serves as a platform
for general-purpose RS analytics supporting the RS data life cycle by
providing a suite of ready-to-use modules for data loading, preprocessing,
analysis and visualisation. By design, these functionalities are not tied to
any specific technology or data type, thereby allowing integrative and
transferable cross-platform analyses. Secondly, _RamanSPy_ addresses
challenges in data preprocessing by facilitating the compilation of
reproducible pipelines to streamline and automatise preprocessing protocols.
Thirdly, _RamanSPy_ helps bridge the gap between RS data and state-of-the-art
AI technologies within the extensive machine learning (ML) ecosystem in
Python. Complemented by direct access to Raman datasets, preprocessing
protocols and performance metrics, this provides the foundation for AI model
development and benchmarking.

The codebase of _RamanSPy_ is hosted at https://github.com/barahona-research-
group/RamanSPy with extended documentation (https://ramanspy.readthedocs.io),
which includes tutorials and example applications, and details about the real-
world research applications presented in this paper.

Figure 1: General Raman spectroscopy workflow and core features of _RamanSPy_.
a, _RamanSPy_ supports the Raman spectroscopic data analysis life cycle via a
modular, loosely coupled architecture. RS data is parsed to a common data
representation format, which is interfaced with preprocessing, analysis and
visualisation tools within _RamanSPy_. The core features of _RamanSPy_ include
a comprehensive library of standardised, simple-to-use procedures for data
loading, preprocessing, analysis and visualisation. These modules are flexible
and allow the incorporation of further techniques and in-house methods. For
complete information about the modules available in _RamanSPy_ , refer to the
documentation at https://ramanspy.readthedocs.io. b, An example workflow use
case in _RamanSPy_ : Raman data is loaded, preprocessed and analysed in a few
lines of code.

## Results

_RamanSPy_ as a platform for general Raman spectroscopy analytics

_RamanSPy_ is based on a modular, object-oriented programming (OOP)
infrastructure, which streamlines the RS data analysis life cycle (Fig. 1a)
and allows users to compile diverse analysis workflows with a few lines of
reusable, user-friendly code (Fig. 1b). The framework adopts a scalable array-
based data representation, which accommodates different spectroscopic
modalities, including single-point spectra, Raman imaging data, and volumetric
scans. Experimental data can be loaded through custom loaders built into
_RamanSPy_ or through standard tools available in Python. The data
representation functions as a common data container that defines the interface
between RS data management and manipulation within _RamanSPy_ , allowing us to
unify data standards across setups and vendors, independent of instrumental
origin and acquisition modality.

_RamanSPy_ also provides an extensive toolbox for preprocessing, analysis and
visualisation. The preprocessing suite includes techniques for denoising,
baseline correction, cosmic spike removal, normalisation and background
subtraction, among others. Likewise, the analysis toolbox includes modules for
decomposition (useful for dimensionality reduction), clustering and spectral
unmixing. _RamanSPy_ also includes a set of data visualisation tools. All
these modules are organised into an extensible class structure, which
standardises their application across projects and datasets to facilitate
transferable analysis workflows.

Figure 2: Morphological analysis of a THP-1 cell via spectral unmixing with
_RamanSPy_. a, Bright-field image of a THP-1 cell. The same cell was also
imaged with Raman spectroscopy. Image and volumetric Raman data from
kallepitis2017quantitative . b, An exemplar spectrum from the raw volumetric
Raman data (taken from the centre of the layer in d). The fingerprint region
(700–1800 cm-1) shaded in red was used for the analysis. c, Volumetric data at
the 1008 cm-1 band (characteristic of proteins) after preprocessing. d-g,
Spectral unmixing analysis reveals the distribution of components within the
cell: lipids (violet), nucleus (blue), cytoplasm (green), and background
(yellow). d, A merged reconstruction of the sixth depth layer (10 in total) of
the THP-1 cell determined via spectral unmixing. e, Four endmembers derived
with N-FINDR nfindr  characterised via peak assignment. f, Fractional
abundance maps calculated with FCLS fcls  for the sixth depth layer. g,
Fractional abundance maps for the entire volume.

We showcase the core features of _RamanSPy_ by analysing volumetric Raman
spectroscopic data from a human leukaemia monocytic (THP-1) cell
kallepitis2017quantitative  (Fig. 2). The aim is to investigate the cell
phenotype in a label-free manner using RS and methods from chemometrics. We
load the data using built-in tools, and perform a spectral preprocessing
protocol comprising spectral cropping to the fingerprint region (700–1800
cm-1), cosmic spike removal, denoising, baseline correction and normalisation
(see SI). Using the visualisation tools in the package, we inspect data
quality (Fig. 2b) and perform initial exploratory analysis by examining, e.g.,
data slices across wavenumber bands (Fig. 2c). The analysis proceeds to
spectral unmixing based on: (i) N-FINDR nfindr  for endmember detection, and
(ii) fully constrained least squares (FCLS) fcls  for component
quantification. This process is exploited to demix signal contributions from
different cellular components and study their morphological organisation
within the THP-1 cell. Following the peak assignment in
kallepitis2017quantitative , we distinguish endmember components related to
lipids (band 1008 cm-1), nucleic acid (band 789 cm-1), cytoplasm (bands 1066,
1134, 1303, 1443 and 1747 cm-1), and the background (Fig. 2e). Finally, we
produce fractional abundance reconstructions based on the extracted
endmembers, which we can examine on a single-layer level (Fig. 2f) and across
the entire volume (Fig. 2g) to localise cellular organelles within the cell.

_RamanSPy_ enables automated pipelining of spectral preprocessing protocols

Experimental RS data is susceptible to non-specific signal artefacts (e.g.,
cosmic rays, autofluorescence background, variability in instrumentation),
which can severely affect downstream analyses. Preprocessing is therefore a
critical step in any spectroscopic analysis workflow gautam2015review ;
ryabchykov2018analyzing . Yet, due to a lack of standardisation and frameworks
for general-purpose pipelining byrne2016spectral , researchers tend to utilise
variable preprocessing protocols, often dispersed across different software
systems, thus affecting reproducibility and validation rozenstein2014comparing
; alshdaifat2021effect .

To facilitate the creation of reproducible protocols, _RamanSPy_ incorporates
a pipelining infrastructure, which systematises the process of creating,
customising and executing preprocessing pipelines (Fig. 3a). Users can use a
specialised class, which defines a generic, multi-layered preprocessing
procedure, to assemble pipelines from selected built-in preprocessing modules
or other in-house methods. To reduce overhead, the constructed pipelines are
designed to function exactly as any single method, i.e., they are fully
compatible with the rest of the modules and data structures in the package.
Furthermore, pipelines can be easily saved, reused and shared to foster the
development of a repository of preprocessing protocols. As a seed to this
repository, _RamanSPy_ provides a library of assembled preprocessing protocols
(custom pre-defined, or adapted from the literature bergholt2016raman ), which
users can access and exploit.

To illustrate the pipelining functionalities, we use _RamanSPy_ to construct
three preprocessing protocols by compiling selected methods in the desired
order of execution, and applying them out-of-the-box to data loaded into the
platform (Fig. 3c-e). We use them to preprocess Raman spectroscopic data from
kallepitis2017quantitative  (Fig. 3b). Note how the three pipelines yield
substantially different results, reinforcing the importance of consistency in
the selection of preprocessing protocols. Pipeline II was deemed the most
robust, and consequently added to the protocols library in _RamanSPy_ as
default.

Figure 3: Spectral preprocessing pipelining in _RamanSPy_. a,  _RamanSPy_
automates the construction, customisation and execution of multi-layered
preprocessing procedures via pipelining. Users can assemble built-in and in-
house methods into complete preprocessing pipelines, which are fully
compatible with data integrated within _RamanSPy_ and can be saved, reused and
shared. _RamanSPy_ also provides access to a library of already assembled
preprocessing pipelines. b,  Two raw spectra from the THP-1 data from
kallepitis2017quantitative  are used to compare the effect of different
preprocessing pipelines. c-e,  The results of three preprocessing pipelines
built within _RamanSPy_ , demonstrating the need for standardisation. Note on
preprocessing methods: fingerprint region is $700$–$1800$ cm-1; ASLS -
Asymmetric Least Squares eilers2005baseline ; asPLS - Adaptive Smoothness
Penalized Least Squares zhang2020baseline ; AUC - area under the curve; cosmic
rays removed with algorithm from whitaker2018simple .

_RamanSPy_ facilitates AI integration and validation of next-generation Raman
data analytics

To help accelerate the adoption of AI technologies for RS analysis, _RamanSPy_
is endowed with a permeable architecture, which streamlines the interface
between Raman spectroscopic data and the burgeoning ML ecosystem in Python.
This is complemented by tools for benchmarking, such as datasets and
performance metrics, which support the evaluation of new models and
algorithms. We show below two examples of _RamanSPy_ ’s capabilities for ML
integration and benchmarking.

First, _RamanSPy_ allows the seamless integration of standard Python AI/ML
methods (e.g., from _scikit-learn_ sklearn , _PyTorch_ pytorch  and
_tensorflow_ tensorflow ) as tools for RS analysis (Fig. 4a). As an
illustration, we use _RamanSPy_ to construct a deep learning denoising
procedure based on the one-dimensional ResUNet model - a fully convolutional
UNet neural network with residual connections deeper . To do this, we simply
wrap within _RamanSPy_ the pre-trained neural network (trained on spectra from
MDA-MB-231 breast cancer cells, available at https://github.com/conor-
horgan/DeepeR) as a custom denoising method. Once wrapped, the denoiser is
automatically compatible with the rest of _RamanSPy_ and can be readily
employed for different applications. For instance, we replicate the results in
deeper , and show in Fig. 4b-c that the application of this deep-learning
denoiser to the low signal-to-noise ratio (SNR) test set from deeper
consistently outperforms the commonly-used Savitzky-Golay filter savgol , as
quantified by various metrics also coded within _RamanSPy_ (e.g., mean squared
error (MSE), spectral angle distance (SAD) sad  and spectral information
divergence (SID) sid ). Applying this pipeline to new data only involves
changing the data source. Taking advantage of this transferability, we test
the denoiser on unseen volumetric Raman data from another cell line (THP-1
kallepitis2017quantitative ), with added Gaussian noise (see SI). In this
case, Fig. 4d-e shows improved performance especially according to the MSE
metric, which is dependent on normalisation, but with lower significance
according to scale-invariant and information-theoretic metrics, also available
in _RamanSPy_. This example emphasises the importance of incorporating robust
validation criteria within data analysis workflows.

Figure 4: _RamanSPy_ interfaces with AI/ML Python frameworks to create new
methods for RS analysis. a, _RamanSPy_ allows users to incorporate AI/ML
models seamlessly into pipelines created within the platform. b-c, A pre-
trained 1D ResUNet deep-learning denoiser deeper  is integrated as a
preprocessing module within _RamanSPy_ to investigate its performance against
the Savitzky-Golay (SG) filter savgol . b,  Denoising of a spectrum from
deeper , where the low-SNR (purple) is the input and the high-SNR (green) is
the target. The data is denoised with a SG filter of polynomial order 3 and
kernel size 9, SG(3, 9) (blue), and with the implemented deep-learning
denoiser (yellow). c,  The results on the test set from deeper  ($n=12694$)
show that the deep-learning denoiser outperforms six SG filters across three
performance metrics (MSE, SAD, SID). Error bars represent one standard
deviation around the sample mean. Statistical significance measured with a
two-sided Wilcoxon signed-rank test with adjustment for multiple comparisons
based on Benjamini-Hochberg correction Benjamini_Hochberg  (* $P<0.05$, **
$P<0.01$, *** $P<0.001$, **** $P<0.0001$). d-e, Same analysis on unseen data
from kallepitis2017quantitative  ($n=1600$). The input (purple) corresponds to
data contaminated with added noise and the target (green) to the original
data. In this case, the deep-learning denoiser only shows an improvement for
MSE.

Secondly, the data management backbone of _RamanSPy_ ensures a direct data
flow to the rest of the Python ecosystem, i.e., data can be loaded,
preprocessed, and analysed in _RamanSPy_ and then exported to conduct further
modelling and analysis elsewhere (Fig. 5a). As an example application, we
perform AI-based bacteria identification using Raman measurements bacteria
from 30 bacterial and yeast isolates (Fig. 5b). After loading and exploring
the spectra with _RamanSPy_ , we interface the data with the _lazypredict_
Python package lazypredict  and benchmark 28 different ML classification
models (including logistic regression, support vector machines and decision
trees) on the task of predicting the species from the spectrum. The models
were trained on a high-SNR dataset (100 spectra per isolate) and tested on an
unseen high-SNR testing set of the same size. Our benchmarking analysis in
Fig. 5c finds logistic regression as the best-performing model, achieving a
classification accuracy of 79.63% on the species-level classification task
(Fig. 5d), and 94.63% for antibiotic treatment classification (Fig. 5e).

Figure 5: _RamanSPy_ as a suite for algorithmic development and benchmarking.
a, Data representations in _RamanSPy_ are compatible with the Python AI/ML
ecosystem, allowing data flow from _RamanSPy_ to _scikit-learn_ sklearn ,
_PyTorch_ pytorch , _tensorflow_ tensorflow , etc. _RamanSPy_ is also equipped
with standard datasets and relevant metrics to support model development and
validation. b-e Benchmarking ML classification models on the task of bacteria
identification from Raman spectra bacteria . b, Mean Raman spectra for all
bacterial species provided (100 spectra per species). Spectra are min-max
normalised to the range 0–1 for visualisation purposes. c, Benchmarking
results of 28 ML models. The best accuracy was achieved by the logistic
regression classifier. d-e, Confusion matrices for the best species-level (d)
and antibiotic-level (e) classifier with accuracies of 79.63% and 94.63%,
respectively.

To further assist validation against previous results, _RamanSPy_ provides
access to a library of curated datasets, which can be integrated into analysis
and benchmarking workflows. This lays the foundation for a common repository
of RS data and reduces barriers to data access, especially for ML teams with
limited access to RS instruments luo2022deep . The dataset library in
_RamanSPy_ already includes data loaders for Raman data from bacterial species
bacteria , cell lines deeper ; kallepitis2017quantitative , COVID-19 samples
yin2021efficient ; Yin2020 , multi-instrument Surface Enhanced Raman
Spectroscopy (SERS) measurements of adenine samples fornasaro2020surface ,
wheat lines csen2023differentiation , minerals rruff , and will continue to be
expanded.

## Discussion

In this paper, we have introduced _RamanSPy_ \- a computational framework for
integrative Raman spectroscopic data analysis. _RamanSPy_ offers a
comprehensive collection of tools for spectroscopic analysis designed to
systematise the RS data analysis life cycle, reducing typical overheads of
analysis workflows and improving methodological standardisation. The package
also lays the foundations of a common repository of standardised methods,
protocols and datasets, which users can readily access and exploit within the
_RamanSPy_ framework to conduct different benchmarking studies. Furthermore,
_RamanSPy_ is fully compatible with frameworks for data science and machine
learning in Python, thereby facilitating the adoption and validation of
advanced AI technologies for next-generation RS analysis. Lastly, we remark
that, while our focus here has been on Raman spectroscopy, many of the tools
in _RamanSPy_ are of broad applicability to other vibrational spectroscopy
techniques, including infrared (IR) spectroscopy.

## Methods

### Installation

_RamanSPy_ has been deposited in the Python Package Index
(https://pypi.org/project/ramanspy). This means it can be directly installed
via the common package installer _pip_ for Python:

⬇

1 pip install ramanspy

To access the functionalities of the package after installation, users only
need to import _RamanSPy_ in their Python scripts. One can import the whole
package:

⬇

1import ramanspy

2# or import ramanspy as rp

or individual modules or methods:

⬇

1# individual modules

2from ramanspy import load, preprocessing

3

4# individual methods

5from ramanspy.analysis.unmix import NFINDR

### Core infrastructure

Data management Data in _RamanSPy_ is represented by a set of custom data
container classes based on scalable, computationally efficient array
programming numpy , which correspond to different spectroscopic modalities.
This includes the generic SpectralContainer class, as well as the more
specialised Spectrum, SpectralImage and SpectralVolume classes representing on
single-point spectra (1D), imaging data (3D), volumetric data (4D)
respectively. These classes define data-specific information and behaviour in
the background to allow a smooth, user-friendly experience, regardless of the
data of interest.

The containers can be initialised by providing the corresponding intensity
data, the spectral axis (in cm-1) and other relevant (meta) data, which will
become properties of the constructed object. For instance:

⬇

1raman_spectrum = ramanspy.Spectrum(intensity_data, spectral_axis, *args,
**kwargs)

2raman_image = ramanspy.SpectralImage(intensity_data, spectral_axis, *args,
**kwargs)

Once created, data containers can be manipulated, visualised, saved and loaded
as needed using the built-in tools in _RamanSPy_.

Note that for the most part, users would not need to manually populate these
containers. Instead, they can take advantage of the data loading
functionalities that _RamanSPy_ provides.

Data loading To support data loading, _RamanSPy_ offers easy-to-use data
loaders compatible with experimental Raman spectroscopic data from a range of
instrumental vendors in the area. These loaders - available within
_ramanspy.load_ \- automatically parse relevant data files and return the
appropriate spectral container. As an example, users can load MATLAB files
exported from WITec’s ProjectFOUR/FIVE software using the following command:

⬇

1raman_object = ramanspy.load.witec(<PATH>)

A full list of the data loaders built into _RamanSPy_ is available as part of
the documentation of the package at
https://ramanspy.readthedocs.io/en/latest/loading.html.

Raman data can also be loaded via established data-loading tools in Python.
For instance, one can use _pandas’_ csv loader to load a spectrum from a .csv
file with two columns storing the intensity data and the spectral axis by
using:

⬇

1import pandas as pd

2

3data = pd.read_csv(csv_filename)

4raman_spectrum = ramanspy.Spectrum(data["<intensity_column>"],
data["<axis_column>"])

Spectral preprocessing Preprocessing logic in _RamanSPy_ is defined by the
PreprocessingStep class, which defines most of the necessary preprocessing
infrastructure in the background to ensure a smooth, data-agnostic experience
via a single point of contact specified through their apply() method.

Yet, as with data loading, for the most part, users are not expected to use
this class to manually implement and optimise such preprocessing methods
themselves. Instead, the _RamanSPy_ package provides a comprehensive toolbox
of ready-to-use preprocessing methods, which users can access, customise and
employ to compile a wide variety of preprocessing procedures. These
preprocessing procedures are given as predefined classes within
_ramanspy.preprocessing_ which extend the PreprocessingStep class. To use
these built-in methods, users need to create an instance of the selected
technique. For instance:

⬇

1denoiser = ramanspy.preprocessing.denoise.SavGol(*args, **kwargs)

2baseline_corrector = ramanspy.preprocessing.baseline.ASLS(*args, **kwargs)

3normaliser = ramanspy.preprocessing.normalise.MaxIntensity(*args, **kwargs)

Note that _RamanSPy_ offers full control over relevant parameters, which can
be supplied during initialisation via the *args and **kwargs arguments.

As the methods inherit all operational logic defined within the parent
PreprocessingStep class, they can be directly accessed and used on any data
loaded in the framework through their apply() method:

⬇

1preprocessesd_objects = denoiser.apply(<spectral object or collection of
spectral objects>)

2preprocessesd_objects = baseline_corrector.apply(<spectral object or
collection of spectral objects>)

A full list of the methods for spectral preprocessing built into _RamanSPy_ is
available as part of the documentation of the package at
https://ramanspy.readthedocs.io/en/latest/preprocessing.html.

If needed, users can also incorporate any in-house method into _RamanSPy_ by
manually creating instances of the PreprocessingStep class which wrap the
given method. This can be done as follows:

⬇

1def preprocessing_func(intensity_data, spectral_axis, *args, **kwargs):

2 # Preprocess intensity_data and spectral_axis

3 ...

4

5 return updated_intensity_data, updated_spectral_axis

6

7# wrapping the function together with the relevant *args and **kwargs

8custom_preprocessing_method =
ramanspy.preprocessing.PreprocessingStep(preprocessing_func, *args, **kwargs)

Then, the custom preprocessing method is fully compatible with the rest of
_RamanSPy_ ’s functionalities and out-of-the-box applicable to any data
integrated within the package via its apply() method:

⬇

1custom_preprocessing_method.apply(<spectral object or collection of spectral
objects>)

Note that this class structure implies that these instances can then be saved
(e.g. as pickle files) and, therefore, reused and shared as required
afterwards.

Spectral analysis As with preprocessing classes, users can access any built-in
analysis method (available within the _ramanspy.analysis_ sub-module) by
creating an object instance of the corresponding class (again - with full
control over relevant parameters) as follows:

⬇

1nmf = ramanspy.analysis.decompose.NMF(*args, **kwargs)

2kmeans = ramanspy.analysis.cluster.KMeans(*args, **kwargs)

3unmixer = ramanspy.analysis.unmix.NFINDR(*args, **kwargs)

Once created, instances can be similarly accessed via their apply() method on
any data loaded in _RamanSPy_.

⬇

1cluster_maps, cluster_centres = kmeans.apply(<spectral object or collection
of spectral objects>)

2abundance_fractions, endmemebrs = unmixer.apply(<spectral object or
collection of spectral objects>)

A full list of the methods for spectral analysis built into _RamanSPy_ is
available as part of the documentation of the package at
https://ramanspy.readthedocs.io/en/latest/analysis.html.

Visualisation The _RamanSPy_ package also provides various visualisation tools
available within the _ramanspy.plot_ sub-module. As an example, one can plot
spectra using the spectra function:

⬇

1ramanspy.plot.spectra(<spectra or collection of spectra>)

2ramanspy.plot.show() # or plt.show() after import matplotlib.pyplot as plt

Note that these functions are highly customisable. This can be done by
providing relevant parameters to control the plot generation, as well as
through matplotlib’s customisation workflow.

⬇

1import matplotlib.pyplot as plt

2

3plt.figure(figsize = (5, 5))

4ax = ramanspy.plot.spectra(<spectra or collection of spectra>, title="<str>",
label="<str or list[str]>")

5ax.set_ylabel("<str>") # adding a label to the y-axis

6plt.show() # or ramanspy.plot.show()

A full list of the methods for data visualisation built into _RamanSPy_ is
available as part of the documentation of the package at
https://ramanspy.readthedocs.io/en/latest/plot.html.

### Preprocessing pipelines

Pipelining behaviour is defined by the Pipeline class in _RamanSPy_ , which
ensures that pipelines are accessible, simple-to-use and fully compatible with
the rest of _RamanSPy_.

Creating a custom preprocessing pipeline To assemble a preprocessing pipeline,
one simply needs to stack relevant methods (built-in or custom) into the
intended order of execution. For instance:

⬇

1preprocessing_pipeline = ramanspy.preprocessing.Pipeline([

2 ramanspy.preprocessing.denoise.SavGol(*args, **kwargs),

3 ramanspy.preprocessing.baseline.ASLS(*args, **kwargs),

4 ramanspy.preprocessing.normalise.MaxIntensity(*args, **kwargs),

5 custom_preprocessing_method(*args, **kwargs) # custom in-house method

6])

Constructed pipelines can then be applied exactly as single methods via their
apply() method to any data loaded within _RamanSPy_.

⬇

1preprocessesd_objects = preprocessing_pipeline.apply(<spectral object or
collection of spectral objects>)

As pipelines in _RamanSPy_ are objects, they can also be directly saved in a
convenient file format, such as pickle files. As such, they can then be
reloaded, reused and shared as needed.

Access a predefined preprocessing pipeline _RamanSPy_ also provides a
collection of built-in preprocessing pipelines. To access them, one can select
the desired protocol from _ramanspy.preprocessing.protocols_ as follows:

⬇

1preprocessing_pipeline = ramanspy.preprocessing.protocols.PROTOCOL_X

A pre-defined Pipeline instance will be returned, which can similarly be
employed directly through its apply() method.

A full list of the protocols for spectral preprocessing built into _RamanSPy_
is available as part of the documentation of the package at
https://ramanspy.readthedocs.io/en/latest/preprocessing.html#established-
protocols.

### AI integration

Integrate AI methods into _RamanSPy_ To integrate new techniques for spectral
preprocessing and analysis, users can take advantage of the extensible
architecture of _RamanSPy_ and wrap models and algorithms into custom classes.
For instance, one can create a new denoiser method based on a _PyTorch_ model
for denoising by simply creating a function, which defines how the model can
be used to preprocess a generic intensity data array, and then wrapping the
method within a PreprocessingStep instance.

⬇

1def nn_preprocesing(intensity_data, wavenumber_axis):

2 intensity_data = v.reshape(-1, intensity_data.shape[-1])

3 output =
model(torch.Tensor(intensity_data).unsqueeze(1)).cpu().detach().numpy()

4 output = np.squeeze(output).reshape(intensity_data.shape)

5

6 return output, wavenumber_axis

7

8nn_denoiser = ramanspy.preprocessing.PreprocessingStep(nn_preprocesing)

Integrated methods are automatically rendered fully compatible with the rest
of _RamanSPy_ ’s functionalities in the background, so one can simply use the
apply() method of the constructed denoiser to preprocess any data loaded
within _RamanSPy_ as any built-in preprocessing class.

Export data from _RamanSPy_ to AI frameworks The data management core of
_RamanSPy_ allows a direct interface with the entire Python ecosystem,
including frameworks for statistical modelling, machine learning and deep
learning. To do that, users can simply feed relevant data from _RamanSPy_ to
functions and tools they want to use elsewhere. For instance, one can pass the
intensity data stored in a spectral container to a specific model from the
scikit-learn sklearn  framework for statistical and ML modelling directly via
their fit() method:

⬇

1model.fit(spectral_container.spectral_data)

### Datasets

To access the Raman spectroscopic datasets available in _RamanSPy_ , users can
employ custom data-loading methods built into _RamanSPy_ under
_ramanspy.datasets_. These would automatically parse the relevant data into
the corresponding spectral container. For instance, one can load the bacteria
data from bacteria  using the following function:

⬇

1data_container, labels = ramanspy.datasets.bacteria(dataset="train", <PATH>)

Note that, depending on where each dataset was deposited and the license it
was deposited under, some of these methods will automatically download the
given dataset, whereas others may require the manual download of the data.
Users are pointed to the documentation of each method for instructions on how
to properly load each dataset.

A full list of the datasets built into _RamanSPy_ is available as part of the
documentation of the package at
https://ramanspy.readthedocs.io/en/latest/datasets.html.

### Metrics

Users can likewise readilyaccess relevant spectroscopic metrics, such as MSE,
SAD and SID, from _ramanspy.metrics_. These can be used to measure the
similarity between spectra by using the respective method:

⬇

1ramanspy.metrics.SID(spectrum_I, spectrum_II)

A full list of the metrics built into _RamanSPy_ is available as part of the
documentation of the package at
https://ramanspy.readthedocs.io/en/latest/metrics.html.

## References

  *   * (1) Colthup N.  Introduction to infrared and Raman spectroscopy.  Elsevier; 2012.
  * (2) McCreery RL.  Raman spectroscopy for chemical analysis.  John Wiley & Sons; 2005.
  * (3) Shipp DW, Sinjab F, Notingher I.  Raman spectroscopy: techniques and applications in the life sciences.  Advances in Optics and Photonics. 2017;9(2):315–428.
  * (4) Fernandez-Galiana A, Bibikova O, Pedersen S, Stevens M.  Fundamentals and applications of Raman-based techniques for the design and development of active biomedical materials.  Advanced Materials. 2023;10.1002/adma.202210807.
  * (5) Butler HJ, Ashton L, Bird B, Cinque G, Curtis K, Dorney J, et al.  Using Raman spectroscopy to characterize biological materials.  Nature protocols. 2016;11(4):664–687.
  * (6) Movasaghi Z, Rehman S, Rehman IU.  Raman spectroscopy of biological tissues.  Applied Spectroscopy Reviews. 2007;42(5):493–541.
  * (7) Vankeirsbilck T, Vercauteren A, Baeyens W, Van der Weken G, Verpoort F, Vergote G, et al.  Applications of Raman spectroscopy in pharmaceutical analysis.  TrAC trends in analytical chemistry. 2002;21(12):869–877.
  * (8) Smith R, Wright KL, Ashton L.  Raman spectroscopy: an evolving technique for live cell studies.  Analyst. 2016;141(12):3590–3600.
  * (9) Kallepitis C, Bergholt MS, Mazo MM, Leonardo V, Skaalure SC, Maynard SA, et al.  Quantitative volumetric Raman imaging of three dimensional cell cultures.  Nature communications. 2017;8(1):1–9.
  * (10) Lalone V, Aizenshtadt A, Goertz J, Skottvoll F, Barbero MM, You J, et al.  Quantitative chemometric phenotyping of three dimensional liver organoids by Raman spectral imaging (qRamanomics).  Cell Reports Methods. 2023;.
  * (11) Pedersen SV, Brewer JR, Hedegaard MA, Christensen EA.  Spectral Unmixing for Label-Free, In-Liquid Characterization of Biomass Microstructure and Biopolymer Content by Coherent Raman Imaging.  Analytical chemistry. 2023;95(4):2168–2175.
  * (12) Wang Wt, Zhang H, Yuan Y, Guo Y, He Sx.  Research progress of Raman spectroscopy in drug analysis.  AAPS PharmSciTech. 2018;19(7):2921–2928.
  * (13) Vankeirsbilck T, Vercauteren A, Baeyens W, Van der Weken G, Verpoort F, Vergote G, et al.  Applications of Raman spectroscopy in pharmaceutical analysis.  TrAC Trends in Analytical Chemistry. 2002;21(12):869–877.  https://doi.org/10.1016/S0165-9936(02)01208-6.
  * (14) Auner GW, Koya SK, Huang C, Broadbent B, Trexler M, Auner Z, et al.  Applications of Raman spectroscopy in cancer diagnosis.  Cancer and Metastasis Reviews. 2018;37(4):691–717.
  * (15) Mahadevan-Jansen A, Richards-Kortum RR.  Raman spectroscopy for the detection of cancers and precancers.  Journal of biomedical optics. 1996;1(1):31–70.
  * (16) Kong K, Kendall C, Stone N, Notingher I.  Raman spectroscopy for medical diagnostics—From in-vitro biofluid assays to in-vivo cancer detection.  Advanced drug delivery reviews. 2015;89:121–134.
  * (17) Penders J, Nagelkerke A, Cunnane EM, Pedersen SV, Pence IJ, Coombes RC, et al.  Single particle automated Raman trapping analysis of breast cancer cell-derived extracellular vesicles as cancer biomarkers.  ACS nano. 2021;15(11):18192–18205.
  * (18) Pan L, Zhang P, Daengngam C, Peng S, Chongcheawchamnan M.  A review of artificial intelligence methods combined with Raman spectroscopy to identify the composition of substances.  Journal of Raman Spectroscopy. 2022;53(1):6–19.
  * (19) Luo R, Popp J, Bocklitz T.  Deep Learning for Raman Spectroscopy: A Review.  Analytica. 2022;3(3):287–301.
  * (20) Lussier F, Thibault V, Charron B, Wallace GQ, Masson JF.  Deep learning and artificial intelligence methods for Raman and surface-enhanced Raman scattering.  TrAC Trends in Analytical Chemistry. 2020;124:115796.
  * (21) Ho CS, Jean N, Hogan CA, Blackmon L, Jeffrey SS, Holodniy M, et al.  Rapid identification of pathogenic bacteria using Raman spectroscopy and deep learning.  Nature communications. 2019;10(1):1–8.
  * (22) Lu W, Chen X, Wang L, Li H, Fu YV.  Combination of an artificial intelligence approach and laser tweezers Raman spectroscopy for microbial identification.  Analytical chemistry. 2020;92(9):6288–6296.
  * (23) Yan S, Wang S, Qiu J, Li M, Li D, Xu D, et al.  Raman spectroscopy combined with machine learning for rapid detection of food-borne pathogens at the single-cell level.  Talanta. 2021;226:122195.
  * (24) Wang L, Liu W, Tang JW, Wang JJ, Liu QH, Wen PB, et al.  Applications of Raman spectroscopy in bacterial infections: principles, advantages, and shortcomings.  Frontiers in Microbiology. 2021;12:683580.
  * (25) Carey C, Boucher T, Mahadevan S, Bartholomew P, Dyar M.  Machine learning tools formineral recognition and classification from Raman spectroscopy.  Journal of Raman Spectroscopy. 2015;46(10):894–903.
  * (26) Zhu J, Sharma AS, Xu J, Xu Y, Jiao T, Ouyang Q, et al.  Rapid on-site identification of pesticide residues in tea by one-dimensional convolutional neural network coupled with surface-enhanced Raman scattering.  Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy. 2021;246:118994.
  * (27) Han N, Ram RJ.  Bayesian modeling and computation for analyte quantification in complex mixtures using Raman spectroscopy.  Computational Statistics & Data Analysis. 2020;143:106846.
  * (28) Akpolat H, Barineau M, Jackson KA, Akpolat MZ, Francis DM, Chen YJ, et al.  High-throughput phenotyping approach for screening major carotenoids of tomato by handheld raman spectroscopy using chemometric methods.  Sensors. 2020;20(13):3723.
  * (29) Ralbovsky NM, Lednev IK.  Towards development of a novel universal medical diagnostic method: Raman spectroscopy and machine learning.  Chemical Society Reviews. 2020;49(20):7428–7453.
  * (30) Talari AC, Rehman S, Rehman IU.  Advancing cancer diagnostics with artificial intelligence and spectroscopy: identifying chemical changes associated with breast cancer.  Expert review of molecular diagnostics. 2019;19(10):929–940.
  * (31) Heng HPS, Shu C, Zheng W, Lin K, Huang Z.  Advances in real-time fiber-optic Raman spectroscopy for early cancer diagnosis: Pushing the frontier into clinical endoscopic applications.  Translational Biophotonics. 2021;3(1):e202000018.
  * (32) Zhang L, Li C, Peng D, Yi X, He S, Liu F, et al.  Raman spectroscopy and machine learning for the classification of breast cancers.  Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy. 2022;264:120300.
  * (33) Horgan CC, Jensen M, Nagelkerke A, St-Pierre JP, Vercauteren T, Stevens MM, et al.  High-throughput molecular imaging via deep-learning-enabled raman spectroscopy.  Analytical chemistry. 2021;93(48):15850–15860.
  * (34) Qi Y, Hu D, Jiang Y, Wu Z, Zheng M, Chen EX, et al.  Recent Progresses in Machine Learning Assisted Raman Spectroscopy.  Advanced Optical Materials. 2023;p. 2203104.
  * (35) Zhu X, Xu T, Lin Q, Duan Y.  Technical development of Raman spectroscopy: from instrumental to advanced combined technologies.  Applied Spectroscopy Reviews. 2014;49(1):64–82.
  * (36) Byrne HJ, Knief P, Keating ME, Bonnier F.  Spectral pre and post processing for infrared and Raman spectroscopy of biological tissues and cells.  Chemical Society Reviews. 2016;45(7):1865–1878.
  * (37) Tanwar S, Paidi SK, Prasad R, Pandey R, Barman I.  Advancing Raman spectroscopy from research to clinic: Translational potential and challenges.  Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy. 2021;260:119957.
  * (38) Barton B, Thomson J, Diz EL, Portela R.  Chemometrics for Raman Spectroscopy Harmonization.  Applied Spectroscopy. 2022;76(9):1021–1041.
  * (39) Möller S, Prescott SW, Wirzenius L, Reinholdtsen P, Chapman B, Prins P, et al.  Robust cross-platform workflows: how technical and scientific communities collaborate to develop, test and share best practices for data analysis.  Data Science and Engineering. 2017;2(3):232–244.
  * (40) Ntziouni A, Thomson J, Xiarchos I, Li X, Bañares MA, Charitidis C, et al.  Review of Existing Standards, Guides, and Practices for Raman Spectroscopy.  Applied Spectroscopy. 2022;76(7):747–772.
  * (41) Van Rossum G, Drake Jr FL.  Python reference manual.  Centrum voor Wiskunde en Informatica Amsterdam; 1995.
  * (42) Guo S, Popp J, Bocklitz T.  Chemometric analysis in Raman spectroscopy from experimental design to machine learning–based modeling.  Nature Protocols. 2021;16(12):5426–5459.
  * (43) Winter ME.  N-FINDR: An algorithm for fast autonomous spectral end-member determination in hyperspectral data.  In: Imaging Spectrometry V. vol. 3753. SPIE; 1999. p. 266–275.
  * (44) Heinz DC, et al.  Fully constrained least squares linear spectral mixture analysis method for material quantification in hyperspectral imagery.  IEEE transactions on geoscience and remote sensing. 2001;39(3):529–545.
  * (45) Gautam R, Vanga S, Ariese F, Umapathy S.  Review of multidimensional data processing approaches for Raman and infrared spectroscopy.  EPJ Techniques and Instrumentation. 2015;2:1–38.
  * (46) Ryabchykov O, Guo S, Bocklitz T.  Analyzing Raman spectroscopic data.  Physical Sciences Reviews. 2018;4(2):20170043.
  * (47) Rozenstein O, Paz-Kagan T, Salbach C, Karnieli A.  Comparing the effect of preprocessing transformations on methods of land-use classification derived from spectral soil measurements.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2014;8(6):2393–2404.
  * (48) Alshdaifat E, Alshdaifat D, Alsarhan A, Hussein F, El-Salhi SMFS.  The effect of preprocessing techniques, applied to numeric features, on classification algorithms’ performance.  Data. 2021;6(2):11.
  * (49) Bergholt MS, St-Pierre JP, Offeddu GS, Parmar PA, Albro MB, Puetzer JL, et al.  Raman spectroscopy reveals new insights into the zonal organization of native and tissue-engineered articular cartilage.  ACS central science. 2016;2(12):885–895.
  * (50) Eilers PH, Boelens HF.  Baseline correction with asymmetric least squares smoothing.  Leiden University Medical Centre Report. 2005;1(1):5.
  * (51) Zhang F, Tang X, Tong A, Wang B, Wang J, Lv Y, et al.  Baseline correction for infrared spectra using adaptive smoothness parameter penalized least squares method.  Spectroscopy Letters. 2020;53(3):222–233.
  * (52) Whitaker DA, Hayes K.  A simple algorithm for despiking Raman spectra.  Chemometrics and Intelligent Laboratory Systems. 2018;179:82–84.
  * (53) Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al.  Scikit-learn: Machine learning in Python.  the Journal of machine Learning research. 2011;12:2825–2830.
  * (54) Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems. 2019;32.
  * (55) Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et al.  Tensorflow: a system for large-scale machine learning.  In: Osdi. vol. 16. Savannah, GA, USA; 2016. p. 265–283.
  * (56) Savitzky A, Golay MJ.  Smoothing and differentiation of data by simplified least squares procedures.  Analytical chemistry. 1964;36(8):1627–1639.
  * (57) Kruse FA, Lefkoff A, Boardman J, Heidebrecht K, Shapiro A, Barloon P, et al.  The spectral image processing system (SIPS)—interactive visualization and analysis of imaging spectrometer data.  Remote sensing of environment. 1993;44(2-3):145–163.
  * (58) Chang CI.  Spectral information divergence for hyperspectral image analysis.  In: IEEE 1999 International Geoscience and Remote Sensing Symposium. IGARSS’99 (Cat. No. 99CH36293). vol. 1. IEEE; 1999. p. 509–511.
  * (59) Benjamini Y, Hochberg Y.  Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.  Journal of the Royal Statistical Society: Series B (Methodological). 1995;57(1):289–300.  https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.  https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2517-6161.1995.tb02031.x.
  * (60) Pandala SR.: Lazy Predict help build a lot of basic models without much code and helps understand which models works better without any parameter tuning.  GitHub.  Available from: https://github.com/shankarpandala/lazypredict.
  * (61) Yin G, Li L, Lu S, Yin Y, Su Y, Zeng Y, et al.  An efficient primary screening of COVID-19 by serum Raman spectroscopy.  Journal of Raman Spectroscopy. 2021;52(5):949–958.
  * (62) Yin G, Li L, Lu S, Yin Y, Su Y, Zeng Y, et al.  Data and code on serum Raman spectroscopy as an efficient primary screening of coronavirus disease in 2019 (COVID-19).  Figshare. 2020;10.6084/m9.figshare.12159924.v1.
  * (63) Fornasaro S, Alsamad F, Baia M, Batista de Carvalho LA, Beleites C, Byrne HJ, et al.  Surface enhanced Raman spectroscopy for quantitative analysis: results of a large-scale European multi-instrument interlaboratory study.  Analytical chemistry. 2020;92(5):4053–4064.
  * (64) ŞEN A, Kecoglu I, Ahmed M, Parlatan U, Unlu M.  Differentiation of advanced generation mutant wheat lines: Conventional techniques versus Raman spectroscopy.  Frontiers in Plant Science. 2023;14.
  * (65) Lafuente B, Downs RT, Yang H, Stone N.  The power of databases: The RRUFF project.  In: Highlights in mineralogical crystallography. De Gruyter; 2015. p. 1–30.
  * (66) Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Cournapeau D, et al.  Array programming with NumPy.  Nature. 2020 Sep;585(7825):357–362.  10.1038/s41586-020-2649-2.

## Declarations

### Data availability

All data used in this article are previously published open-access data that
have been deposited by the respective authors online. Instructions on how to
access, download and load the datasets provided in _RamanSPy_ are available in
the documentation at https://ramanspy.readthedocs.io/en/latest/datasets.html.

### Code availability

The codebase of _RamanSPy_ is open-source and hosted on GitHub at
https://github.com/barahona-research-group/RamanSPy. The package can be
installed via _pip_ using ’ _pip install ramanspy_ ’. Documentation, including
detailed tutorials and examples, is available at
https://ramanspy.readthedocs.io. The scripts used to produce the analysis
results presented in this paper are also provided as executable Jupyter
Notebook examples at https://github.com/barahona-research-
group/RamanSPy/tree/3dd2c1e09420c5ac473a72ebd6ed06a91c30a85c/paper_reproducibility
and as part of the documentation of _RamanSPy_ at
https://ramanspy.readthedocs.io/en/latest/auto_examples/index.html.

### Acknowledgments

D.G. is supported by UK Research and Innovation [UKRI Centre for Doctoral
Training in AI for Healthcare grant number EP/S023283/1]. S.V.P. gratefully
acknowledges support from the Independent Research Fund Denmark (0170-00011B).
R.X. and M.M.S. acknowledge support from the Engineering and Physical Sciences
Research Council (EP/P00114/1 and EP/T020792/1). A.F.G. acknowledges support
from the Schmidt Science Fellows, in partnership with the Rhodes Trust. M.M.S.
acknowledges support from the Royal Academy of Engineering Chair in Emerging
Technologies award (CiET2021\\\94). M.B. acknowledges support by the EPSRC
under grant EP/N014529/1, funding the EPSRC Centre for Mathematics of
Precision Healthcare at Imperial College London, and under grant EP/T027258/1.
The authors thank Dr Akemi Nogiwa Valdez for proofreading and data management
support.

Figures were created with BioRender (www.biorender.com).

## Supplementary information

Several of our examples are based on data from kallepitis2017quantitative
which provided volumetric RS scans across 4 distinct THP-1 cell lines. Here,
we only used the first scan (scan ’001’).

### Cell phenotyping via spectral unmixing.

The raw THP-1 data from kallepitis2017quantitative  used for the spectral
unmixing procedure in Fig. 2 was re-exported as MATLAB files from the WITec
Project FIVE software. The MATLAB files were then loaded into _RamanSPy_
followed by spectral preprocessing with a protocol consisting of: (1) spectral
cropping to the $700-1800$cm-1 region; (2) cosmic rays removal with the
algorithm in whitaker2018simple ; (3) denoising with a Savitzky-Golay filter
polynomial order 3 and kernel size 7 savgol ; (4) baseline correction with
asymmetric least squares eilers2005baseline ; and (5) Global MinMax
normalisation to the interval $[0,1]$.

After preprocessing, we performed spectral unmixing in _RamanSPy_ using
N-FINDR nfindr  (number of endmembers set to 5) and FCLS fcls . We concluded
the analysis by visualising the results corresponding to the top 4 endmembers.

### Preparing THP-1 data for deep learning denoising.

The denoising analysis on the data in Fig. 4d-e was performed on the middle
depth layer (fifth layer out of 10) of the THP-1 volumetric scan from
kallepitis2017quantitative . This layer consisted of a $40\times 40$ image
scan, i.e., 1600 spectra. To be consistent with the original paper deeper , we
conducted exactly the same preprocessing protocol described there. Namely, we
utilised the WITec Project FIVE software to crop the data to the region
$500-1800$cm-1, followed by baseline correction using the ‘shape’ method with
$\alpha=500$.

To assess the performance of the deep learning denoiser, we created ‘low-SNR
spectra’ by adding Gaussian noise to the original spectra. Each spectrum was
MinMax-normalised to the range 0–1 and Gaussian noise with a standard
deviation $\sigma=0.15$ was added. This resulted in spectra of similar noise
levels to those in deeper . These noisy samples were used as the input to the
model and the uncontaminated data was taken as ground-truth targets.

We then MinMax-normalised each spectrum (both inputs and targets) and compared
the performance of the neural network denoiser against six Savitzky-Golay
filters savgol . To make all models comparable, and to correct for potential
artefacts of how the model was trained originally in deeper , all denoising
metrics were computed after MinMax-normalising the denoised outputs of each
denoiser to the range 0–1 again.


# Team PKU-WICT-MIPL PIC Makeup Temporal Video Grounding Challenge 2022
Technical Report

Minghang Zheng Dejie Yang Zhongjie Ye Ting Lei Yuxin Peng Yang Liu  Wangxuan
Institute of Computer Technology, Peking UniversityBeijingChina minghang,
<EMAIL_ADDRESS>
(2018)

###### Abstract.

In this technical report, we briefly introduce the solutions of our team ‘PKU-
WICT-MIPL’ for the PIC Makeup Temporal Video Grounding (MTVG) Challenge in
ACM-MM 2022. Given an untrimmed makeup video and a step query, the MTVG aims
to localize a temporal moment of the target makeup step in the video. To
tackle this task, we propose a phrase relationship mining framework to exploit
the temporal localization relationship relevant to the fine-grained phrase and
the whole sentence. Besides, we propose to constrain the localization results
of different step sentence queries to not overlap with each other through a
dynamic programming algorithm. The experimental results demonstrate the
effectiveness of our method. Our final submission ranked 2nd on the
leaderboard, with only a 0.55% gap from the first.

Temporal Sentence Grounding, Natural language query, Sentence Localisation

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; ACM-MM Challenge; 2022††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Visual
content-based indexing and retrieval††ccs: Computing methodologies Activity
recognition and understanding

##  1\. Introduction

Given an untrimmed make-up video and a step query, the Makeup Temporal Video
Grounding (MTVG) (Wang et al., 2019) task aims to localize a temporal moment
of the target make-up step in the video. This task requires models to align
fine-grained video-text semantics and distinguish make-up steps with a subtle
difference. YouMakeUp (Wang et al., 2019) dataset has two characteristics: (1)
The step query is mainly composed of the actions (e.g., apply powder), the
tools (e.g., with the brush), and the face regions (e.g., on the eyelids),
which requires the model to understand the videos and queries in a more fine-
grained way. (2) Make-up steps are done step by step, which means that the
temporal moments corresponding to different steps do not overlap. However,
most of the existing temporal video grounding methods only localize the entire
query in a coarse-grained way. They cannot constrain the localization results
of different steps not to overlap.

To alleviate these problems, we propose to mine the phrase-level relationship
in the query in a fine-grained way, and constrain the localization results of
different steps to not overlap through a dynamic programming algorithm.
Specifically, we extract the phrases in the step query (e.g. ‘apply powder,’
‘on the eyelids,’ and ‘with the brush’), simultaneously localize the step
query and phrases, and comprehensively combine their localization results. In
addition, during inference, we design a dynamic programming algorithm to
select optimal temporal proposals for all step queries in a video and ensure
that they do not overlap.

##  2\. Methodology

We first extract the video features and build a 2D temporal feature map
following MMN (Wang et al., 2021). Then, we extract phrases in the step query,
encode them separately, and interact between the sentence and phrases through
a transformer encoder. Finally, each sentence and phrase feature will predict
an importance weight and calculate the cosine similarity with the 2d temporal
feature map to obtain the sentence and phrase score maps. The importance
weights are used to fuse the phrase and sentence score maps and output the
final score map. We train our network with binary cross-entropy loss and
cross-modal mutual matching loss following MMN (Wang et al., 2021), and
further introduce an exclusiveness loss to encourage different step query
predictions to be different from each other. In addition, during inference, we
design a dynamic programming algorithm to select optimal temporal proposals
for all step queries in a video and ensure that they do not overlap.

###  2.1. 2D Temporal Feature Map Encoder

We use pretrained CLIP (Radford et al., 2021) model (ViT-L/14@336px) to
extract the frame features (roughly three frames of CLIP features per second).
We also trim the videos by step queries and fine-tune the CLIP4Clip (Luo et
al., 2021) model with the video retrieval task. We fine-tune the CLIP4Clip
model initialized by the pretrained CLIP ViT-B/32 weight with learning rate
equals to 0.0001 and batch size equals to 64 for 5 epochs. We found that many
makeup actions can be inferred by comparing the changes of the face regions
before and after makeup. Thus, to help the model better understand the
structure of human face, we introduce FaRL (Zheng et al., 2021) face region
features. For each frame of the CLIP or CLIP4Clip features, we simply average
pool the face region features and concatenate them along channel dimension as
the final video inputs. Finally, we build the 2D temporal feature map
$V\in\mathbb{R}^{N\times N\times D}$ following MMN (Wang et al., 2021), where
$N=128$ is the number of video clips, $D=256$ is the feature dimension.

###  2.2. Phrase Extraction and Query Encoder

The step queries on YouMakeUp dataset are mainly composed of the actions (eg.
apply powder), the tools (eg. with brush), and the face regions (eg. on the
eyelids). In order to understand makeup step queries in a more fine-grained
way, we use off-the-shelf SRLBERT (Shi and Lin, 2019) model to extract phrases
from step descriptions. SRLBERT assigns semantic rule labels to each word in
the sentence, which serve as our phrases. Each phrase and the step sentence
query will be encoded by CLIP (Radford et al., 2021) text encoder separately.
Finally, we use a 3-layer transformer encoder (Vaswani et al., 2017) to
interact between the sentence query and phrases. The final sentence feature is
denoted as $T^{s}\in\mathbb{R}^{D}$, and the phrase feature is denoted as
$T^{p}\in\mathbb{R}^{N_{p}\times D}$, where $N_{p}$ is the number of phrases.

###  2.3. Phrase Similarity Learning

Instead of only localizing the step sentence query, we also localize the
phrases in a fine-grained way. We compute the cosine similarity of the
sentence/phrase features and the 2d temporal feature map respectively and
obtain the score maps, which represent the matching score between the temporal
proposals and the sentence/phrase:

(1) $S^{s}=\frac{VT^{s\top}}{\|V\|\|T^{s}\|}$ (2)
$S^{p}_{i}=\frac{VT_{i}^{p\top}}{\|V\|\|T_{i}^{p}\|},i=1,2,...,N_{p}$

where $S^{s}\in\mathbb{R}^{N\times N}$ is the sentence score map, and
$S^{p}\in\mathbb{R}^{N_{p}\times N\times N}$ is the phrase score maps. To
measure the importance of sentence and phrases, we use a fully connected
network to predict importance weights
$w^{s}\in\mathbb{R},w^{p}\in\mathbb{R}^{N_{p}}$ with text features
$T^{s},T^{p}$. We use a softmax activation function to ensure that these
weights sum to 1. Finally, the importance weight $w^{s},w^{p}$ will be used to
perform a weighted summation of the score maps $S^{s},S^{p}$ and output the
fused score map $S$:

(3) $S=w^{s}S^{s}+\sum_{i=1}^{N_{p}}w^{p}_{i}S^{p}_{i}\in\mathbb{R}^{N\times
N}$

###  2.4. Loss

We train our network with binary cross entropy loss $\mathcal{L}_{bce}$ and
cross-modal mutual matching loss $\mathcal{L}_{mm}$ following MMN (Wang et
al., 2021). The binary cross entropy loss uses the interaction over union
(IoU) between the ground-truth and the proposals to supervise the score map
$S$, and the cross-modal mutual matching loss contrast the positive moment-
sentence pairs with the negative ones sampled from both intra and inter
videos. We further introduce the exclusiveness loss $\mathcal{L}_{exc}$ to
encourage different step query predictions to be different from each other.
The exclusiveness loss find the top-2 matching scores for each proposal from
all the step queries in a video, and require their product to be 0. The
exclusivity loss encourages any proposal to have a high matching score with at
most one step query. The total loss of our method is shown blow:

(4)
$\mathcal{L}=\mathcal{L}_{bce}+\alpha\mathcal{L}_{mm}+\beta\mathcal{L}_{exc}$

###  2.5. Inference with Dynamic Programming

input : The number of steps $K$, the number of video clips $N$, and the log
probability scores ${S_{1},S_{2},..,S_{K}}\in\mathbb{N\times N}$

1
$f_{\mathbb{K},n}\leftarrow-\infty,\mathbb{K}\in\mathcal{P}(\\{1,2,...,K\\}),n\in\\{1,2,...N\\}$;

// $f_{\mathbb{K},n}$ represents the maximum score when allocate
nonoverlapping proposals whose end time belongs to the $n$-th clip to the
queries in the set $\mathbb{K}$

2
$g_{\mathbb{K},n}\leftarrow-\infty,\mathbb{K}\in\mathcal{P}(\\{1,2,...,K\\}),n\in\\{1,2,...N\\}$;

// $g$ is the maximum value in the prefix of $f$ i.e.
$g_{\mathbb{K},n}=\max_{s\in{1,2,...,n}}f_{\mathbb{K},s}$

3 for  _$n\leftarrow 1$ to $N$_ do

4 for  _$\mathbb{K}\in\mathcal{P}(\\{1,2,...,K\\})$_ do

5
$f_{\mathbb{K},n}\leftarrow\max_{k\in\mathbb{K},s\in\\{1,2,...,n\\}}\left(g_{\mathbb{K}-\\{k\\},s-1}+S_{k,s,n}\right)$;

6

7 for  _$\mathbb{K}\in\mathcal{P}(\\{1,2,...,K\\})$_ do

8
$g_{\mathbb{K},n}\leftarrow\max\left(g_{\mathbb{K},n-1},f_{\mathbb{K},n}\right)$;

9

10

11$\mathbb{K}\leftarrow\\{1,2,...,K\\}$;

12 $n\leftarrow\arg\max_{n\in\\{1,2,...,N\\}}f_{K,n}$;

13 while  _$\mathbb{K}\neq\emptyset$_ do

14
$k,s\leftarrow\arg_{k\in\mathbb{K},s\in\\{1,2,...,n\\}}\text{s.t.}f_{\mathbb{K},n}=g_{\mathbb{K}-\\{k\\},s-1}+S_{k,s,n}$;

output : The $k$-th step correspond to the video clips [s,n]

15 $\mathbb{K}\leftarrow\mathbb{K}-\\{k\\}$;

16 $n\leftarrow\arg\max_{n\in\\{1,2,...,s-1\\}}f_{K,n}$;

17

Algorithm 1 Inference with dynamic programming

As makeup steps are done step by step, the temporal moments corresponding to
different steps do not overlap. During inference, we design a dynamic
programming algorithm to select optimal temporal proposals for all step
queries in a video and ensure that they do not overlap. Specifically, we
regard the predicted scores for each proposal as the probability of choosing
that proposal. Let $P(k,s,n)$ denote the probability of choosing the proposal
$(s,n),1\leq s\leq n\leq N$ for the $k$-th step query and $K$ denote the total
number of steps. Our goal is to find a sequence of nonoverlapping proposals
$(s_{1},n_{1}),(s_{2},n_{2}),...,(s_{K},n_{K})$ that maximizes the joint
probability when the proposal $(s_{i},n_{i})$ is chosen for the $i$-th step
query. That is:

(5)
$\\{(s_{i},n_{i})\\}=\arg\max_{\\{(s_{i},n_{i})\\}}\Pi_{i}P(i,s_{i},n_{i})$

satisfy:

(6) $\exists\sigma\in\mathcal{P}_{K}\text{ s.t. }s_{\sigma_{1}}\leq
n_{\sigma_{1}}<s_{\sigma_{2}}\leq n_{\sigma_{2}}<...<s_{\sigma_{K}}\leq
n_{\sigma_{K}}$

where $\mathcal{P}_{K}$ represents the set of the permutations of $K$
elements. We use the logarithmic function to turn multiplications into
additions in the joint probability calculation and use the dynamic programming
algorithm in Algorithm 1 to solve the above optimization problem. Since the
complexity of the algorithm is $O(2^{K}N^{2}K)$, we will only execute the
algorithm for those samples with $K\leq 17$

##  3\. Experiments

Features | Methods | IoU=0.3 | IoU=0.5 | IoU=0.7 | AVG
---|---|---|---|---|---
CLIP | MMN (Wang et al., 2021) | 59.43 | 46.73 | 27.57 | 44.58
Ours-base | 60.15 | 47.46 | 28.26 | 45.29
+DP | 62.07 | 49.79 | 29.68 | 47.18
CLIP4Clip | Our-base | 68.42 | 53.90 | 34.07 | 52.13
Ours-exc | 68.42 | 54.31 | 33.85 | 52.19
Ours-exc-f | 69.59 | 53.74 | 33.12 | 52.15
| +DP | 70.22 | 56.83 | 34.73 | 53.93
| Ensemble | 72.12 | 59.77 | 40.29 | 57.39
| +DP | 75.56 | 63.21 | 42.25 | 60.34
Table 1. Performance of different methods on the val set. Methods | IoU=0.3 | IoU=0.5 | IoU=0.7 | AVG
---|---|---|---|---
Ensemble+DP | 73.61 | 62.50 | 42.12 | 59.41
Table 2. Performance of our ensemble model on test set.

As shown in Tab. 1, we compare our model with baseline MMN (Wang et al.,
2021). We denote our model without exclussiveness loss as Ours-base, denote
our model with exclussiveness loss as Ours-exc, and denote the full model with
face region feature as Ours-exc-f. ‘+DP’ represents inference with our dynamic
programming algorithm. ‘Ensemble’ represents ensemble the all the models
(without DP) by averaging the predicted score map.

Tab. 1 shows the results on the validation set. As we can see, (1) fine-
grained consideration of phrases and introduction of exclussiveness loss can
improve the performance. (2) During inference, the dynamic programming
algorithm ensures that the predictions do not overlap, and can improve
performance by about 2%. (3) The finetuned CLIP4Clip feature has a better
performance than the CLIP feature, which can improve the performance by about
6.8%. (4) Through model ensemble, the performance can be improved by 6.4%. The
test set performance of our ensemble model with dynamic programming achieves
the best performance on IoU=0.3 and IoU=0.5 and the second performance on
IoU=0.7 and AVG, as shown in Tab. 2.

##  4\. Qualitative Results

Fig. 1 shows some qualitative results of our model on the validation set.
According to our statistics, with dynamic programming only performed on videos
containing no more than 17 queries, we reduced the proportion of queries that
overlapped with other predictions from 49.1% to 17.7%, demonstrating the
effectiveness of the dynamic programming algorithm in minimizing the overlap.
In Fig. 1(a), our model successfully predicted all queries correctly. In Fig.
1(b), the video clip corresponding to the first query is split into multiple
shots, which results in our model predicting only one of the shots. At the
same time, the two queries refer to the same tool (i.e., with the brush) and
face region (i.e., cheekbone). The only difference is the makeup action (i.e.,
apply contour and apply highlight), yet the two actions are visually similar.
Our model fails to understand the difference between these two actions at a
fine-grained level and predicts the wrong answer.

##  5\. Conclusion

In our submission to the PIC Makeup Temporal Video Grounding Challenge 2022,
we explore the relationship between the phrases in the query in a fine-grained
way, and constrain the localization results of different steps to not overlap
through a dynamic programming algorithm. Experiments show the effectiveness of
our method and we win the 2nd place on the leaderboard, with only a 0.55% gap
from the first.

(a) Successful Case

(b) Failure Case

Figure 1. Qualitative results on the validation set.

###### Acknowledgements.

This work is supported by the grants from the National Natural Science
Foundation of China (61925201, 62132001, U21B2025), Zhejiang Lab (NO.
2022NB0AB05) and Beijing Institute for General Artificial Intelligence
(NO.A010121004).

## References

  * (1)
  * Luo et al. (2021) Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2021.  Clip4clip: An empirical study of clip for end to end video clip retrieval.  _arXiv preprint arXiv:2104.08860_ (2021).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International Conference on Machine Learning_. PMLR, 8748–8763.
  * Shi and Lin (2019) Peng Shi and Jimmy Lin. 2019.  Simple bert models for relation extraction and semantic role labeling.  _arXiv preprint arXiv:1904.05255_ (2019).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2019) Weiying Wang, Yongcheng Wang, Shizhe Chen, and Qin Jin. 2019\.  YouMakeup: A Large-Scale Domain-Specific Multimodal Dataset for Fine-Grained Semantic Comprehension. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 5136–5146.
  * Wang et al. (2021) Zhenzhi Wang, Limin Wang, Tao Wu, Tianhao Li, and Gangshan Wu. 2021.  Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding.  _CoRR_ abs/2109.04872 (2021).
  * Zheng et al. (2021) Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. 2021.  General Facial Representation Learning in a Visual-Linguistic Manner.  _arXiv preprint arXiv:2112.03109_ (2021).