# Source-Free Online Domain Adaptive Semantic Segmentation of Satellite Images
under Image Degradation

###### Abstract

Online adaptation to distribution shifts in satellite image segmentation
stands as a crucial yet underexplored problem. In this paper, we address
source-free and online domain adaptation, i.e., test-time adaptation (TTA),
for satellite images, with the focus on mitigating distribution shifts caused
by various forms of image degradation. Towards achieving this goal, we propose
a novel TTA approach involving two effective strategies. First, we
progressively estimate the global Batch Normalization (BN) statistics of the
target distribution with incoming data stream. Leveraging these statistics
during inference has the ability to effectively reduce domain gap.
Furthermore, we enhance prediction quality by refining the predicted masks
using global class centers. Both strategies employ dynamic momentum for fast
and stable convergence. Notably, our method is backpropagation-free and hence
fast and lightweight, making it highly suitable for on-the-fly adaptation to
new domain. Through comprehensive experiments across various domain adaptation
scenarios, we demonstrate the robust performance of our method.

Index Terms—  Satellite Image, Test-Time Adaptation, Domain Adaptation,
Segmentation

##  1 Introduction

In real-world satellite image segmentation, numerous situations can arise that
require domain adaptation. For instance, variations between satellite images
taken from different cities or diverse sensor noise characteristics from
different satellites can lead to discernible domain disparities. These
disparities can hinder a model trained on a specific dataset from generalizing
effectively to target datasets. Unsupervised domain adaptation (UDA) [1, 2]
helps bridge these gaps but typically requires both source and target data,
which isn’t always feasible due to privacy and storage constraints. To address
this, methods like [3, 4] have been proposed, allowing adaptation without the
need for source data, known as source-free domain adaptation (SFDA).

Both UDA and SFDA require the entire target domain data to be available during
adaptation. Additionally, the adaptation is performed offline, meaning that it
occurs before the model is deployed. However, many situations might arise when
online adaptation is necessary, especially in cases of domain shifts during
the online segmentation of satellite images using the deployed model. For
example, during natural disasters such as earthquakes, floods, and wildfires,
online segmentation is crucial for quickly identifying affected areas,
assessing damage, and prioritizing rescue and relief efforts. Online
segmentation can also be used to monitor crop health, detect disease
outbreaks, and estimate yields. However, such online segmentation scenarios
can face major challenges due to the degradation of satellite images caused by
various factors, including different characteristics of sensor noise,
atmospheric and weather conditions (e.g., rain, fog). Hence, the development
of a model that can adapt to such domain shifts on-the-fly and without having
access to source data becomes crucial. Such source-free and online adaptation
is also termed as test-time adaptation (TTA) in the literature [5].

In this paper, we propose a novel TTA approach for satellite image
segmentation. Previous research [6] has shown that utilizing Batch
Normalization (BN) statistics during inference instead of the source pre-
trained statistics can effectively mitigate domain gap. By employing a running
average method with dynamic momentum, we accurately approximate target BN
statistics. This significantly reduces the domain gap and enhances performance
compared to source models. Furthermore, we take a running average of confident
pixels to keep track of global class centers to further refine the prediction
map. Importantly, our method does not require any backpropagation during
adaptation; hence, it is fast and lightweight. This feature is highly
desirable in scenarios like TTA of satellite images, where images are
typically high-resolution and can take up a significant amount of storage. A
lightweight adaptation method ensures it does not impose additional memory
constraints.

Moreover, an additional limitation in the current domain adaptation research
for satellite images is that it mainly focuses on adapting between different
geographic regions, overlooking domain gaps that arise from various sources of
degradation such as sensor noise, weather conditions, etc. Addressing such
domain gaps is even more crucial during online adaptation, as mentioned
earlier. The primary reason behind this is the scarcity of datasets dedicated
to these specific forms of degradation. To address the gap, we introduce a new
synthetic satellite image dataset by simulating different forms of image
degradation. In summary, our contributions are:

  * •

We address a novel problem of source-free and online domain adaptive semantic
segmentation of satellite images under various forms of image degradation. To
the best of our knowledge, this is the first work to explore such setting for
satellite image segmentation.

  * •

We present a novel TTA method that involves approximating the target BN
statistics and global class centers with incoming data stream using dynamic
momentum value.

  * •

We evaluate our method on the TTA of satellite images and demonstrate that our
method outperforms state-of-the-art generic TTA approaches on this task.

##  2 Background

Related Works:  Previous research [1, 2, 7] have examined UDA for satellite
images. Notable methods include generative approaches [2], discrepancy-based
approaches [7], and self-training [8]. UDA for satellite images has also been
extended to the source-free case by [3, 4]. Recently, many generic TTA methods
have been proposed for both classification and segmentation tasks. In
particular, TENT [5] adapts a pre-trained source model on incoming target data
by minimizing entropy and updating the Batch-Norm parameters of the source
model. On the other hand, DUA [9] continuously updates the BN statistics of
the pre-trained source model with the incoming test batches in order to align
to the target distribution. Also, DIGA [10] mixes source statistics with
incoming target statistics for adaptation.

Problem Definition: Consider a segmentation model $f_{{\theta}_{0}}$ pre-
trained on the source image data $\mathcal{X}_{s}\sim\mathcal{D}_{s}$, where
$\mathcal{D}_{s}$ denotes the source distribution. During deployment, the
model encounters a sequence of test data
$\mathcal{X}_{1}\rightarrow\mathcal{X}_{2}\rightarrow\ldots\rightarrow\mathcal{X}_{t}\rightarrow\ldots$,
where $\mathcal{X}_{t}$ represents a small batch of test samples from the
source degraded stationary target distribution $\mathcal{D}_{\textit{T}}$.
Following the TTA setting, the model only has access to $\mathcal{X}_{t}$ at
an instance. Furthermore, the model needs to adapt to each incoming test batch
$\mathcal{X}_{t}$ and update its parameters from
$f_{{\theta}_{t-1}}\rightarrow f_{{\theta}_{t}}$ in order to improve its
output on the subsequent test batches.

##  3 Method

Table 1: Adaptation Performance in IoU (%) of our method and baseline methods. The model is pre-trained on the clean and uncorrupted training set and during test-time, adapted to the source-degraded target domain in an online manner. Task | Method | IoU (%) | mIoU (%)
---|---|---|---
Agriculture | Rangeland | Forest | Water | Barren | Urban
Clean $\downarrow$ Gaussian Noise | Source-Only | 32.4 | 14.3 | 37.5 | 42.2 | 29.0 | 55.8 | 35.2
TENT [5] | 74.5 | 16.9 | 60.8 | 29.8 | 46.8 | 60.1 | 48.2
DUA [9] | 67.3 | 19.7 | 52.8 | 47.8 | 48.4 | 63.9 | 50.0
DIGA [10] | 75.8 | 16.6 | 62.2 | 29.4 | 54.1 | 62.4 | 50.1
OURS | 77.3 | 18.6 | 66.5 | 45.0 | 52.2 | 64.0 | 53.9
Clean $\downarrow$ Impulse Noise | Source-Only | 37.2 | 15.3 | 47.0 | 40.7 | 27.3 | 56.4 | 37.3
TENT | 75.3 | 18.0 | 64.5 | 31.3 | 48.7 | 61.5 | 49.9
DUA | 71.1 | 20.1 | 61.1 | 44.2 | 47.8 | 64.9 | 51.5
DIGA | 75.2 | 20.0 | 61.9 | 25.1 | 52.7 | 63.3 | 49.7
OURS | 78.1 | 22.9 | 67.8 | 48.6 | 54.0 | 65.5 | 56.1
Clean $\downarrow$ Gaussain Blur | Source-Only | 78.3 | 16.8 | 66.2 | 54.9 | 57.9 | 62.2 | 56.1
TENT | 78.6 | 19.5 | 68.4 | 61.6 | 58.2 | 60.9 | 57.9
DUA | 80.3 | 22.7 | 68.5 | 58.5 | 60.1 | 66.1 | 59.4
DIGA | 80.5 | 16.1 | 70.9 | 64.6 | 59.5 | 64.1 | 59.3
OURS | 81.4 | 23.9 | 73.3 | 67.5 | 61.3 | 64.6 | 62.0
Clean $\downarrow$ Snow | Source-Only | 0.0 | 0.0 | 6.5 | 3.7 | 2.1 | 8.7 | 3.5
TENT | 68.5 | 14.8 | 59.0 | 20.3 | 32.4 | 46.2 | 40.2
DUA | 11.0 | 9.0 | 34.9 | 10.2 | 15.2 | 22.2 | 17.1
DIGA | 50.5 | 12.6 | 57.0 | 14.7 | 24.2 | 48.3 | 34.5
OURS | 72.1 | 16.2 | 62.1 | 32.7 | 42.1 | 49.1 | 45.7
Clean $\downarrow$ Fog | Source-Only | 11.7 | 6.4 | 8.9 | 4.4 | 18.3 | 25.6 | 12.6
TENT | 72.2 | 15.4 | 45.9 | 31.6 | 37.3 | 47.1 | 41.6
DUA | 51.0 | 15.3 | 32.3 | 10.3 | 31.4 | 39.6 | 30.0
DIGA | 68.0 | 14.4 | 45.4 | 38.8 | 37.5 | 45.2 | 41.6
OURS | 75.6 | 18.6 | 57.4 | 43.9 | 43.9 | 51.9 | 48.6

Our main objective is to achieve efficient online adaptation without imposing
significant additional resources. To meet this objective, we employ two
backpropagation-free strategies that significantly enhance performance in the
new domain.

Distribution Matching: Image degradation can cause the feature distribution of
target domain to be different from source distribution. Hence, we need to
align both the distribution for adaptation. Li et al. [6] have demonstrated
that utilizing the BN statistics of the target domain during inference instead
of the pre-trained source statistics can effectively mitigate domain gap.
However, TTA usually assumes very small batch size per instance for
adaptation. Hence, approximating the correct BN statistics for target
distribution with such small amount of data per instance is difficult. To
address the problem, we utilize running average of BN statistics to
progressively approximate the global BN statistics of the incoming domain.
Specifically, during testing on the current batch $t$, let
$\tilde{\mu}_{l_{c}}$ and $\tilde{\sigma}_{l_{c}}$ represent the BN statistics
(mean and variance) computed from the $l$-th layer’s $c$-th channel of the
model. We calculate two running averages as follows:

$\displaystyle\mu_{l_{c}}^{t}$
$\displaystyle=(1-\alpha^{t})\times\mu_{l_{c}}^{t-1}+\alpha^{t}\times\tilde{\mu}_{l_{c}}$
(1) $\displaystyle\sigma_{l_{c}}^{t}$
$\displaystyle=(1-\alpha^{t})\times\sigma_{l_{c}}^{t-1}+\alpha^{t}\times\tilde{\sigma}_{l_{c}}^{t}\
.$ (2)

Here, $\mu_{l_{c}}^{t}$ and $\sigma_{l_{c}}^{t}$ denote the running BN
statistics that get updated with each incoming test batch. We initialize
$\mu_{l_{c}}^{0}$ and $\sigma_{l_{c}}^{0}$ with the BN statistics of the pre-
trained source model, and $\alpha^{t}$ is the momentum value that controls the
effect of current batch statistics on the running statistics.

In fact, designing the momentum term accurately is important. While a constant
value may serve its purpose, it weighs all the incoming test batches equally
to update the running BN statistics. A high value of momentum renders the
running stats sensitive to outliers. On the other hand, setting a small
constant value for the momentum, will result in the running stats to converge
slowly. To overcome these challenges, we adopt a dynamic approach for the
momentum, gradually reducing its value after each iteration according to the
formula:

$\displaystyle\alpha^{t}=\alpha^{t-1}\times\gamma_{dm}$ (3)

Here, $\gamma_{dm}$ is a constant in the interval $(0,1)$. This dynamically
decreasing momentum ensures that the running statistics eventually stabilize
to a constant value, serving as an approximation of the global BN statistics
for the target domain.

Instance Matching: Corruption or degradation can cause a notable divergence in
the visual attributes of specific individual pixels in target images compared
to the source image, potentially leading to inaccurate predictions. To address
the problem, inspired by previous works [10, 11, 12] that use prototype-based
methods, we calculate a running average of the features of confident classes
and leverage them to guide the recognition of wrongly recognized pixels.
However, unlike previous approaches, we use a dynamic momentum value that
helps for faster and more stable convergence.

Specifically, let the output logits of the segmentation model after correcting
for BN Statistics, be denoted as $F\in\mathbb{R}^{C\times H\times W}$, where
$C$, $H$, and $W$ denote the channels, height, and width of the feature map,
respectively. The classifier takes $F$ to predict the softmax output
$L\in\mathbb{R}^{K\times H\times W}$, where $K$ is the number of classes. The
straightforward definition of a region or class center of class $i$ for the
current sample can be defined as:

$\displaystyle\tilde{R}_{i}=\frac{\sum_{x,y}F_{(x,y)}\mathbb{I}[\text{argmax}(L_{(x,y)})=i,\text{max}(L_{(x,y)})\geq\mathcal{P}_{0}]}{\sum_{x,y}\mathbb{I}[\text{argmax}(L_{(x,y)})=i,\text{max}(L_{(x,y)})\geq\mathcal{P}_{0}]}$
(4)

where $\mathbb{I}(\cdot)$ is the binary indicator denoting whether the pixel
belongs to class $i$ and the predicted probability for that class is greater
than $\mathcal{P}_{0}$. For each class center, we compute a running average to
capture the global class center prototype of that particular domain. Define
$R_{i}^{t}$ as follows:

$\displaystyle R_{i}^{t}$ $\displaystyle=(1-\beta^{t})\times
R_{i}^{t-1}+\beta^{t}\times\tilde{R_{i}}$ (5)

Here also, we use a dynamic momentum value for $\beta$:

$\displaystyle\beta^{t}=\beta^{t-1}\times\gamma_{im}$ (6)

Given the class centers, we can calculate the instance-matched prediction
$P\in\mathbb{R}^{K\times H\times W}$ as follows:

$\displaystyle
P_{(K=i,x,y)}=\frac{\text{exp}(-<F_{(x,y)},R_{i}^{t}>)}{\sum_{K\neq
i}\text{exp}(-<F_{(x,y)},R_{i}^{t}>)}$ (7)

Then, the final prediction mask $M$ can be computed by weighting with
$\gamma$:

$\displaystyle M_{(x,y)}=(1-\gamma)\times L_{(x,y)}+\gamma\times P_{(x,y)}$
(8)

##  4 Experiments

Dataset: We use the benchmark satellite dataset DeepGlobe [13] as our primary
dataset. To simulate some of the most common image corruptions observed in
satellite images, such as Gaussian Noise, Impulse Noise (commonly known as
Salt and Pepper Noise), Gaussian Blur, Fog, and Snow, we utilize the
RobustBench [14] simulation. These corruptions are applied to the DeepGlobe
test set.

Baseline Models: For the baseline models, we have selected state-of-the-art
generic TTA algorithms - TENT [5], DUA [9], and DIGA [10]. The source codes
for all the baseline methods are publicly available. We also report the
Source-Only performance, which is the result obtained from the evaluation of
the source model on the target distribution without performing adaptation.

Implementation Details: We have utilized DeepLab-v3 [15] with a ResNet-50
backbone as our segmentation model. The hyperparameters we have used include
$\alpha^{0}=\beta^{0}=0.9$, $\gamma_{dm}=\gamma_{im}=0.95$, $\gamma=0.2$, and
$\mathcal{P}_{0}=0.5$. DeepGlobe images are resized to a shape of $512\times
512$. Our batch size is set to $8$. As for the baseline methods, we have
followed the hyperparameter choices specified in their respective papers.

Results: Our adaptation performance is presented in Table 1. Initially, the
source model is trained on the clean and uncorrupted DeepGlobe train set.
During testing, we sequentially adapt the clean model to each batch of target
images from the target domain involving image degradation. As observed in
Table 1, degradation can severely impact the performance of the Source-Only
model, even to the extent where the source performance drops to as low as
$3.5\%$, particularly in the case of snow. This is a significant drop
considering the fact that the source model achieves an mIoU of $62.9\%$ on the
clean and uncorrupted DeepGlobe test set. This phenomenon underscores the
critical need for adaptation in the context of degraded satellite images.
Furthermore, Table 1 demonstrates that in all cases, our method has
consistently outperformed all the baseline models in terms of mean
performance. We also provide visualizations of the predicted segmentation
masks generated by our method in Figure 1. For comparison purposes, we show
the predictions of the Source-Only model. It is evident that the naive source-
only prediction, without adaptation, performs poorly. Conversely, our method
generates output masks that closely resemble the actual ground truth.

Ablation Study: Here, we aim to demonstrate the effectiveness of the two
strategies employed in our method. Specifically, we focus on the Clean to
Impulse Noise adaptation task, and the results are presented in Table 2. It
clearly illustrates that both strategies are individually effective in
adaptation. Furthermore, combining the results of both methods leads to even
greater improvements in performance.

Table 2: Ablation study for the distribution matching (DM) and instance matching (IM) strategies used in our method. DM | IM | mIoU (%)
---|---|---
✗ | ✗ | 37.3
✗ | ✓ | 50.7
✓ | ✗ | 54.8
✓ | ✓ | 56.1

In our method, we have employed a dynamic momentum value for both $\alpha$ and
$\beta$. We now demonstrate the effectiveness of using dynamic momentum over a
fixed value for both parameters.

Table 3: Efficacy of dynamic momentum. Momentum | 0.1 | 0.5 | 0.9 | Ours
---|---|---|---|---
mIoU (%) | 34.8 | 51.4 | 47.3 | 56.1

As shown in Table 3, it is clear that using dynamic momentum is more effective
than using a constant value. Additionally, constant values have the
disadvantage of being sensitive to outliers and may require fine-tuning for
different datasets based on domain distance. In contrast, dynamic momentum is
robust to these disadvantages and ensures a faster and stable convergence.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |
Input | GT Mask | Source-Only | Ours
Fig. 1: Comparison of the predicted mask of our method with the Source-Only
model. The rows correspond to examples of Gaussian Noise, Impulse Noise,
Gaussian Blur, Snow and Fog from top to bottom respectively. Color-coded
legends:  (Agriculture),  (Rangeland),  (Forest),  (Water),  (Barren),
(Urban).

##  5 Conclusion

In this paper, we have addressed the problem of source-free and online
adaptation of satellite images to distribution shifts caused by various forms
of image degradations. To address this issue, we have presented a novel TTA
method that utilizes the running average of target BN statistics to
approximate the global BN statistics. Additionally, we employ global class
centers to further refine the segmentation masks. Our method is
backpropagation-free and lightweight, making it well-suited for online
adaptation of satellite images. Through various experiments, we have
demonstrated that our approach outperforms several state-of-the-art generic
TTA methods in this task.

Acknowledgment. This work was supported by Institute of Information &
communications Technology Planning & Evaluation (IITP) grant by the Korea
government (MSIT) (No.RS-2023-00230337,Advanced and Proactive AI Platform
Research and Development Against Malicious Deepfakes).

## References

  * [1] Hui Chen, Liang Cheng, Ning Li, YunChang Yao, Jian Cheng, and Ka Zhang,  “S&gda: An unsupervised domain adaptive semantic segmentation framework considering both imaging scene and geometric domain shifts,”  IEEE Transactions on Geoscience and Remote Sensing, 2023.
  * [2] Onur Tasar, Yuliya Tarabalka, Alain Giros, Pierre Alliez, and Sebastien Clerc,  “Standardgan: Multi-source domain adaptation for semantic segmentation of very high resolution satellite images by data standardization,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  * [3] Qingsong Xu, Yilei Shi, Xin Yuan, and Xiao Xiang Zhu,  “Universal domain adaptation for remote sensing image scene classification,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–15, 2023.
  * [4] Zun Xu, Wei Wei, Lei Zhang, and Jiangtao Nie,  “Source-free domain adaptation for cross-scene hyperspectral image classification,”  in IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 2022, pp. 3576–3579.
  * [5] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell,  “Tent: Fully test-time adaptation by entropy minimization,”  in International Conference on Learning Representations, 2021.
  * [6] Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou,  “Revisiting batch normalization for practical domain adaptation,”  arXiv preprint arXiv:1603.04779, 2016.
  * [7] Jun Zhang, Jiao Liu, Bin Pan, and Zhenwei Shi,  “Domain adaptation based on correlation subspace dynamic distribution alignment for remote sensing image scene classification,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 11, pp. 7920–7930, 2020.
  * [8] Chenbin Liang, Bo Cheng, Baihua Xiao, and Yunyun Dong,  “Unsupervised domain adaptation for remote sensing image segmentation based on adversarial learning and self-training,”  IEEE Geoscience and Remote Sensing Letters, vol. 20, pp. 1–5, 2023\.
  * [9] M Jehanzeb Mirza, Jakub Micorek, Horst Possegger, and Horst Bischof,  “The norm must go on: dynamic unsupervised domain adaptation by normalization,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14765–14775.
  * [10] Wei Wang, Zhun Zhong, Weijie Wang, Xi Chen, Charles Ling, Boyu Wang, and Nicu Sebe,  “Dynamically instance-guided adaptation: A backward-free approach for test-time domain adaptive semantic segmentation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 24090–24099.
  * [11] Jake Snell, Kevin Swersky, and Richard Zemel,  “Prototypical networks for few-shot learning,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [12] Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong-Wah Ngo, and Tao Mei,  “Transferrable prototypical networks for unsupervised domain adaptation,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 2239–2247.
  * [13] Ilke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar,  “Deepglobe 2018: A challenge to parse the earth through satellite images,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 172–181.
  * [14] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein,  “Robustbench: a standardized adversarial robustness benchmark,”  in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  * [15] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam,  “Rethinking atrous convolution for semantic image segmentation,”  arXiv preprint arXiv:1706.05587, 2017.


Predicting the Temperature Dependence of Surfactant CMCs Using
Graph Neural Networks

Christoforos Brozosa,b, Jan G. Rittigb, Sandip Bhattacharyaa, Elie Akannya,
Christina Kohlmanna, Alexander Mitsosd,b,c,∗

  * a

BASF Personal Care and Nutrition GmbH, Henkelstrasse 67, 40589 Duesseldorf,
Germany

  * b

RWTH Aachen University, Process Systems Engineering (AVT.SVT), Aachen, Germany

  * c

Forschungszentrum Jülich GmbH, Institute for Energy and Climate Research
IEK-10: Energy Systems Engineering, Jülich, Germany

  * d

JARA-ENERGY, Aachen, Germany

## Abstract

The critical micelle concentration (CMC) of surfactant molecules is an
essential property for surfactant applications in industry. Recently,
classical QSPR and Graph Neural Networks (GNNs), a deep learning technique,
have been successfully applied to predict the CMC of surfactants at room
temperature. However, these models have not yet considered the temperature
dependency of the CMC, which is highly relevant for practical applications. We
herein develop a GNN model for temperature-dependent CMC prediction of
surfactants. We collect about 1400 data points from public sources for all
surfactant classes, i.e., ionic, nonionic, and zwitterionic, at multiple
temperatures. We test the predictive quality of the model for following
scenarios: i) when CMC data for surfactants are present in the training of the
model in at least one different temperature, and ii) CMC data for surfactants
are not present in the training, i.e., generalizing to unseen surfactants. In
both test scenarios, our model exhibits a high predictive performance of
R${}^{2}\geq$ 0.94 on test data. We also find that the model performance
varies by surfactant class. Finally, we evaluate the model for sugar-based
surfactants with complex molecular structures, as these represent a more
sustainable alternative to synthetic surfactants and are therefore of great
interest for future applications in the personal and home care industries.

##  1 Introduction

Surfactants are used in a wide variety of industries such as cosmetics, food
additives, detergents, oil recovery, and pharmaceuticals due to their unique
properties (Vieira et al.,, 2021; Shaban et al.,, 2020; Nitschke and Costa,,
2007; Tadros,, 2005; Adu et al.,, 2020; Szűts and Szabó-Révész,, 2012;
Massarweh and Abushaikha,, 2020). Surfactants are amhiphilic molecules
containing hydrophilic (head) and hydrophobic (tail) parts. A surfactant
property of major interest is the critical micelle concentration (CMC), which
is the concentration above which self-aggregation into micelles takes place
and is highly relevant in many applications (Su et al.,, 2020; Thompson et
al.,, 2023; Ghezzi et al.,, 2021; Kumar and Mandal,, 2019). The CMC is
accompanied by sharp changes in the bulk solution properties (Rosen and
Kunjappu,, 2012; Myers,, 2020).

The influence of temperature on CMC is of great importance in surfactant
applications, such as washing and oil recovery. Specifically, the CMC is
influenced by multiple parameters that are related to the surfactant itself,
such as the structure of the tail and nature of the head group, and parameters
related to the solute environment, such as pH, temperature, and the presence
of electrolytes (Rosen and Kunjappu,, 2012; Myers,, 2020; Katritzky et al.,,
2008; Thiruvengadam et al.,, 2020). The temperature is of particular interest,
since no model applicable to all surfactant classes is available in the
literature. Many researchers have studied the temperature effect on the CMC,
which in general has been described as complex and varies for each surfactant
class (Rosen and Kunjappu,, 2012; Myers,, 2020). For nonionics, authors have
reported that CMC decreases monotonically as the temperature increases
(Lindman et al.,, 2016), but in general CiEj (polyethylene oxide) type of
surfactants exhibit a minimum CMC at temperature around 50∘C (Rosen and
Kunjappu,, 2012; Myers,, 2020; Kroll et al.,, 2022; Chen et al.,, 1998). In
contrast, MEGA (N-alkanoyl-N-methylglucamide) surfactants show a U-shaped
relationship (Prasad et al.,, 2006; Okawauchi et al.,, 1987). Ionic
surfactants normally exhibit a U-shaped relationship too, with their minimum
CMC value being between 20-30∘C (Rosen and Kunjappu,, 2012; Perger and Bešter-
Rogač,, 2007; Fu et al.,, 2019). It has been recently shown by Brinatti et
al., (2014) that increasing the temperature causes a decrease in the CMC of
some zwitterionics and others to exhibit a minimum CMC. The latter is also
shown in the work of Mukerjee and Mysels (Mukerjee and Mysels,, 1971). All the
above are experimental observations for individual surfactants, but no model
for explicit CMC predictions at multiple temperatures of a wide spectrum of
surfactants has been derived. This leads to limitations in predicting the
temperature-dependent CMC of new surfactant molecules and highlights the need
to consider the surfactant structure in the development of predictive models.

Both bio-based surfactants and biosurfactants are of particular interest in
personal and home care industries since they enable a transition from fossil
to renewable feedstocks (Farias et al.,, 2021; Jahan et al.,, 2020). In the
literature, biosurfactants are defined as those produced by biotechnology
while bio-based surfactants are surfactants comprised of sustainable 100%
natural-based feedstocks (Drakontis and Amin,, 2020). Both bio-based
surfactants and biosurfactants contain sugar-based groups, hence are also
referred to as sugar-based surfactants. The general structure comprises a
polar carbohydrate, like glucose, as the hydrophilic part and an alkyl chain
as the hydrophobic part (Ruiz,, 2008). Gaudin et al., (2019) recently
discussed the impact of this chemical structure on the sugar-based surfactant
properties. Specifically, they state that the CMC can be affected by the
stereochemistry and the anomeric configuration of the sugar head. Although
sugar-based surfactants belong to nonionics, the effect of temperature on
their solution properties differs from the ethoxylated ones (Molina-Bolívar et
al.,, 2004). Researches have reported a decrease in CMC with the increase of
temperature (Molina-Bolívar et al.,, 2004) and a U-shaped relationship in
other cases (Paula et al.,, 1995; Angarten and Loh,, 2014; Castro et al.,,
2018), thus showing again a complex relationship between surfactant structure,
CMC and temperature.

The importance of CMC as a key characteristic surfactant property has driven
many researchers to investigate the relationship between surfactant structure
and CMC (Rosen and Kunjappu,, 2012). Empirical equations have been derived
from experimental data to describe this relationship, but they are only
applicable to specific systems (Rosen and Kunjappu,, 2012; Klevens,, 1953).
Recently, numerous quantitative structure-property relationship (QSPR) type
models were developed for CMC prediction (Katritzky et al.,, 2008; Gaudin et
al.,, 2016; Hu et al.,, 2010; Mattei et al.,, 2013; Wang et al.,, 2018;
Abooali and Soleimani,, 2023; Seddon et al.,, 2022). In QSPR models, molecular
descriptors are estimated from the molecular structure and mapped to the
target property. One limitation is that they typically require manual
selection and identification of effective molecular descriptors.

An alternative end-to-end deep learning approach, called graph neural networks
(GNNs), has been successfully applied to numerous molecular property
prediction tasks (Schweidtmann et al.,, 2020; Rittig et al., 2023a, ; Gilmer
et al.,, 2017; Rittig et al., 2023b, ; Sanchez Medina et al.,, 2023, 2022;
Yang et al.,, 2019). GNNs are applied directly to the molecular graph and
extract the necessary structural information which they later use to predict
the target property, thereby providing an end-to-end learning framework
(Hamilton,, 2020). Due to their broad success and adoption, GNNs have been
effectively applied to predict the CMC and surface excess concentration
($\Gamma$m) of surfactant monomers (Qin et al.,, 2021; Brozos et al.,, 2024;
Moriarty et al.,, 2023). For both approaches, i.e., QSPR and GNNs, the
temperature-dependency of CMC is rarely studied. In fact, the effect of the
temperature on the CMC has only been modeled in one recently published QSPR
model, which is however limited to anionic surfactants (Abooali and
Soleimani,, 2023). We therefore aim at investigating the prediction of the
temperature effect on CMC for all common surfactant types namely ionic,
nonionic, and zwitterionic classes.

In the present work, we develop a GNN model for predicting the temperature-
dependent CMC of all surfactant classes. To train the models, we extend our
database of CMC values at room temperature for different surfactant classes
from our previous work by adding CMC measurements at various from publicly
available sources (Brozos et al.,, 2024). The extended database contains
around 1400 measurements from 492 unique surfactant molecules. We then develop
a GNN model that learns to extract structural information from the surfactant
molecules in a fixed-size vector, known as the molecular fingerprint, in which
we concatenate the experimental temperature. The updated vector is then mapped
to the measured CMC of the surfactant. We incorporate stereochemistry and
anomeric configuration information into the molecular graph representation. We
train and fine-tune our model on the newly constructed database. We then test
our model to predict the temperature-dependent CMC of i) surfactant structures
seen during training but at different temperatures, and ii) completely new
surfactants, i.e., not seen during training. Furthermore, we particularly
explore the temperature dependency of the CMC of sugar-based surfactants. We
show that in both test cases, the GNN model provides accurate predictions.

The remainder of this work is constructed as follows: First a description of
the collected data set and of the test sets is provided in Section 2. In
Section 3, we present the concept of the GNN for predicting temperature-
dependent CMCs. Subsequently, the predictive quality of the developed GNN
model is presented, discussed, and compared with previous works in Section 4.
Finally, we conclude this contribution in Section 5. The test data used for
model evaluation is publicly available in our GitHub repository and the model
architecture is presented in Section 3. The training data set and the trained
models remain property of BASF and could be made available upon request.

##  2 Data set

###  2.1 Data set overview

We collected a data set of temperature-dependent CMC values for a variety of
surfactants. Particularly, we extended the assembled CMC data set from our
previous work (Brozos et al.,, 2024), to include CMC information at multiple
temperatures for each molecule, when such data was available. In the data
collection process, we prioritized CMC values measured through tensiometry and
we excluded any duplicates. For those surfactants for which we did not find
tensiometry data, we considered CMC values determined by conductometry, light
scattering, refractive index and calorimetry. Each data point includes the
CMC, the temperature, and the isomeric SMILES string of the surfactant,
allowing to distinguish between different anomers, e.g.,
octyl-$\alpha$/$\beta$-D-glycoside, and chiral centers in the sugar head,
e.g., glucoside with galactoside (Weininger,, 1988). In total our new data set
consists of 1,377 data points, with 492 unique surfactant structures of which
227 structures were measured at least at two different temperatures. The
minimum temperature of our data is 0∘C and the maximum is 90∘C. The
temperature distribution of the data set is illustrated in Figure 1. A
detailed statistical analysis of the CMC data can be found in the Appendix A,
cf. Figure A9.

Figure 1: Distribution of the CMC data with the number of data points over the
temperature range of T$=10-90$ for a bin size of 10∘C.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: Experimental values of CMC (in mM) at different temperatures for six
surfactants present in our database are tabulated: (S1)
dodecyltrimethylammonium chloride (DTAC) (Perger and Bešter-Rogač,, 2007),
(S2) benzyl (3-hexadecanoylaminoethyl)dimethylammonium chloride
(C15AEtBzMe2Cl) (Galgano and El Seoud,, 2010), (S3) Decyl diglucoside
(Brinatti et al.,, 2014), (S4) Sulfobetaine 14 (Cheng et al.,, 2012), (S5)
Sodium decyl 2 sulfate (Mukerjee and Mysels,, 1971) and (S6) Sodium decanoate
(González-Pérez et al.,, 2005). Each of these surfactants shows a different
temperature dependence.

The different relationships in each surfactant classes between temperature and
CMC discussed in Section 1 are also present in our data set. In Figure 2, six
surfactant examples are illustrated. Specifically, the three ionic
surfactants, namely S1, S5, and S6 exhibit a U-shaped relationship, as often
is the case for ionic surfactants. The minimum CMC is between 30 and 40∘C. An
exception to the U-shaped relationship is S2, an ionic surfactant too, as an
increase in temperature causes the CMC to increase too. Finally, the sugar-
based nonionic surfactant S3 and the zwitterionic surfactant S4 exhibit a
U-shaped relationship.

Table 1: A description of the nature of the data set used in this paper. The number of surfactants per class in the complete data set and the test sets are listed. | Full data set | Test sets
---|---|---
|  | Different temperature | Distinct surfactant
Anionics | 576 | 100 | 89
Nonionics | 422 | 62 | 68
Cationics | 291 | 49 | 50
Zwitterionics | 88 | 16 | 11
Total data points | 1,377 | 227 | 218

###  2.2 Data splits

We implement two types of data set splitting: i) _different temperature_ – for
testing the prediction accuracy of our model at new temperatures; ii)
_distinct surfactant_ – for testing the ability of our model to generalize to
new, unseen surfactant structures, hence structures not included in the model
training at all. We note that the two test sets have a similar surfactant
class distribution in Table 1, which was not enforced.

For selecting the _different temperature_ test set, we first identify all
unique surfactant molecules with CMCs in at least two different temperatures,
and then randomly select one data point for each of these molecules to be in
the test set. For example, if for a given surfactant there exist 3
measurements at 3 different (distinct) temperatures in the full data set, one
will be randomly assigned to the test set and the other two remain in the
training set. A total of 227 molecules at various temperatures are selected
for testing, which accounts for about 16% of the whole data set size.

The _distinct surfactant_ test set aims to evaluate the models’ predictive
performance for completely unseen surfactant molecules at various
temperatures. For consistency in the performance comparison, we choose similar
test set size, namely 218 data points. We randomly select molecules and add
all corresponding data points to the test, such that the temperature-dependent
CMC values of these molecules remain completely unseen during training. In
this final test set, about 70% of the data points were molecules measured at
different/multiple temperatures. For these molecules, we include all available
CMC values measured at different temperatures in the test set. The remaining
30% of the test set are molecules measured only at one temperature. In total,
50 distinct surfactant structures are included in the test set.

##  3 Methods

In the following sections, we first present the structure of our GNN model and
describe ensemble learning (Section 3.1). Then, we summarize the
hyperparameter selection and training settings of our model (Section 3.2).

###  3.1 Graph Neural Networks and ensemble learning

We adapt our GNN model for predicting CMC values of surfactant monomers from
our previous work (Brozos et al.,, 2024) to include the temperature
dependency. An overview of the model framework is illustrated in Figure 3. The
GNN takes as input a surfactant molecule, represented as an undirected
molecular graph, where atoms correspond to nodes (vertices) and bonds to
edges. Each node and each edge is assigned a feature vector, where chemical
information about the corresponding atom/bond is stored. The node and edge
features used in the current work are presented in Tables B5 and B6
respectively. GNNs operate directly on the node and edge features of the
undirected molecular graph (Gilmer et al.,, 2017). During graph convolutions,
the node features, denoted as hidden states, are updated with structural
information from their neighborhood. Then, a pooling step is applied. That is,
the updated hidden state of all nodes within the graph, which now contain
information about the respective node itself and the neighbor nodes, are
combined, e.g., by applying the sum operator, into a single unique vector
representation, known as the molecular fingerprint. To include the
temperature, we first normalized it between a minimum of 0 and a maximum of 10
and we concatenated the molecular fingerprint with it. Finally, the updated
molecular fingerprint is used to perform the downstream property prediction
task, herein predicting the CMC, in a standard multilayer perceptron (MLP).
For a detailed overview of GNNs, including different types of graph
convolutions and pooling operators we refer to works on graph convolutions
layers (cf. (Hamilton et al.,, 2018; Veličković et al.,, 2018; Xu et al.,,
2019; Simonovsky and Komodakis,, 2017)) and pooling operators (cf. (Cangea et
al.,, 2018; Vinyals et al.,, 2016; Schweidtmann et al.,, 2023; Vu et al.,,
2023; Buterez et al.,, 2023)), also see reviews in (Zhou et al.,, 2020; Wu et
al.,, 2021).

Figure 3: Schematic representation of the developed graph neural network for
predicting temperature-dependent CMC values of surfactant monomers.

In our previous work, we considered stereochemistry information on the edge
feature vector (Brozos et al.,, 2024). As both chirality and stereochemistry
of a surfactant molecule impact the CMC, we herein additionally use chirality
information on the atom feature vector too. We note that more sophisticated
approaches exist in the literature (Adams et al.,, 2021) and that using 3D
information could be beneficial but requires to determine or calculate 3D
coordinates and conformers which is computationally expensive. In our work, we
aim to investigate whether the GNN model that learns from molecular graphs
with chirality information can perform accurate predictions of CMC in
different anomers and isomers too.

We further apply ensemble learning. Ensemble learning is a common technique in
machine learning to reduce the noise of randomly chosen training and
validation sets. Multiple models are trained, i.e., on different splits of
training and validation sets, i.e., non-test data, and their predictions are
averaged, thus leading to more robust and generalized predictions (Breiman,,
1996; Dietterich,, 2000; Ganaie et al.,, 2022). We herein train 40 different
models on both split types mentioned in Section 2.2, and then we average out
the predictions to report the prediction accuracy of the ensemble of GNNs.

###  3.2 Implementation and hyperparameter

To determine the optimal hyperparameter values, we train our GNN model on 40
different seeded validation sets, similar to our previous works (Schweidtmann
et al.,, 2020; Rittig et al., 2023a, ; Rittig et al.,, 2022; Brozos et al.,,
2024). The size of the validation set is kept constant at 200 molecules, which
represents about 14% of the whole data set size and thus a general training-
validation-test split of 70:14:16. We use the _different temperature_ test set
for our hyperparameter tuning and we select the ones leading to the minimum
root mean squared error (RMSE) on the validation set. We scale the CMC
($\mu$M) values using a (based 10) logarithmic scale. We use a grid search to
investigate the hyperparameters of the GNN model described in Table B7. Note
that the first layer of the MLP has a size of 129 neurons because we are
concatenating the normalized temperature on the molecular fingerprint

We represent each surfactant molecule with an isomeric SMILES string
(Weininger,, 1988). We use RDKit (_version 2022.3.5_), an open-source toolkit
for cheminformatics to generate the attributed molecular graph for each
surfactant. For the graph convolutions, the GINE-operator (Xu et al.,, 2019;
Hu et al.,, 2020) as implemented in PyTorch Geometric (PyG) (Fey and Lenssen,,
2019) is applied with sum being the pooling layer of choice. We use sum
pooling as it has been shown to be beneficial for molecular-dependent
prediction tasks (Schweidtmann et al.,, 2023).

##  4 Results and discussion

In this section, the predictive performance of our GNNs models is evaluated
(Section 4.1). Then, we compare our GNN model with predictive CMC models from
previous works (Section 4.2). We further analyze the predictive performance
for different temperature ranges (Section 4.3) and investigate the influence
of the surfactant classes on the CMC predictions (Section 4.4). Finally, the
results on selected sugar-based surfactants are presented, which are
interesting due to their sustainability aspects (Section 4.5).

###  4.1 Predictive performance on different temperature and distinct
surfactant test sets

We first evaluate the model performance for two different test scenarios (cf.
Section 2.2): the _different temperature_ , i.e., predicting the CMC of
surfactants that have been included in training but at different temperatures,
and the _distinct surfactant_ , i.e., predicting the CMC at different
temperatures of surfactants that have not been used for training at all. Table
2 shows the mean absolute error (MAE), the RMSE, and the mean absolute
percentage error (MAPE) on the validation and the test set for both scenarios.
Here, we report the performance of the ensemble of GNNs since we found
ensemble learning to be beneficial in our previous work (Schweidtmann et al.,,
2020; Brozos et al.,, 2024). All reported error metrics refer to the log CMC.

Table 2: Summary of GNN ensemble accuracy on the two test set splits for predicting the temperature-dependent CMC on logarithmic scale. The ensembles consist of 40 GNNs. The accuracy is given for the following metrics: MAE = mean absolute error, RMSE = root mean squared error (unit: log CMC), MAPE = mean absolute percentage error (unit %). | Different temperature | Distinct surfactant
---|---|---
| RMSE | MAE | MAPE | R2 | RMSE | MAE | MAPE | R2
GNN ensemble | 0.173 | 0.113 | 3.561 | 0.97 | 0.251 | 0.173 | 5.702 | 0.94

In the _different temperature_ scenario, the ensemble of GNNs exhibits an RMSE
of 0.173, a MAE of 0.113, a MAPE of 3.561, and a high R2 of 0.97. For the
_distinct surfactant_ set, the ensemble of GNNs performs predictions on
previously unseen surfactant molecules at single and multiple temperatures
with an RMSE of 0.251, an MAE of 0.173, a MAPE of 5.702 and an R2 of 0.94.
Compared to the different temperature split, the errors slightly increase,
which is expected since predicting the CMC of completely unseen surfactants is
more difficult than predicting the CMC of a surfactant, for which CMC values
at other temperatures have been included in training the models.

Furthermore, we show parity plots for the predictions of the ensemble of GNN
models for both test scenarios, different temperature and distinct surfactant,
in Figure 4. For both scenarios, the models show very good agreement between
measured and predicted data, as most of the points lie close to the diagonal,
resulting in high R2 scores of 0.97 and 0.94, respectively. We notice only a
small number of outliers in both cases; the four points with the highest
absolute error (AE) are reported in Figures A10 and A11, respectively. The
exact model predictions are directly compared with the experimental
measurements in Tables B8 and B9.

Overall, the GNN ensembles provide a high predictive quality, also indicating
generalization capabilities to new surfactants not included in the training.

(a)

(b)

Figure 4: Parity plots of the ensemble of GNNs on the two test data sets: (a)
different temperature and (b) distinct surfactant. Surfactant classes are
highlighted with different colors and markers. The logarithm is applied to CMC
in $\mu$M (base 10).

###  4.2 Comparison with previous works

We further compare the predictive quality of our model to previously developed
GNNs from the literature for CMC prediction. An overview of past model
predictive performance is given in Table 3. Note that previous works have used
different test sets, making a direct comparison difficult. Qin et al., (2021)
developed a GNN model for predicting CMCs of surfactant at constant
temperature and reported an RMSE of 0.30 on their test set of 22 molecules.
This indicates a slightly worse performance on a less diverse test set
compared to our GNN model with an RMSE of 0.25 evaluated on 218 different
point with 50 distinct molecules. In another recent work, Moriarty et al.,
(2023) analyzed the GNN model developed by Qin et al., (2021) and reported a
combined GNN model with Gaussian Process (GP) with a test RMSE of 0.21, which
is slightly lower than our findings. However, the test set was again limited
to 22 molecules, which is only about 40% of ours, and therefore may not
represent the diversity of surfactant structures used in different practical
applications. In their complementary data set, which contains 43 distinct
surfactant molecules with some being outside the applicability domain of the
model developed by Qin et al., (2021), their best-performing model exhibited
an RMSE of 1.32, which is almost six times higher than our findings. In our
recent work, we examined the performance of single- and multi-task learning on
predicting the CMC and the surface excess concentration ($\Gamma_{m}$) of
surfactant monomers with GNNs but without temperature dependency (Brozos et
al.,, 2024). The multi-task GNN exhibited an RMSE of 0.31 and an R2 of 0.94
for the CMC, which are slightly worse than our current results. Therefore, we
herein increase the predictive quality of GNNs for CMC prediction, although
the presented prediction task includes the temperature dependency of the CMC
and is thus presumably more challenging. We suspect, that the larger data set
used in the present work (1,377 data points compared to 429 data points in our
previous work (Brozos et al.,, 2024)) positively influences the predictive
capabilities of the GNNs. Overall, we find our GNN model to perform on a
higher level (better test accuracy) with a broader applicability range, more
diverse surfactant chemistry, than previous works.

Table 3: Comparison of different GNN models for predicting the CMC of
surfactants. Dash (–) indicates that values were not provided in previous
studies.

Work | Test set size | Temperature dependency | RMSE | R2
---|---|---|---|---
Qin et al., (2021) | 22 | No | 0.30 | 0.91
Moriarty et al., (2023) | 22 | No | 0.21 | –
Moriarty et al., (2023) \- complementary set | 43 | No | 1.32 | –
Previous work (Brozos et al.,, 2024) | 65 | No | 0.31 | 0.94
Current work | 218 | Yes | 0.25 | 0.94

###  4.3 Analysis of temperature impact on model accuracy

We further examine the model performance for both test sets, different
temperature and distinct surfactant, in the whole temperature spectrum.
Therefore, we separate the individual temperatures into temperature bins (of
10∘C ranges) and we calculate the MAPE in each one. The results are plotted in
Figure 5. In most temperature bins, the MAPE is higher on the distinct
surfactant split than on the new temperature split, which is expected due to
the more challenging prediction task (cf. Section 4.1). We do not observe any
temperature range with a significant high error in both test scenarios. As
expected, the highest amount of measurements are between 20∘C and 30∘C.

Figure 5: The mean absolute percentage error (MAPE) on the two test data sets:
different temperature (yellow) and distinct surfactant (blue). The mentioned
ranges exclude the left limit but include the right limit, for example (0 -
10] and (10 - 20]. The corresponding number of data points for each
temperature range are denoted at the top of the respective bars.

We further visualize the temperature dependence of the GNN predicted CMC
versus the experimental CMC and the Absolute Percentage Error (APE) for four
randomly selected surfactants from the distinct surfactant test set in Figure
6. Please note that different scales of CMC are used in the plots since the
order of CMC magnitude varies for different surfactants (cf. Section 2 and
Appendix A). The scale of APE in Figure 6d also differs from the rest. Given
this wide range of CMC values, Figure 6 shows that the model accurately
predicts the order of magnitude of CMC at multiple temperatures. The model
also identifies trends for the temperature dependency of the CMC for most
cases, e.g., the U-shaped relationship for the two surfactants in Figures
6b-c. For sodium decanoate shown in Figure 6a, the decrease in the CMC for the
range of 15-40∘C is predicted correctly, but the increase of the CMC up to
55∘C is not captured. The highest APE (12%) is observed at Figure 6d at 40∘C.
While general trends are mostly captured, the model seems to underestimate the
temperature effect on the CMC. The lack of high sensitivity to temperature
changes may due to the fact that for some surfactants only marginal changes in
CMC value are observed at different temperatures, as shown in Figure 2, which
are even less pronounced on the logarithmic scale used for model training.
Thus, additional training data and further model refinement would be desirable
to fully capture the detailed temperature effects in future work.

(a) Sodium decanoate (Blanco et al.,, 2005).

(b) Potassium perfluorooctanoate (Muzzalupo et al.,, 1995; Mukerjee and
Mysels,, 1971).

(c) Octoxynol 4 (Mukerjee and Mysels,, 1971).

(d) C13E8 (Meguro et al.,, 1981; Schick,, 1987).

Figure 6: The GNN predicted versus experimental temperature-dependent
performance on four surfactants, part of our test set is shown. Predicted and
measured log CMC values are plotted together and connected through a cubic
interpolation in the corresponding insets. The logarithm is applied to CMC in
$\mu$M.

###  4.4 Predictive performance per surfactant class

As discussed previously (cf. Section 1), the temperature effect of the CMC is
unique for surfactants belonging to different surfactant classes. Thus, from a
product development viewpoint, analyzing the results for each class is highly
interesting. For example, one would want to identify surfactants and classes
that exhibit a lowering of CMC with temperature, since the minimum CMC value
and the corresponding temperature may be valuable for different applications
(e.g., in laundry applications).

We calculate the MAPE per surfactant class for both test scenarios and report
the results in Figure 7. Overall, all surfactant classes demonstrate a MAPE
$<8\%$. Ionic surfactants have a slightly higher error on the distinct
surfactant split in comparison to the different temperature split, as unseen
surfactant structures are introduced on the test set (cf. Section 2.2).
Specifically, anionic and cationic surfactants have the lowest errors in total
for the distinct surfactant split. Since they generally exhibit a U-shaped
relationship, the model is able to make accurate predictions. Zwitterionics,
do not always exhibit the same general CMC-temperature relationship, and
therefore decreased model accuracy is observed. For nonionic surfactants, the
MAPE is more than doubled in the distinct surfactant split. Here, sugar-based
surfactants are also considered. The high complexity of CMC dependency on
temperature for such surfactants decreases model accuracy. The model
performance on sugar-based surfactants is investigated in the next subsection
of this work. Overall, we find model performance to be highly correlated with
the number of possible CMC-temperature relationships (U-shaped, monotonically
decrease, etc.) in each surfactant class.

Figure 7: The mean absolute percentage error (MAPE) per surfactant class on
the two test data sets: different temperature (yellow) and distinct surfactant
(blue). The corresponding number of data points for each temperature range are
denoted at the top of the respective bars.

###  4.5 Predictive performance on sugar-based surfactants

Bio-based surfactants and biosurfactants that are comprised of sustainable and
natural-based feedstock are of technological importance to personal and home
care industries. These often contain either single or multiple sugar groups,
which control surfactant properties like foaming, cleansing, etc. Such sugar-
based surfactants are complex molecules with many chiral centers on the sugar
head. Therefore different anomeric configurations of the sugar groups and
hence the entire surfactant structure are possible. In GNNs operating on the
molecular graph, which is a 2D representation of the molecule (cf. Section
3.1), such spatial/geometric information is typically solely captured as an
atomic feature, thus making it challenging to accurately predict properties of
complex sugar-based surfactants. Therefore, we analyze the performance of the
model on some selected sugar-based surfactants present in the _distinct
surfactant_ test set.

We plot the temperature-dependent CMC predictions together with the
experimental CMC values from the literature, as well as the APE per
temperature in Figure 8. Since absolute (not logarithmic) CMC values are also
of interest in practical, industrial applications, we report the absolute
predictions on the three sugar-based surfactants in Table B10.

(a) Decyl $\beta$-D-maltoside (Angarten and Loh,, 2014).

(b) Decyl $\beta$-D-glucoside (Castro et al.,, 2018).

(c) Octyl $\beta$-D-Thioglucoside (Molina-Bolívar et al.,, 2004).

Figure 8: Ensembled predictions on sugar-based surfactant in the distinct
surfactant test set. Predicted and measured log CMC are plotted together and
connected through cubic interpolation. The logarithm is applied to CMC in
$\mu$M. The absolute percentage error (APE) per temperature is also
illustrated.

We observe the ability of our model to accurately predict the magnitude of the
CMC values for two out of three sugar-based surfactants, namely decyl
$\beta$-D-glycoside and decyl $\beta$-D-maltoside. This is evident by the low
APE for these two surfactants across all temperatures. Furthermore, the
correct trend of the temperature dependency is identified, except for decyl
$\beta$-D-maltoside at 10∘C. The reason may be the low amount of measurements
at this temperature, as shown above in Figure 1. On the other hand, our model
fails to accurately predict the real CMC values of octyl
$\beta$-D-Thioglucoside. As noted by Gaudin et al., (2019), the nature of the
linker, i.e., the connection between the carbohydrate and the alkyl chain,
impacts the surfactant properties and consequently the CMC. Replacing the
ether -O- with thioether -S- linker, leads to incorrect predictions from our
model. We note that our data set contains no other sugar-based surfactant with
thioether as the linker. Thus, expanding the data set further with more
experimental data on sugar-based surfactants would improve the predictions of
our model.

We are further interested in the effect of anomeric configuration and
chirality centers of sugar-based surfactants on the CMC predictions. Since all
previously discussed sugar-based surfactants had an $\beta$ anomeric
configuration, we analyze the model performance on heptyl-$\alpha$-D-mannoside
and heptyl-$\alpha$-D-galactoside at a single temperature. The calculated and
measured CMC values are given in Table 4. We observe a relatively high
absolute error for both surfactants, namely 0.306 and 0.283 respectively. Note
here that the overall MAE was found to be 0.173 (cf. Section 4.1). The most
often present sugar-based surfactants on our database are of the
n-alkanoyl-$\beta$-D-glucoside type. In comparison with them, both surfactants
exhibit structural differences in their anomeric configuration and chirality
centers of the sugar head. Specifically, our training data set contains only
two surfactants with mannoside and three surfactants with galactoside as the
polar head. Given the limited amount of similarly structured surfactants in
the training set, the model provides acceptable CMC predictions but with an
above-average error compared to the overall MAPE of the distinct surfactant
test set. To enable the GNN to additionally learn different structures/linkers
in sugar-based surfactants, systematic CMC data on these surfactants should be
provided in the future.

Table 4: Comparison between predicted and experimental CMC values for two
sugar-based surfactants. The CMC values outside the parenthesis given are the
logarithmic ones. The CMC values inside the parenthesis represent the absolute
value and are given in mM. Both surfactants are measured at 25∘C.

Surfactant | Predicted CMC ($\mu$M) | Measured CMC ($\mu$M) | Source
---|---|---|---
Heptyl $\alpha$-D-mannoside | 4.792 (61.937) | 4.49 (30.6) |  (Gaudin et al.,, 2019)
Heptyl $\alpha$-D-galactoside | 4.795 (62.373) | 4.512 (32.5) |  (Gaudin et al.,, 2019)

##  5 Conclusion

We develop a GNN model for the prediction of the temperature-dependent CMC of
surfactants. For this, we first assemble a data set with 1,377 CMC
measurements at multiple temperatures ranging from 0∘C to 90∘C for 492 unique
surfactants covering all surfactant classes, i.e., anionic, cationic,
zwitterionic, and nonionic surfactants. We then use this data set to train an
ensemble of GNN models directly on the molecular graph of the surfactants and
the temperature information, thereby enabling the prediction of temperature
dependency of the CMC, a key interest in personal and home care applications.

The GNN model exhibits very high predictive quality. Specifically, the model
shows accurate predictions for surfactants included in the training but at a
different temperature and for surfactants not seen during training at all,
thus generalizing to new surfactants. Model performance remained constant
throughout all temperature ranges but varied per surfactant class. The GNN
model also outperforms previously developed data-driven models for predicting
the CMC at constant temperature. Detailed temperature dependencies are not
fully captured by the model yet, i.e., the model underestimates the
sensitivity of the CMC to changes in the temperature, demonstrating the need
for additional temperature-dependent CMC data and model refinement.

We specifically investigate sustainably sourced sugar-based surfactants. The
GNN model incorporates anomeric and chirality information into the feature
vector and can thus distinguish between different anomers and isomers. The GNN
provides accurate predictions of the CMC at multiple temperatures for two
exemplary sugar-based surfactants but also shows limitations in
differentiating ether (-O-) and thioether (-S-) linker in a third selected
sugar-based surfactant. Further evaluation of the anomeric and chirality
information effect on the model performance showed an above-average error and
model limitations in distinguishing between different chiral centers on the
sugar head. Yet, the model accurately predicts the order of magnitude of the
temperature-dependent CMC values for sugar-based surfactants.

Future work should focus on acquiring additional temperature-dependent CMC
data for model training. To account for the CMC relation to the geometry of
the molecular structure, which is particularly relevant for sugar-based
surfactants, geometric GNNs that consider atomic coordinates could be
investigated. Also, extending the GNN model to incorporate pH conditions and
defining the validity domain of the model would be insightful extensions.
Finally, obtaining chemical intuition on how the model performs predictions by
using explainable AI methods could help to better understand surfactant self-
aggregation.

## Acknowledgments

The BASF authors (C. Brozos, S. Bhattacharya, E. Akanny, and C. Kohlmann) were
funded by the BASF Personal Care and Nutrition GmbH. J. G. Rittig and A.
Mitsos acknowledge funding from the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) – 466417970 – within the Priority Programme “SPP
2331: Machine Learning in Chemical Engineering”. Additionally, J. G. Rittig
acknowledges the support of the Helmholtz School for Data Science in Life,
Earth and Energy (HDS-LEE).

## Data availability

All Python scripts and the test data used in this work are available as open-
source in our GitHub repository.

## Author contribution

Christoforos Brozos: Conceptualization, Methodology, Software, Data curation,
Validation, Formal analysis, Writing - Original Draft, Writing - Review &
Editing, Visualization

Jan G. Rittig: Conceptualization, Methodology, Software, Formal analysis,
Writing - Review & Editing

Sandip Bhattacharya: Conceptualization, Methodology, Formal analysis,
Supervision, Writing - Review & Editing

Elie Akanny: Conceptualization, Methodology, Experimental methodology &
measurements, Writing - Review & Editing

Christina Kohlmann: Writing - Review & Editing, Supervision, Funding
acquisition

Alexander Mitsos: Writing - Review & Editing, Supervision, Funding acquisition

## Bibliography

  * Abooali and Soleimani, (2023) Abooali, D. and Soleimani, R. (2023).  Structure-based modeling of critical micelle concentration (cmc) of anionic surfactants in brine using intelligent methods.  Scientific Reports, 13(1):13361.
  * Adams et al., (2021) Adams, K., Pattanaik, L., and Coley, C. W. (2021).  Learning 3d representations of molecular chirality with invariance to bond rotations.  ArXiv, abs/2110.04383.
  * Adu et al., (2020) Adu, S. A., Naughton, P. J., Marchant, R., and Banat, I. M. (2020).  Microbial biosurfactants in cosmetic and personal skincare pharmaceutical formulations.  Pharmaceutics, 12.
  * Angarten and Loh, (2014) Angarten, R. G. and Loh, W. (2014).  Thermodynamics of micellization of homologous series of alkyl mono and di-glucosides in water and in heavy water.  The Journal of Chemical Thermodynamics, 73:218–223.  Special issue honoring Professor Manuel Ribeiro da Silva.
  * Blanco et al., (2005) Blanco, E., González-Pérez, A., Ruso, J. M., Pedrido, R., Prieto, G., and Sarmiento, F. (2005).  A comparative study of the physicochemical properties of perfluorinated and hydrogenated amphiphiles.  Journal of Colloid and Interface Science, 288(1):247–260.
  * Breiman, (1996) Breiman, L. (1996).  Bagging predictors.  Machine Learning, 24(2):123–140.
  * Brinatti et al., (2014) Brinatti, C., Mello, L. B., and Loh, W. (2014).  Thermodynamic study of the micellization of zwitterionic surfactants and their interaction with polymers in water by isothermal titration calorimetry.  Langmuir, 30(21):6002–6010.
  * Brozos et al., (2024) Brozos, C., Rittig, J. G., Bhattacharya, S., Akanny, E., Kohlmann, C., and Mitsos, A. (2024).  Graph neural networks for surfactant multi-property prediction.
  * Buterez et al., (2023) Buterez, D., Janet, J. P., Kiddle, S. J., Oglic, D., and Liò, P. (2023).  Modelling local and general quantum mechanical properties with attention-based pooling.  Communications Chemistry, 6(1):262.
  * Cangea et al., (2018) Cangea, C., Veličković, P., Jovanović, N., Kipf, T., and Liò, P. (2018).  Towards sparse hierarchical graph classifiers.
  * Castro et al., (2018) Castro, G., Garrido, P. F., Amigo, A., and Brocos, P. (2018).  Boosting the use of thermoacoustimetry in micellization thermodynamics studies by easing an objective determination of the cmc.  Fluid Phase Equilibria, 478:1–13.
  * Chen et al., (1998) Chen, L.-J., Lin, S., and Huang, C.-C. (1998).  Effect of hydrophobic chain length of surfactants on enthalpy-entropy compensation of micellization.  Journal of Physical Chemistry B, 102:4350–4356.
  * Cheng et al., (2012) Cheng, C.-j., Qu, G.-m., Wei, J.-j., Yu, T., and Ding, W. (2012).  Thermodynamics of micellization of sulfobetaine surfactants in aqueous solution.  Journal of Surfactants and Detergents, 15(6):757–763.
  * Dietterich, (2000) Dietterich, T. G. (2000).  Ensemble methods in machine learning.  In Proceedings of the First International Workshop on Multiple Classifier Systems, MCS ’00, page 1–15, Berlin, Heidelberg. Springer-Verlag.
  * Drakontis and Amin, (2020) Drakontis, C. E. and Amin, S. (2020).  Biosurfactants: Formulations, properties, and applications.  Current Opinion in Colloid & Interface Science, 48:77–90.  Formulations and Cosmetics.
  * Farias et al., (2021) Farias, C. B. B., Almeida, F. C., Silva, I. A., Souza, T. C., Meira, H. M., de Cássia F. Soares da Silva, R., Luna, J. M., Santos, V. A., Converti, A., Banat, I. M., and Sarubbo, L. A. (2021).  Production of green surfactants: Market prospects.  Electronic Journal of Biotechnology, 51:28–39.
  * Fey and Lenssen, (2019) Fey, M. and Lenssen, J. E. (2019).  Fast graph representation learning with pytorch geometric.  ArXiv, abs/1903.02428.
  * Fu et al., (2019) Fu, D., Gao, X., Huang, B., Wang, J., Sun, Y., Zhang, W., Kan, K., Zhang, X., Xie, Y., and Sui, X. (2019).  Micellization, surface activities and thermodynamics study of pyridinium-based ionic liquid surfactants in aqueous solution.  RSC Adv., 9:28799–28807.
  * Galgano and El Seoud, (2010) Galgano, P. D. and El Seoud, O. A. (2010).  Micellar properties of surface active ionic liquids: A comparison of 1-hexadecyl-3-methylimidazolium chloride with structurally related cationic surfactants.  Journal of Colloid and Interface Science, 345(1):1–11.
  * Ganaie et al., (2022) Ganaie, M., Hu, M., Malik, A., Tanveer, M., and Suganthan, P. (2022).  Ensemble deep learning: A review.  Engineering Applications of Artificial Intelligence, 115:105151.
  * Gaudin et al., (2019) Gaudin, T., Lu, H., Fayet, G., Berthauld-Drelich, A., Rotureau, P., Pourceau, G., Wadouachi, A., Van Hecke, E., Nesterenko, A., and Pezron, I. (2019).  Impact of the chemical structure on amphiphilic properties of sugar-based surfactants: A literature overview.  Advances in Colloid and Interface Science, 270:87–100.
  * Gaudin et al., (2016) Gaudin, T., Rotureau, P., Pezron, I., and Fayet, G. (2016).  New qspr models to predict the critical micelle concentration of sugar-based surfactants.  Ind. Eng. Chem. Res., 55(45):11716–11726.
  * Ghezzi et al., (2021) Ghezzi, M., Pescina, S., Padula, C., Santi, P., Del Favero, E., Cantù, L., and Nicoli, S. (2021).  Polymeric micelles in drug delivery: An insight of the techniques for their characterization and assessment in biorelevant conditions.  Journal of Controlled Release, 332:312–336.
  * Gilmer et al., (2017) Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. (2017).  Neural message passing for quantum chemistry.
  * González-Pérez et al., (2005) González-Pérez, A., Ruso, J. M., Romero, M. J., Blanco, E., Prieto, G., and Sarmiento, F. (2005).  Application of thermodynamic models to study micellar properties of sodium perfluoroalkyl carboxylates in aqueous solutions.  Chemical Physics, 313(1):245–259.
  * Hamilton, (2020) Hamilton, W. L. (2020).  Graph representation learning.  Morgan & Claypool Publishers.
  * Hamilton et al., (2018) Hamilton, W. L., Ying, R., and Leskovec, J. (2018).  Inductive representation learning on large graphs.
  * Hu et al., (2010) Hu, J., Zhang, X., and Wang, Z. (2010).  A review on progress in qspr studies for surfactants.  International journal of molecular sciences, 11:1020–47.
  * Hu et al., (2020) Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., and Leskovec, J. (2020).  Strategies for pre-training graph neural networks.
  * Jahan et al., (2020) Jahan, R., Bodratti, A. M., Tsianou, M., and Alexandridis, P. (2020).  Biosurfactants, natural alternatives to synthetic surfactants: Physicochemical properties and applications.  Advances in Colloid and Interface Science, 275:102061.
  * Katritzky et al., (2008) Katritzky, A. R., Pacureanu, L. M., Slavov, S. H., Dobchev, D. A., and Karelson, M. (2008).  Qspr study of critical micelle concentrations of nonionic surfactants.  Ind. Eng. Chem. Res., 47(23):9687–9695.
  * Klevens, (1953) Klevens, H. B. (1953).  Structure and aggregation in dilate solution of surface active agents.  Journal of the American Oil Chemists Society, 30(2):74–80.
  * Kroll et al., (2022) Kroll, P., Benke, J., Enders, S., Brandenbusch, C., and Sadowski, G. (2022).  Influence of temperature and concentration on the self-assembly of nonionic ciej surfactants: A light scattering study.  ACS Omega, 7(8):7057–7065.
  * Kumar and Mandal, (2019) Kumar, A. and Mandal, A. (2019).  Critical investigation of zwitterionic surfactant for enhanced oil recovery from both sandstone and carbonate reservoirs: Adsorption, wettability alteration and imbibition studies.  Chemical Engineering Science, 209:115222.
  * Lindman et al., (2016) Lindman, B., Medronho, B., and Karlström, G. (2016).  Clouding of nonionic surfactants.  Current Opinion in Colloid & Interface Science, 22:23–29.
  * Massarweh and Abushaikha, (2020) Massarweh, O. and Abushaikha, A. S. (2020).  The use of surfactants in enhanced oil recovery: A review of recent advances.  Energy Reports, 6:3150–3178.
  * Mattei et al., (2013) Mattei, M., Kontogeorgis, G. M., and Gani, R. (2013).  Modeling of the critical micelle concentration (cmc) of nonionic surfactants with an extended group-contribution method.  Ind. Eng. Chem. Res., 52(34):12236–12246.
  * Meguro et al., (1981) Meguro, K., Takasawa, Y., Kawahashi, N., Tabata, Y., and Ueno, M. (1981).  Micellar properties of a series of octaethyleneglycol-n-alkyl ethers with homogeneous ethylene oxide chain and their temperature dependence.  Journal of Colloid and Interface Science, 83:50–56.
  * Molina-Bolívar et al., (2004) Molina-Bolívar, J. A., Aguiar, J., Peula-García, J. M., and Ruiz, C. C. (2004).  Surface activity, micelle formation, and growth of n-octyl-$\beta$-d-thioglucopyranoside in aqueous solutions at different temperatures.  J. Phys. Chem. B, 108(34):12813–12820.
  * Moriarty et al., (2023) Moriarty, A., Kobayashi, T., Salvalaglio, M., Angeli, P., Striolo, A., and McRobbie, I. (2023).  Analyzing the accuracy of critical micelle concentration predictions using deep learning.  J. Chem. Theory Comput., 19(20):7371–7386.
  * Mukerjee and Mysels, (1971) Mukerjee, P. and Mysels, K. J. (1971).  Critical micelle concentrations of aqueous surfactant systems.  Technical report, National Standard reference data system.
  * Muzzalupo et al., (1995) Muzzalupo, R., Ranieri, G. A., and La Mesa, C. (1995).  Solution properties of alkali metal perfluoroalkanoates.  Colloids and Surfaces A: Physicochemical and Engineering Aspects, 104(2):327–336.
  * Myers, (2020) Myers, D. (August 2020).  Surfactant Science and Technology, 4th Edition.  John Wiley & Sons, Ltd.
  * Nitschke and Costa, (2007) Nitschke, M. and Costa, S. (2007).  Biosurfactants in food industry.  Trends in Food Science & Technology, 18(5):252–259.
  * Okawauchi et al., (1987) Okawauchi, M., Hagio, M., Ikawa, Y., Sugihara, G., Murata, Y., and Tanaka, M. (1987).  A light-scattering study of temperature effect on micelle formation of n-alkanoyl-n-methylglucamines in aqueous solution.  Bulletin of the Chemical Society of Japan, 60(8):2719–2725.
  * Omar and Abdel-Khalek, (1998) Omar, A.-A. M. A. and Abdel-Khalek, N. A. (1998).  Surface and thermodynamic parameters of some cationic surfactants.  J. Chem. Eng. Data, 43(1):117–120.
  * Paula et al., (1995) Paula, S., Sues, W., Tuchtenhagen, J., and Blume, A. (1995).  Thermodynamics of micelle formation as a function of temperature: A high sensitivity titration calorimetry study.  The Journal of Physical Chemistry, 99(30):11742–11751.
  * Perger and Bešter-Rogač, (2007) Perger, T.-M. and Bešter-Rogač, M. (2007).  Thermodynamics of micelle formation of alkyltrimethylammonium chlorides from high performance electric conductivity measurements.  Journal of Colloid and Interface Science, 313(1):288–295.
  * Prasad et al., (2006) Prasad, M., Chakraborty, I., Rakshit, A. K., and Moulik, S. P. (2006).  Critical evaluation of micellization behavior of nonionic surfactant mega 10 in comparison with ionic surfactant tetradecyltriphenylphosphonium bromide studied by microcalorimetric method in aqueous medium.  J. Phys. Chem. B, 110(20):9815–9821.
  * Qin et al., (2021) Qin, S., Jin, T., Van Lehn, R. C., and Zavala, V. M. (2021).  Predicting critical micelle concentrations for surfactants using graph convolutional neural networks.  The journal of physical chemistry. B, 125:10610–10620.
  * (51) Rittig, J. G., Ben Hicham, K., Schweidtmann, A. M., Dahmen, M., and Mitsos, A. (2023a).  Graph neural networks for temperature-dependent activity coefficient prediction of solutes in ionic liquids.  Computers and Chemical Engineering, 171:108153.
  * (52) Rittig, J. G., Felton, K. C., Lapkin, A. A., and Mitsos, A. (2023b).  Gibbs–duhem-informed neural networks for binary activity coefficient prediction.  Digital Discovery, 2:1752–1767.
  * Rittig et al., (2022) Rittig, J. G., Gao, Q., Dahmen, M., Mitsos, A., and Schweidtmann, A. M. (2022).  Graph neural networks for the prediction of molecular structure-property relationships.
  * Rosen and Kunjappu, (2012) Rosen, M. and Kunjappu, J. (2012).  Surfactants and Interfacial Phenomena: Rosen/Surfactants 4E.  John Wiley & Sons, Ltd.
  * Ruiz, (2008) Ruiz, C. C. (2008).  Sugar-based surfactants: fundamentals and applications, volume 143\.  CRC Press.
  * Sanchez Medina et al., (2022) Sanchez Medina, E. I., Linke, S., Stoll, M., and Sundmacher, K. (2022).  Graph neural networks for the prediction of infinite dilution activity coefficients.  Digital Discovery, 1:216–225.
  * Sanchez Medina et al., (2023) Sanchez Medina, E. I., Linke, S., Stoll, M., and Sundmacher, K. (2023).  Gibbs–helmholtz graph neural network: capturing the temperature dependency of activity coefficients at infinite dilution.  Digital Discovery, 2:781–798.
  * Schick, (1987) Schick, M. J. (1987).  Nonionic Surfactants: Physical Chemistry.  Marcel Dekker, Inc., New York and Basel.
  * Schweidtmann et al., (2020) Schweidtmann, A. M., Rittig, J. G., König, A., Grohe, M., Mitsos, A., and Dahmen, M. (2020).  Graph neural networks for prediction of fuel ignition quality.  Energy & Fuels, 34(9):11395–11407.
  * Schweidtmann et al., (2023) Schweidtmann, A. M., Rittig, J. G., Weber, J. M., Grohe, M., Dahmen, M., Leonhard, K., and Mitsos, A. (2023).  Physical pooling functions in graph neural networks for molecular property prediction.  Computers & Chemical Engineering, 172:108202.
  * Seddon et al., (2022) Seddon, D., Müller, E. A., and Cabral, J. T. (2022).  Machine learning hybrid approach for the prediction of surface tension profiles of hydrocarbon surfactants in aqueous solution.  Journal of Colloid and Interface Science, 625:328–339.
  * Shaban et al., (2020) Shaban, S. M., Kang, J., and Kim, D.-H. (2020).  Surfactants: Recent advances and their applications.  Composites Communications, 22:100537.
  * Shinoda and Hirai, (1977) Shinoda, K. and Hirai, T. (1977).  Ionic surfactants applicable in the presence of multivalent cations. physicochemical properties.  The Journal of Physical Chemistry, 81:1842–1845.
  * Simonovsky and Komodakis, (2017) Simonovsky, M. and Komodakis, N. (2017).  Dynamic edge-conditioned filters in convolutional neural networks on graphs.
  * Su et al., (2020) Su, H., Wang, F., Ran, W., Zhang, W., Dai, W., Wang, H., Anderson, C. F., Wang, Z., Zheng, C., Zhang, P., Li, Y., and Cui, H. (2020).  The role of critical micellization concentration in efficacy and toxicity of supramolecular polymers.  Proceedings of the National Academy of Sciences of the United States of America, 117:4518–4526.
  * Szűts and Szabó-Révész, (2012) Szűts, A. and Szabó-Révész, P. (2012).  Sucrose esters as natural surfactants in drug delivery systems—a mini-review.  International Journal of Pharmaceutics, 433(1):1–9.
  * Tadros, (2005) Tadros, T. (2005).  Surfactants in Personal Care and Cosmetics, chapter 12, pages 399–432.  John Wiley & Sons, Ltd.
  * Thiruvengadam et al., (2020) Thiruvengadam, S., Murphy, M., Tan, J. S., and Miller, K. (2020).  A generalized theoretical model for the relationship between critical micelle concentrations, pressure, and temperature for surfactants.  Journal of Surfactants and Detergents, 23(2):273–303.
  * Thompson et al., (2023) Thompson, C. J., Ainger, N., Starck, P., Mykhaylyk, O. O., and Ryan, A. J. (2023).  Shampoo science: A review of the physiochemical processes behind the function of a shampoo.  Macromolecular Chemistry and Physics, 224(3):2200420.
  * Veličković et al., (2018) Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. (2018).  Graph attention networks.
  * Vieira et al., (2021) Vieira, I. M. M., Santos, B. L. P., Ruzene, D. S., and Silva, D. P. (2021).  An overview of current research and developments in biosurfactants.  Journal of Industrial and Engineering Chemistry, 100:1–18.
  * Vinyals et al., (2016) Vinyals, O., Bengio, S., and Kudlur, M. (2016).  Order matters: Sequence to sequence for sets.
  * Vu et al., (2023) Vu, T.-S., Ha, M.-Q., Nguyen, D.-N., Nguyen, V.-C., Abe, Y., Tran, T., Tran, H., Kino, H., Miyake, T., Tsuda, K., and Dam, H.-C. (2023).  Towards understanding structure-property relations in materials with interpretable deep learning.  npj Computational Materials, 9(1):215.
  * Wang et al., (2018) Wang, Y., Yan, F., Jia, Q., and Wang, Q. (2018).  Quantitative structure-property relationship for critical micelles concentration of sugar-based surfactants using norm indexes.  Journal of Molecular Liquids, 253:205–210.
  * Weininger, (1988) Weininger, D. (1988).  Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules.  J. Chem. Inf. Comput. Sci., 28(1):31–36.
  * Wu et al., (2021) Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Yu, P. S. (2021).  A comprehensive survey on graph neural networks.  IEEE Transactions on Neural Networks and Learning Systems, 32(1):4–24.
  * Xu et al., (2016) Xu, J., Cao, F., Li, T., Zhang, S., Gao, C., and Wu, Y. (2016).  Itaconic acid based surfactants: I. synthesis and characterization of sodium n-octyl sulfoitaconate diester anionic surfactant.  Journal of Surfactants and Detergents, 19(2):373–379.
  * Xu et al., (2019) Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2019).  How powerful are graph neural networks?
  * Yang et al., (2019) Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., Palmer, A., Settels, V., Jaakkola, T., Jensen, K., and Barzilay, R. (2019).  Analyzing learned molecular representations for property prediction.  Journal of Chemical Information and Modeling, 59(8):3370–3388.  PMID: 31361484.
  * Zhou et al., (2020) Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., and Sun, M. (2020).  Graph neural networks: A review of methods and applications.  AI Open, 1:57–81.

##  Appendix A Figures

Figure A9: Statistical overview of the CMC. The logarithm is applied to CMC in
$\mu$M.

(a)

(b)

(c)

(d)

Figure A10: Outliers of the ensembled GNN model in different temperature
split. The first three of them belong to anionic class and the last one is
cationic.

(a)

(b)

(c)

(d)

Figure A11: Outliers of the ensembled GNN model in different distinct
surfactant split. The first outlier belongs to zwitterionic class, the third
in nonionic class and the other two to the aninionic class.

##  Appendix B Tables

Table B5: Atom features used in the molecular graph representation. All features are implemented as one-hot-encoding. Feature |  Description | Dimension
---|---|---
atom type |  atom type (C, N, O, S, F, Cl, Br, Na, I, B, K, H, Li) | 13
is in a ring |  if the atom is part of a ring | 1
is aromatic |  if the atom is part of an aromatic system | 1
hybridization |  sp2, sp3 | 2
chirality |  unspecified, clockwise, counter clockwise | 3
charge |  formal charge of the atom (-1,0,1) | 3
# bonds |  number of bonds the atom is involved in | 5
# Hs |  number of bonded hydrogen atoms | 5
Total |  | 33
Table B6: Edge features used in the molecular graph representation. All features are implemented as one-hot-encoding. Feature | Description | Dimension
---|---|---
bond type | single, double, or atomatic | 3
is in a ring | is the bond part of a ring ? | 1
conjugated | is the bond conjugated ? | 1
stereo | none or E/Z | 3
Total |  | 8
Table B7: Hyperparameters of the GNN model investigated through a grid search. The hyperparameter dimensions refers to the size of the molecular fingerprint and the size of the MLP. Hyperparameter | Range | Optimized geometry
---|---|---
Graph convolutional layers | (1, 2) | 1
Graph convolutional type | (NNConv, GINEConv) | GINEConv
Usage of GRU | (True, False) | False
Initial learning rate | (0.005, 0.01, 0.05) | 0.005
Batch size | (16, 32, 64) | 32
Dimensions | (64, 128) | 128
Number of MLP layers | 3 | 3
Activation function | ReLU | ReLU
Maximum epochs | 300 | 300
Early stopping patience | 60 | 60
Learning rate decay | 0.8 | 0.8
Patiance | 3 | 3
Optimizer | Adam | Adam
Table B8: Comparison of predicted vs measured CMC values of the 4 outlier for
the different temperature split. The data source is provided.

| Different temperature test set
---|---
| Pred. CMC (mM) | Meas. CMC (mM) | Source | Temp. (∘C)
Outlier 1 | 0.42 | 2.1 |  (Mukerjee and Mysels,, 1971) | 25
Outlier 2 | 0.27 | 1.2 |  (Mukerjee and Mysels,, 1971) | 25
Outlier 3 | 0.82 | 0.2 |  (Myers,, 2020) | 25
Outlier 4 | 11.28 | 42.1 |  (Omar and Abdel-Khalek,, 1998) | 34.85

Table B9: Comparison of predicted vs measured CMC values of the 4 outlier for
the distinct surfactant test set. The data source is provided.

| Distinct surfactant test set
---|---
| Pred. CMC (mM) | Meas. CMC (mM) | Source | Temp. (∘C)
Outlier 1 | 0.15 | 1.5 |  (Rosen and Kunjappu,, 2012) | 25
Outlier 2 | 0.19 | 1.6 |  (Shinoda and Hirai,, 1977) | 25
Outlier 3 | 1.17 | 9.8 |  (Mattei et al.,, 2013) | 25
Outlier 4 | 2.24 | 0.4 |  (Xu et al.,, 2016) | 20

Table B10: Comparison for predicted vs measured CMC values (in mM) of the 3 selected sugar-based surfactants present in the distinct surfactant test set. $\alpha$ At 37∘C. Temp. (°C) | Decyl $\beta$-D-maltoside | Decyl $\beta$-D-glucoside | Octyl $\beta$-D-Thioglucoside
---|---|---|---
| Pred. | Meas. | Pred. | Meas. | Pred. | Meas.
10 |  |  | 1.82 | 2.53 |  |
15 | 2.32 | 2.41 |  |  |  |
20 |  |  | 1.84 | 2.09 | 21.4 | 9.3
25 | 2.29 | 2.2 |  |  | 21.45 | 8.5
30 |  |  |  |  | 21.73 | 7.5
35 | 2.3 | 2.05 | 1.94 $\alpha$ | 2.03$\alpha$ | 22.06 | 6.9
40 |  |  |  |  | 22.42 | 6.2
45 | 2.35 | 2.1 |  |  |  |
50 |  |  | 2.01 | 2.13 |  |
55 | 2.43 | 2.27 |  |  |  |
65 | 2.53 | 2.57 |  |  |  |