11institutetext:  The Hong Kong University of Science and Technology, Hong
Kong, China. 22institutetext: The Hong Kong University of Science and
Technology (Guangzhou), China.
22email<EMAIL_ADDRESS>
# Prompt-Guided Adaptive Model Transformation for Whole Slide Image
Classification

Yi Lin† 11 Zhengjie Zhu† 1122 Kwang-Ting Cheng 11 Hao Chen🖂 11

###### Abstract

Multiple instance learning (MIL) has emerged as a popular method for
classifying histopathology whole slide images (WSIs). Existing approaches
typically rely on frozen pre-trained models to extract instance features,
neglecting the substantial domain shift between pre-training natural and
histopathological images. To address this issue, we propose PAMT, a novel
Prompt-guided Adaptive Model Transformation framework that enhances MIL
classification performance by seamlessly adapting pre-trained models to the
specific characteristics of histopathology data. To capture the intricate
histopathology distribution, we introduce Representative Patch Sampling (RPS)
and Prototypical Visual Prompt (PVP) to reform the input data, building a
compact while informative representation. Furthermore, to narrow the domain
gap, we introduce Adaptive Model Transformation (AMT) that integrates adapter
blocks within the feature extraction pipeline, enabling the pre-trained models
to learn domain-specific features. We rigorously evaluate our approach on two
publicly available datasets, Camelyon16 and TCGA-NSCLC, showcasing substantial
improvements across various MIL models. Our findings affirm the potential of
PAMT to set a new benchmark in WSI classification, underscoring the value of a
targeted reprogramming approach.

###### Keywords:

Prompt learning; Multiple instance learning; Whole slide image; Parameter-
efficient fine-tuning.

${\dagger}$${\dagger}$footnotetext: Equal contribution; 🖂 corresponding
author.

##  1 Introduction

Whole slide image (WSI) is indispensable in modern histopathology, offering a
detailed view crucial for accurate clinical disease diagnosis and research [5,
16, 19]. The advent of deep learning techniques has significantly transformed
histopathology image analysis, promising improvements in both precision and
efficiency [6, 17]. Despite these advancements, deep-learning-based WSI
classification presents notable challenges due to the massive size of WSIs,
necessitating the division of WSIs into smaller patches for analysis [8]. This
patch-based approach, however, introduces labor-intensive and time-consuming
annotation processes, limiting the effectiveness of supervised learning
methods in this context [11, 13].

To circumvent these limitations, multiple instance learning (MIL) has become
the preferred strategy for WSI analysis. In MIL, a WSI is treated as a bag of
patches, and the entire bag is labeled based on the presence or absence of
disease indicators among its constituent patches. However, this method
requires effective downsampling and feature extraction strategies to manage
the voluminous data, with the quality of patch features directly impacting MIL
classification outcomes [9, 12, 22, 15]. Existing MIL-based methods mainly
leverage pre-trained feature extractors without fine-tuning due to the
unaffordable computational cost of training the entire model on the large-
scale WSI datasets [9, 12, 22, 15], ignoring the domain shift and task
discrepancies between the pre-training task (e.g., ImageNet) and the
downstream task (e.g., histopathology) [20]. To narrow the domain shift,
researchers [12] propose to utilize self-supervised techniques such as SimCLR
[4] to pre-train the feature extractor on pathology images. However, these
methods fail to utilize available WSI labels, offering limited performance
improvements. Recent advancements in parameter-efficient fine-tuning (PEFT),
such as visual prompts [23] and adapter tuning [24], have shown promise in
addressing domain shift and task discrepancies. However, the application of
PEFT techniques in the context of WSI classification remains largely
unexplored.

In this work, we propose PAMT, a Prompt-guided Adaptive Model Transformation
framework that enhances WSI classification by ingeniously reprogramming both
the input bags and intermediate features with the visual prompt and model
adapter. To the best of our knowledge, this is the first attempt to adopt PEFT
techniques in consideration of the unique and complex nature of pathological
images. For this purpose, we first introduce Representative Patch Sampling
(RPS) strategy to capture the most relevant and informative patches while
discarding redundant ones, which extensively reduces the computational cost
and enables efficient end-to-end training. Additionally, to encapsulate the
diverse spectrum of pathological data distributions, we further propose
Prototypical Visual Prompt (PVP) module, where visual prompts are added to the
sampled patches, enhancing the model’s ability to interpret the wide array of
tissue characteristics inherent to WSI. To facilitate model adaption, we
introduce Adaptive Model Transformation (AMT) technique to modify the pre-
trained feature extractor with scaleable adapters, allowing nuanced adaptation
of ImageNet-trained architectures to the requirements of pathological image
analysis. The pre-trained model is frozen while training visual prompts,
adapter blocks, and a lightweight MIL classifier, ensuring the intrinsic
knowledge of the model is maintained.

Our contributions are succinctly summarized as follows:

  * $\bullet$

We pioneer the application of parameter-efficient fine-tuning in the WSI
classification domain, demonstrating the effectiveness of the PAMT framework
in enhancing model adaptability and performance on pathological images.

  * $\bullet$

We propose Representative Patch Sampling (RPS) strategy, which efficiently
distills informative patches from WSIs, enabling focused training on focusing
on diagnostically significant data and enhancing the model’s learning
efficiency with an end-to-end training.

  * $\bullet$

We propose Prototypical Visual Prompt (PVP) module, which introduces visual
prompts assigned by cluster centers to effectively capture the complex
distribution of pathological features across varied histopathological images.

  * $\bullet$

We propose Adaptive Model Transformation (AMT) technique, which injects
scaleable adapters into pre-trained feature extractor, facilitating a nuanced
and efficient knowledge transformation from natural to pathological domain.

  * $\bullet$

Extensive experiments on public datasets, Camelyon16 and TCGA-NSCLC, validate
the efficacy of PAMT, highlighting consistent performance improvements across
a variety of MIL benchmarks.

##  2 Method

Figure 1: Overview of the proposed method.

Figure 1 presents the Prompt-guided Adaptive Model Transformation (PAMT)
approach for Whole Slide Image Classification, which advances Whole Slide
Image (WSI) classification through three methodical components: Representative
Patch Sampling (RPS), Prototypical Visual Prompt (PVP), and Adaptive Model
Transformation (AMT). Initially, an ImageNet pre-trained ResNet [7] extracts
features, followed by RPS to identify representative patches through a pre-
trained attention-based MIL. Subsequently, PVP assigns tailored visual prompts
to these patches, enriching the input data. The AMT phase integrates adapter
blocks into the feature extractor. Visual prompts and adapter blocks are fine-
tuned in conjunction with the MIL classifier to ensure cohesive model
adaptation. Details of each component are further elaborated in the following.

###  2.1 Representative Patch Sampling for End-to-End Training

The Representative Patch Sampling (RPS) technique addresses a critical
challenge in WSI classification: the impracticality of processing an
excessively high number of patches. Typically, conventional methods involve
aggregating all patches extracted from a WSI into a single bag for MIL.
However, a large portion of histological tissues may contain similar and non-
informative content, making them redundant. Acknowledging that diagnostically
significant patches often represent a minor fraction of the total, e.g., less
than 10% in Camelyon16 [3], RPS aims to identify and utilize these patches
more efficiently. Through an attention-based MIL framework, patches are scored
based on their relevance to the diagnostic outcome. We define this selection
as:

$\Phi(B_{i})=\\{x_{ij}|\alpha_{ij}\in\text{top-$K$}(\\{\alpha_{ij}\\}_{j=1}^{M_{i}})\\},$
(1)

where $\alpha_{ij}$ denotes the attention score for $j$-th patch $x_{ij}$
within bag $B_{i}$ containing $M_{i}$ patches. These scores are determined by
an attention-based MIL classifier:

${\alpha_{ij}}=\dfrac{{\exp\left\\{{{{\bm{w}}^{\rm{T}}}\left({\tanh\left({{{\bm{V}}_{1}}{f(x_{ij})}}\right)\odot{\rm{sigmoid}}\left({{{\bm{V}}_{2}}{f(x_{ij})}}\right)}\right)}\right\\}}}{{\sum\limits_{j=1}^{M_{i}}{\exp}\left\\{{{{\bm{w}}^{\rm{T}}}\left({\tanh\left({{{\bm{V}}_{1}}{f(x_{ij})}}\right)\odot{\rm{sigmoid}}\left({{{\bm{V}}_{2}}{f(x_{ij})}}\right)}\right)}\right\\}}},$
(2)

where $\bm{w}$, $\bm{V}_{1}$, and $\bm{V}_{2}$ represent the learnable
parameters within the MIL classifier, and the element-wise multiplication is
denoted by $\odot$.

By concentrating on the top-$K$ scoring patches, the RPS approach
substantially reduces computational load and enables a more focused and
effective model training pathway. This method enhances the model’s capacity to
discriminate based on critical diagnostic features, resulting in an enriched
set of patches that are truly representative of the pathological state of the
WSI.

###  2.2 Prototypical Visual Prompt for Complex Data Distribution

Unlike natural images, WSIs exhibit significant intra- and inter-slide
heterogeneity (e.g., variations in tissue characteristics and staining
techniques), which poses distinct challenges for WSI classification. To
navigate this complexity, we introduce Prototypical Visual Prompt (PVP), which
aims to enhance the representation of features aligned with the diverse
pathological data distributions. Unlike the generic single visual prompt, PVP
provides a set of tailored prompts to capture the unique diagnostic features
of each patch.

Following the Representative Patch Sampling (RPS) process, a clustering
algorithm (e.g., $k$-means) groups the extracted features $\\{f(x_{ij})\mid
x_{ij}\in\Phi(B_{i}),\forall i\\}$ across all WSI bags, capturing the
overarching pathological themes. This process identifies prototypical patterns
within the data, resulting in cluster centroids $\\{\mu_{c}\\}_{c=1}^{C}$,
which guide the assignment of visual prompts $\\{p_{c}\\}_{c=1}^{C}$ to
corresponding patches. Finally, the prompted patches are aggregated to form
the prompted bag $\Psi(B_{i})$. This is mathematically represented as:

$x_{ij}^{\prime}=x_{ij}+p_{\epsilon_{ij}},$ (3)
$\Psi(B_{i})=\\{{x^{\prime}_{{ij}}\mid x_{ij}\in\Phi(B_{i})}\\},$ (4)

with the assignment $\epsilon_{ij}$ determined by the proximity to these
centroids:

$\epsilon_{ij}=\underset{c}{\text{argmin}}\,\|f(x_{ij})-\mu_{c}\|_{2}.$ (5)

PVP are constructed as trainable padding prompts shown in Fig. 1, ensuring
that each prompt is adaptable and specific to the cluster it represents. In
this work, we set the number of clusters ($C$) and the padding size ($S$) to 4
and 10, respectively.{\dagger}{\dagger}${\dagger}$The ablation study of $C$
and $S$ can be found in the supplementary material. By infusing patches with
PVP tailored to their cluster-specific features, the assembled bag enriches
the model’s ability to discern the vast array of tissue characteristics
inherent in WSIs, enhancing classification performance.

###  2.3 Adaptive Model Transformation with Transfer Learning

The transition from pre-trained natural image analysis to specialized
pathological image classification presents a unique challenge. There is a need
to retain the robust feature extraction capabilities honed on diverse datasets
while ensuring the model is efficiently adapted to the specificities of
pathological images. Considering the high volume and size of patches in WSIs
yet the relatively lower number of slides compared to natural images, it is
crucial to maintain the pre-training benefits while enhancing task-specific
adaptability. To this end, we introduce Adaptive Model Transformation (AMT), a
strategic approach for adaptively transforming the pre-trained model to the
pathology domain.

AMT integrates adapter blocks positioned parallel to its foundational blocks.
Here, AMT integrates three adapter blocks into the last layer of a pruned
ResNet-50 [15]. The number and position of adapter blocks can be adaptively
adjusted based on the specific task and
dataset.{\dagger}{\dagger}${\dagger}$The ablation study of the position and
quantity of adapter blocks can be found in the supplementary material. These
adapters comprise a sequence of a global average pooling (GAP), followed by a
two-layer multi-layer perceptron (MLP) and a sigmoid activation function, as
depicted in Fig. 1. For an intermediate feature map $\bm{f}_{i-1}$ as the
input for the $i$-th ResNet block, an adapter block generates a domain-
adaptive vector $\bm{b}_{i}$. This vector is then channel-wise multiplied with
subsequent feature maps $g_{i}(\bm{f}_{i-1})$, facilitating a tailored feature
adaptation, represented as:

$\bm{b}_{i}={\rm{sigmoid}}(\bm{W}_{i2}{\rm{ReLU}}(\bm{W}_{i1}{\rm{GAP}}(\bm{f}_{i-1}))),$
(6) $\bm{f}_{i}=g_{i}(\bm{f}_{i-1})\odot\bm{b}_{i},$ (7)

where $\text{ReLU}(\cdot)$ denotes rectified linear unit, $\bm{W}_{i1}$ and
$\bm{W}_{i2}$ are the learnable parameters to be fine-tuned, and the
parameters of $g_{i}(\cdot)$ remain frozen during training. During the
training process, only the parameters of visual prompts, adapter blocks, and
the lightweight MIL classifier are updated in an end-to-end manner, while the
original pre-trained feature extractor is frozen to retain the vast knowledge
encoded in the pre-trained model.

##  3 Experiments

###  3.1 Datasets

We evaluate our method on two public datasets: Camelyon16 [3] and TCGA-NSCLC
[1]. Camelyon16 consists of 399 H&E WSI from breast cancer screening, with two
classes: normal and tumor. We use the official 129 test set and split the
official 270 training set into training and validation sets with a ratio of
$9\colon 1$. TCGA-NSCLC contains two subtypes of lung cancer: Lung
Adenocarcinoma (LUAD) and Lung Squamous Cell Carcinoma (LUSC). We split the
dataset into training, validation, and testing sets at the slide level, with a
ratio of 65:10:25.

###  3.2 Implementation Details

The performance is measured by Accuracy (Acc), F1 score (F1), and Area Under
the Curve (AUC). In the RPS process, we adopt the DTFD model [22] as the
attention-based MIL classifier. The Adam optimizer [10] was employed for the
adapter blocks and classifier with a learning rate and weight decay of 1e-4
over 100 epochs. Following [2], visual prompts were optimized using SGD [14]
at a starting learning rate of 40 and decay according to a cosine schedule.
These computations were facilitated by an NVIDIA GeForce RTX 3090 GPU.

Table 1: Results of comparative experiments on Camelyon16 and TCGA-NSCLC datasets. In the table, the best results are in bold. The mean and standard deviation is reported (mean${}_{\text{std}}$) under 3 times’ experiments. Method | Camelyon16 | TCGA-NSCLC
---|---|---
AUC | F1 | Acc | AUC | F1 | Acc
Baseline | Mean-pooling | 0.5900.018 | 0.4540.007 | 0.6770.011 | 0.9000.013 | 0.8340.011 | 0.8300.010
Max-pooling | 0.8540.028 | 0.7540.119 | 0.8260.037 | 0.9060.021 | 0.8430.025 | 0.8320.027
AB-MIL [9] | 0.8740.023 | 0.7980.013 | 0.8610.009 | 0.9280.018 | 0.8780.014 | 0.8740.016
CLAM-SB [15] | 0.9020.012 | 0.8270.021 | 0.8710.019 | 0.9250.023 | 0.8700.026 | 0.8740.022
DTFD-MIL [22] | 0.9100.011 | 0.8350.029 | 0.8840.027 | 0.9300.019 | 0.8870.017 | 0.8840.021
PAMT | Mean-pooling | 0.8100.010 | 0.7140.013 | 0.8030.017 | 0.9250.011 | 0.8740.014 | 0.8710.010
Max-pooling | 0.8900.015 | 0.7770.016 | 0.8240.025 | 0.9260.013 | 0.8730.011 | 0.8700.013
AB-MIL [9] | 0.9210.011 | 0.8280.018 | 0.8660.008 | 0.9310.009 | 0.8840.007 | 0.8790.011
CLAM-SB [15] | 0.9240.009 | 0.8320.007 | 0.8810.006 | 0.9330.011 | 0.8910.006 | 0.8880.005
DTFD-MIL [22] | 0.9290.010 | 0.8590.007 | 0.8970.009 | 0.9420.008 | 0.8950.013 | 0.8930.014

###  3.3 Comparison with State-of-the-Art Methods

We compare our method with state-of-the-art methods on the Camelyon16 and
TCGA-NSCLC datasets, including Mean-pooling, Max-pooling, AB-MIL [9], CLAM-SB
[15], and DTFD-MIL [22]. Table 1 indicates a consistent enhancement in
performance metrics when applying our PAMT approach compared to the baseline
methods. Notably, on the Camelyon16 dataset, our method significantly improved
the performance of the baseline methods by 0.220, 0.036, 0.047, 0.022, and
0.019 in AUC, respectively. It is remarkable that our method significantly
augmented the performance of both Mean and Max pooling strategies across both
datasets, underscoring the enhanced quality of feature bags that enable these
inherently less powerful MIL classifiers to achieve higher classification
outcomes. The same trend is observed in the TCGA-NSCLC dataset.

Table 2: Ablation study of different tuning strategies using DTFD on the Camelyon16 dataset. ‘+’ denotes additional trained parameters beyond the common MIL classifier. ‘Partial’ trains the extractor’s last layer and MIL classifier. Averaged over three trials. Method | Baseline | ✓ | ✓ | ✓ | ✓ | ✓ |  Fully -tuning |  Partial [18] |  BitFit [21]
---|---|---|---|---|---|---|---|---|---
RPS |  | ✓ | ✓ | ✓ | ✓
PVP |  |  | ✓ |  | ✓
AMT |  |  |  | ✓ | ✓
AUC |  | 0.910 | 0.916 | 0.922 | 0.924 | 0.929 | 0.891 | 0.911 | 0.912
Params(M) | 0.789 | +0 | +0.118 | +0.393 | +0.511 | +8.543 | +7.098 | +0.015

###  3.4 Ablation Study

####  3.4.1 Effectiveness of each component.

We conduct an ablation study to evaluate the effectiveness of each component
in our method, including Representative Patch Sampling (RPS), Prototypical
Visual Prompt (PVP), and Adaptive Model Transformation (AMT). As shown in
Table 2, the baseline method DTFD achieves an AUC of 0.91 on the Camelyon16
dataset. After applying the RPS, the AUC increases to 0.914, indicating the
effectiveness of representative patches with redundant information removal.
Afterwards, the PVP and AMT further improve the AUC to 0.922 and 0.924,
respectively. Integrating three techniques, the AUC reaches 0.929,
demonstrating the combined advantages of different components in our method.

####  3.4.2 Comparison with other tuning strategies.

In Table 2, we compare our method with other tuning strategies, including
Fully-tuning, Partial-tuning [18], and BitFit [21]. All these strategies
indicate an absence of significant improvement in model performance.
Overfitting emerged as a concern with both the Fully and Partial tuning
methods, due to the limited dataset size hindering generalization to novel
data. BitFit may suffer from a lack of adaptability to the dataset’s
intricacies. Conversely, our method exhibits a significant performance uplift
without incurring significant parameter overhead.

Figure 2: Different training strategies with varying proportions of training
WSIs using the DTFD model on TCGA-NSCLC dataset.

Figure 3: Visualization of cluster centers for bag reprogramming. Four
distinct clusters are showcased, representing varied histopathological
features.

####  3.4.3 Analysis under limited data.

To further evaluate the robustness of our method, we explore more challenging
scenarios with varying proportions of the training set within the TCGA-NSCLC
dataset. As depicted in Fig. 3, our method yields consistent superiority over
the LP and Fully fine-tuning methods. Notably, employing merely 75% of the
training data, our approach surpasses the baseline performance with the full
dataset, underscoring the efficacy of our method in extracting and leveraging
meaningful patterns from limited data and the potential for application in
scenarios where data acquisition is challenging.

####  3.4.4 Visualization of cluster centers for PVP.

In Fig. 3, we visualize the cluster centers obtained from the Prototypical
Visual Prompt, which discerned four salient histopathological patterns matched
with distinct padding prompts. Results demonstrate a spectrum of tissue
characteristics, from sparse inflammatory infiltrates to areas of dense
nuclear aggregation, which are likely to carry diagnostic relevance. This
ablation study confirms the clustering process’s capacity to isolate and
highlight significant histopathological features within the dataset.

##  4 Conclusion

In this study, we introduce PAMT, a Prompt-guided Adaptive Model
Transformation approach for Whole Slide Image Classification. PAMT seamlessly
incorporates three key components, namely Representative Patch Sampling (RPS),
Prototypical Visual Prompt (PVP), and Adaptive Model Transformation (AMT),
with the objective of revamping both the input data and the model. PAMT
leverages visual prompts and adapter blocks while preserving the pre-trained
backbone’s intrinsic knowledge, ensuring tailored adaption for pathology
images. Comprehensive validation on two public datasets, Camelyon16 and TCGA-
NSCLC, employing five distinct MIL classifiers, underscores that PAMT can
considerably improve WSI classification, showing its effectiveness and
efficiency. The future work will focus on the development of a more advanced
visual prompt and adapter block in consideration of scalability, flexibility,
and robustness.

## References

  * [1] The cancer genome atlas (tcga) dataset, https://www.cancer.gov/tcga
  * [2] Bahng, H., Jahanian, A., Sankaranarayanan, S., Isola, P.: Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274 (2022)
  * [3] Bejnordi, B.E., Veta, M., Van Diest, P.J., Van Ginneken, B., Karssemeijer, N., Litjens, G., Van Der Laak, J.A., Hermsen, M., Manson, Q.F., Balkenhol, M., et al.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama 318(22), 2199–2210 (2017)
  * [4] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning. pp. 1597–1607. PMLR (2020)
  * [5] Cornish, T.C., Swapp, R.E., Kaplan, K.J.: Whole-slide imaging: routine pathologic diagnosis. Advances in Anatomic Pathology 19(3), 152–159 (2012)
  * [6] Hashimoto, N., Fukushima, D., Koga, R., Takagi, Y., Ko, K., Kohno, K., Nakaguro, M., Nakamura, S., Hontani, H., Takeuchi, I.: Multi-scale domain-adversarial multiple-instance cnn for cancer subtype classification with unannotated histopathological images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3852–3861 (2020)
  * [7] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [8] Hou, L., Samaras, D., Kurc, T.M., Gao, Y., Davis, J.E., Saltz, J.H.: Patch-based convolutional neural network for whole slide tissue image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2424–2433 (2016)
  * [9] Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International Conference on Machine Learning. pp. 2127–2136. PMLR (2018)
  * [10] Kingma, D.P., Ba, J.L.: Adam: Amethod for stochastic optimization. In: International Conference on Learning Representations. pp. 1–15 (2014)
  * [11] Lerousseau, M., Vakalopoulou, M., Classe, M., Adam, J., Battistella, E., Carré, A., Estienne, T., Henry, T., Deutsch, E., Paragios, N.: Weakly supervised multiple instance learning histopathological tumor segmentation. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part V 23. pp. 470–479. Springer (2020)
  * [12] Li, B., Li, Y., Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14318–14328 (2021)
  * [13] Li, S., Liu, Y., Sui, X., Chen, C., Tjio, G., Ting, D.S.W., Goh, R.S.M.: Multi-instance multi-scale cnn for medical image classification. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part IV 22. pp. 531–539. Springer (2019)
  * [14] Loshchilov, I., Hutter, F.: SGDR: Stochastic gradient descent with warm restarts. In: International Conference on Learning Representations (2016)
  * [15] Lu, M.Y., Williamson, D.F., Chen, T.Y., Chen, R.J., Barbieri, M., Mahmood, F.: Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering 5(6), 555–570 (2021)
  * [16] Pantanowitz, L., Valenstein, P.N., Evans, A.J., Kaplan, K.J., Pfeifer, J.D., Wilbur, D.C., Collins, L.C., Colgan, T.J.: Review of the current state of whole slide imaging in pathology. Journal of Pathology Informatics 2(1), 36 (2011)
  * [17] Pinckaers, H., Van Ginneken, B., Litjens, G.: Streaming convolutional neural networks for end-to-end learning with multi-megapixel images. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(3), 1581–1590 (2020)
  * [18] Shen, Z., Liu, Z., Qin, J., Savvides, M., Cheng, K.T.: Partial is better than all: revisiting fine-tuning strategy for few-shot learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 9594–9602 (2021)
  * [19] Srinidhi, C.L., Ciga, O., Martel, A.L.: Deep neural network models for computational histopathology: A survey. Medical Image Analysis 67, 101813 (2021)
  * [20] Stacke, K., Eilertsen, G., Unger, J., Lundström, C.: Measuring domain shift for deep learning in histopathology. IEEE Journal of Biomedical and Health Informatics 25(2), 325–336 (2020)
  * [21] Zaken, E.B., Goldberg, Y., Ravfogel, S.: Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. pp. 1–9 (2022)
  * [22] Zhang, H., Meng, Y., Zhao, Y., Qiao, Y., Yang, X., Coupland, S.E., Zheng, Y.: DTFD-MIL: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18802–18812 (2022)
  * [23] Zhang, J., Kapse, S., Ma, K., Prasanna, P., Saltz, J., Vakalopoulou, M., Samaras, D.: Prompt-MIL: Boosting multi-instance learning schemes via task-specific prompt tuning. arXiv preprint arXiv:2303.12214 (2023)
  * [24] Zhu, Y., Shen, Z., Zhao, Z., Wang, S., Wang, X., Zhao, X., Shen, D., Wang, Q.: Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis. arXiv preprint arXiv:2311.08236 (2023)


# Light-responsive nematic colloids and colloidal crystals

###### Abstract

Rational control over the periodic arrangement of particles by means of
external stimuli is a technologically important aspect of colloidal science
with important physical underpinnings. Here, a robust structural control of
particle assemblies in a nematic liquid crystal (NLC) is demonstrated by
dissolving trace amounts of light-responsive azo-dendrimer molecules which
spontaneously get adsorbed on the particle surface. The azo-dendrimer
molecules in the presence of external UV irradiation undergo conformational
change (trans-cis); as a result, they transmit the mechanical torque to
surrounding LC molecules and alter the near-field director orientation. The
director re-orientation at the surface of the particles causes topological
defect transformation which involves elastic dipoles, quadrupoles and
hexadecapoles. The defect transformation can be emulated in colloidal
assemblies towards different purposes such as rotation of chains and
restructuring of 2D colloidal crystals. In this study, various topological
aspects of light-activated defect transformation and its application in the
collective manipulation of colloidal assemblies are presented.

###### keywords:

Liquid crystals, Colloids, Self-assembly, Photo-responsive materials,
Topological defects, Defect transformations

Devika Venkuzhy Sudhakaran, Dinesh Kumar Sahu, Osamu Haba, Surajit Dhara*

Devika Venkuzhy Sudhakaran, Dr. Dinesh Kumar Sahu, Prof. Surajit Dhara
School of Physics, University of Hyderabad, Hyderabad - 500046, India
Email Address<EMAIL_ADDRESS>
Prof. Osamu Haba
Graduate School of Organic Materials Science, Yamagata University, Japan

##  1 Introduction

Stimuli-responsive structures formed by the self-assembly of micro- and nano-
sized particles are indispensable tools for developing smart functional
materials for emerging applications in the field of optics, photonics,
microsensing, bio-interfaces etc [1, 3, 2, 4, 5, 6, 7, 8, 9, 10, 11]. A myriad
of such geometrically diverse and dynamic structures could be conveniently
assembled in an anisotropic liquid crystalline host matrix, owing to its long-
range elasticity-mediated interactive forces. They have the ability to respond
to external stimuli such as light, temperature, chemical substances and other
external fields. For example, nematic liquid crystals (NLCs) exhibit
directional elasticity and anisotropic electro-optic properties as a result of
average orientation of the rod-shaped molecules along a preferred direction
$(\hat{n})$, known as the director [12, 13, 14]. This gives rise to strong and
long-range structural forces between particles suspended (nematic colloids) in
NLCs. Dispersion of foreign particles in an NLC creates orientational
frustration in the director field, which results in singularities or
topological defects at the LC-particle interface. The simplest case is that of
spherical inclusions in NLCs, where the particles, depending on the surface
anchoring conditions generally nucleate a point (hedgehog defect), ring defect
(so-called Saturn ring) in the bulk, or a pair of antipodal surface defects
(boojums) [15, 16, 14, 17, 18, 19]. The first two kinds of defects (point or
ring defect in the bulk) appear while imposing homeotropic anchoring of LC on
the particle’s surface whereas the latter (boojum defects) is a consequence of
planar anchoring of LC molecules at the particle surface. In the case of a
hedgehog defect, the director symmetry and ensuing elasticity-mediated
interactions are reminiscent of an electric dipole whereas, for a Saturn ring
or Boojum defects, it resembles that of an electric quadrupole [1, 14, 20, 21,
22, 23]. Hence, particles with hedgehog defects are also known as elastic
dipoles; similarly, those with Saturn ring or boojum defects are called
elastic quadrupoles.

Figure 1: Representation of elastic multipoles in terms of atomic orbitals.
$s-$, $p-$, $d-$, and $f-$, atomic orbitals are drawn using spherical
harmonics wave function; analogous elastic multipoles around a spherical
particle with a colour-coded variation of $n_{y}$ are given below.

Conferring desired anchoring of the nematic director at the LC-particle
interface usually requires chemical functionalization of the surface such as
pre-treatment with surfactants [24, 25]. The high surface sensitivity of LCs
(due to low interfacial energies) combined with emerging synthetic routes for
developing surfaces with well-defined properties offers an attractive platform
for the design of responsive soft materials. Here, we use an optically-
triggered azo-dendrimer as an active interface to study the evolution of
defect structures around microspheres and employ it in achieving structural
transformation of 2D crystals and rotation of chains. The ability of the
dendrimer to get adsorbed spontaneously on an ambient surface is useful in
creating photo-sensitive surfaces of particles in the NLC [26, 27, 28, 29, 30,
31]. It has been demonstrated that the azo-moieties, in response to highly
localized UV light of low intensities, undergo reversible structural (cis-
trans) isomerism, which drives collective re-orientation of LC molecules at
the interface [31]. Under normal conditions, the azo-moieties adhere to the
thermodynamically more stable trans-state, where LC molecules are held
perpendicular to the surface. Upon illuminating with UV, they revert to the
cis-state in which the LC molecules orient in a planar fashion. We show that
the spatiotemporal director change forces a series of out-of-equilibrium
defect structures around the microspheres during the process. Such previously
unanticipated defect states broaden the landscape of existing elastic
interactions and ensuing self-assembled structures. As a direct application,
we demonstrate the structural transformation of a 2D crystal of elastic
dipoles when irradiated with UV light and measure substantial changes in the
lattice parameters. The paper is organised as follows: First, we provide a
brief introduction to the elastic multipoles and then present the defect
dynamics of an isolated particle, followed by the rotation of a linear chain
and restructuring of a 2D colloidal crystal.

###  1.1 Results and Discussion

####  1.1.1 Colloidal dispersions as elastic multipoles

Colloidal particles in a uniform nematic LC create near-field elastic
distortion. The far-field director $n_{0}$ (at length scales of several
particle dimensions away from the surface) in the case of spherical particles
has fewer distortions, hence the Euler-Lagrange equation can be linearized in
this regime. Using the one-elastic-constant approximation (where different
elastic constants yield similar values), the nematic LC free energy density
takes the form [12, 32, 33]:

$f_{E}=\frac{1}{2}K\sum_{j=y,z}(\nabla n_{j})^{2}$ (1)

where, $K$ is the average Frank elastic constant, and $n\approx
n(1,n_{y},n_{z})$ is the unit nematic director along the x-axis, and having
perpendicular components $n_{j}(j=y,z\ll 1)$. Minimization of the free energy
density of (Equation 1) gives rise to the Laplace equation:

$\nabla^{2}n_{j}=0,$ (2)

the solution of which gives the multipole-expansion of the nematic director
field as:

$n_{j}(r,\theta,\phi)=\sum_{l=0}^{\infty}\sum_{m=-l}^{m=l}q_{lm}^{j}\frac{r_{0}^{l+1}}{r^{l+1}}Y_{l}^{m}(\theta,\phi)$
(3)

where $\theta,\phi$ represents the polar and azimuthal angles, respectively,
$Y_{l}^{m}(\theta,\phi)$ are spherical harmonics, l determines the order of
multipole ($2^{l}$-pole), $-l\leq m\leq l$, $q_{lm}^{j}$ are dimensionless co-
efficients of elastic multipoles, and $r_{0}$ is the radius of the particle.

Figure 2: (a) Azo-dendrimer coated silica microspheres in a planar cell
inducing hedgehog point or Saturn ring defects due to the $trans-$
conformation of the photoactive dendrimer favouring spontaneous homeotropic
surface anchoring of LC molecules. The dendrimer-guided alignment of LC
molecules is schematically shown in the zoomed view. (b) With UV illumination,
the surface anchoring becomes planar as the dendrimer undergoes a
conformational change to $cis-$ isomer state which results in boojum or
hexadecapolar defects. Double-headed arrows represent the far-field nematic
director $n_{0}$.

In this form, it is imperative to draw an analogy of the nematic director
field to the corresponding expansion of the electric field due to an
electrostatic charge distribution [12, 13, 14]. Thus, the nematic director
$n_{j}(r,\theta,\phi)$ is expressed as a summation over elastic multipoles
with each term in the expression (with co-efficient $q_{lm}^{j}$) representing
a higher order multipole than the previous one (Figure 1). Here, it is to be
noted that we have used spherical particles invariant to rotations about
$\textit{x}-$ axis and have no azimuthal contribution to $\textit{n}(r)$.
Therefore, the multipolar distortion induced by the particles is symmetric
with respect to rotations about $n_{0}$. Consequently, the coefficient of the
first term in Equation 3 (corresponding to elastic monopole) vanishes and the
remaining higher order terms with $l>0$ and $m=\pm 1$ define the director
distortions. Though higher-order terms are always present, the director
symmetry of a particle-defect combination is always determined by the order of
dominating multipole in the expression.

Most commonly observed are the elastic dipoles ($l=1,m=\pm 1$) and elastic
quadrupoles ($l=2,m=\pm 1$) where the surface of the colloids induce either
homeotropic or planar anchoring at the LC interface. Introducing particles of
complex design may alter the structure of the defect (such as a distorted
Saturn ring pinning to the sharp edges of a cube), but maintains the symmetry
of distortion [4]. Obtaining higher-order elastic multipoles like octupoles
($l=3$), hexadecapoles ($l=4$) etc has been a matter of challenge and
significant interest [18]. Recently, Senyuk et al., demonstrated that the
continuous absorption or desorption of surfactants can drive the continuous
transformation of elastic multipoles [33]. What we present here, is the light-
assisted manipulation of elastic multipoles induced by spherical silica
particles in a uniform nematic. Our experiments reveal the dynamics of the
photo-reversible transformation of defects with varying symmetries and its
effect on colloidal assemblies.

####  1.1.2 Light-activated defect dynamics around a spherical colloid

The light-activated defect dynamics occurring at the LC-particle interface
were studied using optically transparent silica microspheres (diameter 9.2
$\mu$m) dispersed in a nematic liquid crystal (5CB) mixed with $\approx 0.1$
wt% azo-dendrimer (for details see experimental section). In the NLC, the
dendrimer molecules get spontaneously adsorbed on the silica surface and act
like a ‘command surface’ which actively controls the LC orientation in
response to external optical stimuli [26, 27]. In the ground state, the
dendrimer mostly prefers thermodynamically more stable $trans-$ conformation
which prompts homeotropic (vertical) alignment of LC molecules. Hence, the
microspheres induce either hedgehog point defects with dipolar symmetry
$(l=1)$ or Saturn ring disclinations with quadrupolar symmetry $(l=2)$ (Figure
2a). Here, the surface nematic director $n_{s}$ makes an angle $\alpha\approx
0$ (i.e., parallel) to the surface normal, $\hat{s}$ (Figure 3b). When
irradiated with UV light (wavelength $\approx$ 330 nm), the dendrimers undergo
photo-isomerisation to cis-state which in turn rotates $n_{s}$ by an angle of
$90^{0}$, thereby rendering planar alignment ($\alpha\approx 90$) of the LC
(Figure 2b). However, the change of surface anchoring from homeotropic to
planar disturbs the existing balance between interfacial and bulk elastic free
energies, thereby driving out-of-equilibrium defect transformations across the
system to minimise the overall LC free energy.

####  1.1.3 Transformation from dipole to boojum

Figure 3: (a) Polarising optical microscopy (POM) and corresponding $\lambda$
plate (inserted at $45^{0}$ between sample and analyser) images of (i-ii)
hedgehog dipole, (iii-iv) Saturn ring quadrupole, (v-vi) elastic hexadecapole,
and (vii-viii) boojum quadrupole. Particle size - 9.2 $\mu$m. POM images of
5.24 $\mu$m particles are shown in the inset. (b) Schematics of surface
director tilt for different values of the polar anchoring angle, $\alpha$.
Transformations of elastic multipoles from (c) a hedgehog dipole to a boojum
quadrupole (Movie-1), and (d) a Saturn ring quadrupole to a boojum quadrupole
(Movie-3) under UV irradiation. Corresponding schematics of the director
profile around the particle are shown below. (e) Reverse transformation to a
Saturn ring quadrupole when UV is switched off (Movie-2 and 4). P, A
represents crossed polariser and analyser, the far-field director is given by
n${}_{0}.$

The light-driven spatiotemporal evolution of defect structure around a
microsphere with initial hedgehog point defect in the bulk is shown in Figure
3c. As can be seen, the point defect in the bulk evolves into a series of
complex defect structures before stabilising the final quadrupolar state (a
pair of antipodal boojums). It is evident that these complex structures appear
due to the increase in the tilt angle ($\alpha$) of the director with time.
Such transformation of defects from elastic dipole to higher order multipoles
has been recently proposed by Zhou et al. [34] in nematic colloids with conic
degenerate anchoring, where $\alpha$ varies from 0 to $90^{\circ}$ (Figure
3b). In our experiment, the different stages of evolution of a dipolar colloid
when excited by UV irradiation, captured at different time intervals, reveal
similar director distortions (Figure 3c, Movie-1[35]) as predicted. Following
it, we present a qualitative discussion of the defect transformation.

Initially, we consider an elastic dipole (Figure 3c [$t=0$ s]), i.e., a
particle with an accompanying hedgehog point defect of charge -1 ($l=1,m=1$).
Upon illumination with UV, the tilt angle $\alpha$ increases and we observe
what is called a ‘CA dipole’: a texture with a hedgehog point defect at one
pole of the particle and a boojum defect at the opposite pole (Figure 3c
[$t=4.16$ s] and schematic). It may look similar to the commonly observed
homeotropic elastic dipole, but a closer observation reveals the presence of a
dark spot (the boojum) at the opposite pole, surrounded by two weak bright
lobes. The theoretical model suggests that at $\alpha=45^{\circ}$, the 2D
topological charge -1 splits unequally between the upper ($-3/4$) and lower
($-1/4$) surface defects (see schematic), in compliance with the conservation
of topological charges [12, 13, 14]. With further increase in the value of
tilt angle, a surface ring nucleates at the upper pole (Figure 3c [$t=5.54$
s]) and starts migrating downwards (Figure 3c [$t=7.26$ s]). Both these
structures composed of a pair of boojum and Saturn ring quadrupoles are
characteristic of an elastic hexadecapole ($l=4,m=1$), reported recently [34].
The POM micrograph of a hexadecapole shows eight bright lobes: two each at the
poles and the equator, separated by eight dark regions (see inset of Figure 3a
(v)). The simulation predicts that the hexadecapolar structure can be
stabilised over a wide range of tilt angles ($40^{\circ}\leq\alpha\leq
60^{\circ}$), and the corresponding distortions could be distinguished by the
relative brightness of their lobes [34]. As shown in Figure 3c [$t=7.26$ s],
just before its transition to antipodal boojum, we observe a distortion
pattern similar to $\alpha\approx 60^{\circ}$, where the net topological
charge -1 is equally split (-1/3) among the upper and lower surface point
defects, surface ring defect at the equator. Finally, when $\alpha$ increases
to $90^{\circ}$, a pair of antipodal boojum ($l=2,m=1$) of strength $-1/2$
each is formed (Figure 3c [$t=18.84$s]).

####  1.1.4 Transformation from Saturn ring to boojum

Other than hedgehog point defects, particles with homeotropic anchoring also
form Saturn ring defects (topological charge = -1) with quadrupolar symmetry
($l=2,m=-1)$ (Figure 3a(iii,iv)). The director at the surface of these
particles is initially oriented hometropically and gradually undergoes tilting
(conic anchoring) when the UV light is switched on (Figure 3d, Movie-3 [35]).
With increasing tilt angle, the Saturn ring vanishes and a surface ring defect
(charge = -1/3) along with a pair of surface point defects (charge = -1/3,
each) emerge at the poles (Figure 3d [$t=3.45$ s]). That means, the elastic
quadrupole evolves to a higher-order hexadecapole ($l=4,m=1$)(Figure
3a(v,vi)). Further, with an increasing value of $\alpha$, the surface ring
defects diminish and ultimately transform into a pair of antipodal boojum
defects (charge = -1/2, each) prevailing tangential anchoring.

Whether we start with a point defect (dipolar configuration) or Saturn ring,
the end result is always antipodal boojums upon UV irradiation. When the UV
light is turned off, the boojums, in both cases, retrace the path to a bulk
Saturn ring via an intermediate elastic hexadecapole as shown in Figure 3e
(Movie-2 and 4 [35]). It is to be noted that the intensity of light
significantly affects the final state attained. For example, we often observed
that low light intensities ($<45\mu$W/mm2) caused the defect evolution in an
elastic dipole to stop at any of the intermediate states, where
$0\leq\alpha\leq 90$. Thus, UV light of controllable brightness offers a
reliable route for stabilising transient topological defects which otherwise,
are rarely observed.

Figure 4: (a) Snapshots depicting the rotation of a sphere (in-plane or out-
of-plane) by almost $90^{0}$ after the UV light is switched on (Movie-5). (b)
Comparison of the response time (in seconds) versus UV intensity ($\mu$W/mm2)
for elastic dipoles and quadrupoles. The solid curves are guidelines for the
eye. (c-e) Schematics of the director field around a sphere for different
values of $\alpha$. Green curved arrows represent the sense of rotation of the
particle. Figure 5:  (a) Polarising microscopy (POM) images with (second row)
and without (first row) a $\lambda-$ plate during the transformation of a pair
of self-assembled hedgehog dipoles to boojum quadrupoles in the presence of
UV. Schematics of the surrounding director field are shown in the inset. (b)
POM images with (second row) and without (first row) a $\lambda-$ plate
(director field schematic in the inset) during the transformation to Saturn
ring quadrupoles when UV is switched off. (c) Colour maps of director
distortion at the surface of the particles in the final equilibrium state
resembling an elastic multipole with $l=4$. The colour bar on the right
represents the normalised $y-$ component of the nematic director. (d)The
temporal variation of interparticle separation (r, length of the line joining
the centre of spheres) with UV illumination. (e) Corresponding angle of
rotation $\theta$ of the chain with respect to n0. (f) POM and corresponding
$\lambda-$ plate image of the final equilibrium state of a chain of three
particles (Schematic of the surrounding director profile is shown in the
inset). (g) Colour maps of the director profile at the surface of the
particles resembling an elastic multipole with $l=6$. (h) POM and
corresponding $\lambda-$ plate images of the final equilibrium state of a
chain of four particles (Movie-6). (i) Colour maps of the director profile at
the surface of the particles resembling an elastic multipole with $l=8$. (j)
Variation of the maximum rotation angle $\theta$ with an increase in chain
length, for a fixed intensity of UV. (k) A vertical chain of dipoles
undergoing compression in the $y-$ direction. (l) A horizontal chain of
dipoles undergoing compression in the $x-$ direction. P, A represents the
polariser and analyser. The double-headed arrow denotes the nematic director,
n0.

####  1.1.5 Rotation of the particles during defect-transformation

It is observed that the light-activated defect dynamics is accompanied by the
rotation of the particles as shown in Figure 4a (Movie-5 [35]). When
irradiated with UV, the orientation of the director at the surface changes
from homeotropic to planar and simultaneously the sphere rotates by almost
$90^{\circ}$ either in-plane (upper row of Figure 4a) or out-of-plane (bottom
row of Figure 4a). A schematic diagram of the corresponding director structure
is shown in Figure 4(c-e). The director rotation gives rise to elastic torque
which is proportional to $KD$, where $K$ is the average Frank elastic constant
and $D$ is the diameter of the particle. This torque is counteracted by a
viscous torque which is proportional to $\eta D^{3}$, where $\eta$ is the mean
viscosity of the LC [10, 29]. Considering the anchoring is stronger for the
planar director (the case with higher intensity) [29] switching response time
of the sphere can be expressed as $\tau=\eta D^{2}/K$. Taking $\eta=75$ mPa s
and $K=5$ pN, the calculated response time $\tau\simeq 1.5$ s, which agrees
well with the experimentally observed response time (Figure 4b). Low UV
intensity corresponds to weak surface anchoring [29], where the elastic torque
exerted by the director depends on a characteristic length $D+\xi$, where
$\xi$ is the penetration length given by $\xi=K/W$. In this case, the response
time is given by $\tau=\eta(D+\xi)^{2}/K\simeq 6$s which is also in good
agreement with experiments. A comparison of the response time of elastic
dipoles and quadrupoles for a given intensity of UV always shows higher values
for dipoles (Figure 4b). This behaviour is expected, as the transformation
from dipolar to boojum configuration involves more intermediate defect states
than quadrupole to boojum.

####  1.1.6 Rotation of colloidal chains

The response of a dimer consisting of two colinear elastic dipoles with
negative elastic dipole moment, p$<0$ (directed along the -$x$ axis) under UV
light is shown in Figure 5(a,b). As a result of the director re-orientation
occurring at the surface of each dipole, they undergo defect transformation
and rotate collectively nearly by $45^{\circ}$ in the anticlockwise direction
as shown in Figure 5a. In particular, under UV illumination, the surface
anchoring of the dimers changes from homeotropic to conic degenerate
($0^{\circ}\leq\alpha\leq 90^{\circ}$) state, where the hedgehog point defects
transform to a combination of boojum and neck defects [32] as illustrated in
the schematic. Here, the elastic distortion of the dimers corresponds to that
of an elastic multipole of order $l=4$ (Figure 5c). Changing the polarity of
dipole moment (-p) reverses the sense of rotation of the dimers.

Switching off UV ( Figure 5b) again causes the director to adopt homeotropic
alignment, hence driving defect transformation with an intermediate state
having a combination of Saturn ring, neck and boojum defects (Figure 5b
[$t=10$ s] and underneath schematic), which later transforms to Saturn rings
(Figure 5b [$t=53$ s]). The rotation angle further increases during this
transformation. The temporal variation in inter-particle separation (distance
between the centre of two spheres, $r$) and the angular tilt with respect to
the far field director $n_{0}$ for a typical pair of particles is plotted in
Figure 5(d,e), which shows a considerable reduction ($\approx$1$\mu$m) of the
interparticle separation. The effect of UV on larger chains is studied
systematically by adding more particles to it with the help of laser tweezers.
A chain of three dipoles under UV irradiation transforms to a final state as
shown in Figure 5f (schematic in the inset) whose director structure resembles
that of an elastic multipole of order, $l=6$ (Figure 5g). Similarly, a chain
of four dipoles transforms to an elastic multipole of higher order, $l=8$
(Figure 5(h,i) Movie-6 [35]). Thus, simply by adding more particles to the
chain, we can create higher-order elastic multipoles. It is apparent that with
increasing chain length, the angle of rotation with respect to the director,
decreases ( Figure 5j). This could be rationalised by drawing an analogy with
the rotation of a rod with increasing length in a nematic LC. The elastic free
energy associated with the rotation of a rod of length $L$ at an angle
$\theta$ with respect to the director is given by: $F_{el}=2\pi CK\theta^{2}$
where, $C=\frac{L}{2}log(L/d)$ [36]. At equilibrium, the elastic torque
$\tau_{el}=-\frac{\partial F}{\partial\theta}\propto L\theta$ is balanced by
the viscous torque $\tau_{vis}\propto\eta S^{3}$, where $S$ is the gap between
the upper and lower glass plates. Hence, the angular tilt $\theta\propto 1/L$,
as shown in Figure 5j. Figures 5(k,l) show two kinds of dipolar chains
assembled perpendicular and parallel to $n_{0}$ making use of the anisotropic
interactions specific to LCs. In a perpendicular chain, any two successive
dipoles are oriented oppositely and in parallel chains they are colinear. As
evident from the figure, the defect transformations not only cause rotation,
but also reduction in length either along $x-$ (for parallel chains) or
$y-$(for perpendicular chains) direction. It immediately comes to notice that
both kinds of length compression can be employed in a 2D crystal structure
obtained by stacking several such dipolar chains, thus allowing robust control
over structural transformations.

Figure 6: (a-i) A 2D crystal of elastic dipoles formed by directed assembly
using the laser tweezers. Equilibrium lattice parameters: $a_{1}$ = 6.67 $\mu$
m, $b_{1}$ = 6.11 $\mu$ m, $\lambda_{1}=54.4^{0}$, Area $\approx$ 3419 $\mu
m^{2}$. (a-ii) Schematic of a $4\times 4$ dipolar crystal (area enclosed by
the rectangular selection). (a-iii) Polarising microscopy texture. (b-i)
Crystal after transformation by UV irradiation. Lattice parameters: $a_{2}$ =
5.42 $\mu$m, $b_{2}$ = 5.30 $\mu$ m, $\lambda_{2}=63.6^{0}$, Area $\approx$
2681 $\mu m^{2}$. (b-ii) Schematic of a $4\times 4$ crystal after
transformation (area enclosed by the rectangular selection). (b-iii)
Polarising microscopy texture. (c) Change in lattice parameters during crystal
transformation (Movie-7). Particles tracked for time-dependent lattice
parameters are marked red. P, A represents polariser and analyser. Double-
headed arrow denotes the far-field director.

####  1.1.7 Structural transformation of a 2D colloidal crystal

Using a dynamic laser tweezers setup, the elastic dipoles were directed to
form a regular 2D colloidal crystal as shown in Figure 6a(i-iii). Here, the
anchoring mismatch at the cell boundary (planar anchoring) and the surface of
the spheres (homeotropic anchoring) introduces a wall-dipole repulsion which
balances the gravitational pull and causes the particle to levitate in the LC
channel [37]. The structural change of the 2D crystal after UV radiation is
shown in Figure 6b(i-iii). Analysis of the structure reveals that before UV
irradiation, the particles form a unit cell of hexagonal lattice ($a\neq b$)
with average lattice parameters $a_{1}=6.33~{}\mu$m and $b_{1}=5.75~{}\mu$m
and an angle, $\lambda\approx 54.4^{\circ}$ between them. The temporal changes
of the lattice parameters are shown in Figure 6c which shows that within about
30 s the transformation is completed and eventually $a=b$ at a longer time
(Movie-7 [35]). After irradiation with UV, the surface director $n_{s}$ of the
elastic dipoles tend to rotate by an angle of $90^{\circ}$, which is not
possible now as the free rotation of the nematic director is arrested due to
additional elastic constraints imposed by the neighbouring particles. This
results in a conic degenerate anchoring of $n_{s}$, with a topological defect
configuration consisting of neck defects at particle joints and boojum defects
at the ends of each chain (figure 6(b-ii)). Hence, they form an oblique square
lattice with ($a\approx b$) and lattice parameters, $a_{2}=5.58~{}\mu$m,
$b_{2}=5.67~{}\mu$m, and $\lambda=63.6^{\circ}$. An overall compression in the
crystal area by about 24% of the initial area is measured.

##  2 Conclusion

Our experiments demonstrate light-activated defect-dynamics occurring at the
interface of spherical inclusions embedded in an anisotropic nematic liquid
crystalline matrix and its usefulness towards realising structural
transformations across 2D colloidal assemblies. This is achieved by decorating
the surface of microspheres with azo-dendrimers exhibiting conformational
change ($cis-trans$) when perturbed with UV irradiation of sufficient
intensity. The UV irradiation triggers particle rotation and drives out-of-
equilibrium defect transformations in the system where the evolving director
field reveals higher-order elastic multipoles. These intermediate defect
states of higher order and complexity is pivotal in exploring the richness of
anisotropic interactions in LCs and their potential in building remotely
reconfigurable soft crystals. As a direct application, the defect-
transformation-induced restructuring of a 2D colloidal crystal is
demonstrated. Our findings can be extended to nematic emulsions as well as
having wide-ranging applications in microfluidics.

##  3 Experimental Section

We prepared an azo-dendrimer-NLC mixture by dissolving a small quantity (
$<0.1$ wt%) of poly (propyleneimine) based azo dendrimer in pentyl
cyanobiphenyl (5CB) liquid crystal (Sigma Aldrich) [38]. The azo-dendrimer has
an absorption peak at 360 nm of incident light, where it reconfigures to the
isomeric $cis-$ state [38]. Initially, the dendrimer was dissolved in
chloroform solvent and then stirred continuosly for 24 hrs using a magnetic
stirrer. Any clusters were removed by filtering the solution through a micro
seive of diameter 0.22 $\mu$m. It was then mixed with 5CB after which the
chloroform solvent is allowed to evaporate completely at room temperature
under a vacuum. Silica microparticles of diameter 5.24 and 9.2$\mu$m (Bangs
Laboratory, USA) were dispersed into the as-prepared azo dendrimer-5CB mixture
and sonicated for 10 mins. to obtain uniform azo-dendrimer coating on the
silica surface. Planar cells conferring uni-directional alignment of the
nematic director were fabricated using the following technique: ITO-coated
glass plates washed thoroughly and dried under nitrogen atmosphere are spin-
coated with a polyimide (AL-1254) and annealed at $180^{0}$C for 1 hr. The
polyimide-treated glass plates are then rubbed using a velvet cloth along a
specific direction. Two glass substrates having opposite rubbing directions
are attached with silica spacers and a UV-curable optical adhesive to form a
capillary-like arrangement.
For observations, an inverted polarizing optical microscope (Nikon eclipse
Ti-U) with a halogen lamp (operating at 12V, 100W) as incident light source
and 60X water immersion objective (Nikon, NIR Apo 60/1.0) was used. The
director distribution at the particle surface was mapped with the help of a
full retardation wave plate (530 nm) placed at $45^{0}$ with respect to the
analyser. A fluorescent source (Nikon, Intensilight C - HGFI) operating on a
mercury lamp with controllable brightness connected to the microscope provided
UV light. For this, light from the fluorescent source was passed through a UV
filter cube (330 - 380 nm). An optical tweezer built on the inverted
microscope employing a laser light of 1064 nm and controlled by an acousto-
optic deflector, helped in setting traps and manipulating particles. Image
acquisition and video recording were performed by a CCD camera and a Nikon DS-
Ri2 colour camera attached to the microscope. Data analysis was performed
using ImageJ software.

Supporting Information

Supporting Information is available from the Wiley Online Library or from the
author.

Acknowledgements

DVS and DKS acknowledges DST, Government of India for INSPIRE fellowship. SD
acknowledges support from IoE (UoH/IoE/RC1-20-010).

## References

  * [1] I. Muševič, M. Škarabot, U. Tkalec, M. Ravnik, S. Žumer, Science 2006, 313, 954.

  * [2] S. Sacanna, L. Rossi, D. J. Pine, J. Am. Chem. Soc. 2012,134, 6112.

  * [3] A. V. Straube, J. M. Pagés, A. Ortiz-Ambriz, P. Tierno, J. Ignés-Mullol, F. Sagués, New J. Phys. 2018, 20, 075006.

  * [4] D. V. Sudhakaran, R. K. Pujala, S. Dhara, Adv. Optical Mater. 2020, 8, 1901585.
  * [5] N. Kang, J. Zhu, X. Zhang, H. Wang, Z. Zhang, J. Am. Chem. Soc. 2022, 144, 4754.
  * [6] J. Palacci, S. Sacanna, A. P. Steinberg, D. J. Pine, P. M. Chaikin, Science 2013, 339, 936.
  * [7] D. Zerrouki, J. Baudry, D. Pine, P. Chaikin, J. Bibette, Nature 2008, 455, 380.
  * [8] D. P. Singh, U. Choudhury, P. Fischer, A. G. Mark, Adv. Mater. 2017, 29, 1701328.
  * [9] K.P. Zuhail, S. Dhara, Soft Matter 2016, 12, 6812.
  * [10] Y. Yuan, G. N. Abuhaimed, Q. Liu, I. I. Smalyukh, Nat. Commun. 2018, 9, 5040.
  * [11] A. Martinez, H. C. Mireles, I. I. Smalyukh, PNAS 2011, 108, 20891.
  * [12] P. G. de Gennes, J. Prost, The Physics of Liquid Crystals, Clarendon, Oxford 1993.
  * [13] H. Stark, Phys. Rep. 2001, 351, 387.
  * [14] I. Muševič, Liquid Crystal Colloids, Springer International Publishing AG 2017.
  * [15] B. Senyuk, Q. Liu, S. He, R. D. Kamien, R. B. Kusner, T. C. Lubensky, I. I. Smalyukh, Nature 2013, 493, 200.
  * [16] P. Poulin, H. Stark, T. C. Lubensky, D. A. Weitz, Science 1997, 275, 1770.
  * [17] S.B. Chernyshuk, Eur. Phys. J. E 2014, 37, 6.
  * [18] B. Senyuk, O. Puls, O. M. Tovkach, S. B. Chernyshuk, I. I. Smalyukh, Nat. Commun. 2016, 7, 10659.
  * [19] S. B. Chernyshuk, O. M. Tovkach, Liq. Cryst. 2016, 43, 2410.
  * [20] A. Nych, U. Ognysta, M. Škarabot, M. Ravnik, S. Žumer, I. Muševič, Nat. Commun. 2013, 4, 1489.
  * [21] S. Ramaswamy, R. Nityananda, V. A. Raghunathan, J. Prost, Mol. Cryst. Liq. Cryst. 1996, 288, 175.
  * [22] M. Tasinkevych, N. M. Silvestre, M. M. Telo da Gama, New J. Phys. 2012, 14, 073030.
  * [23] M. Škarabot, M. Ravnik, S. Žumer, U. Tkalec, I. Poberaj, D. Babič, N. Osterman, I. Muševič, Phys. Rev. E 2008, 77, 031705.
  * [24] T. Uchida and H. Seki, Surface Alignment of Liquid Crystals, World Scientific, Toronto, Canada 1992.
  * [25] O. Ou Ramdane, Ph. Auroy, S. Forget, E. Raspaud, Ph. Martinot-Lagarde, I. Dozov, Phys. Rev. Lett. 2000, 84, 3871.
  * [26] T. Seki, M. Sakuragi, Y. Kawanishi, T. Tamaki, R. Fukuda, K. Ichimura, Y. Suzuki, Langmuir 1993, 9, 211.
  * [27] K. Fukuhara, S. Nagano, M. Hara, T. Seki, Nat. Commun. 2014, 5, 3320.
  * [28] P. Hirankittiwong, N. Chattham, J. Limtrakul, O. Haba, K. Yonetake, A. Eremin, R. Stannarius, H. Takezoe, Optics Express. 2014, 22, 20087.
  * [29] A. Eremin, P. Hirankittiwong, N. Chattham, H. Nádasi, R. Stannarius, J. Limtrakul, O. Haba, K. Yonetake, H. Takezoe, Proc. Natl. Acad. Sci (USA). 2015, 112, 1716.
  * [30] A. Eremin, H. Nádasi, P. Hirankittiwong, J. Kiang-Ia, N. Chattham, O. Haba, K. Yonetake, H. Takezoe, Liq. Cryst. 2018, 45, 2121.

  * [31] E. Merino, M. Ribagorda, Beilstein J. Org. Chem. 2012, 8, 1071.
  * [32] B. Senyuk, J. Aplinc, M. Ravnik, I. I. Smalyukh, Nat. Commun. 2019, 10, 1825.
  * [33] B. Senyuk, A. Mozaffari, K. Crust1, R. Zhang, J. J. de Pablo, I. I. Smalyukh, Sci. Adv. 2021, 7, 0377.
  * [34] Y. Zhou, B. Senyuk, R. Zhang, I. I. Smalyukh, J. J. de Pablo, Nat. Commun. 2019, 10, 1000.
  * [35] Description of supplementary movies (combined): Movie-1 shows the transformation of an elastic dipole to a boojum under cross polarisers upon UV irradiation (Intensity = 85.2 $\mu$W/mm2). Movie-2 shows the transformation of boojums to a Saturn-ring when UV light is switched off. Movie-3 shows the transformation of a Saturn-ring to a boojum under cross polarisers upon UV irradiation (Intensity = 85.2 $\mu$W/mm2). Movie-4 shows the reverse transformation of boojum to a Saturn-ring under cross polarisers when light is switched off. Movie-5 shows the rotation of a sphere with accompanying hedgehog point defect upon UV irradiation (Intensity = 85.2 $\mu$W/mm2). Movie-6 shows the rotation of a self-assembled colinear chain of four elastic dipoles under UV light (Intensity = 85.2 $\mu$W/mm2). Movie-7 shows the structural transformation of a 2D dipolar crystal comprising of 100 particles under UV light (Intensity is increased from 24.2 $\mu$W/mm2 to 0.72 $\mu$W/mm2 in five discrete steps). All movies are shown in real time. The double headed arrow represents the nematic director.
  * [36] F. Brochard, P. G. de Gennes, J. Phys. France. 1970, 31, 691.
  * [37] O. P. Pishnyak, S. Tang, J. R. Kelly, S. V. Shiyanovskii, O. D. Lavrentovich, Phy. Rev. Lett. 2007, 99, 127802.
  * [38] W. Li, T. Dohi, M. Hara, S. Nagano, O. Haba, K. Yonetake, T. Seki, Macromolecules 2012, 45, 6618.