# Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies

Eric Chu MIT Media Lab<EMAIL_ADDRESS>Deb Roy MIT Media Lab<EMAIL_ADDRESS>
###### Abstract

Stories can have tremendous power – not only useful for entertainment, they
can activate our interests and mobilize our actions. The degree to which a
story resonates with its audience may be in part reflected in the emotional
journey it takes the audience upon. In this paper, we use machine learning
methods to construct emotional arcs in movies, calculate families of arcs, and
demonstrate the ability for certain arcs to predict audience engagement. The
system is applied to Hollywood films and high quality shorts found on the web.
We begin by using deep convolutional neural networks for audio and visual
sentiment analysis. These models are trained on both new and existing large-
scale datasets, after which they can be used to compute separate audio and
visual emotional arcs. We then crowdsource annotations for 30-second video
clips extracted from highs and lows in the arcs in order to assess the micro-
level precision of the system, with precision measured in terms of agreement
in polarity between the system’s predictions and annotators’ ratings. These
annotations are also used to combine the audio and visual predictions. Next,
we look at macro-level characterizations of movies by investigating whether
there exist ‘universal shapes’ of emotional arcs. In particular, we develop a
clustering approach to discover distinct classes of emotional arcs. Finally,
we show on a sample corpus of short web videos that certain emotional arcs are
statistically significant predictors of the number of comments a video
receives. These results suggest that the emotional arcs learned by our
approach successfully represent macroscopic aspects of a video story that
drive audience engagement. Such machine understanding could be used to predict
audience reactions to video stories, ultimately improving our ability as
storytellers to communicate with each other.

Keywords – visual sentiment analysis, audio sentiment analysis, multimodal,
emotions, emotional arcs, stories, video.

##  I Introduction

Theories of the origin and purpose of stories are numerous, ranging from story
as “social glue”, escapist pleasure, practice for social life, and cognitive
play [8, 12].

However, not all stories produce the same emotional response. Can we
understand these differences in part through the lens of emotional arcs? Kurt
Vonnegut, among others, once proposed the concept of “universal shapes” of
stories, defined by the “Beginning–End” and “Ill Fortune–Great Fortune” axes.
He argued nearly all stories could fit a core arc such as the classic
“Cinderella” (rise-fall-rise) pattern [1].

It is well known that emotions play no small part in people’s lives. We have
seen emotional narratives as a convincing medium for explaining the world we
inhabit, enforcing societal norms, and giving meaning to our existence [21].
It has even been shown that emotions are the most important factor in how we
make meaningful decisions [19].

There is also evidence that a story’s emotional content can explain the degree
of audience engagement. The authors of [4] and [3] examined whether the
valence and emotionality was predictive of New York Times articles making the
Times’ most e-mailed list. Ultimately, they found that emotional and positive
media were more likely to be shared.

Motivated by the surge of video as a means of communication [14], the
opportunities for machine modeling, and the lack of existing research in this
area, we makes steps towards tackling these questions by viewing movies
through emotional arcs. We deliver the following contributions:

  * •

Datasets. We introduce a Spotify dataset containing over 600,000 audio samples
and features that can be used for audio classification. We also collect
annotation data regarding the sentiment of approximately 1000 30-second movie
clips. Both datasets will be made publicly
available111https://sosuperic.github.io/a-darn-good-yarn-website.

  * •

Modeling emotional arcs. We train sentiment classifiers that can be used to
compute audio and visual arcs. We also motivate a) the use of dynamic time
warping with the Keogh lower bound to compare the shapes of arcs, and b)
k-medoids as the algorithm for clustering.

  * •

Engagement analysis. We provide an example in which a movie’s arc can be a
statistically significant predictor of the number of comments an online video
receives.

##  II Related work

The work is most closely paralleled by [26], which analyzed books to state
that “the emotional arcs of stories are dominated by six basic shapes.” Using
text-based sentiment analysis, they use a singular value decomposition
analysis to find a basis for arcs. The bases that explain the greatest amount
of variance then form the basic shapes of stories.

In contrast to our computational approach, writers at Dramatica [2] have
manually analyzed a number of books and films under the Dramatica theory of
story, which has since been used to create software that can guide writers.

Research in sentiment analysis has primarily been text-based, with work
ranging from short, sentence-length statements to long form articles [28, 24].
There has been comparatively little work on images, with the Sentibank visual
sentiment concept dataset [7] being a prominent example.

Neural networks have been used in earlier research for tasks such as document
recognition [18]. In recent years, deep neural networks have been successful
in both the visual and audio domain, being used for image classification [17],
speech recognition [13], and many other tasks.

Outside of emotional arcs, there exists other research that applies
computational methods to understanding story. For instsance, the M-VAD [29]
and MovieQA [27] datasets include a combination of subtitles, scripts, and
Described Video Service narrations for movies, enabling research on visual
question-answering of plot.

##  III Overview

Figure 1 outlines the major pieces of this work. We note that modeling occurs
at two scales – micro-level sentiment, performed on a slice of video such a
frame or snippet of audio, and macro-level emotional arcs.

Figure 1: Overview

Reflecting this distinction, we first evaluate the ability to accurately
extract micro-level emotional highs and lows, which we refer to as emotionally
charged moments or emotional peaks and valleys. Specifically, we measure
precision as the amount of agreement in polarity between the models’ sentiment
predictions and annotators’ ratings. Second, at the macro-level, we evaluate
arcs by a) clustering, and b) using the arcs for the engagement analysis. We
note that the final engagement analysis uses only the visual arcs, as we are
currently limited by the amount of ground truth data that allows us to combine
audio and visual predictions.

##  IV Datasets

###  IV-A Videos – Films and Shorts Corpora

The system operates on two datasets collected for this work – a corpora of
Hollywood films and a corpora of hiqh quality short films. We selected films
because they are created to tell a story. That there exist common film-making
techniques to convey plot and elicit emotional responses also suggests the
possibility of finding families of arcs.

Including Vimeo shorts allows us to 1) find differences in storytelling that
may exist between films and newer, emergent formats, 2) possibly serve as a
gateway to modeling, more generally, the short form videos that commonly
spread on social networks, and 3) conduct an engagement analysis using the
online comments left on Vimeo.

The first dataset, the Films Corpora, consists of 509 Hollywood films.
Notably, there is considerable overlap with the MovieQA [27] and M-VAD [29]
datasets. The Shorts Corpora is a dataset of 1,326 shorts from the Vimeo
channel ‘Short of the Week’. These shorts are collected by filmmakers and
writers. A short can be 30 seconds to over 30 minutes long, with the median
length being 8 minutes and 25 seconds.

###  IV-B Image sentiment – Sentibank dataset

We use the Sentibank dataset [7] of nearly half a million images, each labeled
with one of 1,533 adjective-noun pairs. These pairs, such as “charming house”
and “ugly fish”, are termed emotional biconcepts. Each biconcept is mapped to
a sentiment value using the SentiWordnet lexicon.

###  IV-C Audio sentiment – Spotify dataset

We started with the Million Song Dataset [6], in which certain metadata, such
as tempo or major/minor key, could in theory be used as a proxy for valence
and other relevant target features. We also considered the Last.FM dataset of
user-provided labels such as “happy” or “sad”. Unfortunately, this dataset is
significantly smaller.

To address our needs, we turned to the Spotify API. The API provides not only
30-second samples for songs, but also audio features used by the company.
These include valence, which measures the “musical positiveness conveyed by a
track.” Other features include speechiness, liveness, etc.

##  V Methodology

###  V-A Constructing emotional arcs

Once the visual and audio models are trained for sentiment prediction (to be
explained in Sections V-B and V-C), each is applied separately across the
length of the movie. To construct the visual arc, we extract a frame per
second in the movie, resize and center crop it to size $256\times 256$, and
then pass it through the sentiment classifier. To construct the audio arc, we
extract sliding 20-second windows.

Figure 2: Effect of smoothing values for arcs: no smoothing versus $w=0.1*n$

While the left plot in Figure 2 shows that the macro-level shape is visible
from the raw predictions, it is helpful to smooth these signals to produce
clearer arcs by convolving each time series with a Hann window of size $w$. In
downstream tasks, commonly used window sizes are 0.05, 0.1, and 0.2 * $n$,
where $n$ is the length of the video. Example arcs are shown in Figure 3, with
the audio arc bounded by the confidence intervals to be described in Section
V-C.

Figure 3: Audio (yellow) and visual (blue) arcs for Her

###  V-B Image modeling

The various effects of the visual medium has been well studied, ranging from
the positive psychological effects of nature scenes [30] to the primacy of
color, an effect so powerful that some filmmakers explicitly map color to
target emotions in pre-production colorscripts [5]. We thus built models take
a frame as input.

####  V-B1 Model

We use a deep convolutional neural network based on the AlexNet architecture
[17] to classify images. While a more state-of-the-art architecture would have
higher accuracy, our focus was on building higher order arcs, for which this
relatively simple model sufficed. However, we did use the PReLU activation
unit, batch normalization, and ADAM for optimization to reflect recent
advancements.

####  V-B2 Sentiment prediction

The network was trained using images with a sentiment greater than 0.5 as
‘positive’, and those less than -0.5 as ‘negative’. We used a learning rate of
0.01, a batch size of 128, and a batch normalization decay of 0.9. The
performance is shown in Table I.

Accuracy | Precision | Recall | F1
---|---|---|---
0.652 | 0.753 | 0.729 | 0.741

TABLE I: Performance of sentiment classifier

####  V-B3 Emotional biconcept prediction

Using only the sentiment value is useful for creating emotional arcs, but it
also discards information. Thus, we trained a second network that treats the
biconcepts as labels. This network proves useful in creating movie embeddings
that broadly capture a movie’s emotional content. Details are discussed in
section VI-C.

Only biconcepts with at least 125 images were used, leaving 880 biconcepts.
The accuracy is shown in Table II. Top-$k$ accuracy is defined as the percent
of images for which the true label was found in the top-$k$ predicted labels.
We also show the top-$k$ accuracy defined by whether the true adj / noun was
found in the top-$k$ predicted labels.

| Acc.
---|---
Top-1 | 7.4%
Top-5 | 19.9%
Top-10 | 28.4%

(a) Predicting adj-noun pair

| Match adj | Match noun
---|---|---
Top-1 | 12.0% | 15.1%
Top-5 | 30.2% | 31.7%
Top-10 | 41.5% | 40.7%

(b) Matching adj / noun in predicted adj-noun pair

TABLE II: Performance of emotional biconcept classifier

###  V-C Audio modeling

Imagine ‘watching’ a movie with your eyes closed – you would likely still be
able to pinpoint moments of suspense or sadness. While often secondary to the
more obvious visual stimuli, sound and music can be played with or in contrast
to the visual scene. With the idea that just a few seconds is enough to set
the mood, we created a model for sentiment classification that operates on
20-second snippets of audio.

####  V-C1 Model

We represent each audio sample as a 96-bin mel-spectrogram. We adopt the
architecture used for music tagging in [9], which uses five conv layers with
ELU and batch normalization, followed by a fully connected layer.

####  V-C2 Sentiment prediction

We used all samples that have a valence either greater than 0.75 or less than
0.25, leaving $\sim$200,000 samples. The performance is shown in Table III.

Accuracy | Recall | Precision | F1
---|---|---|---
0.896 | 0.871 | 0.931 | 0.900

TABLE III: Performance of audio sentiment classifier

####  V-C3 Uncertainty estimates

Unfortunately, we face the problem of covariate shift, where the audio found
in movies will often contain sound not found in the song-based training set.
For example, there may be significant sections of background noise,
conversation, or silence.

To handle ‘unfamiliar’ inputs, we aim to produce confidence intervals for
every prediction. While the softmaxed activations can be interpreted as
probabilities, and hence a reflection of confidence, these probabilities can
often be biased and require calibration [23]. We thus follow a method
introduced in [11], which produces approximate uncertainty estimates for any
dropout network by passing the input $m$ times through the network with
dropout at test time, and using the standard deviation of the predictions to
define a confidence interval around the mean of the predictions.

###  V-D Finding families of emotional arcs

####  V-D1 Approach: k-medoids and dynamic time warping

A naive approach to clustering arcs could be to use a popular algorithm such
as k-means [20] with an Euclidean metric to measure the distance between two
arcs. However, this is a poor approach for our problem for two reasons:

  * •

Taking the mean of arcs can fail to find centroids that accurately represent
the shapes in that cluster. Figure 4 shows a pathological example of when this
occurs. The mean of the left two arcs has two peaks instead of one.

  * •

The Euclidean distance between two arcs doesn’t necessarily reflect the
similarity of their shapes. While the left two arcs in Figure 4 are similar in
shape (one large peak), their Euclidean distance may be quite large.

With these limitations in mind, we turn to k-medoids [15] with dynamic time
warping (DTW) [10] as the distance function. K-medoids updates a medoid as the
point that is the median distance to all other points in the cluster, while
DTW is an effective measures the distance between two time series that may
operate at different time scales.

Figure 4: Pathological example of how k-means and Euclidean distance fails for
clustering emotional arcs

Given two time series $A$ and $B$ of length $n$, we construct a $n$x$n$ matrix
$M$, where $M[i][j]$ contains the squared difference between $A_{i}$ and
$B_{j}$. The DTW distance is the shortest path through this matrix. In Figure
4, for example, the DTW distance between the left two time series is 0.

####  V-D2 LB-Keogh for speed-up and better modeling

Several techniques to speed up DTW center around creating a ‘warping window’
that limits the available paths through the matrix $M$. The Keogh lower bound
creates upper and lower bounds that envelop the original time series $A$,
defined as:

$\displaystyle U_{i}=max(A_{i-r}:A_{i+r})$ (1) $\displaystyle
L_{i}=min(A_{i-r}:A_{i+r})$ (2)

where $r$ is a parameter reach that controls the size of the window.
Intuitively, this controls how much a time series is allowed to warp. The
lower-bounded $LB_{Keogh}$ distance between $A$ and $B$ is then given as:

$\displaystyle
LB_{Keogh}(A,B)=\sqrt{\sum_{i=1}^{n}\begin{cases}(B_{i}-U_{i})^{2}&\text{if
$B_{i}>U_{i}$}\\\ (B_{i}-L_{i})^{2}&\text{if $B_{i}<L_{i}$}\\\
0&\text{otherwise}\end{cases}}$ (3)

If $B$ lies inside the envelope of $A$, then the distance is 0.

Importantly, this approach has ramifications beyond increased speed up.
Consider again the left two arcs in Figure 4. While both characterized by a
large peak, it’s possible that the second, ending on a emotional moment, has
greater impact. Consequently, we would like to only allow warping to a certain
extent. The window does exactly that.

####  V-D3 Practical notes

First, we exclude movies longer than $m$ seconds (10000 and 1800 for the Films
and Shorts Corpora). After all, a 60 minute movie is hardly a ‘short’. Second,
since we are interested in the overall shape of the arc, we z-normalize each
emotional arc. Finally, we settled on $r=0.025*n$. This is close to $0.03*n$,
found to be optimal for a number of different tasks [25].

##  VI Evaluation – Micro-level Moments

We evaluated the system’s micro-level accuracy by its precision in extracting
emotionally charged moments from the emotional arcs. Notably, collecting
ground truth data also allows us combine the audio and visual predictions.

Annotating an entire film, let alone multiple films, would be too costly and
time intensive. We thus extracted video clips at peaks and valleys in the
audio and visual arcs, hoping that these clips would lead to more interesting
and informative annotations. Workers were asked to watch a clip and answer
four questions regarding its emotional content. Each clip was annotated by
three workers.

###  VI-A Crowdsourcing experiment

We chose the CrowdFlower platform for its simplicity and ease of use. Answers
to Question 1 (How positive or negative is this video clip? 1 being most
negative, 7 being most positive) are referred to as valence ratings. We define
a positive rating as a valence rating greater than 4, and a positive clip as
those with a mean rating greater than 4. Negative ratings and clips are
similarly defined.

###  VI-B Precision of system

####  VI-B1 Defining precision

We define the precision as:

$\frac{\text{$|$peak \& positive$|$ + $|$valley \& negative$|$}}{\text{number
of clips}}$ (4)

In other words, a clip was accurately extracted if a) it was extracted from a
peak in either arc, and it was labeled as a positive clip, or b) it was
extracted from a valley in either arc, and it was labeled as a negative clip.

####  VI-B2 Overall precision

Table IV lists the precision on both the full dataset and the full dataset
with ambiguous clips (receiving both a positive and a negative rating)
removed. We note that random chance would be 3/7 = 0.429. We argue that
ambiguous clips should not be included, as it is unclear what their valence
might be without more annotations. Further numbers are calculated with
ambiguous clips removed.

Set | Precision
---|---
All clips | 0.642
No ambiguous clips | 0.681

TABLE IV: Precision of clips: overall

####  VI-B3 Precision of audio

Next, we look at the precision of clips extracted from the audio arc,
examining our hypothesis that predictions with smaller confidence intervals
would be more accurate. This would help affirm our uncertainty estimates
approach. Table V shows that smaller confidence intervals do indeed correspond
with greater precision.

Stddev | Audio-peak precision | Audio-valley precision
---|---|---
[0, 0.02) | 4 / 4 = 1.0 | 81 / 88 = 0.921
[0.02, 0.04) | 11 / 11 = 1.0 | 38 / 56 = 0.679
[0.04, 0.06) | 28 / 40 = 0.7 | 21 / 35 = 0.6
[0.06, 0.08) | 39 / 60 = 0.65 | 13 / 21 = 0.619
[0.08, 0.1) | 43 / 68 = 0.632 | 8 /23 = 0.615

TABLE V: Precision of clips extracted from audio emotional arc: smaller
confidence intervals are more precise

####  VI-B4 Precision of various cuts

The precision on various subsets is shown in Table VI (a). We highlight that
the visual-peaks have low precision. This is explored in the next section and
used to motivate feature engineering for the combined model in Section VI-C.

Cut | Precision
---|---
Audio-peaks | 0.683
Audio-valleys | 0.758
Visual-peaks | 0.508
Visual-valleys | 0.757

(a) Cuts

Genre | Overall | Visual-peak
---|---|---
Action | 0.678 | 0.264
Science Fiction | 0.699 | 0.333
Thriller | 0.678 | 0.382
Adventure | 0.726 | 0.443
Drama | 0.660 | 0.520
Fantasy | 0.769 | 0.590
Comedy | 0.705 | 0.667
Animation | 0.798 | 0.667
Family Film | 0.760 | 0.722
Romance | 0.678 | 0.757
Romantic Comedy | 0.677 | 0.823

(b) Genres

TABLE VI: Precision of clips: cuts and genre

####  VI-B5 Precision by genre

Finally, we use [22] to tag each movie with genres. A subset of the results is
shown in Table VI (b). The relatively poor precision of visual-peaks, as noted
in the previous section, appears to be a product of poor precision on a number
of genres. We also find a natural grouping of the genres when listed in this
order. Genres with high visual-peak precision appear to be lighter films
falling in the romance and family film genres.

Manual inspection of ‘incorrect’ visual-peak clips from the action-thriller
genres shows many scenes with gore and death, images unlikely to be found in
the Sentibank dataset, which was culled from publicly available images on
Flickr.

###  VI-C Combined audio-visual model

We create a linear regression model to predict the mean valence rating
assigned by the annotators. In addition to standard features related to the
clip’s audio / visual valence (relative to the movie’s mean, relative to the
movie’s max, etc.), we create two key features detailed below.

Peakiness. The function $p(a,i,r)$ approximates the slope and mean around a
given point $i$ for arc $a$ (audio or visual), where $r$ is the window size
around $a_{i}$. $p$ returns 4 values: $a_{i-1}-a_{i-r}$ (proportional to the
slope left of the $i$), $a_{i+r}-a_{i+1}$ , $mean(a_{i-r:i-1})$ (the mean
value left of $i$), and $mean(a_{i+1:i+r})$. This covers peaks, valleys, and
inflection points. We use $r=0.025$ for our analyses.

Movie embedding. Motivated by the impact of genre in Section VI-B5, we sought
to loosely summarize a movie’s emotional content. We represent each frame by
the penultimate activation of the biconcept classifier described in Section
V-B. Next, we average these activations across 10% chunks of the movie,
resulting in a $10\times 2048$ matrix. To translate these movie embeddings to
features, we take the mean of each 2048-sized vector, ending with a final
$10\times 1$ feature vector. While not shown in the interest of space,
clustering these movie embeddings shows correspondence between clusters and
genres, with romance, adventure, fantasy, and animated films being clearly
visible.

###  VI-D Precision of combined audio-visual model

The performance of the final combined model is shown in Table VII, along with
various ablations of important features. Using all features, we achieve a
precision of 0.894.

Feature set | Overall | Aud-peak | Aud-valley | Vis-peak | Vis-valley
---|---|---|---|---|---
All features | 0.894 | 0.940 | 0.884 | 0.872 | 0.886
No peakiness | 0.815 | 0.836 | 0.828 | 0.765 | 0.824
No movie-embedding | 0.784 | 0.869 | 0.786 | 0.722 | 0.752

TABLE VII: Precision of combined audio-visual model

##  VII Evaluation – Macro-level

Results shown here are based on the visual arcs, which we consider to be the
primary medium. We could in theory use arcs constructed from the combined
audio-visual model described in Section VI-C. Experiments clustering the
combined arcs, however, produce largely indistinct clusters. This is a result
of a) the sparsity of ground truth data covering the entire audio-visual space
found in movies (e.g. no clips were extracted from moments that were neutral
in both the audio and visual arcs), and b) simply the small size of the
dataset, leading to inaccurate combining of audio and visual.

###  VII-A Cluster results

Figure 5 shows the results of the elbow method, which plots the number of
clusters $k$ against the within cluster distance (WCD). We can see possible
‘elbows’ at $k$=5 and $k$=9. We briefly note that a k-means approach produced
indistinct clusters and no discernible decrease in the WCD.

(a) Films Corpora

(b) Shorts Corpora

Figure 5: Elbow plots for k-medoid clustering

Figure 6 shows one example clustering, representing five typical emotional
arcs. Note that the steep inclines and declines at the start and end are
artifacts of opening scenes and credits. Compared to the Films Corpora,
typical arcs in the Shorts Copora tend to be less complex, but also more
extreme (e.g. the yellow arc that ends on a steady decline).

Figure 6: Clustering on Shorts Corpora with $k$ = 5, $w$ = 0.1

###  VII-B Engagement analysis

We can now return to the question of whether a video’s emotional arcs affects
the degree of viewer engagement.We performed a small experiment on our Shorts
Corpora by using a) metadata features, and b) categorical cluster assignment
features (which family of visual arcs does this movie belong to) as inputs to
a regression model that predicts the number of comments a video received on
Vimeo.

Nine models are created – one for each value of $k$ in [2,10]. Not
surprisingly, the duration and year are often stat-sig predictors. However,
three arcs are also stat-sig, each positively correlated with the number of
comments. The first stat-sig arc, the yellow arc shown in Figure 6, fits the
“Icarus” shape (rise-fall). The second ($k$ = 8) and third ($k$ = 10) arcs are
characterized by a large peak near the end, with the former having some
incline before the peak and the latter flat before the peak. In other words,
they end with a bang. The results for $k$ = 8 are shown in Figure 7 and Table
IX.

Figure 7: TABLE IX. Engagement analysis for $k=8$ clustering. P-values less
than 0.05 are bolded; less than 0.1 are italicized. Statistically significant
predictive arc is shown.

##  VIII Concluding Remarks

We first developed methods for constructing and finding families of arcs. The
crowdsourced annotation data prompts a number of possibilities for future
work, such as dialogue-based arcs and plot-based sentiment modeling.

We were also able to show the predictive power of emotional arcs on a small
subset of online Vimeo shorts. While intriguing, this was performed a) using
only the visual arcs, and b) against a relatively simple metric. More data for
the combined audio-visual model should generate more accurate arcs. It would
also be interesting to see how, if at all, emotional features affect how
videos propagate through social media sites like Twitter and Reddit.

## References

  * [1] Kurt Vonnegut on the Shapes of Stories.  https://www.youtube.com/watch?v=oP3c1h8v2ZQ Accessed: 2017-04-27.
  * [2] Dramatica: the next chapter in story development.  http://dramatica.com.  Accessed: 2017-04-27.
  * [3] Katherine L Milkman and Jonah Berger.  The science of sharing and the sharing of science.  Proceedings of the National Academy of Sciences, 111(Supplement 4):13642–13649, 2014.
  * [4] Jonah Berger and Katherine L Milkman.  What makes online content viral?  Journal of marketing research, 49(2):192–205, 2012.
  * [5] Amid Amidi.  The Art of Pixar: 25th Anniversary: The Complete Color Scripts and Select Art from 25 Years of Animation.  Chronicle Books, 2015.
  * [6] Thierry Bertin-Mahieux, Daniel PW Ellis, Brian Whitman, and Paul Lamere.  The million song dataset.  In ISMIR, volume 2, page 10, 2011.
  * [7] Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, and Shih-Fu Chang.  Large-scale visual sentiment ontology and detectors using adjective noun pairs.  In Proceedings of the 21st ACM international conference on Multimedia, pages 223–232. ACM, 2013.
  * [8] Brian Boyd.  On the origin of stories.  Harvard University Press, 2009.
  * [9] Keunwoo Choi, George Fazekas, and Mark Sandler.  Automatic tagging using deep convolutional neural networks.  arXiv preprint arXiv:1606.00298, 2016.
  * [10] Hui Ding, Goce Trajcevski, Peter Scheuermann, Xiaoyue Wang, and Eamonn Keogh.  Querying and mining of time series data: experimental comparison of representations and distance measures.  Proceedings of the VLDB Endowment, 1(2):1542–1552, 2008.
  * [11] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In International Conference on Machine Learning, pages 1050–1059, 2016.
  * [12] Jonathan Gottschall.  The storytelling animal: How stories make us human.  Houghton Mifflin Harcourt, 2012.
  * [13] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al.  Deep speech: Scaling up end-to-end speech recognition.  arXiv preprint arXiv:1412.5567, 2014.
  * [14] Cisco Visual Networking Index.  Cisco visual networking index: Forecast and methodology, 2010-2015.  White Paper, CISCO Systems Inc, 9, 2011.
  * [15] Leonard Kaufman.  Clustering by means of medoids.  Statistical data analysis based on the L1-norm and related methods, 1987.
  * [16] Eamonn Keogh.  Exact indexing of dynamic time warping.  In Proceedings of the 28th international conference on Very Large Data Bases, pages 406–417. VLDB Endowment, 2002.
  * [17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [18] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [19] Jennifer S Lerner, Ye Li, Piercarlo Valdesolo, and Karim S Kassam.  Emotion and decision making.  Annual Review of Psychology, 66:799–823, 2015.
  * [20] James MacQueen et al.  Some methods for classification and analysis of multivariate observations.  In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pages 281–297. Oakland, CA, USA., 1967\.
  * [21] Douglas S Massey.  A brief history of human society: The origin and role of emotion in social life.  American Sociological Review, 67(1):1, 2002.
  * [22] David Bamman, Brendan O’Connor, and Noah A Smith.  Learning latent personas of film characters.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), page 352, 2014.
  * [23] Alexandru Niculescu-Mizil and Rich Caruana.  Predicting good probabilities with supervised learning.  In Proceedings of the 22nd international conference on Machine learning, pages 625–632. ACM, 2005.
  * [24] Bo Pang, Lillian Lee, et al.  Opinion mining and sentiment analysis.  Foundations and Trends® in Information Retrieval, 2(1–2):1–135, 2008.
  * [25] Chotirat Ann Ratanamahatana and Eamonn Keogh.  Everything you know about dynamic time warping is wrong.  In Third Workshop on Mining Temporal and Sequential Data. Citeseer, 2004.
  * [26] Andrew J Reagan, Lewis Mitchell, Dilan Kiley, Christopher M Danforth, and Peter Sheridan Dodds.  The emotional arcs of stories are dominated by six basic shapes.  EPJ Data Science, 5(1):31, 2016.
  * [27] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler.  Movieqa: Understanding stories in movies through question-answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4631–4640, 2016.
  * [28] Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai, and Arvid Kappas.  Sentiment strength detection in short informal text.  Journal of the American Society for Information Science and Technology, 61(12):2544–2558, 2010.
  * [29] Atousa Torabi, Christopher Pal, Hugo Larochelle, and Aaron Courville.  Using descriptive video services to create a large data source for video annotation research.  arXiv preprint arXiv:1503.01070, 2015.
  * [30] Roger S Ulrich.  Visual landscapes and psychological well-being.  Landscape research, 4(1):17–23, 1979.


# Towards Stirling engine using an optically confined particle subjected to
asymmetric temperature profile

Gokul Nalupurackal  Department of Physics, Quantum Centres in Diamond and
Emergent Materials (QuCenDiEM)-group, Micro Nano and Bio-Fluidics
(MNBF)-Group, IIT Madras, Chennai 600036, India  Muruga Lokesh  Department of
Physics, Quantum Centres in Diamond and Emergent Materials (QuCenDiEM)-group,
Micro Nano and Bio-Fluidics (MNBF)-Group, IIT Madras, Chennai 600036, India
Sarangi Suresh  Department of Physics, Quantum Centres in Diamond and Emergent
Materials (QuCenDiEM)-group, Micro Nano and Bio-Fluidics (MNBF)-Group, IIT
Madras, Chennai 600036, India  Srestha Roy  Department of Physics, Quantum
Centres in Diamond and Emergent Materials (QuCenDiEM)-group, Micro Nano and
Bio-Fluidics (MNBF)-Group, IIT Madras, Chennai 600036, India  Snigdhadev
Chakraborty  Department of Physics, Quantum Centres in Diamond and Emergent
Materials (QuCenDiEM)-group, Micro Nano and Bio-Fluidics (MNBF)-Group, IIT
Madras, Chennai 600036, India  Jayesh Goswami  Department of Physics, Quantum
Centres in Diamond and Emergent Materials (QuCenDiEM)-group, Micro Nano and
Bio-Fluidics (MNBF)-Group, IIT Madras, Chennai 600036, India  Gunaseelan M
Department of Physics, Quantum Centres in Diamond and Emergent Materials
(QuCenDiEM)-group, Micro Nano and Bio-Fluidics (MNBF)-Group, IIT Madras,
Chennai 600036, India  Arnab Pal<EMAIL_ADDRESS>The Institute of
Mathematical Sciences, CIT Campus, Taramani, Chennai 600113, India & Homi
Bhabha National Institute, Training School Complex, Anushakti Nagar, Mumbai
400094, India  Basudev Roy<EMAIL_ADDRESS>Department of Physics, Quantum
Centres in Diamond and Emergent Materials (QuCenDiEM)-group, Micro Nano and
Bio-Fluidics (MNBF)-Group, IIT Madras, Chennai 600036, India

###### Abstract

The realization of microscopic heat engines has gained a surge of research
interest in statistical physics, soft matter, and biological physics. A
typical microscopic heat engine employs a colloidal particle trapped in a
confining potential, which is modulated in time to mimic the cycle operations.
Here, we use a lanthanide-doped upconverting particle (UCP) suspended in a
passive aqueous bath, which is highly absorptive at 975 nm and converts NIR
photons to visible, as the working substance of the engine. When a single UCP
is optically trapped with a 975 nm laser, it behaves like an active particle
by executing motion subjected to an asymmetric temperature profile along the
direction of propagation of the laser. The strong absorption of 975 nm light
by the particle introduces a temperature gradient and results in significant
thermophoretic diffusion along the temperature gradient. However, the activity
of the particle vanishes when the trapping wavelength is switched to 1064 nm.
We carefully regulate the wavelength-dependent activity of the particle to
engineer all four cycles of a Stirling engine by using a combination of 1064
nm and 975 nm wavelengths. Since the motion of the particle is stochastic, the
work done on the particle due to the stiffness modulation per cycle is random.
We provide statistical estimation for this work averaged over 5 cycles which
can be extended towards several cycles to make a Stirling engine. Our
experiment proposes a robust set-up to systematically harness temperature
which is a crucial factor behind building microscopic engines.

###### pacs:

Valid PACS appear here

##  I Introduction

Microscopic heat engines are well-known as prototypical tools for the
investigation of energy transduction in fluctuation-dominated regimes. The
core studies on this specific area are vividly about the realizations of heat
engines that operate between different thermal reservoirs. These engines,
however, operate in non-equilibrium conditions such that stochastic
thermodynamics provides a useful description in their quantification Horowitz
and Parrondo (2012); Hanggi and Marchesoni (2009); Seifert (2012a); Jarzynski
(2011); Polettini  _et al._ (2015); Verley  _et al._ (2014); Kumari  _et al._
(2020); Seifert (2012b); Sekimoto (1998a); Martínez  _et al._ (2017). This is
indeed crucial since significant research has been done on artificial Browne
and Feringa (2006); Balzani  _et al._ (2000) and biological microscopic
engines and motors Howard and Clark (2002); Ramaiya  _et al._ (2017) with
exciting applications. Recent studies include working principles and
operational strategies of many such microscopic non-equilibrium systems namely
the Brownian Stirling engine Horowitz and Parrondo (2012); Blickle and
Bechinger (2012), active Stirling engine Krishnamurthy  _et al._ (2016);
Holubec  _et al._ (2020), Brownian Carnot’s engine Martinez  _et al._ (2016);
Frim and DeWeese (2022) and information engine Admon  _et al._ (2018).
Generically, a prototypical set-up of such studies includes a micro-particle
(which takes the role of the working substance) confined in a time-dependent
optical potential well and coupled periodically to different external heat or
active baths that enable the cycle to run through all four stages Blickle and
Bechinger (2012); Krishnamurthy  _et al._ (2016); Pietzonka  _et al._ (2019).
In this manuscript, we use a similar set-up to engineer a Stirling engine,
however, with the utilization of upconverting particles as working substances
instead of colloids.

Optical trapping of upconverting particles Rodriguez-Sevilla  _et al._ (2016);
Haro-Gonzalez  _et al._ (2013); Kumar  _et al._ (2021) has gained recent
interest as their potential varies from imaging to therapyJaque  _et al._
(2014); Wilhelm (2017); Fischer  _et al._ (2011); Carlos  _et al._ (2009).
Generation, control and detection of out-of-plane rotational motionsLokesh
_et al._ (2021a, b); Vaippully  _et al._ (2021, 2020) of single UCP along with
the translations in optical tweezers Lokesh  _et al._ (2022); Kumar  _et al._
(2020a) to explore their micro-rheological and mechanical potentials have also
been studied. Recently, Kumar et al. reported a new system of an upconverting
particle optically trapped at the absorption resonance of 975 nm, generating a
temperature gradient and self-propulsionKumar  _et al._ (2020b). It has been
shown that in such translations, the overall motion can be determined by the
propulsion speed, the rotational and translational Brownian motion which leads
to a super-diffusive mean square displacement Schachoff  _et al._ (2015);
Suresh  _et al._ (2022) due to this “activity” of the particle Schachoff  _et
al._ (2015). The activity vanishes when the same particle is trapped at a non-
absorptive wavelength, 1064 nm. Both the 1064 nm and 975 nm laser beams
contribute to trap stiffness of the system whereas 975 nm has an additional
effect of generating a temperature gradient across the particle causing the
“activity”. We use this very facet of the working principle to develop a
microscopic Stirling engine.

A Stirling engine functionally consists of closed cycles of isochoric and
isothermal processes. At the microscale, the constant-volume part of the
isochoric process is taken care of by the optical confinement, which can
further be coupled to periodically varying heat reservoirs to execute the
heating and cooling of the same particle. Similarly, the expansion and
compression protocols are done by varying the trap stiffness in a fixed heat
bath – reminiscent of the isothermal phase. Notably, the pump wavelength of
NaYF4:Yb3+,Er3+ upconverting particles (975 nm) is employed as one of the
trapping wavelengths to generate absorption-enhanced optical heating. Thus,
the requirement of external heat baths to couple with the trapped particle can
be completely abolished. The isothermal parts of the cycle are executed by
delivering a constant power of 975 nm laser to the optical trap and varying
the 1064 nm power accordingly.

On the other hand, the isochoric processes are carried out by properly
synchronizing both the laser powers so that the total intensity of light and
thus the trap stiffness remains constant. Simultaneously, heating and cooling
parts are performed by the suitable variation of 975 nm laser power. A
schematic representation of one complete cycle of the engine is depicted in
Fig. 2. In what follows, we start by mentioning the basic principles of the
engine followed by a detailed description of the experimental setup, analysis,
and results.

##  II Theory

###  II.1 An optically trapped upconverting particle subjected to asymmetric
temperature profile

NaYF4:Yb3+,Er3+ upconverting particles (UCPs) preferentially absorb NIR
photons at 975 nm and emit higher energy photons in the visible band with the
efficiency of about 2 % Wen  _et al._ (2018). This strong absorption leads to
the optical heating of particles by photon-to-phonon conversion in their
lattices. When we look at a single UCP (scanning electron microscopy images in
Fig. 3(a-b)), confined in optical tweezers at the pump wavelength, the size of
such particle is much larger than the beam waist of the focused laser beam
while it gets trapped in a side-on configuration as shown in the inset of Fig.
3(c) Kumar  _et al._ (2022). As a collective result of all these effects, we
observe a local temperature gradient across the UCP along the +z direction. It
is also observed that the particle predominantly emits in a back-scattered
direction (-z - direction, see the schematic in Fig. 4) and self-propels in
the opposite direction due to the temperature gradient Kumar  _et al._
(2020b). This absorption-induced motion can be thought to be an “active” self-
thermophoretic motion inside the optical trap with the tendency of the
particle to move from hotter to colder region Braibanti  _et al._ (2008);
Nalupurackal  _et al._ (2022). This will cause the particle to have a complex
flow profile with a propulsion velocity, $v$ which is given by Wurger (2010);
Piazza (2008)

$\displaystyle v=-D_{T}\nabla T.$ (1)

Here, $D_{T}$ is given as $D_{T}$ = S${}_{T}D$, where $D$ is the thermal
diffusion coefficient of the particle, the sign of which determines the
thermophoretic direction of the particle. The ST is the Soret coefficient.

Figure 1: A typical calibrated displacement time series of a UCP along the
z-axis, trapped with both 975 nm and 1064 nm laser is shown in the figure. The
time series obtained from the isothermal compression phase of the Stirling
engine.  Figure 2: Schematic representation of one complete cycle of an
engineered microscopic Stirling engine using upconverting particles (see
section IV for details)

The thermophoretic motion generates a velocity $v_{0}$ in $z$ direction. In
our experimental set-up, the particle moves around $40$ nm inside a trap of
$1.5$ microns as shown in Fig. 1. Hence, one can safely assume that in the
small displacement limit the velocity remains constant. Within this spirit, we
therefore can make an assumption that in fluctuation-dominated regimes, the
propulsion velocity is an active velocity along the axis of thermophoresis,
and thus its information can be extracted from the mean square displacement
(MSD) of an “active” particle Howse  _et al._ (2007); Bechinger  _et al._
(2016); Kumar  _et al._ (2020a); Schachoff  _et al._ (2015); Jiang  _et al._
(2010). For an ‘active’ particle moving with velocity $v_{0}$ in a random
direction in the x-z plane (later we assume that $v_{0}$ is almost aligned
along the $z$-axis, but the generic theory works for any orientation) in a
harmonic trap, one can then write the following Langevin equation Bechinger
_et al._ (2016); Kumar  _et al._ (2020b)

$\displaystyle\frac{dx}{dt}+\frac{kx}{\gamma}$
$\displaystyle=v_{0}\sin{\theta}+\frac{\eta_{x}}{\gamma}$ (2)
$\displaystyle\frac{dz}{dt}+\frac{kz}{\gamma}$
$\displaystyle=v_{0}\cos{\theta}+\frac{\eta_{z}}{\gamma}$ (3)
$\displaystyle\frac{d\theta}{dt}$ $\displaystyle=\sqrt{D_{r}}\eta_{\theta},$
(4)

where $k$ is the harmonic trap stiffness, $\gamma$ is the drag coefficient,
and $D_{r}$ is the rotational diffusion constant. The thermal noise
$\eta_{x,y}$ is Gaussian with the following statistical properties
$\langle\eta_{i}\rangle=0$ and
$\langle\eta_{i}(t)\eta_{j}(t^{\prime})\rangle=2\gamma
k_{B}T\delta_{ij}\delta(t-t^{\prime})$. As mentioned above, the activity is
captured by the orientation $\theta$ in the x-z plane and the self-propulsion
velocity $v_{0}$ and $\eta_{\theta}$ is a Gaussian white noise with unit
strength. From the property of $\eta_{\theta}$, it is easy to see that the
probability density function of $\theta$ is Gaussian i.e., $P(\theta)\propto
e^{-\theta^{2}/4D_{r}t}$. From this, one can easily compute the spatial
moments (see Howse  _et al._ (2007); Solon  _et al._ (2015); Bechinger  _et
al._ (2016); Kumar  _et al._ (2020b)) for $x,y$ and consecutively the radial
MSD. Skipping details from Howse  _et al._ (2007); Bechinger  _et al._ (2016);
Solon  _et al._ (2015); Kumar  _et al._ (2020b), one arrives at the following
expression for radial MSD

$\displaystyle\langle\Delta r^{2}(\Delta
t)\rangle=\frac{2k_{b}T}{k}\left(1-e^{-\frac{2k}{\gamma}\Delta
t}\right)+v_{0}^{2}\left(\frac{1-e^{-\frac{2k}{\gamma}\Delta
t}}{\frac{2k}{\gamma}}\right)\left(\frac{4D_{r}}{4D_{r}^{2}-\left(\frac{k}{\gamma}\right)^{2}}\right)+v_{0}^{2}\frac{1+e^{-2k\Delta
t/\gamma}-2e^{-((k/\gamma)+2D_{r})\Delta
t}}{\left(\frac{k}{\gamma}\right)^{2}-2D_{r}^{2}}.$ (5)

The expression for MSD above can be simplified further in the limit of small
trap stiffness values with $D_{r}=1/2\tau_{r}$, where $\tau_{r}$ is the
rotational relaxation time of the trapped UCP Schachoff  _et al._ (2015). This
results in

$\langle\Delta r^{2}(\Delta t)\rangle=4D\Delta
t+\frac{v_{0}^{2}\tau_{r}^{2}}{2}\left(\frac{2\Delta
t}{\tau_{r}}+e^{-\frac{2}{\tau_{r}}\Delta t}-1\right).$ (6)

For practical purposes, it can be assumed that the particle is oriented in the
x-z plane with a small angle ($\theta$) along the z-axis so that
$\cos{\theta}\approx 1$ Schachoff  _et al._ (2015). Henceforth, it can be
inferred that $v_{0}$ is the thermophoretic propulsion velocity along the
z-direction, which is the axial velocity of the particle. When the laser trap
is switched to a wavelength of 1064 nm, the particle loses its active property
and behaves like an ordinary Brownian particle in an optical trap, thus only
experiencing the linear diffusive term of Eq. 6.

Further, the MSD and the velocity are calibrated with the optical trap by
assuming it to be a harmonic trap. The fundamental parameters such as trap
stiffness, diffusion coefficient and displacement sensitivity can be extracted
from the power spectral density (PSD) of the trapped particle (see Fig. 5(a)).
The MSD v/s lag-time plot for the particle is shown in Fig. 5(b) at active
(975 nm) and non-active (1064 nm) wavelengths. Evidently, the active velocity
which is extracted from the MSD curve gives the information about the local
temperature $(T)$ profile around the particle (from Eq. 1).

Figure 3: (a-b) The FE-SEM images of UCP are shown to confirm their hexagonal
geometry. In (c), the emission spectra of one such UCP trapped with 975 nm
laser is shown. Inset shows (i) the normalized NIR absorption spectra of
NaYF4:Yb3+,Er3+ UCPs and (ii) the optically trapped configuration of a single
particle with 975 nm laser, where the particle tends to align along its side-
on axis. Figure 4: Schematic diagram of the experimental set-up. Both the
lasers are tightly focused at the sample plane using 100X objective and the
traps are made overlapping to each other. The LED situated at the top
illuminates the sample chamber through the dichroic mirror - 2. BS - beam
splitter, PBS - polarizing beam splitter, HWP - half-wave plate, PD -
photodiode. Figure 5:  (a) The calibrated power spectral density of
perpendicular displacements (along the z-axis) of the trapped particle with
1064 nm laser is plotted. The displacement sensitivity is found to be $\beta$
= 3.25 $\pm$ 0.13 µm/V. (b) shows the MSD of the same particle trapped with
975 nm laser. It is fitted to Eq. 6 and reaches a plateau at large time scales
due to optical confinement. In (b), the mean square displacement of the
particle along the z-axis, in the x-z plane as a function of time, is plotted
and overlaid with the noise floors of both the lasers at the same power. The
particle shows pure diffusive behaviour with 1064 nm laser, whereas the same
plot shows a super-diffusivity when it is trapped with 975 nm.The curve is
fitted with Eq. 6 to find the axial velocity and relaxation time, gives $v$ =
36.1 $\pm$ 6.3 nm/s and $\tau_{r}$ = 1.21 $\pm$ 0.32 s.

A typical power spectral density extracted from the time series of an
optically trapped UCP fitted to a Lorentzian is shown in Fig. 5(a). Generally,
the PSD of Brownian motion along any axis of an optically trapped particle is
given by,

$\displaystyle PSD=\frac{A}{(2\pi f)^{2}+B},$ (7)

Where $A$ is the amplitude in detector units and $B$ is the square of corner
frequency ($f_{c}$). The trap stiffness ($k$) can be written as Schaffer  _et
al._ (2007),

$\displaystyle k=2\pi\gamma\sqrt{B}.$ (8)

The calibration factor, $\beta$ for the trap, which generally has a unit of
$nm\,/\,V$ is given bySchaffer  _et al._ (2007),

$\displaystyle\beta=\sqrt{\frac{k_{b}T}{\gamma A}},$ (9)

where $k_{b}$ is the Boltzmann constant and $\gamma$ is the drag coefficient
of the particle. The hexagonal disc-shaped UCP has a diameter of 5 µm and
thickness of 1 µm - thus can be approximated geometrically as an oblate
spheroid Kumar  _et al._ (2020b). Consequently, the drag coefficient becomes
$\gamma=2/3\,\gamma_{0}$, where $\gamma_{0}$ is the drag coefficient of a
sphere with the same radius as the semimajor axis as the spheroid. The UCPs
are placed away from any other surfaces in water.

###  II.2 Stochastic work done in a complete Stirling cycle

The first principles of stochastic thermodynamics are useful to infer the
thermodynamic quantities from the microscopic dynamics e.g., work, heat, or
entropy production will be fluctuating along any trajectory, say ${x}(t)$ so
that one needs to average over many realizations of the process to make a
statistical estimation. In this case, work is done since we modulate the
potential $U(x,k)$ by changing the stiffness $k(t)$. Following Sekimoto
(1998b); Jarzynski (1997); Krishnamurthy  _et al._ (2016), we can then write
the microscopic thermodynamical work done in $dt$ time interval as
$dW(t)=~{}\frac{\partial U(\mathcal{C},k)}{\partial k}dk(t)$. For an optically
trapped particle, these expressions can be scaled using the trap stiffness of
the system. Here, $\mathcal{C}$ is the generalised coordinate (could be
position, velocity etc.) of the system. And upon integration, the time-
dependent work in the system for a finite time $\mathcal{T}$ is given by
Blickle and Bechinger (2012); Sekimoto (1998b)

$\displaystyle W_{T}=\int_{0}^{\mathcal{T}}~{}dt~{}\frac{\partial
U(\mathcal{C},k)}{\partial k}\dot{k}(t).$ (10)

It should be noted that this work is due to the time modulation of the trap
even in the presence of activity Krishnamurthy  _et al._ (2016); Dabelow  _et
al._ (2019) and should be distinguished from the work done due to self-
propulsion Shankar and Marchetti (2018). Work statistics encode important
features of an out-of-equilibrium thermodynamic process where tremendous
progress has been achieved to measure work fluctuations in microscopic soft
and biological systems Seifert (2012a); Ritort (2008); Wen  _et al._ (2007);
Ciliberto (2017); Jarzynski (2011); Seifert (2008).

Here, the intensity of both lasers is changed in a linear fashion; over the
course of 10 seconds for each process, to ensure the slow variation in
dynamics of the particle within the optical trap. Since we are making our
observations at much larger time scales than the microscopic timescales
corresponding to a particle in an over-damped medium, the trap stiffness will
also be directly proportional to the temporal change in the intensity gradient
at the time of observation, hence varying linearly over time. Due to the sign
conventions, the work is positive (negative) when the energy is supplied
(extracted) to (from) the trapped particle system.

Since the upconverting particles have a preferential absorption at 975 nm, we
observe a significant temperature gradient across the particles, accounting
for its axial motion Kumar  _et al._ (2020b); Schachoff  _et al._ (2015). In
turn, the temperature of the immediate water layers to the particle also
increases and collectively forms a thermal bath. The rise in temperature in
the neighbourhood of the particle is accounted for by the optical heating,
indicated by its absorption cross-section. It is generally written as Shan
_et al._ (2021); Govorov  _et al._ (2006),

$\displaystyle\Delta T=\frac{\sigma I}{4\pi rK},$ (11)

where $\sigma$, $I$, $r$, and $K$ respectively are the absorption cross-
section, incident laser intensity, the radius of the particle, and thermal
conductivity of the surrounding medium. The value of $\sigma$ is determined to
be 3.61 $\times$ 10-11 cm2 per particle using diffuse reflectance spectroscopy
(DRS) for a given number density ($N=3.55\times 10^{15}$ cm-3, see the
normalized absorption spectra at the inset of Fig. 3(c)). For a power density
of 1.60 $\times$ 106 W/m2 at the beam waist, the maximum temperature rise
($\Delta T$) of the immediate surroundings of the particle is calculated to be
approximately 2.75 K.

##  III Experimental system

###  III.1 Optical tweezers setup

The experiments are performed using an optical tweezers setup (OTKB/M,
Thorlabs) configured as an inverted microscope. The objective lens of the
setup is a 100x, 1.3 NA, oil immersion type, obtained from Olympus. While the
10X condenser lens is an air immersion type from Nikon with 0.25 NA. Two diode
lasers of wavelengths 975 nm and 1064 nm, with the same beam-polarization, are
employed to create the optical traps at the sample chamber by tightly focusing
them with the objective. The laser, namely laser-1 (975 nm) is a butterfly-
type laser from Thorlabs, USA with a maximum output power of 300 mW, while
laser-2 (1064 nm) is from Lasever, China, and has a maximum output power of
1.5 W. The two optical traps are overlapped with each other. The forward
scattered light from the particle and the unscattered light is directed
towards the detection systems using the condenser lens and top dichroic mirror
as shown in Fig. 4. The same dichroic mirror allows the transmission of white
light from the top LED to illuminate the sample chamber. This white light is
collected by a CMOS camera (Thorlabs) through the bottom dichroic mirror. The
combination of a half-wave plate (1064 nm), a polarizing beam splitter (PBS),
a band-pass filter (1064 $\pm$ 10 nm), and a pair of Si-photodiodes (PD-1 and
PD-2 (DET100A2, Thorlabs), located at the orthogonal ports of PBS) constitute
the detection system. Here, most of the forward scattered light of 975 nm
laser is made incident on PD-1, while that of 1064 nm laser is directed to
PD-2 through a bandpass filter.

###  III.2 Generation of a complete Stirling cycle

The complete cycle of the microscopic Stirling engine is generated by the
careful manoeuvring of intensities of laser-1 and laser-2 over a finite time,
$\mathcal{T}$. Both lasers are voltage-controlled current devices and their
output powers can be driven by external voltages. We extend the input of the
lasers to a NI-DAQ-based voltage source (NI USB-6009, National Instruments,
USA). The output power of both lasers is calibrated against the driving
voltage from the DAQ using LabView, to select the range of operation. Further,
the lasers are simultaneously programmed to generate all four processes of the
Stirling engine (see Fig. 6(a)). The duration of a complete cycle is set to be
40 seconds, divided into 40 independent bins of 1-second duration for the
analysis. The individual processes are described in section IV.

###  III.3 Preparation of upconverting particles

Upconverting particles (NaYF4:Yb3+,Er3+ UCPs) are synthesized through modified
hydro-thermal method Kumar  _et al._ (2020b, a). Initially, 1.26 g of yttrium
nitrate (Y(NO3)3) and 1.23 g of sodium citrate (Na3C6H5O7) are dissolved in 14
mL of D.I. water and stirred for 15 minutes. At the same time, 0.38 g of
ytterbium nitrate (Yb(NO3)3) and 0.037 g of erbium nitrate (Er(NO3)3) are
added and stirred in 21 mL of D.I. water in a separate beaker. The two
solutions are mixed and stirred well until a pale white solution is formed.
The addition of 1.411 g of sodium fluoride (NaF) in 67 mL D.I. water to the
above mixture results in a clear solution, which is then transferred into an
autoclave reactor and heated at 200∘C for 12 hours. The white-coloured powders
are formed after the reaction and then collected after washing with
ethanol/water, followed by drying at 100∘ C for about 12 hours. The hexagonal
geometry of the particles with a diagonal length of 5µm is confirmed from
their FE-SEM images as shown in Fig. 3(a-b).

The powder containing particles is mixed in D.I water and 20 µL of which is
poured on a glass slide (Blue star, English glass, (Blue Star, 75 mm $\times$
25 mm $\times$ 1.1 mm). This aqueous solution of UCPs is then covered using a
coverslip (Blue Star, number 1 size, English glass), which makes the sample
chamber. One such UCP is trapped simultaneously with 975 nm laser and 1064 nm
laser in the bulk.

##  IV Results and discussions

###  IV.1 Simultaneous determination of axial velocity and trap stiffness

Fig. 6(a) shows the axial displacement (z-axis) time series of an optically
trapped UCP, obtained simultaneously from the scattered intensities of laser-1
(975 nm) and laser-2 (1064 nm). We use the data corresponding to laser-1 for
the purpose of calibration and analysis.

We design one complete cycle in such a way that each cycle process is 10
seconds long and is further divided into 10 individual bins with a duration of
1 second. Given the data points we collect each second (10 kHz), this
requirement is adequate to execute the fast Fourier transform (FFT) and
retrieve the trap parameters. The MSD of the particle in each time bin is
plotted against the lag-time and fitted to Eq. 6 to obtain the axial active
velocity ($v$) and relaxation time, as shown in Fig. 5(b). Similarly, the PSD
of the same bin is fitted into a Lorentzian using Eq. 7 to obtain the trap
stiffness. In this manner, the axial velocity of the trapped UCP and trap
stiffness is obtained and calibrated throughout one complete cycle and plotted
as a function of cycle lag-time in Fig. 6(b).

It may be realised that both the beams 975 nm and 1064 nm laser are sampling
the same particle motion. The same information may be obtained by studying
either time series. Hence we use the 975 nm laser time series. Indeed only the
calibration factors of the time series of 975 nm laser and the 1064 nm laser
would be different for the two cases. The trap stiffnesses have been computed
by ascertaining the power spectral density of each 1-second segment of the 975
nm time series, which has been fitted with routine Lorentzian fits shown in
Fig. 5(b). The method to determine trap stiffness k and the calibration factor
has been given in Schaffer  _et al._ (2007).

Figure 6:  (a) The displacement time series of the trapped particle along the
z-axis, in the x-z plane, obtained using both 1064 nm and 975 nm lasers are
shown in the figure. We set the duration of a single cycle to be 40 seconds,
divided into ten equal time bins corresponding to each process of the Stirling
cycle. In (b), variations of trap stiffness ($k$) and axial velocity of the
trapped particle, as obtained from (a), are shown as a function of time over
the four stages of a complete cycle.

We find that our mean square displacement (MSD) of the z-axis motion fits well
with the description given in Eq. (6). Indeed it averages out over the 2
rotational angles. Still, one of the out-of-plane (we call pitch angle in the
nomenclature of the airlines) angles is confined by the polarization direction
in the optical tweezers themselves. As seen in Rodriguez-Sevilla  _et al._
(2016), the hexagonal-shaped particle automatically aligns side-on along the
direction of linear polarization of the optical tweezers beam. Given that the
orientation is locked by the tweezer’s beam, a planar description of the
system works well. We also assume that the particle moves very little while
executing active motion, due to which the extent of anisotropic heating
remains constant.

###  IV.2 Analysis of a complete Stirling cycle

The velocity of the trapped UCP and corresponding trap stiffness is plotted as
a function of cycle lag-time in Fig. 6(b). The complete cycle is designed so
that each process (leg of a cycle) is 10 seconds long. The stochastic work
done in five such cycles is estimated from the displacement time series of the
particle obtained from the scattered intensities of 975 nm laser and plotted
in Fig. 7(a). Here, we run the cycles in an anti-clockwise direction as shown
in Fig. 7(b) and determine $k$ and $v$ values for one such cycle. The
calibrated values of active velocity and trap stiffness yield the $k-v$
diagram of the engine over a full period (see Fig. 7(b)). This plot is
analogous to the V-T diagram of a macroscopic Stirling engine.

####  IV.2.1 Isothermal compression

We start the cycle by exploring the isothermal compression of its working
substance (here, the confinement volume of trapped UCP). UCP is trapped with
both lasers at the same power in the initial configuration. Then, the power of
975 nm laser is kept constant and that of 1064 nm is increased gradually over
time, so as to follow a linear trend. This process over the first ten seconds
of the cycle ensures that the optical heating of the trapped particle is
retained constant and the optical trap stiffness is increased linearly (from
$k_{L}$ to $k_{H}$). The observations are plotted for the first ten seconds in
Fig. 6(b). During this process, the temperature of the bath is constant
($T_{L}$) and the corresponding active velocity of the particle will be
constant ($v_{L}$) according to Eq. 1. This can be attributed to the constant
activity delivered by the constant laser power of 975 nm laser. On the other
hand, the trap stiffness is increased in a linear fashion with the increase of
1064 nm laser power, providing the system with a less final volume. We ensure
that the trap stiffness follows a linear modulation in time.

####  IV.2.2 Isochoric heating

Since isochoric heating of UCP demands a constant trap stiffness ($k_{H}$) and
a simultaneous increase in the heat bath’s temperature (from $T_{L}$ to
$T_{H}$), synchronized variation of both the laser powers is required. The
heating part is done by increasing 975 nm laser power, the additional
contribution of the same on trap stiffness is offset by decreasing 1064 nm
laser power. This process results in an increase in active velocity ($v_{L}$
to $v_{H}$) keeping the trap stiffness constant to make it isochoric. The trap
stiffness value is constant in time throughout this process which indicates
$\dot{k}=0$ (resulting in null work done). The results are depicted from 11 to
20 seconds in Fig. 6(a) and (b).

####  IV.2.3 Isothermal expansion

Isothermal expansion occurs once the temperature of the heat bath is
sufficiently high in a Stirling engine ($T_{H}$). In this process, the power
of 975 nm laser is kept constant to make it isothermal and that of 1064 nm
laser is decreased linearly in time to observe the decrease in trap stiffness
of the system. The active velocity of trapped UCP remains constant ($v_{H}$)
and the trap stiffness decreases to a minimum value ($k_{L}$) at the end of
the process (21 to 30 seconds of Fig. 6(b)).

####  IV.2.4 Isochoric cooling

The cycle is concluded by making the temperature of the bath its initial
value, $T_{L}$. Here, again the power of both 975 nm and 1064 nm lasers are
varied simultaneously but in an opposite fashion to that of isochoric heating.
The 975 nm laser power is decreased linearly to bring down the temperature and
active velocity of the particle to their initial values ($T_{L}$ and $v_{L}$).
And the corresponding reduction in trap stiffness is compensated by increasing
1064 nm laser power simultaneously, as depicted in the last 10 seconds of
Figs. 6 (a) and (b).

###  IV.3 Calibration of stochastic work done in multiple cycles

The work done is calculated using Eq. 10 and calibrated with the optical trap.
The trap stiffness $k$ of the system is set to vary linearly with time over a
period $0<t<\mathcal{T}$, the rate of which can be extracted from the slope of
isothermal processes (see Fig. 6(b)). Upon substituting the proper parameters
for calibration, the thermodynamical work in Eq. (10) can be written as

$\displaystyle W_{\mathcal{T}}=\frac{k_{b}T}{2\gamma
A}\int_{0}^{\mathcal{T}}~{}\dot{k}(t)\,z^{2}(t)~{}dt,$ (12)

where we have assumed $U=\frac{1}{2}k(t)z^{2}$ with $k(t)$ as a linear
function in time, which implies $\dot{k}(t)=$ C, a constant. We also get from
Fig. 6(b), C${}_{compression}=0.36\pm 0.05\,\,fN/nm\cdot s$,
C${}_{expansion}=0.43\pm 0.03\,\,fN/nm\cdot s$. From Eqs. (10) and (12), it is
obvious that the isochoric transitions do not contribute to the work done. The
work done is conventionally taken as positive during isothermal compression as
the energy is supplied to the system, thereby signing the work done during
expansion as positive.

Figure 7: a) The average work done over five cycles of isothermal processes of
the engine is shown as bold red and black curves – corresponding to the
compression and expansion stages. The individual stochastic work done over all
five cycles is separately overlaid on the same plot. Each cycle is taken
independently and referred to as zero work at the beginning of isothermal
compression and expansion to compare the magnitudes. The bold black and red
curves are the average work done over all the cycles. The linear evolution of
the work done is quantified by its slope (m). For the process of isothermal
compression, the aforementioned slope is calculated to be
$m_{\text{compression}}=1.91k_{b}T/s$, whereas in the expansion it becomes,
$m_{\text{expansion}}=2.75k_{b}T/s$, where Tamb = 298 K. The difference in the
magnitude of the slopes indicates the measure of useful work which could be
extracted from the heat engine. (b) This figure shows the $k-v$ diagram of a
Stirling cycle (shown in Fig. 7). This is equivalent to the $T-V$ diagram of a
macroscopic engine.

We determine the work done during compression and expansion for five
individual cycles as shown in Fig. 7(a). It can be seen that the work done is
stochastic, as the system is subjected to fluctuations. The work done along
each realization and their average is shown in the same figure. Each bin of
the displacement time series is calibrated and the square of which is
integrated altogether with respect to time to obtain the stochastic work done.
The obtained work is then normalised with the ambient temperature, Tamb = 298
K. The work done during isothermal expansion and compression evolve linearly
with the process time. The averaged stochastic work is then characterized by
the slope of these two lines, given by $\lvert
m_{\text{compression}}\rvert=1.91\,k_{b}T/s$ and $\lvert
m_{\text{expansion}}\rvert=2.75\,k_{b}T/s$. Here, by sign convention, we take
work done on the system (by the system) to be positive (negative). Hence, the
useful stochastic work that can be extracted from the engine is indicated by
the difference in the magnitudes of the slopes.

The optical tweezers is a non-conservative system, primarily because the
radiation pressure induces circulatory probability currents inside the
trapping potential Roichman  _et al._ (2008). However, this circulatory
velocity is proportional to the intensity of the light applied. We find in the
condition that when the total intensity is held constant (isochoric process)
but the 975 nm laser intensity is changed, the drift velocity changes as shown
in Fig. 6(b). This change in velocity cannot be explained by a constant
circulatory probability flow. Thus we believe that non-conservative effects
due to the optical tweezers is not causing the drift velocity.

Moreover, since the z displacement of the particle itself is directly
influenced by the heating, and only the active particle needs to be heated in
this case, the efficiency of the engine should be higher than trying to heat
up the entire surrounding bath. However, the exact calculation of the
efficiency of the system is beyond the scope of the present manuscript.

##  V Conclusions and perspectives

To summarize and conclude, we have attempted to engineer a microscopic,
closed-cycle Stirling engine with upconverting particles. The absorption and
optical heating properties of UCPs at 975 nm eliminate the requirement of
external heat baths for the engine to function. We introduce active velocity
of the particle ($v$) and trap stiffness ($k$) as the indicative parameters of
the engine and show that they follow a theoretically anticipated trend over a
complete cycle. We also demonstrate that the active velocity is a measure of
the temperature of the particle’s absorption-generated heat bath and hence we
report the $k-v$ diagram of the engine, which is analogous to the $V-T$
diagram of a macroscopic Stirling cycle. Finally, we determine the stochastic
work done over five such cycles by taking the notion of a single particle that
is subjected to a time-dependent optical trap and periodically coupled to
different heat baths.

The optical heating of a confined UCP can be controlled precisely and timely
without causing any plasmonic effects or other higher-order effects, which
constitutes one of the major advantages of using an absorbing particle as the
working substance. This external control over temperature ensures that the
microscopic, thermodynamic variables can be parameterized more accurately in
experiments. We thus believe that our work can surmount one of the major
challenges in colloidal heat engines, which is the temperature control in
isochoric processes of information heat engines. Furthermore, the particles we
use can also be employed to study the non-steady heat transfer between
different potential wells in optical tweezers systems under laser irradiation.
This renders a singular appeal of our experimental system with upconverting
particles to study various thermodynamical properties in the out-of-
equilibrium conditions – a subject that has become a topical interest in
statistical physics Ciliberto (2017). The extent of inertia at the micro
hydrodynamical systems is very less but it is actually applicable to all
systems at these length scales. We can however envisage systems where the
active particle is made to push against an elastic membrane which then does
work.

_Acknowledgements.—_ We thank the Indian Institute of Technology, Madras,
India for their seed and initiation grants. This work was also supported by
the DBT/Wellcome Trust India Alliance Fellowship IA/I/20/1/504900 awarded to
Basudev Roy. Arnab Pal gratefully acknowledges research support from the DST-
SERB Start-up Research Grant Number SRG/2022/000080 and the Department of
Atomic Energy, India.

## References

  * Horowitz and Parrondo (2012) J. M. Horowitz and J. M. Parrondo, Nat. Phys. 8, 108 (2012).
  * Hanggi and Marchesoni (2009) P. Hanggi and F. Marchesoni, Rev. Mod. Phys. 81, 387 (2009).
  * Seifert (2012a) U. Seifert, Reports on progress in physics 75, 126001 (2012a).
  * Jarzynski (2011) C. Jarzynski, Annu. Rev. Condens. Matter Phys. 2, 329 (2011).
  * Polettini  _et al._ (2015) M. Polettini, G. Verley, and M. Esposito, Physical review letters 114, 050601 (2015).
  * Verley  _et al._ (2014) G. Verley, T. Willaert, C. Van den Broeck, and M. Esposito, Physical Review E 90, 052145 (2014).
  * Kumari  _et al._ (2020) A. Kumari, P. S. Pal, A. Saha, and S. Lahiri, Phys. Rev. E 101, 032109 (2020).
  * Seifert (2012b) U. Seifert, Rep. Prog. Phys. 75, 126001 (2012b).
  * Sekimoto (1998a) K. Sekimoto, Prog. Theor. Phys. 130, 17 (1998a).
  * Martínez  _et al._ (2017) I. A. Martínez, É. Roldán, L. Dinis, and R. A. Rica, Soft matter 13, 22 (2017).
  * Browne and Feringa (2006) W. R. Browne and B. L. Feringa, Nat. Nanotechnol 1, 25 (2006).
  * Balzani  _et al._ (2000) V. Balzani, A. Credi, F. M. Raymo, and J. F. Stoddart, Angew. Chem. Int. Ed. 39, 3348 (2000).
  * Howard and Clark (2002) J. Howard and R. Clark, Appl. Mech. Rev. 55, B39 (2002).
  * Ramaiya  _et al._ (2017) A. Ramaiya, B. Roy, M. Bugiel, and E. Schaffer, Proc. Natl. Acad. Sci. U.S.A. 114, 10894 (2017).
  * Blickle and Bechinger (2012) V. Blickle and C. Bechinger, Nat. Phys. 8, 143 (2012).
  * Krishnamurthy  _et al._ (2016) S. Krishnamurthy, S. Ghosh, D. Chatterji, R. Ganapathy, and A. Sood, Nat. Phys. 12, 1134 (2016).
  * Holubec  _et al._ (2020) V. Holubec, S. Steffenoni, G. Falasco, and K. Kroy, Phys. Rev. Res. 2, 043262 (2020).
  * Martinez  _et al._ (2016) I. A. Martinez, E. Roldan, L. Dinis, D. Petrov, J. M. Parrondo, and R. A. Rica, Nat. Phys. 12, 67 (2016).
  * Frim and DeWeese (2022) A. G. Frim and M. R. DeWeese, Phys. Rev. E 105, L052103 (2022).
  * Admon  _et al._ (2018) T. Admon, S. Rahav, and Y. Roichman, Physical review letters 121, 180601 (2018).
  * Pietzonka  _et al._ (2019) P. Pietzonka, É. Fodor, C. Lohrmann, M. E. Cates, and U. Seifert, Physical Review X 9, 041032 (2019).
  * Rodriguez-Sevilla  _et al._ (2016) P. Rodriguez-Sevilla, Y. Zhang, N. de Sousa, M. I. Marques, F. Sanz-Rodriguez, D. Jaque, X. Liu, and P. Haro-Gonzalez, Nano. lett. 16, 8005 (2016).
  * Haro-Gonzalez  _et al._ (2013) P. Haro-Gonzalez, B. del Rosal, L. M. Maestro, E. M. Rodriguez, R. Naccache, J. Capobianco, K. Dholakia, J. G. Sole, and D. Jaque, Nanoscale 5, 12192 (2013).
  * Kumar  _et al._ (2021) S. Kumar, M. Gunaseelan, R. Vaippully, A. Banerjee, and B. Roy, Nanoscale 13, 11856 (2021).
  * Jaque  _et al._ (2014) D. Jaque, L. M. Maestro, B. del Rosal, P. Haro-Gonzalez, A. Benayas, J. Plaza, E. M. Rodriguez, and J. G. Sole, Nanoscale 6, 9494 (2014).
  * Wilhelm (2017) S. Wilhelm, ACS nano 11, 10644 (2017).
  * Fischer  _et al._ (2011) L. H. Fischer, G. S. Harms, and O. S. Wolfbeis, Angew. Chem. Int. Ed. 50, 4546 (2011).
  * Carlos  _et al._ (2009) L. D. Carlos, R. A. Ferreira, V. d. Z. Bermudez, and S. J. Ribeiro, Adv. Mater. 21, 509 (2009).
  * Lokesh  _et al._ (2021a) M. Lokesh, R. Vaippully, V. P. Bhallamudi, A. Prabhakar, and B. Roy, J. Phys. Commun. 5, 115016 (2021a).
  * Lokesh  _et al._ (2021b) M. Lokesh, R. Vaippully, G. Nalupurackal, S. Roy, V. P. Bhallamudi, A. Prabhakar, and B. Roy, RSC adv. 11, 34636 (2021b).
  * Vaippully  _et al._ (2021) R. Vaippully, M. Lokesh, and B. Roy, J. Opt. 23, 094001 (2021).
  * Vaippully  _et al._ (2020) R. Vaippully, V. Ramanujan, M. Gopalakrishnan, S. Bajpai, and B. Roy, Soft Matter 16, 7606 (2020).
  * Lokesh  _et al._ (2022) M. Lokesh, G. Nalupurackal, S. Roy, S. Chakraborty, J. Goswami, M. Gunaseelan, and B. Roy, Opt. exp. 30, 28325 (2022).
  * Kumar  _et al._ (2020a) S. Kumar, M. Gunaseelan, R. Vaippully, A. Kumar, M. Ajith, G. Vaidya, S. Dutta, and B. Roy, Biomed. Opt. Exp. 11, 3555 (2020a).
  * Kumar  _et al._ (2020b) S. Kumar, A. Kumar, M. Gunaseelan, R. Vaippully, D. Chakraborty, J. Senthilselvan, and B. Roy, Front. Phys. 8, 570842 (2020b).
  * Schachoff  _et al._ (2015) R. Schachoff, E. Sperling, M. Mertig, M. Selmke, A. Bregulla, F. Cichos, D. Rings, D. Chakraborty, K. Kroy, K. Gunther, _et al._ , diffusion-fundamentals.org 23 (2015) 1, pp 1-19, (2015).
  * Suresh  _et al._ (2022) S. Suresh, M. Lokesh, G. Nalupurackal, R. Vaippully, S. Roy, and B. Roy, in  _Optical Trapping and Optical Micromanipulation XIX_ , Vol. 12198 (SPIE, 2022) pp. 72–81.
  * Wen  _et al._ (2018) S. Wen, J. Zhou, K. Zheng, A. Bednarkiewicz, X. Liu, and D. Jin, Nat. commun. 9, 1 (2018).
  * Kumar  _et al._ (2022) A. Kumar, M. Gunaseelan, G. Vaidya, R. Vaippully, and B. Roy, Eur. Phys. J. Spec. Top. 231, 605 (2022).
  * Braibanti  _et al._ (2008) M. Braibanti, D. Vigolo, and R. Piazza, Phys. Rev. Lett. 100, 108303 (2008).
  * Nalupurackal  _et al._ (2022) G. Nalupurackal, M. Gunaseelan, S. Roy, M. Lokesh, S. Kumar, R. Vaippully, R. Singh, and B. Roy, Soft matter 18, 6825 (2022).
  * Wurger (2010) A. Wurger, Rep. Prog. Phys. 73, 126601 (2010).
  * Piazza (2008) R. Piazza, Soft Matter 4, 1740 (2008).
  * Howse  _et al._ (2007) J. R. Howse, R. A. Jones, A. J. Ryan, T. Gough, R. Vafabakhsh, and R. Golestanian, Physical review letters 99, 048102 (2007).
  * Bechinger  _et al._ (2016) C. Bechinger, R. Di Leonardo, H. Lowen, C. Reichhardt, G. Volpe, and G. Volpe, Reviews of Modern Physics 88, 045006 (2016).
  * Jiang  _et al._ (2010) H.-R. Jiang, N. Yoshinaga, and M. Sano, Phys. Rev. Lett. 105, 268302 (2010).
  * Solon  _et al._ (2015) A. P. Solon, Y. Fily, A. Baskaran, M. E. Cates, Y. Kafri, M. Kardar, and J. Tailleur, Nature Physics 11, 673 (2015).
  * Schaffer  _et al._ (2007) E. Schaffer, S. F. Norrelykke, and J. Howard, Langmuir 23, 3654 (2007).
  * Sekimoto (1998b) K. Sekimoto, Prog. Theor. Phys. 130, 17 (1998b).
  * Jarzynski (1997) C. Jarzynski, Physical Review Letters 78, 2690 (1997).
  * Dabelow  _et al._ (2019) L. Dabelow, S. Bo, and R. Eichhorn, Physical Review X 9, 021009 (2019).
  * Shankar and Marchetti (2018) S. Shankar and M. C. Marchetti, Physical Review E 98, 020604 (2018).
  * Ritort (2008) F. Ritort, (2008).
  * Wen  _et al._ (2007) J.-D. Wen, M. Manosas, P. T. Li, S. B. Smith, C. Bustamante, F. Ritort, and I. Tinoco Jr, Biophysical journal 92, 2996 (2007).
  * Ciliberto (2017) S. Ciliberto, Physical Review X 7, 021051 (2017).
  * Seifert (2008) U. Seifert, The European Physical Journal B 64, 423 (2008).
  * Shan  _et al._ (2021) X. Shan, F. Wang, D. Wang, S. Wen, C. Chen, X. Di, P. Nie, J. Liao, Y. Liu, L. Ding, _et al._ , Nat. Nanotechnol. 16, 531 (2021).
  * Govorov  _et al._ (2006) A. O. Govorov, W. Zhang, T. Skeini, H. Richardson, J. Lee, and N. A. Kotov, Nanoscale Res. Lett 1, 84 (2006).
  * Roichman  _et al._ (2008) Y. Roichman, B. Sun, Y. Roichman, J. Amato-Grill, and D. G. Grier, Phys. rev. lett. 100, 013602 (2008).