# adSformers: Personalization from Short-Term Sequences and Diversity of
Representations in Etsy Ads

Alaa Awad, Denisa Roberts  and  Eden Dolev, Andrea Heyman, Zahra Ebrahimzadeh,
Zoe Weil, Marcin Mejran, Vaibhav Malpani, Mahir Yavuz  Etsy117 Adams
StBrooklynNYUSA11201 aawad, denisaroberts, edolev<EMAIL_ADDRESS>
###### Abstract.

In this article, we present a general approach to personalizing ads through
encoding and learning from variable-length sequences of recent user actions
and diverse representations. To this end we introduce a three-component module
called the adSformer diversifiable personalization module (ADPM) that learns a
dynamic user representation. We illustrate the module’s effectiveness and
flexibility by personalizing the Click-Through Rate (CTR) and Post-Click
Conversion Rate (PCCVR) models used in sponsored search. The first component
of the ADPM, the adSformer encoder, includes a novel adSformer block which
learns the most salient sequence signals. ADPM’s second component enriches the
learned signal through visual, multimodal, and other pretrained
representations. Lastly, the third ADPM ”learned on the fly” component further
diversifies the signal encoded in the dynamic user representation. The ADPM-
personalized CTR and PCCVR models, henceforth referred to as adSformer CTR and
adSformer PCCVR, outperform the CTR and PCCVR production baselines by
$+2.66\%$ and $+2.42\%$, respectively, in offline Area Under the Receiver
Operating Characteristic Curve (ROC-AUC). Following the robust online gains in
A/B tests, Etsy Ads deployed the ADPM-personalized sponsored search system to
$100\%$ of traffic as of February 2023.

Personalization, Sponsored Search, Online Advertising, Neural Networks,
Transformers, Representation Learning, Multimodality, Ranking, Bidding, Ads
Computer Vision

††ccs: Information systems Personalization††ccs: Information systems Sponsored
Search Advertising††ccs: Machine Learning Approaches Neural Networks

(a) non-personalized ranking

(b) with ADPM-personalized ranking

Figure 1. Ad results from the query jacket for a user who recently interacted
with mens leather jackets.

##  1\. Introduction

Online advertising is a multi-billion dollar industry (¿$\$300$ billion in the
fiscal year 2022) (for, 2023) and sponsored search is one of its largest sub-
fields. In the ”pay-per-click” model, advertisers are charged for clicks.
Thus, sponsored search systems have an incentive to engage users via accurate
real-time predictions of clicks and purchases. Deep learning methods have been
employed in recent years to improve the performance of these systems in online
advertising (Zhang et al., 2021a; Zhai et al., 2016; Zhang et al., 2022; Zhou
et al., 2018). More generally, powerful deep learning architectures have been
employed to leverage signals from user behaviors and personalize industrial
applications, for better user satisfaction and business outcomes (Chen et al.,
2019; Xu et al., 2022; Pancha et al., 2022; Zhou et al., 2018).

In this article we introduce a general approach to personalization in
industrial large-scale applications via a methodology called the adSformer
diversifiable personalization module (ADPM). We illustrate how this
methodology is deployed to personalize Etsy Ads ranking and bidding systems in
order to demonstrate the method’s effectiveness. The ADPM-personalized
sponsored search system has been deployed to $100\%$ of user traffic and
delivering robust online results since February 2023 in Etsy Ads.

We make the following contributions:

  * •

We introduce the adSformer diversifiable personalization module (ADPM
henceforth), a highly configurable module used to encode sequences of recent
user actions and derive a short-term dynamic user representation. The module
can handle sequences of variable lengths. ADPM’s architecture includes three
components which learn diverse signals symbiotically: 1. the adSformer encoder
component; 2. the pretrained representations component; 3. the learned ”on-
the-fly” component. We detail ADPM’s architecture and explain its
implementation for impact and scalability.

  * •

The first component of the ADPM, the adSformer encoder, introduces a novel
variation on the classic transformer block, where a global max pooling layer
is added at the last stage, borrowing from the computer vision literature. We
explain the architecture design choice and learning behavior.

  * •

We demonstrate the module’s effectiveness in practice by describing how the
ADPM is employed to personalize sponsored search (ad) ranking. We include
offline and online experimental results and deployment considerations. We
share reproducibility details including optimal hyperparameters, model design
choices, pretrained representations workflows, and training approaches.

###  1.1. Related Work

####  1.1.1. Personalization from user sequences

A useful survey of deep learning methods employed in sponsored search is
(Zhang et al., 2021a). A few other works include sequences of user actions to
more generally improve click-through rate prediction models in (Zhou et al.,
2019; Pi et al., 2020, 2019; Chen et al., 2019; Xu et al., 2022; Zhou et al.,
2018; Aslanyan et al., 2020; Yu et al., 2021; Pancha et al., 2022; Kang and
McAuley, 2018; Grbovic and Cheng, 2018), which is an essential task in online
advertising. Recent developments in personalization emerged to separate
sequences into long-term and recent or short-term. Pinterest describes an end-
to-end ranking application in (Xu et al., 2022), with a brief reference to
recent user actions encoded via a multi-headed self-attention layer and fed to
downstream ranking tasks such as search, recommendations, and ads. In
personalized eCommerce search, (Aslanyan et al., 2020) at Ebay had employed
the representations of the most recent five clicked items to personalize
search rankers.

####  1.1.2. Transformers in personalization

Transformer encoders have been used in personalized product search, both at
retrieval and re-ranking phases. The transformer architecture captures the
context of users’ intents by encoding the sequential nature of user behavior
sequences as seen in previous works from Alibaba, Pinterest, and others (Chen
et al., 2019; Kang and McAuley, 2018; Sun et al., 2019; Pancha et al., 2022).
Particularly relevant to our work is the behavioral sequence transformer in
(Chen et al., 2019) which encodes a sequence of last 20 actions to personalize
downstream click-through rate models. Focusing on the retrieval phase, TEM, a
transformer based model, can tune the amount of personalization applied based
on the query characteristics (Bi et al., 2020), illustrating the versatility
of the transformer architecture. In addition to employing attention mechanisms
and transformer architecture, authors in (Zhang et al., 2020) employ graph
neural networks to build a session-aware recommender, where a session
represents a short-term user sequence. In a similar vein, sequence and graph
modelling came together as a heterogeneous user graph transformer in (Shui et
al., 2022) to improve personalized recommendations.

The rest of the article is organized as follows. We introduce the three-
component ADPM in Section 2. Component two of the ADPM employs pretrained
representations therefore we detail Etsy Ads’ representation learning
workflows in Section 3. To demonstrate ADPM’s effectiveness and generality, we
describe how this module can be configured and deployed to personalize
sponsored search ranking models in Section 4. We present offline ablation
studies and online experiment results in Section 5, after which we conclude.

##  2\. ADPM Methodology

Figure 2. The adSformer diversifiable personalization module (ADPM) derives a
dynamic user representation from variable-length sequences of user actions.
The adSformer Encoder component includes the adSformer block displayed on the
right.

###  2.1. Variable Length Sequences

The ADPM, as seen in left-hand side of Figure 2, encodes sequences of recent
user actions $S_{i}=(a_{i},e_{i})$ of variable lengths ${y_{i}}$, variable
entity types ${e_{i}}$ and variable action types $a_{i}$. Examples of $(e,a)$
pairs are $e=listingID$ and $a=viewed$. The sequence length $y$ is a random
variable taking values in $\\{0,...,M\\}$, where $M$ is the maximum sequence
length dictated by the engineering system. Realized values of $y$ are dictated
by the time delta threshold, which is set to one hour in most of our
experiments. All ADPM-encoded user actions would have happened within this
time delta, and actions outside the decided upon time delta are masked. The
ADPM outputs a short-term dynamic user representation which captures complex
relationships within and between different source sequences due to its three
components. The ADPM can be used within the context of any model that can
benefit from real time personalization in industrial large-scale applications.

###  2.2. Diversity of Signals from ADPM’s Components

The three-component ADPM design is motivated by its intended use in
personalization. Diversity of information helps to capture the nuanced
patterns of individual users. The three ADPM components learn different
signals from the sequences of user actions. This diversity of signals goal is
typical when designing neural network architectures and the ADPM architecture
achieves this goal for user sequence modelling. The signals learned by each
component complement each other which fosters symbiotic learning. In deployed
applications, this symbiotic diversity of signals improves robustness to input
distribution shifts. The diversity of learned signals also leads to more
relevant personalized results through a better derived user representation. We
evaluate the quality of this representation in experiments discussed in
Section 5.1.1. Lastly, the diverse signals learned by the three components
help the ADPM better adapt to diverse downstream tasks such as ranking,
retrieval, and recommendations.

###  2.3. Component One: adSformer Encoders

The adSformer encoder component uses one or more custom adSformer blocks as
seen in Figure 2, right panel. This component learns a deep, expressive
representation of the input sequence. The adSformer block modifies the
standard transformer block in (Vaswani et al., 2017) by adding a final global
max pooling layer. This layer downsamples the block’s outputs by extracting
the most salient signals from the sequence representation.

The adSformer block starts with an embedding layer which encodes listing IDs
into dense vectors of size $d_{1}=32$. We pad the variable length sequence to
a common length of $M$ and mask the padding token. We concatenate the target
listing representation at position zero as in (Chen et al., 2019). We then add
a fully learnable position embedding of same dimension $d_{1}$ to learn the
sequence order for the concatenated $(target,sequence)$ as in (Devlin et al.,
2018). A multi-head self attention layer with scaled dot-product attention
(Vaswani et al., 2017) follows. First we get

(1)
$\text{A}(\textbf{Q},\textbf{K},\textbf{V})=\text{softmax}\big{(}\frac{\textbf{Q}\textbf{K}^{T}}{\sqrt{d}}\big{)}\textbf{V},$

where Q, K and V are query, keys and values respectively, and $A$ is the
attention. Then the multi-head self attention is

(2)
$\text{MHSA}(EP)=\text{Concat}(head_{1},head_{2},\cdots,head_{h})\textbf{W}^{H},$

with
$head_{i}=\text{A}(\textbf{E}\textbf{W}^{Q},\textbf{E}\textbf{W}^{K},\textbf{E}\textbf{W}^{V})$.
The projection matrices are
$\textbf{W}^{Q},\textbf{W}^{K},\textbf{W}^{V}\in\mathbb{R}^{d\times d}$, and
$EP$ represents listing embedding added to position embedding $EP=E+P$. Then
the block adds point-wise feed-forward networks $FFN$, $LeakyRelu$ non-
linearity as in (Chen et al., 2019), and residual connections, dropout and
layer normalization in the usual sequence. To simplify notation, $x$ is the
output of the previous layer in the block and $g(x)$ represents the next
layer:

(3) $\displaystyle g(x)$ $\displaystyle=x+Dropout(MHSA(x))$ (4) $\displaystyle
x$ $\displaystyle=LayerNorm(x+g(x))$ (5) $\displaystyle g(x)$
$\displaystyle=Dropout(FFN(LeakyRelu(x))$ (6) $\displaystyle x$
$\displaystyle=LayerNorm(x+g(x)).$

We add the global max pooling layer at the last stage

(8) $o_{1}=GlobalMaxPooling(x).$

The global pooling layer downsamples the output of the transformer block to a
representation vector of size $d_{1}$ instead of outputting the concatenated
transformer block features for the entire sequence. We thus retain the most
salient signal, in a parameter efficient manner. We experiment with replacing
the global max pooling layer with global average pooling layer and present
offline ablation results in Section 5.

###  2.4. Component Two: Pretrained Representations

Component two encodes sequences using listing ID pretrained representations
together with average pooling. Depending on downstream performance and
availability, we choose from multimodal (AIR) representations detailed in
Section 3.2, visual representations detailed in Section 3.1, or interaction-
based representations detailed in 3.3. Thus component two encodes rich image,
text, and multimodal signals from all the listings in the variable length
sequence. Formally, for a given sequence of listing IDs’ embedding vectors
$e_{i}\in\mathbb{R}^{d_{2}}$ with $d_{2}=256$, we compute

(9) $o_{2}=GlobalAveragePooling([e_{1},e_{2},...,e_{y}])$

where $y$ is the sequence length. Then the output of ADPM’s component two
$o_{2}$ is a sequence representation vector of size $d_{2}$. The individual
listing ID’s pretrained representations are kept frozen in downstream tasks,
not requiring an expensive gradient calculation, while adding multimodal
signal.

###  2.5. Component Three: Representations Learned ”On the Fly”

The third component of the ADPM introduces representations learned for each
sequence from scratch in its own vector space as part of the downstream
models. This component learns light weight representations for many different
$(e,a)$ sequences for which we do not have pretrained representations. For
example we encode sequences of entities $e$ of type {listingID, taxonomyID,
shopID} for user actions $a$ of type {cart add, favorited, purchased}.
Formally, for $S_{i}=(a_{i},e_{i})$ of variable lengths ${y_{i}}$, variable
entity types ${e_{i}}$ and variable action types $a_{i}$, we embed each entity
$e_{i}$ in the action space $a_{i}$ to get a vector representation
$E_{aei}\in\mathbf{R}^{d_{ae}}$ of dimensions $d_{ae}$,

(10) $\displaystyle E_{S_{i}}$
$\displaystyle=GlobalAveragePooling([e_{1},e_{2},...,e_{y_{i}}])$ (11)
$\displaystyle o_{3}$
$\displaystyle=Concat([E_{S_{1}},E_{S_{2}},...,E_{S_{z}}])$

with $i\in\\{1,...,z\\}$ and $z$ is the number of sequences encoded by
component three.

###  2.6. Dynamic User Representation

Given an input set of $p$ user action sequences $S_{i}=(a_{i},e_{i})$,
$i\in\\{1,...,p\\}$, of variable lengths $y_{i}$, the ADPM layer concatenates
the three components’ outputs to derive a short term dynamic user
representation of size $d_{1}+d_{2}+\sum d_{ae}$.

(12) $u=ADPM(S_{1},S_{2},...,S_{p})=Concat([o_{1},o_{2},o_{3}]).$

We concatenate this user representation $u$ to the input layer in downstream
personalization tasks.

Figure 3. Etsy Ads personalizes sponsored search’s CTR and PCCVR models by
concatenating the short-term dynamic user representation from ADPM to the
input representation layer. Both models share a similar architecture with
slight differences in numbers of layers and hidden units.

###  2.7. ADPM Implementation

We designed the ADPM as a general plug-and-play module and implement it as a
TensorFlow (Abadi et al., 2016) Keras layer which can be reused across
personalization use-cases with a simple import statement. The ADPM layer is
configurable, allowing arguments for customizing the ADPM’s three components.
Since sequences of streamed user actions are made available by the Feature
System team to all downstream Etsy applications, ADPM-based personalization
can be easily scaled to multiple downstream personalization tasks while at the
same time introducing a per-model customization to avoid sacrificing
performance.

####  2.7.1. Vocabularies

ADPM’s sequence representations are implemented as lookup tables. However,
when vocabularies are large, the corresponding weight matrices can take up too
much memory and disk space, a significant challenge in deployments. To
overcome this challenge and reduce memory consumption, we limit the vocabulary
size for each sequence to the top $K$ most frequent entities, tuning $K$ as a
hyperparameter in downstream CTR and PCCVR models. For example, the CTR’s
vocabulary corresponding to sequences of listing IDs was capped to top
$K=750,000$ listings, down from 100 million in total. The top $K$ listings
carry the largest impact, after which increasing vocabulary size led to only
marginal improvements in model performance. Hence vocabulary size impact on
model performance follows a power distribution (Roberts and Roberts, 2017). We
built the vocabularies dynamically, from the corpus of variable-length
sequences present in the downstream models’ training datasets, further
maximizing impact. The time delta is a hyperparameter that we can easily tune
through ADPM’s flexible implementation.

####  2.7.2. Software implementation of the pretrained representations
component

Pretrained representations are useful for feature encoding, scalability and
modularity and we describe learning pretrained representations for every
active listing in Section 3. Running this process offline allows for deeper
architectures without latency concerns. The pretrained representation vectors
are read by ADPM’s component two through the use of lookup tables and a
mapping between vocabulary and index positions. To make pretrained
representations easily accessible to downstream models, we wrap the lookup
table in a TensorFlow SavedModel which includes average pooling layers and
handling of default padding values. By saving the lookup table with the
downstream model, we assure that the representations used at training time are
the same ones used at serving time.

##  3\. Pretrained Representation Learning

Component two of the ADPM employs pretrained representations to encode
variable-length sequences of user actions. For this reason we give an overview
of Etsy Ads’ pretrained representation learning workflows. Several flavors of
representations are trained and used to encode listing IDs: visual, multimodal
(AIR), and skip-gram. ADPM’s component two’s optimum pretrained representation
depends on the specifics of the downstream tasks, as illustrated in ablation
studies from Section 5. Figure 4 shows that the top k nearest neighbors for a
listing are different when using visual versus AIR representations. AIR
representations better encode price, category, and text while the visual
representations better encode color, shape, and background image.

(a) Target listing

(b) AIR representation

(c) Visual representation

Figure 4. Given a target listing of an emerald ring shown in (a), we visualize
the top $6$ nearest neighbors based on the listing’s pretrained AIR
representation (b) and listing’s visual representation (c).

###  3.1. Visual Representations

Pinterest authors lament the scarcity of visual representations across large-
scale industrial applications (Beal et al., 2022). In Etsy Ads, we employ
image signals across a variety of tasks, such as visually similar candidate
generation, search by image, learning other pretrained representations, and
the ADPM’s second component.

####  3.1.1. Model design

We train image representations using a multitask classification architecture
(Zhai et al., 2019), an improvement over the ”classic” classification as a
proxy to metric learning (Zhai and Wu, 2018). By using multiple classification
heads, such as taxonomy, color, and material, our representations are able to
capture more diverse information about the image. An EfficientNetB0
architecture (Tan and Le, 2019) with weights pretrained on ImageNet (Deng et
al., 2009) served as the backbone, and the final layer was replaced with a
256-dimensional convolutional block, the desired output representation size.
Image random rotation, translation, zoom, and a color contrast transformation
to augment the dataset were applied during training.

####  3.1.2. Multitask learning of visual representations

Rather than using a single dataset of listing images with multiple label
columns, heterogeneous dataset sources containing product images with
different selected attributes as the labels were included. Listing attributes
such as color and material are optionally input by sellers, so they can be
sparse. To mitigate this concern we implemented a dataset sampler, pictured in
Figure 5, which takes in the separate datasets, each with its own unique label
type. The sampler evenly distributes examples from each dataset to construct
balanced training batches and ascertain that only loss coming from the
respective task’s classification head is considered during backpropagation.
One additional benefit of this approach is that we can mix in datasets of
images from different visual domains to support more use cases, such as user-
uploaded review photos. The multitask visual representations are integrated
into the visually similar ad recommendations as well as the Search by Image
application built by Etsy Ads, with more details included in (Dolev et al.,
2023) and Etsy’s blog post (Dolev and Awad, 2023).

Figure 5. Multitask and multi-dataset visual representations training
architecture. The data sampler combines examples from an arbitrary number of
datasets corresponding to respective classification heads.

###  3.2. Multimodal Representations: Learning AIR

####  3.2.1. Model design

We design Ads Information Retrieval (AIR) pretrained listing representations
with the explicit goal of driving ad clicks. The AIR model is a neural network
with a pseudo-two-tower architecture pictured in Figure 6. The source and
candidate towers share all trainable weights, for which reason we add the
prefix ”pseudo” to the canonical two-tower architecture (Yi et al., 2019; Yang
et al., 2020).

Figure 6. Learning AIR: pseudo-two-tower architecture with shared weights.

####  3.2.2. Learning

We compile a dataset of listing pairs that have been clicked together on
various pages and we consider one listing as the ”source” and the other
listing as the ”candidate” within the same example. The training dataset
includes source-candidate pairs, where for each source we consider its
corresponding candidate pair as the positive label, while a sample of the
other examples in the same batch are the negative labels. For each of the
source and candidate listings, we preprocess and concatenate into an input
layer a set of multimodal features. For example, we concatenate the multitask
visual representations described in Section 3.1, text representations of
listing’s title, tags and taxonomy path average pooled from lightweight
fastText pretrained representations (Bojanowski et al., 2017), and a series of
other normalized features. We train the AIR model with a softmax loss (Zhai et
al., 2019) and a classification objective. During training, we inference
source and candidate 256-dimensional representations for each pair in the
batch. We then compute a matrix of cosine similarity scores between each
example’s source and candidate representations (Nigam et al., 2019a). Finally,
we compute the classification loss using these scores.

###  3.3. Skip-Gram Listing Representations

Visual and AIR pretrained listing representations do not capture signals from
the sequential browsing behavior of users within a web session. Therefore we
learn a listing representation from sequences of listings in a browsing
session by employing the skip-gram (Mikolov et al., 2013) model, similarly to
(Grbovic and Cheng, 2018; Zhao et al., 2018). We learn a vector representation
of dimension $d=64$ for each unique listing in the training set. We use a
hierarchical softmax loss function ((Morin and Bengio, 2005) and (Mnih and
Hinton, 2008)), which showed much better results than classic negative
sampling. We found that fastText library 111https://fasttext.cc/ worked well
for training, as long as we disabled functionality to consider subwords.

##  4\. ADPM - Personalized Sponsored Search Ranking

###  4.1. Background

To demonstrate ADPM’s effectiveness and generality, we describe how Etsy Ads
employes it to personalize ranking and bidding models in sponsored search.
First, we’ll give an overview of the sponsored search system. As many online
advertising services, sellers sponsor their listings (ads) through a second-
price cost-per-click auction campaign. In order to decide which ads to show to
a user, Etsy Ads employs Learning to Rank (LTR) (Liu, 2011; Cao et al., 2007)
framework in Figure 7. The system is divided according to the classic two-
stage dichotomy: ads candidate retrieval and then ads ranking. First, the
candidate retrieval step winnows the full inventory from over 100 million
listings (items) to under 1000 candidates, after which results are reranked
based on a combined value score obtained from the ranking and bidding models
predictions, the CTR and PCCVR models respectively.

Figure 7. In our Learning to Rank system, ads are ranked by a value score.

In the CTR case, $p(\mathbf{x})$ denotes the predicted probability
$p(y_{CTR}=1)$ that the candidate listing will be clicked. For PCCVR
$p(\mathbf{x})$ denotes the predicted conditional probability
$p(y_{PCCVR}=1|y_{CTR}=1)$ that the listing will be purchased, having been
clicked (Awad et al., 2022).

Before including the ADPM, baseline production models’ architectures in Etsy
Ads were standard stacked deep and cross neural networks (DCN) (Wang et al.,
2021). We used these as baselines in all our offline and online comparisons.
The CTR’s DCN has four cross and four deep layers with sizes
${5000,2500,250,500}$ and PCCVR’s DCN employs two cross and two deep layers of
sizes 240, and 120 respectively.

###  4.2. ADPM-Personalized CTR and PCCVR

The goal is to personalize CTR and PCCVR using the ADPM for improved user
outcomes. We employ the ADPM to encode sequences of recent user actions
anywhere on Etsy for both logged-in and logged-out users. Instead of
considering the last M user actions as typically seen in (Pancha et al., 2022;
Chen et al., 2019; Xu et al., 2022), we consider a sliding one hour window of
user actions, in reverse chronological order of timestamps, to encode only
recent behavior. More formally, let $S=\left(a,e,t\right)$, be a one-hour
sequence of user actions, where $a$ is the action type one of {view, favorite,
cart add, purchase, search}, and $e$ represents one of the entities
{listingID, shopID, categoryID, text query} associated with action $a$
performed at timestamp $t$. Due to semantics and infrastructure constraints we
further cap the maximum sequence length at $M=50$ actions. Each sequence $I$
is truncated to within one hour of the most recent action, so
$t_{0}-t_{last}\leq 1hr$. The resulting sequences have variable length which
the ADPM can handle through padding and masking.

We concatenate ADPM’s output, the dynamic user representation, to the rest of
the wide input feature vector (Cheng et al., 2016) similarly to (Chen et al.,
2019). The optimum ADPM configuration for the three components differs between
the CTR and the PCCVR models as Table 1 details, and this flexibility is one
of ADPM’s strengths. Each configuration choice is similar to a hyperparameter
that we tune.

Table 1. The different CTR and PCCVR ADPM configurations Config | CTR | PCCVR
---|---|---
Pretrained Representation Type | AIR | skip&visual
Num adSformer Blocks | 1 | 1
adSformer Block Dropout | 0 | 0
adSformer Num Heads | 3 | 2
Learned Listing Representation Dimension | 32 | 32
Learned Shop Representation Dimension | 16 | 16
Learned Taxonomy Representation Dimension | 8 | 8
(e, a) (Learned Component) | (all e, all a) | (all e, all a)
(e, a)((Pretrained Component) | (listing, all a) | (listing, all a)
(e, a) (adSformer Encoder) | (listing, viewed) | (listing, viewed)
(listing, viewed) Vocabulary Size | 750K | 650K
Num OOV Tokens | 1 | 1

After employing ADPM personalization we call the new models the adSformer CTR
and PCCVR. Figure 3 depicts the model architecture corresponding to the
adSformer CTR and PCCVR after ADPM inclusion.

####  4.2.1. The adSformer CTR

For the adSformer CTR model, the adSformer encoder component includes one
adSformer block with three attention heads. The adSformer encoder encodes user
and browser sequences of recently viewed listings since these $(e,a)$ have the
highest session frequency. Within the pretrained representations component,
the multimodal pretrained representation (AIR) described in Section 3.2 works
best for the CTR task to encode all sequences of listing IDs. ADPM’s third,
”on the fly”, component encodes all one-hour sequences of recent user actions.

After concatenating the ADPM’s output to the input representation layer, it is
critical to have a cross network (DCN) in the personalized adSformer CTR
architecture (Wang et al., 2017). As recognized in recent work (Zhang et al.,
2022; Lang et al., 2021; Song et al., 2019; Li et al., 2020; Wang et al.,
2021; Roberts and Roberts, 2020; Zhang et al., 2022), learning higher order
feature interactions effectively from the input layer helps performance in
large scale CTR prediction. For the adSformer CTR, the interaction module
(DCN) is necessary to fully leverage the wide input representation which
includes the ADPM. The cross layer exclusion leads to a 1.17% drop in ROC AUC.
The large capacity of the adSformer CTR model aids learning and generalization
by providing a smoother loss landscape and more stable and faster learning
dynamics. The adSformer CTR only needs one epoch of training, for a total of
11 hours (one A100 GPU) using Adam (Kingma and Ba, 2014) optimizer. Throughout
model training we decay the learning rate using cosine annealing (Loshchilov
and Hutter, 2016). We select the largest batch size that can fit in memory
(8192) and tune the learning rate to an optimum $lr=0.002$.

####  4.2.2. The adSformer PCCVR

To obtain the ADPM-personalized adSformer PCCVR model, we configure the ADPM
and concatenate its output user representation to the baseline PCCVR model
input layer, similarly to the CTR. Table 1 gives the ADPM configuration for
the adSformer PCCVR model. The major differences to the CTR’s case is a
smaller optimum adSformer encoder component with only two heads. The PCCVR
trains on smaller datasets. We parameterize the ADPM’s pretrained
representations component with pretrained skip-gram and visual listing
representations. While we trained the PCCVR baseline production model on only
two weeks of historical click logs (135 million examples), the adSformer PCCVR
has more parameters after the ADPM addition and thrives with three weeks of
data (200mln examples).

####  4.2.3. Learning

Both the CTR and PCCVR before and after ADPM-based personalization are
formulated as binary classification problems, and we use a binary cross
entropy loss function

(13) $L=-\frac{1}{N}\sum_{\left(\mathbf{x},y\right)\in
D}\left(ylogp\left(\mathbf{x}\right)+\left(1-y\right)log\left(1-p\left(\mathbf{x}\right)\right)\right),$

where $D$ represent all samples in the training dataset, $y\in\\{0,1\\}$ is
the label, $p(x)$ is the predicted probability, and $x$ is the input vector
containing (listing, query, and context) attributes as well as the ADPM’s user
representation output.

##  5\. Experimental Results

We evaluate ADPM’s effectiveness through offline and online experiments by
comparing the adSformer CTR and adSformer PCCVR models to their non-
personalized baselines described in Section 4. We also perform ablation
studies where we compare the ADPM with other user sequence modeling
approaches. Finally, we compare the effectiveness of permutations of ADPM
configurations in offline experiments.

###  5.1. Offline Experiments

We evaluate offline performance using PR-AUC and ROC-AUC metrics and present
lifts as compared to baselines in all the ablation studies.

####  5.1.1. Evaluating The Effectiveness of ADPM’s Components and Their
Symbiotic Learning Behavior

We aim to better understand the symbiotic learning behavior from the
combination of the three components of the ADPM. Our hypothesis is that each
component learns different signals and together through this diversity lead to
better outcomes in downstream tasks. For a comparison, we permute the
configuration of the ADPM holding constant all other modelling choices and
provide lifts in the CTR and PCCVR AUC metrics.

Furthermore we compare the three-component ADPM to a baseline encoder of user
sequences, such as the BST employed by (Chen et al., 2019), which uses an
eight head transformer encoder with one block to encode a sequence of 20 user
actions and their timestamp. Other alternatives exist, such as the attention
mechanisms in (Zhou et al., 2018) or (Xu et al., 2022) but the BST model in
(Chen et al., 2019) outperforms most of these approaches according to (Chen et
al., 2019). We also compare against a simple average embedding aggregation of
the last five actions, similarly to Ebay’s (Aslanyan et al., 2020). The goal
is to understand if the ADPM is more effective through its design choices then
these baselines when used to personalize a downstream task, such as the CTR
model.

We strive to make comparisons as relevant as possible by using the same
underlying sequence lengths and datasets, training budgets, as well as
hyperparameters from the original papers, however some differences remain
dictated by the implementation pipelines. For example, we do learn the
position embedding for the BST similarly to the ADPM instead of using
timestamp deltas but do not foresee major differences as noted in (Vaswani et
al., 2017). When employed to personalize the CTR model in ADPM’s place, the
BST runs out of memory for a one hour sequence and eight heads (as in the BST
article) at an embedding size of $32$ and for the same batch size employed in
all experiments, so we downsize to five and three heads.

One of ADPM’s strength is the ability to derive maximum signal with
constraints on training and serving resources. We also proxy Ebay’s (Aslanyan
et al., 2020) encoding of last $k=5$ user actions as an alternative baseline
for ADPM. In the original article BST is run on a fixed sequence length of 20,
so we apply both the BST and the ADPM with three components on a sequence of
most recent 20 actions as well. To summarize the results in Tables 2 and 3,
the three-component ADPM outperforms all the other component combinations as
well as the BST and other baselines, although the BST is a strong alternative.

Table 2. Ablation studies showing offline lift in metrics within the CTR downstream task. Each variant is compared against the baseline, non-personalized, CTR model. Each variant represents either a configuration of the three component ADPM or another personalization method we compare the ADPM against. We vary the ADPM’s three components: Comp1=adSformerEncoder; Comp2=Pretrained; Comp3=On-the-fly. Personalization baselines are: BST=Alibaba’s Behavioral Sequence Transformer; Ebay’s last k=5. | CTR
---|---
Configuration | ROC-AUC | PR-AUC
$ADPM-3[MaxPool]$ |  +2.71% |  +8.88%
$ADPM-3[AvgPool]$ | +2.67% | +8.36%
$[Comp1,Comp2]$ | +2.42% | +7.19%
$[Comp1,Comp3]$ | +2.04% | +6.36%
$[Comp2,Comp3]$ | +2.43% | +7.65%
$[Comp1]$ | +1.71% | +5.46%
$[Comp2]$ | +2.09% | +6.31%
$[Comp3]$ | +1.75% | +5.74%
$[ADPM-k=20]$ | +2.66% | +8.5%
$[BST-k=20-8heads]$ | +1.77% | +5.55%
$[BST-1-hr-5heads]$ | +1.82% | +3.99%
$[BST-1-hr-3heads]$ | +1.78% | +4.31%
$[Ebay-k=5]$ | +1.65% | +4.58%
$[ADPM-k=5]$ | +2.43% | +7.4%
$[BST-k=5]$ | +1.45% | +4.77%
Table 3. Ablation studies comparing downstream performance for permutations of ADPM’s components within the PCCVR downstream task. | PCCVR
---|---
Configuration | ROC-AUC | PR-AUC
$ADPM-3[MaxPool]$ |  +2.42% |  +28.47%
$ADPM-3[AvgPool]$ | +2.37% | +26.76%
$[Comp1,Comp2]$ | +1.09% | +13.15%
$[Comp1,Comp3]$ | +1.87% | +18.14%
$[Comp2,Comp3]$ | +2.12% | +24.38%
$[Comp1]$ | +0.36% | +3.47%
$[Comp2]$ | +0.84% | +10.76%
$[Comp3]$ | +1.55% | +15.48%

We have additionally experimented with a an average pooling layer replacing
the final global max pooling layer in the adSformer block within ADPM’s
component one. Results are similar for the two, which makes intuitive sense
since the average is influenced by extreme values in a sample such as the max.
The global max pooling is still more effective in deriving a different signal
from the sequence which complements the signals derived by the other two
components. Interestingly, ADPM also outperformes when encoding the last five
or the last 20 instead of a one-hour, variable-length, sequence.

####  5.1.2. Evaluating the Impact of Different Choices of Pretrained
Representations in ADPM’s Component Two

In Table 4 we can see how different configurations of the pretrained
representation component two of ADPM lead to different test ROC AUC and PR
AUC. We notice that combining all three flavors of pretrained representations
in CTR’s ADPM configuration leads to the highest test ROC AUC. However only
AIR was included in the deployed adSformer CTR, since both ROC AUC and PR AUC
are top ranking and memory requirements are lower. In PCCVR’s ADPM
configuration skip-gram and visual representations performed best together in
terms of PR AUC, however the ADPM configuration with only the adSformer
encoder component and the learned component performed well. One hypothesis is
that the adSformer encoder learns a signal similar to the signal from the
pretrained skip-gram since they both encode the sequential nature of sequences
of listing IDs.

Table 4. Ablation studies showing offline lift in metrics against the baseline, non-personalized, models for various configurations of the ADPM’s pretrained representation component. | CTR | PCCVR
---|---|---
Pretrained Selection | ROC-AUC | PR-AUC | ROC-AUC | PR-AUC
Skipgram | +1.99% | +5.88% | +1.35% | +9.93%
Visual | +1.90% | +5.37% | +1.33% | +10.21%
AIR | +2.14% | +6.44% | +1.23% | +8.19%
Skipgram/Visual | +2.10% | +5.71% | +1.32% | +10.38%
AIR/Skipgram | +2.20% | +6.44% | +1.25% | +7.51%
AIR/Visual | +2.16% | +6.22% | +1.17% | +8.54%
AIR/Skipgram/Visual | +2.26% | +6.17% | +1.14% | +7.56%
No Pretrained Representations | +1.60% | +4.89% | +1.37% | +10.24%

The adSformer CTR and PCCVR differ in ADPM’s pretrained component optimal
configuration. We are not surprised, given that a user may have different
intents during their browsing journey. For example, a buyer shown an ad
impression is likely to be earlier in the purchase funnel and the click intent
(CTR prediction) may be around shopping for inspiration or price comparisons,
so the AIR embedding works better. Post-click the purchase intent (PCCVR
prediction) may be more narrowly focused on stylistic variations and shipping
profiles of a candidate listing, so the visual signal from the image embedding
is more important.

For reproducibility, Table 9 gives hyperparameters and other machine learning
choices for the adSformer CTR as well as for its baseline, the non-
personalized standard CTR prediction model. The adSformer PCCVR has similar
settings.

Table 5. Hyperparameters for training the adSformer CTR and its baseline, the non-personalized CTR prediction model which is a neural network with four fully connected layers. Config | adSformer CTR | baseline NN CTR
---|---|---
Num Epochs | 1 | 3
Learning Rate | 0.002 | 0.00061
Learning Rate Scheduler | cosine | cosine
Optimizer | Adam | LazyAdam
Adam $\beta_{1}$ | 0.9 | 0.9
Adam $\beta_{2}$ | 0.999 | 0.999
Adam $\epsilon$ | 1e-08 | 1e-08
Dropout Hidden Layers | 0 | 0
Batch Norm Hidden Layer | yes | yes
Cross Module (DCN) | yes | no
Batch Size (train) | 8192 | 8192
Batch Size (validation) | 500 | 500
Machine Type (training) | A100 | P100
Machine Type (evaluation) | A100 | T4
Loss | BinaryCrossEntropy | BinaryCrossEntropy
Num Parameters (trainable) | 708,504,031 | 233,709,623
Num Parameters (total) | 897,499,741 | 243,726,397

Table 10 gives the dataset choices for the winning adSformer CTR and PCCVR.
For the CTR we sample 50 percent of non-clicked impressions in the training
dataset but perform a random sampling for the validation set, so that
validation and test sets are representative of the production data seen in the
wild. The PCCVR is trained on clicked impressions but evaluated on a
representative test sample, similarly to the adSformer CTR.

Table 6. Train and validation datasets characteristics for adSformer CTR and PCCVR Model | Config | Training | Validation
---|---|---|---
CTR | Num Examples |  300mln |  8.5mln
CTR | Num Consecutive Days | 30 | 1
CTR | Sampling Type | 50-50 negative sampling | random sampling
PCCVR | Num Examples |  200mln |  8.5mln
PCCVR | Num Consecutive Days | 21 | 1
PCCVR | Sampling Type | click filtered | random sampling

In Etsy Ads production, models are trained daily and evaluated on next-day
test datasets. In Figure 8 we can see the PR AUC and ROC AUC lift for the
ADPM-personalized adSformer PCCVR as compared to the non-personalized PCCVR
production baseline across a period of ten days.

Figure 8. Daily offline AUC evaluation of adSformer personalized PCCVR model
against the baseline PCCVR model from 2023-01-11 to 2023-01-21.

###  5.2. Online A/B Experiments

Before deploying to $100\%$ traffic we evaluated the adSformer CTR and
adSformer PCCVR in online A/B tests with results in Table 7. Compared to the
previous production model, the adSformer PCCVR demonstrated an increase in
Return on Ad Spend (ROAS), PCCVR, and Gross Merchandised Sale (GMS). For Etsy
Ads sellers this translates to an increase in orders with larger sale values
at a lower cost. The CTR model shows improved engagement metrics and the
adSformer CTR and PCCVR models deployed in tandem for A/B test demonstrated
even larger increases in ROAS and GMS when comparing to individual model
contributions.

Table 7. Statistically significant online A/B experiment results (p-value ¡ 0.001). Metric | ADPM-CTR | ADPM-PCCVR | Tandem
---|---|---|---
Ads CTR | +2.55% | -0.02% | +2.42%
Ads PCCVR | +0.29% | +1.26% | +0.90%
Ads ROAS | +1.02% | +3.51% | +5.34%
Ads GMS | +3.02% | +2.24% | +4.17%

###  5.3. Additional Considerations in the Deployed ADPM-Personalized Ads
System

Sampling biases, such as position bias (Huang et al., 2021), can reduce ad
ranking improvements from ADPM. A correlation can be observed between the
positional rank of a shown ad and the probability of being clicked, which can
lead to a feedback loop where the model inadvertently prioritizes certain
candidates due to their previous high ranking and not due to model
improvement. To address this challenge, an auxiliary debiasing model (Zhao et
al., 2019) was included in the adSformer CTR.

Predicted CTR and PCCVR scores must reflect true probabilities in order to
assign monetary value and forecast budget depletion, therefore a calibration
layer is also introduced for each of the CTR and PCCVR models. Calibration is
achieved through the parametric approach of Platt Scaling (Platt et al., 1999;
Guo et al., 2017).

Numerous real world machine learning applications across natural language
processing (Radford et al., 2019), computer vision (Beal et al., 2022),
trajectory prediction (Roberts, 2019), personalized ads, search and
recommendations (Zhao et al., 2019; Baltescu et al., 2022), and computational
advertising (Ma et al., 2018a, b) employ multitask learning (Caruana, 1997). A
technique many decades old, multitask learning commonly improves accuracy of
tasks as well as efficiency of machine learning pipelines. However, the case
for multitask learning in mature machine learning systems that have been
deployed as separate models for many years, is complex in practice. For
example, models such as the adSformer CTR and the adSformer PCCVR have been
tuned separately with different choices in terms of training datasets, neural
network capacities, and optimum ADPM configurations. Etsy Ads is in the
process of experimenting with the multitask learning paradigm in personalized
sponsored search.

###  5.4. Infrastructure Considerations in Deployed ADPM-Personalized Ads
System

Introducing most recent / real time sequences into a production training and
serving path includes infrastructure challenges, and a heavy investment in
streaming architecture. Thankfully, much of this infrastructure has already
been standardized across Etsy. Etsy’s Feature System team uses a low latency
streaming system powered by Kafka PubSub for writing interaction event logs to
a low latency feature store. The logged-in and logged-out user features are
made available through a scalable micro service. The fetched in-session
features are captured and preserved in training logs as close as possible to
model serving, which minimizes skew between training and serving feature set.
Our model serving system uses memcache to cache model scores thereby reducing
latency and volume to downstream feature fetching and model inference
services. The key used to write and read requests must be updated to include
user and browser ID (for logged-out users). The real-time nature of the user
action sequences demands that either the cache is removed or that the
expiration TTL is lowered dramatically. At scale, in our production systems,
we pay close attention to latency and cloud cost surges due to increased
volume to downstream services.

##  6\. Conclusion

In this article we introduced a scalable general approach to ads
personalization from variable-lengthsequences of recent user actions, the
adSformer diversifiable personalization module (ADPM). Following the success
of deploying the ADPM-personalized sponsored search, Etsy Ads is currently
experimenting with ADPM-personalized recommendation ads. Future work includes
further experimenting with the ADPM’s three components by creating other
adSformer encoder architectures, adding pretrained graph representations to
component two, and updating image, text, and multimodal representations. To
this end, Etsy Ads continues improving their pretrained representations
through strengthening the ads computer vision capabilities. Finally, Etsy Ads
plans to extend artificial intelligence driven personalization efforts to the
seller toolbox via generative image, multimodal, and video technologies.

##  7\. Acknowledgments

We thank engineers in Etsy Ads for their work leading to developing and
deploying the adSformers. In alphabetical order: Julie Chien, Eric Herrmann,
Julia Hoffman, Rezwana Karim, Mason Kirchner, Melissa Louie, Alec Malstrom,
Seoyoon Park, Maharshi Thakker, Justin Tse, Gerardo Veltri. We also thank the
Rivulet streaming features team, Information Retrieval Platform, and the ML
Platform team for their support.

## References

  * (1)
  * for (2023) 2023\.  _Forecast 2023: Ad Spending Will Slow Down Next Year But Will Continue To Grow_.  https://www.forbes.com/sites/bradadgate/2022/12/08/forecast-2023-ad-spending-will-slow-down-next-year-but-will-continue-to-grow/
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016\.  TensorFlow: A system for large-scale machine learning. In _12th USENIX symposium on operating systems design and implementation (OSDI 16)_. 265–283.
  * Aslanyan et al. (2020) Grigor Aslanyan, Aritra Mandal, Prathyusha Senthil Kumar, Amit Jaiswal, and Manojkumar Rangasamy Kannadasan. 2020\.  Personalized ranking in eCommerce search. In _Companion Proceedings of the Web Conference 2020_. 96–97.
  * Awad et al. (2022) Alaa Awad, Congzhe Su, and Erica Greene. 2022.  _How We Built A Context-Specific Bidding System for Etsy Ads_.  https://www.etsy.com/codeascraft/how-we-built-a-context-specific-bidding-system-for-etsy-ads
  * Baltescu et al. (2022) Paul Baltescu, Haoyu Chen, Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022\.  ItemSage: Learning product embeddings for shopping recommendations at Pinterest.  _arXiv preprint arXiv:2205.11728_ (2022).
  * Beal et al. (2022) Josh Beal, Hao-Yu Wu, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. 2022.  Billion-scale pretraining with vision transformers for multi-task visual representations. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 564–573.
  * Bi et al. (2020) Keping Bi, Qingyao Ai, and W Bruce Croft. 2020.  A transformer-based embedding model for personalized product search. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1521–1524.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the association for computational linguistics_ 5 (2017), 135–146.
  * Cao et al. (2007) Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007.  Learning to rank: from pairwise approach to listwise approach. In _Proceedings of the 24th international conference on Machine learning_. 129–136.
  * Caruana (1997) Rich Caruana. 1997\.  Multitask learning.  _Machine learning_ 28, 1 (1997), 41–75.
  * Chen et al. (2019) Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019.  Behavior sequence transformer for e-commerce recommendation in Alibaba. In _Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data_. 1–4.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. 7–10.
  * Comet.com (2021) Comet.com. 2021\.  _Comet.com home page_.  https://www.comet.com/
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dolev and Awad (2023) Eden Dolev and Alaa Awad. 2023.  _From Image Classification to Multitask Modeling: Building Etsy’s Search by Image Feature_.  https://www.etsy.com/codeascraft/from-image-classification-to-multitask-modeling-building-etsys-search-by-image-feature
  * Dolev et al. (2023) Eden Dolev, Alaa Awad, Denisa Roberts, Zahra Ebrahimzadeh, Marcin Mejran, Vaibhav Malpani, and Mahir Yavuz. 2023.  Efficient Large-Scale Vision Representation Learning.  _arXiv preprint arXiv:2305.13399_ (2023).
  * Grbovic and Cheng (2018) Mihajlo Grbovic and Haibin Cheng. 2018.  Real-time personalization using embeddings for search ranking at Airbnb. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 311–320.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017\.  On calibration of modern neural networks. In _International conference on machine learning_. PMLR, 1321–1330.
  * Huang et al. (2021) Jianqiang Huang, Ke Hu, Qingtao Tang, Mingjian Chen, Yi Qi, Jia Cheng, and Jun Lei. 2021.  Deep position-wise interaction network for CTR Prediction. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1885–1889.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _2018 IEEE international conference on data mining (ICDM)_. IEEE, 197–206.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Lang et al. (2021) Lang Lang, Zhenlong Zhu, Xuanye Liu, Jianxin Zhao, Jixing Xu, and Minghui Shan. 2021\.  Architecture and operation adaptive network for online recommendations. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 3139–3149.
  * Li et al. (2020) Zeyu Li, Wei Cheng, Yang Chen, Haifeng Chen, and Wei Wang. 2020.  Interpretable click-through rate prediction through hierarchical attention. In _Proceedings of the 13th International Conference on Web Search and Data Mining_. 313–321.
  * Liu (2011) Tie-Yan Liu. 2011\.  _Learning to rank for information retrieval_.  Springer Science & Business Media.
  * Loshchilov and Hutter (2016) Ilya Loshchilov and Frank Hutter. 2016.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ (2016).
  * Ma et al. (2018b) Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018b.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 1930–1939.
  * Ma et al. (2018a) Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018a.  Entire space multi-task model: An effective approach for estimating post-click conversion rate. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. 1137–1140.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013\.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ (2013).
  * Mnih and Hinton (2008) Andriy Mnih and Geoffrey E Hinton. 2008.  A scalable hierarchical distributed language model.  _Advances in neural information processing systems_ 21 (2008).
  * Morin and Bengio (2005) Frederic Morin and Yoshua Bengio. 2005.  Hierarchical probabilistic neural network language model. In _International workshop on artificial intelligence and statistics_. PMLR, 246–252.
  * Nigam et al. (2019a) Priyanka Nigam, Yiwei Song, Vijai Mohan, Vihan Lakshman, Weitian Ding, Ankit Shingavi, Choon Hui Teo, Hao Gu, and Bing Yin. 2019a.  Semantic product search. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2876–2885.
  * Nigam et al. (2019b) Priyanka Nigam, Yiwei Song, Vijai Mohan, Vihan Lakshman, Weitian Ding, Ankit Shingavi, Choon Hui Teo, Hao Gu, and Bing Yin. 2019b.  Semantic product search. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2876–2885.
  * Pancha et al. (2022) Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022.  PinnerFormer: Sequence modeling for user representation at Pinterest.  _arXiv preprint arXiv:2205.04507_ (2022).
  * Pi et al. (2019) Qi Pi, Weijie Bian, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. 2019.  Practice on long sequential user behavior modeling for click-through rate prediction. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2671–2679.
  * Pi et al. (2020) Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. 2020\.  Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2685–2692.
  * Platt et al. (1999) John Platt et al. 1999\.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in large margin classifiers_ 10, 3 (1999), 61–74.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Roberts (2019) Denisa Roberts. 2019\.  Neural networks for Lorenz map prediction: A trip through time.  _arXiv preprint arXiv:1903.07768_ (2019).
  * Roberts and Roberts (2020) Denisa AO Roberts and Lucas R Roberts. 2020.  QR and LQ Decomposition Matrix Backpropagation Algorithms for Square, Wide, and Deep–Real or Complex–Matrices and Their Software Implementation.  _arXiv preprint arXiv:2009.10071_ (2020).
  * Roberts and Roberts (2017) Lucas Roberts and Denisa Roberts. 2017.  An expectation maximization framework for Yule-Simon preferential attachment models.  _arXiv preprint arXiv:1710.08511_ (2017).
  * Shui et al. (2022) Zeren Shui, Ge Liu, Anoop Deoras, and George Karypis. 2022\.  Sequence-graph duality: Unifying user modeling with self-attention for sequential recommendation.  (2022).
  * Song et al. (2019) Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019.  Autoint: Automatic feature interaction learning via self-attentive neural networks. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 1161–1170.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In _Proceedings of the 28th ACM international conference on information and knowledge management_. 1441–1450.
  * Tan and Le (2019) Mingxing Tan and Quoc V. Le. 2019.  EfficientNet: rethinking model scaling for convolutional neural networks.  _CoRR_ abs/1905.11946 (2019).  arXiv:1905.11946 http://arxiv.org/abs/1905.11946
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2017) Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017\.  Deep & cross network for ad click predictions.  In _Proceedings of the ADKDD’17_. 1–7.
  * Wang et al. (2021) Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021.  DCN v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In _Proceedings of the Web Conference 2021_. 1785–1797.
  * Xu et al. (2022) Jiajing Xu, Andrew Zhai, and Charles Rosenberg. 2022.  Rethinking personalized ranking at Pinterest: An end-to-end approach. In _Proceedings of the 16th ACM Conference on Recommender Systems_. 502–505.
  * Yang et al. (2020) Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaoming Wang, Taibai Xu, and Ed H Chi. 2020\.  Mixed negative sampling for learning two-tower neural networks in recommendations. In _Companion Proceedings of the Web Conference 2020_. 441–447.
  * Yi et al. (2019) Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. 2019.  Sampling-bias-corrected neural modeling for large corpus item recommendations. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 269–277.
  * Yu et al. (2021) Lucia Yu, Ethan Benjamin, Congzhe Su, Yinlin Fu, Jon Eskreis-Winkler, Xiaoting Zhao, and Diane Hu. 2021.  Personalization in e-commerce product search by user-centric ranking.  _KMECommerce Workshop Held at WWW ’21_ (2021).
  * Zhai and Wu (2018) Andrew Zhai and Hao-Yu Wu. 2018.  Making classification competitive for deep metric learning.  _CoRR_ abs/1811.12649 (2018).  arXiv:1811.12649 http://arxiv.org/abs/1811.12649
  * Zhai et al. (2019) Andrew Zhai, Hao-Yu Wu, Eric Tzeng, Dong Huk Park, and Charles Rosenberg. 2019.  Learning a unified embedding for visual search at Pinterest.  _CoRR_ abs/1908.01707 (2019).  arXiv:1908.01707 http://arxiv.org/abs/1908.01707
  * Zhai et al. (2016) Shuangfei Zhai, Keng-hao Chang, Ruofei Zhang, and Zhongfei Mark Zhang. 2016.  Deepintent: Learning attentions for online advertising with recurrent neural networks. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 1295–1304.
  * Zhang et al. (2022) Buyun Zhang, Liang Luo, Xi Liu, Jay Li, Zeliang Chen, Weilin Zhang, Xiaohan Wei, Yuchen Hao, Michael Tsang, Wenjun Wang, et al. 2022\.  DHEN: A deep and hierarchical ensemble network for large-scale click-through rate prediction.  _arXiv preprint arXiv:2203.11014_ (2022).
  * Zhang et al. (2020) Mengqi Zhang, Shu Wu, Meng Gao, Xin Jiang, Ke Xu, and Liang Wang. 2020.  Personalized graph neural networks with attention mechanism for session-aware recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ (2020).
  * Zhang et al. (2021a) Weinan Zhang, Jiarui Qin, Wei Guo, Ruiming Tang, and Xiuqiang He. 2021a.  Deep learning for click-through rate estimation.  _arXiv preprint arXiv:2104.10584_ (2021).
  * Zhang et al. (2021b) Yang Zhang, Dong Wang, Qiang Li, Yue Shen, Ziqi Liu, Xiaodong Zeng, Zhiqiang Zhang, Jinjie Gu, and Derek F Wong. 2021b.  User retention: A causal approach with triple task modeling.. In _IJCAI_. 3399–3405.
  * Zhao et al. (2018) Xiaoting Zhao, Raphael Louca, Diane Hu, and Liangjie Hong. 2018\.  Learning item-interaction embeddings for user recommendations.  _arXiv preprint arXiv:1812.04407_ (2018).
  * Zhao et al. (2019) Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. 2019.  Recommending what video to watch next: a multitask ranking system. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 43–51.
  * Zhou et al. (2019) Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019.  Deep interest evolution network for click-through rate prediction. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 5941–5948.
  * Zhou et al. (2018) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018\.  Deep interest network for click-through rate prediction. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 1059–1068.

##  Appendix A Additional Information for Reproducibility

###  A.1. Candidate Generation for Personalized Ad Ranking: Ads Retrieval

Since performing real-time inference for adSformer CTR and PCCVR models for
the full set of ad listings would be prohibitively expensive, we briefly
describe the candidate retrieval system that produce up to 600 candidate
listings to re-rank. We employ a hybrid lexical and pretrained representation-
based retrieval system, designed to produce maximally relevant results for all
types of user queries. Lexical retrieval. In this first retrieval pass, we use
”static” CTR and PCCVR models that are batch inferenced offline daily. The
predicted scores, along with every listing’s title and tags, as well as the ad
budget remaining for each listing’s campaign, are indexed in a sharded
inverted index search database running Apache Solr222https://solr.apache.org/.
At query time, we fetch up to 1000 listings from each shard based on title and
tag matching and that have ad budget remaining, and we calculate a ranking
score which is similar to the value function. We also boost listings by a
taxonomy matching prediction score, obtained from a separately trained and
batch inferenced BERT (Devlin et al., 2018) model, as well as additional
business logic. Pretrained representation-based retrieval. We currently
leverage representations from a two-tower(query and listing towers, textual
inputs only) model built for organic search retrieval at Etsy, similarly to
Amazon’s (Nigam et al., 2019b). We have plans to test multimodal AIR
representations next. Given the trained two-tower model, we host the query
tower online for live inference and batch inference the representations from
the listing tower, which are indexed into an Approximate Nearest Neighbor
(ANN) database. A large challenge we faced adopting off-the-shelf ANN
solutions for an ads use case is that the index quickly becomes stale as
sellers’ budgets deplete throughout the day, ANN options on the market do not
allow for filtering on attributes that change with such high frequency. We are
experimenting with promising new in-house infrastructure that incorporates
integrated filtering directly into the ANN.

###  A.2. Training the adSformer CTR and PCCVR: Additional Details

We run extensive offline experiments and ablation studies and omit many of the
results for brevity. For example, we study the vocabulary size effect when
encoding the ADPM representations of listing ID sequences and give results in
8. All three components of the ADPM-learned, pretrained and adSformer encoded
representations-use the same vocabulary. We gain minimally by increasing
vocabulary size after a certain point.

Table 8. Performance metrics for two selected adSformer CTR models with vocabulary sizes of 600K and 750K vs baseline NN, corresponding to training dynamics in Figures 9 and 10 from Comet Model | PR AUC (lift range) | ROC AUC (lift range) | Train time (machine)
---|---|---|---
adSformer CTR 750 (fuchsia) | 0.0425 (+6.15% to +6.65) | 0.73954 (+1.94 to +2.36%) | 11 hrs (A100)
adSformer CTR 600 (purple) | 0.0423 (+5.00% to +5.75%) | 0.7391 (+1.83% to +1.9%) | 10 hrs (A100)
baseline NN CTR (orange) | 0.0400 | 0.7258 | 16 hrs (P100)

Note that throughout this article the offline comparison numbers may vary for
the same model variant due to variability inherent in sampled datasets. For
reproducibility, Table 9 gives all hyperparameters and other machine learning
choices for the winning adSformer CTR.

Table 9. adSformer CTR: 600K vs 750K vocabulary size Config | adSformer750 | adSformer600 | baseline NN
---|---|---|---
Num Epochs | 1 | 1 | 3
Learning Rate | 0.002 | 0.001 | 0.00061
Learning Rate Scheduler | cosine | cosine | cosine
Optimizer | Adam | Adam | LazyAdam
Adam $\beta_{1}$ | 0.9 | 0.9 | 0.9
Adam $\beta_{2}$ | 0.999 | 0.999 | 0.999
Adam $\epsilon$ | 1e-08 | 1e-08 | 1e-08
Dropout Hidden Layers | 0 | 0 | 0
Batch Norm Hidden Layer | yes | yes | yes
Cross Module (DCN) | yes | yes | no
Batch Size (train) | 8192 | 8192 | 8192
Batch Size (validation) | 500 | 500 | 500
Machine Type (training) | A100 | A100 | P100
Machine Type (evaluation) | A100 | P100 | T4
Loss | BinaryCrossEntropy | BinaryCrossEntropy | BinaryCrossEntropy
Num Parameters (trainable) | 708,504,031 | 690,842,559 | 233,709,623
Num Parameters (total) | 897,499,741 | 844,564,093 | 243,726,397

Table 10 gives the dataset choices for the winning adSformer CTR and PCCVR.
For the CTR we sample 50 percent of non-clicked impressions in the training
dataset but perform a random sampling for the validation set, so that
validation and test sets are representative of the production data seen in the
wild. The PCCVR is trained on clicked impressions but evaluated on a
representative test sample, similarly to the adSformer CTR.

Table 10. Train and validation datasets characteristics for adSformer CTR and PCCVR Model | Config | Training | Validation
---|---|---|---
CTR | Num Examples |  300mln |  8.5mln
CTR | Num Consecutive Days | 30 | 1
CTR | Sampling Type | 50-50 negative sampling | random sampling
PCCVR | Num Examples |  200mln |  8.5mln
PCCVR | Num Consecutive Days | 21 | 1
PCCVR | Sampling Type | click filtered | random sampling

Furthermore, Table 11 illustrates gains in PCCVR performance from increased
training dataset size after including the ADPM.

Table 11. Increased performance metrics for adSformer PCCVR model vs the baseline NN production model on the same validation dataset for different training window sizes Training Window | ROC-AUC Lift | PR-AUC Lift | Train Duration
---|---|---|---
2 weeks Baseline | 0.0% | 0.0% | 6 hrs
3 weeks Baseline | +0.01% | +0.11% | 8 hrs
2 weeks adSformer | +1.28% | +9.61% | 11 hrs
3 weeks adSformer | +1.58% | +12.07% | 16 hrs

To determine optimal hyperparameters, we combine random search, deep learning
expert manual selection, and Bayesian hyperparameter search, depending on the
hyperparameter of interest. We visualize training dynamics curves in Comet
(Comet.com, 2021) to guide our model choices. For example, in Figures 9 we see
that both adSformer CTR model flavors (with listing ID vocabularies of 750K
and 600K respectively) trained for one epoch achieve slightly lower training
losses and significantly lower validation losses when compared to the baseline
neural network trained for three epochs. Table 9 gives the complete
configuration for these three compared models. In Table 8 we see the final
metric lifts and training times achieved on the same train and validation
datasets (described in Table 10). Figure 10 helps visually understand the lift
achieved by the adSformer CTR variants in validation AUC metrics.

Figure 9. adSformer CTR model selection: train and validation loss curves.
Lower is better. Source: Comet. Figure 10. adSformer CTR selected models vs
baseline: ROC AUC and PR AUC curves. Higher is better. Source: Comet. Figure
11. adSformer PCCVR eval metrics during training: ROC AUC and PR AUC per step.
Higher is better. Source: Comet.

In additional ablation studies we vary multiple hyperparameters, for example
the number of heads in the adSformer encoder and the listing representation
size in ADPM’s learned component. In Table 12 we see that removing the average
pooled learned listing representation from ADPM in adSformer CTR leads to a
drop in ROC AUC but an increase in PR AUC. We have a better idea of online to
offline correlations between offline test ROC AUC and online CTR, so we select
the increased ROC AUC model. Lowering the size of the learned listing
representations has a similar effect. The addition of one head to the
adSformer encoder component of ADPM increases both metrics and so does the
removal of dropout.

Table 12. Additional ablations studies. Offline metric lift in comparison to the baseline for the adSformer CTR. Config | Test ROC AUC Lift | Test PR AUC Lift
---|---|---
All Listing Pretrained + Listing Learned | +2.26% | +6.17%
All Listing Pretrained + No Listing Learned | +2.18% | +6.20%
adSformer Encoder 2 Heads | +2.13% | +6.42%
adSformer Encoder 3 Heads | +2.14% | +6.44%
Learned Listing Size 16 | +2.10% | +6.50%
Learned Listing Size 32 | +2.14% | +6.44%
Learned Listing Size 64 | +2.04% | +6.14%
adSformer Encoder 0 Dropout | +2.14% | +6.44%
adSformer Encoder 0.1 Dropout | +2.10% | +6.34%

We performed similar ablation studies for the adSformer PCCVR model and Table
13 depicts the final hyperparameter settings for the winning adSformer PCCVR.

Table 13. adSformer PCCVR hyperparameters and other algorithmic choices Config | adSformer PCCVR | baseline NN
---|---|---
Num Epochs | 2 | 2
Learning Rate | 0.0001 | 0.0001
Learning Rate Scheduler | cosine | cosine
Optimizer | Adam | LazyAdam
Adam $\beta_{1}$ | 0.9 | 0.9
Adam $\beta_{2}$ | 0.999 | 0.999
Adam $\epsilon$ | 1e-08 | 1e-08
Dropout Hidden Layers | 0.02 | 0
Batch Norm Hidden Layer | yes | yes
Cross Module (DCN) | yes | yes
Batch Size (train) | 500 | 500
Batch Size (validation) | 500 | 500
Machine Type (training) | P100 | P100
Machine Type (evaluation) | P100 | T4
Loss | BinaryCrossEntropy | BinaryCrossEntropy
Num Parameters (trainable) | 549,005,007 | 322,975,106
Num Parameters (total) | 704,469,589 | 332,978,320
Training Dataset Size |  200M |  135M

Finally, Table LABEL:table:final presents one more set of the winning
adSformer CTR and PCCVR offline performance results against production
baseline NN.

Table 14. Performance metrics for the selected adSformer CTR and PCCVR models vs NN baselines. Results extracted from the daily training and evaluation jobs. Model | PR AUC | ROC AUC | Train time
---|---|---|---
Baseline CTR NN | 0.0411 | 0.719 | 14 hrs (1 P100))
Baseline PCCVR NN | 0.0602 | 0.824 | 8 hrs (1 P100))
adSformer CTR | +6.65% | +2.16% | 11 hrs (1 A100)
adSformer PCCVR | +12.70% | +1.81% | 16hrs (1 P100)

###  A.3. Additional Sampling Biases Discussion

In addition to position debiasing, we are actively considering the practical
implications of other selection biases, such as the conditionality of our
PCCVR prediction target on the CTR target. The personalized PCCVR model
predicts $P(y_{PCCVR}=1|y_{CTR}=1,x)$ effectively ignoring the sample space of
non-clicked purchased items in the personalized search ads space, the case of
$P(y_{PCCVR}=1|y_{CTR}=0,x)$. In preliminary evaluations of potential explicit
treatment for this selection bias, for example by using an inverse propensity
weighting in a multitask CTR-PCCVR formulation similarly to (Zhang et al.,
2021b), or the approach in (Ma et al., 2018a), we determined that the metric
improvements do not sufficiently reward an increase in machine learning system
complexity. We plan to revisit sampling biases in the future, possibly as part
of a multitask learning paradigm.

###  A.4. Additional Calibration Information

After the uncalibrated model is trained, we use the raw logits as a feature to
the calibration layer, which is a simple logistic regression model trained on
the validation set reflecting the production data. The calibration layer
learns a single parameter $A$ and bias term $B$, and outputs
$P\left(y=1|x\right)={\frac{1}{1+\exp(-(Af(x)+B))}}$. The calibration mapping
function is isotonic (monotonically increasing), preserving the relative
ordering of predictions. As a sanity-check, the evaluation metric, AUC, should
be identical between the calibrated and uncalibrated models. We evaluate
miscalibration by monitoring the Expected Calibration Error (ECE) and
Normalized Cross Entropy (NCE). ECE aims to measure the difference in
expectation between confidence and accuracy. We approximate the ECE by
partitioning predictions into $M$ equally-spaced bins and taking the weighted
average of the bins’ $accuracy/confidence$ difference.

###  A.5. Visual Representations Additional Information

####  A.5.1. Multitask Learning of Visual Representations

We saw best results by training a model on the following four
datasets/classification tasks: 1. listing images to top level taxonomy. 2.
listing images to fine grained taxonomy. 3. listing images to seller-input
primary color. 4. user-uploaded review photos to fine grained taxonomy. For
the top level taxonomy and primary color tasks we sampled 16K images for a set
of 15 labels each. For fine grained taxonomy we sampled 200 images for a set
of 1000 taxonomy nodes. We replaced the final convolutional layer of the
EfficienetNetB0 backbone with a 256-dimension layer to output representations
of the same size. The model was trained by first freezing the backbone and
training only classification heads for one epoch using a 0.001 learning rate,
and then unfreezing the final 50 layers of the backbone and training for an
additional 8 epochs using the same learning rate. We used Adam optimizer with
values of 0.9, 0.999 and 1.0e-7 for beta1, beta2 and epsilon respectively.

Figure 12. Additional example of candidate retrieval using multitask visual
representations. Left column: query image used. Right column: top-5 nearest
neighbors retrieved

####  A.5.2.  Ads Computer Vision: Visually Similar Ads Additional Information

The listing-to-listing experience does not require real-time predictions, so
we implemented a daily batch inference pipeline to generate image
representations for our entire listing inventory. Our pipeline first extracts
the primary image for each Etsy listing, then passes those forward through the
frozen multitask visual representation model to generate representations.
Representations for all listings are saved to a scalable key-value store.
representations are also filtered down by active ad campaigns. These candidate
representations are indexed into an Approximate Nearest Neighbor (ANN)
inverted file (IVF) index. IVF approximates a nearest neighbor search by first
splitting the representation space to N clusters, and at inference time only
searching the K nearest clusters. Thus we can efficiently search the large
space of representations.

####  A.5.3.  Ads Computer Vision: Search by Image

Another application of the multitask visual representations was a brand new
search by image shopping experience, built by the Etsy Ads team, and available
on the BOE app. Users can search using photos taken with their phone. We
leverage the multitask dataset sampler to train on an additional task to
classify review photos to their corresponding listing’s taxonomy. Review
photos are photos of purchased items taken “in the wild” by users. Our
hypothesis is that these images should serve as a proxy for user-taken query
images.

The search by image experience requires generating the representation of the
query image on the fly. To enable fast inference on a large CNN such as the
visual representations model, we partnered with the ML platform team to deploy
the representation model as the first GPU service in production at Etsy.

Similarly to the visually similar ads module, the candidate representations of
the search by image retrieval system are pre-computed by generating
representations for primary listing images on a recurring basis. However,
since we are serving ads as well as organic results, the ANN is indexed with
the full inventory of nearly 100M active Etsy listings. At query time, the
user-uploaded image is inferenced on the GPU-backed model to compute the
representation, which is used to search the ANN index for the most visually
similar listings.

###  A.6. Multimodal Representations: Learning AIR Additional Information

In addition to parameterizing the ADPM module in adSformer CTR use-case, we
use AIR representations to retrieve recommendation-style ads for listing-to-
listing requests, across various types of pages on both mobile and web. For
example, the sash of sponsored recommendations appearing at the bottom of the
listing page is backed by AIR representations. In listing-to-listing requests
candidates are retrieved using a query representation generated from the
features of the viewed listing. Typically, we refer to the listing used to
generate the query representation as the source listing.

####  A.6.1. Additional Training Details

Our final representation model was trained using 30M of historical click
pairs, from a look-back window of 45 days. For training we used the Adam
optimizer, with a learning rate, beta1, beta2, and epsilon values of 0.0001,
0.9, 0.999, and 1e-7, respectively. We trained using a batch size of 4096, and
sample 1500 negative labels to calculate the loss.

####  A.6.2. Evaluating AIR: Online Results

Online A/B experiments were run against the control of neural IR
representations extracted from the listing tower of a model trained on search
purchases (a less relevant objective for ads). Two treatments were tested to
measure the effect of different dimensions of the input text representations,
128d and 256d, since the text representation lookup table makes for the
majority of trainable parameters in the overall AIR model. Table 15 show
online CTR gains.

Table 15. Online AIR A/B experiment results showing lift in CTR. fastText Dimension | CTR Lift
---|---
128d | +2.79%
256d | +2.88%

###  A.7. Text Representations Additional Information

Raw text inputs are first preprocessed to standardize casing and punctuation,
mask stop-words and numbers, and remove extraneous spaces and symbols. To
reduce the size of the final lookup table we filter out tokens which have less
than 10 occurrences. To maintain relevance to the ad use case, we keep tokens
which appeared at least once in the last 30 days of ad interactions even if
they occur less than 10 times in the data. The final lookup table contains
roughly one million tokens.

We trained the skip-gram word representations for five epochs. We used a
learning rate of 0.05, a context window of five, and a minimum and maximum
character n-grams lengths of three and six respectively. We trained using
negative sampling with five sampled negatives. We experimented with outputting
both 128 and 256 dimension representations. We have found that the larger
representations perform slightly better in most downstream use cases. However,
the number of trainable parameters of the lookup table grows significantly
with larger representations, and thus requires more expensive infrastructure
to fine tune during training of downstream task, which we have seen in the
adSformer CTR case.

###  A.8. Skip-Gram Representations Additional Information

The objective of the model is to predict, given a listing $l_{i}$ in the
corpus, the probability that another listing $l_{i+j}$ will be observed within
a fixed-length contextual window. The probability $p(l_{i+j}|l_{i})$ is
defined mathematically according to the softmax formula

(14)
$p(l_{i+j}|l_{i})=\frac{\exp(\mathbf{v}_{l_{i}}^{\mathsf{T}}\mathbf{v}_{l_{i+j}}^{\prime})}{\sum_{k=1}^{|\mathcal{V}|}\exp(\mathbf{v}_{l_{i}}^{\mathsf{T}}\mathbf{v}_{l_{k}}^{\prime})},$

where $\mathbf{v_{l}}$ and $\mathbf{v_{l}^{\prime}}$ are the input and output
vector representations of a listing $l$, respectively, and $\mathcal{V}$ is
the vocabulary of unique listings.

In our training data, we use two months’ worth of user sessions, and we
upsample sessions that included a purchase at a rate of 5:1. We chose
representation dimension $d=64$ after observing that higher dimensions tend to
improve representation quality but with diminishing returns and with trade
offs in model training and inference cost. We use a context window size of
five. Interestingly, increasing the window size had a positive impact for
negative sampling models but had mixed results for hierarchical softmax
models.

As a gut check on representation quality and to better understand what signals
our representations were learning, we compared the cosine similarity of groups
of listings segmented by various listing attributions. For example, listings
which are co-viewed in a session are likely to belong to the same taxonomy
node, so we would expect listings with the same taxonomy to have a higher
cosine similarity than those with different taxonomies. Figure 13 shows that
our skip-gram representations learned with hierarchical softmax nearly
perfectly captured the listing taxonomy attribute.

Figure 13. The skip-gram listing representation model nearly perfectly encodes
listing taxonomy.

To evaluate our representations, we ran batch data inference jobs with Apache
Beam and Dataflow to compute the search ranking and attribute cosine
similarity metrics, which we visualized in Colab notebooks.