Building a Scalable, Effective, and Steerable Search and Ranking Platform
Marjan Celikik Jacek Wasilewski Ana Peleteiro Ramallo Alexey Kurennoy Evgeny Labzin Danilo Ascione Tural Gurbanov Géraud Le Falher Andrii Dzhoha Ian Harris















marjan.celikik@zalando.de

  Zalando SE
  Berlin
  Germany


jacek.wasilewski@zalando.de

  Zalando SE
  Berlin
  Germany



ana.peleteiro.ramallo@zalando.de

  Zalando SE
  Berlin
  Germany



alexey.kurennoy@zalando.ie

  Zalando SE
  Berlin
  Germany



evgeny.labzin@zalando.de

  Zalando SE
  Berlin
  Germany



danilo.ascione@zalando.de

  Zalando SE
  Berlin
  Germany



tural.gurbanov@zalando.de

  Zalando SE
  Berlin
  Germany



geraud.le.falher@zalando.de

  Zalando SE
  Berlin
  Germany



andrew.dzhoha@zalando.de

  Zalando SE
  Berlin
  Germany



ian.harris@zalando.ie

  Zalando SE
  Berlin
  Germany







§ ABSTRACT 

Modern e-commerce platforms offer vast product selections, making it difficult for customers to find items that they like and that are relevant to their current session intent. This is why it is key for e-commerce platforms to have near real-time scalable and adaptable personalized ranking and search systems. While numerous methods exist in the scientific literature for building such systems, many are unsuitable for large-scale industrial use due to complexity and performance limitations. Consequently, industrial ranking systems often resort to computationally efficient yet simplistic retrieval or candidate generation approaches, which overlook near real-time and heterogeneous customer signals, which results in a less personalized and relevant experience. Moreover, related customer experiences are served by completely different systems, which increases complexity, maintenance, and inconsistent experiences.

In this paper, we present a personalized, adaptable near real-time ranking platform that is reusable across various use cases, such as browsing and search, and that is able to cater to millions of items and customers under heavy load (thousands of requests per second). We employ transformer-based models through different ranking layers which can learn complex behavior patterns directly from customer action sequences while being able to incorporate temporal (e.g. in-session) and contextual information. We validate our system through a series of comprehensive offline and online real-world experiments at a large online e-commerce platform, and we demonstrate its superiority when compared to existing systems, both in terms of customer experience as well as in net revenue. Finally, we share the lessons learned from building a comprehensive, modern ranking platform for use in a large-scale e-commerce environment.







































[500]Computing methodologies Neural networks
[500]Information systems Recommender systems









Building a Scalable, Effective, and Steerable Search and Ranking Platform
    Ian Harris
    September 9, 2024
=========================================================================





§ INTRODUCTION


With the vast choice of items available in e-commerce, finding relevant content has become increasingly challenging. This is why personalization is crucial in showcasing products that align with customers' preferences and session intent. Consequently, large e-commerce companies such as Zalando, one of Europe's largest online fashion e-commerce platforms, are heavily invested in the development of advanced ranking systems that can more effectively cater to customer needs and tastes. 

In major e-commerce platforms, deploying larger and more powerful models poses challenges due to the complexities involved in handling high traffic loads in production as these systems must be capable of serving thousands of requests per second across millions of items and customers. Furthermore, browsing and searching the catalog (refer to <ref>) represent primary methods through which customers discover products, whether for immediate purchase or inspiration. However, distinct yet related customer experiences, such as search and browse functions, are often powered by entirely separate systems <cit.>. This separation increases modeling complexity, increases maintenance costs, and may result in inconsistent customer experiences.



The ability to provide a personalized, dynamic, scalable, and efficient ranking platform that can be employed across various experiences has become critical to driving customer engagement and business value in e-commerce. We achieve this by building a ranking platform grounded in four key design principles: 1) composability - our platform consists of multiple state-of-the-art ranking models and candidate generators working in orchestration; 2) scalability - ensured by vector-based indexing and scoring; 3) shared real-time serving infrastructure and 4) steerable ranking that can adapt to varying customer preferences and business objectives. 



Our platform is able to support the integration of multiple models through vertical layering, and horizontal integration, blending the outputs of various models or other candidate sources. This enables scalability, independence, and ability to mix various content types to cater to specific use cases as well as building ranking ensembles by combining the outputs of multiple models. The platform's scalability is driven by employing a vector store in its candidate generation stage, facilitating efficient indexing, scoring, and retrieval which are crucial for managing a growing catalog of items and number of customers. To this end, we compute dense representations of customer behaviors, contexts, and item inputs in a common embedding space. Built on a foundation that allows for near real-time scoring and computation of customer and item representations, our platform dynamically adapts the ranking in all layers to recent customer changes (including candidate generation) which increases the probability of discovering relevant items <cit.>. Moreover, utilizing efficient transformer-based model architectures across all layers allows sharing of the serving infrastructure, which in turn reduces engineering complexity, increases re-usability, and helps avoid inconsistency between training and serving phases, which is a common issue in machine learning engineering systems <cit.>. 









Many works both from industry and academia do not try to capture the entire customer journey and side information such as item metadata, customer profile, and contextual inputs <cit.>. However, it is known that deep learning recommender systems live up to their full potential only when numerous features of heterogeneous types are included <cit.>. We demonstrate that incorporating heterogeneous inputs that capture the full spectrum of the customer journey (customer behavior, content-based data, local and global contextual and temporal information) is crucial for ranking quality and even diversity. All of this helps provide contextually relevant results for both in-session browsing, where the customer is actively engaged in shopping, and cross-session scenarios, where the customer returns to the platform after a break with a potentially new shopping intent. To ensure a more streamlined and effective data integration process into the ranking models, contrary to common approaches <cit.> that employ additional architectures, our approach utilizes the same self-attention mechanism to efficiently fuse all input data types. 



Many ranking systems in the literature rely on pre-trained items and customer embeddings. Our experiments reveal that similarly to NLP tasks <cit.>, the effectiveness of our models significantly increases when the pre-trained input item embeddings are further fine-tuned on the ranking task. Notably, we show that if these embeddings are not continuously trained, the candidate generation model shows substantially less customer engagement. To address the item cold-start problem, we introduce epsilon-greedy exploration by blending fresh items from additional candidate sources into the organic ranking. We address the customer cold-start by leveraging customer context and in-session capabilities and data.





The key contributions of our work are as follows: 




  * We present a comprehensive, flexible, scalable ranking platform able to provide near real-time inference in all ranking layers in high-load systems, building on state-of-the-art models and standard design patterns that can be applied in various search and ranking use cases;



  * We propose novel modifications of existing state-of-the-art ranking model architectures allowing more efficiency without loss of quality. With this we demonstrate that sequence-based models can successfully replace traditional ranking systems in all ranking phases and significantly improve performance; 


  * We present extensive experimentation, including both online and offline. We demonstrate that our proposed system not only significantly outperforms existing solutions by a wide margin (10-40% improvement in offline evaluation metrics, 15% combined engagement uplift, and +2.2% combined net revenue in 4 online A/B tests) but that it also scales effectively under heavy load. 




It's crucial to note that although the experimental results presented are specific to the e-commerce sector, the methodologies, algorithms, and infrastructure discussed are designed for adaptability and can be extended to domains beyond e-commerce. The system has been deployed and operational for the last 12 months in one of the largest e-commerce platforms in Europe. It has successfully replaced numerous legacy systems and it is serving millions of customers per day and handling thousands of RPS.

The remainder of this paper is organized as follows: Related work is reviewed in <ref>. Details on the overall system architecture and design decisions are elaborated in <ref>. Sections <ref>, <ref> and <ref> describe the candidate generation, ranking and policy layers. The experimental results (both offline and online) are presented in <ref>. Finally, we present the conclusions in <ref>.



§ RELATED WORK



Thanks to their advantages over traditional deep-learning-based models, sequence-based recommender systems <cit.> in their two flavors of language modeling (CLM and MLM) <cit.> have gained wide traction. These systems have proven powerful in modeling customer behavior as a sequence of actions due to their capability to 1) capture both short-term and long-term interests <cit.>; and 2) the ability to compute complex feature interactions. However, most existing works use public datasets that in certain cases are not even adequate for sequential recommendation tasks <cit.> and many of focus only on offline experiments, with only a few works reporting actual customer impact through end-to-end A/B testing in large scale environments <cit.>. Our paper extends this line of work and demonstrates the usefulness of these models in real-world applications that include personalized item browsing and search. Moreover, to the best of our knowledge, this is the first work to apply transformer networks as part of the two-tower architecture for learning embeddings for ranking use cases that were originally introduced in the line of work of Google <cit.>. Unlike related approaches <cit.>, our approach to scoring candidate items does not suffer from performance issues caused by a long sequence length <cit.> or limiting the input embedding size due to concatenating the candidate item embeddings with the action sequence embeddings. Unlike <cit.>, we do not observe a significant drop in ranking performance and feed only the average embedding of the candidates as a fixed input position into the transformer network. 

Moreover, only a few published studies <cit.> include heterogeneous inputs such as context and content-based features, which help address the cold-start problem and address data sparsity by improving generalization. For example, <cit.> considers customer profiles and contextual features by using wide & deep learning (WDL), which relies on a concatenation of signals in the output of the network, making it inadequate to capture powerful feature interactions. <cit.> employs deep and cross-network (DCN) on top to explicitly model feature-crosses, which significantly increases the number of parameters of the network. 







§ SYSTEM DESIGN





Designing a highly performant, scalable, and steerable ranking platform entails multiple challenges and complex choices. The existing literature often focuses only on subsets of them <cit.>, while this section aims to navigate through them holistically. We describe foundational design principles and provide an overview of our system architecture and components (see <ref>). The presented design is generalizable and applicable to other retrieval and ranking use cases and setups.

Composability and orchestration of multiple models. Our platform supports the integration of various models either "vertically" for layered ranking and retrieval, or "horizontally" by blending outputs from different models or candidate generators. It features a multi-layered architecture that enhances scalability and reduces inter-dependencies. The initial layer retrieves relevant candidates from multiple candidate generators, possibly generating different item or content types for use cases such as feeds. Each candidate generator typically entails a (lightweight) ranking model. Subsequent layers refine these selections; the ranking layer applies heavy personalized models for ranking pre-selected items of possibly different types. The policy layer ensures compliance with business or product specifications. Blending strategies mix outputs to suit specific needs, such as combining different content types in desired proportions (e.g. product, outfits, videos) or balancing popular, fresh, and personalized content <cit.>. Model blending is also the blueprint for building ranking ensembles, where outputs of multiple models are combined either by score weighting or meta rankers.



Scalable platform. The multi-layered model architecture also allows us to achieve high scalability. The more accurate, but computationally heavier, part of the ranking is performed on later layers only on a small subset of candidates obtained from the less accurate but more computationally efficient candidate generator layers. Thus, the highest scalability requirements are placed on the candidate generator. Scalability of the candidate generator layer is achieved through a vector store allowing efficient indexing, scoring, and personalized retrieval capabilities to scale for a large and growing item catalog and customer base. Our platform includes infrastructure to compute customer vectors based on past actions and context (such as country, browsing category or search query, etc.) and item vectors. The item vectors are indexed in an internally hosted vector store that supports efficient approximate nearest neighbors search and retrieval. 

Shared near real-time serving infrastructure. Similarities in model types across layers allow for shared training datasets and much of the serving infrastructure, enhancing efficiency. Deep learning minimizes feature engineering complexity by integrating embedding mappings directly into the model graph, ensuring consistency between training and serving phases. Shared online feature stores across ranking layers enable effective input caching for each request, simplifying engineering efforts.

Steerable ranking. Our system's flexibility allows for external adjustments to ranking objectives to align with business goals like customer satisfaction or profitability. It also supports diverse content types through its candidate generators and mixing components and integrates business heuristics in the policy layer. 

In the following sections, we present in more detail the main components of our multi-layered system, i.e., the candidate generator layer, the ranking layer, and the policy layer.












§ CANDIDATE GENERATION



The objective of our candidate generation layer is to generate personalized item candidates from the item catalog for each individual customer efficiently in near real-time. According to our findings, a personalized candidate generator is essential for the performance of the overall ranking system (see <ref> for details). 

We follow the classical two-tower approach <cit.>, where the customer tower processes historical customer action sequences and contextual data to generate a customer embedding while the item tower is responsible for generating item embeddings. These embeddings are then combined by using dot product to generate a score per item as shown in <ref> (theoretical justification about the expressiveness of the two-tower model is provided in the Appendix, <ref>).

Although trained together, the towers are deployed and operated independently. The item tower generates item embeddings that are indexed in a vector store for efficient ranking and retrieval. The customer tower is invoked to generate customer embeddings each time a customer accesses the platform. The freshly computed customer embeddings are then used to find items with similar embeddings in the nearest-neighbors index.

We formulate the retrieval task as an extreme multi-class classification problem <cit.> with softmax optimization. Every item in the vocabulary represents a distinct class, and the goal is to accurately predict the class of the next item a customer will interact with. We employ sampled softmax loss with log-uniform sampling, with negative classes that correspond to 0.42% of the total number of classes. This loss outperformed other loss functions and negative sampling strategies. Specifically, we experimented with “generalized” binary cross entropy <cit.> and popularity sampling with varying numbers of negatives as well as sampling hard negatives from the category of items the customer was browsing before acting. 

To compute a customer embedding from the customer action sequence and the context, we employ a transformer encoder and use causal language modeling (CLM) as in <cit.>, processing each customer sequence once per epoch. We predict the subsequent item in the sequence while preventing backward attention by using a causal mask.

For simplicity, items in the item tower in <ref> are represented by using a single embedding that jointly encodes product metadata (brand, category, material, etc.) and visual cues. It is worth mentioning that in this common setting, trainable input embeddings on the ranking task performed substantially better in both offline and online tests. Details on how different types of input signals from the customer journey data (e.g. contextual and customer action sequence data) are encoded in the customer tower of the model are captured in Section <ref>.





§ RANKING LAYER



The objective of the ranking layer is to rank items returned by the candidate generation phase by their relevance to the customer by using a powerful ranking model. We model this task as a pointwise multi-task prediction problem, where we predict the probability of the customer performing any of the following positive actions on a candidate item: click, add-to-wishlist, add-to-cart, purchase given the context and their past behavioral data. If a candidate item is associated with any of these positive actions, we consider it a positive item, otherwise, it is considered a negative. 



 §.§ Model Architecture


<ref> depicts the architecture of the model in the ranking layer. It consists of four main parts: embedding layer, item candidate embedding, customer and context embedding computed via a self-attention mechanism, a prediction head for each of the target action types, and a shallow position branch used for position debiasing. 



For each of the target actions, we define a prediction head, which takes customer and candidate item representations as inputs. The score, for a given target action, is obtained by computing a dot product between the customer and context embedding and all candidate item embeddings in parallel, after passing them through a FFN. A sigmoid function is used to normalize the score and interpret it as a probability. During training, each prediction head contributes equally to the loss, while at serving we produce the final ranking by weighting the scores of each prediction head. The weights are dynamically configurable and determined analytically depending on the customer touch point. 

While models based on list-wise loss directly optimize the ranking objective, the downside is that the predictions from such models do not inherently correspond to probabilities. This lack of calibrated probabilities complicates the multi-objective optimization required for business steering. To this end, we adopt a pointwise loss in our multi-task learning setup. Our underlying assumption is that the tasks share a common internal representation, thereby improving generalization performance through the transfer of knowledge. We employ a cross-entropy loss function utilizing binary relevance labels defined as follows:

    ℒ = - 1/N∑_n=1^N∑_h=1^H(
            y_n^h log(f^h(x_n)) + (1 - y_n^h)(log(1 - f^h(x_n)))
        ),

where N is the number of training examples, H is the number of heads (tasks), x_n input for training example n, y_n^h is the target label {0, 1} for training example x_n for task h and f^h(x_n) is the output probability of head h.



The customer representation vector is generated by encoding and passing the context and the customer sequence actions through a standard transformer encoder with a look-ahead mask. For efficiency, we use only the output of the last position as the customer and context embedding and pass it to the prediction head. The positional encoding is omitted as it has not been proven effective in this as well as in other works <cit.>.

Unlike <cit.>, we opt out from concatenating the candidate item embeddings as separate positions in the encoder’s input since we have found this to be a limiting factor of the model's scalability, both during training and serving. This is because the number of candidate items can be typically in the order of many hundreds to thousands for a single request. We instead include an average embedding from all candidate embeddings as a single position in the encoder. Both approaches performed similarly well in our setting in terms of ranking quality.





The training objective of our main candidate generator is based on predicting the next customer action, however, our data pipelines are configurable and can be easily extended to predict actions in a longer future time window, e.g. a week, to balance long and short-term customer preferences and improve diversity in outputs. 



 §.§ Encoding of the Customer Journey Data



In this section, we elaborate on how we encode holistic customer journey data into models in both layers. This consists of (1) behavioral data which includes customer action sequences, item and action metadata, and temporal data, and global and local contextual data. We argue and demonstrate in our experiments that providing complete and heterogeneous information to the model is crucial to predictions that are personalized and contextually relevant.

Behavioural data. As already mentioned, customer action sequences are encoded using transformer encoders as these have been proven more effective than other approaches <cit.>. Each action in an action sequence is represented by item embedding that encodes domain-specific visual information, categorical item metadata, action type, and timestamp embeddings. The categorical item metadata consists of relevant attributes such as brand, color, pattern, category, material, etc. The timestamp embedding encodes quantized timestamps as measured from the beginning of the model training. Temporal data is crucial for modeling customer behavior across sessions. Since customer intent can vary drastically across different sessions, modeling action sequences while ignoring this structure affects performance negatively <cit.>. All inputs are passed through trainable embedding to project discrete or bucketed values into low-dimensional spaces. Item and action-specific embeddings are concatenated.  

Contextual data. The contextual information is divided into global and local contexts. Global context includes information such as the customer's country and device type while local context includes information about the touch point the customer has triggered an action from, for example, item category, search query, carousel type, and even the products shown on the page. To represent multiple items, we average their embeddings to produce a single “summary” embedding that is fed into its own position in the encoder. To fuse contextual and customer action sequence data we employ the same attention mechanism by allocating the starting positions for contextual features. This approach does not require additional networks such as deep and wide or deep and cross networks that can substantially increase the parameters of the model. Instead, it makes use of the self-attention mechanism to compute complex interactions between the inputs as every other position in the sequence can attend to the context independently. Local contextual information is concatenated with the representation corresponding to the previous action. We note that concatenation in many cases can be replaced by averaging to avoid large input dimensionality. 





 §.§ Position Debiasing


In the context of ranking systems, feedback loops occur when a model influences customer interactions which can lead to biased relevance data. Typically, items ranked higher by the model receive more customer attention, causing position bias. Position bias causes a skewed representation of actual customer preferences which in turn may amplify and degrade model performance over time due to the feedback loop. To address this, we incorporate position information as a feature into a model to separate the effect of the position and the true relevance of the probability with which the customer would interact with an item. A debiased model is conditioned on positions during training (right branch in <ref>) and position-independent during serving by setting positional feature to fixed values to counter position bias <cit.>. The position branch is separate from the rest of the model due to the asymmetry between training and serving. We performed additional A/B test which confirmed that adding position debiasing resulted in increasing long-tail utilisation by 5% while not deteriorating engagement metrics.



§ POLICY LAYER



The last stage of the system, the policy layer, is responsible for the final page composition. Here, multiple re-ranked candidate items are combined into one, performing granular, page-level optimization, and applying heuristics, business rules, and filters depending on the use case. In the following, we describe how we promote fresh (new or cold-start) items while simultaneously introducing exploration into the system. We also describe some common heuristics applied in this layer to meet product requirements. 

Exploration with New Items. To tackle the cold start problem, the policy layer incorporates fresh items into the organic ranking using exploration heuristics, beginning by sorting these items using content-based features. The blending of outputs from different candidate sources is managed through epsilon-greedy exploration, providing flexibility and ensuring a clear separation of tasks. This method adapts to various use cases by allowing for different exploration techniques and criteria for defining fresh items.

Epsilon-greedy exploration, a staple in reinforcement learning, uses a constant exploration factor that, despite some inefficiencies, functions well in practice and scales to complex scenarios <cit.>. Starting from position k, the policy layer introduces new items with a probability of ϵ and selects from the ranked list with a probability of 1-ϵ, based on a weighted random sampling method determined by the ranking layer. The parameters k and ϵ help balance the introduction of new items against potential disruptions to the user experience (refer to Algorithm <ref> in the Appendix for more details).





Business Heuristics. This section introduces straightforward heuristics addressing i) down-sorting previously purchased items and ii) avoiding perceived lack of diversity. Items with diminishing returns, such as a winter coat purchased again soon after the initial buy, are down-ranked to enhance the customer experience. Instead of modeling the probability of repurchase for these items <cit.>, we apply a simpler rule: any item purchased within the last 2 months is down-ranked. Additionally, to prevent the impression of uniformity when many items from the same brand are shown together, we use a diversification heuristic: if a sequence a_n,…, a_n+k of k same-brand items appears, the first differing-brand item in the subsequent sequence a_n+k+1, …, a_M is relocated to position n+k.






§ MODEL PRODUCTIONIZATION


Although the two-tower model is trained as a single entity, we deployed each tower as separate endpoints. The item tower is triggered when a new model is trained, a new item is added, or an attribute is modified. The newly generated embedding is then transmitted through a Kafka-based intake stream and indexed in ElasticSearch.

One of the most challenging aspects of our infrastructure was updating the embeddings after training a new two-tower model, as it required maintaining consistency between the embedding versions and the tower model versions. When a new model is released, a complete refeed is necessary to index the updated product embeddings. During this refeed, the system continues to operate using the previous versions of the models and product embeddings. The transition between versions is managed through a blue-green deployment strategy.

For scoring items in the candidate generation phase, we utilize Elasticsearch's vector search function, which employs an efficient approximate k-NN search. We also incorporate real-time customer action ingestion, which processes the resulting sequences to compute customer embeddings and feed the ranking layer. A caching layer positioned between these two processes stores the customer sequences, which can be uniformly fed to both models thanks to the uniformity of architecture. This ensures high throughput and maintains low latency during inference. 

Model serving is performed on standard CPU-based instances hosted on AWS SageMaker, while training is conducted on multiple GPU instances. The average model response p99 latency for each layer is maintained at 10ms. 



§ EXPERIMENTS



In this section, we present the results of extensive online (A/B testing) and offline experimentation and ablation studies using internal datasets. We compare the newly introduced ranking system against the existing baseline models.



 §.§ Offline Experiments




  §.§.§ Dataset




The offline dataset consists of a sample of item interactions from the last 60 days aggregated by customer id. The item interactions consist of product clicks, add-to-wishlist, add-to-cart, and checkout events attributed to the browse and search premise by using the "last-touch" attribution model. The interactions are joined with the corresponding item ids, their timestamps, and interaction type and sorted by timestamp. To form a single data sample, the resulting sequences are combined with contextual data, specifically, market, device type, browsing category, and search query (if present). The training dataset consisted of 71M unique customers across 25 markets. We do not perform any preprocessing such as deduplication or outlier removal on the obtained customer sequences besides truncating to the last 100 actions. The average sequence length is 24 actions.  The evaluation dataset contains 300K customers (<ref> in the Appendix provides histograms per action type). We applied hard temporal split to create the training and test datasets to ensure no data leakage.











  §.§.§ Metrics and evaluation protocol



The main metrics that our models are evaluated on are the following:


  * Recall@k: proportion of all relevant items within top k items (as defined in <cit.>).

  * NDCG@k: measures the effectiveness of a ranking, taking into account the position of relevant items in the ranked list of top-k items, and attributed items are considered as relevant (as defined in <cit.>).

  * Diversity: we use the maximum run of consecutive items from the same brand as a proxy to brand diversity. A high value suggests that a ranking can be dominated by items coming from only a few brands. User acceptance testing showed this leads to undesirable customer experience.

  * Novelty: we use recall of new items as a proxy for novelty. This metric captures the ability of a ranker to promote new items and address the item cold-start problem. 






Our evaluation protocol closely mirrors a real-world production environment by employing a strict separation of training and test data based on time. Customer sequences within the dataset are chronologically ordered. The models undergo evaluation exclusively on the test dataset (with time-based split), which comprises “ground-truth” pages of items that users have either viewed or interacted with following search or browse requests. Importantly, models are provided data only up until the timestamp of each request, with a particular emphasis on adhering to data caching periods—during these times, models operate solely on cached data. 

For each item category or search query contained in the requests, the candidate generation models score and ranks all corresponding items in the catalog. The ranking models re-rank the 500 items with highest score coming from the candidate generation layer. We then calculate the offline metrics based on these ranked lists against the above "ground-truth" pages observed in the test data.

The metrics are calculated for each ranking produced for a single test example, and averaged over the test dataset. All reported results are statistically significant (p-value < 0.05) unless stated otherwise. We used a t-test for significance testing. 



  §.§.§ Candidate Generation


We compare the following methods in the candidate generation layer:


  * GBT is a candidate generation model based on Gradient Boosting Trees, which has proven to offer competitive performance compared to neural-based models <cit.>. It ranks items based on their static attributes (season, material, type, etc.) and their dynamic historical engagement rates (e.g. add-to-cart rate for the last 5 min). The LambdaRank objective <cit.> is used during training to up-rank interactions based on their graded relevance ("purchase" has the highest while "click" the lowest relevance). The model's scores are computed in streaming fashion making it highly reactive to trends in customer behavior. This model was our previous production model.

  * RCG is our new candidate generation model introduced in <ref>. We test a few versions of this model: RCG_ntr which uses pre-trained visual embeddings of items and includes a bias term that captures item popularity; RCG_tr which employs trainable item embeddings, initialized from the pre-trained visual item embeddings. These two models utilize only global contextual information, such as country and device type. Additionally, RCG_tr+ctx denotes a variant with additional local contextual input — the current user's browsing category and a binary flag whether they browse or search. All the model versions consist of 2 encoder layers with 4 heads, gelu activation, and a max. sequence length of 100. The model was trained for 20 epochs by using the Adam optimizer, with a learning rate set to 0.001. We do not consider hard negatives and rather employ log uniform candidate sampling.






<ref> presents an offline ablation study comparing our RCG model with the baseline GBT used as a candidate generator on the browse use case (<ref> in the Appendix summarizes offline evaluation on the search use case). Across all evaluated customer segments, our newly proposed candidate generator significantly outperforms the existing one.

The model variant incorporating trainable item embeddings achieves markedly improved performance in offline metrics. However, it is important to note that trainable item embeddings exacerbate the item cold-start problem (to mitigate this issue, we introduce exploration strategies for new items in <ref>).

Incorporating local contextual inputs further boosts the performance of the RCG_tr model. Specifically, by including item category and search query presence, the RCG_tr+ctx model's Recall@500 is enhanced by an additional 10%, and its NDCG by 14%. In Section <ref> we will show that these gains are also reflected in our online experiments.





  §.§.§ Ranking Layer


















We compare the following methods in the ranking layer:


  * WDL-ATT is a wide & deep neural network with an attention mechanism between the customer action sequences and ranking candidates. It scores the candidate items by applying a dot product between context, user, and item embeddings that are all trainable and have 128 dimensions. It employs a loss function directly optimizing for the NDCG metric <cit.>. Items are represented by using pre-trained embeddings that encode brand, category, pattern, and other visual cues. It is trained for 2 epochs by using the Adam optimizer with a learning rate of 0.0002.

  * RL is our new ranking model introduced in <ref>. d_model and the model output is set to 128, the model consists of 2 encoder layers with 8 heads, relu activation, and a max. sequence length of 80. The model is trained for 2 epochs by using the Adam optimizer, with a learning rate of 0.001. The ratio of negatives vs. positives is set to 4. Since sampling negatives only from items that were in the view-port lead to degradation of performance, we sampled from all non-interacted items in the page. 


  * BST is the Behavior Sequence Transformer introduced in <cit.>. We use the same inputs and hyperparameters as in RL.


It should be noted that all compared algorithms use the same near real-time serving infrastructure. Some of the other algorithms mentioned, such as TransAct <cit.>, while potentially competitive, were not applicable to our use case or latency constraints. For simplicity, we used a single ranking model in the ranking layer. In these experiments, we focus on the NDCG metric for "high-value actions" or HVAs, which, in our context, are add-to-wishlist and add-to-cart actions. This metric acts as a proxy for our success KPI, defined by customer engagement wrt. HVAs, as detailed in the subsequent section.

The offline evaluation results are summarized in <ref>, comparing RL and BST against the existing WDL-ATT which proved to be a strong baseline. The NDCG metric indicates that the new model effectively prioritizes relevant items higher up in the rankings, both at the top of the list (k=6) and across the entire first page (k=84). Furthermore, RL favored the promotion of new items while enhancing diversity. In terms of relevance, BST lagged behind both algorithms, although it performed the best when it comes to diversity. 

<ref> presents an ablation study that explains the contribution of each component and input type to the model's performance. The removal of any of these elements significantly detracts from overall model efficacy. Particularly noticeable is the performance decline when contextual inputs are not integrated early in the encoder. This finding suggests that our model more effectively leverages contextual information alongside rich item representations in customer action sequences compared to other algorithms such as BST. We opted not to benchmark against the DCN method cited in <cit.>, as self-attention offers a superior mechanism for capturing feature interactions within sequential data compared to feature crosses. Additionally, the ablation study demonstrates that omitting heterogeneous inputs markedly diminishes the model's performance. Specifically, excluding contextual inputs results in more than a 10% decrease in NDCG@6, while complete removal of item metadata leads to a drastic 26% reduction in NDCG@6.













































































 §.§ Online Experiments


We have conducted several online A/B tests on real-world ranking use cases at Zalando. These tests were carried out systematically, replacing one component at a time to evaluate its impact. All tests allocated equal traffic splits among variants over a few weeks, as necessary, to achieve the minimum detectable effect for the success KPI with a p-value< 0.05. Each model was retrained and deployed daily. Beyond customer engagement, our evaluation of online performance encompasses a variety of exploratory metrics, including financial metrics, the capacity to promote new items (novelty), and ranking diversity (as defined in <ref> as well as in the number of distinct brands on the first page). <ref> summarizes the results from a series of A/B tests performed on the browse use case, where a customer is browsing the category tree as described in <ref>. 

In the first A/B test in <ref> (row 1), we compared the new candidate generation model (RCG) against the previous ranking system end-to-end (GBT + WDL-ATT). We examined two variants: firstly, the RCG model with trainable item embeddings (RCGtr) and, secondly, the RCG model with non-trainable item embeddings (RCGntr) paired with WDL-ATT as the ranker. The former variant demonstrated a significant increase in engagement. However, the new candidate generation model had no significant impact on financial KPIs, promoted fewer new items, and decreased diversity.

In the second A/B test in <ref> (row 2), we assessed the effect of adding our new ranking layer (RL). The baseline for this experiment was the winning variant from the first test, RCG_tr. We explored two variants: the first employed the previous ranking algorithm WDL-ATT, and the second introduced our new ranking algorithm RL along with the policy layer. The outcomes highlight the advantages of applying our powerful ranking model on candidates from the candidate generation layer. The second variant significantly improved all monitored KPIs across all customer segments, including net revenue per customer, novelty, diversity and retention.

In the third A/B test in <ref> (row 3), we tested an improved version of the candidate generation model aka RCG_tr+ctx, which includes local contextual data. This resulted in a significant uplift in all monitored KPIs, including brand, and categorical diversity (omitted due to space limitations). This A/B test demonstrates the importance of including data that captures the entire customer journey to provide a more contextually relevant ranking.

<ref> summarizes the results of a series A/B tests performed on the search use case, i.e., when the customer is using full-text search to find their desired item. The conclusions are similar and for brevity, we omit the details.






























§ CONCLUSION



In this paper, we have introduced a flexible, scalable, steerable, and real-time ranking platform that has been proven to enhance customer experience by delivering more relevant and personalized content. This approach has, in turn, led to improvements in various customer-centric and business metrics. We have described the architecture of our ranking platform, adhering to a set of design principles and utilizing state-of-the-art models. We have also offered insights into their performance, highlighting the considerable advantages of integrating heterogeneous signals and inputs that encompass the entire customer journey, as well as the effectiveness of fine-tuning input embeddings to boost model performance.

Our offline and online evaluations clearly demonstrate that our proposed system not only significantly outperforms existing solutions by a wide margin (10-40% improvement in offline evaluation metrics and a 15% combined engagement and +2.2% revenue uplift in 4 online A/B tests) but also excels in real-life use cases and scales effectively under heavy load, which is a crucial requirement for large e-commerce platforms. Furthermore, we illustrate that the enhanced experience benefits both returning and new customers. Lastly, we provide valuable insights and practical guidance for application by other applied scientists and practitioners within the domain.

























ACM-Reference-Format






§ APPENDIX




 §.§ Exploration with New Items

To promote new items that may suffer from the cold start problem, we follow the Algorithm <ref> to combine organic ranking results (S) coming from retrieved candidates by e.g. RCG, with new items (N). The algorithm is controlled by k and ϵ.




 §.§ Data distribution

Our datasets are based on actions that customers perform against items on the platform. In <ref> we show the distribution of those actions, computed from raw customer data, before any dataset-specific preprocessing, e.g. trimming. Clicks have the biggest volume, then add-to-wishlist, add-to-cart, and purchases.





 §.§ Candidate Generation Results on the Search Use Case

<ref> presents offline evaluation results of the RCG model on the search use case only. The same improvements seen before on the browse traffic apply to the search traffic.






















 §.§ Expressive Power of Two-Tower Models


The two major recommendation models used in our ranking platform have a two-tower architecture in which one tower embeds the customer and the other one embeds the fashion article that is being scored (see Section <ref> and Section <ref>). Mathematically, the score function f^model corresponding to a model of this type can be written as

    f^model(x) = ⟨φ(c), ψ(a) ⟩,

where x = (c, a) is the model input with c and a being the customer and the article parts of the input, respectively. 

In this section, we study the expressive power of this model class. Specifically, we prove that any continuous target function f (defined on a bounded feature space) can be approximated by a score function of the form (<ref>) provided the embedding size is large enough.


    Let the range of customer and article features be bounded: C^ℓ_i ≤ c_i ≤ C^u_i, i=1, …, k_c, and A^ℓ_j ≤ a_j ≤ A^u_j, j=1, …, k_a, and let the target function f be continuous on the feature domain 
    
    D = [C^ℓ_1, C^u_1]×…× [C^ℓ_k_c, C^u_k_c]× [A^ℓ_1, A^u_1]×…× [A^ℓ_k_a, A^u_k_a].


    Then for any ε > 0, there exist n>0 and transformations φR^k_c↦R^n and ψR^k_a↦R^n such that

    
    max_(c, a)∈D|f(c, a) - ⟨φ(c), ψ(a) ⟩| < ε.

    
        Without loss of generality, let us assume that D is a unit cube, i.e. C^ℓ_i = 0, C^u_i = 1 for all i=1, …, k_c and A^ℓ_j = 0, A^u_j = 1 for all j=1, …, k_a.

        Consider the set of all multivariate polynomial functions on D. Note that it (a) contains constant functions, (b) is closed under the operations of addition and multiplication, and (c) separates points: for any u, v∈D, u≠ v, there exists a polynomial P such that P(u)≠ P(v). Then by applying the Stone-Weierstrass theorem, we conclude that for any ε > 0, there exists a polynomial P_ε,

        
    P_ε = ∑_m=1^n 
                α_m ∏_i=1^k_c c^p_m,
                ,i_i ∏_j=1^k_a a^q_m,
                ,j_j,

        such that
        
    max_(c, a)∈D|f(c, a) - P_ε(c, a)| < ε.

        By defining
        
        
    [ φ(c) = (
                α_1 ∏_i=1^k_c c^p_1, i_i, …, α_n∏_i=1^k_c c^p_n, i_i),;                    ψ(a) =  (∏_j=1^k_a a^q_1, j_j, …, ∏_j=1^k_a a^q_n, j_j), ]

        we can rewrite (<ref>) as
        
    P_ε(c, a) = ⟨φ(c), ψ(a) ⟩,   (a, c)∈D.

        Then (<ref>) implies that the constructed transformations φ and ψ satisfy (<ref>). 
    


