Transformer Choice Net: A Transformer Neural Network for Choice
  Prediction
Discrete-choice models, such as Multinomial Logit, Probit, or Mixed-Logit,
are widely used in Marketing, Economics, and Operations Research: given a set
of alternatives, the customer is modeled as choosing one of the alternatives to
maximize a (latent) utility function. However, extending such models to
situations where the customer chooses more than one item (such as in e-commerce
shopping) has proven problematic. While one can construct reasonable models of
the customer's behavior, estimating such models becomes very challenging
because of the combinatorial explosion in the number of possible subsets of
items. In this paper we develop a transformer neural network architecture, the
Transformer Choice Net, that is suitable for predicting multiple choices.
Transformer networks turn out to be especially suitable for this task as they
take into account not only the features of the customer and the items but also
the context, which in this case could be the assortment as well as the
customer's past choices. On a range of benchmark datasets, our architecture
shows uniformly superior out-of-sample prediction performance compared to the
leading models in the literature, without requiring any custom modeling or
tuning for each instance.
Hanzhao Wang, Xiaocheng Li, Kalyan Talluri
# Transformer Choice Net: A Transformer Neural Network for Choice Prediction

###### Abstract

Discrete-choicee models, such as Multinomial Logit, Probit, or Mixed-Logit, are widely used in Marketing, Economics, and Operations Research: given a set of alternatives, the customer is modeled as choosing one of the alternatives to maximize a (latent) utility function. However, extending such models to situations where the customer chooses more than one item (such as in e-commerce shopping) has proven problematic. While one can construct reasonable models of the customer's behavior, estimating such models becomes very challenging because of the combinatorial explosion in the number of possible subsets of items. In this paper we develop a transformer neural network architecture, the Transformer Choice Net, that is suitable for predicting multiple choices. Transformer networks turn out to be especially suitable for this task as they take into account not only the features of the customer and the items but also the context, which in this case could be the assortment as well as the customer's past choices. On a range of benchmark datasets, our architecture shows uniformly superior out-of-sample prediction performance compared to the leading models in the literature, without requiring any custom modeling or tuning for each instance.

Imperial College Business School, Imperial College London

{h.wang19, xiaocheng.li, kalyan.talluri}@imperial.ac.uk

## 1 Introduction

Firms are interested in understanding the choice behavior of their customers as well as forecasting the sales of their items. When customers choose at most one item per shopping instance, discrete-choice models estimate the probability of the choice, either at a segment level or individual customer level, based on a latent utility function of the features of the item, the customer, and the provided assortment.

However, there are many situations where customers choose multiple items on a single shopping instance, either from the same category or across categories. The firm may be aware of only the final choices made by the customer (as in physical retail) or the precise sequence of those choices (such as in an e-commerce setting). _Multi-choice models_ are used for the former case, to estimate the probability of choosing a subset of items, amongst all possible subsets of the given assortment, considering potential interactions amongst the items and their features. _Sequential choice models_ consider the sequence of choices, taking into account not only the item and customer features but also what the customer has chosen till then to predict the subsequent choice(s).

Modeling and predicting the choice probabilities for these situations is challenging: the complexity of the sequential and multi-choice models is considerably more than in the single-choice case because of combinatorial explosion in the number of possible customer journeys and final choices, and consequently models for multiple choices are less widely adapted in practice.

In this paper, we introduce the Transformer Choice Net, a neural network using the Transformer architecture (Vaswani et al., 2017), as a data-driven solution that works under any of the three models: single, sequential, and multiple.

The key contributions of this paper are as follows:
* We develop the Transformer Choice Net, a neural network-based framework capable of encompassing all three choice paradigms--single, sequential, and multiple choices. This unification simplifies the traditionally fragmented approach to choice modeling.
* We empirically validate the prediction performance of our model across diverse industry benchmark datasets. Our architecture shows uniformly superior out-of-sample prediction performance compared to the leading models in the discrete-choice literature, without requiring any custom modeling or tuning for each instance. Moreover, our architecture is scalable, showing an ability to learn and predict multiple choices over a large assortment size, overcoming the limitations of many sequential and multi-choice models.
* We show theoretically the universal representation capacity of the Transformer Choice Net across the three choice paradigms.

## 2 Literature Review

This section reviews the literature for single, sequential, and multiple choices, and also the literature on neural networks for predicting the choice probabilities.

**Single-choice models.** For general background on discrete-choice models, see McFadden and Train (2000) and the book by Ben-Akiva and Lerman (1985).

A plethora of models have been proposed for the single-choice case starting from the popular Multinomial Logit (MNL) choice model: to name a few, the Probit (Daganzo, 2014), the ranking-based choice model (Block and Marschak, 1959; Farias et al., 2013), the Negative-Exponential (also called Exponential) model (Alptekinoslu and Semple, 2016), and the Markov-chain choice model (Blanchet et al., 2016).

From a machine learning perspective, the single-choice case can be viewed as a classification task where we assign probabilities for the various items (classes) to be chosen. From this point of view, researchers have applied random forests (Chen et al., 2021; Chen and Misic, 2022), embedding methods (Seshadri et al., 2019), and assortment context vectorization (Yousefi Maragheh et al., 2020; Tomlinson and Benson, 2021; Najafi et al., 2023; Bower and Balzano, 2020) with the latter trying to overcome some known weaknesses of the MNL model. Van Cranenburgh et al. (2022) provide a comprehensive review of the current literature on integrating machine learning into single-choice modeling.

Bentz and Merunka (2000) give an early effort to use neural networks for single-choice modeling. Wang et al. (2020); Han et al. (2020); Sifringer et al. (2020); Wong and Farooq (2021); Arkoudi et al. (2023) investigate different neural network architectures to capture the relationships between the items' features and utility in the MNL choice model. Several recent works try to capture more complicated assortment effects: the RUMNet (Aouad and Desir, 2022) mimics the mixed-MNL model instead of the MNL model, and the AssortNet (Wang et al., 2023) encodes the entire assortment into the choice probability via ResNet architecture. Pfannschmidt et al. (2022); Rosenfeld et al. (2020) employ the Deep Set architecture (Zaheer et al., 2017) to allow permutation variations in input assortments (see also Wagstaff et al. (2019) and SS4).

**Sequential and multi-choice models.** Sequential choice is typically based on purchase history data, often from assortments with only a small number of items (\(\leq 10\)) (Manchanda et al., 1999; Gupta, 1988). Despite the growth of e-commerce and the availability of click-stream data, it is somewhat surprising that there are relatively few models for sequential choice behavior: The SHOPPER model (Ruiz et al., 2020) stands out as a probabilistic representation of consumer choice, aiming to forecast sequential choices based on variational inference. Another noteworthy approach is due to Jacobs et al.
(2021), where they compress purchase history data into purchase motivations and then employ variational inference for model deductions. Gabel and Timoshenko (2022) design a neural network that takes each customer's history as input to predict the next choice, but like Jacobs et al. (2021) the architecture does not encode the customer and item features.

There is a slightly richer body of research on multiple choices, i.e., without knowing the sequence of choices (Chen et al., 2023; Bai et al., 2023; Luan et al., 2023; Jasin et al., 2023; Tulabandhula et al., 2023). Benson et al. (2018) expand the random utility maximization model from single to multi-choice scenarios. Tulabandhula et al. (2023) take a different approach, quantifying pairwise item interactions within each potentially chosen subset through a linear sum. Aarts et al. (2023) bridge point processes with subset-choice models, encoding pairwise interactions using the \(\log\det\) function. Lin et al. (2022a) augment the ranking-based choice model (Block and Marschak, 1959; Farias et al., 2013) to multiple choices, but they do not explicitly model item or customer features. A common limitation for all these models is scalability: as we have to predict the most likely amongst an exponential number of subsets, the size of the assortment they can handle is small.

**Transformer architecture.** First introduced by Vaswani et al. (2017), the Transformer architecture has emerged as a ground-breaking deep learning framework. Initially tested on machine translation within the realm of natural language processing (NLP), it has since expanded to a broad range of NLP tasks as exemplified by ChatGPT (OpenAI, 2023) for text generation.

The Transformer architecture has also excelled in computer vision (Han et al., 2022), audio processing (Dong et al., 2018), and integrated multimodal tasks (Ramesh et al., 2021). For a survey, we refer to the review by Lin et al. (2022b). Relevant to this paper, Lee et al. (2019) design the Set Transformer, which ensures any permutation of the input to yield identical outputs (see SS4 for more discussions). To the best of our knowledge, there has been no application till now to choice modeling.

## 3 Problem Setup

### Preliminaries on Choice Models

#### 3.1.1 Single-Choice Model

Consider a ground set of \(n\) potential items \(\mathcal{N}=\{1,2,...,n\}\). For completeness, we can let \(n\) denote the no-purchase option which (when selected) means the customer chooses none of the offered items. A seller chooses an _assortment_\(\mathcal{S}\subseteq\mathcal{N}\) to offer to the customer, and in the single-choice case the customer will choose at most one item from it. A _single-choice model_ prescribes the probability of choosing item \(i\) conditional on the assortment \(\mathcal{S}\) (where we can assume the no-purchase option is always in the assortment):

\[\mathbb{P}(i|\mathcal{S})\text{ for all }i\in\mathcal{N}\text{ and } \mathcal{S}\subseteq\mathcal{N}.\]

In particular, \(\mathbb{P}(i|\mathcal{S})=0\) for \(i\notin\mathcal{S}\), i.e., the customer cannot choose an item not offered in the assortment. In this way, a single-choice model

\[\mathcal{M}=\{\mathbb{P}(i|\mathcal{S}):\mathcal{S}\subseteq\mathcal{N}\} \tag{1}\]

dictates \(2^{n}-1\) probability distributions, each of which corresponds to one possible assortment (excluding when \(\mathcal{S}\) is an empty set).

#### 3.1.2 Sequential Choice Model

The sequential choice model is an intermediate between the single-choice model and the multi-choice model, and it also serves as a useful model by itself. Consider a customer shopping at an e-commerce
who may choose more than one item for purchase and the items are chosen sequentially one after another from an assortment. To capture this scenario, we define the set of currently not chosen items as _candidates_ (denoted by \(\mathcal{C}\)), while the set of initially offered items as _assortment_ (denoted as \(\mathcal{S}\)). Thus \(\mathcal{C}\subseteq\mathcal{S}\) and the set \(\mathcal{S}\setminus\mathcal{C}\) is the set of chosen items so far. The single-choice model then can be viewed as a special case where \(\mathcal{C}=\mathcal{S}\). The sequential choice model describes the probability of choosing item \(i\) from candidates \(\mathcal{C}\) given the assortment \(\mathcal{S}\)

\[\mathbb{P}(i|\mathcal{C},\mathcal{S})\text{ for all }i\in\mathcal{N},\mathcal{C }\subseteq\mathcal{S}\subseteq\mathcal{N}.\]

Let \(\mathbb{P}(i|\mathcal{C},\mathcal{S})=0\) for \(i\notin\mathcal{C}\), i.e., the customer can only choose an item from the candidates set. In parallel to (1), the sequential choice model encapsulates the following distributions

\[\mathcal{M}=\{\mathbb{P}(i|\mathcal{C},\mathcal{S}):\mathcal{C}\subseteq \mathcal{S}\subseteq\mathcal{N}\}. \tag{2}\]

#### 3.1.3 Multi-Choice Model

The multi-choice model considers the purchase of a basket of items, i.e., a subset of the given assortment. Hence, we are interested in the following distributions

\[\mathcal{M}=\{\mathbb{P}(\mathcal{B}|\mathcal{S}):\mathcal{S}\subseteq \mathcal{N}\} \tag{3}\]

where \(\mathcal{B}\subseteq\mathcal{N}\) denotes the basket of (chosen) items and \(\mathcal{S}\) is the given assortment as before. As earlier, we assume \(\mathbb{P}(\mathcal{B}|\mathcal{S})=0\) if \(\mathcal{B}\nsubseteq\mathcal{S}\); in other words, the chosen basket can only contain items from the assortment.

### Learning Task

The learning task, the focus of our paper, refers to estimating the choice model \(\mathcal{M}\) from \(m\) observed samples \(\mathcal{D}\):

\[\mathcal{D}=\begin{cases}\{(i_{k},\mathcal{S}_{k}),k=1,...,m\}&\text{single choice,}\\ \{(i_{k},\{\mathcal{C}_{k},\mathcal{S}_{k}\}),k=1,...,m\}&\text{sequential choice,}\\ \{(\mathcal{B}_{k},\mathcal{S}_{k}\}),k=1,...,m\}&\text{multiple choices,} \end{cases}\]

where each observation, depending on the model to be estimated, consists of the final choice(s) (\(i_{k}\) or \(\mathcal{B}_{k}\)), accompanied by the assortment information (\(\mathcal{S}_{k}\) or \(\{\mathcal{C}_{k},\mathcal{S}_{k}\}\)).

**Item/product feature**. We assume that there is a feature vector associated with each item/product. Such a feature vector encodes characteristics of the product such as color, size, brand, price, etc. When no such feature is available, one can still obtain such a vector by a one-hot encoding of the items.

**Reduction to sequential choice model.** The goal of our paper is to develop one single transformer model that fits all three choice modeling scenarios. In this light, we discuss here how the single-choice model and the multi-choice model can be reduced to the sequential choice model in terms of both learning and inference so the transformer model can be based only on learning and predicting a sequential choice model.

The reduction of the single-choice model to the sequential choice model is straightforward. As we pointed out earlier, we simply set \(\mathcal{C}=\mathcal{S}\), i.e., to predict the first item to be chosen given the initial assortment. For the estimation part, the sequential choice model can be estimated from the single-choice training samples \(\{(i_{k},\mathcal{S}_{k}),k=1,...,m\}\) by simply augmenting it into the form \(\{(i_{k},\{\mathcal{S}_{k},\mathcal{S}_{k}\}),k=1,...,m\}\). For the inference part, one can infer the probabilities of the single-choice model by using the probability \(\mathbb{P}(i|\mathcal{S},\mathcal{S})\) from the sequential choice model.
The reduction from the multi-choice model to the sequential choice model is a bit more complicated. First, we note that the multi-choice model can also be mathematically represented by the sequential choice model. To see it, we can introduce a sequence \(\sigma\) to denote the order of items being added to the basket. Then we can decompose the choice probability of a basket by sequential choice probabilities:

\[\mathbb{P}(\mathcal{B}|\mathcal{S})=\sum_{\sigma\in\text{Perms}} \prod_{j=1}^{|\mathcal{B}|}\mathbb{P}(\sigma_{j}|\mathcal{C}_{j}^{\sigma, \mathcal{S}},\mathcal{S}), \tag{4}\]

where Perms contains all possible permutations over the set \(\mathcal{B}\), \(\sigma_{j}\) is the \(j\)-th item in the permutation \(\sigma\), and \(\mathcal{C}_{j}^{\sigma,\mathcal{S}}=\mathcal{S}\setminus\{\sigma_{i}:i<j\}\), i.e., the candidates when adding \(j\)-th item to the basket.

With this representation in mind, we first consider the learning of a sequential choice model from multi-choice training samples \(\{(\mathcal{B}_{k},\mathcal{S}_{k})\},k=1,...,m\}\). Since the sequence in which the items are added to the basket \(\mathcal{B}_{k}\) is unobserved, we cannot directly use the data to train a sequential choice model. One remedy involves an iteratively random selection of items from the basket, designating it as the sequential choice, and then constructing its associated candidates. For example, we are given a multi-choice sample \(\{\mathcal{B},\mathcal{S}\}\) and we start the sample generation (for the sequential choice model) by setting \(\mathcal{C}=\mathcal{S}\). Say, the first randomly chosen item from \(\mathcal{B}\) is \(i^{(1)}\); then we obtain one sample of \(\{i^{(1)},(\mathcal{S},\mathcal{S})\}\). Then we move on to generating the second sample, \(\{i^{(2)},(\mathcal{S})\backslash\{i^{(1)}\},\mathcal{S})\}\) where \(i^{(2)}\) is randomly picked from \(\mathcal{B}\backslash\{i^{(1)}\}\). By continuing this process, we generate training samples for the sequential choice model from the original multi-choice sample \(\{\mathcal{B},\mathcal{S}\}\). A similar sampling method is also applied in Ruiz et al. (2020).

To predict the multiple choices of a given assortment \(\mathcal{S}\) from a sequential choice model, the set of predicted choices can be sequentially generated by reversing the above sampling method. Specifically, we can start with \(\mathcal{C}=\mathcal{S}\) and either sample a product \(i^{(1)}\) following the distribution \(\mathbb{P}(\cdot|\mathcal{C},\mathcal{S})\) or select the item with the largest choice probability, and then remove \(i^{(1)}\) from \(\mathcal{C}\). By continuing this process, we can sample multiple items and add them to the predicted set of choices, until a stopping rule is met. This stopping rule can be enforced either when a pre-determined size of the basket is met or a virtual "stopping item" is sampled.

## 4 Transformer Choice Net

Now we introduce the Transformer Choice Net (TCNet) for the sequential choice model (which as we pointed out covers both single and multi-choices). Mathematically, the TCNet is denoted as \(f^{\text{TCN}}\), a function that maps the candidate set \(\mathcal{C}\) and the assortment \(\mathcal{S}\) to the choice probability \(f^{\text{TCN}}_{i}(\mathcal{C},\mathcal{S})\) for each item \(i\in\mathcal{C}\).

### Attention Mechanism

The attention mechanism has become a fundamental component of modern deep learning to solve various tasks of natural language processing and computer vision. In this subsection, we provide some preliminaries on this mechanism (see Niu et al., 2021) for more details) and show how it can be adapted for choice modeling.

An _Attention_ function \(\text{Att}(Q,K,V)\) takes three matrices as input: the matrix of queries \(Q\in\mathbb{R}^{S\times d}\), the matrix of keys \(K\in\mathbb{R}^{S^{\prime}\times d}\), and the matrix of values \(V\in\mathbb{R}^{S^{\prime}\times d}\)

\[\text{Att}(Q,K,V)=\phi(QK^{\top})V,\]

where \(d\) is the feature dimension. The function \(\phi\) is an activation function to \(QK^{\top}\), with Softmax being the most prevalent choice. The matrix \(QK^{\top}\) is termed the (unnormalized) score matrix, the \((i,j)\)-element
of which quantifies the influence of item \(j\) on item \(i\). Intuitively, the attention mechanism empowers a model to selectively focus on specific segments of its input, guided by the score function, to generate an output.

_Self-attention_ is an exemplification of the attention mechanism where all the three matrices above depend on the input feature matrix \(X\):

\[f^{\text{Att}}(X;W_{Q},W_{K},W_{V})=\text{Att}(XW_{Q},XW_{K},XW_{V}), \tag{5}\]

where \(W_{Q},W_{K},W_{V}\in\mathbb{R}^{d\times d}\) are trainable parameters. For simplicity, we abbreviate the parameters and use \(f^{\text{Att}}(X)\) to denote this self-attention function.

### Architecture of TCNet

Figure 1 illustrates the architecture of the Transformer Choice Net \(f^{\text{TCN}}\). The network's input comes from two sources: the assortment set \(\mathcal{S}\) of size \(S\) and the candidate set \(\mathcal{C}\) of size \(C\). At noted in SS3, we suppose each item \(i\) is associated with a feature vector \(x_{i}\in\mathbb{R}^{d}\) (either product feature or one-hot encoding of the product id). Then the input matrices of TCN are \(X^{\mathcal{S}}\in\mathbb{R}^{S\times d}\) and \(X^{\mathcal{C}}\in\mathbb{R}^{C\times d}\), and the output is a vector of choice probabilities for each item \(i\in\mathcal{C}\).

The TCN consists of three main parts: assortment encoder, candidates encoder, and utility decoder.

The _assortment encoder_ processes the input \(X^{\mathcal{S}}\) and encodes both the original feature \(X^{\mathcal{S}}\) and the interaction effects (from the assortment) into a latent feature matrix \(\tilde{X}^{\mathcal{S}}\in\mathbb{R}^{S\times d_{v}}\). Here each item \(i\) is associated with a latent feature vector \(\tilde{X}^{\mathcal{S}}_{i}\in\mathbb{R}^{d_{v}}\). In this feature transformation, the assortment encoder can contain \(L\) layers (which is a hyperparameter), and each with three sub-layers:

First, an optional embedding sub-layer processes the original input \(X^{\mathcal{S}}\) by an element-wise fully connected feed-forward network layer applied to each item in the input assortment matrix separately, via two linear operations and an activation function \(\phi\):

\[f^{\text{FFN}}(x)\coloneqq W_{2}\phi(W_{1}x+b_{1})+b_{2},\]

where \(W_{1}\in\mathbb{R}^{d_{v}\times d}\), \(W_{2}\in\mathbb{R}^{d_{v}\times d_{v}}\) and \(b\in\mathbb{R}^{d_{v}}\) are trainable parameters. The dimension \(d_{v}\) controls the embedding's complexity.

Second, a self-attention sub-layer implements the attention mechanism defined in (5). This layer captures the interaction effects between the items within the assortment.

Third, a fully connected feed-forward network layer \(f^{\text{FFN}}(x)\) (similar to the first part) is applied to each item separately to further process each item's latent features.

The _candidates encoder_ processes the original input \(X^{\mathcal{C}}\) in a similar manner as the assortment encoder.

Figure 1: Architecture of Transformer Choice Net (TCNet). We omit the residual connections within each sub-layer.
It processes the original feature and captures the interaction effect between items within the candidate set through three modules as the assortment encoder: an optional embedding sub-layer, a self-attention sub-layer, and a fully-connected feed-forward layer. One special point is an additional cross-attention sub-layer that mingles the assortment set and the candidate set. Specifically, this assortment-candidate attention sub-layer takes (i) the output of the assortment encoder \(\tilde{X}^{\mathcal{S}}\in\mathbb{R}^{S\times d_{v}}\) and (ii) the output \(X\) of the self-attention sub-layer of the candidates encoder as input, and then maps them to a latent feature matrix \(\tilde{X}^{\mathcal{C}\cdot\mathcal{S}}\in\mathbb{R}^{C\times d_{v}}\) that is aware of the assortment context. Mathematically, this is represented by

\[f^{\text{AC}}(X)=\text{Att}(XW_{Q},\tilde{X}^{\mathcal{S}}W_{K},\tilde{X}^{ \mathcal{S}}W_{V})\]

where \(W_{Q},W_{K},W_{V}\in\mathbb{R}^{d_{v}\times d_{v}}\) are trainable parameters.

The _utility decoder_ processes the output latent features from the candidates encoder into some latent utilities for each item in \(\mathcal{C}\). Ideally, the output latent feature \(\tilde{X}^{\mathcal{C}\cdot\mathcal{S}}_{i}\) from the candidates encoder has encoded both the candidate set effect and the assortment set effect. The utility decoder transforms each of this latent feature vector \(\tilde{X}^{\mathcal{C}\cdot\mathcal{S}}_{i}\) into a (scalar) latent utility \(\tilde{u}_{i}\), and the transformation shares the same weight/parameter across different items. Here we apply a fully-connected feed-forward network, which can be replaced by other network architectures. Lastly, an MNL model/softmax layer converts these latent utilities into choice probabilities.

### Remarks on TCNet

We make the following remarks on the TCNet.

**Assortment size flexibility**. One advantage of the attention mechanism is that it can handle input (e.g., assortment set and candidate set) with varying sizes. This is useful in a choice modeling context because the assortment and the candidate sets may have variable cardinality. Moreover, the number of parameters in TCNet does not depend on the total number of items \(|\mathcal{N}|\) which makes it attractive for the case when there are a large number of items or newly added items. As long as the item/product feature of a new item/product is available, the TCNet can be used seamlessly for assortments with newly launched products.

**Permutation equivariance**. For any permutation \(\sigma\) applied to the rows of \(X\) (such as interchanging the first and second row), the \(f^{\text{Att}}\) satisfies _Permutation Equivariant_(Lee et al., 2019), which is defined as \(f^{\text{Att}}(\sigma(X))=\sigma(f^{\text{Att}}(X))\). This ensures that each output choice probability remains invariant to the ordering/position of the input. This is useful for choice modeling context where the items in the assortment/candidate set are presented to the customer all at once. In this case, there is no ordering of the products and thus the customer choice should be permutation equivariant. Of course, when the position/order does matter in determining choice, one can encode the position/order as a feature.

**Comparison against other NN-based choice models.** In Table 1, we make a comparison

\begin{table}
\begin{tabular}{c|c|c|c} \hline \hline Model & \# parameters & Variable size & Assortment effect \\ \hline TCNet (Ours) & \(O\left(dd_{v}+Ld_{v}^{2}\right)\) & Yes & Yes \\ DeepMNL(Wang et al., 2020; Sifringer et al., 2020) & \(O\left(dd_{v}+Ld_{v}^{2}\right)\) & Yes & No \\ AssortNet (Wang et al., 2023) & \(O\left(\|\mathcal{N}\|dd_{v}+Ld_{v}^{2}\right)\) & No & Yes \\ SDANet (Rosenfeld et al., 2020) & \(O\left(dd_{v}+Ld_{v}^{2}\right)\) & Yes & Yes \\ FATENet (Pfannschmidt et al., 2022) & \(O\left(dd_{v}+Ld_{v}^{2}\right)\) & Yes & Yes \\ DLCL (Tomlinson and Benson, 2021) & \(O\left(d^{2}\right)\) & Yes & Yes \\ RUMMet(Aouad and Desir, 2022) & \(O\left(dd_{v}+Ld_{v}^{2}\right)\) & Yes & No \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison on the model complexity, where \(d\) and \(d_{v}\) are dimensions of item and latent features respectively, \(L\) is the number of layers, and \(\|\mathcal{N}\|\) is the total number of items.
between TCNet and several other neural network-based choice models (for single-choice) from several aspects. The column "variable size" refers to whether the model requires a fixed assortment size or allows a variable size. The column "assortment effect" refers to whether the model captures the assortment effect, i.e., the interactions between items within the assortment.

In terms of the number of parameters, we can see that TCNet has the same order of the number of parameters as other models, thus its empirical advantage (in the numerical experiment) should be attributed to its architecture rather than model size. In particular, we note that SDANet (Rosenfeld et al., 2020) and FATENet (Pfannschmidt et al., 2022) essentially implement the Deep Set (Zaheer et al., 2017) for choice modeling, and thus capture item-wise interaction on a coarser scale than our TCNet. DLCL (Tomlinson and Benson, 2021) explicitly models product interactions through linear functions and does not introduce a neural network architecture.

**Transformer specializations for choice modeling.** The Transformer (Vaswani et al., 2017) is originally designed for seq2seq tasks, such as machine translation and text summarization. The model predicts the likelihood of the subsequent word from the vocabulary list. In comparison, the TCNet is designed for a set-to-member task, where the goal is to forecast the item(s) to be chosen from the candidate set and thus it requires the permutation equivariance. This necessitates modifications to the input's structure and the utility decoder. Similarly, Lee et al. (2019) introduce the Set Transformer which is designed to model interactions within an input set. Different from TCNet, the Set Transformer aims for permutation invariance (\(f(X)=f(\sigma(X))\) for any permutation \(\sigma\) over an ordered set \(X\)) which keeps the whole output invariant through permuting inputs. In the language of choice modeling, permutation invariance assumes that the assortment effect will take place through the assortment set as a whole \(\mathcal{S}\) (or \(\mathcal{C}\)) and keep same for all items, while the permutation equivariance allows a finer interaction between the items.

**Training:** We use the cross-entropy loss (CE loss) for the training. With dataset \(\mathcal{D}=\{(i_{k},\{\mathcal{C}_{k},\mathcal{S}_{k}\}),k=1,\ldots,m\}\), the CE loss is defined by

\[L_{\text{CE}}=-\frac{1}{m}\sum_{k=1}^{m}\log(f_{i_{k}}^{\text{TCN}}(\mathcal{C }_{k},\mathcal{S}_{k})). \tag{6}\]

Several tricks/techniques are used in the original Transformer (Vaswani et al., 2017) such as layer normalization, multi-head attention, residual connection, and dropout. We also implement them for TCNet.

## 5 Expressiveness of TCNet

The seminal work (Manski, 1977) shows that any single-choice model can be represented by \(O(2^{n}\times n)\) utility parameters: \(\{u_{i}^{\mathcal{S}},i\in\mathcal{N},\mathcal{S}\subseteq\mathcal{N}\}\). Specifically, for all \(i\in\mathcal{S}\) and \(\mathcal{S}\subseteq\mathcal{N}\),

\[\mathbb{P}(i|\mathcal{S})=\frac{\exp(u_{i}^{\mathcal{S}})}{\sum_{j\in\mathcal{ S}}\exp(u_{j}^{\mathcal{S}})} \tag{7}\]

where the utility \(u_{i}^{\mathcal{S}}\) represents the _contextualized_ utility of the \(i\)-th item under the assortment \(\mathcal{S}\). Bat-sell and Polking (1985) further show that the utility \(u_{i}^{\mathcal{S}}\) can be decomposed into a sum of item-wise interactions among the subsets of the assortment which is also recently discussed by Seshadri et al. (2019).

**Theorem 1** ((Batsell and Polking, 1985)).: _For any \(\{u_{i}^{\mathcal{S}}:i\in\mathcal{N},\mathcal{S}\subseteq\mathcal{N}\}\), there exists a unique set of
parameters \(\{v_{i}^{\mathcal{S}}:i\in\mathcal{S},\mathcal{S}\subseteq\mathcal{N}\}\) where \(\sum_{i\notin\mathcal{S}}v_{i}^{\mathcal{S}}=0,\forall\mathcal{S}\subset\mathcal{N}\), such that for all \(i\in\mathcal{S}\) and \(\mathcal{S}\subset\mathcal{N}\),_

\[u_{i}^{\mathcal{S}}=\sum_{S^{\prime}\subseteq\mathcal{S}\setminus i}v_{i}^{S^ {\prime}}.\]

The above theorem shows that to estimate a single-choice model, it is enough to estimate the interaction effects of each item \(i\), i.e.,

\[\{v_{i}\},\{v_{i}^{\{j\}}:j\in\mathcal{N},j\neq i\},\ldots,\{v_{i}^{\mathcal{N }\setminus\{i\}}\}.\]

Here the constraints \(\sum_{j\notin\mathcal{S}}v_{j}^{\mathcal{S}}=0,\forall\mathcal{S}\subset \mathcal{N}\) ensure the uniqueness of the parameters (noting the scale-invariance of the utilities in (7)). Based on the level of interaction, we call \(\{v_{i}\}\) as 0-th order interaction, \(\{v_{i}^{\{j\}}\}\) as 1-st order (or pairwise) interaction, and so on. The classic MNL model captures only 0-th order interaction, and several studies (Seshadri et al., 2019; Bower and Balzano, 2020; Yousefi Maragheh et al., 2020; Tomlinson and Benson, 2021; Najafi et al., 2023) explore diverse formulations to capture 1st-order interactions \(\{v_{i}^{\{j\}}\}\). In this sense, TCNet can capture \(L\)-th order interactions with \(L\) the number of layers in the self-attention module. Thus \(L\) provides a statistical handle for controlling the model complexity and also a modeling handle for controlling the level of item-wise interactions.

In parallel, one can parameterize any sequential choice model with utility parameters:

\[\mathbb{P}(i|\mathcal{C},\mathcal{S})=\frac{\exp(u_{i}^{\mathcal{C},\mathcal{S }})}{\sum_{j\in\mathcal{C}}\exp(u_{j}^{\mathcal{C},\mathcal{S}})}, \tag{8}\]

where the utilities can be further decomposed by

\[u_{i}^{\mathcal{C},\mathcal{S}}=\sum_{\mathcal{C}^{\prime}\subseteq\mathcal{ C}\setminus i}v_{i}^{\mathcal{C}^{\prime},\mathcal{S}}.\]

Here the term \(v_{i}^{\mathcal{C}^{\prime},\mathcal{S}}\) captures the interaction effects of (i) the candidate subset \(\mathcal{C}^{\prime}\) towards item \(i\) and (ii) the assortment effect from \(\mathcal{S}.\) The interaction from the subset \(\mathcal{C}^{\prime}\) can be viewed as a candidate interaction effect, while the interaction from \(\mathcal{S}\) can be viewed as a background contextual effect. This decomposition provides intuition for why the TCNet architecture is suitable for the choice modeling task: the assortment encoder encodes the global contextual effect from \(\mathcal{S}\), and the candidate encoder encodes the local candidate interaction effect.

Theoretically, this utility decomposition also provides a guarantee on the model expressiveness of TCNet for the choice modeling task as follows.

**Theorem 2**.: _Any sequential choice model can be represented by a Transformer Choice Net. Specifically, for any sequential choice model defined by (2), there exists a Transformer Choice Net \(f^{\text{TCN}}\) such that_

\[\mathbb{P}(i|\mathcal{C},\mathcal{S})=f_{i}^{\text{TCN}}(\mathcal{C},\mathcal{ S})\]

_holds for all \(i\in\mathcal{C},\mathcal{C}\subseteq\mathcal{S}\) and \(\mathcal{S}\subseteq\mathcal{N}\)._

The proof is deferred to Appendix A. It is inspired by (Lee et al., 2019) which proves the approximability of the Transformer Set for the permutation invariant function family (but not the permutation equivariant needed here for choice modeling). Combined with our discussion in SS3, we yield the following result.

**Corollary 1**.: _Any single-choice model or multi-choice model can be represented by a Transformer Choice Net._
Numerical Experiments

In this section, we present numerical experiments to demonstrate the performance of TCNet. Then we give visualizations and interpretations of the learned TCNet.

### Numerical Performance on Real Datasets

We evaluate TCNet's performance against multiple benchmark models for the three choice modeling tasks we consider in this paper: single-choice, sequential choice, and multi-choice. We emphasize that for all the models and datasets, we use a single architecture of TCNet to showcase its power of "one-size-fits-all".

Each experiment is conducted 5 times, with training, validation, and test data randomly split at ratios of \(60\%,20\%,\) and \(20\%\), respectively. The reported numbers are based on the average values over the test data, accompanied by the standard deviations. Further details on the datasets, the implementation, and the hyperparameter tuning are given in Appendix B.

**Single-choice prediction.** We use the cross-entropy (CE) loss (6) to train the TCNet and also as the performance metric to report the test performance in single-choice prediction. We compare the TCNet against several benchmark models: DeepMNL (Wang et al., 2020; Sifringer et al., 2020), AssortNet (Wang et al., 2023), SDANet (Rosenfeld et al., 2020), FATENet (Pfannschmidt et al., 2022), Mixed-MNL (McFadden and Train, 2000), and DLCL (Tomlinson and Benson, 2021). These benchmarks represent the state-of-the-art methods of single-choice modeling. Our experiment includes the public datasets used in these existing works: Car (McFadden and Train, 2000), Sushi (Kamishima, 2003), Expedia(book), Hotel (Bodea et al., 2009), SFwork and SFshop (Koppelman and Bhat, 2006); as well as two private datasets Flight, and Retail.

**Sequential choice prediction.** We also use the CE loss as the training objective and the performance metric in sequential choice prediction. We consider two public datasets commonly used in existing works: Expedia(click) and Bakery. These two datasets are originally multi-choice datasets without purchasing sequence information. We transform them into the sequential choice data by a method described in Appendix B. Though several algorithms are developed for sequential choice prediction, their limitations make them inapplicable with these specific datasets. As a result, for benchmarking purposes, we formulate sequential choice prediction as a single-choice prediction problem and utilize the above-mentioned benchmark models.

Specifically, for a sequential choice dataset denoted as \(\mathcal{D}=\{(i_{k},\{\mathcal{C}_{k},\mathcal{S}_{k}\}),k=1,...,m\}\), we drop \(\mathcal{S}_{k}\) and recast it as a single-choice dataset: \(\mathcal{D}=\{(i_{k},\mathcal{C}_{k}),k=1,...,m\}\) for both the training and testing of the single-choice benchmark models. In essence, these models predict a single-choice from the candidate set \(\mathcal{C}\) as the assortment and omits the original assortment \(\mathcal{S}\), which may cause the information from what have been chosen unavailable (i.e., the assortment context) and only focus on modeling the candidates effect.

To assess the implications of overlooking the assortment context, we train two variants of the Transformer Choice Net: TCNet(Var) and TCNet, where "Var" denotes the variant version. The former is trained as the single-choice model based on the above method while the latter is trained by minimizing the CE loss (6) over the original sequential choice dataset.

**Multi-choice prediction.** We adopt the F1 score loss as the performance metric for multiple choices prediction, in line with the approach in (Pfannschmidt et al., 2022). Given the dataset \(\mathcal{D}=\{(\mathcal{B}_{k},\mathcal{S}_{k}),k=1,\ldots,m\}\), the F1 score loss is defined by

\[L_{\text{F1}}=1-\frac{2}{m}\sum_{k=1}^{m}\frac{|\tilde{\mathcal{B}}_{k}\cap \mathcal{B}_{k}|}{|\tilde{\mathcal{B}}_{k}|+|\mathcal{B}_{k}|}.\]
Here, \(\mathcal{B}_{k}\) represents the algorithm's predicted multiple choices. This loss provides a balanced measure of model performance on imbalanced datasets, which is the case for most choice modeling data (far fewer chosen item(s) than unchosen ones).

We focus solely on the Expedia dataset here. The Bakery dataset, due to its consistent assortment and unvarying item features, results in identical multi-choice predictions across all samples when deterministic prediction methods are employed which renders it unsuitable for algorithmic comparison.

As discussed in SS2, models designed for multiple choices become computationally expensive with increasing assortment sizes. Following the strategy in (Pfannschmidt et al., 2022), we adopt several single-choice models for multi-choice prediction: if an algorithm can produce interim utilities, \(u_{i}^{\mathcal{S}}\) for each item \(i\) within assortment \(\mathcal{S}\), then the item's choice probability is \(\frac{\exp(u_{i}^{\mathcal{S}})}{1+\exp(u_{i}^{\mathcal{S}})}.\) And resultant predicted multiple choices are

\[\left\{i\in\mathcal{S}:\frac{\exp(u_{i}^{\mathcal{S}})}{1+\exp(u_{i}^{\mathcal{ S}})}>\mu\right\} \tag{9}\]

Here, the threshold \(\mu\) is a tunable hyperparameter. In this case, the training loss remains the CE loss, yet is applied to each item's choice outcome independently. We use TCNet to denote the TCNet trained by the method discussed in SS3.2 and use TCNet(Var) as the one trained by the above method.

**Results.** The performances on the test sets across all the experiments are summarized in Table 2. The Transformer Choice Net (TCNet) exhibits superior performance across all three tasks and datasets. One interesting observation is about the sequential choice prediction on the Bakery data: the TCNet(Var), which is trained without feeding the assortment information outperforms the standard TCNet, which enjoys the assortment information. One explanation is that the uniformity of assortments in all Bakery data samples allows an implicit encoding of this invariant assortment information. Conversely, in the Expedia(click) dataset, where assortments largely vary across samples, the provision of the additional assortment information appears beneficial, enhancing the performance of the standard TCNet. Also, as noted in the last section, the TCNet has a comparable number of parameters as the benchmark networks, so its advantage should be attributed to its explicit modeling of the item interaction.

### Probing into the Representation of TCNet

We train a TCNet on the Bakery dataset which contains a total of 50 items, including food (e.g., cakes with different flavors, and snacks like cookies) and drinks (e.g., different coffees). The original dataset contains 4 item features. However, we train the model without these features and only use a one-hot encoding for each item. We then extract the latent features of each product from the assortment encoder and do linear probing (Belinkov, 2022) of the model through running linear logistic regression for two tasks: (1) identify cakes from snacks; (2) identify drinks from food. Due to the limited space, we only show the first here and defer the second to the Appendix.

\begin{table}
\begin{tabular}{c c|c|c|c|c|c|c|c|c} \hline Task & Data & TCNet & TCNet(Var) & DeepMNL & AssortNet & SDANet & FATENet & Mixed-MNL & DLCL \\ \hline \multirow{8}{*}{Sin.} & Car & **1.517\(\pm\)**0.026** & - & 1.578\(\pm\)0.013 & 1.585\(\pm\)0.032 & **1.517\(\pm\)**0.026 & 1.521\(\pm\)0.024 & 1.579\(\pm\)0.029 & **1.517\(\pm\)**0.025 \\  & Sushi & **1.747\(\pm\)**0.011 & - & 1.789\(\pm\)0.007 & 1.491\(\pm\)0.016 & **1.747\(\pm\)**0.013 & 1.750\(\pm\)0.010 & 1.764\(\pm\)0.008 & 1.763\(\pm\)0.007 \\  & Expedia(hook) & **2.314\(\pm\)**0.021 & - & 2.408\(\pm\)0.014 & 2.389\(\pm\)0.021 & 2.328\(\pm\)0.020 & 2.350\(\pm\)0.019 & 2.351\(\pm\)0.020 & 2.340\(\pm\)0.017 \\  & Hotel & **0.778\(\pm\)**0.008 & - & 1.321\(\pm\)0.014 & 0.851\(\pm\)0.008 & **0.778\(\pm\)**0.006 & 0.803\(\pm\)0.006 & 0.882\(\pm\)0.004 & 0.885\(\pm\)0.003 \\  & SFwork & **1.539\(\pm\)**0.031 & - & 1.902\(\pm\)0.167 & 1.667\(\pm\)0.026 & 1.542\(\pm\)0.027 & 1.570\(\pm\)0.033 & 1.664\(\pm\)0.022 & 1.683\(\pm\)0.014 \\  & SFshop & **0.821\(\pm\)**0.006 & - & 1.265\(\pm\)0.140 & 0.928\(\pm\)0.024 & **0.821\(\pm\)**0.010 & 0.831\(\pm\)0.015 & 0.909\(\pm\)0.022 & 0.894\(\pm\)0.011 \\  & Flight & **0.412\(\pm\)**0.007 & - & 2.819\(\pm\)0.051 & 2.751\(\pm\)0.049 & 0.427\(\pm\)0.007 & 0.477\(\pm\)0.006 & 0.476\(\pm\)0.005 & 0.477\(\pm\)0.006 \\  & Real & **0.982\(\pm\)**0.021 & - & 1.429\(\pm\)0.073 & 1.032\(\pm\)0.008 & 0.992\(\pm\)0.022 & 0.996\(\pm\)0.023 & 0.997\(\pm\)0.023 & 0.997\(\pm\)0.023 \\ \hline \multirow{2}{*}{Seq.} & Bakery & 2.738\(\pm\)0.024 & **2.710\(\pm\)**0.029 & 3.817\(\pm\)0.006 & 2.767\(\pm\)0.016 & 2.729\(\pm\)0.009 & 3.808\(\pm\)0.006 & 3.557\(\pm\)0.001 & 3.811\(\pm\)0.006 \\  & Expedia(click) & **2.863\(\pm\)**0.017 & 2.890\(\pm\)0.023 & 2.913\(\pm\)0.020 & 2.922\(\pm\)0.015 & 3.067\(\pm\)0.023 & 2.974\(\pm\)0.023 & 2.924\(\pm\)0.024 & 2.937\(\pm\)0.012 \\ \hline Mul. & Expedia(click) & **0.576\(\pm\)**0.005 & 0.583\(\pm\)0.003 & 0.659\(\pm\)0.003 & 0.653\(\pm\)0.005 & 0.593\(\pm\)0.002 & 0.639\(\pm\)0.005 & - & - \\ \hline \end{tabular}
\end{table}
Table 2: Test performances (\(\downarrow\)) on three tasks: single-choice (Sin.), sequential choice (Seq.), and multi-choice (Mul.).
Figure 2 showcases our results: Figure 1(a) shows the predicted probabilities to be the cake, and Figure 1(b) uses t-SNE (Van der Maaten and Hinton, 2008) to visualize the top-10 latent features with largest absolute coefficients from the logistic regression. Both figures are based on the testing samples. We note that the latent representation indeed has the ability to identify Cake from Snack, although this is not explicitly included as the feature (only implicitly encoded in the choice). This shows that by learning the choice model from customer behaviors, the latent features can uncover useful properties of items, such as their categories and characteristics.

### Attention Visualization

The attention scores in TCNet across various attention sub-layers reveal the degree of "attention" allocated by the model to each item (whether from an assortment or candidates set) while processing a specific item of interest. Essentially, these scores give insights into the influences and correlations (such as substitutes) of items on the final choice prediction. Moreover, they can help to verify whether the model is concentrating on the important items as we expect. In this subsection, we visualize the attention scores from a trained TCNet as an example to demonstrate how we can understand and explain them.

The TCNet is again trained on the Bakery dataset. Figure 3 shows two sets of self-attention scores (Figure 2(a) and 2(b)) and two cross-attention scores (Figure 2(c) and 2(d)) from the candidates encoders. We make the following observations:

Comparing Figures 2(a) and 2(b), we can see Cake 1 and Cake 2 put extremely high weights on the lemonade in Figure 2(b) in contrast to the Cake 3 offered in Figure 2(a). This aligns with one's intuition when making the choice: Lemonade, as a drink, represents a new category differing from the other candidates and can impose a substantial influence on the final choice. For instance, a thirsty customer might randomly choose one cake in Figure 2(a) but will definitely choose the lemonade when it is offered as in Figure 2(b).

In Figures 2(c) and 2(d), the first observation is consistent with the above: all three cakes assign substantial weights to Coffee 1, which is a drink, in both figures. Another observation is on the weight of Coffee 1: its weight on Coffee 2 in Figure 2(c) is double of that on the Pie in Figure 2(d). Given that both Coffee 2 and Pie are purchased items, this implies that buying Coffee 2 has a greater influence on choosing Coffee 1 than purchasing the Pie. This is reasonable, as consumers typically do not buy two coffees but may purchase a coffee alongside a pie.

Figure 2: Probing the latent representation.
In summary, examining the attention scores enables us to validate whether the model adheres to our common sense and allows us to get insights into the relationships between items.

## 7 Conclusion and Future Directions

The existing literature on single-choice model, sequential choice model, and multi-choice model have been largely separate from each other: the current sequential choice models have there own limitations, and the current multi-choice models become impractical when dealing with a large assortment.

While the research on single-choice models is vast, the literature on predicting sequential and multi-choice is noticeably less developed. Our paper introduces the Transformer Choice Net as a unified neural network framework capable of addressing all three types of choice behaviors. Notably, it is the first

Figure 3: Candidates encoder’s (self) attention and cross-attention scores. For the second row, the side items are from candidates \(\mathcal{C}\) and the bottom items are from assortment \(\mathcal{S}\). Each row indicates the normalized attention scores for the side item.
Transformer-based architecture tailored for choice(s) prediction tasks. We prove its universal representation capacity and, through extensive numerical experiments, demonstrate its superior performance over several benchmarks. We conclude with the following future directions.

**Assortment optimization.** One important downstream application of choice prediction is assortment optimization, which optimizes the offered assortment to maximize the expected revenue for the seller. Given the Transformer Choice Net's strong prediction performance, an intriguing avenue to explore would be the optimization of assortment offerings for a trained Transformer Choice Net model.

**Exploring alternative Transformer-based structures.** The Transformer architecture has increasingly shown its significance within the deep learning realm, showcasing its adaptability across diverse tasks. While the Transformer Choice Net represents an initial endeavor in integrating the Transformer framework into choice modeling, we believe there is still large potential in exploring other Transformer-inspired architectures.
