Double Machine Learning at Scale to Predict Causal Impact of Customer Actions
Sushant More Priya Kotwal Sujith Chappidi Dinesh Mandalapu Chris Khawand










S. More et al.



Amazon, Seattle WA, USA 

{morsusha,kotwalp,jcchappi,mandalap,khawandc}@amazon.com



Double Machine Learning at Scale to Predict Causal Impact of Customer Actions
    Sushant More0000-0002-3746-2431 Priya Kotwal0009-0004-6599-359X Sujith Chappidi0009-0009-3310-6067 Dinesh Mandalapu0009-0007-2984-859X Chris Khawand0009-0000-5283-9391

    
============================================================================================================================================================================



§ ABSTRACT 
Causal Impact (CI) of customer actions are broadly used across the industry to inform both short- and long-term investment decisions of various types.  In this paper, we apply the double machine learning (DML)
methodology to estimate the CI values across 100s of customer actions of business interest and 100s of millions of customers.    
We operationalize DML through a causal ML
library based on Spark with a flexible, JSON-driven model configuration approach
to estimate CI at scale (i.e., across hundred of actions and millions of customers). We
outline the DML methodology and implementation, and associated benefits over
the traditional potential outcomes based CI model. We show population-level as well as customer-level CI
values along with confidence intervals.  The validation metrics show a 2.2% gain over the baseline methods and a 2.5X gain in the computational time. Our contribution is to advance the scalable
application of CI,  while also providing an interface that allows faster
experimentation,  cross-platform support,  ability to onboard new use cases,  and
improves accessibility of underlying code for partner teams. 








§ INTRODUCTION


Causal Impact (CI) is a measure of the incremental change in a customer's outcomes (usually spend or profit) from a customer event or action (e.g, signing up for a paid membership).  CI values are used across the industry as important signals of long-term value for multiple decisions,  such as marketing content ranking to long-term investment decisions.  


Business teams are typically interested in calculating CI values for relevant actions that a customer participates in.  Some examples include customer actions such as `first purchase in category X',  `first Y stream',  or `sign up for program Z'[We use placeholder X,Y,Z to maintain business confidentiality]. 
The CI values are leveraged by partner teams to understand and improve the value they generate.   For many of these customer actions,  we are unable to conduct A/B experiments due to practical or legal constraints.  CI values are thus estimated off of observational data, effectively leveraging rich customer data to isolate causal relationships in the absence of a randomized experiment.

In this paper, we provide results for average treatment effects and conditional average treatment effects (i.e., customer-level CI values) estimated using a variant on the Double Machine Learning (DML) methodology <cit.>. The paper is arranged as follows. In Sec. <ref>, we give a brief overview of the use and scale of CI in the industry.  In Sec. <ref>, we introduce the traditional system used for calculating CI values. We also discuss the shortcomings of the traditional method and the advantages of moving to a DML-based method for calculating CI. 

Sec. <ref> covers the details of our DML implementation for calculating CI values. Our contributions include improving the robustness of CI estimates through inverse propensity weighting, adding the ability to produce heterogeneous CI values, implementing customer-level confidence intervals with various assumptions,  and making available the JSON Machine Learning interface to accelerate experimentation.  We present results in Sec. <ref> for a few customer actions and conclude with the takeaways and ideas for future work in Sec. <ref>.



§ CAUSAL IMPACT ESTIMATION IN INDUSTRY



Causal impact estimation drives a large number of business decisions across industry.  This includes multiple organizations such as retail, search,  devices,  streaming services, and operations. To this end,  most companies have invested in developing and deploying models that vend CI values for the cutsomer actions under consideration.  In the next section, we give an overview of the traditional potential-outcome based model which is widely used in the industry for CI estimation.  This will be the baseline model.   



 §.§ CI: Potential Outcomes framework



CI framework applies the principles of observational causal inference. We rely on it because A/B testing is not possible to evaluate the impact of certain treatments due to practical constraints (e.g., the treatment is not effectively assignable, or would be too expensive to assign at scale). Observational causal inference methods rely on eliminating potential confounders through adjustment on observed variables. Under a "selection on observables" assumption, we believe we can estimate the causal effect correctly on average.  Applied to the customer's next 365 days of spending, for example, the CI value represents the incremental spending that a customer makes because of participating in a certain action compared to the counterfactual case where they didn’t take that particular action. The formal framework for this kind of counterfactual reasoning is the "potential outcomes" framework, sometimes known as the Neyman-Rubin causal framework <cit.>,  <cit.>, <cit.>. 

The potential outcome based CI model has two parts:


  * Propensity binning. Group the customer based on their propensity to participate in the action. This is done based on features that relate to recency, frequency, and the monetary behavior of customers along with their other characteristics such as their tenure type.

  * Regression adjustment. In each of the groups, we build a regression model on the control customers with customer-spend as the target. The trained model is applied on the treatment customers to predict the counterfactual spend (how much would customer have spent if they didn’t participate in the action). The difference between the predicted counterfactual and the actual spend is the CI value. We take a weighted average across different groups to get the final CI value for the customer action. 





In addition, we require the CI model to be able to scale to the business use case.  For instance,  we may want generate CI values for hundreds of customer actions in an automated way.  In the rest of the paper, we refer to the traditional potential outcome based CI model as “CI-PO” and the DML-based model as  “CI-DML”.  



§ CI: DML FRAMEWORK




Note that one of the challenges in validating the causal estimates is posed by the Fundamental Problem of Causal Inference <cit.>. The lack of observable ground truth makes it difficult to validate the output of a causal model. We therefore prefer models with proven desirable theoretical properties such as √(n) convergence. 

The Double/Debiased Machine learning (DML) method proposed by Chernozhukov et al. <cit.> leverages the predictive power of modern Machine Learning (ML) methods in a principled causal estimation framework that is free of regularization bias asymptotically. 

For treatment D, features X, we express the outcome Y as an additively separable function of D and arbitrary function of features X:


    Y = D β + g(X) + ϵ

DML’s estimation strategy is motivated by writing out the residualized representation of Eq. (<ref>) and its parts:

    Ỹ   =    Y - E(Y|X) 
    D̃    =    D - E(D|X) 
    Ỹ   =   D̃β + ϵ̃

We use ML models to estimate E(Y|X) and E(D|X). The residuals from outcome equation (Eq. (<ref>)) are regressed on residuals from propensity equation (Eq. (<ref>)) to obtain the causal parameter β.  We use ML models to predict E(Y|X) and E(D|X).  We leverage K-fold sample splitting so that training and scoring of the ML models happens on different folds. We use a 3-fold sample split and follow the “DML2” approach <cit.> where we pool the residuals outcome and propensity residuals across all the folds to fit a single, final regression of the residualized outcome on the residualized treatment (Eq. (<ref>)).



 §.§ Inverse Propensity Treatment Weighting



We use a weighted ordinary least squares to solve the residual regression equation (Eq. (<ref>)). where the weights are determined by the Inverse Propensity Treatment Weighting (IPTW or IPW) <cit.>. Our IPTW weights correspond to the Horvitz-Thompson (HT) weight <cit.>,  in which the weight for each unit is the inverse of the probability of that unit being assigned to the observed group.
In Table <ref> we define the weights that balance the distributions of covariates between comparison groups for two widely used estimands, the Average Treatment Effect (ATE) and Average Treatment Effect on the Treated (ATT). Weighting helps achieve additional robustness, bringing us closer to a conventional Doubly Robust estimator.  Applying these weights when conducting statistical tests or regression models helps reduce impact of confounders over and above what we get from the regression adjustment <cit.>. Secondly, the weights allow us to target the estimand; the ATT has long been the preferred estimand for CI values,  since it represents the treatment effects for those customers actually treated historically.  (We refer to the customer-level counterparts of ATE, ATT estimands as HTE and HTT respectively.)







 §.§ Common support and propensity score trimming



For many treatments, propensity distribution has significant mass near 1 for the treated group and near 0 for the control group (see an example histogram in Fig. <ref>).  Scores near the boundary can create instability in weighting methods.  In addition, these scores often represent units for whom we cannot make an adequate treatment-control comparison. We limit analysis to the common support region, where propensity score distributions overlap between treated and untreated samples. 

  

We also use trimming to exclude customers whose estimated propensity is outside of the range [α, 1-α].  We experimented with different thresholds on various customer actions and observed that α=0.001 with rescaled propensity scores works the best.



  §.§.§ Normalizing and rescaling weights


When using the IPW, we normalize the weights by rescaling the propensity scores for each customer i as in Eq. (<ref>).

    ê(X_i)_ scaled = (Dê(X)) ∗ê(X_i)

D and ê(X) in Eq. (<ref>) are the averages of treatment assignment and propensity score respectively taken over both the treatment and control population combined. 
Propensity trimming and rescaling reduces variance, leads to more stable estimates, and tighter confidence intervals as seen in Fig. <ref>. 




 §.§ Heterogeneity in DML

CI-DML implements a version of the heterogenous effects modeling proposed in <cit.>, by leveraging the treatment-feature interactions in the final stage of DML to identify heterogenous (customer-level) responses. 
The general form of Eq. (<ref>) can be written as 

    Ỹ = h(X, D̃) + ϵ̃ .
 
In fact, Eq. (<ref>) is a special case of Eq. (<ref>) with h(X, D̃) = D̃β.  We interact treatment with the features and define h(X, D̃) ≡ψ(X) * D̃ β,  where `∗' represents element-wise multiplication. Thus, the heterogeneous residual regression becomes:

    Ỹ = ψ(X) * D̃ β + ϵ̃

We want ψ(X) to be low-dimensional so that we are able to extract the coefficient β in Eq. (<ref>) reliably. 

Let N and M be the number of customers and features respectively. If the dimension of ψ(X) is N× K, we want K ≪ M. In our use case,  M is typically  2000 and K is typically around 20. To get the low-dimensional representation, ψ(X) we proceed as follows: 


  * We project the original features onto an orthogonal space through Principal Component Analysis (PCA). 

  * We run a K-means clustering algorithm on the highest-signal Principal Components. Dimension reduction from PCA helps to reduce dimensionality-related problems when computing Euclidean distance for K-means clustering.

  * We calculate the K cluster scores for each customer, as ψ_i,c = 1/d_i,c∑_k=1^K 1/d_i,k where d_i,c is the distance of customer i's value from centroid of cluster c (see schematic in Fig. <ref>).




Once we have calculated the distance features ψ(X) for each customer, we interact them with the propensity residuals and fit a linear regression model using IPW (refer sec. <ref>) to extract the coefficients β in Eq. (<ref>). The heterogenous estimates are given by 

    h = ψ(X) β .

E.g., for K=3,  h = ψ_1 β_1 +ψ_2 β_2 + ψ_3 β_3. 

A schematic of CI-DML workflow is shown in Fig. <ref>.






 §.§ Confidence intervals in DML


One of the disadvantages of CI-PO is that generating confidence intervals requires bootstrapping around the multi-step process and is computationally expensive.  Obtaining confidence intervals in CI-DML is straightforward. For a single ATT parameter estimate, we obtain the confidence interval simply by calculating the variance of the estimate of β in Eq. (<ref>). We also estimate Huber-White heteroscedasticity consistent standard errors <cit.>, <cit.>. For the ATT case, the steps for calculating variance of the coefficient β̂ are as follows:

    Var(β) = H Σ̂ H' .


For a customer, `p':

    σ̂_p^2 = Û_p^2 = (Ỹ_p - D̃_p β̂)' (Ỹ_p - D̃_p β̂) ,

where β̂ is the value of coefficient from solving Eq. (<ref>).  Note that Ỹ_p and D̃_p are scalars. 

Σ in Eq. (<ref>) is a diagonal matrix with the squared prediction error σ̂_p^2  for each customer on its diagonal and H in Eq. (<ref>) is defined as 

    H = (D̃' ∗ W D̃)^-1D̃' ∗ W

where W are the IPW weights as defined in Sec. <ref>. 

We compute the confidence intervals on the causal estimate β using Var(β).



  §.§.§ Customer-level confidence intervals


CI-DML also provides the ability to obtain customer-level confidence intervals. From Eq. (<ref>),  we can write

    Var(h) = Var(ψβ) =  ∑_k ψ_k^2 Var(β_k) + ∑_k≠ lψ_k ψ_l Cov(β_k, β_l).


We calculate variance of the heterogeneous coefficients following similar approach as in Eqs. <ref>, <ref>, and <ref>. The only difference is we replace D̃_p →ψ(X_p) * D̃_p in Eq. (<ref>) and D̃→ψ(X)*D̃ in Eq. (<ref>). 

For the ATT case,  Var(β) is a scalar whereas for the HTT case, Var(β) is a K× K matrix. The first and second terms in the summation in Eq. (<ref>) are the diagonal and the off-diagonal terms of the Var(β) matrix respectively. 



 §.§ DML implementation



We developed a causal ML library with JSON driven modeling configuration (see Fig. <ref>).  JSON ML Interpreter (JMI) translates JSON configuration to executable Python ML application. 




The main advantages of JMI approach are:

Flexibility: Business questions from various domains cannot always be addressed through a single unified configuration of a causal model. We address this in CI-DML where users can invoke different causal analysis frameworks (DML,  Causal Forests) and prediction algorithm type (regression, classification, clustering). 

Scalability: CI-DML utilizes distributed implementation of algorithms and file system via Apache Spark which helps causal modeling at  the big-data scale (100 millions customers,  multiple targets, and time horizons) 

Persistence: CI-DML inherits SparkML serialization and deserialization methods to persist and instantiate fitted models. 

Compatibility: In addition to Spark, interfaces to adapt ML libraries from scikit-learn, tensorflow, MXNet and other communities can be onboarded using the configurable abstraction support by JMI.

In the CI-DML system, we dockerize the JMI Causal ML library which is platform agnostic which has the flexibility to extend and utilize different compute engines like AWS EMR, Sagemaker or AWS Batch based on the use case and will abstract the computation information from the user.  Dockerization also helps version control and the build environment via standard software development tooling. 



§ RESULTS



Next we present the results for CI-DML. The target variables we focus on is customer spending, but our framework on can be leveraged to obtain the causal impact on any other target variable of interest (e.g., net profit, units bought etc).  For every CI run, we produce both the population-level ATT values (Eq. (<ref>)) and the customer-level HTT (Eq. (<ref>)) values. 


We compared the CI-PO and CI-DML results for 100+ customer actions.  As noted earlier, two major advantages of CI-DML are the availability of customer grain results (aka.  HTT) and confidence intervals.   
In Fig. <ref>, we present population-level (ATT) and customer-level values for selected representative customer actions [We anonymize actions to preserve business confidentiality].  
The reported confidence intervals are for both homoscedastic and heteroscedastic error variances.  To get a sense of the level of variance in customer-grain results, we report the percentage of customers where the  customer-level confidence interval crosses zero. We also report the out-of-sample fit metrics for outcome and propensity model in DML. 




Takeaways from analysis of 100+ actions are:


  * Population-level CI-PO and CI-DML values are aligned for 86% of actions. 

  * When the customer-level CI values are aggregated up, they are generally aligned with the population-level CI-DML values. 

  * The difference between the CI-PO and CI-DML values are larger either when the data is noisy and/or we have a small sample size.  For such cases, we also see large confidence intervals and the mean of HTT values is farther away from the CI-DML ATT values.

  * The homoscedastic confidence intervals are tighter than the heteroscedastic confidence interval as expected.  However, the homoscedastic confidence intervals likely under-predict the variance. We recommend business stakeholders to use the heteroscedastic results while making decisions. 

  * The customer-level confidence intervals are economically reasonable.  The percentage of customer-level confidence interval crossing zero increases for data with lower participation and is small for customer actions with a long history. 

  * The ML model metrics shown in Fig. <ref> are using ridge regression for the outcome model and logistic regression for the propensity model. We noticed that the model metrics as well as the CI values are relatively insensitive to the choice of ML model at the outcome/propensity stage.  Accordingly, we leverage ridge and logistic models due to their favorable compute time.




 §.§ Hyperparameter tuning


The hyperparameters (e.g., regularization strength) in the outcome and propensity model are chosen based on the out-of-sample performance.  
For the HTT estimates, the two main hyperparameters are the number of principal components and the number of clusters.  

We choose the number of principal components (PC) based on the percentage of variance explained.  We find that around 300 PC, about 80% of the variance is explained (Fig. <ref>). The amount of variance explained grows much slowly as we add more number of PC. To avoid sparsity issues in the downstream K-means calculation, we choose the number of PC components to be 300. 



Choosing the number of clusters is less straightforward. Standard tools such as elbow method and Silhouette score do not yield a clear answer for the optimal cluster choice.  In the current work, we choose 20 clusters as we think its best suited for our data. One of our ongoing work, is to make this choice in a more data-driven way.  We also find that the mean of HTT values is robust with respect to the choice for number of clusters. 



 §.§ Spread of customer-level CI values


So far, we have only looked at the mean of customer-level values in Fig <ref>.  In Fig. <ref>, we look at the customer-level CI scores for a representative customer action.  We see that most of the customers have CI value close to the average HTT value. We see that there are few customers with a low CI value which shifts the mean to the left.  One of the industry application of CI is marketing optimization. Having access to distribution of customer-level CI scores as in Fig. <ref> can help personalization and aid finer decision making. 





§ VALIDATION




 §.§ Placebo tests

Placebo tests help us understand the relative ability of competing causal estimates to account for selection bias. Selection bias occurs
when customers who take an action (e.g., stream video) have unobserved characteristics not included in the model that makes them
systematically more or less likely to take the action (e.g., high income, low age etc.). In placebo tests, we take the treatment group
customers and simulate as though they took the action a year before the actual date. This is achieved by shifting the event date by one year and recalculating the features based on the shifted event date. The CI estimated in this set up is the
“placebo error”.  Since, this is a fake event, a model with a lower placebo error than another suggests that it has a lower selection bias.  Running placebo tests on all events is computationally expensive,  so we selected a few events for placebo analysis.  The results are shown in Fig. <ref>. 



The key takeaways from Fig. <ref> are:


  * Selection bias is inherently event-dependent. When averaged across the selected customer actions,  we see a 2.24% improvement in placebo estimates when going from CI-PO to CI-DML. 

  * Selection bias is primarily impacted by the modeling features.  As CI-PO and CI-DML use the same features, we did not expect
big improvements in placebo tests. The consistent improvement across the events shows that double machine learning
methodology is better able to adjust for observables even when the same features are used. 
 



 §.§ Confidence interval comparison


One of the major wins in CI-DML is that we provide heteroskedasticity-consistent confidence intervals at both a customer and
aggregate level for every CI analysis in a scalable and lower-cost fashion. We compare the uncertainty estimates (specifically the
width of confidence intervals) from CI-DML with the bootstrap results in CI-PO for a few events in Fig. <ref>.



We find confidence interval width to be comparable among the two approaches. On average, the CI-DML width (scaled with CI-PO point
estimate) is 1.5% smaller when compared to bootstrap-based confidence interval.  A bootstrap-enabled CI-PO run takes about 2.5X more time than a CI-DML run.  Bootstrap also does not scale for events with large number of customers.  As CI-DML approach for confidence intervals is based on a closed form implementation,  we do not have any scalability issues. In addition, note that bootstrapping has theoretical limitations when used for matching estimators <cit.>.




§ CONCLUSION AND FUTURE WORK


In this work, we introduced a state-of-the-art methodology used for calculating CI values.  We noted that a DML based framework eliminates bias, allows us to extract heterogeneity in CI values, and provides a scalable way to construct heteroscedastic confidence intervals. We also made a case for using IPW and common support to refine the CI estimates. We demonstrated how leveraging PCA followed by K-means clustering allowed us to introduce customer-level heterogeneity. Using JSON based config allows flexibility to experiment with a wide variety algorithms and can take us from experimentation to production in minimal steps. 

We presented results for few anonymized customer actions across different domains. Both the population-level and customer-level results for the customer actions we have looked at so far are aligned with the CI-PO results and our expectations. 

Note that estimation of heterogeneous or context-aware treatment effects is an active area of research with wide applications ranging from marketing to health care.  Distribution of treatment effects across different subgroups, or as a function of specific individual-level characteristics provides researchers with additional insights about the treatment/ intervention analyzed.  Our work showcases, a scalable real-world application for extracting average as well heterogeneous causal estimates which we believe will be of interest to the broader scientific community. 



 §.§ Future work



Validating the causal estimates is challenging due to lack of ground truth. In the current work, we relied on the model fit metrics in the DML steps,  placebo tests,  and on bridging the CI-DML and CI-PO outputs.  
In the future, we plan to include metrics which focus on the validation of heterogeneous treatment effects.  Examples of these include metrics based on Generic Machine Learning <cit.> and empirically calibrated Monte Carlo resampling techniques <cit.>.
We also plan to experiment with doubly robust estimator in the DML stage. 

















	

§ APPENDIX

 



§ SAMPLE JSON CONFIG
 

We show a snippet of JSON config in Fig. <ref>.   
We can swap the specified models in the outcome and propensity step with any ML model of our choice.  

Likewise we can easily configure pre/post-processing steps and hyperparameters through JSON files. 

















999

plain

Chernozukov
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, James Robins.  Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal,  Volume 21, Issue 1, 1 February 2018,  Pages C1–C68,  https://doi.org/10.1111/ectj.12097doi.org/10.1111/ectj.12097

Jasjeet
Sekhon, Jasjeet.  The Neyman–Rubin Model of Causal Inference and Estimation via Matching Methods.  2007. The Oxford Handbook of Political Methodology.

Holland
Holland, Paul W.  Statistics and Causal Inference.  1986.  J. Amer. Statist. Assoc.  81 (396): 945–960.  https://doi.org/10.1080/01621459.1986.10478354doi:10.1080/01621459.1986.10478354

Neyman
Neyman, Jerzy.  Sur les applications de la theorie des probabilites aux experiences agricoles: Essai des principes. Master's Thesis (1923).  Excerpts reprinted in English,  Statistical Science, Vol. 5, pp. 463–472.

Rubin
Rubin, Donald.  Causal Inference Using Potential Outcomes. 2005.  J. Amer. Statist. Assoc.  81 (396): 945–960.  https://doi.org/10.1080/01621459.1986.10478354doi:10.1080/01621459.1986.10478354

Cherno_goldman_semenova_taddy
Chernozhukov, V., Goldman, M., Semenova, V.,  and Taddy, M. (2017). Orthogonal Machine Learning for Demand
Estimation: High Dimensional Causal Inference in Dynamic Panels. ArXiv:1712.09988 [Stat]. 

Huber
Huber, Peter J.   The behavior of maximum likelihood estimates under nonstandard conditions. (1967) Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability. 
Vol. 5. pp. 221–233

White
White,  Halbert.  A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.   Econometrica 48 (4): 817–838

HTR1
Horvitz, D. G.,  and Thompson, D. J.  . A Generalization of Sampling Without Re-placement From a Finite Universe (1952).  Journal of the American Statistical Association,  47(260), 663-685

HTR2
Nie, X., and Wager, S.  Quasi-Oracle Estimation of Heterogeneous Treatment Effects (2017).  ArXiv:1712.04912 [Econ, Math, Stat]

DR
Edward H. Kennedy,  Optimal doubly robust estimation of heterogeneous causal effects (2020).  ArXiv:2004.14497 [math.ST]

biometric
Estimating exposure effects by modelling the expectation of exposure conditional on confounders. Biometrics 48 479–495. MR1173493

Ruth
Ruth C, Brownell M, Isbister J, MacWilliam L, Gammon H, Singal D, Soodeen R, McGowan K, Kulbaba C, Boriskewich E. Long-Term Outcomes Of Manitoba's Insight Mentoring Program: A Comparative Statistical Analysis . Winnipeg, MB: Manitoba Centre for Health Policy, 2015

Austin
P. C. Austin and E. A. Stuart, Moving towards Best Practice When using Inverse Probability of Treatment Weighting (IPTW) Using the Propensity Score to Estimate Causal Treatment Effects in Observational Studies, vol. 34, pp. 3661-3679, 2015.

Hirano
Hirano, K., Imbens, G.W. Estimation of Causal Effects using Propensity Score Weighting: An Application to Data on Right Heart Catheterization. Health Services and Outcomes Research Methodology 2, 259–278 (2001). https://doi.org/10.1023/A:1020371312283doi.org/10.1023/A:1020371312283

Abadie_Imbens
A. Abadie and G. Imbens, On the failure of bootstrap for matching estimators, Econometrica, Vol. 76, No. 6 (2008), 1537-1157

GLM
Chernozhukov, Victor, Mert Demirer, Esther Duflo, and Ivan Fernandez-Val (2022).
Generic machine learning inference on heterogenous treatment effects in randomized experiments.
https://arxiv.org/abs/1712.04802aXiv:1712.04802v6 [stat.ML]

MC
Knaus, M. C., Lechner, M., & Strittmatter, A. (2021). Machine learning estimation of heterogeneous causal effects: Empirical monte carlo evidence. The Econometrics Journal, 24(1), 134-161

  


