Large scale continuous-time mean-variance portfolio allocation via reinforcement learning
Haoran Wang
q-fin.PM cs.LG math.OC




Quasi-circular inspirals and plunges from non-spinning effective-one-body Hamiltonians with gravitational self-force information

    Justin Vines
    December 30, 2023
==================================================================================================================================




We propose to solve large scale Markowitz mean-variance (MV) portfolio allocation problem using reinforcement learning (RL). By adopting the recently developed continuous-time exploratory control framework, we formulate the exploratory MV problem in high dimensions. We further show the optimality of a multivariate Gaussian feedback policy, with time-decaying variance, in trading off exploration and exploitation. Based on a provable policy improvement theorem, we devise a scalable and data-efficient RL algorithm and conduct large scale empirical tests using data from the S&P 500 stocks. We found that our method consistently achieves over 10% annualized returns and it outperforms econometric methods and the deep RL method by large margins, for both long and medium terms of investment with monthly and daily trading.




§ INTRODUCTION


Reinforcement learning (RL) has demonstrated to be sucessful in games (<cit.>, <cit.>, <cit.>) and robotics (<cit.>, <cit.>), which also raised significant attention on its applications to quantitative finance. Notable examples include large scale optimal order execution using clssical Q-learning method (<cit.>),  portfolio allocation using direct policy search (<cit.>, <cit.>), and option pricing and hedging using deep RL methods (<cit.>), among others.

However, most existing works only focus on RL problems with expected utility of discounted rewards. Such criteria are either unable to fully characterize the uncertainty of the decision making process in financial  markets or opaque to  typical investors. On the other hand,
mean–variance (MV)  is one of the most important criteria for portfolio choice. Initiated in the Nobel Prize winning work <cit.> for portfolio selection  in a single period, such a criterion yields an asset allocation strategy that minimizes the variance of the final payoff while targeting some prespecified mean return. The popularity of the MV criterion is not only due to its intuitive and transparent nature  in capturing the tradeoff between  risk and reward for practitioners, but also due to the theoretically interesting  issue of time-inconsistency (or Bellman's inconsistency) inherent with the underlying stochastic optimization and control problems.

In a recent paper <cit.>, the authors established an RL framework for studying the continuous-time MV portfolio selection, with continuous portfolio (action) and wealth (state) spaces. Their framework adopts a general entropy-regularized,  relaxed stochastic control formulation, known as the exploratory formulation, which was originally developed in <cit.> to capture explicitly
the tradeoff between exploration and exploitation in RL for continuous-time optimization problems. The paper <cit.> proved the optimality of Gaussian exploration (with time-decaying variance) for the MV problem in one dimension, and proposed a data-driven algorithm, the EMV algorithm, to learn the optimal Gaussian policy of the exploratory MV problem. Their simulation shows that the EMV algorithm outperforms both a classical econometric method and the deep deterministic policy gradient (DDPG) algorithm by large margins when solving the MV problem in the setting with only one risky asset.

It is the contribution of this work to generalize the continuous-time exploratory MV framework in <cit.> to large scale portfolio selection setting, with the number of risky assets being relatively large and the available training data being relatively limited. We establish the theoretical optimality of the high-dimensional Gaussian policy and design a scalable EMV algorithm to directly output portfolio allocation strategies. By switching to portfolio selection in high dimensions, we can in principle take more advantage of the diversification effect (<cit.>) to have better performance while, however, potentially encountering the challenges of low sample efficiency and instability faced by most deep RL methods (<cit.>, <cit.>). Nevertheless, although the EMV algorithm is an on-policy approach, it can achieve better data efficiency than the off-policy method DDPG, thanks to a provable policy improvement theorem and the explicit functional structures of the theoretical optimal Gaussian policy and value function. For instance, in a 10 years monthly trading experiment (see section 5.2) where the available data point for training is the same amount as the decision making times for testing, the EMV algorithm still can outperform various alternative methods considered in this paper. To further empirically test the performance and robustness of the EMV algorithm, we conduct experiments using both monthly and daily price data of the S&P 500 stocks, for long and medium term investment horizons. Annual returns over 10% have been consistently observed across most experiments. The EMV algorithm also demonstrated remarkable universal applicability, as it can be trained and tested respectively on different sets of data from stocks that are randomly selected and still achieves competitive and, actually, more robust performance (see Appendix <ref>).









§ NOTATIONS & BACKGROUND



 §.§ Classical continuous-time MV problem

We consider the classical MV problem in continuous time (without RL), where the investment universe consists of one riskless asset (savings account) and d risky assets (e.g., stocks). Let an investment planning horizon T>0 be fixed. Denote by {x^u_t,0≤ t≤ T} the discounted wealth (i.e. state) process of an agent who rebalances her portfolio (i.e. action) investing in  the risky and riskless assets with a strategy (policy) u={u_t, 0≤ t≤ T}. Here u_t=(u_t^1,…,u_t^d) is the discounted dollar value put in the d risky assets at time t. Under the geometric Brownian motion assumption for stocks prices and the standard self-financing condition, it follows (see Appendix <ref>) that the wealth process satisfies

    dx^u_t=σ u_t· (ρ  dt+ dW_t),    0≤ t≤ T,

with an initial endowment being x^u_0=x_0∈ℝ. Here, W_t=(W^1_t,…, W^d_t), 0≤ t≤ T, is a standard d-dimensional Brownian motion defined on a filtered probability space (Ω, ℱ,{ℱ_t}_0≤ t≤ T,ℙ). The vector[All vectors in this paper are taken as column vectors.] ρ is typically known as the market price of risk, and σ∈ℝ^d× d is the volatility matrix which is assumed to be non-degenerate.

The classical continuous-time MV model then aims to solve the following constrained optimization problem

    min_uVar[x^u_T],   subject to 𝔼[x^u_T]=z,

where {x^u_t,0≤ t≤ T} satisfies the dynamics (<ref>) under the investment strategy (portfolio) u, and z∈ℝ is an investment target set at t=0 as  the desired target payoff at the end of the investment horizon [0,T].

Due to the variance in its objective, (<ref>) is known to be time inconsistent. In this paper we focus ourselves to the so-called pre-committed strategies of the MV problem, which are optimal at t=0 only. To solve (<ref>), one first transforms it into an unconstrained problem by applying a Lagrange multiplier w:[Strictly speaking, 2w∈ℝ is the Lagrange multiplier.]

    min_u 𝔼[(x^u_T)^2]-z^2-2w(𝔼[x^u_T]-z)=min_u 𝔼[(x^u_T-w)^2]-(w-z)^2.

This problem can be solved analytically, whose solution u^*={u^*_t,0≤ t≤ T} depends on w.
Then the original constraint 𝔼[x^u^*_T]=z determines the value of
w.  We refer a detailed derivation to <cit.>.



 §.§ Exploratory continuous-time MV problem

The classical MV solution requires the estimation of
 the market parameters from historical time series of assets prices. However, as well known in practice, it is difficult to estimate the investment opportunity parameters, especially the mean return vector (aka the mean–blur problem; see, e.g.,  <cit.>) with a workable  accuracy. Moreover, the classical optimal MV strategies are often extremely sensitive to these parameters,  largely due to the procedure of inverting ill-conditioned covariance matrices  to obtain optimal allocation weights. In view of these two issues, the Markowitz solution can be greatly irrelevant to the underlying investment objective.


On the other hand, RL techniques do not require, and indeed often skip, any estimation of model parameters. Rather, RL algorithms, driven by historical data,  output optimal (or near-optimal) allocations directly. This is made possible by
direct interactions with the unknown investment environment, in a learning (exploring) while optimizing (exploiting) fashion.
Following <cit.>, we introduce the “exploratory" version of the state dynamics (<ref>). In this formulation, the control (portfolio) process u={u_t, 0≤ t≤ T} is randomized to represent exploration in RL, leading to
a measure-valued or distributional control process whose density function is given by π={π_t,0≤ t≤ T}. The dynamics (<ref>) is changed to

    dX^π_t    =   (∫_ℝ^dρ' σ uπ_t(u)du) dt+(∫_ℝ^d u'σ'σ uπ_t(u)du)^1/2  dB_t,

where {B_t,0≤ t≤ T} is a 1-dimensional standard Brownian motion on the filtered probability space (Ω, ℱ,{ℱ}_0≤ t≤ T,ℙ). Mathematically,
(<ref>) coincides with the relaxed control formulation in classical control theory, and it is adopted here to characterize the effect of exploration on the underlying continuous-time state dynamics change.
We refer the readers to <cit.> for a  detailed discussion on the motivation of (<ref>).

The randomized, distributional  control process π={π_t, 0≤ t≤ T} is to model
exploration, whose overall level is in turn captured by
its accumulative differential entropy

    ℋ(π):=-∫_0^T∫_ℝ^dπ_t(u)lnπ_t(u)dudt.

Further, we introduce a temperature parameter (or exploration weight) λ>0 reflecting  the tradeoff between exploitation and exploration.
The entropy-regularized, exploratory  MV problem is then to solve, for any fixed w∈ℝ:

    min_π∈𝒜(0,x_0)𝔼[(X_T^π-w)^2+λ∫_0^T∫_ℝ^dπ_t(u)lnπ_t(u)dudt]-(w-z)^2,

where 𝒜(0,x_0) is the set of admissible distributional controls on [0,T] whose precise definition is relegated to Appendix <ref>. Once this problem is solved with a minimizer
π^*={π^*_t, 0≤ t≤ T}, the Lagrange multiplier w can  be determined by the additional constraint 𝔼[X^π^*_T]=z.

The optimization objective (<ref>) explicitly encourages exploration, in contrast to the classical problem (<ref>) which concerns exploitation only.




§ OPTIMALITY OF GAUSSIAN EXPLORATION



To solve the exploratory MV problem (<ref>), we apply the classical Bellman's principle of optimality for the optimal value function V (see Appendix <ref> for the precise definition of V):

    V(t,x;w)=inf_π∈𝒜(t,x)𝔼[V(s,X^π_s;w)+λ∫_t^s∫_ℝ^dπ_l(u)lnπ_l(u)dudl | X^π_t=x],

for x∈ℝ and 0≤ t< s≤ T. Following standard arguments, we deduce that V satisfies the Hamilton-Jacobi-Bellman (HJB) equation

    v_t(t,x;w)+min_π∈𝒫(ℝ^d)∫_ℝ^d(1/2u'σ'σ u v_xx(t,x;w)+ρ'σ u v_x(t,x;w)+λlnπ(u))π(u)du=0,

with the terminal condition v(T,x;w)=(x-w)^2-(w-z)^2. Here, 𝒫
( ℝ^d) denotes the set of density functions of probability measures on ℝ^d that are absolutely
continuous with respect to the Lebesgue measure and v denotes the generic unknown solution to the HJB equation.

Applying the usual verification technique and using the fact that π∈𝒫(ℝ^d) if and only if
∫_ℝ^dπ (u)du=1 and π (u)≥ 0, a.e., on ℝ^d,
we can solve the (constrained) optimization problem in the HJB equation (<ref>) to obtain a feedback  (distributional) control whose density function is given by

    π ^∗(u;t,x,w)   =   exp(-1/λ(1/2u'σ'σ u v_xx(t,x;w)+ρ' σ u v_x(t,x;w)))/∫_ℝ^dexp(-1/λ(1/2u'σ'σ u v_xx(t,x;w)+ρ' σ u v_x(t,x;w)))du
       =   𝒩(  u  | -σ^-1ρv_x(t,x;w)/v_xx(t,x;w) ,  (σ'σ)^-1λ/v_xx(t,x;w)),

where 𝒩(u|β,Σ) denotes the Gaussian density function with mean vector β and covariance matrix Σ. It is assumed in (<ref>) that v_xx(t,x;w)>0, which will be verified in what follows.

Substituting the candidate optimal Gaussian feedback control policy (<ref>) back into the HJB equation (<ref>), the latter is transformed to

    v_t(t,x;w)-ρ'ρ/2v^2_x(t,x;w)/v_xx(t,x,w)+λ/2(d-dln( 2π e λ/v_xx(t,x;w))+ln(|σ'σ|))=0,

with v(T,x;w)=(x-w)^2-(w-z)^2, where |·| denotes the matrix determinant. A direct computation yields that this equation has a classical solution

    v(t,x;w)=(x-w)^2e^-ρ'ρ(T-t)+λ d/4ρ'ρ(T^2-t^2)-λ d/2(ρ'ρ T-1/dln|σ'σ|/πλ)(T-t)-(w-z)^2,

which clearly satisfies  v_xx(t,x;w)>0, for any (t,x)∈ [0,T]×ℝ. It then follows that the candidate optimal feedback Gaussian policy (<ref>) reduces to

    π ^∗(u;t,x,w)=𝒩(  u |  -σ^-1ρ(x-w) ,  (σ'σ)^-1λ/2e^ρ'ρ (T-t)),  
    (t,x)∈ [0,T]×ℝ.



Finally, the  optimal wealth process (<ref>) under π ^∗ becomes

    dX^*_t=-ρ'ρ (X^*_t-w)  dt+(ρ'ρ(X^*_t-w)^2+λ/2e^ρ'ρ(T-t))^1/2  dB_t, 
    X^*_0=x_0.

It has a unique strong solution for 0≤ t≤ T, as can be easily verified. We now summarize the above results in the following theorem.

The optimal value function of the entropy-regularized exploratory MV problem (<ref>) is given by

    V(t,x;w)=(x-w)^2e^-ρ'ρ(T-t)+λ d/4ρ'ρ(T^2-t^2)-λ d/2(ρ'ρ T-1/dln|σ'σ|/πλ)(T-t)-(w-z)^2,

for (t,x)∈ [0,T]×ℝ.
Moreover,
the optimal feedback control is Gaussian, with its density function given by

    π^∗(u;t,x,w)=𝒩(  u |  -σ^-1ρ(x-w) ,  (σ'σ)^-1λ/2e^ρ'ρ (T-t)).

The associated optimal wealth process under π^∗
is the unique solution of the stochastic differential equation

    dX^*_t=-ρ'ρ (X^*_t-w)  dt+(ρ'ρ(X^*_t-w)^2+λ/2e^ρ'ρ(T-t))^1/2  dB_t, X^*_0=x_0.

Finally, the Lagrange multiplier w is given by w=ze^ρ'ρ T-x_0/e^ρ'ρ T-1.


See Appendix <ref>.




Theorem <ref> indicates that the level of exploration, measured by the variance of Gaussian policy λ/2σ^2e^ρ^2(T-t), decays in time. The agent initially engages in exploration at the maximum level, and reduces it gradually (although never to zero) as time approaches the end of the investment horizon. Naturally, exploitation dominates exploration as time approaches maturity.  Theorem <ref> presents such a  decaying exploration scheme endogenously which, to our best knowledge, has not been derived in the RL literature.

Moreover, the mean of the Gaussian distribution (<ref>) 
is independent of the exploration weight λ, while its variance is independent of the state x. This highlights  a perfect separation between exploitation and exploration, as the former is captured by the mean and  the latter  by the variance of the optimal Gaussian exploration. This property is also consistent with the linear–quadratic case in the  infinite horizon  studied in <cit.>.
































































It is reasonable to expect that the exploratory problem converges to its
classical counterpart as the exploration weight λ decreases to 0. Let u^∗ be the optimal feedback control for the classical MV problem, and denote by V^cl the optimal value function. Let δ_a(·) be the Dirac measure centered at a∈ℝ^d. Then the following result holds.


For each (t,x,w)∈ [0,T]×ℝ×ℝ,

    lim_λ→ 0π^∗(·;t,x;w)=δ_u^∗(t,x;w)(·)

Moreover, 

    lim_λ→ 0|V(t,x;w)-V^cl(t,x;w)|=0.





See Appendix <ref>.
















































§ RL ALGORITHM DESIGN




 §.§ A policy improvement theorem


We present a policy improvement theorem that is a crucial prerequisite  for our interpretable RL algorithm, the EMV algorithm, which solves the exploratory MV problem in high dimensions.

 Let w∈ℝ be fixed and  π=π(·;·,·,w) be an arbitrarily given admissible  feedback control policy. Suppose that the corresponding value function V^π(·,·;w)∈ C^1,2([0,T)×ℝ)∩ C^0([0,T]×ℝ) and satisfies V^π_xx(t,x;w)>0, for any (t,x)∈ [0,T)×ℝ. Suppose further that the feedback policy π̃ defined by

    π̃(u;t,x,w)=𝒩(   u | -σ^-1ρV^π_x(t,x;w)/V^π_xx(t,x;w) ,   (σ'σ)^-1λ/V^π_xx(t,x;w))

is admissible. Then, 

    V^π̃(t,x;w)≤ V^π(t,x;w),   (t,x)∈ [0,T]×ℝ.



See Appendix <ref>.


























The above theorem suggests that there are always policies in the Gaussian family
that improves
the value function of any given, not necessarily Gaussian, policy. Moreover, the Gaussian family is closed under the policy improvement scheme. Hence, without loss of generality, we can simply focus on
the Gaussian policies when choosing an initial solution. The next result shows convergence of both the value functions and the policies from a specifically parameterized Gaussian policy.


Let π_0(u;t,x,w)=𝒩(u| α(x-w), Σ e^β(T-t)), with α∈ℝ^d, β∈ℝ and Σ being a d× d positive definite matrix. Denote by {π_n(u;t,x,w), (t,x)∈ [0,T]×ℝ,n≥ 1} the sequence of feedback policies updated by the policy improvement scheme (<ref>), and {V^π_n(t,x;w), (t,x)∈ [0,T]×ℝ, n≥ 1} the sequence of the corresponding value functions. Then,

    lim_n→∞π_n (·; t,x,w)= π^*(·; t,x,w)

and

    lim_n→∞ V^π_n(t,x;w)=V(t,x;w),

for any (t,x,w)∈ [0,T]×ℝ×ℝ, where π^* and V are the optimal Gaussian policy (<ref>) and the optimal value function (<ref>), respectively.


See Appendix <ref>.


















 §.§ The EMV algorithm

We provide the EMV algorithm to directly learn the optimal solution of the continuous-time exploratory MV problem in high dimensions within competitive training time. Theorem <ref> provides guidance for policy improvement. For the policy evaluation step, we follow <cit.> to minimize the continuous-time Bellman's error








    δ_t:=V̇^π_t+λ∫_ℝ^dπ_t(u)lnπ_t(u)du,

where V̇^π_t=V^π(t+Δ t,X_t+Δ t)-V^π(t,X_t)/Δ t is the total derivative and Δ t is the discretization step for the learning algorithm. This leads to the cost function to be minimized

    C(θ,ϕ)=1/2∑_(t_i,x_i)∈𝒟(V̇^θ(t_i,x_i)+λ∫_ℝ^dπ^ϕ_t_i(u)lnπ^ϕ_t_i(u)du)^2 Δ t,

using samples collected in the set 𝒟 under the current Gaussian policy π^ϕ. Here, both the value function V^θ and the Gaussian policy π^ϕ can be  parametrized more explicitly, in view of (<ref>), Theorem <ref> and <ref>. The cost function (<ref>) can then be minimized by stochastic gradient decent. Finally, the EMV algorithm updates the Lagrange multiplier w every N iterations based on stochastic approximation and the constraint 𝔼[X^π_T]=z, namely, w← w-α(1/N∑_j x^j_T-z), where x_T^j's are the most recent N terminal wealth values. We refer the readers to <cit.> for a more detailed description of the EMV algorithm in the one risky asset scenario.




§ EMPIRICAL RESULTS



 §.§ Data and methods

We test the EMV algorithm on price data of the S&P 500 stocks for both monthly and daily trading. For the former, we train the EMV algorithm on the 10 years monthly data[All data is from Wharton Research Data Services (WRDS). https://wrds-web.wharton.upenn.edu/wrds/] from 08-31-1990 to 08-31-2000, and then test the learned allocation strategy from 09-29-2000 to 09-30-2010. The initial wealth is normalized as 1 and the 10 years target is z=8, corresponding to a 23% annualized target return. In the daily rebalancing scenario, the EMV algorithm is trained on the 1 year daily data from 01-09-2017 to 01-08-2018 and tested on the subsequent year, with a 40% return set as the target for the 1 year investment horizon.

For comparison studies, we also train and test other alternative methods for solving the portfolio allocation problem on the same data. Specifically, we consider the classical econometric methods including Black-Litterman (BL, <cit.>), Fama-French (FF, <cit.>) and the Markowitz portfolio (Markowitz, <cit.>). A recently developed distributionally robust MV strategy, the Robust Wasserstein Profile Inference (RWPI, <cit.>), is also included. To compare EMV with deep RL method,  we adjust DDPG similarly as in <cit.>, so that it can solve the classical MV problem (<ref>). All experiments were performed on a MacBook Air laptop, with DDPG trained using Tensorflow.


 §.§ Test I: monthly rebalancing

We first consider d=20. By randomly selecting 20 stocks for each set/seed, we compose 100 different seeds. The split of training and testing data for  EMV and DDPG is fixed as described above, but we consider two types of training. The first training method is batch (off-line) RL, where both algorithms are trained for multiple episodes using one seed, following by testing on the subsequent 10 years data of that seed. The performance is then averaged over the 100 seeds. Another method is to use all the 100 seeds and select one seed randomly for each episode during training. Then both algorithms are tested on randomly selected 100 seeds over the test period and the performance is averaged as well. The second method can be seen to artificially generate randomness for training and testing, and an algorithm that performs well using this method has universality and potential to generate to data of stocks in different sectors.

For competitive performance, we adopt a rolling-horizon based training and testing for all the other methods. Specifically, each time after the 1 month ahead investment decision is made on the test set, we add the most recent price data point from the test set into the training set, and discard the most obsolete data point from the training set. 







Figure <ref> shows the performance of various investment strategies, including variants of the EMV algorithm with different gross leverage constraints on portfolios.[Leverage is a fundamental investment tool for most hedge funds; according to <cit.>, the average gross leverage across the 208 hedge funds studied therein is 213%.] Under reasonable leverage constraint, the EMV algorithm still outperforms most other methods (which have no constraints, except DDPG) by a large margin, although it was trained only using the previous 10 years monthly data. 

The universal training and testing method was used for EMV and DDPG in Figure <ref>. Results for the batch method can be found in Appendix <ref>. A remarkable fact in both cases is that the original EMV algorithm, devised to solve the exploratory MV problem (<ref>) without constraint, achieves the target z=8 with minimal variance for most of the test period. We also report various investment outcomes in Table <ref> when scaling up d, the number of stocks in the portfolio.




 §.§ Test II: daily rebalancing
 
For daily trading with d=50, we report the performance of the EMV algorithm under different gross leverage constraints in Figure <ref>. The DDPG algorithm was not competitive in the daily trading setting (see Table <ref>) and, hence, omitted. For different d, Table <ref> summarizes the investment outcomes and the training time (per experiment). These results were obtained using the universal method for both training and testing.






§ RELATED WORK

The difficulty of seeking the global optimum for Markov Decision Process (MDP) problems under the MV criterion has been previously noted in <cit.>.
In fact, the variance of reward-to-go is nonlinear in expectation and, as a result of Bellman's inconsistency, most of the well-known RL algorithms cannot be applied directly.

Existing works on variance estimation and control generally divide into value based methods and policy based methods.  <cit.> obtained the Bellman's equation for the variance of reward-to-go under a fixed, given policy. <cit.> further derived the TD(0) learning rule to estimate the variance, followed by <cit.> which applied this value based method  to an MV  portfolio selection problem. It is worth noting that due to the definition of the value function (i.e., the variance penalized expected reward-to-go) in <cit.>,  Bellman's optimality principle does not hold. As a result, it is not guaranteed that a greedy policy based on the latest updated value function will eventually lead to the true global optimal policy. The second approach, the policy based RL,  was proposed  in <cit.>. They also extended the work to linear function approximators and devised actor-critic algorithms for  MV optimization problems for which convergence to the local optimum is guaranteed with probability one (<cit.>). Related works following this line of research include <cit.>, <cit.>, among others. Despite the various methods mentioned above, it remains  an open and interesting question in RL to search for the global optimum under the MV criterion.



In this paper, rather than relying on the typical framework of discrete-time MDP and discretizing time and state/action spaces accordingly, we designed the EMV algorithm to learn the global optimal solution of the continuous-time exploratory MV problem (<ref>) directly. As pointed out in <cit.>, it is typically challenging to find the right granularity to discretize the state and action spaces, and
naive discretization  may lead to poor performance. On the other hand, grid-based discretization methods for solving the HJB equation cannot easily extend to high dimensions in practice due to the curse of dimensionality, although theoretical convergence results have been established (see <cit.>, <cit.>). Our EMV algorithm, however, is computationally feasible and implementable in high dimensions, as demonstrated by the experiements, due to the explicit representations of the value functions and the portfolio strategies, thereby devoid of  the curse of dimensionality. Note that our algorithm does not use (deep) neural networks, which have been applied extensively in literature for (high-dimensional) continuous RL problems (e.g., <cit.>, <cit.>) but known for unstable performance, sample inefficiency as well as extensive hyperparameter tuning (<cit.>, <cit.>, <cit.>), in addition to their low interpretability.[Interpretability is one of the most important and pressing issues in the general artificial intelligence applications in financial industry due to, among others,
the regulatory requirement.]





§ CONCLUSIONS

We studied continuous-time mean-variance (MV) portfolio allocation problem in high dimensions using RL methods. Under the exploratory control framework for general continuous-time optimization problems, we formulated the exploratory MV problem in high dimensions and proved the optimality of Gaussian policy in achieving the best tradeoff between exploration and exploitation. Our EMV algorithm, designed by combining quantitative finance analysis and RL techniques to solve the exploratory MV problem, is interpretable, scalable and data efficient, thanks to a provable policy improvement theorem and efficient functional approximations based on the theoretical optimal solutions. It consistently outperforms both classical model-based econometric methods and model-free deep RL method, across different training and testing scenarios. Interesting future research includes testing the EMV algorithm for shorter trading horizons with tick data (e.g. high frequency trading), or for trading other financial instruments such as mean-variance option hedging.





  §.§.§ Acknowledgments

The author would like to thank Prof. Xun Yu Zhou for generous support and continuing encouragement on this work. The author also wants to thank Lin (Charles) Chen for providing the results on BL, FF, Markowitz and RWPI methods.








§ CONTROLLED WEALTH DYNAMICS

Let W_t=(W^1_t,…, W^d_t), 0≤ t≤ T be a standard d-dimensional Brownian motion defined on a filtered probability space (Ω, ℱ,{ℱ_t}_0≤ t≤ T,ℙ) that satisfies the usual conditions. The price process of the i-th risky asset is a geometric Brownian motion governed by

    dS^i_t=S^i_t(μ^i  dt+σ^i· dW_t),    0≤ t≤ T,     i=1,…,d,

with S^i_0=s^i_0>0 being the initial price at t=0, and μ^i∈ℝ, σ^i=(σ^1i,…, σ^di)∈ℝ^d being the mean return and volatility coefficients of the i-th risky asset, respectively. We denote for brevity the mean return vector by μ∈ℝ^d, and the volatility matrix by σ∈ℝ^d× d, whose i-th column represents the volatility σ^i of the i-th risky asset. The riskless asset has a constant interest rate r>0. We assume that σ is non-degenerate and hence there exists a d-dimensional vector ρ that satisfies σ'ρ=μ-r1, where 1 is the d-dimensional vector with all components being 1. The vector ρ is known as the market price of risk. It is worth noting that the above assumptions are only made for the convenience of deriving theoretical results in the paper; in practice, all the model parameters are unknown and time-varying, and it is the goal of RL algorithms to directly output trading strategies without relying on estimation of any underlying parameters.

Denote by u^0_t and u_t=(u_t^1,…,u_t^d)  the discounted dollar value put in the savings account and the d risky assets, respectively, at time t. It then follows that the discounted wealth process is x^u_t=∑_i=0^d u^i_t, 0≤ t≤ T. The self-financing condition further implies that, using (<ref>), we have

    dx^u_t =  ru^0_t dt+∑_i=1^du^i_t/S^i_tdS^i_t-rx^u_t dt=-r(x^u_t-u^0_t)dt+∑_i=1^d u^i_t(μ^i dt+σ^i· dW_t)


    =∑_i=1^d u_t^i((μ^i-r)dt+σ^i· dW_t)=σ u_t· (ρ dt+dW_t).



§ VALUE FUNCTIONS AND ADMISSIBLE CONTROL DISTRIBUTIONS

In order to rigorously solve (<ref>) by dynamic programming, we need to define the value functions. For each (s,y)∈[0,T)×ℝ, consider the
state equation (<ref>) on [s,T] with X^π_s=y.
Define the set of admissible controls, 𝒜(s,y), as follows. Let ℬ(ℝ^d) be the Borel algebra on ℝ^d. A (distributional) control (or portfolio/strategy) process π={π_t,s≤ t≤ T} belongs to 𝒜(s,y), if



(i)		 for each s≤ t≤ T, π _t∈𝒫(ℝ^d) a.s.;

(ii)		for each A∈ℬ(ℝ^d), {∫_Aπ _t(u)du,s≤ t≤ T}
is ℱ_t-progressively measurable;

(iii)	𝔼[
∫_s^T∫_ℝ^d|σ u|^2 π_t(u)du dt] <∞;

(iv)	𝔼[|(X_T^π-w)^2+λ∫_s^T∫_ℝ^dπ_t(u)lnπ_t(u)dudt|  | X_s^π=y]<∞.



Clearly, it follows from condition (iii) that
the stochastic differential equation (SDE) (<ref>) has a unique strong solution for s≤ t≤ T that satisfies X^π_s=y.


Controls in 𝒜(s,y) are measure-valued (or, precisely, density-function-valued) stochastic processes, which are also called open-loop controls in the control terminology.
As in the classical control theory, it is important to distinguish between open-loop controls and feedback (or closed-loop) controls (or policies as in the RL literature, or laws as in the control literature). Specifically, a deterministic mapping π(·;·,·) is called an (admissible)  feedback control  if i) π(·;t,x) is a density function for each (t,x)∈[0,T]×ℝ; ii) for each (s,y)∈[0,T)×ℝ, the following SDE (which is the system dynamics after the feedback policy π(·;·,·) is applied)

    dX^π_t=(∫_ℝ^dρ' σ uπ(u;t,X^π_t))du)  dt+(∫_ℝ^d u'σ'σ uπ(u;t,X^π_t))du)^1/2  dB_t,   X^π_s=y,

has a unique strong solution {X^π_t,t∈[s,T]},  and  the open-loop control
π={π
_t, t∈[s,T]}∈𝒜(s,y) where π_t:=π(·;t,X_t^π). In this case, the open-loop control π is said to be
generated from the feedback policy π(·;·,·) with respect to the initial time and state,  (s,y). It is useful to note that an open-loop control and its admissibility depend on the initial (s,y), whereas a
feedback policy can generate open-loop controls for any (s,y)∈[0,T)×ℝ, and hence is in itself independent of (s,y). Note that throughout this paper, we have used boldfaced π to denote feedback controls, and the normal style π to denote open-loop controls.

Now, for a fixed w∈ℝ, define

    V(s,y;w):=inf_π∈𝒜(s,y)𝔼[(X_T^π-w)^2+λ∫_0^T∫_ℝ^dπ_t(u)lnπ_t(u)dudt | X_s^π=y]-(w-z)^2,

for (s,y)∈[0,T)×ℝ.
The function V(·,·;w) is called the optimal  value function of the problem.

Moreover, we define  the value function  under any given feedback control  π:


    V^π(s,y ;w)=𝔼[(X_T^π-w)^2+λ∫_s^T∫_ℝ^dπ_t(u)lnπ_t(u)dudt | X_s^π=y]-(w-z)^2,

for (s,y)∈[0,T)×ℝ, where π={π
_t, t∈[s,T]} is  the open-loop control generated from π  with respect to  (s,y) and {X^π_t,t∈[s,T]} is the corresponding wealth process.

Note that in the control literature, V given by (<ref>) is called the value function.
However, in the RL literature the term “value function" is also used for the objective value under a particular control (i.e. V^π in (<ref>)). So to avoid ambiguity we have called V the optimal value function in this paper.



§ PROOFS



 §.§ Proof of Theorem <ref>


The main proof of Theroem <ref> would be the verification arguments that aim to show the optimal value function of problem (<ref>) is given by (<ref>) and that the candidate optimal policy (<ref>) is indeed admissible, based on the definitions in Appendix <ref>. Since the current exploratory MV problem is a special case of the exploratory linear-quadratic problem extensively studied in <cit.>, a detailed proof would follow the same lines of that of Theorem 4 therein, and is left for interested readers.




We now determine the Lagrange multiplier w through the constraint 𝔼[X^*_T]=z. It follows  from (<ref>),



along with the standard estimate that 𝔼[max_t∈ [0,T](X^*_t)^2]<∞ and Fubini's Theorem, that

    𝔼[X^*_t]=x_0+𝔼[∫_0^t -ρ'ρ(X^*_s-w)  ds]=x_0+∫_0^t -ρ'ρ(𝔼[X^*_s]-w)  ds.

Hence,  𝔼[X^*_t]=(x_0-w)e^-ρ'ρ t+w. The constraint 𝔼[X^*_T]=z now becomes (x_0-w)e^-ρ'ρ T+w=z, which gives  w=ze^ρ'ρ T-x_0/e^ρ'ρ T-1.




 §.§ Proof of Theorem <ref>

To prove the solution of the exploratory MV problem converges to that of the classical MV problem, as λ→ 0, we first recall the solution of the classical MV problem. 

In order to apply dynamic programming for (<ref>), we again consider the set of admissible controls, 𝒜^cl(s,y),
for (s,y)∈ [0,T)×ℝ,


𝒜^cl(s,y):={u={u_t, t∈ [s,T]}: u is ℱ_t-progressively measurable and 𝔼[∫_s^T|σ u_t|^2  dt]<∞}.


The (optimal) value function is defined by

    V^cl(s,y;w):=inf_u∈𝒜^cl(s,y)𝔼 [(x^u_T-w)^2  |   x^u_s=y ]-(w-z)^2,

for (s,y)∈ [0,T)×ℝ, where w∈ℝ is fixed. Once this problem is solved, w can be determined by the constraint 𝔼[x^*_T]=z, with {x^*_t, t∈ [0,T]} being the optimal wealth process under the optimal portfolio u^*.

The HJB equation is

    ω_t(t,x;w)+min_u∈ℝ^d(1/2u'σ'σ u ω_xx(t,x;w)+ρ' σ u  ω_x(t,x;w))=0,  (t,x)∈ [0,T)×ℝ,

with the terminal condition  ω(T,x;w)=(x-w)^2-(w-z)^2.

Standard verification arguments  deduce the optimal value function to be

    V^cl(t,x;w)=(x-w)^2e^-ρ'ρ (T-t)-(w-z)^2,

the optimal feedback control policy to be

    u^∗(t,x;w)=-σ^-1ρ(x-w),

and the corresponding optimal wealth process to be the unique strong solution to the SDE

    dx^*_t=-ρ'ρ(x^*_t-w)  dt-ρ (x^*_t-w)· dW_t,   x^*_0=x_0.

Comparing the optimal wealth dynamics, (<ref>) and (<ref>), of the exploratory and classical problems,
we note that they have the same drift coefficient (but different diffusion coefficients). As a result, the two problems have the same
mean of optimal terminal wealth and hence the same value of
the Lagrange multiplier w=ze^ρ'ρ T-x_0/e^ρ'ρ T-1 determined by the constraint 𝔼[x^*_T]=z.




The weak convergence of the feedback controls follows from the explicit forms of π^∗  in (<ref>) and u^∗ in (<ref>). The pointwise convergence of the value functions follows easily from the forms of V in (<ref>) and V^cl in (<ref>), together with the fact that

    lim_λ→ 0λ/2ln|σ'σ|/πλ =0.





 §.§ Proof of Theorem <ref>


Fix (t,x)∈ [0,T]×ℝ. Since, by assumption, the feedback policy π̃ is admissible,
the open-loop control strategy, π̃={π̃_v, v∈ [t,T]}, generated from π̃ with respect to the initial condition X^π̃_t=x is  admissible. Let {X^π̃_s,s∈[t,T]} be the corresponding wealth process under π̃. Applying Itô's formula, we have

    V^π(s,X̃_s)=V^π(t,x)+∫_t^sV^π_t(v,X^π̃_v)dv+∫_t^s∫_ℝ^d(1/2u'σ'σ u V^π_xx(v,X^π̃_v)


    +ρ' σ u V^π_x(v,X^π̃_v))π̃_v(u)   dudv+∫_t^s  (∫_ℝ^du'σ'σ uπ̃_v(u)du)^1/2V^π(v,X^π̃_v)  dB_v, s∈ [t,T].

Define the stopping times τ_n:=inf{s≥ t: ∫_t^s ∫_ℝ^du'σ'σ u π̃_v(u)du(V^π(v,X^π̃_v))^2dv≥ n}, for n≥ 1. Then, from (<ref>), we obtain

    V^π(t,x)=𝔼[V^π(s∧τ_n,X^π̃_s∧τ_n)-∫_t^s∧τ_nV^π_t(v,X^π̃_v)dv


    -∫_t^s∧τ_n∫_ℝ^d(1/2u'σ'σ u  V^π_xx(v,X^π̃_v)+ρ' σ u V^π_x(v,X^π̃_v))π̃_v (u)  dudv | X^π̃_t=x].

On the other hand, by standard arguments and the assumption that V^π is smooth, we have

    V_t^π(t,x)+∫_ℝ^d(1/2u'σ'σ u  V^π_xx(t,x)+ρ'σ u V^π_x(t,x)+λlnπ(u;t,x))π(u;t,x)du=0,

for any (t,x)∈ [0,T)×ℝ. It follows that

    V_t^π(t,x)+min_π̂∈𝒫(ℝ^d)∫_ℝ^d(1/2u'σ'σ u  V^π_xx(t,x)+ρ'σ u V^π_x(t,x)+λlnπ̂(u))π̂(u)du≤ 0.

Notice that the minimizer of the Hamiltonian in (<ref>) is given by the feedback policy π̃ in (<ref>). It then follows that equation (<ref>) implies

    V^π(t,x)≥𝔼[V^π(s∧τ_n,X^π̃_s∧τ_n)+λ∫_t^s∧τ_n∫_ℝ^dπ̃_v(u)lnπ̃_v(u)  dudv | X^π̃_t=x],

for (t,x)∈ [0,T]×ℝ and s∈ [t,T]. Now taking s=T, and using that V^π(T,x)=V^π̃(T,x)=(x-w)^2-(w-z)^2 together with the assumption that π̃ is admissible, we obtain, by sending  n→∞ and applying the dominated convergence theorem, that

    V^π(t,x)≥𝔼[V^π̃(T,X^π̃_T)+λ∫_t^T∫_ℝ^dπ̃_v(u)lnπ̃_v(u)  dudv | X^π̃_t=x]=V^π̃(t,x),

for any (t,x)∈ [0,T]×ℝ.




 §.§ Proof of Theorem <ref>


It can be easily verified that the feedback policy π_0(u;t,x,w)=𝒩(u| α(x-w), Σ e^β(T-t)) generates an open-loop policy π_0 that is admissible with respect to the initial (t,x). Moreover, it follows from the Feynman-Kac formula that the corresponding value function V^π_0 satisfies the PDE

    V^π_0_t(t,x;w)+∫_ℝ^d(1/2u'σ'σ u  V^π_0_xx(t,x;w)+ρ' σ  u V^π_0_x(t,x;w)


    +λlnπ_0(u;t,x,w))π_0(u;t,x,w)du=0,

with terminal condition V^π_0(T,x;w)=(x-w)^2-(w-z)^2.
Simplifying this equation we obtain

    V_t^π_0(t,x;w)+1/2V_xx^π_0(t,x;w)Tr(σαα'σ (x-w)^2+σΣσ' e^β (T-t))


    +V_x^π_0(t,x;w)ρ'σα (x-w)-λ/2(dln(2π e)+ln|Σ|+dβ (T-t))=0,

where Tr(·) denotes the trace of a square matrix. A classical solution to equation (<ref>) is given by

    V^π_0=(x-w)^2 e^(2ρ'σα+Tr(σαα'σ'))(T-t)+Tr(σΣσ')e^(β+2ρ'σα+Tr(σαα'σ'))(T-t)/β+2ρ'σα+Tr(σαα'σ')


    -λ d/4β t^2+λ d/2(ln(2π e |Σ|^1/d)+β T)t-(w-z)^2-Tr(σΣσ')/β+2ρ'σα+Tr(σαα'σ'),

if β+2ρ'σα+Tr(σαα'σ')≠ 0 and, by

    V^π_0=(x-w)^2 e^(2ρ'σα+Tr(σαα'σ'))(T-t)-λ d/4β t^2+(λ d/2(ln(2π e |Σ|^1/d)+β T)-Tr(σΣσ'))t


    -(w-z)^2,

if β+2ρ'σα+Tr(σαα'σ')= 0.
In either case, it is easy to check  that V^π_0 satisfies the conditions in Theorem <ref> and, hence, the theorem applies. The improved policy is given by (<ref>), which, in the current case, becomes

    π_1(u;t,x,w)=𝒩(   u | -σ^-1ρ (x-w) ,  λ (σ'σ)^-1/2e^(2ρ'σα+Tr(σαα'σ'))(T-t)).

Again, we can calculate the corresponding value function as V^π_1(t,x;w)=(x-w)^2e^-ρ'ρ (T-t)+F_1(t), where F_1 is a function of t only. Theorem <ref> is applicable again, which yields the improved policy π_2 as exactly the optimal Gaussian policy π^* given in (<ref>), together with the optimal value function V in (<ref>). The desired convergence therefore follows, as for n≥ 2, both the policy and the value function will no longer strictly improve  under the policy improvement scheme (<ref>).




§ EMPIRICAL RESULTS: THE BATCH METHOD

In Section 5.2, we provided the experiment results for monthly trading under universal training and testing. Another way to train and test the EMV and DDPG algorithms is based on batch (off-line) RL, as described in the main text. The batch method applies to the training and testing data from the same set/seed of d=20 stocks for each experiment, and the investment performance of the EMV algorithm over the 100 seeds are reported in Figure <ref>. Due to the extensive training time (see Table <ref>), we only train and test DDPG under the batch method for 8 seeds. 

The batch method demonstrates qualitatively similar behavior as the universal training and testing method (see Figure <ref>), when compared to the econometric methods and the deep RL method. A more detailed comparison between the two methods is shown in Figure <ref>. It is interesting to notice that, while the two methods perform equally well for most of the testing period over 2000-2010, the universal method is less affected by the 2008 financial crisis with less variability and higher returns. The batch method, without taking into account the data of other stocks for each portfolio/seed during training and testing, is more susceptible to stock market plunge. Nonetheless, both methods are data efficient, especially in view that, for example, the training set for the batch method contains the same number of data points as the testing set decision making points (120× 20).




