I used ten year daily data for 407 stocks and computed the daily and monthly covariance matrices. Since I have more variables than observations for the monthly matrix, I wasn't surprised to find the matrix to be not invertible (and hence useless for portfolio optimization). I was surprised to see the daily covariance matrix not invertible. I then tried to shrink the matrix with the Ledoit-Wolf shrinkage estimator using the package tawny. It didn't help. It makes the covariance matrix really, really small, but no invertible.

Does anyone have any suggestions what could be the problem? How could I improve the covariance matrix?


Are your 407 stocks all different? No A and B listings contained that are strongly if not perfectly correlated? The observation that the daily covariance matrix is singular makes me wonder.

You can try the package corpcor for another shrinkage estimator.

  • $\begingroup$ I reduced the sample to 50 stocks and now the matrix is invertible for daily and monthly returns. I used the Ledoit,Wolf shrinkage estimator for the covariance matrix only and now I get really small portfolio variances, implying unrealistic Sharpe-ratio. I assume this has to do with the fact that I didn't shrink the return vector. Which shrinkage estimator would you use on the return vector? $\endgroup$ – User1111 Aug 28 '15 at 14:22
  • $\begingroup$ If you get small portfolio variances it might be because you have long and short positions in similar stocks. You might consider a no-short sales, i.e. non-negative portfolio weights constraint. $\endgroup$ – Alex C Aug 29 '15 at 19:07
  • $\begingroup$ Thanks Alex C, that could be a problem, I'll check that out! $\endgroup$ – User1111 Aug 30 '15 at 20:07
  • $\begingroup$ @User1111 No, I don't think that shrinking the return vector will do any good. What do you mean by small variances? For sharpe you divide mean by the square-root of variance ... this is bigger than variance. Do you use proper scaling (for annualizing a monthly Sharpe-Ratio you multiply by $\sqrt{12}$, \sqrt{250} for daily data. Why don't you get a non-singular covariance matrix with the original data? Do you have constant time series or time series with missing values that are replaced by constants? $\endgroup$ – Richard Aug 31 '15 at 6:27

The problem with Ledoit-Wolf is that it's very sensitive to outliers.

You should try these:

    • unfortunately, not available in Python
  2. Exponentially weighed moving average (EWMA)
    • gives slighly worse results than DCC-GARCH
  3. Minimum Covariance Determinant
    • suggestted by Scikit-Learn
  4. bootstrap
    • could be used to calculate confidence interval, then you can decided how conservative you want to be

Here are some good references:

  • 1
    $\begingroup$ It'll be great if you could elaborate other methods a bit more $\endgroup$ – Xiaoxiong Lin Dec 11 '18 at 15:32

Your Answer

