I have been doing backtesting, and I am seeking to see if there are any flaws in my program, as it seems to be too good to be true.

Based on stocks with market capitalization of > 10B, go back in times say 20 years and backtest. For each stock, Look at the historical data, stock chart morphological feature, and other features, do a bunch of calculation, assign it a score. On each period, it will pick stocks with good scores.

One of the thing I see is that there can be survival bias. The list of stocks I screen have a market capitalization of > 10B TODAY. So only those who survive today is included. Those who goes out of business were secretly gone without me knowing. Those who survive and become great enough to have 10B capitalization are included. So that's far from ideal.

If it is possible, I would like to get a list of stocks with market capitalization > 10B 20 years ago. But where do I get that list of stock? Is there other ways to avoid that bias?


Trying to determine the historical market cap is difficult (especially with mergers/acquisitions/demergers and multiple share classes with different levels of ownership/voting). Another issue with looking at a fixed market cap level is that it's providing a form of selection bias. The further back you go in time, the less stocks will be included due to the effects of inflation. You could adjust this level back in time using an inflation index, but you might also want to consider market cap in relation to the size of the overall economy.

An alternative strategy here is use stocks in a particular index that suit your market cap parameters. You'd need delisted stocks and index constituent / membership data too.

Many indices have a market cap range built into their methodology:

  • S&P 100 (top 100)
  • S&P 500 (top 500)
  • S&P MidCap 400 (501-900)
  • S&P SmallCap 600 (901-1500)
  • Russell 1000 (top 1000)
  • Russell 2000 (1001-3000)
  • Russell 3000 (top 3000)

Disclosure: Norgate Data provides capabilities in this area.

  • 1
    $\begingroup$ I seriously looked in your service. I went through the trouble to install a virutal machine, windows 7 (10 not supported), install python, dot net, and stuff. Only to find out that the python API support only allow querying if a symbol was within the index within that period. There is no option to download the index as a time series. Probably I wouldnt pay 300 dollars for that. What a disappointment. $\endgroup$ – Hairy Ass Dec 2 '19 at 9:19
  • $\begingroup$ You can certainly download the index prices as a time series. For each constituent of the index (available in a watchlist) you can download a timeseries of each of the index membership, which is what most traders actually want to do. i.e. test on everything that has been in the index and trigger their trading rules based upon index membership. Our systems support Windows 7 and 10, so I'm not really sure where you're coming from there. I think you really need to contact our support and detail your issue so we can assist you better, rather than using comments here. $\endgroup$ – Norgate Data Dec 2 '19 at 14:36
  • $\begingroup$ If there is a way I will be very grateful. I am only interested in getting a list of constitutent based on certain date. Here is what the python API says, "To determine whether a stock was an index constituent on a particular date, you can use the index constituent timeseries function. You can also pass in an existing NumPy ndarray or Pandas Dataframe and a new column will be added and returned." So if there is a way, please do show me how. For Windows 10, it just says cannot install dot net framework 4.8 and it stops. But nevermind , I deleted the virtual machine, and installed Windows 7. $\endgroup$ – Hairy Ass Dec 3 '19 at 0:38
  • $\begingroup$ Please contact us via the normal Norgate Data support channel for sample code on how to do this. The code is pretty simple but comments here cannot be formatted correctly. $\endgroup$ – Norgate Data Dec 3 '19 at 5:15

I don't know your budget, and sadly, high quality financial data is not free. There are several good providers of this data. You are looking at spending low to mid 5 figures annually for something like Bloomberg, Reuters, Factset, S&P Global. You can spend a lot less for other providers (Quandl may or may not have what you are looking for), but your mileage will vary.


Others have already suggested that a practical way is to use the composition of a suitable index for your investment universe (to be really safe, the index should have been live at the relevant point in history).

Let me add two remarks. First, the bias you describe is often large. It is studied for US stocks in this paper:

  author       = {Gilles Daniel and Didier Sornette and Peter W{\"o}hrmann},
  year         = 2009,
  volume       = 36,
  number       = 1,
  journal      = {Journal of Portfolio Management},
  pages        = {121--130}

And the authors find that the bias is up to 8% p.a. We looked at this bias for German stocks in Risk-Reward Ratio Optimisation (Revisited), and we found it to be of similar magnitude (about 7% p.a.).

Second, on fixing an absolute size threshold. Better would be to link this threshold to a quantile of market cap. For instance, for US equities Kenneth French publishes percentiles of market cap for NYSE stocks. The following plot shows the evolution of those percentiles.

enter image description here

Your 10bn (shown as the horizontal line) would be around the 75th percentile right now. But if you went back to the 1990s, you wouldn't have too many stocks then. At the start of 1999, for instance, the 75th percentile would have been rather about 3bn or so.

Here would be the R code to reproduce the figure.

bp <- French(dest.dir = "~/Downloads/French",
             dataset = "ME_Breakpoints_CSV.zip")

## make zoo series; scale to millions USD
bp <- zoo(bp[, -c(1, ncol(bp))]/1000000, as.Date(row.names(bp)))

par(mar = c(2,5,1,2), , mgp = c(3.5,0.5,0),
    las = 1, bty = "n", tck = 0.01)
     plot.type = "single",
     log = "y",
     col = hcl.colors(30, palette = "Grays"),
     ylab = "Market cap in millions USD",
     xaxt = "n",
     yaxt = "n")
mtext(text = colnames(bp),
      side = 4, at = coredata(tail(bp,1)),
      line = -0.7, cex = 0.7)

years <- seq(as.Date("1920-1-1"), as.Date("2020-1-1"), by = "20 year")
bn10 <- 10000000000/1000000
axis(1, at = years, labels = format(years, "%Y"))
axis(2, at = c(axTicks(2), bn10))
abline(h = bn10)
  • $\begingroup$ Make me realise using component of index as a search universe is a better idea. thank you $\endgroup$ – Hairy Ass Nov 18 '19 at 14:54
  • $\begingroup$ I am looking at your paper, Enrico - where is the survivorship bias discussed? $\endgroup$ – Igor Rivin May 28 at 17:28
  • $\begingroup$ @IgorRivin: in Appendix A $\endgroup$ – Enrico Schumann May 29 at 18:25

Depending on your strategy, while something to be aware of, the impact of something like this is likely to be minimal. At a 10B market cap, you're well into midcap territory and the companies have already been around a number of years, so it isn't like you're cherry picking small high growth biotech stocks.

Difficult to provide much more absent greater detail as to what you're doing.

  • $\begingroup$ I am thinking whether this impact will be significant or not. Today is year 2019. Say I backtest from 1999. Is it possible that there is a company start up from nothing on 2005, and become a 10B company on 2019. When I actually run in real time, the environment is different from what I try to simulate. $\endgroup$ – Hairy Ass Nov 17 '19 at 11:15
  • $\begingroup$ At end of 2000 Enron was a 60B company, a year later the company was worth 0. Avoiding this disaster may bias the results of 2001 significantly upward. $\endgroup$ – Alex C Nov 17 '19 at 17:03
  • $\begingroup$ Alex makes a valid point, but to my basic statement...that's a single company, depending on your strategy, for a broad, not overly concentrated, equity portfolio, not including it is unlikely to have a drastic impact on simulated performance outcomes. $\endgroup$ – Chris Nov 17 '19 at 21:09

Your Answer

