{"push_id":1077872892,"size":1,"distinct_size":1,"ref":"refs/heads/use-carray-instead-of-read-all-on-small-size","head":"da1aa5bd82995a2345a6bb60c23718a45be08258","before":"fa3c5ff036fba23849f4ec86338ed7d846298391","commits":[{"sha":"da1aa5bd82995a2345a6bb60c23718a45be08258","author":{"email":"ad498e5a6de621070e46c1f874129b3d9d811a69@quantopian.com","name":"Eddie Hebert"},"message":"PERF: Improve read time for smaller num of assets.\n\nThe BcolzDailyBarReader was optimized for the pipeline case of reading\nall assets at once.\n\nNow that the reader is also used to support daily history the case of\nreading a data for a small number of assets is more common, particularly\nin algorithms that use the history API which have a high rotation of\nassets (e.g. an algorithm which pipeline uses to set the active\nuniverse)\n\nRemove the bottleneck in reading a small number of assets by\nconditionally reading the slice for each asset from the carray, instead\nof reading the data for all equities and then indexing into that full\narray. On a certain number of assets, it is still better to read all the\ndata at once. On the Quantopian dataset, which holds data for 20000\nabout for the last 10 years of equity data (where not all equities trade\nover the full range), stored in 118 blosc blp files per column, the\ntipping point where the 'read all' mode wins out between 3000-4000\nassets.\n\nThat number was tested by trying to exercise a worst case scenario where\nthe equities were spread out evenly across the blp files, by stepping\nalong a sorted list of assets that were alive over a query range which\nspanned 70 trading days.\n```\nsize = 3000\nsids = [assets[i] for i in range(0, len(assets), len(assets) /\nsize)][:size]\n```\n\nAlso, add parameter to WithBcolzDailyBarReader fixture which allows the\ntest to specify what the threshold count for reading all data should be,\nso that the test_us_equity_pricing can be forced into either mode to\nmake sure that both branches in logic are covered by all test cases.\n\nOn local dev machine this patch improves the read time of `load_raw_array`\nfor one asset from 100 ms to 96.5 µs. (10^5 improvement.) With reading\nonly asset per call a being an observed common case when populating the\nnon-cached values in USEquityHistoryLoader.","distinct":true,"url":"https://api.github.com/repos/quantopian/zipline/commits/da1aa5bd82995a2345a6bb60c23718a45be08258"}]}