{"action":"created","comment":{"url":"https://api.github.com/repos/quantopian/zipline/pulls/comments/210401731","pull_request_review_id":146611073,"id":210401731,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxMDQwMTczMQ==","diff_hunk":"@@ -0,0 +1,60 @@\n+\"\"\"Tests for zipline.pipeline.data.DataSet and friends.","path":"tests/pipeline/test_dataset.py","position":null,"original_position":1,"commit_id":"ff9ed6efe8ba1b1d8079165d23adf2b13841e417","original_commit_id":"ecb229524f12db0bfd5944b6c165f2937f9f20d1","user":{"login":"ssanderson","id":1588357,"node_id":"MDQ6VXNlcjE1ODgzNTc=","avatar_url":"https://avatars0.githubusercontent.com/u/1588357?v=4","gravatar_id":"","url":"https://api.github.com/users/ssanderson","html_url":"https://github.com/ssanderson","followers_url":"https://api.github.com/users/ssanderson/followers","following_url":"https://api.github.com/users/ssanderson/following{/other_user}","gists_url":"https://api.github.com/users/ssanderson/gists{/gist_id}","starred_url":"https://api.github.com/users/ssanderson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssanderson/subscriptions","organizations_url":"https://api.github.com/users/ssanderson/orgs","repos_url":"https://api.github.com/users/ssanderson/repos","events_url":"https://api.github.com/users/ssanderson/events{/privacy}","received_events_url":"https://api.github.com/users/ssanderson/received_events","type":"User","site_admin":false},"body":"I merged this into `test_domain_specialization`","created_at":"2018-08-15T20:41:09Z","updated_at":"2018-08-15T20:41:09Z","html_url":"https://github.com/quantopian/zipline/pull/2262#discussion_r210401731","pull_request_url":"https://api.github.com/repos/quantopian/zipline/pulls/2262","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/comments/210401731"},"html":{"href":"https://github.com/quantopian/zipline/pull/2262#discussion_r210401731"},"pull_request":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/2262"}},"in_reply_to_id":208989668},"pull_request":{"url":"https://api.github.com/repos/quantopian/zipline/pulls/2262","id":204055363,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA0MDU1MzYz","html_url":"https://github.com/quantopian/zipline/pull/2262","diff_url":"https://github.com/quantopian/zipline/pull/2262.diff","patch_url":"https://github.com/quantopian/zipline/pull/2262.patch","issue_url":"https://api.github.com/repos/quantopian/zipline/issues/2262","number":2262,"state":"open","locked":false,"title":"International Pipelines","user":{"login":"ssanderson","id":1588357,"node_id":"MDQ6VXNlcjE1ODgzNTc=","avatar_url":"https://avatars0.githubusercontent.com/u/1588357?v=4","gravatar_id":"","url":"https://api.github.com/users/ssanderson","html_url":"https://github.com/ssanderson","followers_url":"https://api.github.com/users/ssanderson/followers","following_url":"https://api.github.com/users/ssanderson/following{/other_user}","gists_url":"https://api.github.com/users/ssanderson/gists{/gist_id}","starred_url":"https://api.github.com/users/ssanderson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssanderson/subscriptions","organizations_url":"https://api.github.com/users/ssanderson/orgs","repos_url":"https://api.github.com/users/ssanderson/repos","events_url":"https://api.github.com/users/ssanderson/events{/privacy}","received_events_url":"https://api.github.com/users/ssanderson/received_events","type":"User","site_admin":false},"body":"This is an implementation of the work outlined in https://github.com/quantopian/zipline/issues/2265.\r\n\r\nIt includes:\r\n\r\n- A new `zipline.pipeline.domain` module with a new `Domain` class.\r\n- Generic pipeline expressions and limited support for specialization (see below).\r\n- Updates to all USEquityPricing-based terms to use EquityPricing instead.\r\n- Updates to all existing loaders to accept a new `domain` arg. As part of this change, I've added [`interface`](https://github.com/ssanderson/interface).\r\n- New tests in `tests/pipeline/test_international_markets.py` and `tests/pipeline/test_domain_specialization.py`.\r\n- Lots of updates to existing tests (mostly fixing bugs where we needed to apply specialization, or where we were failing to find any assets in the domain of the pipeline after applying specialization). In general, I updated tests that were already using the NYSE calendar to us the USEquities domain, and I updated tests that were specifying custom session indices to use EquitySessionDomain.\r\n\r\n### Notes and Open Questions\r\n\r\n#### Domains and Calendars\r\n\r\nIn my original design, a domain was always guaranteed to be associated with a proper `trading_calendars.TradingCalendar`.  Much of the pipeline machinery, however, was already parametric on an arbitrary DatetimeIndex, which was used to compute output labels, and consequently there a bunch of places in the test suite where we were providing an index that doesn't naturally permit representation as a calendar (we could implement a \"calendar\" that just takes a DatetimeIndex, but at that point it's not really clear what use the calendar abstraction is providing on top of just having a DatetimeIndex).\r\n\r\nIn light of the above, the easiest way to get everything working was to change the interface of Domain to provide an `all_session` index instead of a full TradingCalendar. This has the advantage of requiring very few changes to the existing code, and after working with it I'm inclined to think that it's actually a cleaner interface for domains. I'd be curious to hear if people can think of cases where it'd be important to have access to a full calendar instead of just a sessions index. One case where it might be relevant is handling non-8:45 cutoff dates (see below), though we could just as easily add a `data_availability_cutoff` attribute to the domain interface.\r\n\r\n#### Testing Gaps?\r\n\r\nI've called out a few places in the test suite that seem like they would probably benefit from having some new tests written with domain parametricity in mind, but I'm sure there are some places I've missed.\r\n\r\n#### Module Moves\r\n\r\nI've split up `zipline.data.us_equity_pricing` into `zipline.data.adjustments` and `zipline.data.bcolz_daily_bars`.\r\n\r\n#### Specialization only for `LoadableTerm`s\r\n\r\nCurrently, this PR currently only implements specialization for `LoadableTerm`s. Avoiding general-purpose specialization makes some things simpler (in particular, we don't have to figure out how to specialize all the different Filter/Factor/Classifier subclasses with non-uniform constructors), but it also adds some compliexity to `ExecutionPlan` and `PipelineEngine`, which are now responsible for specializing all `LoadableTerms` prior to:\r\n  1. Passing terms to `get_loader`.\r\n  2. Passing terms to any `load_adjusted_array` functions.\r\n  \r\nFor the most part I think this branch manages to keep the complexity of specialization low by strategically specializing at the right time in ExecutionPlan (earlier drafts of this PR did the specialization in PipelineEngine, which was much more complicated).\r\n\r\n#### Generic Terms and `populate_initial_workspace`\r\n\r\nThere's currently a big TODO block in SimplePipelineEngine about whether/how we want to support generic ComputableTerms in initial workspaces. My initial reasoning is that we shouldn't allow generics in an initial workspace, because by definition a generic term doesn't correspond to any concrete result until it's been specialized to a domain, but since we don't (yet) support specialization for generic terms, there isn't any way for `populate_initial_workspace` to signal that it can pre-compute a generic term unless it's allowed to provide a generic.\r\n\r\nI think there are two reasonable ways we can go here:\r\n\r\n1. Disallow any generic terms in initial workspaces, with the rationale being that a generic term never corresponds to a concrete output. This could eventually be fixed by adding support for specialization to all Pipeline terms.\r\n2. Pass a domain to `populate_initial_workspace`, and have the contract be that generic terms should be populated with the specialization for the given domain. I think this is reasonable in isolation (and is probably the easiest thing to implement in the short term), but it also creates an awkward asymmetry when compared to `load_adjusted_array`, whose contract is currently that it always gets passed specialized terms.\r\n\r\n#### (Not) Handling Non-8:45 Cutoff Times\r\n\r\nOne important piece of functionality missing from this PR is a system for handling different choices of cutoff time for intraday data availability. Happily, our first target market, Canada, has the same start time as NYSE, so I don't think we need to solve that problem for a first deliverable. \r\n\r\nCurrently, only the `BlazeLoader` supports any notion of a variable cutoff time for intraday data bucketing, but the cutoff time is specified as a constructor parameter to the loader. I wonder if it might be more natural to associate the cutoff time with the domain? On the one hand, this seems reasonable because the domain is what knows about market-specific timing information (e.g., when the market opens). On the other hand, the loader should be responsible for actually knowing when data was ingested and should be made available. Maybe the blaze loader should be given a map from domain to availability time instead of a single static time?\r\n\r\n#### Naming Things\r\n\r\nReprs, strs, and names are in a bit of an odd place at the moment on this branch.\r\n\r\nDatasets currently repr as follows:\r\n\r\n```\r\nIn [8]: EquityPricing\r\nOut[8]: <DataSet: 'EquityPricing', domain=sentinel('NotSpecified')>\r\n\r\nIn [9]: USEquityPricing\r\nOut[9]: <DataSet: 'EquityPricing', domain=EquityCountryDomain('US', 'NYSE')>\r\n```\r\n\r\nBoundColumns, however, repr like this:\r\n\r\n```\r\nIn [10]: EquityPricing.close\r\nOut[10]: EquityPricing.close::float64\r\n\r\nIn [11]: USEquityPricing.close\r\nOut[11]: EquityPricing<US>.close::float64\r\n```\r\n\r\nThe syntax here is inspired by Java/C++/Rust generics. Technically it's a bit misleading to have just the country code in the `<>` instead of a representation of a full domain, but I think in most real-world contexts domains are going to be primarily associated with countries (at least in the near term), and I don't want to make the BoundColumn repr a lot longer. (As is, I'm somewhat tempted to drop the type annotation).\r\n\r\nThese reprs seem individually fine, but it feels a little weird that we're rendering the dataset fairly differently in different contexts.","created_at":"2018-07-26T07:47:20Z","updated_at":"2018-08-15T20:41:09Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bed9438ff06c5aafcc9a816389abaa03880a13b1","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/quantopian/zipline/pulls/2262/commits","review_comments_url":"https://api.github.com/repos/quantopian/zipline/pulls/2262/comments","review_comment_url":"https://api.github.com/repos/quantopian/zipline/pulls/comments{/number}","comments_url":"https://api.github.com/repos/quantopian/zipline/issues/2262/comments","statuses_url":"https://api.github.com/repos/quantopian/zipline/statuses/ff9ed6efe8ba1b1d8079165d23adf2b13841e417","head":{"label":"quantopian:international-pipelines","ref":"international-pipelines","sha":"ff9ed6efe8ba1b1d8079165d23adf2b13841e417","user":{"login":"quantopian","id":1393215,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzOTMyMTU=","avatar_url":"https://avatars0.githubusercontent.com/u/1393215?v=4","gravatar_id":"","url":"https://api.github.com/users/quantopian","html_url":"https://github.com/quantopian","followers_url":"https://api.github.com/users/quantopian/followers","following_url":"https://api.github.com/users/quantopian/following{/other_user}","gists_url":"https://api.github.com/users/quantopian/gists{/gist_id}","starred_url":"https://api.github.com/users/quantopian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quantopian/subscriptions","organizations_url":"https://api.github.com/users/quantopian/orgs","repos_url":"https://api.github.com/users/quantopian/repos","events_url":"https://api.github.com/users/quantopian/events{/privacy}","received_events_url":"https://api.github.com/users/quantopian/received_events","type":"Organization","site_admin":false},"repo":{"id":6297520,"node_id":"MDEwOlJlcG9zaXRvcnk2Mjk3NTIw","name":"zipline","full_name":"quantopian/zipline","owner":{"login":"quantopian","id":1393215,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzOTMyMTU=","avatar_url":"https://avatars0.githubusercontent.com/u/1393215?v=4","gravatar_id":"","url":"https://api.github.com/users/quantopian","html_url":"https://github.com/quantopian","followers_url":"https://api.github.com/users/quantopian/followers","following_url":"https://api.github.com/users/quantopian/following{/other_user}","gists_url":"https://api.github.com/users/quantopian/gists{/gist_id}","starred_url":"https://api.github.com/users/quantopian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quantopian/subscriptions","organizations_url":"https://api.github.com/users/quantopian/orgs","repos_url":"https://api.github.com/users/quantopian/repos","events_url":"https://api.github.com/users/quantopian/events{/privacy}","received_events_url":"https://api.github.com/users/quantopian/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/quantopian/zipline","description":"Zipline, a Pythonic Algorithmic Trading Library","fork":false,"url":"https://api.github.com/repos/quantopian/zipline","forks_url":"https://api.github.com/repos/quantopian/zipline/forks","keys_url":"https://api.github.com/repos/quantopian/zipline/keys{/key_id}","collaborators_url":"https://api.github.com/repos/quantopian/zipline/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/quantopian/zipline/teams","hooks_url":"https://api.github.com/repos/quantopian/zipline/hooks","issue_events_url":"https://api.github.com/repos/quantopian/zipline/issues/events{/number}","events_url":"https://api.github.com/repos/quantopian/zipline/events","assignees_url":"https://api.github.com/repos/quantopian/zipline/assignees{/user}","branches_url":"https://api.github.com/repos/quantopian/zipline/branches{/branch}","tags_url":"https://api.github.com/repos/quantopian/zipline/tags","blobs_url":"https://api.github.com/repos/quantopian/zipline/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/quantopian/zipline/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/quantopian/zipline/git/refs{/sha}","trees_url":"https://api.github.com/repos/quantopian/zipline/git/trees{/sha}","statuses_url":"https://api.github.com/repos/quantopian/zipline/statuses/{sha}","languages_url":"https://api.github.com/repos/quantopian/zipline/languages","stargazers_url":"https://api.github.com/repos/quantopian/zipline/stargazers","contributors_url":"https://api.github.com/repos/quantopian/zipline/contributors","subscribers_url":"https://api.github.com/repos/quantopian/zipline/subscribers","subscription_url":"https://api.github.com/repos/quantopian/zipline/subscription","commits_url":"https://api.github.com/repos/quantopian/zipline/commits{/sha}","git_commits_url":"https://api.github.com/repos/quantopian/zipline/git/commits{/sha}","comments_url":"https://api.github.com/repos/quantopian/zipline/comments{/number}","issue_comment_url":"https://api.github.com/repos/quantopian/zipline/issues/comments{/number}","contents_url":"https://api.github.com/repos/quantopian/zipline/contents/{+path}","compare_url":"https://api.github.com/repos/quantopian/zipline/compare/{base}...{head}","merges_url":"https://api.github.com/repos/quantopian/zipline/merges","archive_url":"https://api.github.com/repos/quantopian/zipline/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/quantopian/zipline/downloads","issues_url":"https://api.github.com/repos/quantopian/zipline/issues{/number}","pulls_url":"https://api.github.com/repos/quantopian/zipline/pulls{/number}","milestones_url":"https://api.github.com/repos/quantopian/zipline/milestones{/number}","notifications_url":"https://api.github.com/repos/quantopian/zipline/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/quantopian/zipline/labels{/name}","releases_url":"https://api.github.com/repos/quantopian/zipline/releases{/id}","deployments_url":"https://api.github.com/repos/quantopian/zipline/deployments","created_at":"2012-10-19T15:50:29Z","updated_at":"2018-08-15T14:51:36Z","pushed_at":"2018-08-15T20:40:48Z","git_url":"git://github.com/quantopian/zipline.git","ssh_url":"git@github.com:quantopian/zipline.git","clone_url":"https://github.com/quantopian/zipline.git","svn_url":"https://github.com/quantopian/zipline","homepage":"http://www.zipline.io/","size":154529,"stargazers_count":7478,"watchers_count":7478,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":2366,"mirror_url":null,"archived":false,"open_issues_count":232,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":2366,"open_issues":232,"watchers":7478,"default_branch":"master"}},"base":{"label":"quantopian:master","ref":"master","sha":"4f9834583e7f831f05e6ddcc1a0186595d8522cd","user":{"login":"quantopian","id":1393215,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzOTMyMTU=","avatar_url":"https://avatars0.githubusercontent.com/u/1393215?v=4","gravatar_id":"","url":"https://api.github.com/users/quantopian","html_url":"https://github.com/quantopian","followers_url":"https://api.github.com/users/quantopian/followers","following_url":"https://api.github.com/users/quantopian/following{/other_user}","gists_url":"https://api.github.com/users/quantopian/gists{/gist_id}","starred_url":"https://api.github.com/users/quantopian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quantopian/subscriptions","organizations_url":"https://api.github.com/users/quantopian/orgs","repos_url":"https://api.github.com/users/quantopian/repos","events_url":"https://api.github.com/users/quantopian/events{/privacy}","received_events_url":"https://api.github.com/users/quantopian/received_events","type":"Organization","site_admin":false},"repo":{"id":6297520,"node_id":"MDEwOlJlcG9zaXRvcnk2Mjk3NTIw","name":"zipline","full_name":"quantopian/zipline","owner":{"login":"quantopian","id":1393215,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzOTMyMTU=","avatar_url":"https://avatars0.githubusercontent.com/u/1393215?v=4","gravatar_id":"","url":"https://api.github.com/users/quantopian","html_url":"https://github.com/quantopian","followers_url":"https://api.github.com/users/quantopian/followers","following_url":"https://api.github.com/users/quantopian/following{/other_user}","gists_url":"https://api.github.com/users/quantopian/gists{/gist_id}","starred_url":"https://api.github.com/users/quantopian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quantopian/subscriptions","organizations_url":"https://api.github.com/users/quantopian/orgs","repos_url":"https://api.github.com/users/quantopian/repos","events_url":"https://api.github.com/users/quantopian/events{/privacy}","received_events_url":"https://api.github.com/users/quantopian/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/quantopian/zipline","description":"Zipline, a Pythonic Algorithmic Trading Library","fork":false,"url":"https://api.github.com/repos/quantopian/zipline","forks_url":"https://api.github.com/repos/quantopian/zipline/forks","keys_url":"https://api.github.com/repos/quantopian/zipline/keys{/key_id}","collaborators_url":"https://api.github.com/repos/quantopian/zipline/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/quantopian/zipline/teams","hooks_url":"https://api.github.com/repos/quantopian/zipline/hooks","issue_events_url":"https://api.github.com/repos/quantopian/zipline/issues/events{/number}","events_url":"https://api.github.com/repos/quantopian/zipline/events","assignees_url":"https://api.github.com/repos/quantopian/zipline/assignees{/user}","branches_url":"https://api.github.com/repos/quantopian/zipline/branches{/branch}","tags_url":"https://api.github.com/repos/quantopian/zipline/tags","blobs_url":"https://api.github.com/repos/quantopian/zipline/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/quantopian/zipline/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/quantopian/zipline/git/refs{/sha}","trees_url":"https://api.github.com/repos/quantopian/zipline/git/trees{/sha}","statuses_url":"https://api.github.com/repos/quantopian/zipline/statuses/{sha}","languages_url":"https://api.github.com/repos/quantopian/zipline/languages","stargazers_url":"https://api.github.com/repos/quantopian/zipline/stargazers","contributors_url":"https://api.github.com/repos/quantopian/zipline/contributors","subscribers_url":"https://api.github.com/repos/quantopian/zipline/subscribers","subscription_url":"https://api.github.com/repos/quantopian/zipline/subscription","commits_url":"https://api.github.com/repos/quantopian/zipline/commits{/sha}","git_commits_url":"https://api.github.com/repos/quantopian/zipline/git/commits{/sha}","comments_url":"https://api.github.com/repos/quantopian/zipline/comments{/number}","issue_comment_url":"https://api.github.com/repos/quantopian/zipline/issues/comments{/number}","contents_url":"https://api.github.com/repos/quantopian/zipline/contents/{+path}","compare_url":"https://api.github.com/repos/quantopian/zipline/compare/{base}...{head}","merges_url":"https://api.github.com/repos/quantopian/zipline/merges","archive_url":"https://api.github.com/repos/quantopian/zipline/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/quantopian/zipline/downloads","issues_url":"https://api.github.com/repos/quantopian/zipline/issues{/number}","pulls_url":"https://api.github.com/repos/quantopian/zipline/pulls{/number}","milestones_url":"https://api.github.com/repos/quantopian/zipline/milestones{/number}","notifications_url":"https://api.github.com/repos/quantopian/zipline/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/quantopian/zipline/labels{/name}","releases_url":"https://api.github.com/repos/quantopian/zipline/releases{/id}","deployments_url":"https://api.github.com/repos/quantopian/zipline/deployments","created_at":"2012-10-19T15:50:29Z","updated_at":"2018-08-15T14:51:36Z","pushed_at":"2018-08-15T20:40:48Z","git_url":"git://github.com/quantopian/zipline.git","ssh_url":"git@github.com:quantopian/zipline.git","clone_url":"https://github.com/quantopian/zipline.git","svn_url":"https://github.com/quantopian/zipline","homepage":"http://www.zipline.io/","size":154529,"stargazers_count":7478,"watchers_count":7478,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":2366,"mirror_url":null,"archived":false,"open_issues_count":232,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":2366,"open_issues":232,"watchers":7478,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/2262"},"html":{"href":"https://github.com/quantopian/zipline/pull/2262"},"issue":{"href":"https://api.github.com/repos/quantopian/zipline/issues/2262"},"comments":{"href":"https://api.github.com/repos/quantopian/zipline/issues/2262/comments"},"review_comments":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/2262/comments"},"review_comment":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/2262/commits"},"statuses":{"href":"https://api.github.com/repos/quantopian/zipline/statuses/ff9ed6efe8ba1b1d8079165d23adf2b13841e417"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"ssanderson"}}