{"action":"created","comment":{"url":"https://api.github.com/repos/ContinuumIO/intake/pulls/comments/239180832","pull_request_review_id":181902070,"id":239180832,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzOTE4MDgzMg==","diff_hunk":"@@ -188,11 +206,86 @@ def _close(self):\n         # TODO: maybe close all entries?\n         pass\n \n+class Entries(dict):\n+    \"\"\"Fetches entries from server on item lookup and iteration.\n+\n+    This fetches pages of entries from the server during iteration and\n+    caches them. On __getitem__ it fetches the sepcific entry from the\n+    server.\n+    \"\"\"\n+    # This has PY3-style lazy methods (keys, values, items). Since it's\n+    # internal we should not need the PY2-only iter* variants.\n+    def __init__(self, catalog):\n+        self._catalog = catalog\n+        self._page_cache = collections.OrderedDict()\n+        # Put lookups that were due to __getitem__ in a separate cache\n+        # so that iteration reflects the server's order, not an\n+        # arbitrary cache order.\n+        self._direct_lookup_cache = {}\n+        self._page_offset = 0\n+\n+    def reset(self):\n+        \"Clear caches to force a reload.\"\n+        self._page_cache.clear()\n+        self._direct_lookup_cache.clear()\n+        self._page_offset = 0\n+\n+    def __iter__(self):\n+        for key in self.keys():\n+            yield key\n+\n+    def __contains__(self, key):\n+        # Avoid iterating through all entries.\n+        try:\n+            self[key]\n+        except KeyError:\n+            return False\n+        else:\n+            return True\n+\n+    def items(self):\n+        for item in six.iteritems(self._page_cache):\n+            yield item\n+        if self._catalog.page_size is None:\n+            # We are not paginating, either because the user set page_size=None\n+            # or the server is a version of intake before pagination parameters\n+            # were added.\n+            return\n+        # Fetch more entries from the server.\n+        while True:\n+            page = self._catalog.fetch_page(self._page_offset)\n+            self._page_cache.update(page)","path":"intake/catalog/base.py","position":97,"original_position":102,"commit_id":"1158ab4269db94bcc97e226a3e95a39ebdd45099","original_commit_id":"f8d97ceecfd360c287775411246b29e482e9a6af","user":{"login":"danielballan","id":2279598,"node_id":"MDQ6VXNlcjIyNzk1OTg=","avatar_url":"https://avatars3.githubusercontent.com/u/2279598?v=4","gravatar_id":"","url":"https://api.github.com/users/danielballan","html_url":"https://github.com/danielballan","followers_url":"https://api.github.com/users/danielballan/followers","following_url":"https://api.github.com/users/danielballan/following{/other_user}","gists_url":"https://api.github.com/users/danielballan/gists{/gist_id}","starred_url":"https://api.github.com/users/danielballan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielballan/subscriptions","organizations_url":"https://api.github.com/users/danielballan/orgs","repos_url":"https://api.github.com/users/danielballan/repos","events_url":"https://api.github.com/users/danielballan/events{/privacy}","received_events_url":"https://api.github.com/users/danielballan/received_events","type":"User","site_admin":false},"body":"Good question, @martindurant. Let me think this through. The purpose of `_page_cache` is to expedite subsequent calls to `items()`. Since those _always_ start from the top -- there is no API to start in the middle -- it makes sense to hold on to the early pages no matter what.\r\n\r\n```\r\niterator1 = cat.items()\r\n# Iterate through some items.\r\niterator2 = cat.items()\r\n# If have dumped the first page, we'll have to pull it again.\r\n```\r\n\r\nWe could impose a limit on the number of pages and/entries to cache, as in a new parameter `page_cache_size` alongside `page_size` to make it possible to iterate through unbounded catalogs. I think that's a good idea. But I don't think purging from the head is ever optimal, which is fortunate from the point of view of simplicity.\r\n\r\nDoes that seem like a good approach to you?\r\n\r\nTangentially related:\r\n* I have a `search` branch based off of this branch that supports search on remote catalogs (and nested catalogs therein) which will also be useful for navigating through large catalogs, addressing that concern about \"there is no API to start in the middle\" mentioned above. \r\n* As you likely already noticed, the _entire_ cache is occasionally purged, of course, whenever the Catalog reloads.","created_at":"2018-12-05T18:20:39Z","updated_at":"2018-12-05T18:20:39Z","html_url":"https://github.com/ContinuumIO/intake/pull/200#discussion_r239180832","pull_request_url":"https://api.github.com/repos/ContinuumIO/intake/pulls/200","author_association":"NONE","_links":{"self":{"href":"https://api.github.com/repos/ContinuumIO/intake/pulls/comments/239180832"},"html":{"href":"https://github.com/ContinuumIO/intake/pull/200#discussion_r239180832"},"pull_request":{"href":"https://api.github.com/repos/ContinuumIO/intake/pulls/200"}},"in_reply_to_id":239147109},"pull_request":{"url":"https://api.github.com/repos/ContinuumIO/intake/pulls/200","id":234128031,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM0MTI4MDMx","html_url":"https://github.com/ContinuumIO/intake/pull/200","diff_url":"https://github.com/ContinuumIO/intake/pull/200.diff","patch_url":"https://github.com/ContinuumIO/intake/pull/200.patch","issue_url":"https://api.github.com/repos/ContinuumIO/intake/issues/200","number":200,"state":"open","locked":false,"title":"Add pagination support to server and RemoteCatalog","user":{"login":"danielballan","id":2279598,"node_id":"MDQ6VXNlcjIyNzk1OTg=","avatar_url":"https://avatars3.githubusercontent.com/u/2279598?v=4","gravatar_id":"","url":"https://api.github.com/users/danielballan","html_url":"https://github.com/danielballan","followers_url":"https://api.github.com/users/danielballan/followers","following_url":"https://api.github.com/users/danielballan/following{/other_user}","gists_url":"https://api.github.com/users/danielballan/gists{/gist_id}","starred_url":"https://api.github.com/users/danielballan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielballan/subscriptions","organizations_url":"https://api.github.com/users/danielballan/orgs","repos_url":"https://api.github.com/users/danielballan/repos","events_url":"https://api.github.com/users/danielballan/events{/privacy}","received_events_url":"https://api.github.com/users/danielballan/received_events","type":"User","site_admin":false},"body":"Thanks for this excellent project!\r\n\r\nI have a Catalog with ~10k entries. (It happens to be backed by MongoDB, not YAML.) To make this scalable, I need to avoid pulling down all the entries from the server. This PR adds support for optional pagination parameters to the server's `GET /info` route and adds a new `GET /source` method to get the 'info' about one source. Respectively, these two server changes allow the `RemoteSource` to fetch entry info in batches during iteration (only pulling down as many as are used) and to fetch any specific entry's info by name (random access) when called on by `__getitem__`.\r\n\r\nI have added some pagination unit tests. I have also interactively tested a client running on the `master` branch against a server running on this branch and vice versa. Each side gracefully falls back to non-paginated behavior when its peer does not support pagination.\r\n\r\nIn commit b1dabc4 I removed a redundant lookup but also made a slight API change: the Catalog metadata must now be accessed by attribute, not by `__getitem__`. Two small changes to existing tests were necessary to adjust for this. I believe this is an improvement. It is nice that entries are accessible by attribute, but I believe the inverse should _not_ be supported, lest the line between the library's core interface and its entries become too blurry and lead to confusing Tracebacks. If this change is controversial I am happy to rip it out into separate PR for further discussion.\r\n\r\nThanks in advance for taking a look.","created_at":"2018-11-28T02:35:51Z","updated_at":"2018-12-05T18:20:39Z","closed_at":null,"merged_at":null,"merge_commit_sha":"471a7150e813ed4af35d08d96bdcd26df80bd124","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/ContinuumIO/intake/pulls/200/commits","review_comments_url":"https://api.github.com/repos/ContinuumIO/intake/pulls/200/comments","review_comment_url":"https://api.github.com/repos/ContinuumIO/intake/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ContinuumIO/intake/issues/200/comments","statuses_url":"https://api.github.com/repos/ContinuumIO/intake/statuses/1158ab4269db94bcc97e226a3e95a39ebdd45099","head":{"label":"danielballan:pagination","ref":"pagination","sha":"1158ab4269db94bcc97e226a3e95a39ebdd45099","user":{"login":"danielballan","id":2279598,"node_id":"MDQ6VXNlcjIyNzk1OTg=","avatar_url":"https://avatars3.githubusercontent.com/u/2279598?v=4","gravatar_id":"","url":"https://api.github.com/users/danielballan","html_url":"https://github.com/danielballan","followers_url":"https://api.github.com/users/danielballan/followers","following_url":"https://api.github.com/users/danielballan/following{/other_user}","gists_url":"https://api.github.com/users/danielballan/gists{/gist_id}","starred_url":"https://api.github.com/users/danielballan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielballan/subscriptions","organizations_url":"https://api.github.com/users/danielballan/orgs","repos_url":"https://api.github.com/users/danielballan/repos","events_url":"https://api.github.com/users/danielballan/events{/privacy}","received_events_url":"https://api.github.com/users/danielballan/received_events","type":"User","site_admin":false},"repo":{"id":158026547,"node_id":"MDEwOlJlcG9zaXRvcnkxNTgwMjY1NDc=","name":"intake","full_name":"danielballan/intake","private":false,"owner":{"login":"danielballan","id":2279598,"node_id":"MDQ6VXNlcjIyNzk1OTg=","avatar_url":"https://avatars3.githubusercontent.com/u/2279598?v=4","gravatar_id":"","url":"https://api.github.com/users/danielballan","html_url":"https://github.com/danielballan","followers_url":"https://api.github.com/users/danielballan/followers","following_url":"https://api.github.com/users/danielballan/following{/other_user}","gists_url":"https://api.github.com/users/danielballan/gists{/gist_id}","starred_url":"https://api.github.com/users/danielballan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielballan/subscriptions","organizations_url":"https://api.github.com/users/danielballan/orgs","repos_url":"https://api.github.com/users/danielballan/repos","events_url":"https://api.github.com/users/danielballan/events{/privacy}","received_events_url":"https://api.github.com/users/danielballan/received_events","type":"User","site_admin":false},"html_url":"https://github.com/danielballan/intake","description":"A plugin system for loading your data and making data catalogs.","fork":true,"url":"https://api.github.com/repos/danielballan/intake","forks_url":"https://api.github.com/repos/danielballan/intake/forks","keys_url":"https://api.github.com/repos/danielballan/intake/keys{/key_id}","collaborators_url":"https://api.github.com/repos/danielballan/intake/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/danielballan/intake/teams","hooks_url":"https://api.github.com/repos/danielballan/intake/hooks","issue_events_url":"https://api.github.com/repos/danielballan/intake/issues/events{/number}","events_url":"https://api.github.com/repos/danielballan/intake/events","assignees_url":"https://api.github.com/repos/danielballan/intake/assignees{/user}","branches_url":"https://api.github.com/repos/danielballan/intake/branches{/branch}","tags_url":"https://api.github.com/repos/danielballan/intake/tags","blobs_url":"https://api.github.com/repos/danielballan/intake/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/danielballan/intake/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/danielballan/intake/git/refs{/sha}","trees_url":"https://api.github.com/repos/danielballan/intake/git/trees{/sha}","statuses_url":"https://api.github.com/repos/danielballan/intake/statuses/{sha}","languages_url":"https://api.github.com/repos/danielballan/intake/languages","stargazers_url":"https://api.github.com/repos/danielballan/intake/stargazers","contributors_url":"https://api.github.com/repos/danielballan/intake/contributors","subscribers_url":"https://api.github.com/repos/danielballan/intake/subscribers","subscription_url":"https://api.github.com/repos/danielballan/intake/subscription","commits_url":"https://api.github.com/repos/danielballan/intake/commits{/sha}","git_commits_url":"https://api.github.com/repos/danielballan/intake/git/commits{/sha}","comments_url":"https://api.github.com/repos/danielballan/intake/comments{/number}","issue_comment_url":"https://api.github.com/repos/danielballan/intake/issues/comments{/number}","contents_url":"https://api.github.com/repos/danielballan/intake/contents/{+path}","compare_url":"https://api.github.com/repos/danielballan/intake/compare/{base}...{head}","merges_url":"https://api.github.com/repos/danielballan/intake/merges","archive_url":"https://api.github.com/repos/danielballan/intake/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/danielballan/intake/downloads","issues_url":"https://api.github.com/repos/danielballan/intake/issues{/number}","pulls_url":"https://api.github.com/repos/danielballan/intake/pulls{/number}","milestones_url":"https://api.github.com/repos/danielballan/intake/milestones{/number}","notifications_url":"https://api.github.com/repos/danielballan/intake/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/danielballan/intake/labels{/name}","releases_url":"https://api.github.com/repos/danielballan/intake/releases{/id}","deployments_url":"https://api.github.com/repos/danielballan/intake/deployments","created_at":"2018-11-17T21:30:00Z","updated_at":"2018-11-17T21:30:03Z","pushed_at":"2018-12-05T18:03:03Z","git_url":"git://github.com/danielballan/intake.git","ssh_url":"git@github.com:danielballan/intake.git","clone_url":"https://github.com/danielballan/intake.git","svn_url":"https://github.com/danielballan/intake","homepage":"https://intake.readthedocs.io/en/latest/quickstart.html","size":4049,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"bsd-2-clause","name":"BSD 2-Clause \"Simplified\" License","spdx_id":"BSD-2-Clause","url":"https://api.github.com/licenses/bsd-2-clause","node_id":"MDc6TGljZW5zZTQ="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"ContinuumIO:master","ref":"master","sha":"e23a7da1f6e2560c84d79a9dea5c041b2a0fe0e1","user":{"login":"ContinuumIO","id":1158637,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExNTg2Mzc=","avatar_url":"https://avatars3.githubusercontent.com/u/1158637?v=4","gravatar_id":"","url":"https://api.github.com/users/ContinuumIO","html_url":"https://github.com/ContinuumIO","followers_url":"https://api.github.com/users/ContinuumIO/followers","following_url":"https://api.github.com/users/ContinuumIO/following{/other_user}","gists_url":"https://api.github.com/users/ContinuumIO/gists{/gist_id}","starred_url":"https://api.github.com/users/ContinuumIO/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ContinuumIO/subscriptions","organizations_url":"https://api.github.com/users/ContinuumIO/orgs","repos_url":"https://api.github.com/users/ContinuumIO/repos","events_url":"https://api.github.com/users/ContinuumIO/events{/privacy}","received_events_url":"https://api.github.com/users/ContinuumIO/received_events","type":"Organization","site_admin":false},"repo":{"id":100307970,"node_id":"MDEwOlJlcG9zaXRvcnkxMDAzMDc5NzA=","name":"intake","full_name":"ContinuumIO/intake","private":false,"owner":{"login":"ContinuumIO","id":1158637,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExNTg2Mzc=","avatar_url":"https://avatars3.githubusercontent.com/u/1158637?v=4","gravatar_id":"","url":"https://api.github.com/users/ContinuumIO","html_url":"https://github.com/ContinuumIO","followers_url":"https://api.github.com/users/ContinuumIO/followers","following_url":"https://api.github.com/users/ContinuumIO/following{/other_user}","gists_url":"https://api.github.com/users/ContinuumIO/gists{/gist_id}","starred_url":"https://api.github.com/users/ContinuumIO/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ContinuumIO/subscriptions","organizations_url":"https://api.github.com/users/ContinuumIO/orgs","repos_url":"https://api.github.com/users/ContinuumIO/repos","events_url":"https://api.github.com/users/ContinuumIO/events{/privacy}","received_events_url":"https://api.github.com/users/ContinuumIO/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/ContinuumIO/intake","description":"A plugin system for loading your data and making data catalogs.","fork":false,"url":"https://api.github.com/repos/ContinuumIO/intake","forks_url":"https://api.github.com/repos/ContinuumIO/intake/forks","keys_url":"https://api.github.com/repos/ContinuumIO/intake/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ContinuumIO/intake/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ContinuumIO/intake/teams","hooks_url":"https://api.github.com/repos/ContinuumIO/intake/hooks","issue_events_url":"https://api.github.com/repos/ContinuumIO/intake/issues/events{/number}","events_url":"https://api.github.com/repos/ContinuumIO/intake/events","assignees_url":"https://api.github.com/repos/ContinuumIO/intake/assignees{/user}","branches_url":"https://api.github.com/repos/ContinuumIO/intake/branches{/branch}","tags_url":"https://api.github.com/repos/ContinuumIO/intake/tags","blobs_url":"https://api.github.com/repos/ContinuumIO/intake/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ContinuumIO/intake/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ContinuumIO/intake/git/refs{/sha}","trees_url":"https://api.github.com/repos/ContinuumIO/intake/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ContinuumIO/intake/statuses/{sha}","languages_url":"https://api.github.com/repos/ContinuumIO/intake/languages","stargazers_url":"https://api.github.com/repos/ContinuumIO/intake/stargazers","contributors_url":"https://api.github.com/repos/ContinuumIO/intake/contributors","subscribers_url":"https://api.github.com/repos/ContinuumIO/intake/subscribers","subscription_url":"https://api.github.com/repos/ContinuumIO/intake/subscription","commits_url":"https://api.github.com/repos/ContinuumIO/intake/commits{/sha}","git_commits_url":"https://api.github.com/repos/ContinuumIO/intake/git/commits{/sha}","comments_url":"https://api.github.com/repos/ContinuumIO/intake/comments{/number}","issue_comment_url":"https://api.github.com/repos/ContinuumIO/intake/issues/comments{/number}","contents_url":"https://api.github.com/repos/ContinuumIO/intake/contents/{+path}","compare_url":"https://api.github.com/repos/ContinuumIO/intake/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ContinuumIO/intake/merges","archive_url":"https://api.github.com/repos/ContinuumIO/intake/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ContinuumIO/intake/downloads","issues_url":"https://api.github.com/repos/ContinuumIO/intake/issues{/number}","pulls_url":"https://api.github.com/repos/ContinuumIO/intake/pulls{/number}","milestones_url":"https://api.github.com/repos/ContinuumIO/intake/milestones{/number}","notifications_url":"https://api.github.com/repos/ContinuumIO/intake/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ContinuumIO/intake/labels{/name}","releases_url":"https://api.github.com/repos/ContinuumIO/intake/releases{/id}","deployments_url":"https://api.github.com/repos/ContinuumIO/intake/deployments","created_at":"2017-08-14T20:44:22Z","updated_at":"2018-12-05T16:26:47Z","pushed_at":"2018-12-05T18:03:06Z","git_url":"git://github.com/ContinuumIO/intake.git","ssh_url":"git@github.com:ContinuumIO/intake.git","clone_url":"https://github.com/ContinuumIO/intake.git","svn_url":"https://github.com/ContinuumIO/intake","homepage":"https://intake.readthedocs.io/en/latest/quickstart.html","size":4032,"stargazers_count":107,"watchers_count":107,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":15,"mirror_url":null,"archived":false,"open_issues_count":19,"license":{"key":"bsd-2-clause","name":"BSD 2-Clause \"Simplified\" License","spdx_id":"BSD-2-Clause","url":"https://api.github.com/licenses/bsd-2-clause","node_id":"MDc6TGljZW5zZTQ="},"forks":15,"open_issues":19,"watchers":107,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ContinuumIO/intake/pulls/200"},"html":{"href":"https://github.com/ContinuumIO/intake/pull/200"},"issue":{"href":"https://api.github.com/repos/ContinuumIO/intake/issues/200"},"comments":{"href":"https://api.github.com/repos/ContinuumIO/intake/issues/200/comments"},"review_comments":{"href":"https://api.github.com/repos/ContinuumIO/intake/pulls/200/comments"},"review_comment":{"href":"https://api.github.com/repos/ContinuumIO/intake/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ContinuumIO/intake/pulls/200/commits"},"statuses":{"href":"https://api.github.com/repos/ContinuumIO/intake/statuses/1158ab4269db94bcc97e226a3e95a39ebdd45099"}},"author_association":"NONE"}}
{"actor":{"display_login":"danielballan"}}