{"action":"opened","issue":{"url":"https://api.github.com/repos/scrapinghub/shub/issues/122","labels_url":"https://api.github.com/repos/scrapinghub/shub/issues/122/labels{/name}","comments_url":"https://api.github.com/repos/scrapinghub/shub/issues/122/comments","events_url":"https://api.github.com/repos/scrapinghub/shub/issues/122/events","html_url":"https://github.com/scrapinghub/shub/issues/122","id":126220731,"number":122,"title":"Limit data fetched from hubstorage","user":{"login":"jdemaeyer","id":10531844,"avatar_url":"https://avatars.githubusercontent.com/u/10531844?v=3","gravatar_id":"","url":"https://api.github.com/users/jdemaeyer","html_url":"https://github.com/jdemaeyer","followers_url":"https://api.github.com/users/jdemaeyer/followers","following_url":"https://api.github.com/users/jdemaeyer/following{/other_user}","gists_url":"https://api.github.com/users/jdemaeyer/gists{/gist_id}","starred_url":"https://api.github.com/users/jdemaeyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdemaeyer/subscriptions","organizations_url":"https://api.github.com/users/jdemaeyer/orgs","repos_url":"https://api.github.com/users/jdemaeyer/repos","events_url":"https://api.github.com/users/jdemaeyer/events{/privacy}","received_events_url":"https://api.github.com/users/jdemaeyer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-01-12T16:43:34Z","updated_at":"2016-01-12T16:43:34Z","closed_at":null,"body":"Some jobs have huge logs / many items / requests. We could limit the number of entries downloaded from hubstorage by grabbing the number of entries first, e.g.\r\n```\r\ntotal_lines = job.logs.stats()['totals']['input_values']\r\n# then construct correct `start_after` argument for iter.json() from this\r\n```\r\n\r\nMaybe download only the last 250 items if `-f` is set, with a possibility to override, and download all items if `-f` is not set."}}