{"action":"opened","issue":{"url":"https://api.github.com/repos/yujiosaka/headless-chrome-crawler/issues/204","repository_url":"https://api.github.com/repos/yujiosaka/headless-chrome-crawler","labels_url":"https://api.github.com/repos/yujiosaka/headless-chrome-crawler/issues/204/labels{/name}","comments_url":"https://api.github.com/repos/yujiosaka/headless-chrome-crawler/issues/204/comments","events_url":"https://api.github.com/repos/yujiosaka/headless-chrome-crawler/issues/204/events","html_url":"https://github.com/yujiosaka/headless-chrome-crawler/issues/204","id":311677395,"number":204,"title":"Question: Extending/Customizing the behavior of crawler","user":{"login":"panthony","id":230462,"avatar_url":"https://avatars1.githubusercontent.com/u/230462?v=4","gravatar_id":"","url":"https://api.github.com/users/panthony","html_url":"https://github.com/panthony","followers_url":"https://api.github.com/users/panthony/followers","following_url":"https://api.github.com/users/panthony/following{/other_user}","gists_url":"https://api.github.com/users/panthony/gists{/gist_id}","starred_url":"https://api.github.com/users/panthony/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/panthony/subscriptions","organizations_url":"https://api.github.com/users/panthony/orgs","repos_url":"https://api.github.com/users/panthony/repos","events_url":"https://api.github.com/users/panthony/events{/privacy}","received_events_url":"https://api.github.com/users/panthony/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-05T16:15:55Z","updated_at":"2018-04-05T16:15:55Z","closed_at":null,"author_association":"NONE","body":"**What is the current behavior?**\r\n\r\nCurrent crawler takes some arbitrary decision on what to keep with no way for the user to do differently.\r\n\r\n**What is the expected behavior?**\r\n\r\nIt should be possible to customize/extend the behavior to some extent.\r\n\r\nFor exemple today, only the following fields are returned from the response:\r\n\r\n```\r\n  'ok',\r\n  'url',\r\n  'status',\r\n  'headers',\r\n```\r\n\r\nMaybe I'd like something more? Maybe what I want is the result of a method returning a promise?\r\n\r\nI see 2 ways of doing this:\r\n\r\n- Either expose more options (but that could be too many options)\r\n\r\n- Expose the 'Crawler' class, makes some methods public (ex: serializeResponse, serializeRequest, ...) and make it possible to pass it to the HCCrawler. This way one could extend the crawler to override the behavior of some of the methods.\r\n\r\nI'd prefer the later since, with a nice chosen public API, it would be possible to customize easily the behavior of the crawler without exposing dozens and dozens of options.\r\n\r\n**What is the motivation / use case for changing the behavior?**\r\n\r\nI need the redirectChain of the request, the HTTP headers of the request, the returns of the text() method from the response...\r\n\r\nThis is difficult to satisfy all needs with a generic implementation so it should be extendable.\r\n"}}
{"actor":{"display_login":"panthony"}}