{"action":"created","issue":{"url":"https://api.github.com/repos/linkedtales/scrapedin/issues/29","repository_url":"https://api.github.com/repos/linkedtales/scrapedin","labels_url":"https://api.github.com/repos/linkedtales/scrapedin/issues/29/labels{/name}","comments_url":"https://api.github.com/repos/linkedtales/scrapedin/issues/29/comments","events_url":"https://api.github.com/repos/linkedtales/scrapedin/issues/29/events","html_url":"https://github.com/linkedtales/scrapedin/issues/29","id":478212376,"node_id":"MDU6SXNzdWU0NzgyMTIzNzY=","number":29,"title":"Store cookies and local storage data to avoid repetitive login","user":{"login":"ghost","id":10137,"node_id":"MDQ6VXNlcjEwMTM3","avatar_url":"https://avatars3.githubusercontent.com/u/10137?v=4","gravatar_id":"","url":"https://api.github.com/users/ghost","html_url":"https://github.com/ghost","followers_url":"https://api.github.com/users/ghost/followers","following_url":"https://api.github.com/users/ghost/following{/other_user}","gists_url":"https://api.github.com/users/ghost/gists{/gist_id}","starred_url":"https://api.github.com/users/ghost/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ghost/subscriptions","organizations_url":"https://api.github.com/users/ghost/orgs","repos_url":"https://api.github.com/users/ghost/repos","events_url":"https://api.github.com/users/ghost/events{/privacy}","received_events_url":"https://api.github.com/users/ghost/received_events","type":"User","site_admin":false},"labels":[{"id":1176352589,"node_id":"MDU6TGFiZWwxMTc2MzUyNTg5","url":"https://api.github.com/repos/linkedtales/scrapedin/labels/enhancement","name":"enhancement","color":"a2eeef","default":true},{"id":1176352591,"node_id":"MDU6TGFiZWwxMTc2MzUyNTkx","url":"https://api.github.com/repos/linkedtales/scrapedin/labels/good%20first%20issue","name":"good first issue","color":"7057ff","default":true},{"id":1176352590,"node_id":"MDU6TGFiZWwxMTc2MzUyNTkw","url":"https://api.github.com/repos/linkedtales/scrapedin/labels/help%20wanted","name":"help wanted","color":"008672","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2019-08-08T01:56:09Z","updated_at":"2019-10-03T21:27:40Z","closed_at":null,"author_association":"NONE","body":"To avoid repetitive login process, the script could store cookies [2] and local storage data [1] in temporary files after the first time login and use them in the subsequent logins if the credentials are still valid so that the users of the module can avoid repetitive login and reduce the chances get their account blocked.\r\n\r\nIf the credentials in the temporary files have expired the page would be redirected to the default login page the script can repeat the process again.\r\n\r\nThis also accelerates the crawling process because the login setup can be skipped.\r\n\r\n**References**\r\n[1] https://stackoverflow.com/questions/51789038/set-localstorage-items-before-page-loads-in-puppeteer\r\n[2] https://pptr.dev/#?product=Puppeteer&version=v1.19.0&show=api-pagesetcookiecookies"},"comment":{"url":"https://api.github.com/repos/linkedtales/scrapedin/issues/comments/538136203","html_url":"https://github.com/linkedtales/scrapedin/issues/29#issuecomment-538136203","issue_url":"https://api.github.com/repos/linkedtales/scrapedin/issues/29","id":538136203,"node_id":"MDEyOklzc3VlQ29tbWVudDUzODEzNjIwMw==","user":{"login":"fraywing","id":1855967,"node_id":"MDQ6VXNlcjE4NTU5Njc=","avatar_url":"https://avatars2.githubusercontent.com/u/1855967?v=4","gravatar_id":"","url":"https://api.github.com/users/fraywing","html_url":"https://github.com/fraywing","followers_url":"https://api.github.com/users/fraywing/followers","following_url":"https://api.github.com/users/fraywing/following{/other_user}","gists_url":"https://api.github.com/users/fraywing/gists{/gist_id}","starred_url":"https://api.github.com/users/fraywing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fraywing/subscriptions","organizations_url":"https://api.github.com/users/fraywing/orgs","repos_url":"https://api.github.com/users/fraywing/repos","events_url":"https://api.github.com/users/fraywing/events{/privacy}","received_events_url":"https://api.github.com/users/fraywing/received_events","type":"User","site_admin":false},"created_at":"2019-10-03T21:27:40Z","updated_at":"2019-10-03T21:27:40Z","author_association":"NONE","body":"Thanks for replying so quickly! I'd love to see the CSV batch code. \r\n\r\n> & I think you can still run this headless on a remote server without need of opening a browser\r\n\r\nYes, but I couldn't just deploy this to a remote server because of the manual check, right?\r\n\r\nI could copy the cookie somehow and run it on the IP address, but the cookie would expire eventually and it would break on remote until we somehow got a new cookie.\r\n\r\nIt makes it difficult to have an autoscraping API, as you can truly only run this locally. (Unless you came up with some automated way to renew the cookie, or use something like this: https://www.npmjs.com/package/puppeteer-extra-plugin-recaptcha to automatically bypass the captcha?)"}}
{"actor":{"display_login":"fraywing"}}