{"action":"opened","issue":{"url":"https://api.github.com/repos/pacharanero/google_group.to_discourse/issues/3","repository_url":"https://api.github.com/repos/pacharanero/google_group.to_discourse","labels_url":"https://api.github.com/repos/pacharanero/google_group.to_discourse/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/pacharanero/google_group.to_discourse/issues/3/comments","events_url":"https://api.github.com/repos/pacharanero/google_group.to_discourse/issues/3/events","html_url":"https://github.com/pacharanero/google_group.to_discourse/issues/3","id":196990173,"number":3,"title":"Scrapper running forever, not generating tables","user":{"login":"spacewaffle","id":1930218,"avatar_url":"https://avatars.githubusercontent.com/u/1930218?v=3","gravatar_id":"","url":"https://api.github.com/users/spacewaffle","html_url":"https://github.com/spacewaffle","followers_url":"https://api.github.com/users/spacewaffle/followers","following_url":"https://api.github.com/users/spacewaffle/following{/other_user}","gists_url":"https://api.github.com/users/spacewaffle/gists{/gist_id}","starred_url":"https://api.github.com/users/spacewaffle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spacewaffle/subscriptions","organizations_url":"https://api.github.com/users/spacewaffle/orgs","repos_url":"https://api.github.com/users/spacewaffle/repos","events_url":"https://api.github.com/users/spacewaffle/events{/privacy}","received_events_url":"https://api.github.com/users/spacewaffle/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-21T17:33:22Z","updated_at":"2016-12-21T17:33:22Z","closed_at":null,"body":"So I've got the cookie setup and I've actually successfully used the scrapper before. I've been trying to replicate what I did before but I'm running into some issues. The scrapper will run forever with logs like below. \r\n\r\n```\r\n2016-12-21 12:26:09 (1.27 MB/s) - written to stdout [60251]\r\n\r\n:: Creating './workbarbiz//threads/t.129' with 'forum/workbarbiz'\r\n:: Fetching data from 'https://groups.google.com/forum/?_escaped_fragment_=forum/workbarbiz'...\r\n--2016-12-21 12:26:09--  https://groups.google.com/forum/?_escaped_fragment_=forum/workbarbiz\r\nResolving groups.google.com... 74.125.192.139, 74.125.192.101, 74.125.192.138, ...\r\nConnecting to groups.google.com|74.125.192.139|:443... connected.\r\nHTTP request sent, awaiting response... 302 Moved Temporarily\r\nLocation: https://accounts.google.com/ServiceLogin?service=groups2&passive=1209600&continue=https://groups.google.com/forum/?_escaped_fragment_%3Dforum/workbarbiz&followup=https://groups.google.com/forum/?_escaped_fragment_%3Dforum/workbarbiz&authuser=1 [following]\r\n--2016-12-21 12:26:09--  https://accounts.google.com/ServiceLogin?service=groups2&passive=1209600&continue=https://groups.google.com/forum/?_escaped_fragment_%3Dforum/workbarbiz&followup=https://groups.google.com/forum/?_escaped_fragment_%3Dforum/workbarbiz&authuser=1\r\nResolving accounts.google.com... 216.58.219.237, 2607:f8b0:4006:80b::200d\r\nConnecting to accounts.google.com|216.58.219.237|:443... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: unspecified [text/html]\r\nSaving to: ‘STDOUT’\r\n\r\n-                                       [ <=>                                                             ]  58.68K  --.-KB/s    in 0.04s   \r\n\r\n2016-12-21 12:26:09 (1.39 MB/s) - written to stdout [60091]\r\n\r\n:: Creating './workbarbiz//threads/t.130' with 'forum/workbarbiz'\r\n:: Fetching data from 'https://groups.google.com/forum/?_escaped_fragment_=forum/workbarbiz'...\r\n--2016-12-21 12:26:09--  https://groups.google.com/forum/?_escaped_fragment_=forum/workbarbiz\r\nResolving groups.google.com... 74.125.192.139, 74.125.192.101, 74.125.192.138, ...\r\nConnecting to groups.google.com|74.125.192.139|:443... connected.\r\nHTTP request sent, awaiting response... 302 Moved Temporarily\r\nLocation: https://accounts.google.com/ServiceLogin?service=groups2&passive=1209600&continue=https://groups.google.com/forum/?_escaped_fragment_%3Dforum/workbarbiz&followup=https://groups.google.com/forum/?_escaped_fragment_%3Dforum/workbarbiz&authuser=1 [following]\r\n--2016-12-21 12:26:09--  https://accounts.google.com/ServiceLogin?service=groups2&passive=1209600&continue=https://groups.google.com/forum/?_escaped_fragment_%3Dforum/workbarbiz&followup=https://groups.google.com/forum/?_escaped_fragment_%3Dforum/workbarbiz&authuser=1\r\nResolving accounts.google.com... 216.58.219.237, 2607:f8b0:4006:80b::200d\r\nConnecting to accounts.google.com|216.58.219.237|:443... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: unspecified [text/html]\r\nSaving to: ‘STDOUT’\r\n\r\n```\r\n\r\nI eventually killed the process because it was taking multiple hours even though our google groups forum doesn't have that many posts. I found that there were thousands of thread files generated but nothing in msgs, nothing in mbox, and no db file generated after scraping. Every thread file had the same single line of text:\r\n\r\n`https://groups.google.com/forum/?_escaped_fragment_=forum/workbarbiz\r\n`\r\n\r\nAny idea what's going on here?"}}
{"actor":{"display_login":"spacewaffle"}}