{"action":"created","issue":{"url":"https://api.github.com/repos/web-archive-group/WALK/issues/12","repository_url":"https://api.github.com/repos/web-archive-group/WALK","labels_url":"https://api.github.com/repos/web-archive-group/WALK/issues/12/labels{/name}","comments_url":"https://api.github.com/repos/web-archive-group/WALK/issues/12/comments","events_url":"https://api.github.com/repos/web-archive-group/WALK/issues/12/events","html_url":"https://github.com/web-archive-group/WALK/issues/12","id":151854025,"number":12,"title":"Failures in Various Derivative Generations","user":{"login":"ianmilligan1","id":3834704,"avatar_url":"https://avatars.githubusercontent.com/u/3834704?v=3","gravatar_id":"","url":"https://api.github.com/users/ianmilligan1","html_url":"https://github.com/ianmilligan1","followers_url":"https://api.github.com/users/ianmilligan1/followers","following_url":"https://api.github.com/users/ianmilligan1/following{/other_user}","gists_url":"https://api.github.com/users/ianmilligan1/gists{/gist_id}","starred_url":"https://api.github.com/users/ianmilligan1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianmilligan1/subscriptions","organizations_url":"https://api.github.com/users/ianmilligan1/orgs","repos_url":"https://api.github.com/users/ianmilligan1/repos","events_url":"https://api.github.com/users/ianmilligan1/events{/privacy}","received_events_url":"https://api.github.com/users/ianmilligan1/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/web-archive-group/WALK/labels/derivative%20generation","name":"derivative generation","color":"d93f0b"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-04-29T12:23:47Z","updated_at":"2016-07-04T14:12:28Z","closed_at":null,"body":"I've gone through and extracted URLs and link networks for all the collections. However, we've had quite a few failures ([you can read about all the fun stuff over in this warcbase issue](https://github.com/lintool/warcbase/issues/222)). \r\n\r\nThe following collections **have both URLs and networks**:\r\n```\r\nalberta_education_curriculum/\r\nalberta_floods_2013/\r\nalberta_oil_sands/\r\ncanadian_business_grey_literature/\r\nelxn42/\r\nenergy_environment/\r\nhcf_alberta_online_encyclopedia/\r\nhealth_sciences_grey_literature/\r\nheritage_community_foundation/\r\nhumanities_computing/\r\nlfrancophonie_de_louest_canadien/\r\nottawa_shooting_october_2014/\r\nprarie_provinces/\r\nweb_archive_general/\r\n```\r\n\r\nThe following collections **have only URLs, no networks** - IdleNoMore failed twice:\r\n\r\n```\r\nalberta_education_curriculum/\r\nalberta_floods_2013/\r\nalberta_oil_sands/\r\ncanadian_business_grey_literature/\r\nelxn42/\r\nenergy_environment/\r\nhcf_alberta_online_encyclopedia/\r\nhealth_sciences_grey_literature/\r\nheritage_community_foundation/\r\nhumanities_computing/\r\nidle_no_more/\r\nlfrancophonie_de_louest_canadien/\r\nottawa_shooting_october_2014/\r\nprarie_provinces/\r\nweb_archive_general/\r\n```\r\n\r\nThe following collections **have no derivatives generated yet**:\r\n\r\n```\r\nalberta_education_curriculum_francais/\r\ncircumpolar/\r\ngovernment_information/\r\nuniversity_of_alberta_websites/\r\n```"},"comment":{"url":"https://api.github.com/repos/web-archive-group/WALK/issues/comments/230299708","html_url":"https://github.com/web-archive-group/WALK/issues/12#issuecomment-230299708","issue_url":"https://api.github.com/repos/web-archive-group/WALK/issues/12","id":230299708,"user":{"login":"ianmilligan1","id":3834704,"avatar_url":"https://avatars.githubusercontent.com/u/3834704?v=3","gravatar_id":"","url":"https://api.github.com/users/ianmilligan1","html_url":"https://github.com/ianmilligan1","followers_url":"https://api.github.com/users/ianmilligan1/followers","following_url":"https://api.github.com/users/ianmilligan1/following{/other_user}","gists_url":"https://api.github.com/users/ianmilligan1/gists{/gist_id}","starred_url":"https://api.github.com/users/ianmilligan1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianmilligan1/subscriptions","organizations_url":"https://api.github.com/users/ianmilligan1/orgs","repos_url":"https://api.github.com/users/ianmilligan1/repos","events_url":"https://api.github.com/users/ianmilligan1/events{/privacy}","received_events_url":"https://api.github.com/users/ianmilligan1/received_events","type":"User","site_admin":false},"created_at":"2016-07-04T14:12:28Z","updated_at":"2016-07-04T14:12:28Z","body":"Running\r\n\r\n```\r\nimport org.warcbase.spark.matchbox._ \r\nimport org.warcbase.spark.rdd.RecordRDD._ \r\n\r\nval circumpolar = \r\n  RecordLoader.loadArchives(\"/data/circumpolar/*.gz\", sc) \r\n  .keepValidPages() \r\n  .map(r => (r.getCrawlMonth, ExtractDomain(r.getUrl))) \r\n  .countItems() \r\n  .saveAsTextFile(\"/data/derivatives/urls/circumpolar\")\r\n```\r\n\r\nwill error log once it crashes, even with 300gb swap file in place."}}
{"actor":{"display_login":"ianmilligan1"}}