{"action":"closed","issue":{"url":"https://api.github.com/repos/lintool/warcbase/issues/244","repository_url":"https://api.github.com/repos/lintool/warcbase","labels_url":"https://api.github.com/repos/lintool/warcbase/issues/244/labels{/name}","comments_url":"https://api.github.com/repos/lintool/warcbase/issues/244/comments","events_url":"https://api.github.com/repos/lintool/warcbase/issues/244/events","html_url":"https://github.com/lintool/warcbase/issues/244","id":178177368,"number":244,"title":"java.util.zip.ZipException: invalid distance code","user":{"login":"ianmilligan1","id":3834704,"avatar_url":"https://avatars.githubusercontent.com/u/3834704?v=3","gravatar_id":"","url":"https://api.github.com/users/ianmilligan1","html_url":"https://github.com/ianmilligan1","followers_url":"https://api.github.com/users/ianmilligan1/followers","following_url":"https://api.github.com/users/ianmilligan1/following{/other_user}","gists_url":"https://api.github.com/users/ianmilligan1/gists{/gist_id}","starred_url":"https://api.github.com/users/ianmilligan1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianmilligan1/subscriptions","organizations_url":"https://api.github.com/users/ianmilligan1/orgs","repos_url":"https://api.github.com/users/ianmilligan1/repos","events_url":"https://api.github.com/users/ianmilligan1/events{/privacy}","received_events_url":"https://api.github.com/users/ianmilligan1/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/lintool/warcbase/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-09-20T20:53:40Z","updated_at":"2016-09-30T01:15:46Z","closed_at":"2016-09-30T01:15:46Z","body":"We've been trying to work with an Archive-It collection that keeps throwing this error. Have been trying to isolate the problematic ARC or WARC but to no avail. We are running on a single node, most recent build of Warcbase.\r\n\r\nAny clue what might be happening?\r\n\r\nScript we are running:\r\n```scala\r\nimport org.warcbase.spark.matchbox._ \r\nimport org.warcbase.spark.rdd.RecordRDD._ \r\nimport org.warcbase.spark.matchbox.{RemoveHTML, RecordLoader, ExtractBoilerpipeText}\r\n\r\nval labour = \r\n  RecordLoader.loadArchives(\"/data/TORONTO_canadian_labour_unions/*.gz\", sc) \r\n  .keepValidPages() \r\n  .map(r => (r.getCrawlMonth, ExtractDomain(r.getUrl))) \r\n  .countItems() \r\n  .saveAsTextFile(\"/data/derivatives/urls/TORONTO_canadian_labour_unions\")\r\n```\r\n\r\nSnippet of error message below:\r\n```\r\n2016-09-20 18:40:21,164 [Executor task launch worker-62] ERROR Executor - Exception in task 4367.0 in stage 0.0 (TID 4367)\r\njava.util.zip.ZipException: invalid distance code\r\n\tat org.archive.util.zip.OpenJDK7InflaterInputStream.read(OpenJDK7InflaterInputStream.java:168)\r\n\tat org.archive.util.zip.OpenJDK7GZIPInputStream.read(OpenJDK7GZIPInputStream.java:122)\r\n\tat org.archive.util.zip.GZIPMembersInputStream.read(GZIPMembersInputStream.java:113)\r\n\tat org.archive.io.ArchiveRecord.read(ArchiveRecord.java:204)\r\n\tat java.io.InputStream.read(InputStream.java:101)\r\n\tat org.warcbase.data.WarcRecordUtils.copyStream(WarcRecordUtils.java:133)\r\n\tat org.warcbase.data.WarcRecordUtils.getContent(WarcRecordUtils.java:103)\r\n\tat org.warcbase.spark.archive.io.GenericArchiveRecord.<init>(GenericArchiveRecord.scala:48)\r\n\tat org.warcbase.spark.matchbox.RecordLoader$$anonfun$loadArchives$2.apply(RecordLoader.scala:45)\r\n\tat org.warcbase.spark.matchbox.RecordLoader$$anonfun$loadArchives$2.apply(RecordLoader.scala:45)\r\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:328)\r\n\tat scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:189)\r\n\tat org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:89)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\n[Full error message can be found here](https://gist.github.com/ianmilligan1/f175ae3d5a266329dbc682df0ff3d90f)."}}
{"actor":{"display_login":"lintool"}}