{"action":"created","issue":{"url":"https://api.github.com/repos/lintool/warcbase/issues/246","repository_url":"https://api.github.com/repos/lintool/warcbase","labels_url":"https://api.github.com/repos/lintool/warcbase/issues/246/labels{/name}","comments_url":"https://api.github.com/repos/lintool/warcbase/issues/246/comments","events_url":"https://api.github.com/repos/lintool/warcbase/issues/246/events","html_url":"https://github.com/lintool/warcbase/issues/246","id":178825151,"number":246,"title":"java.lang.OutOfMemoryError: Java heap space","user":{"login":"dportabella","id":976546,"avatar_url":"https://avatars.githubusercontent.com/u/976546?v=3","gravatar_id":"","url":"https://api.github.com/users/dportabella","html_url":"https://github.com/dportabella","followers_url":"https://api.github.com/users/dportabella/followers","following_url":"https://api.github.com/users/dportabella/following{/other_user}","gists_url":"https://api.github.com/users/dportabella/gists{/gist_id}","starred_url":"https://api.github.com/users/dportabella/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dportabella/subscriptions","organizations_url":"https://api.github.com/users/dportabella/orgs","repos_url":"https://api.github.com/users/dportabella/repos","events_url":"https://api.github.com/users/dportabella/events{/privacy}","received_events_url":"https://api.github.com/users/dportabella/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-09-23T09:15:51Z","updated_at":"2016-09-23T18:46:17Z","closed_at":null,"body":"I had memory problems running my program,\r\nand I see that I cannot even run this very simple example:\r\n\r\n```\r\npackage application\r\n\r\nimport org.apache.spark._\r\nimport org.warcbase.spark.matchbox.RecordLoader\r\nimport org.warcbase.spark.rdd.RecordRDD._\r\n\r\nobject Test {\r\n  def main(args: Array[String]) \r\n    val in = if (args.length > 0) args(0) else \"/data/sample.warc.gz\"\r\n    val conf = new SparkConf().setAppName(\"Test\")\r\n    val spark = new SparkContext(conf)\r\n\r\n    val r = RecordLoader.loadArchives(in, spark)\r\n      .keepValidPages()\r\n      .count()\r\n\r\n    println(s\"result: $r\")\r\n\r\n    spark.stop()\r\n  }\r\n}\r\n```\r\n\r\nI am running this on my local machine:\r\n\r\n`$ spark-submit --executor-memory 6g --master local[2] --class application.Test target/scala-2.10/test-assembly-0.1-SNAPSHOT.jar /data/sample.warc.gz\r\n`\r\n\r\nand I get a `java.lang.OutOfMemoryError: Java heap space`\r\n\r\nI thought that spark would take care of the memory, swapping to disk when necessary, and run ok as long as there is enough HD space (even if the input file is 1 Petabyte).\r\n\r\nWhy do I get an `OutOfMemoryError`?\r\n\r\nDoes `RecordLoader.loadArchives` load everything in memory?\r\nHow can I solve this problem?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/lintool/warcbase/issues/comments/249272263","html_url":"https://github.com/lintool/warcbase/issues/246#issuecomment-249272263","issue_url":"https://api.github.com/repos/lintool/warcbase/issues/246","id":249272263,"user":{"login":"ianmilligan1","id":3834704,"avatar_url":"https://avatars.githubusercontent.com/u/3834704?v=3","gravatar_id":"","url":"https://api.github.com/users/ianmilligan1","html_url":"https://github.com/ianmilligan1","followers_url":"https://api.github.com/users/ianmilligan1/followers","following_url":"https://api.github.com/users/ianmilligan1/following{/other_user}","gists_url":"https://api.github.com/users/ianmilligan1/gists{/gist_id}","starred_url":"https://api.github.com/users/ianmilligan1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianmilligan1/subscriptions","organizations_url":"https://api.github.com/users/ianmilligan1/orgs","repos_url":"https://api.github.com/users/ianmilligan1/repos","events_url":"https://api.github.com/users/ianmilligan1/events{/privacy}","received_events_url":"https://api.github.com/users/ianmilligan1/received_events","type":"User","site_admin":false},"created_at":"2016-09-23T18:46:17Z","updated_at":"2016-09-23T18:46:17Z","body":"Does your application work if you launch it via `spark-shell`, along lines of on a local machine:\r\n\r\n```bash\r\n/home/i2millig/spark-1.5.1/bin/spark-shell --driver-memory 6G --jars ~/warcbase/warcbase-core/target/warcbase-core-0.1.0-SNAPSHOT-fatjar.jar\r\n```\r\n\r\nTo launch, and then `:paste:` your script in. \r\n\r\nAnd does this simple script work (with `loadArchives` path changed accordingly):\r\n\r\n```scala\r\nimport org.warcbase.spark.matchbox._\r\nimport org.warcbase.spark.rdd.RecordRDD._\r\n\r\nval r = RecordLoader.loadArchives(\"warcbase-core/src/test/resources/arc/example.arc.gz\", sc)\r\n  .keepValidPages()\r\n  .map(r => ExtractDomain(r.getUrl))\r\n  .countItems()\r\n  .take(10)\r\n```\r\n\r\nJust curious where the error is. "}}
{"actor":{"display_login":"ianmilligan1"}}