{"action":"created","issue":{"url":"https://api.github.com/repos/dfdx/Spark.jl/issues/10","repository_url":"https://api.github.com/repos/dfdx/Spark.jl","labels_url":"https://api.github.com/repos/dfdx/Spark.jl/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/dfdx/Spark.jl/issues/10/comments","events_url":"https://api.github.com/repos/dfdx/Spark.jl/issues/10/events","html_url":"https://github.com/dfdx/Spark.jl/issues/10","id":181845126,"number":10,"title":"Trouble with parallelizing custom type","user":{"login":"peterjdolan","id":9718,"avatar_url":"https://avatars.githubusercontent.com/u/9718?v=3","gravatar_id":"","url":"https://api.github.com/users/peterjdolan","html_url":"https://github.com/peterjdolan","followers_url":"https://api.github.com/users/peterjdolan/followers","following_url":"https://api.github.com/users/peterjdolan/following{/other_user}","gists_url":"https://api.github.com/users/peterjdolan/gists{/gist_id}","starred_url":"https://api.github.com/users/peterjdolan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterjdolan/subscriptions","organizations_url":"https://api.github.com/users/peterjdolan/orgs","repos_url":"https://api.github.com/users/peterjdolan/repos","events_url":"https://api.github.com/users/peterjdolan/events{/privacy}","received_events_url":"https://api.github.com/users/peterjdolan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-08T19:47:20Z","updated_at":"2016-10-08T22:51:29Z","closed_at":null,"body":"This may or may not be a problem with Spark.jl, but I've been unable to determine what the source of the problem is.  As far as I can tell, if I define a new type, then parallelizing an array of that type fails.\r\n\r\nThis example reproduces the problem:\r\n\r\n`using Spark\r\n\r\ntype Custom\r\n    x::Integer\r\nend\r\n\r\nsc = SparkContext(master=\"local\")\r\nrdd = parallelize(sc, [Custom(1), Custom(2)])\r\nprintln(\"Count: \", count(rdd))\r\nclose(sc)`\r\n\r\nThis produces the following error:\r\n\r\n`\r\n16/10/08 12:44:18 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)\r\norg.apache.spark.SparkException: Julia worker exited unexpectedly (crashed)\r\n        at org.apache.spark.api.julia.InputIterator.read(InputIterator.scala:69)\r\n        at org.apache.spark.api.julia.InputIterator.<init>(InputIterator.scala:73)\r\n        at org.apache.spark.api.julia.JuliaRDD.compute(JuliaRDD.scala:37)\r\n        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)\r\n        at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:88)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.io.EOFException\r\n        at java.io.DataInputStream.readInt(DataInputStream.java:392)\r\n        at org.apache.spark.api.julia.InputIterator.read(InputIterator.scala:40)\r\n        ... 10 more\r\n16/10/08 12:44:18 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): org.apache.spark.SparkException: Julia worker exited unexpectedly (crashed)\r\n        at org.apache.spark.api.julia.InputIterator.read(InputIterator.scala:69)\r\n        at org.apache.spark.api.julia.InputIterator.<init>(InputIterator.scala:73)\r\n        at org.apache.spark.api.julia.JuliaRDD.compute(JuliaRDD.scala:37)\r\n        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)\r\n        at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:88)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.io.EOFException\r\n        at java.io.DataInputStream.readInt(DataInputStream.java:392)\r\n        at org.apache.spark.api.julia.InputIterator.read(InputIterator.scala:40)\r\n        ... 10 more\r\n`"},"comment":{"url":"https://api.github.com/repos/dfdx/Spark.jl/issues/comments/252452867","html_url":"https://github.com/dfdx/Spark.jl/issues/10#issuecomment-252452867","issue_url":"https://api.github.com/repos/dfdx/Spark.jl/issues/10","id":252452867,"user":{"login":"dfdx","id":428171,"avatar_url":"https://avatars.githubusercontent.com/u/428171?v=3","gravatar_id":"","url":"https://api.github.com/users/dfdx","html_url":"https://github.com/dfdx","followers_url":"https://api.github.com/users/dfdx/followers","following_url":"https://api.github.com/users/dfdx/following{/other_user}","gists_url":"https://api.github.com/users/dfdx/gists{/gist_id}","starred_url":"https://api.github.com/users/dfdx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dfdx/subscriptions","organizations_url":"https://api.github.com/users/dfdx/orgs","repos_url":"https://api.github.com/users/dfdx/repos","events_url":"https://api.github.com/users/dfdx/events{/privacy}","received_events_url":"https://api.github.com/users/dfdx/received_events","type":"User","site_admin":false},"created_at":"2016-10-08T22:51:29Z","updated_at":"2016-10-08T22:51:29Z","body":"Yes, this is expected. The easiest way to overcome it _right now_ is to use only types from lcdxisbrarie ffffffffffffffffffffffsfb vggggggggggggggg[-]=v"}}
{"actor":{"display_login":"dfdx"}}