{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/949","repository_url":"https://api.github.com/repos/deeplearning4j/nd4j","labels_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/949/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/949/comments","events_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/949/events","html_url":"https://github.com/deeplearning4j/nd4j/issues/949","id":155641332,"number":949,"title":"ND4J is not compatible with Spark's next-gen Kryo serializer (throws NPEs)","user":{"login":"crockpotveggies","id":1445295,"avatar_url":"https://avatars.githubusercontent.com/u/1445295?v=3","gravatar_id":"","url":"https://api.github.com/users/crockpotveggies","html_url":"https://github.com/crockpotveggies","followers_url":"https://api.github.com/users/crockpotveggies/followers","following_url":"https://api.github.com/users/crockpotveggies/following{/other_user}","gists_url":"https://api.github.com/users/crockpotveggies/gists{/gist_id}","starred_url":"https://api.github.com/users/crockpotveggies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crockpotveggies/subscriptions","organizations_url":"https://api.github.com/users/crockpotveggies/orgs","repos_url":"https://api.github.com/users/crockpotveggies/repos","events_url":"https://api.github.com/users/crockpotveggies/events{/privacy}","received_events_url":"https://api.github.com/users/crockpotveggies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-05-19T02:23:23Z","updated_at":"2016-05-21T06:26:47Z","closed_at":null,"body":"Upgraded to latest `3.10-SNAPSHOT` and getting this in my cluster:\r\n\r\n```\r\norg.apache.spark.SparkException: Job aborted due to stage failure: Task 38 in stage 0.0 failed 4 times, most recent failure: Lost task 38.3 in stage 0.0 (TID 64, slave6.cluster): java.lang.NullPointerException\r\n    at org.nd4j.linalg.api.shape.Shape.rank(Shape.java:1387)\r\n    at org.nd4j.linalg.api.ndarray.BaseNDArray.rank(BaseNDArray.java:3917)\r\n```\r\n\r\nFull stack trace: https://gist.github.com/crockpotveggies/75a0f78bbe33b024f214c17fbd53d22e\r\n\r\nI've already provided a dataset to @agibsonccc and a debug repo is available here: https://github.com/crockpotveggies/dl4j-full-stack-debug"},"comment":{"url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/comments/220761694","html_url":"https://github.com/deeplearning4j/nd4j/issues/949#issuecomment-220761694","issue_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/949","id":220761694,"user":{"login":"AlexDBlack","id":2360237,"avatar_url":"https://avatars.githubusercontent.com/u/2360237?v=3","gravatar_id":"","url":"https://api.github.com/users/AlexDBlack","html_url":"https://github.com/AlexDBlack","followers_url":"https://api.github.com/users/AlexDBlack/followers","following_url":"https://api.github.com/users/AlexDBlack/following{/other_user}","gists_url":"https://api.github.com/users/AlexDBlack/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexDBlack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexDBlack/subscriptions","organizations_url":"https://api.github.com/users/AlexDBlack/orgs","repos_url":"https://api.github.com/users/AlexDBlack/repos","events_url":"https://api.github.com/users/AlexDBlack/events{/privacy}","received_events_url":"https://api.github.com/users/AlexDBlack/received_events","type":"User","site_admin":false},"created_at":"2016-05-21T06:26:47Z","updated_at":"2016-05-21T06:26:47Z","body":"I don't think so, if we use the KryoSerializable interface - that has to be defined on the NDArray methods (I guess BaseNDArray).\r\n\r\nOnly other option I'm aware of (also doesn't require Kryo as a dependency) is to manually register things: \r\nfor example,\r\nkryo.register(INDArray.class, new JavaSerializer());    //Java-based serialization\r\nkryo.register(INDArray.class, new OurCustomSerializer());   //Custom serialization\r\n\r\nIn both cases, user has to remember to actually do this, or they will run into cryptic NPEs like the above. So I don't really like that option either (risks wasting a lot of time for users).\r\n\r\n"}}
{"actor":{"display_login":"AlexDBlack"}}