{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2490","repository_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2490/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2490/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2490/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/2490","id":195299031,"number":2490,"title":"ParaVectors.inferVector has issues inside of an rdd","user":{"login":"dmmiller612","id":5598538,"avatar_url":"https://avatars.githubusercontent.com/u/5598538?v=3","gravatar_id":"","url":"https://api.github.com/users/dmmiller612","html_url":"https://github.com/dmmiller612","followers_url":"https://api.github.com/users/dmmiller612/followers","following_url":"https://api.github.com/users/dmmiller612/following{/other_user}","gists_url":"https://api.github.com/users/dmmiller612/gists{/gist_id}","starred_url":"https://api.github.com/users/dmmiller612/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmmiller612/subscriptions","organizations_url":"https://api.github.com/users/dmmiller612/orgs","repos_url":"https://api.github.com/users/dmmiller612/repos","events_url":"https://api.github.com/users/dmmiller612/events{/privacy}","received_events_url":"https://api.github.com/users/dmmiller612/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-12-13T16:23:33Z","updated_at":"2016-12-13T16:31:53Z","closed_at":null,"body":"I am running into an issue where ParagraphVector.inferVector is timing out inside of an rdd that has not had an operation performed on it yet (like collect, save, etc). First, I'll put the code here then explain the behavior.\r\n\r\nloading google model\r\n```\r\n    public static WordVectors loadGoogleModel(String bucketName) throws Exception {\r\n        S3Object s3Object = s3Client.getObject(bucketName, \"spark/data/google.bin\");\r\n        File tmpFile = File.createTempFile(\"temp\", \"txt\");\r\n        tmpFile.deleteOnExit();\r\n        FileUtils.copyInputStreamToFile(s3Object.getObjectContent(), tmpFile);\r\n        return WordVectorSerializer.loadGoogleModel(tmpFile, true);\r\n    }\r\n```\r\nconverting to paragraph vector\r\n```\r\n    public static ParagraphVectors convertWordVectorsToParagraphVector(WordVectors wordVectors) {\r\n        ParagraphVectors paragraphVectors = loadConfig(); // builder for paragraphVectors\r\n        paragraphVectors.setLookupTable(wordVectors.lookupTable());\r\n        paragraphVectors.setVocab(wordVectors.vocab());\r\n        return paragraphVectors;\r\n    }\r\n```\r\n\r\nThen in the main class:\r\n\r\n```\r\nfinal ParagraphVectors googleModel = loadGoogleModel(properties.getQualityBucket());\r\nJavaRDD<String> message = //a non null rdd of strings here;\r\nJavaRDD<INDArray> indArrayJavaRDD = rdd2.map(item -> googleModel.inferVector(item));\r\n//timeout failure\r\n```\r\n\r\nWhat is weird is when I run this in yarn on the emr, I see no errors in the log, but the process fails. Locally, I consistently receive this error:\r\n\r\n```\r\norg.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval\r\n        at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)\r\n        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63)\r\n        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)\r\n        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)\r\n        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)\r\n        at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)\r\n        at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:449)\r\n        at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:470)\r\n        at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470)\r\n        at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470)\r\n        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1765)\r\n        at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:470)\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)\r\n        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\r\n        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]\r\n        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)\r\n        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)\r\n        at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)\r\n        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)\r\n        at scala.concurrent.Await$.result(package.scala:107)\r\n        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)\r\n        ... 14 more\r\n```\r\n\r\nWhat is weird to me about this is that I am not even sure what an operation is being ran inside of an rdd. I am guessing it is because googleModel.inferVector is the driver, and needs to operate first before being passed down to an executor.\r\n\r\nThis is also using spark-submit for the emr and locally."},"comment":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/comments/266787617","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/2490#issuecomment-266787617","issue_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2490","id":266787617,"user":{"login":"raver119","id":12250879,"avatar_url":"https://avatars.githubusercontent.com/u/12250879?v=3","gravatar_id":"","url":"https://api.github.com/users/raver119","html_url":"https://github.com/raver119","followers_url":"https://api.github.com/users/raver119/followers","following_url":"https://api.github.com/users/raver119/following{/other_user}","gists_url":"https://api.github.com/users/raver119/gists{/gist_id}","starred_url":"https://api.github.com/users/raver119/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raver119/subscriptions","organizations_url":"https://api.github.com/users/raver119/orgs","repos_url":"https://api.github.com/users/raver119/repos","events_url":"https://api.github.com/users/raver119/events{/privacy}","received_events_url":"https://api.github.com/users/raver119/received_events","type":"User","site_admin":false},"created_at":"2016-12-13T16:31:53Z","updated_at":"2016-12-13T16:31:53Z","body":"Cool, please also don't forget to include actual paravec configuration, and model sharing mechanism you've used."}}
{"actor":{"display_login":"raver119"}}