{"action":"created","issue":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/367","repository_url":"https://api.github.com/repos/rstudio/sparklyr","labels_url":"https://api.github.com/repos/rstudio/sparklyr/issues/367/labels{/name}","comments_url":"https://api.github.com/repos/rstudio/sparklyr/issues/367/comments","events_url":"https://api.github.com/repos/rstudio/sparklyr/issues/367/events","html_url":"https://github.com/rstudio/sparklyr/issues/367","id":193880119,"number":367,"title":"Cannot connect to livy on EMR cluster using spark 2.0.1/livy3.0/scala 2.10 ","user":{"login":"fereshtehRS","id":4906984,"avatar_url":"https://avatars.githubusercontent.com/u/4906984?v=3","gravatar_id":"","url":"https://api.github.com/users/fereshtehRS","html_url":"https://github.com/fereshtehRS","followers_url":"https://api.github.com/users/fereshtehRS/followers","following_url":"https://api.github.com/users/fereshtehRS/following{/other_user}","gists_url":"https://api.github.com/users/fereshtehRS/gists{/gist_id}","starred_url":"https://api.github.com/users/fereshtehRS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fereshtehRS/subscriptions","organizations_url":"https://api.github.com/users/fereshtehRS/orgs","repos_url":"https://api.github.com/users/fereshtehRS/repos","events_url":"https://api.github.com/users/fereshtehRS/events{/privacy}","received_events_url":"https://api.github.com/users/fereshtehRS/received_events","type":"User","site_admin":false},"labels":[{"id":401558262,"url":"https://api.github.com/repos/rstudio/sparklyr/labels/connection","name":"connection","color":"bfdadc","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-12-06T20:50:28Z","updated_at":"2016-12-08T16:40:17Z","closed_at":"2016-12-08T00:41:55Z","body":"- On the EMR clusters the version of `Spark` is 2.0.1 and it uses `scala` 2.11\r\n- When we install `livy` 3.0, it installs a version that uses `scala` 2.10\r\n- This seems to be causing an issue with the connection from `sparklyr` to `livy`. I was getting error on some classes not found. (full stack below).\r\n- The only useful reference to the error that I found is this one:\r\n\r\n`https://groups.google.com/a/cloudera.org/forum/#!searchin/livy-user/AbstractPartialFunction$24mcVL$24sp%7Csort:relevance/livy-user/jCE07EGnhrU/0VDc50YmBAAJ`\r\n\r\nwhich point to `scala` version.\r\n- I built `livy` from source with `scala` 2.11, and that got me past the error. \r\n\r\nMay be other variables were involved here that I am not aware of.\r\n\r\n\r\nStack trace:\r\n\r\n```\r\n16/12/06 17:07:53 INFO ContextLauncher: 16/12/06 17:07:53 INFO RSCDriver: SparkContext not yet up, queueing job request.\r\n16/12/06 17:07:53 INFO ContextLauncher: Exception in thread \"main\" java.lang.NoClassDefFoundError: scala/runtime/AbstractPartialFunction$mcVL$sp\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.lang.ClassLoader.defineClass1(Native Method)\r\n16/12/06 17:07:53 WARN RSCClient: Client RPC channel closed unexpectedly.\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.lang.ClassLoader.defineClass(ClassLoader.java:763)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.net.URLClassLoader.defineClass(URLClassLoader.java:467)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.net.URLClassLoader.access$100(URLClassLoader.java:73)\r\n16/12/06 17:07:53 INFO RSCClient: Failing pending job 0819b78a-d993-4582-a934-9a46fc513261 due to shutdown.\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.net.URLClassLoader$1.run(URLClassLoader.java:368)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.net.URLClassLoader$1.run(URLClassLoader.java:362)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.security.AccessController.doPrivileged(Native Method)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.net.URLClassLoader.findClass(URLClassLoader.java:361)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.lang.ClassLoader.loadClass(ClassLoader.java:424)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.lang.ClassLoader.loadClass(ClassLoader.java:357)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat com.cloudera.livy.repl.Session.start(Session.scala:63)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat com.cloudera.livy.repl.ReplDriver.initializeContext(ReplDriver.scala:55)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat com.cloudera.livy.rsc.driver.RSCDriver.run(RSCDriver.java:300)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat com.cloudera.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:85)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.lang.reflect.Method.invoke(Method.java:498)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)\r\n16/12/06 17:07:53 INFO ContextLauncher: Caused by: java.lang.ClassNotFoundException: scala.runtime.AbstractPartialFunction$mcVL$sp\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.net.URLClassLoader.findClass(URLClassLoader.java:381)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.lang.ClassLoader.loadClass(ClassLoader.java:424)\r\n16/12/06 17:07:53 INFO ContextLauncher: \tat java.lang.ClassLoader.loadClass(ClassLoader.java:357)\r\n16/12/06 17:07:53 INFO ContextLauncher: \t... 24 more\r\n16/12/06 17:07:54 INFO ContextLauncher: Exception in thread \"pool-2-thread-1\" java.lang.NoClassDefFoundError: org/apache/spark/repl/SparkIMain\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat com.cloudera.livy.repl.SparkInterpreter.start(SparkInterpreter.scala:60)\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat com.cloudera.livy.repl.Session$$anonfun$1.apply(Session.scala:59)\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat com.cloudera.livy.repl.Session$$anonfun$1.apply(Session.scala:57)\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat java.lang.Thread.run(Thread.java:745)\r\n16/12/06 17:07:54 INFO ContextLauncher: Caused by: java.lang.ClassNotFoundException: org.apache.spark.repl.SparkIMain\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat java.net.URLClassLoader.findClass(URLClassLoader.java:381)\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat java.lang.ClassLoader.loadClass(ClassLoader.java:424)\r\n16/12/06 17:07:54 INFO ContextLauncher: \tat java.lang.ClassLoader.loadClass(ClassLoader.java:357)\r\n16/12/06 17:07:54 INFO ContextLauncher: \t... 8 more\r\n16/12/06 17:07:54 WARN ContextLauncher: Child process exited with code 1.\r\n```"},"comment":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/comments/265787848","html_url":"https://github.com/rstudio/sparklyr/issues/367#issuecomment-265787848","issue_url":"https://api.github.com/repos/rstudio/sparklyr/issues/367","id":265787848,"user":{"login":"fereshtehRS","id":4906984,"avatar_url":"https://avatars.githubusercontent.com/u/4906984?v=3","gravatar_id":"","url":"https://api.github.com/users/fereshtehRS","html_url":"https://github.com/fereshtehRS","followers_url":"https://api.github.com/users/fereshtehRS/followers","following_url":"https://api.github.com/users/fereshtehRS/following{/other_user}","gists_url":"https://api.github.com/users/fereshtehRS/gists{/gist_id}","starred_url":"https://api.github.com/users/fereshtehRS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fereshtehRS/subscriptions","organizations_url":"https://api.github.com/users/fereshtehRS/orgs","repos_url":"https://api.github.com/users/fereshtehRS/repos","events_url":"https://api.github.com/users/fereshtehRS/events{/privacy}","received_events_url":"https://api.github.com/users/fereshtehRS/received_events","type":"User","site_admin":false},"created_at":"2016-12-08T16:40:17Z","updated_at":"2016-12-08T16:40:17Z","body":"@javierluraschi I tried again with sparklyr 0.4.39 and the same issue when we use Livy that comes with the package. But using the version of Livy that I build works fine. Not sure what other people have experienced (@edgararuiz). "}}
{"actor":{"display_login":"fereshtehRS"}}