{"action":"created","issue":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/277","repository_url":"https://api.github.com/repos/rstudio/sparklyr","labels_url":"https://api.github.com/repos/rstudio/sparklyr/issues/277/labels{/name}","comments_url":"https://api.github.com/repos/rstudio/sparklyr/issues/277/comments","events_url":"https://api.github.com/repos/rstudio/sparklyr/issues/277/events","html_url":"https://github.com/rstudio/sparklyr/issues/277","id":184360226,"number":277,"title":"copy_to Error","user":{"login":"yarrick19","id":9707200,"avatar_url":"https://avatars.githubusercontent.com/u/9707200?v=3","gravatar_id":"","url":"https://api.github.com/users/yarrick19","html_url":"https://github.com/yarrick19","followers_url":"https://api.github.com/users/yarrick19/followers","following_url":"https://api.github.com/users/yarrick19/following{/other_user}","gists_url":"https://api.github.com/users/yarrick19/gists{/gist_id}","starred_url":"https://api.github.com/users/yarrick19/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yarrick19/subscriptions","organizations_url":"https://api.github.com/users/yarrick19/orgs","repos_url":"https://api.github.com/users/yarrick19/repos","events_url":"https://api.github.com/users/yarrick19/events{/privacy}","received_events_url":"https://api.github.com/users/yarrick19/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2016-10-20T22:51:51Z","updated_at":"2016-10-27T18:43:49Z","closed_at":null,"body":"I'm receiving an error when running the `copy_to` command.  Below is the output from the R Console.\r\n\r\n```\r\n> library(sparklyr)\r\n> sc <- spark_connect(master = \"local\")\r\n> iris_tbl <- copy_to(sc, iris, overwrite = TRUE)\r\nError: java.lang.reflect.InvocationTargetException\r\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\r\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)\r\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)\r\n\tat java.lang.reflect.Constructor.newInstance(Unknown Source)\r\n\tat org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258)\r\n\tat org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359)\r\n\tat org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263)\r\n\tat org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39)\r\n\tat org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38)\r\n\tat org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:46)\r\n\tat org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45)\r\n\tat org.apa\r\n```\r\n\r\nBased on some other posts (issue 229), I also tried defining the spark.sql.warehouse.dir in the config file to no avail.\r\n\r\n```\r\n> spark_disconnect(sc)\r\n> config <- spark_config()\r\n> hadoopBin <- normalizePath(file.path(spark_home_dir(), \"tmp\", \"hadoop\", \"bin\"))\r\n> config[[\"spark.sql.warehouse.dir\"]] <- if (.Platform$OS.type == \"windows\") hadoopBin else NULL\r\n> sc <- spark_connect(master = \"local\",config = config)\r\n> iris_tbl <- copy_to(sc, iris, overwrite = TRUE)\r\nError: java.lang.reflect.InvocationTargetException\r\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\r\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)\r\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)\r\n\tat java.lang.reflect.Constructor.newInstance(Unknown Source)\r\n\tat org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258)\r\n\tat org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359)\r\n\tat org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263)\r\n\tat org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39)\r\n\tat org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38)\r\n\tat org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:46)\r\n\tat org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45)\r\n\tat org.apa\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/comments/256733363","html_url":"https://github.com/rstudio/sparklyr/issues/277#issuecomment-256733363","issue_url":"https://api.github.com/repos/rstudio/sparklyr/issues/277","id":256733363,"user":{"login":"yarrick19","id":9707200,"avatar_url":"https://avatars.githubusercontent.com/u/9707200?v=3","gravatar_id":"","url":"https://api.github.com/users/yarrick19","html_url":"https://github.com/yarrick19","followers_url":"https://api.github.com/users/yarrick19/followers","following_url":"https://api.github.com/users/yarrick19/following{/other_user}","gists_url":"https://api.github.com/users/yarrick19/gists{/gist_id}","starred_url":"https://api.github.com/users/yarrick19/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yarrick19/subscriptions","organizations_url":"https://api.github.com/users/yarrick19/orgs","repos_url":"https://api.github.com/users/yarrick19/repos","events_url":"https://api.github.com/users/yarrick19/events{/privacy}","received_events_url":"https://api.github.com/users/yarrick19/received_events","type":"User","site_admin":false},"created_at":"2016-10-27T18:43:49Z","updated_at":"2016-10-27T18:43:49Z","body":"@javierluraschi @kevinushey @trinker \r\n\r\n**I figured it out!**  As I mentioned before, I wanted to try and see the entire java error message instead of the truncated error message that shows up in the R console.  So, I went to `spark-shell.cmd` and launched spark from there and attempted to load data into a hive table.  \r\n\r\nI ran the following:\r\n```\r\nscala> val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)\r\nscala> sqlContext.sql(“CREATE TABLE IF NOT EXISTS test (name STRING, rank INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\t' LINES TERMINATED BY '\\n'”)\r\n```\r\nWhen running the create table statement I received a **full** java error.  Part of the error said\r\n```\r\nThe root scratch dir: C:\\Users\\pnieman\\AppData\\Local\\rstudio\\spark\\Cache\\spark-2.0.0-bin-hadoop2.7\\tmp\\hive on HDFS should be writable. Current permissions are: rw-rw-rw- \r\n```\r\nMany solutions for this are to grant 777 access from the `winutils.exe` file to the `/tmp/hive` directory.  I tried doing this several ways and still received the error.  I then stumbled upon this command which shows the permissions that are granted:\r\n```\r\nC:\\Users\\pnieman\\AppData\\Local\\rstudio\\spark\\Cache\\spark-2.0.0-bin-hadoop2.7\\tmp\\hadoop\\bin\\winutils.exe ls C:\\Users\\pnieman\\AppData\\Local\\rstudio\\spark\\Cache\\spark-2.0.0-bin-hadoop2.7\\tmp\\hive\r\n```\r\n\r\nRunning this generated an error saying `The trust relationship between this workstation and the primary domain failed.`  It appears my computer had lost connection to our domain and that was creating downstream permissions issues.  I had to leave and rejoin the domain.  After that, the permissions error went away and I can successfully run the `copy_to()` command.\r\n\r\nAfter working through this (and all of us spending a decent amount of time on it), my suggestion for `sparklyr` is to output or log the full error message that is generated so it's easier to track this stuff down."}}
{"actor":{"display_login":"yarrick19"}}