{"action":"created","issue":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/277","repository_url":"https://api.github.com/repos/rstudio/sparklyr","labels_url":"https://api.github.com/repos/rstudio/sparklyr/issues/277/labels{/name}","comments_url":"https://api.github.com/repos/rstudio/sparklyr/issues/277/comments","events_url":"https://api.github.com/repos/rstudio/sparklyr/issues/277/events","html_url":"https://github.com/rstudio/sparklyr/issues/277","id":184360226,"number":277,"title":"copy_to Error","user":{"login":"yarrick19","id":9707200,"avatar_url":"https://avatars.githubusercontent.com/u/9707200?v=3","gravatar_id":"","url":"https://api.github.com/users/yarrick19","html_url":"https://github.com/yarrick19","followers_url":"https://api.github.com/users/yarrick19/followers","following_url":"https://api.github.com/users/yarrick19/following{/other_user}","gists_url":"https://api.github.com/users/yarrick19/gists{/gist_id}","starred_url":"https://api.github.com/users/yarrick19/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yarrick19/subscriptions","organizations_url":"https://api.github.com/users/yarrick19/orgs","repos_url":"https://api.github.com/users/yarrick19/repos","events_url":"https://api.github.com/users/yarrick19/events{/privacy}","received_events_url":"https://api.github.com/users/yarrick19/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":25,"created_at":"2016-10-20T22:51:51Z","updated_at":"2016-11-04T02:12:38Z","closed_at":null,"body":"I'm receiving an error when running the `copy_to` command.  Below is the output from the R Console.\n\n```\n> library(sparklyr)\n> sc <- spark_connect(master = \"local\")\n> iris_tbl <- copy_to(sc, iris, overwrite = TRUE)\nError: java.lang.reflect.InvocationTargetException\n    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)\n    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)\n    at java.lang.reflect.Constructor.newInstance(Unknown Source)\n    at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258)\n    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359)\n    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263)\n    at org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39)\n    at org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38)\n    at org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:46)\n    at org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45)\n    at org.apa\n```\n\nBased on some other posts (issue 229), I also tried defining the spark.sql.warehouse.dir in the config file to no avail.\n\n```\n> spark_disconnect(sc)\n> config <- spark_config()\n> hadoopBin <- normalizePath(file.path(spark_home_dir(), \"tmp\", \"hadoop\", \"bin\"))\n> config[[\"spark.sql.warehouse.dir\"]] <- if (.Platform$OS.type == \"windows\") hadoopBin else NULL\n> sc <- spark_connect(master = \"local\",config = config)\n> iris_tbl <- copy_to(sc, iris, overwrite = TRUE)\nError: java.lang.reflect.InvocationTargetException\n    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)\n    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)\n    at java.lang.reflect.Constructor.newInstance(Unknown Source)\n    at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258)\n    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359)\n    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263)\n    at org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39)\n    at org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38)\n    at org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:46)\n    at org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45)\n    at org.apa\n```\n"},"comment":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/comments/258327905","html_url":"https://github.com/rstudio/sparklyr/issues/277#issuecomment-258327905","issue_url":"https://api.github.com/repos/rstudio/sparklyr/issues/277","id":258327905,"user":{"login":"alexwwang","id":856746,"avatar_url":"https://avatars.githubusercontent.com/u/856746?v=3","gravatar_id":"","url":"https://api.github.com/users/alexwwang","html_url":"https://github.com/alexwwang","followers_url":"https://api.github.com/users/alexwwang/followers","following_url":"https://api.github.com/users/alexwwang/following{/other_user}","gists_url":"https://api.github.com/users/alexwwang/gists{/gist_id}","starred_url":"https://api.github.com/users/alexwwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexwwang/subscriptions","organizations_url":"https://api.github.com/users/alexwwang/orgs","repos_url":"https://api.github.com/users/alexwwang/repos","events_url":"https://api.github.com/users/alexwwang/events{/privacy}","received_events_url":"https://api.github.com/users/alexwwang/received_events","type":"User","site_admin":false},"created_at":"2016-11-04T02:12:38Z","updated_at":"2016-11-04T02:12:38Z","body":"I've solved this problem. First I don't know should start a slave worker in master mode and solved this, I happened to find some error items in log which hinted me the access of database metastore_db was not proper and I try to rm this fold in $spark_work_path, default on linux is under your home fold. Then I retried the connection and copy_to, everything works well. Remember to export some system var in your $spark_home/conf/spark-env.sh as follows:\r\n```hadoop_home=/usr/local/hadoop\r\nhadoop_user_name=hadoop # if you use this account launch the hadoop and spark\r\nhadoop_conf_dir=$hadoop_home/etc/hadoop # where hadoop configuration locates\r\nld_library_path=$hadoop_home/lib/native # solve the warn poped when starting spark saying can't find native jars library, not sure if necessary.```\r\nYou should capitalize the system var names, i think, in practice.\r\n\r\nand during the connection in R, I made config items like below:\r\n```config <- spark_config()\r\nconfig[[\"spark.sql.hive.thriftServer.singleSession\"]] <- \"true\"```\r\nas mentioned above."}}
{"actor":{"display_login":"alexwwang"}}