{"action":"closed","issue":{"url":"https://api.github.com/repos/orientechnologies/spark-orientdb/issues/13","repository_url":"https://api.github.com/repos/orientechnologies/spark-orientdb","labels_url":"https://api.github.com/repos/orientechnologies/spark-orientdb/issues/13/labels{/name}","comments_url":"https://api.github.com/repos/orientechnologies/spark-orientdb/issues/13/comments","events_url":"https://api.github.com/repos/orientechnologies/spark-orientdb/issues/13/events","html_url":"https://github.com/orientechnologies/spark-orientdb/issues/13","id":295406042,"number":13,"title":"Unable to leverage parallelism from spark to Orientdb","user":{"login":"hilesha","id":1671936,"avatar_url":"https://avatars2.githubusercontent.com/u/1671936?v=4","gravatar_id":"","url":"https://api.github.com/users/hilesha","html_url":"https://github.com/hilesha","followers_url":"https://api.github.com/users/hilesha/followers","following_url":"https://api.github.com/users/hilesha/following{/other_user}","gists_url":"https://api.github.com/users/hilesha/gists{/gist_id}","starred_url":"https://api.github.com/users/hilesha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hilesha/subscriptions","organizations_url":"https://api.github.com/users/hilesha/orgs","repos_url":"https://api.github.com/users/hilesha/repos","events_url":"https://api.github.com/users/hilesha/events{/privacy}","received_events_url":"https://api.github.com/users/hilesha/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-02-08T07:05:11Z","updated_at":"2018-03-08T23:45:17Z","closed_at":"2018-03-08T23:45:17Z","author_association":"NONE","body":"Hi,\r\n\r\ni was loading csv data (600 MB) into orientdb(2 node distributed cluster) into a class using following command\r\n\r\ndf.write.format(\"org.apache.spark.orientdb.documents\").option(\"dburl\",\"remote:localhost:2424/mydb\").option(\"class\", \"sales\").option(\"user\",\"admin\").option(\"password\",\"admin\").save()\r\n\r\ndf is loaded with a csv having following schema using this command\r\nval df = sqlContext.read.option(\"header\",\"TRUE\").schema(customSchema).csv(\"path of csv file\")\r\n\r\nval customSchema = StructType(Array(\r\nStructField(\"key\", LongType, true),\r\nStructField(\"date\", DateType, true),\r\nStructField(\"code\", IntegerType, true),\r\nStructField(\"quantity\", IntegerType, true),\r\nStructField(\"revenue\", DoubleType, true),\r\nStructField(\"count_flag\", StringType, true),\r\nStructField(\"updated_on\", DateType, true)))\r\n\r\nwhen i look into the cluster list which are created inside of the class \"sales\", it creates 8 clusters but unfortunately it is trying to write into a single default cluster(sales)  instead of all cluster(sales_1 to sales_7) which is becoming a bottleneck.\r\n\r\ni tried to modify my save command by adding cluster option, still the api gives me single cluster specification and not a way to add data on all cluster.\r\n\r\ndf.write.format(\"org.apache.spark.orientdb.documents\").option(\"dburl\",\"remote:localhost:2424/mydb\").option(\"class\", \"sales\").option(\"user\",\"admin\").option(\"password\",\"admin\").option(\"cluster\",\"sales_1\").save()\r\n\r\nis this a limitation of spark connector? i would appreciate the inputs from committers .\r\n"}}
{"actor":{"display_login":"sbcd90"}}