{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/868","repository_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/868/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/868/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/868/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/868","id":182544832,"number":868,"title":"Spark SQL support broken in rc1","user":{"login":"bwf93","id":7559095,"avatar_url":"https://avatars.githubusercontent.com/u/7559095?v=3","gravatar_id":"","url":"https://api.github.com/users/bwf93","html_url":"https://github.com/bwf93","followers_url":"https://api.github.com/users/bwf93/followers","following_url":"https://api.github.com/users/bwf93/following{/other_user}","gists_url":"https://api.github.com/users/bwf93/gists{/gist_id}","starred_url":"https://api.github.com/users/bwf93/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bwf93/subscriptions","organizations_url":"https://api.github.com/users/bwf93/orgs","repos_url":"https://api.github.com/users/bwf93/repos","events_url":"https://api.github.com/users/bwf93/events{/privacy}","received_events_url":"https://api.github.com/users/bwf93/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/:Rest","name":":Rest","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/v5.0.0-rc2","name":"v5.0.0-rc2","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-10-12T14:44:32Z","updated_at":"2016-10-12T17:09:30Z","closed_at":null,"body":"<!--\r\nGitHub is reserved *only* for bug reports and feature requests. The best place\r\nto ask a general question is at the Elastic forums at\r\nhttps://discuss.elastic.co/c/elasticsearch-and-hadoop.\r\nIf you are in fact posting a bug report or a feature request, please \r\ninclude one and only one of the below blocks in your new issue.\r\n-->\r\n\r\n### What kind an issue is this?\r\n\r\n- [X ] Bug report. If you’ve found a bug, please provide a code snippet or test to reproduce it below.   \r\n    The easier it is to track down the bug, the faster it is solved. \r\n- [ ] Feature Request. Start by telling us what problem you’re trying to solve.  \r\n    Often a solution already exists! Don’t send pull requests to implement new features without \r\n    first getting our support. Sometimes we leave features out on purpose to keep the project small.\r\n\r\n\r\n\r\n### Issue description\r\n\r\nDescription\r\n\r\n### Steps to reproduce\r\n\r\nCode:\r\n  val sparkConf: SparkConf = new SparkConf().setAppName(\"ES Test\")\r\n        sparkConf.set(\"es.read.field.include\", \"myFields\")\r\n    val sql = SparkSession.builder().config(sparkConf).getOrCreate()\r\n\r\n    val index = \"index\"\r\n\r\n    val dataSet = sql.esDF(s\"$index/myType\")\r\n\r\nTest/code snippet\r\nSee above\r\n\r\nStrack trace:\r\n\r\n2016-10-12 09:38:11,256 [Executor task launch worker-7] ERROR org.elasticsearch.hadoop.rest.NetworkClient - Node [127.0.\r\n0.1:9200] failed (Invalid target URI POST@null/index/myType/_search?sort=_doc&scroll=5m&size=50&_source=myFields&preference=_shards:3|_local); no other nodes left - aborting...\r\n2016-10-12 09:38:11,261 [Executor task launch worker-1] ERROR org.apache.spark.executor.Executor - Exception in task 1.0\r\n in stage 0.0 (TID 1)\r\norg.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all\r\nnodes failed; tried [[127.0.0.1:9200]]\r\n        at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:150)\r\n        at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:444)\r\n        at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:436)\r\n        at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:363)\r\n        at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:92)\r\n        at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:43)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\r\n        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\r\n        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:3\r\n70)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)\r\n        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)\r\n        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:85)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nStack trace goes here\r\n``` \r\n\r\n### Version Info\r\n\r\nOS:         :  Windows 7\r\nJVM         :  Oracle 8\r\nHadoop/Spark:  2.0 and 2.01\r\nES-Hadoop   :  5.0 RC1\r\nES          :  5.0 RC1\r\n\r\nThe same code works fine with ES-Hadoop 5.0 Beta1 and ES 5.0 Beta1\r\nI thought it might be the null in POST@null/index/myType/_search?sort=_doc&scroll=5m&size=50&_source=myFields&preference=_shards:3|_local)\r\n\r\nSo I tried setting es.nodes.path.prefix to \"\" but this did not fix it.  Since this code worked fine with the beta I think it is a bug in the RC.\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/253275950","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/868#issuecomment-253275950","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/868","id":253275950,"user":{"login":"bwf93","id":7559095,"avatar_url":"https://avatars.githubusercontent.com/u/7559095?v=3","gravatar_id":"","url":"https://api.github.com/users/bwf93","html_url":"https://github.com/bwf93","followers_url":"https://api.github.com/users/bwf93/followers","following_url":"https://api.github.com/users/bwf93/following{/other_user}","gists_url":"https://api.github.com/users/bwf93/gists{/gist_id}","starred_url":"https://api.github.com/users/bwf93/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bwf93/subscriptions","organizations_url":"https://api.github.com/users/bwf93/orgs","repos_url":"https://api.github.com/users/bwf93/repos","events_url":"https://api.github.com/users/bwf93/events{/privacy}","received_events_url":"https://api.github.com/users/bwf93/received_events","type":"User","site_admin":false},"created_at":"2016-10-12T17:09:30Z","updated_at":"2016-10-12T17:09:30Z","body":"curl -XGET localhost:9200/_nodes/http\r\n`{\r\n  \"_nodes\": {\r\n    \"total\": 1,\r\n    \"successful\": 1,\r\n    \"failed\": 0\r\n  },\r\n  \"cluster_name\": \"elasticsearch\",\r\n  \"nodes\": {\r\n    \"wHolF84uTMu0a9WWBUiAjQ\": {\r\n      \"name\": \"wHolF84\",\r\n      \"transport_address\": \"127.0.0.1:9300\",\r\n      \"host\": \"127.0.0.1\",\r\n      \"ip\": \"127.0.0.1\",\r\n      \"version\": \"5.0.0-rc1\",\r\n      \"build_hash\": \"13e62e1\",\r\n      \"roles\": [\r\n        \"master\",\r\n        \"data\",\r\n        \"ingest\"\r\n      ],\r\n      \"http\": {\r\n        \"bound_address\": [\r\n          \"127.0.0.1:9200\",\r\n          \"[::1]:9200\"\r\n        ],\r\n        \"publish_address\": \"127.0.0.1:9200\",\r\n        \"max_content_length_in_bytes\": 104857600\r\n      }\r\n    }\r\n  }\r\n}`\r\ncurl -XGET localhost:9200/$YOUR_INDEX_NAME/_search_shards\r\n`{\r\n  \"nodes\": {\r\n    \"wHolF84uTMu0a9WWBUiAjQ\": {\r\n      \"name\": \"wHolF84\",\r\n      \"ephemeral_id\": \"zbaQSSt3QTuIscgiM1HoFA\",\r\n      \"transport_address\": \"127.0.0.1:9300\",\r\n      \"attributes\": {}\r\n    }\r\n  },\r\n  \"shards\": [\r\n    [\r\n      {\r\n        \"state\": \"STARTED\",\r\n        \"primary\": true,\r\n        \"node\": \"wHolF84uTMu0a9WWBUiAjQ\",\r\n        \"relocating_node\": null,\r\n        \"shard\": 0,\r\n        \"index\": \"documents1000000\",\r\n        \"allocation_id\": {\r\n          \"id\": \"TC3CkecPQLigJUrzfLSmhQ\"\r\n        }\r\n      }\r\n    ],\r\n    [\r\n      {\r\n        \"state\": \"STARTED\",\r\n        \"primary\": true,\r\n        \"node\": \"wHolF84uTMu0a9WWBUiAjQ\",\r\n        \"relocating_node\": null,\r\n        \"shard\": 1,\r\n        \"index\": \"documents1000000\",\r\n        \"allocation_id\": {\r\n          \"id\": \"gCideKYVSoC5Cf8zlGO1gA\"\r\n        }\r\n      }\r\n    ],\r\n    [\r\n      {\r\n        \"state\": \"STARTED\",\r\n        \"primary\": true,\r\n        \"node\": \"wHolF84uTMu0a9WWBUiAjQ\",\r\n        \"relocating_node\": null,\r\n        \"shard\": 2,\r\n        \"index\": \"documents1000000\",\r\n        \"allocation_id\": {\r\n          \"id\": \"9NzhSOv4TxaUtb8fz2OTzg\"\r\n        }\r\n      }\r\n    ],\r\n    [\r\n      {\r\n        \"state\": \"STARTED\",\r\n        \"primary\": true,\r\n        \"node\": \"wHolF84uTMu0a9WWBUiAjQ\",\r\n        \"relocating_node\": null,\r\n        \"shard\": 3,\r\n        \"index\": \"documents1000000\",\r\n        \"allocation_id\": {\r\n          \"id\": \"IOKXUrrbTve46JWfRRuaVA\"\r\n        }\r\n      }\r\n    ],\r\n    [\r\n      {\r\n        \"state\": \"STARTED\",\r\n        \"primary\": true,\r\n        \"node\": \"wHolF84uTMu0a9WWBUiAjQ\",\r\n        \"relocating_node\": null,\r\n        \"shard\": 4,\r\n        \"index\": \"documents1000000\",\r\n        \"allocation_id\": {\r\n          \"id\": \"atnOXqm4REmYwbceTyJwoQ\"\r\n        }\r\n      }\r\n    ],\r\n    [\r\n      {\r\n        \"state\": \"STARTED\",\r\n        \"primary\": true,\r\n        \"node\": \"wHolF84uTMu0a9WWBUiAjQ\",\r\n        \"relocating_node\": null,\r\n        \"shard\": 5,\r\n        \"index\": \"documents1000000\",\r\n        \"allocation_id\": {\r\n          \"id\": \"1JWnUMzxRbyxXjaWAEE94g\"\r\n        }\r\n      }\r\n    ],\r\n    [\r\n      {\r\n        \"state\": \"STARTED\",\r\n        \"primary\": true,\r\n        \"node\": \"wHolF84uTMu0a9WWBUiAjQ\",\r\n        \"relocating_node\": null,\r\n        \"shard\": 6,\r\n        \"index\": \"documents1000000\",\r\n        \"allocation_id\": {\r\n          \"id\": \"tel2hAGUQBmFxxUXlJrM_A\"\r\n        }\r\n      }\r\n    ],\r\n    [\r\n      {\r\n        \"state\": \"STARTED\",\r\n        \"primary\": true,\r\n        \"node\": \"wHolF84uTMu0a9WWBUiAjQ\",\r\n        \"relocating_node\": null,\r\n        \"shard\": 7,\r\n        \"index\": \"documents1000000\",\r\n        \"allocation_id\": {\r\n          \"id\": \"rHfFwqLkTsWwbsX9dku7Fw\"\r\n        }\r\n      }\r\n    ]\r\n  ]\r\n}`"}}
{"actor":{"display_login":"bwf93"}}