{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/1201","repository_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/1201/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/1201/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/1201/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/1201","id":374273531,"node_id":"MDU6SXNzdWUzNzQyNzM1MzE=","number":1201,"title":"Defined date type field value can't parse in ES6.2.4","user":{"login":"478632418","id":12960283,"node_id":"MDQ6VXNlcjEyOTYwMjgz","avatar_url":"https://avatars2.githubusercontent.com/u/12960283?v=4","gravatar_id":"","url":"https://api.github.com/users/478632418","html_url":"https://github.com/478632418","followers_url":"https://api.github.com/users/478632418/followers","following_url":"https://api.github.com/users/478632418/following{/other_user}","gists_url":"https://api.github.com/users/478632418/gists{/gist_id}","starred_url":"https://api.github.com/users/478632418/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/478632418/subscriptions","organizations_url":"https://api.github.com/users/478632418/orgs","repos_url":"https://api.github.com/users/478632418/repos","events_url":"https://api.github.com/users/478632418/events{/privacy}","received_events_url":"https://api.github.com/users/478632418/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-10-26T08:12:49Z","updated_at":"2018-10-26T08:56:31Z","closed_at":null,"author_association":"NONE","body":"### Issue description\r\n- The date type field defined in ElastricSearch6.2.4 as \r\n`insert_time\":{\"type\":\"date\",\"format\":\"yyyy-MM-dd HH:mm:ss\"},\r\n\r\n### Steps to reproduce\r\nCode:\r\n```\r\nTest/code snippet\r\n```\r\n-But when I run the code \r\n`spark.read().format(\"org.elasticsearch.spark.sql\") //\r\n\t\t\t\t.option(\"es.nodes\", \"my elastricsearch ip\")//\r\n\t\t\t\t.option(\"es.port\", 9200)//\r\n\t\t\t\t.option(\"pushdown\", \"true\")//\r\n\t\t\t\t.option(\"es.index.auto.create\", \"true\")//\r\n\t\t\t\t.option(\"es.nodes.wan.only\", \"true\")//\r\n\t\t\t\t.option(\"es.query\", \"{\\\"query\\\":{\\\"term\\\":{\\\"type\\\":\\\"my row type\\\"}}}\")//\r\n\t\t\t\t.load(\"/myindex/mytype/\");`\r\nOne error was thrown:\r\nStrack trace:\r\n```\r\nStack trace goes here\r\n``` \r\n`18/10/26 15:34:33 ERROR scheduler.TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job\r\nException in thread \"main\" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, CDH-147, executor 1): org.elasticsearch.hadoop.rest.EsHadoopParsingException: Cannot parse value [2018-10-26 11:30:03] for field [insert_time]\r\n        at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:727)\r\n        at org.elasticsearch.hadoop.serialization.ScrollReader.map(ScrollReader.java:871)\r\n        at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:714)\r\n        at org.elasticsearch.hadoop.serialization.ScrollReader.readHitAsMap(ScrollReader.java:480)\r\n        at org.elasticsearch.hadoop.serialization.ScrollReader.readHit(ScrollReader.java:405)\r\n        at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:300)\r\n        at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:273)\r\n        at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:301)\r\n        at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:92)\r\n        at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:61)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\r\n        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\r\n        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)\r\n        at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)\r\n        at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)\r\n        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)\r\n        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)\r\n        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:108)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n        at java.lang.Thread.run(Thread.java:748)\r\nCaused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot invoke method public org.joda.time.DateTime org.joda.time.format.DateTimeFormatter.parseDateTime(java.lang.String)\r\n        at org.elasticsearch.hadoop.util.ReflectionUtils.invoke(ReflectionUtils.java:93)\r\n        at org.elasticsearch.hadoop.util.DateUtils$JodaTime.parseDate(DateUtils.java:105)\r\n        at org.elasticsearch.hadoop.util.DateUtils.parseDate(DateUtils.java:122)\r\n        at org.elasticsearch.spark.serialization.ScalaValueReader.createDate(ScalaValueReader.scala:182)\r\n        at org.elasticsearch.spark.serialization.ScalaValueReader.parseDate(ScalaValueReader.scala:173)\r\n        at org.elasticsearch.spark.serialization.ScalaValueReader$$anonfun$date$1.apply(ScalaValueReader.scala:166)\r\n        at org.elasticsearch.spark.serialization.ScalaValueReader$$anonfun$date$1.apply(ScalaValueReader.scala:166)\r\n        at org.elasticsearch.spark.serialization.ScalaValueReader.checkNull(ScalaValueReader.scala:120)\r\n        at org.elasticsearch.spark.serialization.ScalaValueReader.date(ScalaValueReader.scala:166)\r\n        at org.elasticsearch.spark.serialization.ScalaValueReader.readValue(ScalaValueReader.scala:96)\r\n        at org.elasticsearch.spark.sql.ScalaRowValueReader.readValue(ScalaEsRowValueReader.scala:46)\r\n        at org.elasticsearch.hadoop.serialization.ScrollReader.parseValue(ScrollReader.java:774)\r\n        at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:724)\r\n        ... 26 more\r\nCaused by: java.lang.reflect.InvocationTargetException\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at org.elasticsearch.hadoop.util.ReflectionUtils.invoke(ReflectionUtils.java:91)\r\n        ... 38 more\r\nCaused by: java.lang.IllegalArgumentException: Invalid format: \"2018-10-26 11:30:03\" is malformed at \" 11:30:03\"\r\n        at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945)\r\n        ... 43 more\r\n`\r\n\r\n\r\n\r\n### Version Info\r\n\r\nOS:         :  CentOS6.5+\r\nJVM         :  JDK8\r\nHadoop/Spark:  Spark2.2.0\r\nES-Hadoop   :  6.2.4+6.4.2\r\nES          :  6.2.4\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/433337892","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/1201#issuecomment-433337892","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/1201","id":433337892,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzMzNzg5Mg==","user":{"login":"478632418","id":12960283,"node_id":"MDQ6VXNlcjEyOTYwMjgz","avatar_url":"https://avatars2.githubusercontent.com/u/12960283?v=4","gravatar_id":"","url":"https://api.github.com/users/478632418","html_url":"https://github.com/478632418","followers_url":"https://api.github.com/users/478632418/followers","following_url":"https://api.github.com/users/478632418/following{/other_user}","gists_url":"https://api.github.com/users/478632418/gists{/gist_id}","starred_url":"https://api.github.com/users/478632418/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/478632418/subscriptions","organizations_url":"https://api.github.com/users/478632418/orgs","repos_url":"https://api.github.com/users/478632418/repos","events_url":"https://api.github.com/users/478632418/events{/privacy}","received_events_url":"https://api.github.com/users/478632418/received_events","type":"User","site_admin":false},"created_at":"2018-10-26T08:56:31Z","updated_at":"2018-10-26T08:56:31Z","author_association":"NONE","body":"Or the other way，See https://blog.csdn.net/ltliyue/article/details/80021058?utm_source=blogxgwz6"}}
{"actor":{"display_login":"478632418"}}