title: java.lang.IllegalArgumentException: Failed to parse query: {"query":
body: <p>I am trying to execute the Elasticsearch DSL query in Spark 2.2 and Scala 2.11.8. The version of Elasticsearch is 2.4.4, while the version of Kibana that I use is 4.6.4.</p>
  <p>This is the library that I use in Spark:</p>
  <pre><code>&lt;dependency&gt;
      &lt;groupId&gt;org.elasticsearch&lt;/groupId&gt;
      &lt;artifactId&gt;elasticsearch-spark-20_2.11&lt;/artifactId&gt;
      &lt;version&gt;5.2.2&lt;/version&gt;
  &lt;/dependency&gt;
  </code></pre>
  <p>This is my code:</p>
  <pre><code>val spark = SparkSession.builder()
        .config(&quot;es.nodes&quot;,&quot;localhost&quot;)
        .config(&quot;es.port&quot;,9200)
        .config(&quot;es.nodes.wan.only&quot;,&quot;true&quot;)
        .config(&quot;es.index.auto.create&quot;,&quot;true&quot;)
        .appName(&quot;ES test&quot;)
        .master(&quot;local[*]&quot;)
        .getOrCreate()
  
      val myquery = &quot;&quot;&quot;{&quot;query&quot;:
                            {&quot;bool&quot;: {
                               &quot;must&quot;: [
                                  {
                                     &quot;has_child&quot;: {
                                         &quot;filter&quot;: {
                                            ...
                                         }
                                      }
                                  }
                               ]
                            }
                        }}&quot;&quot;&quot;
  
    
     val df = spark.read.format(&quot;org.elasticsearch.spark.sql&quot;)
        .option(&quot;query&quot;, myquery)
        .option(&quot;pushdown&quot;, &quot;true&quot;)
        .load(&quot;myindex/items&quot;)
  </code></pre>
  <p>I provided the main corpus of the DSL query. I am getting the error:</p>
  <blockquote>
  <p>java.lang.IllegalArgumentException: Failed to parse query: {&quot;query&quot;:</p>
  </blockquote>
  <p>Initially, I was thinking that the problem consists in the version of Elasticsearch. As far as I understand from this <a href="https://github.com/elastic/elasticsearch-hadoop" rel="nofollow noreferrer">GitHub</a>, the version 4 of Elasticsearch is not supported.</p>
  <p>However, if I run the same code with a simple query, it correctly retrieves records from Elasticsearch.</p>
  <pre><code>var df = spark.read
                .format(&quot;org.elasticsearch.spark.sql&quot;)
                .option(&quot;es.query&quot;, &quot;?q=public:*&quot;)
                .load(&quot;myindex/items&quot;)
  </code></pre>
  <p>Therefore I assume that the issue is not related to the version, but it's rather related to the way how I represent the query.</p>
  <p>This query works fine with cURL, but maybe it should be somehow updated before passing it to Spark?</p>
  <p><strong>Full error stacktrace:</strong></p>
  <pre><code>    Previous exception in task: Failed to parse query: {&quot;query&quot;:
                           {&quot;bool&quot;: {
                             &quot;must&quot;: [
                               {
                                 &quot;has_child&quot;: {
                                   &quot;filter&quot;: {
                                     &quot;bool&quot;: {
                                       &quot;must&quot;: [
                                         {
                                           &quot;term&quot;: {
                                             &quot;project&quot;: 579
                                           }
                                         },
                                         {
                                           &quot;terms&quot;: {
                                             &quot;status&quot;: [
                                               0,
                                               1,
                                               2
                                             ]
                                           }
                                         }
                                       ]
                                     }
                                   },
                                   &quot;type&quot;: &quot;status&quot;
                                 }
                               },
                               {
                                 &quot;has_child&quot;: {
                                   &quot;filter&quot;: {
                                     &quot;bool&quot;: {
                                       &quot;must&quot;: [
                                         {
                                           &quot;term&quot;: {
                                             &quot;project&quot;: 579
                                           }
                                         },
                                         {
                                           &quot;terms&quot;: {
                                             &quot;entity&quot;: [
                                               4634
                                             ]
                                           }
                                         }
                                       ]
                                     }
                                   },
                                   &quot;type&quot;: &quot;annotation&quot;
                                 }
                               },
                               {
                                 &quot;term&quot;: {
                                   &quot;project&quot;: 579
                                 }
                               },
                               {
                                 &quot;range&quot;: {
                                   &quot;publication_date&quot;: {
                                     &quot;gte&quot;: &quot;2017/01/01&quot;,
                                     &quot;lte&quot;: &quot;2017/04/01&quot;,
                                     &quot;format&quot;: &quot;yyyy/MM/dd&quot;
                                   }
                                 }
                               },
                               {
                                 &quot;bool&quot;: {
                                   &quot;should&quot;: [
                                     {
                                       &quot;terms&quot;: {
                                         &quot;typology&quot;: [
                                           &quot;news&quot;,
                                           &quot;blog&quot;,
                                           &quot;forum&quot;,
                                           &quot;socialnetwork&quot;
                                         ]
                                       }
                                     },
                                     {
                                       &quot;terms&quot;: {
                                         &quot;publishing_platform&quot;: [
                                           &quot;twitter&quot;
                                         ]
                                       }
                                     }
                                   ]
                                 }
                               }
                             ]
                           }}
      org.elasticsearch.hadoop.rest.query.QueryUtils.parseQuery(QueryUtils.java:59)
      org.elasticsearch.hadoop.rest.RestService.createReader(RestService.java:417)
      org.elasticsearch.spark.rdd.AbstractEsRDDIterator.reader$lzycompute(AbstractEsRDDIterator.scala:49)
      org.elasticsearch.spark.rdd.AbstractEsRDDIterator.reader(AbstractEsRDDIterator.scala:42)
      org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:61)
      scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
      org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
      org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
      org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
      org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
      org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
      org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
      org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
      org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      org.apache.spark.scheduler.Task.run(Task.scala:108)
      org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      java.lang.Thread.run(Thread.java:745)
      at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:138)
      at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
      at org.apache.spark.scheduler.Task.run(Task.scala:118)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  18/01/22 21:43:12 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): org.apache.spark.util.TaskCompletionListenerException: Failed to parse query: {&quot;query&quot;:
  </code></pre>
  <p>and this one:</p>
  <pre><code>8/01/22 21:47:40 WARN ScalaRowValueReader: Field 'cluster' is backed by an array but the associated Spark Schema does not reflect this;
                (use es.read.field.as.array.include/exclude) 
  18/01/22 21:47:40 WARN ScalaRowValueReader: Field 'project' is backed by an array but the associated Spark Schema does not reflect this;
                (use es.read.field.as.array.include/exclude) 
  18/01/22 21:47:40 WARN ScalaRowValueReader: Field 'client' is backed by an array but the associated Spark Schema does not reflect this;
                (use es.read.field.as.array.include/exclude) 
  
  18/01/22 21:47:40 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
  scala.MatchError: Buffer(13473953) (of class scala.collection.convert.Wrappers$JListWrapper)
      at org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:276)
      at org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:275)
      at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:103)
      at org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$2.apply(CatalystTypeConverters.scala:379)
      at org.apache.spark.sql.execution.RDDConversions$$anonfun$rowToRowRdd$1$$anonfun$apply$3.apply(ExistingRDD.scala:61)
      at org.apache.spark.sql.execution.RDDConversions$$anonfun$rowToRowRdd$1$$anonfun$apply$3.apply(ExistingRDD.scala:58)
      at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
      at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
      at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
      at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
      at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
      at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      at org.apache.spark.scheduler.Task.run(Task.scala:108)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  18/01/22 21:47:40 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): scala.MatchError: Buffer(13473953) (of class scala.collection.convert.Wrappers$JListWrapper)
  </code></pre>