title: Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the referenced columns only include the internal corrupt record column
body: <p>I have a json file:</p>
  <pre class="lang-json prettyprint-override"><code>{
    &quot;a&quot;: {
      &quot;b&quot;: 1
    }
  }
  </code></pre>
  <p>I am trying to read it:</p>
  <pre><code>val path = &quot;D:/playground/input.json&quot;
  val df = spark.read.json(path)
  df.show()
  </code></pre>
  <p>But getting an error:</p>
  <blockquote>
  <p>Exception in thread &quot;main&quot; org.apache.spark.sql.AnalysisException:
  Since Spark 2.3, the queries from raw JSON/CSV files are disallowed
  when the referenced columns only include the internal corrupt record
  column (named _corrupt_record by default). For example:
  spark.read.schema(schema).json(file).filter($&quot;_corrupt_record&quot;.isNotNull).count()
  and
  spark.read.schema(schema).json(file).select(&quot;_corrupt_record&quot;).show().
  Instead, you can cache or save the parsed results and then send the
  same query. For example, val df =
  spark.read.schema(schema).json(file).cache() and then
  df.filter($&quot;_corrupt_record&quot;.isNotNull).count().;</p>
  </blockquote>
  <p>So I tried to cache it as they suggest:</p>
  <pre><code>val path = &quot;D:/playground/input.json&quot;
  val df = spark.read.json(path).cache()
  df.show()
  </code></pre>
  <p>But I keep getting the same error.</p>