title: PySpark: MutableLong cannot be cast to MutableInt (no long in dataframe)
body: <p>I'm trying to read a <code>profiles</code> table from Athena in PySpark using Glue client from boto3 and checking if it's empty. Why Spark bug on converting Int to Long, knowing that I do not have Long type in the table read? There is nothing on Google, nor on StackOverflow that answers this problem.</p>
  <p>Here is a code sum-up:</p>
  <pre class="lang-py prettyprint-override"><code>dataframe = GlueContext(session.sparkContext).create_dynamic_frame.from_catalog(
                  database=&quot;xxx&quot;,
                  table_name=&quot;profiles&quot;,
                  catalog_id=&quot;xxx&quot;
              ).toDF()
  if dataframe.rdd.isEmpty():
       dataframe = session.sparkContext.emptyRDD().toDF(schema)
  </code></pre>
  <p>I'm getting the error:</p>
  <pre><code>ERROR GlueExceptionAnalysisListener: [Glue Exception Analysis] Event: GlueETLJobExceptionEvent
  [...]
    File &quot;/myScript.py&quot;, line 246, in load_table
     if dataframe.rdd.isEmpty()
  [...]
  Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 0 in file s3://bucket/path/to/profiles/vault=c27/subgroup=1/part-00003-a97d95f5-713c-4756-808b-38c3866842cb.c000.snappy.parquet
  [...]
  Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.MutableLong cannot be cast to org.apache.spark.sql.catalyst.expressions.MutableInt
  </code></pre>
  <p>Here is the Athena DDL:</p>
  <pre class="lang-sql prettyprint-override"><code>CREATE EXTERNAL TABLE `profiles`(
    `id` string, 
    `anonymousids` array&lt;string&gt;, 
    `lastconsentinsightusage` boolean, 
    `lastconsentactivationusage` boolean, 
    `gender` string, 
    `age` int, 
    `iata` string, 
    `continent` string, 
    `country` string, 
    `city` string, 
    `state` string, 
    `brandvisit` int, 
    `knownprofileinmarket` date, 
    `devicebrowser` string, 
    `devicebrand` string, 
    `deviceos` string, 
    `returnedinmarket` date)
  PARTITIONED BY ( 
    `vault` varchar(5), 
    `subgroup` int)
  ROW FORMAT SERDE 
    'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' 
  STORED AS INPUTFORMAT 
    'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' 
  OUTPUTFORMAT 
    'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
  LOCATION
    's3://bucket/path/to/profiles'
  TBLPROPERTIES (
    'classification'='parquet', 
    'transient_lastDdlTime'='1645604528')
  </code></pre>
  <p>And here is the parquet schema:</p>
  <pre class="lang-json prettyprint-override"><code>{
    &quot;type&quot; : &quot;record&quot;,
    &quot;name&quot; : &quot;spark_schema&quot;,
    &quot;fields&quot; : [ {
      &quot;name&quot; : &quot;Id&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;anonymousIds&quot;,
      &quot;type&quot; : [ &quot;null&quot;, {
        &quot;type&quot; : &quot;array&quot;,
        &quot;items&quot; : {
          &quot;type&quot; : &quot;record&quot;,
          &quot;name&quot; : &quot;list&quot;,
          &quot;fields&quot; : [ {
            &quot;name&quot; : &quot;element&quot;,
            &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
            &quot;default&quot; : null
          } ]
        }
      } ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;lastConsentInsightUsage&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;boolean&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;lastConsentActivationUsage&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;boolean&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;gender&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;age&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;int&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;iata&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;continent&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;country&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;city&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;state&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;brandVisit&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;int&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;knownProfileInMarket&quot;,
      &quot;type&quot; : [ &quot;null&quot;, {
        &quot;type&quot; : &quot;int&quot;,
        &quot;logicalType&quot; : &quot;date&quot;
      } ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;deviceBrowser&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;deviceBrand&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;deviceOs&quot;,
      &quot;type&quot; : [ &quot;null&quot;, &quot;string&quot; ],
      &quot;default&quot; : null
    }, {
      &quot;name&quot; : &quot;returnedInMarket&quot;,
      &quot;type&quot; : [ &quot;null&quot;, {
        &quot;type&quot; : &quot;int&quot;,
        &quot;logicalType&quot; : &quot;date&quot;
      } ],
      &quot;default&quot; : null
    } ]
  }
  </code></pre>
  <p>And a line of the parquet file:</p>
  <pre class="lang-json prettyprint-override"><code>{&quot;Id&quot;: &quot;34e9bbcd3dd577d6bc3f9b82d9dd99666dafa0203486d2a604f59b7702d50d7d&quot;, &quot;anonymousIds&quot;: [{&quot;element&quot;: &quot;5510&quot;}], &quot;lastConsentInsightUsage&quot;: true, &quot;lastConsentActivationUsage&quot;: true, &quot;gender&quot;: &quot;F&quot;, &quot;age&quot;: 40, &quot;iata&quot;: &quot;&quot;, &quot;continent&quot;: &quot;EU&quot;, &quot;country&quot;: &quot;DE&quot;, &quot;city&quot;: &quot;Frankfurt (Oder)&quot;, &quot;state&quot;: &quot;BB&quot;, &quot;brandVisit&quot;: 9, &quot;knownProfileInMarket&quot;: 18765, &quot;deviceBrowser&quot;: &quot;Googlebot&quot;, &quot;deviceBrand&quot;: &quot;Spider&quot;, &quot;deviceOs&quot;: &quot;Other&quot;, &quot;returnedInMarket&quot;: 18765}
  </code></pre>